Application Firewall Vendors with something to prove?

InfoWorld has an article about the launch of the Application Security Consortium (no link, see note directly below), a group of application Firewall vendors who want to establish minimum requirements for what can be called an application Firewall.

Note: Amazingly the InfoWorld folks linked the Application Security Consortium to a profile for a New York based company, Application Security Incorporated (which seems to be about developing application test tools).

The members of this "consortium" are F5 Networks, Imperva, Netcontinuum, and Teros. In their start up announcment that coincided with this years CSI Conference in Washington D.C. the consortium issued a challenge to a number of other Firewall vendors (Symantec, McAfee, Juniper, Check Point) to submit their products for testing at ICSA Labs.

While this is an interesting idea it falls short of actually being useful. The Application Security Consortium has developed their own test criteria which has not been published. Where is the peer review here? They made the test criteria available to ICSA Labs and pay the Labs (or more accurately have the challenged vendors pay the Labs) to test other vendors products. This consortium should make their criteria freely availble for everyone to look at. They should define a means for other to challenge items in the criteria which don't make sense.

I was also bothered by the fact that this group has taken to calling themselves a "consortium". The ICSA Labs folks have developed a number of really good security industry consortiums (Anti Virus, Firewall, IPSec Interoperability consortiums to name a few) where vendors come to together with ICSA staff and discuss and agree on testing criteria that ICSA Labs later tests against. This isn't what has happened here.

Until this group gets their act(s) together this really isn't helping improve anyone's security.