Independent Lab Tests; Maybe We Don’t Like Them Too Independent
I have been watching with some amusement this week as the furor over the NSS Labs Firewall report continued to grow. Now as my friend Bill Brenner details, both Fortinet and Sonicwall are griping about the findings of Rick Moy and his teams testing.
This is not that different than when NSS released their first IPS report a few years back. In that one many of the leading IPS players did a lousy job on real world type of attacks. One vendor (I think it was Tipping Point) actually failed to identify and block a large percentage of attacks. Again the vendors were up in arms. But you know what by the time NSS did a second IPS test about a year later many of these IPS vendors did a much better job. Maybe having the truth out there forced them to shape up and actually improve the product.
I had a chance to speak with Rick Moy the other day about the testing and results. Rick pretty much confirmed what I already knew. Very similar to the IPS test, they took these boxes into the lab and ran their tests. The results are what they found.
You may ask why is that different than any other independent tests or product reviews? Well therein lies issues. As someone who has been there, most of the lab tests and product reviews are anything but real world. Usually the vendor insists on having an engineer present during the test so he can tweak, tune and frankly do whatever it takes to make the box pass the test.
Another way these tests are usually skewed is the testing is not in real world scenarios. They may use large packets, but not small packet or vice-versa but not real world traffic. Other times exactly what kind of tests and attacks will be tested is agreed on before hand so that the vendor can tune up their box and make sure it handles everything the lab is going to throw at it before it is sent over.
Then on top of this, how many lab or reviews have you seen where no one fails anything, there are just varying degrees of great. No one wants to say a bad word about a vendors product.
Why would they? If they did the vendor may not want pay for the lab to test the product again or not take ads in the magazine or what have you.
That is why I like the NSS approach. I remember speaking about it with Rick years ago at RSA. NSS has turned the model on its side. Vendors don’t pay for the testing, the readers of the reports do. So now the vendor has no control over what the lab does or writes.
I think it is refreshing to get a truly independent test and view for a change. Many in the security industry have been calling for this type of thing for years. But it seems when it the test is on your product it is a case of “be careful what you wish for”