The Ashimmy Blog

Independent Lab Tests; Maybe We Don’t Like Them Too Independent

I have been watching with some amusement this week as the furor over the NSS Labs Firewall report continued to grow.  Now as my friend Bill Brenner details, both Fortinet and Sonicwall are griping about the findings of Rick Moy and his teams testing.

This is not that different than when NSS released their first IPS report a few years back. In that one many of the leading IPS players did a lousy job on real world type of attacks.  One vendor (I think it was Tipping Point) actually failed to identify and block a large percentage of attacks. Again the vendors were up in arms.  But you know what by the time NSS did a second IPS test about a year later many of these IPS vendors did a much better job. Maybe having the truth out there forced them to shape up and actually improve the product.

I had a chance to speak with Rick Moy the other day about the testing and results. Rick pretty much confirmed what I already knew. Very similar to the IPS test, they took these boxes into the lab and ran their tests. The results are what they found.

You may ask why is that different than any other independent tests or product reviews? Well therein lies issues. As someone who has been there, most of the lab tests and product reviews are anything but real world.  Usually the vendor insists on having an engineer present during the test so he can tweak, tune and frankly do whatever it takes to make the box pass the test.

Another way these tests are usually skewed is the testing is not in real world scenarios. They may use large packets, but not small packet or vice-versa but not real world traffic. Other times exactly what kind of tests and attacks will be tested is agreed on before hand so that the vendor can tune up their box and make sure it handles everything the lab is going to throw at it before it is sent over.

Then on top of this, how many lab or reviews have you seen where no one fails anything, there are just varying degrees of great. No one wants to say a bad word about a vendors product.

Why would they? If they did the vendor may not want pay for the lab to test the product again or not take ads in the magazine or what have you.

That is why I like the NSS approach. I remember speaking about it with Rick years ago at RSA. NSS has turned the model on its side. Vendors don’t pay for the testing, the readers of the reports do. So now the vendor has no control over what the lab does or writes.

I think it is refreshing to get a truly independent test and view for a change. Many in the security industry have been calling for this type of thing for years. But it seems when it the test is on your product it is a case of “be careful what you wish for”

Related articles

• Firewall Vendors Challenge Findings of NSS Labs Report (pcworld.com)
• Firewall Security Issue Raised in Report Angers Vendors (pcworld.com)
• Study Finds Firewalls From Cisco, Fortinet, Others Vulnerable To Old Attack (blogs.forbes.com)
• NSS Labs Finds Most Firewalls Vulnerable to Attack (pcworld.com)
• Hacker 'handshake' hole found in common firewalls (networkworld.com)

Open Source IDS Wars: You Were There

Image via Wikipedia

Like Walter Cronkite, Mitchell and I play  “You Were There”.  With all of the hullaballoo lately around the open source IDS wars between Sourcefire and OSIF, I thought it would be cool to give people some historical perspective on this from two people who were there. Mitchell Ashley and I.  Back when we were helping to launch StillSecure we were one of the first commercial providers using Snort in our IPS and were one of the first sponsors of the Bleeding Snort / Bleeding Edge communities.

Sit back and relax as Mitchell and I reminisce about those days early in the open source security space.  I have also written a recap of this on my Network World blog here.

Okena twisting in its grave, as Cisco kills CSA

Well MARS is not the only Cisco security product to be killed off. Today we hear that Cisco is killing off its host based intrusion detection/prevention product, CSA off as well.  CSA is the old Okena Storm Watch product that Cisco paid I think 150million for back about 6 or 7 years.

CSA was always a bear to work with. By default everything was shut down and tuning it to allow most people to work reasonably was a chore. But on the other hand I had always heard that it was a quality HIDS/HIPS.

Cisco is apparently not replacing it with anything, telling customers to go find another company. Not a very customer friendly thing to do, but better then stringing them along I guess.

I guess this means Cisco is getting out of device based security and will concentrate on network based security. But then again who knows. Cisco’s security strategy has been muddled to say the least since Jayshree Ullal left IMHO.

What does this mean for Cisco NAC? If they can walk away from HIPS like this, what makes you think they won’t walk from NAC too. The NAC strategy there has been foggy at best. I would not be surprised to see them abandon it.

Related articles by Zemanta

• Cisco IT's Journey to Enterprise 2.0 and Beyond - #E2Conf-16 (slideshare.net)
• How Ciscos infighting put customers last and almost killed the NAC Market (ashimmy.com)

IPS vendors need to walk and run at the same time

Hot on the heels of yesterdays news about some dismal results in the NSS IPS labs in detecting attacks, we see today’s news about new capabilities in IPS. Now, one may ask what are the IPS guys doing adding all of this new stuff when the base detection doesn’t seem to work.  From what I know about the NSS testing, its not like they were trying to trick the IPS boxes. It was basically meat and potatoes, block and tackling type of attacks. Some of them were old and perhaps some of the IPS vendors had dropped detection of them. But in any event, clearly we would expect better detection of attacks from the existing technology.  I am sure the IPS vendors will be hard at work on this and those that are in the next round of NSS testing will do better.  It should also be noted that many IPS vendors opted out of the NSS testing. You can only assume that they did because their results would not have been much better.

But at the same time, it seems they are all busy adding new bells, whistles and features. First comes the “big” announcement from McAfee about a major refresh to their platform. It seems they have added a NBA to the mix. It is a separate box evidently.  They have also added the ability for their IPS to get real time threat information from their cloud based technology. Of course it all ties into ePO to mesh with their endpoint stuff.  It sounds interesting, combining all of this security technology but it is just a means into locking you into McAfee?  Will it result in detecting more real attacks in the next NSS test? I guess we will have to see.

Next up, Dark Reading’s Kelly Jackson Higgins has an article up about the virtualization gold rush in the IPS market.  Over at StillSecure we had our IPS running in a VMware image years ago. Sourcefire made a lot of noise about their gear running in a virtual machine a while ago too. What I found most interesting in Kelly’s article was the IBM/ISS disclosure about putting the IPS in the hypervisor and actually protecting the inter-VM.  That is different then just running an IPS in a VM environment versus a stand alone box. 

Tipping Point is even trying to get into the act according to the Dark Reading article. They claim that in their own boxes, they are actually virtualizing the IPS. Of course being so dependent on custom hardware making the move to mainstream VM is going to be hard for the TippingPoints and McAfee’s of the IPS world.

All in all though, the IPS vendors have a double barreled shotgun staring them in the face. They have to get better at the block and tackle stuff the NSS tests exposed, while keeping pace with an ever growing threat landscape and new technology!

More fallout from NSS IPS tests

The fallout continues from Rick Moy’s NSS Labs IPS tests.  Now Ellen Messmer in Network World has an article up about how miserable some of the IPS did in the test. Previously I had written that Tipping Point did not fare well in this test. Well according to the article, Juniper may have done even worse, detecting only 17% of the attacks.

While I think holding these devices to a 100% detection rate may be unrealistic, numbers below 50% and as low as 17% are pretty miserable. Either something was flawed about this test or many of today’s IPS are just not all that!

Gartner calls the Xie brothers Pixies and Fortinet a Unicorn

You have to give some points to Greg Young over at Gartner for boldness. Perhaps not since Rich Stiennon’s IDS is dead has a Gartner analyst taken such a bold and controversial stand.  Greg has called out the whole so called “enterprise UTM” space and by inference Fortinet as the leader of that bunch in this blog on the myth of enterprise UTM.  Greg seeks to debunk the 4 myths surrounding so called enterprise class UTM.

Most of it has to do with defining the enterprise. Frankly I think Greg doesn’t go far enough in his definition of enterprise. He seems to say that 1,000 employees is the magic number for a true enterprise. I think up to 5,000 employees is mid-market.  I define SMB as up to 500 workers.  501 to 5000 mid-markets.  While I agree with Greg that true enterprises will not be satisfied with UTM boxes, I do think that shops of 1,000 to 1,500 people can use a UTM type box and it will fill the bill.  I don’t think the latency that Greg talks about is what it was a few years ago.

Forget what I say though. It should be interesting to see what the Xie brothers (Fortinet co-founders) and some of the other enterprise UTM vendors have to say about what Greg has laid out.

Related articles by Zemanta

• Knock NAC, Knack for NAC, Who gives a NAC? (stillsecureafteralltheseyears.com)
• Fortinet plots rare IT security IPO (theregister.co.uk)

Is something rotten at TippingPoint?

Would you buy an IPS if its chances of detecting and stopping an attack were less than 50/50?  Probably not.  But that evidently is exactly the case with TippingPoint IPS according to the latest test results from NSS Labs. In this interview in TechWorld, Rick Moy, NSS President said that though it won the gold award 5 years ago, this latest round of tests showed a major slippage in coverage compared to other leading IPS providers.

What could it be that resulted in this lackluster result?  Could the fact that 3Com has exerted a stronger influence as a corporate parent led to a brain drain? Has TippingPoint concentrated too much on trying to make their NAC product competitive or integrate into 3Com switches. Maybe it is the H3C stuff in China. Frankly I don’t have any inside information. What I do know is that this could be a major crack in the armor of a leading IPS manufacturer. How much weight this is given and what the fall out will be remains to be seen, but usually where there is smoke there is fire.

Related articles by Zemanta

• Update: 3Com to blend security brains, enterprise brawn (infoworld.com)
• 3Com Switches Back to Selling Switches (blogs.wsj.com)

A busy Monday in the security-sphere

You can tell that Summer is starting to wind down (how did that happen, it was just the 4th of July). Today’s news was full of stuff that bear watching.  Watching the news as closely as I do it is easy to to see the rhythms and currents in the quality and quantity of the news.  Like the bears smelling the first snow of the season as a signal to hibernate, the pace of developments is telling me that we are heading into the back from vacation, lets get busy time of year.  So hope your vacations were fun, but better buckle down and get busy!

1. Fortinet to test the IPO waters. President Obama says the worst is behind us and the markets have been picking up.  Sounds like a great time to try a security based company IPO? Maybe not. The last few have not exactly blown the doors off of anyone, but maybe this time will be different. Fortinet certainly has the revenue to justify it, though I don’t think a 7.4 million profit on revenue over 200 million in 2008 is a great margin. Of course the question is what does 2009 look like for them. Has the recession hit them hard? Have the continued to grow? In any event, this may nudge along other less qualified security and technology companies to put their toes in the IPO waters. With valuations in money raising and M&A so low, something has to give to change the equations.  Good luck to the folks at Fortinet!
2. Marrying NAC to IPS? Why didn’t I think of that? Wait a second I did! Sourcefire and Bradford put out a press release detailing integration between Sourcefire’s IPS and 3D system and the Bradford NAC product. Funny I remember Bradford announcing this same functionality a year or two ago with Snort and then Top Layer and then Nitro. I don’t know this sounds too much like a me too Barney release for my tastes. But Bradford has been silent for a while, so I guess this means they are still in business.  The functionality addressed is actually very similar to how StillSecure’s Safe Access NAC product works with our Strata Guard IDS/IPS system. Safe Access can use Strata Guard as a dedicated post-connect NAC sensor or as a stand alone IDS/IPS with added functionality. StillSecure has been offering this type of functionality for years already, as has Bradford for that matter. This would have been a good announcement in 2007 but today, not so much guys.
3. Lifetime warranties, the new black in networking gear – Extreme Networks announced this a while back but just came across this article in AMEInfo detailing it. Extreme now joins HP ProCurve and Cisco is offering some type of lifetime warranty.  HP ProCurve started this trend but it looks like it is becoming the de facto standard in the network gear space.  I think it is great if there is not a lot fine print that renders them useless.
4. Seeding the clouds? Terremark which is supposed to be growing pretty rapidly (at least my friend Pete Nicoletti says so), reported a run rate of 2.5 million dollars a year in this article on MSPMentor. While more than 2 million dollars is a lot of money (I could use that right now), as an indicator of big things to come, it just doesn’t get my juices flowing.  I would expect more revenue from cloud services than that. Hell, I would like to see more from cloud security alone than that.  BTW, I am doing a podcast on cloud services and cloud security very soon, so stay tuned!

Anyway that is all I have for today.  Have a great day!

Is StillSecure really secure? You betcha!

This is in response to a blog from a William Lee blog article the other day. I don’t know William and I am sure he has the best of intentions. Anyway William wanted to see if our Strata Guard Lite product really worked. He tried to set it up on a virtual machine on Windows and ran some Metasploit exploits at it. He claims that Strata Guard didn’t even see the attack and let it through.

When I first read William’'s post the other day a few things jumped right out at me that screamed here is a mistake waiting to happen.

1. He was running a 3 way VM set up with virtual switching. While that in and of itself is not indicative of a trouble, if you don’t know the finer points of this, it could be trouble. Also running virtual desktops is a lot different then virtual servers. If William made any configuration mistakes, it would prevent Strata Guard from seeing the traffic.

2. He boasts about turning all 6000+ rules onto block. That doesn’t sound like a very experienced IDS/IPS strategy.

Anyway, I didn’t want to jump to conclusions. I asked our security engineers to have a look.  I am happy to report that recreating William’s set up, Strata Guard can of course detect the exploit William references. In fact there is something like 49 rules dealing with that particular exploit.  The problem is in the VM set up.  William does not have SG seeing the traffic! So in answer to William’s question: Is StillSecure really secure? Set up right and you tell me.

Reputation based service is great for stopping spam email, but does it help against targeted attacks?

I saw an interesting Google Alert in my mailbox today. Titled “Cybercriminals In The Cloud”, I thought it was going to be about how cybercriminals were now using cloud services to access confidential information.  So I bit and found myself at a “welcome screen” from Forbes.  Why call it a welcome screen, lets call it an ad page as that is the only thing on there and you can watch the ad or click thru. After clicking through I was greeted by an article by Charlotte Dunlop (nice picture Charlotte).  The gist of the article was that the big thing at RSA was how top tier security vendors were going to use reputation services to make IDS/IPS, UTM and other technologies better able to stop the new more sophisticated attacks that CIOs are dealing with.

I say poppycock! Yes reputation services in the cloud are great for picking up IPs that have been used as spam homes or spewing other malicious content, but in targeted attacks cybercriminals are smart enough to use fresh IPs, not ones that are already tainted.  If these bad guys are smart enough to devices the techniques they do to break in, lets not be naive enough to think that they are going to then go out and use the same old IP addresses to launch new targeted attacks.  Reputation type of defenses are great against mass market type activity, but for targeted exploits that CIOs reading Forbes are worried about I don’t think it offers much hope.  Sounds to me more like yet another person bought the security in the cloud story hook, line and sinker.

Related articles by Zemanta

• Clearing The Confusion On Security And Clouds (cloudave.com)
• Telcos Will Be a Force in Cloud Computing. (cloudave.com)
• Federal policies could make or break cloud computing, experts say (news.cnet.com)