[WEB SECURITY] SATE?

Tom Brennan tomb at proactiverisk.com
Thu May 27 17:57:18 EDT 2010


For anyone that wants to catch up here is a video from the OWASP Archive  http://video.google.com/videoplay?docid=-7567012344169452280&hl=en

On May 27, 2010, at 5:31 PM, Jim Manico wrote:

> I feel that NIST made a few errors in the first 2 SATE studies.
> 
> After the second round of SATE, the results were never fully released to the public - even when NIST agreed to do just that at the inception of the contest. I do not understand why SATE censored the final results - I feel such censorship hurts the industry.
> 
> And even worse, I felt that vendor pressure encouraged NIST to not release the final results. If the results (the real deep data, not the executive summary that NIST release) were favorable to the tool vendors, I bet they would have welcomed the release of the real data. But instead, vendor pressure caused NIST to block the release of the final data set.
> 
> The problems that the data would have revealed is:
> 
> 1) false positive rates from these tools are overwhelming
> 2) the work load to triage results from ONE of these tools were man-years
> 3) by every possible measurement, manual review was more cost effective
> 
> Even worse were the methods around the process of this "study". For example, all of the Java app's in this "study" contained poor hash implementations. But because the tools (none of them) could see this, that "finding" was completely ignored. The coverage was limited ONLY to injection and data flow problems that tools have a chance of finding. In fact, the NIST team chose only a small percentage of the automated findings to review, since it would have taken years to review everything due to the massive number of false positives. Get the problem here?
> 
> I'm discouraged by SATE. I hope some of these problems are addressed in the third study.
> 
> - Jim
> 
> ----------------------------------------------------------------------------
> Join us on IRC: irc.freenode.net #webappsec
> 
> Have a question? Search The Web Security Mailing List Archives: http://www.webappsec.org/lists/websecurity/archive/
> 
> Subscribe via RSS: http://www.webappsec.org/rss/websecurity.rss [RSS Feed]
> 
> Join WASC on LinkedIn
> http://www.linkedin.com/e/gis/83336/4B20E4374DBA
> 


----------------------------------------------------------------------------
Join us on IRC: irc.freenode.net #webappsec

Have a question? Search The Web Security Mailing List Archives: 
http://www.webappsec.org/lists/websecurity/archive/

Subscribe via RSS: 
http://www.webappsec.org/rss/websecurity.rss [RSS Feed]

Join WASC on LinkedIn
http://www.linkedin.com/e/gis/83336/4B20E4374DBA



More information about the websecurity mailing list