[WEB SECURITY] SATE?

Okun, Vadim vadim.okun at nist.gov
Fri May 28 01:16:18 EDT 2010


We have been releasing the real deep data. There have been delays, but there are no sinister reasons for the delays.

The results of the 2nd SATE (our report and all data) will be released in June (We promised to release between February and May, but we're late with the report).

We released the results of the 1st SATE last summer: our report, the raw tool reports, and our analysis of the reports. The data is available (below the list of cautions) from
http://samate.nist.gov/SATE2008.html

or a direct link:
http://samate.nist.gov/SATE2008/resources/sate2008.tar.gz

I will answer some specific points in Jim's email below, but first, let me describe some limitations of SATE and how we are addressing them. SATE 2008 had a number of big limitations, including:

1) We analyzed a non-random subset of tool warnings
2) Determining correctness of tool warnings turned out to be more complicated than a binary true/false decision. Also, determining relevance of a warning to security turned out more difficult than we thought.
3) In most cases, we did not match warnings from different tools that refer to the same weakness. When we started SATE, we thought that we could match warnings by line number and weakness name or CWE id. In fact, most weaknesses are more complex - see Section 3.4 of our report.
4) Analysis criteria were applied inconsistently.

In our publicly released analysis, we used the confirmed/unconfirmed markings instead of true/false markings. We describe the reasons for this in our report - Section 4.2, page 29 of
http://samate.nist.gov/docs/NIST_Special_Publication_500-279.pdf

In SATE 2009, we made some improvements, including:

1) Randomly select a subset of tool warnings for analysis
2) We also looked at tool warnings that were related to human findings by security experts.
3) Use 4 categories for analysis of correctness: true, true but insignificant (for security), false, unknown.  It is an improvement, but there are still problems: for example distinguishing true from true but insignificant is often hard.

> 1) false positive rates from these tools are overwhelming

First, defining a false positive is tough.  Also in SATE 2008, the criteria that we used for analysis of correctness were inconsistent, we did not analyze a random sample of warnings, our analysis had errors. Steve gave a good example in his reply. We corrected some of these problems in 2009, but still way to go.

> 2) the work load to triage results from ONE of these tools were
> man-years

We are not the developers of the test cases, our knowledge of the test case code is very limited. Also, we used tools differently from their use in practice. We analyzed tool warnings for correctness and looked for related warnings from other tools, whereas developers use tools to determine what changes need to be made to software, auditors look for evidence of assurance.

> 3) by every possible measurement, manual review was more cost effective

As Steve said, SATE did not consider cost. In SATE 2009, we had security contractors analyze two of the test cases and report the most important security weaknesses. We then looked at tool warnings that report the same (or related) weakness. This will be released as part of 2009 release (The data set is too small for statistical conclusions.)

A big limitation of SATE has been the lack of ground truths about what security weaknesses really are in the test cases. This determination is hard for reasonably large software. We are trying to address this: manual analysis by security contractors, "CVE-selected" test cases.

> the NIST team chose only a small percentage of the automated findings to review

A small percentage by itself should not be a problem if the selection of tool warnings is done correctly (it was not done correctly in SATE 2008).

Vadim

________________________________________
From: Jim Manico [jim at manico.net]
Sent: Thursday, May 27, 2010 5:31 PM
To: 'Webappsec Group'
Subject: [WEB SECURITY] SATE?

I feel that NIST made a few errors in the first 2 SATE studies.

After the second round of SATE, the results were never fully released to
the public - even when NIST agreed to do just that at the inception of
the contest. I do not understand why SATE censored the final results - I
feel such censorship hurts the industry.

And even worse, I felt that vendor pressure encouraged NIST to not
release the final results. If the results (the real deep data, not the
executive summary that NIST release) were favorable to the tool vendors,
I bet they would have welcomed the release of the real data. But
instead, vendor pressure caused NIST to block the release of the final
data set.

The problems that the data would have revealed is:

1) false positive rates from these tools are overwhelming
2) the work load to triage results from ONE of these tools were man-years
3) by every possible measurement, manual review was more cost effective

Even worse were the methods around the process of this "study". For
example, all of the Java app's in this "study" contained poor hash
implementations. But because the tools (none of them) could see this,
that "finding" was completely ignored. The coverage was limited ONLY to
injection and data flow problems that tools have a chance of finding. In
fact, the NIST team chose only a small percentage of the automated
findings to review, since it would have taken years to review everything
due to the massive number of false positives. Get the problem here?

I'm discouraged by SATE. I hope some of these problems are addressed in
the third study.

- Jim

----------------------------------------------------------------------------
Join us on IRC: irc.freenode.net #webappsec

Have a question? Search The Web Security Mailing List Archives:
http://www.webappsec.org/lists/websecurity/archive/

Subscribe via RSS:
http://www.webappsec.org/rss/websecurity.rss [RSS Feed]

Join WASC on LinkedIn
http://www.linkedin.com/e/gis/83336/4B20E4374DBA


----------------------------------------------------------------------------
Join us on IRC: irc.freenode.net #webappsec

Have a question? Search The Web Security Mailing List Archives: 
http://www.webappsec.org/lists/websecurity/archive/

Subscribe via RSS: 
http://www.webappsec.org/rss/websecurity.rss [RSS Feed]

Join WASC on LinkedIn
http://www.linkedin.com/e/gis/83336/4B20E4374DBA



More information about the websecurity mailing list