3. Tool Coverage:
   Suggest making "Coverage of Industry Standard Vulnerability Categories
   (OWASP Top 10, SANS Top 25…etc) " a subgroup of "Support for
   benchmarking."

2.7 Industry Standards Aided Analysis (support for benchmarking)
Comment: vendors will make a claim, but quality of coverage may be
left up to the user to establish. Facilitation?

5. Reporting Capabilities
   Support for benchmarking reporting - Is the intent that reporting from
   2011 CWE/SANS Top 25 Most Dangerous Software Errors, OWASP Top 10,
   etc. be captured in template based support?

What about SCAP content support?

_jm

On 11/26/11, Sherif Koussa sherif.koussa@gmail.com wrote:

JM,

Please find my replies below

Regards,
Sherif
On Sun, Nov 27, 2011 at 3:23 PM, jm sysvar0@gmail.com wrote:

Are you suggesting to change "Industry Standard Aided Analysis" to
"Coverage of Industry Vulnerability Categories" or do you mean introduce a
new sub-category?

Absolutely, I mainly look at this project as informative to SCA clients
more than providing a benchmark for the tools. I like to look at it as what
are those important things that you should be looking at, ask the vendor
these questions, and then they make their own judgement according to their
own environment.

If the tool scans according to a certain criteria (2011 CWE/SANS 25, OWASP
10...etc) then the report should be able to reflect only that, no?

SCAP is not covered in this Phase unless most people think there is a value
in adding it?

Please find my replies below.

On Mon, Nov 28, 2011 at 9:12 AM, Alen Zukich alen.zukich@klocwork.comwrote:

So for example, let's say that I have non-typical file extensions in my
application, it is necessary to be able to tell the tool to configure the
tool to understand these extensions, for example, let's say I have JSPFs
(Java Server Pages Fragments) in my app, these are just JSP files, I would
want to re-configure the tool to treat them as so, now this is a trivial
example, some applications would create their own extensions, and it would
be important for the tool to understand what these files are otherwise they
will just be ignored.

So the distributed analysis and multi-core are supposed to be covered in
1.3, it might not show that but it will once it is fleshed out. For
analysis time, it would really depend on the combination of the following:
tool + application + environment. This is something that would be left for
the buyer to test for themselves

accuracy, how the tool achieves this, they don't really care.

That's an important one, I think we need to add this.

All,

5 more days available for voting on the second round. Please take a few
moments to look at the updated version of the categories and
sub-categories. Your opinion really does matter.

http://projects.webappsec.org/w/page/42093482/Static%20Analysis%20Tool%20Evaluation%20Criteria%20Working

Regards,
Sherif

On Sat, Nov 26, 2011 at 8:21 PM, Sherif Koussa sherif.koussa@gmail.comwrote:

On Mon, Dec 12, 2011 at 12:49:48PM -0500, Sherif Koussa wrote:

I think these categories are fine. Nothing to add. Are these something that absolutely can't be changed after writing criteria is on-going? Please contact me if you need any kind of help regarding this project. I am more than happy to help.

• Henri Salo

Changes after locking down the criteria would be very minimal but not
impossible.

The next step after locking down the criteria would be to flesh them out
and maybe create assisting tools (like WASSEC's excel sheet). After this is
done, probably and if there is a need we can revisit the criteria based on
feedback and open them again for changes.

Sherif

On Tue, Dec 13, 2011 at 2:36 PM, Henri Salo henri@nerv.fi wrote: