wasc-satec@lists.webappsec.org

WASC Static Analysis Tool Evaluation Criteria

View all threads

Static Analysis Tool Evaluation Criteria (SATEC) project

SR
Srikanth Ramu
Sat, Jul 2, 2011 9:49 PM

Hello Everybody,

I am excited to be part of the SATEC project. Please find below my initial
draft, I would like to know your comments:

Categories:

  1.   Syntactic level
    
  2.   Semantic level
    
  3.   Supported Languages
    
  4.   Supported vulnerabilities
    
  5.   Support for library scanning
    
  6.   Configuration
    
  7.   Interface
    
  8.   Integration support
    
  9.   Platform support
    
  10. Reporting

  11. Performance

  12. Formal Method support

  13. Regression support

  14. Logging

  15. License type

  16. Documentation/Support

  17. Maturity of products

Features:

  1.   Syntactic level
    

a.      Analyze vulnerable syntax

b.      Compliance to standards like -
https://www.securecoding.cert.org/confluence/display/seccode/CERT+Secure+Coding+Standards

c.      Detection of unauthorized content (used for testing) like credit
card details

d.      Minimising false negatives and false positives

  1.   Semantic level
    

a.      Analyze the code flow

  1.   Supported Languages: Multi Language support
    

a.      C, C++, Java, Perl, Python, JavaScript, CSS

  1.   Supported vulnerabilities
    

a.      Types of vulnerabilities

b.      Identify standard vulnerabilities like –

                                                           i.

https://www.owasp.org/index.php/Top_10_2010-Main,

                                                         ii.

http://cwe.mitre.org/top25/

  1.   Support for library scanning:
    

a.      Support for scanning built-in libraries/binaries of language used

  1.   Configuration options:
    

a.      Support for loading new vulnerability database

b.      Specifying levels of severity to be scanned

c.      Option to ignore some vulnerabilities

  1.   Interface
    

a.      Command line

b.      Intuitive GUI

  1.   Integration support:
    

a.      Integration with existing IDE like Eclipse, NetBeans etc.

b.      Integration with Version control system to identify issues during
Check-In process

c.      Integration with issue/bug management system to raise issues

  1.   Platform Support
    

a.      Operating System: Windows, Linux, Mac

b.      Different hardware support: 32 bit, 64 bit processors

  1. Reporting:

a.      Support for HTML, text, XML based reports

b.      Reports with severity of the issue

c.      Email support for sending the reports

  1. Performance:

a.      Scanning of number of lines of code per second

  1. Formal Method support:

a.      Use of formal methods identifying the vulnerabilities (
http://en.wikipedia.org/wiki/Formal_methods)

  1. Regression support

a.      Run the tool in batch script and produce comparing with previous
testing

  1. Logging:  Good logging support

  2. License type

a.      Open source : GPL

b.      Commercial

  1. Documentation/Support

  2. Maturity of products

References:

http://projects.webappsec.org/w/page/13246983/WAFEC-1-HTML-Version

http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis

https://www.owasp.org/index.php/Source_Code_Analysis_Tools

https://www.owasp.org/index.php/Web_Application_Firewall

http://www.cert.org/secure-coding/tools.html

Regards,
Srikanth

Hello Everybody, I am excited to be part of the SATEC project. Please find below my initial draft, I would like to know your comments: *Categories:* 1. Syntactic level 2. Semantic level 3. Supported Languages 4. Supported vulnerabilities 5. Support for library scanning 6. Configuration 7. Interface 8. Integration support 9. Platform support 10. Reporting 11. Performance 12. Formal Method support 13. Regression support 14. Logging 15. License type 16. Documentation/Support 17. Maturity of products *Features:* 1. Syntactic level a. Analyze vulnerable syntax b. Compliance to standards like - https://www.securecoding.cert.org/confluence/display/seccode/CERT+Secure+Coding+Standards c. Detection of unauthorized content (used for testing) like credit card details d. Minimising false negatives and false positives 2. Semantic level a. Analyze the code flow 3. Supported Languages: Multi Language support a. C, C++, Java, Perl, Python, JavaScript, CSS 4. Supported vulnerabilities a. Types of vulnerabilities b. Identify standard vulnerabilities like – i. https://www.owasp.org/index.php/Top_10_2010-Main, ii. http://cwe.mitre.org/top25/ 5. Support for library scanning: a. Support for scanning built-in libraries/binaries of language used 6. Configuration options: a. Support for loading new vulnerability database b. Specifying levels of severity to be scanned c. Option to ignore some vulnerabilities 7. Interface a. Command line b. Intuitive GUI 8. Integration support: a. Integration with existing IDE like Eclipse, NetBeans etc. b. Integration with Version control system to identify issues during Check-In process c. Integration with issue/bug management system to raise issues 9. Platform Support a. Operating System: Windows, Linux, Mac b. Different hardware support: 32 bit, 64 bit processors 10. Reporting: a. Support for HTML, text, XML based reports b. Reports with severity of the issue c. Email support for sending the reports 11. Performance: a. Scanning of number of lines of code per second 12. Formal Method support: a. Use of formal methods identifying the vulnerabilities ( http://en.wikipedia.org/wiki/Formal_methods) 13. Regression support a. Run the tool in batch script and produce comparing with previous testing 14. Logging: Good logging support 15. License type a. Open source : GPL b. Commercial 16. Documentation/Support 17. Maturity of products *References:* http://projects.webappsec.org/w/page/13246983/WAFEC-1-HTML-Version http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis https://www.owasp.org/index.php/Source_Code_Analysis_Tools https://www.owasp.org/index.php/Web_Application_Firewall http://www.cert.org/secure-coding/tools.html Regards, Srikanth
AG
Andre Gironda
Sat, Jul 2, 2011 10:41 PM

On Sat, Jul 2, 2011 at 2:49 PM, Srikanth Ramu srikanthr@gmail.com wrote:

Hello Everybody,

I am excited to be part of the SATEC project. Please find below my initial
draft, I would like to know your comments:

Hi Srikanth,

Great work so far. Here is some of my initial input.

Features:

2.       Semantic level
a.       Analyze the code flow

It's important to directly address the specific types of external flow
sensitivity and slicing direction capabilities.

E.g. SAST Product A requires working data flow in order to analyze
control flow and allows data flow analysis from sources to sinks
SAST Product B works with data flow only but allows tracing
from both sources-to-sinks as well as sinks-to-sources

  1.   Supported Languages: Multi Language support
    

a.      C, C++, Java, Perl, Python, JavaScript, CSS

It would also be nice to speak to the point that some products are
focused on specific verticals e.g. financial, medical, aviation,
mobile apps, embedded systems, OS kernels, etc

4.       Supported vulnerabilities
a.       Types of vulnerabilities

I think you should check out the NSA CSA paper on Static Analysis
Tools in order to gain insight into the vulnerability categories that
they use. It's also my opinion that we should probably use the
terminology "software weaknesses" instead of "vulnerabilities" or
"vulnerability categories" when talking about these products.

5.       Support for library scanning:
a.       Support for scanning built-in libraries/binaries of language used

It is extremely important to speak about the base class library
support for mapped sources-passthrus-sinks (source-sink database), in
addition to support for common frameworks and popular third-party
components.

I found a large list of third-party components here --
http://smartbear.com/products/qa-tools/automated-testing/survey/

On Sat, Jul 2, 2011 at 2:49 PM, Srikanth Ramu <srikanthr@gmail.com> wrote: > Hello Everybody, > > I am excited to be part of the SATEC project. Please find below my initial > draft, I would like to know your comments: Hi Srikanth, Great work so far. Here is some of my initial input. > Features: > > 2.       Semantic level > a.       Analyze the code flow It's important to directly address the specific types of external flow sensitivity and slicing direction capabilities. E.g. SAST Product A requires working data flow in order to analyze control flow and allows data flow analysis from sources to sinks SAST Product B works with data flow only but allows tracing from both sources-to-sinks as well as sinks-to-sources > 3. Supported Languages: Multi Language support > a. C, C++, Java, Perl, Python, JavaScript, CSS It would also be nice to speak to the point that some products are focused on specific verticals e.g. financial, medical, aviation, mobile apps, embedded systems, OS kernels, etc > 4.       Supported vulnerabilities > a.       Types of vulnerabilities I think you should check out the NSA CSA paper on Static Analysis Tools in order to gain insight into the vulnerability categories that they use. It's also my opinion that we should probably use the terminology "software weaknesses" instead of "vulnerabilities" or "vulnerability categories" when talking about these products. > 5.       Support for library scanning: > a.       Support for scanning built-in libraries/binaries of language used It is extremely important to speak about the base class library support for mapped sources-passthrus-sinks (source-sink database), in addition to support for common frameworks and popular third-party components. I found a large list of third-party components here -- http://smartbear.com/products/qa-tools/automated-testing/survey/
SR
Srikanth Ramu
Mon, Jul 4, 2011 12:01 AM

Thank you Andre for the comments.

On Sat, Jul 2, 2011 at 3:41 PM, Andre Gironda andreg@gmail.com wrote:

On Sat, Jul 2, 2011 at 2:49 PM, Srikanth Ramu srikanthr@gmail.com wrote:

Hello Everybody,

I am excited to be part of the SATEC project. Please find below my

initial

draft, I would like to know your comments:

Hi Srikanth,

Great work so far. Here is some of my initial input.

Features:

  1.   Semantic level
    

a.      Analyze the code flow

It's important to directly address the specific types of external flow
sensitivity and slicing direction capabilities.

E.g. SAST Product A requires working data flow in order to analyze
control flow and allows data flow analysis from sources to sinks
SAST Product B works with data flow only but allows tracing
from both sources-to-sinks as well as sinks-to-sources

  1.   Supported Languages: Multi Language support
    

a.      C, C++, Java, Perl, Python, JavaScript, CSS

It would also be nice to speak to the point that some products are
focused on specific verticals e.g. financial, medical, aviation,
mobile apps, embedded systems, OS kernels, etc

  1.   Supported vulnerabilities
    

a.      Types of vulnerabilities

I think you should check out the NSA CSA paper on Static Analysis
Tools in order to gain insight into the vulnerability categories that
they use. It's also my opinion that we should probably use the
terminology "software weaknesses" instead of "vulnerabilities" or
"vulnerability categories" when talking about these products.

  1.   Support for library scanning:
    

a.      Support for scanning built-in libraries/binaries of language

used

It is extremely important to speak about the base class library
support for mapped sources-passthrus-sinks (source-sink database), in
addition to support for common frameworks and popular third-party
components.

I found a large list of third-party components here --
http://smartbear.com/products/qa-tools/automated-testing/survey/

Thank you Andre for the comments. On Sat, Jul 2, 2011 at 3:41 PM, Andre Gironda <andreg@gmail.com> wrote: > On Sat, Jul 2, 2011 at 2:49 PM, Srikanth Ramu <srikanthr@gmail.com> wrote: > > Hello Everybody, > > > > I am excited to be part of the SATEC project. Please find below my > initial > > draft, I would like to know your comments: > > Hi Srikanth, > > Great work so far. Here is some of my initial input. > > > Features: > > > > 2. Semantic level > > a. Analyze the code flow > > It's important to directly address the specific types of external flow > sensitivity and slicing direction capabilities. > > E.g. SAST Product A requires working data flow in order to analyze > control flow and allows data flow analysis from sources to sinks > SAST Product B works with data flow only but allows tracing > from both sources-to-sinks as well as sinks-to-sources > > > 3. Supported Languages: Multi Language support > > a. C, C++, Java, Perl, Python, JavaScript, CSS > > It would also be nice to speak to the point that some products are > focused on specific verticals e.g. financial, medical, aviation, > mobile apps, embedded systems, OS kernels, etc > > > 4. Supported vulnerabilities > > a. Types of vulnerabilities > > I think you should check out the NSA CSA paper on Static Analysis > Tools in order to gain insight into the vulnerability categories that > they use. It's also my opinion that we should probably use the > terminology "software weaknesses" instead of "vulnerabilities" or > "vulnerability categories" when talking about these products. > > > 5. Support for library scanning: > > a. Support for scanning built-in libraries/binaries of language > used > > It is extremely important to speak about the base class library > support for mapped sources-passthrus-sinks (source-sink database), in > addition to support for common frameworks and popular third-party > components. > > I found a large list of third-party components here -- > http://smartbear.com/products/qa-tools/automated-testing/survey/ >