In reply to Dinis's blog post: The Need for Standards to evaluate Static Analysis Tools
1. You unfortunately list few types of SAST. Many of tools don't implement taint analysis -- if you go in the Ada/C/C++ world, you won't see much of taintbased analysis, but other technologies such as symbolic execution (Grammatech), abstract interpretation (Coverity, ASTREE, PolySpace, etc.), and more. A list of SAST can be found on the NIST SAMATE website: List of Source Code Security Analyzers
2. As said on twitter, concerningthe WASSEC, I don't believe it's important to have public evaluation ofcommercial/open-source tools. Also, WASSEC lists some vulnerabilities that the tool should look for, we don'tprovide test cases so it's not nearly possible to claim that a tool effectivelytest for a given problem, e.g. difference between two tools:
- Only test XSS with few payloads and does regexp matching of the rendered html
- A smarter engine that automagically crafts attacks and look at the resulting html with a JS engine (or so, that leads to fewer FP).
Depending on who you are and what you want, you might very well say thatthose two tools have the same support for XSS...
Moreover, tools are changing so quickly that an evaluation would only beaccurate at the time you make it.
3. NIST SATE is literally an exposition. NIST choose test cases (real open-source program that coversdifferent type of functionalities and technologies) and ask tool makers to runtheir SAST on those programs. The goal isn't to compare the tool to claim thatone is better than the other for a type of techno, but it's too see how tools(in general) performs, to see how many types of weaknesses the tools find andalso what is the overlap of tool findings (which resulted in a very littleamount of findings).
More generally, as Andrew said, a SAST isn't only an analysis engine thatfinds weaknesses in a program; it's a suite of functionalities:
- support technologies
- allows users to develop custom checks (or custom rules)
- displays the weaknesses to the user (allow to rank/prune and explainproblem) and reporting capabilities
Ultimately, every one of those elements are important and need to be tested,but again, the importance of those depend on who you are and how you want touse the SAST (from simple compliance type of scan to exhaustive securitytesting).
Just to tell you, NIST SAMATE (organizers of SATE) have been thinking a lot of those problem and there is noeasy solution for evaluating SAST... But the last SATE report explains some ofthe problems we (I was part of the SAMATE team at the time) faced: SATE 2008 -NIST Special Publication 500-279