deep inside: security and tools

Yes, we need a standard to evaluate SAST, but it ain't easy...

In reply to Dinis's blog post: The Need for Standards to evaluate Static Analysis Tools

​1. You unfortunately list few types of SAST. Many of tools don't implement taint analysis -- if you go in the Ada/C/C++ world, you won't see much of taintbased analysis, but other technologies such as symbolic execution (Grammatech), abstract interpretation (Coverity, ASTREE, PolySpace, etc.), and more. A list of SAST can be found on the NIST SAMATE website: List of Source Code Security Analyzers

​2. As said on twitter, concerningthe WASSEC, I don't believe it's important to have public evaluation ofcommercial/open-source tools. Also, WASSEC lists some vulnerabilities that the tool should look for, we don'tprovide test cases so it's not nearly possible to claim that a tool effectivelytest for a given problem, e.g. difference between two tools:

Depending on who you are and what you want, you might very well say thatthose two tools have the same support for XSS...

Moreover, tools are changing so quickly that an evaluation would only beaccurate at the time you make it.

​3. NIST SATE is literally an exposition. NIST choose test cases (real open-source program that coversdifferent type of functionalities and technologies) and ask tool makers to runtheir SAST on those programs. The goal isn't to compare the tool to claim thatone is better than the other for a type of techno, but it's too see how tools(in general) performs, to see how many types of weaknesses the tools find andalso what is the overlap of tool findings (which resulted in a very littleamount of findings).

More generally, as Andrew said, a SAST isn't only an analysis engine thatfinds weaknesses in a program; it's a suite of functionalities:

Ultimately, every one of those elements are important and need to be tested,but again, the importance of those depend on who you are and how you want touse the SAST (from simple compliance type of scan to exhaustive securitytesting).

Just to tell you, NIST SAMATE (organizers of SATE) have been thinking a lot of those problem and there is noeasy solution for evaluating SAST... But the last SATE report explains some ofthe problems we (I was part of the SAMATE team at the time) faced: SATE 2008 -NIST Special Publication 500-279

All entries

  1. February 2013 — RSA 2013 speaking session
  2. February 2013 — HTML5 tokenization visualization
  3. September 2011 — PHP, Variable variables, Oh my!
  4. July 2011 — Dissection of a SQL injection challenge
  5. January 2010 — Yes, we need a standard to evaluate SAST, but it ain't easy...
  6. November 2009 — Data driven factory: I give you data, you give me an object...
  7. June 2009 — NIST Static Analysis Tool Exposition special publication released
  8. December 2008 — Every-day's CSRF: Sorry, I turned off your christmas tree lights
  9. August 2008 — Why the "line of code" is indeed a good metric
  10. May 2008 — Accelerate the convergence to the bug: Running the test in 16-bit
  11. February 2008 — Code review tools: the missing link (so far)
  12. January 2008 — Talk: Problems and solutions for testing web application security scanners
  13. October 2007 — IE6 And IE7 don't have compatible CSS tricks
  14. September 2007 — Source Code Obfuscation
  15. February 2007 — The return of the SVG XSS
  16. February 2007 — How you should design a test suite for Web Apps Scanners
  17. January 2007 — Test Suites for Web Application Scanners
  18. December 2006 — SVG Files: XSS attacks