“The NIST SAMATE project conducted the first Static Analysis Tool Exposition (SATE) in 2008 to advance research in static analysis tools thatfind security defects in source code. The main goals of SATE were to enable empirical research based on large test sets and to encourage improvement andspeed adoption of tools. The exposition was planned to be an annualevent.”
SATE 2008 was one of my last project at NIST. I really enjoyed working on this project from the beginning, it was challenging especially because we hadto create so many artifacts to make the tool reporting the weaknesses the sameway, integrate them all together and provide ways for assessors to makemeaningful reviews.
In a nutshell, we selected 6 different open-source programs (3 in C, 3 in Java) and made tool vendors running their tool on these test cases. Tool vendors were allowed to customize their tool if their tool provide such capability. Fortify was the only vendor who created a custom rule (to help the tool with a validation routine for MVNForum). Our goal was then to combine theresults all together and analyze: provide information on the correctness of thetool.
If you are interested, you can download the SATE data andthe NIST SATE Special Publication.
Thanks to all the SAMATE team for this effort, and especially Vadim Okun andPaul E. Black.
For more information, you can reach the SATE page at NIST.