deep inside: security and tools

How you should design a test suite for Web Apps Scanners

If you have ever think about using a web application scanner for testing the security of your website, you certainly made a choice: Which web apps scanner should I buy/use ?

In this post, I will not tell you what is the better black box tester for whatever kind of web application.

The web applications may be very different, the tools are different and thus they could have different efficiency (i think it's non countable noun)... If you read this, you probably know that I am talking about scanners such as WebInspect, AppScan, Acunetix, Hailstorm, Pantera, Grabber... In the following sections, I will explain a main idea that should be used for testing such a tool.

A test suite for our tools is a website, this website has typically vulnerabilities; you can see this kind of website by watchfire, spi but also WebGoat, SiteGenerator or others. But all of these websites are not realistic and do not consider that the vulnerabilities may exist in different instances: variants.
If you don't know what is a variant check at the XSS Cheat Sheet (RSnake) or at the Attack Patterns (Sean Barnum). A variant is what the hacker use to perform his exploit.

A simple example for XSS is: Let's say you protect your website against XSS by checking the <script> tag; this is not perfect and not good because there is some way to insert other type of XSS strings (onmouseover).

Here comes the concept of Level Of Defense. If you are a developer you think about filters, if you are an attacker you think about variants of vulnerability and attack patterns. The level of defense of a website is the strength of its filters again a given vulnerability.

For a SQL Injection you can have multiple type of filters... Here is a possible list of levels for the SQL Injection:

In the concept of the level of defense, it's important to not that depending of the type of vulnerability (weakness, failure...) the level n-1 is also performed in the level n or the level n is stronger (for the same variants) than the level n-1 (for instance, for Weak Hash Function it's not possible but using SHA-1 instead of MD5 is a level of defense higher).

A Key point: When you are implementing a level of defense for a vulnerbility, you must be sure that your implementation does the whole thing for that type of filter. For example, if you are escaping the HTML entities, you need to do all not only <, > and in the next LoD escaping ' and ".

Why is the level of defense better than a simple system with vulnerabilities?

With the level of defense, you can calibrate a type of website which may be close to yours; you can construct a test suite with your kind of level of defense and see how the tool detect the vulnerabilities when the LoD increase.It is also a good way to know the state of the art of the tools for detecting vulnerabilities...

The idea was developed to create a test suite in order to evaluate web apps scanners; in this test suite we can select the current type of vulnerability and its level of defense (the hardness to break):

All entries

  1. February 2013 — RSA 2013 speaking session
  2. February 2013 — HTML5 tokenization visualization
  3. September 2011 — PHP, Variable variables, Oh my!
  4. July 2011 — Dissection of a SQL injection challenge
  5. January 2010 — Yes, we need a standard to evaluate SAST, but it ain't easy...
  6. November 2009 — Data driven factory: I give you data, you give me an object...
  7. June 2009 — NIST Static Analysis Tool Exposition special publication released
  8. December 2008 — Every-day's CSRF: Sorry, I turned off your christmas tree lights
  9. August 2008 — Why the "line of code" is indeed a good metric
  10. May 2008 — Accelerate the convergence to the bug: Running the test in 16-bit
  11. February 2008 — Code review tools: the missing link (so far)
  12. January 2008 — Talk: Problems and solutions for testing web application security scanners
  13. October 2007 — IE6 And IE7 don't have compatible CSS tricks
  14. September 2007 — Source Code Obfuscation
  15. February 2007 — The return of the SVG XSS
  16. February 2007 — How you should design a test suite for Web Apps Scanners
  17. January 2007 — Test Suites for Web Application Scanners
  18. December 2006 — SVG Files: XSS attacks