It has been some time since I haven't post on my blog... well, I've been busy especially with the end of SATE, and oh well! had vacation :)

Anyway, at the next Static Analysis Workshop this Thursday, we're gonna talk about the SATE experiment and the observations/results we could get from this. I am then gonna talk about a tool I wrote in order to probe if a reported weakness is a false-positive: this is the Automated Evaluation.

The main idea of the Automated Evaluation, is to get some information on the source code and, under some assumptions, try to make a conclusion on the correctness of the piece of code. Behind all the reasoning from that particular tool, my approach had to be radically different than a classical SCA otherwise this would have been like creating a new SCA and this would have been obviously useless. The context of this automated evaluation is limited to the buffer overflows and this can only work for proving false-positive only!

So basically, I am reading the source code from the reported sink to the possibles sources and grabbing the actions that possibly affect the variable which have a role in the code.

These actions are like:

• Allocation of a destination buffer
• Computing the size of the source buffer(s)
• Test for NULL
• Test that involves the size of the buffers...
• ... and some others

Then, once these actions are detected, the tool increments a global score of false-positiveness to this reported weakness. We then only have to set a threshold in order to know what correctness we want to have; this is really tied to the source code and how the program is developed.

Even though this evaluation method is not perfect, this was adapted to the C test cases we had in SATE 2008 since the global code quality was good. We can even say that the software were well written; it was then okay to make some assumption on the code such as:

• If the size of the destination buffer is computed with the size of the source buffer, the size is good (basically: no off-by-one)

Also, the tool itself needs some information on the source code such since it uses regular expression to match the "actions"...

Here we are for a quick explanation and here are the slides: SAW: Automated Evaluation of SCA output

Last Tuesday, I went to the Verify conference to give a talk about Web application scanners evaluation: what we are actually doing at NIST. I'm gonna make a simple entry reviewing what I actually talked about. The slides are here.

First of all, the evaluation was made with a test suite I made. The choices for the test suite are kinda simple, I wanted something really close to a real website. So I decided to use a real website (not a couple of test cases). The website contains multiple seeded vulnerabilities from different kinds (XSS, SQLi, RFi, CSRF, etc.). The website is actually configurable in a sense of vulnerability: you can choose what vulnerabilities will be in the website or not (let's say, I only want to have XSS vulnerabilities). Moreover, in order to see the web apps scanners capabilities, we can select a type of defense for the current protection: the level of defense.

Level of defenses

Programmers are different. They have different background, knowledge and approach to solve security problems. The filters we can see in wild web applications are not equivalents, some are good, some are just bad and we have the full shade of effectiveness. So, in order to test web apps scanner with different difficulties (for them) we implemented different level of protection around the vulnerabilities: the level of defenses.

A simple example: SQL Injection

• Level 0: No protection
• Level 1: Typecasting (in order to convert integer, boolean, double, strings, dates etc.). This protection will limit the SQL Injection on SQL native number types (integers will be converted as integer: 1' OR 1=1-- will be converted into 1).
• Level 2: Escaping the meta-characters. We are protecting about quote injection, etc.
• Level 3: Hiding the MySQL errors, we will now have Blind SQL Injections.
• Level 4: Restricted user management.
• Level 5: Using prepared statements.

Since the level of defenses will be use in combination, the order is important. (combination: level 2 = level 2(level 1(level 0))). So, using these level of defenses we are able to select the difficulty that the tool will have to break the vulnerabilities. For the results, if you are looking at the slides, in the detection rate slide, you'll see that there is not result for the level 2 which means that no tools were able to find vulnerabilities in the level of defense 2.

Attack Surface Coverage

Another point I have been working on is the attack surface coverage. A webapps scanner is not a simple piece of software which launches attacks! The crawling/parsing step is actually really important maybe the most important since it will try to understand the application. The attack surface of the test application is the places where the user has a direct interaction, means no algorithms etc. just inputs handling, error messages etc.

Here is an example of attack surface coverage check points (with numbers) for a login function:

(1) Touch the file [login.php]
if ( all fields are set ) then
	(2) All fields are set [login.php]
	Boolean goodCredentials = checkThisUser(fields)
	if ( goodCredentials ) then
		(3) Credentials are correct; the User is now log in [login.php]
		registerCurrentUser()
	else
		if ( available login test > 0 ) then
			(4) Login information incorrect [login.php]
			displayErrorLogin()
			available login test -= 1
		else
			(5) Too much try with wrong credential [login.php]
			displayErrorLogin()
			askUserToSolveCAPTCHA()
		endif
	endif
endif

Basically, we would like the scanner to use the normal behavior paths and also the abnormals (errors etc.) in order to find vulnerabilities there such as Information Leakage etc. Just a note about the attack surface coverage rate: this number cannot be interpreted alone. You need to use this with the detection rate and the false positive rate. In the slides you can see that the tool A as a 25% attack surface coverage of the application, but this is also the tool with best findings and no false positive. This means that the tool were able to find 33% of vulnerabilities (best results from all the 4 scanner we tested) in 25% of the application which can be considered as accurate compared to the others.

The attack surface coverage may have an important impact, depending on what type of testing you are doing with your webapps scanner. If you want a tool to run at the end, doing a full assessement, then you will need a tool which as a very good coverage (since you only rely on that). But if you are looking for a tool which is fully integrated in your testing process (testing == quality and security) then, I think it's better to have an accurate tool which will cover a lower surface, but the tool will cover the important points.

Conclusions

This is actually hard to make a real strong conclusion about the results given in the slides. The test application is a real simple website (banking application) and is far from a real company website; this is a huge confounding factor. Another problem is that I did the evaluation one vulnerability at the time (and one level of defense at the time). This prevent a couple of real life behaviors...

Here is a new interesting project: WASSEC. This WASC's project is run by Anurag Agarwal and is about the evaluation of web application scanners such as Watchfire's AppScan, SPI's WebInspect etc.

If you are in the field, don't wait to help us :). Here is Anurag's words:

Thank you all for your patience. We have received an overwhelming response from the WASSEC (Web Application Security Scanner Evaluation Criteria) project. To proceed with the project please

1. Please email wasc-wassec-subscribe(AT)webappsec(DOT)org and reply to confirmation email.

2. It is moderated subscription so every contributor has to be approved to send messages to the list.

3. Once you are subscribed to the list, then email wasc-wassec(AT)webappsec(DOT)org to post messages.

All further communication will be done through the mailing list. Please keep checking your junk mail folder in case some messages might go there. We are also in the process of setting up a wiki for the length of the project to post updates, etc. Until then I will be updating my blog with the project details.

Once again, thank you for your participation.

You can checkout the project here: http://webappsec.org/projects/wassec

In one of the last post, I made a comparison between two PHP Source Code Security Analyzers: SWAAT and PHP-SAT. The results was close to say that SWAAT was really better than PHP-SAT.
I started working on the configuration of PHP-SAT and it looks to be quite powerful (well, after talking with Eric Bouwers, I'm waiting for the next release) and I think I will be able to have good results with combining a security oriented configuration and some additional bugpatterns.
On the other hand, SWAAT is really limited for now as example, I've made a simple php script with only SQL queries inside: every lines are highlighted as flawed (and with a MEDIUM level)!! This is simply stupid and they would better don't report anything than doing that... just tell that you don't support SQL Injection for now... Anyway, SWAAT is for me the tool to keep an eye on, I will try to develop some features on it, especially for XSS detection and SQL Injection findings...

For a while, I've been working on a test suite for evaluating web application scanners. Now I have a test suite (PHP/MySQL/AJAX) with a bunch of variable vulnerabilities:

• XSS
• SQL Injection
• Remote File Injection
• Weak hash functions
• Session/Cookies problems
• CSRF
• others...

But there is a problem for a full evaluation. Web Application are not only a simple schema of scripts and databases and complex relation, there is also server configuration, infrastructure, different type of databases etc. Thus, I really have to create different test suites for a good coverage of what web apps could be.
I plan to use:

• Ruby On Rails framework
• ASP.NET/MS SQL based application
• JSP application

This should cover the differnt type of application but I still have to think about server types, architectures,multiple databases etc.

You: What a sense of humour!
Me: I know

By the way, this thread is only to give a URL: http://wapiti.sourceforge.net This is a quite simple web apps scanner. I have to test it for wednesday (when I'll give a presentation on Web Apps Scanners with Demos).

And because I'm glad you're reading theses lines, here is a new 'stuff' on OWASP website (meaning Cool Stuff For Web Developpers/Security) : Pantera