I just came back from the DHS Forum on the Software Assurance where I gave a talk about testing web application security scanners, and especially, the problems and some solutions for testing the scanners.

The presentation is an introduction to a methodology I've been developing at NIST for a while now. This presentation is the follow-up of the Verify Conference slides and also the talk I gave at HICSS (I will release the slides from this presentation when engadget.com will fix the vulnerabilities that I used in order to show the different variation of attacks for introducing the level of defenses)

You can reach the DHS Forum slides as a Google presentation.

Since I started to work on my static analyzer using php-ast/oracle, I realized that looking for vulnerabilities need a lot of hard coded/database entries. This is really sad, since, in order to get something correct you would need a huge knowledge database. So I started thinking of generalization of vulnerabilities and way to express it. It's tough. Really.

The most realistic (if I can say so) idea I had is to actually handle vulnerabilities definition using a given taxonomy. I still need a lot of knowledge, especially on the language (PHP) I'm analyzing, especially the output functions, global variable, filters, resources etc. but the big advantage with rules is that you can generalize the definition.

Anyway, I started dealing with natural language, will try to make this fitting into my model in order to communicate with the future static analyzer engine of php-oracle... and thanks to the AIMA project, I was able to get some fast results on the processing:

# source definition:
unvalidated input go to sink in html context
# parse tree:
2 possiblities
##
  02NP[('Adjective', 'unvalidated'), ('Noun', 'input')][]
      23VP[('Verb', 'go')][]
        45NP[('Noun', 'sink')][]
       ('Preposition', 'to')
      35PP[]
     
    25VP[]
      68NP[('Name', 'html'), ('Noun', 'context')][]
     ('Preposition', 'in')
    58PP[]
   
  28VP[]

08S[]
##
  02NP[('Adjective', 'unvalidated'), ('Noun', 'input')][]
    23VP[('Verb', 'go')][]
        45NP[('Noun', 'sink')][]
          68NP[('Name', 'html'), ('Noun', 'context')][]
         ('Preposition', 'in')
        58PP[]
       
      48NP[]
     ('Preposition', 'to')
    38PP[]
   
  28VP[]
 
08S[]

And the taxonomy I used is the following (which needs to be extended to handle more than "input validation"):

IV = Grammar('InputValidation',
	Rules(
		S = 'NP VP | S Conjunction S',
		NP = 'Pronoun | Noun | Article Noun | Adjective Noun | NP PP | NP RelClause | Name Noun',
		VP = 'Verb | VP NP | VP Adjective | VP PP',
		PP = 'Preposition NP',
		RelClause = 'That VP'
	),
	Lexicon(
		Noun = "input | output | privilege | context | header | user | sink | file",
		Verb = "is | go | write | print",
		Adjective = "validated | unvalidated | asynchronous",
		Pronoun = "me | you | i | it",
		Name = "html | database | http | sql | ldap",
		Article = "the | a | an",
		Preposition = "to | in | on",
		Conjunction = "and | or | but | not",
		That = "that"
	))

Now, I only have to finish my model of a vulnerability (I do not think about building something really general, but a model that can handle injection flaws, privilege, communication would be awesome). Once this is finish, lots of things would be possible such as generating attacks directly from the definition (this would be more like a generalized attack generator) and vulns. checkers for the source code analyzer.

I know this is a kinda tough project and I really have lots of other things to do, but I really want to give this a try... just to see where it goes...

For fuckthespam!, I wanted to add a nice feature due to the content of this website: a listing of keywords that people used to come on this website.

Well, the code is pretty simple bust just wanted to share it; it's working for google, msn and yahoo (the 3 most important search engine), I don't really care about having everything and just wanted to share this PHP snippet.

$referer = $_SERVER["HTTP_REFERER"];
if (strpos($referer,"search") > 0) {
	// look for google, yahoo and MSN
	$key = 0;
	if (strpos($referer,"google.") > 0 || strpos($referer,"msn.") > 0)
		$key = "q";
	else if (strpos($referer,"yahoo.") > 0)
		$key = "p";

	if ($key) {
		$parse_url = parse_url (urldecode($referer));
		if (array_key_exists("query",$parse_url)) {
			$query = $parse_url['query'];
			// extract (.+)$key=(.*)&
			$t = explode("&", $query);
			foreach($t as $k=>$e) {
				if ($e[0] == $key && $e[1] == '=') {
					$k = "$key=";
					$keyword = str_replace($k,'',$e);
					if (strlen($keyword) > 2) {
						// $keyword is actually the whole content of the search
					}
					break;
				}
			}
		}
	}
}

While surfing the web, I found this website: http://opensourcetesting.org/.

Just the perfect repository of testing tools, there are a bunch of them on different testing area (security, functional, quality, unit testing and so on!).

Edit: Added in my security planet!

I used to work a bit on spam bot protection, whether it is for protecting the email disclosure or the spam in the website itself. I then, started a stupid website called http://fuckthespam.com where I will gather some spam (the funny one) but also listing some anti-spam techniques :)

Hopefully I will be able to also make an history of spam to see how techniques and also content evolved.

Months ago, I was talking about and doing some small tests with the php source code security analyzer that I was able to find on the web.

I was able to quickly test the new Fortify SCA 5.0 which is handling PHP application now. I can tell you that I am really exciting about this tool. First of all, it beats from far all the tools I've tested previously (for PHP), which is fair since it's a commercial tool.

But what I'm really excited about now is that I will be able to make more tests on my test suites, compare with my security metrics & basic security analyzer, looking at the behavior of SCA tools when the source code is obfuscated, and so on. You're on the good track Fortify, now, open an API and I will be able to make an hybrid tool...

Since I also have some plan of testing real PHP applications with both testing approaches (static/dynamic), I'd like to see the difference of application coverage, vulnerability finding and false-positive rates (yeah, the last one is obvious, but still interesting).

I'm also glad to see that vendors are taking PHP as a serious language and not only for script kiddies.

Tomorrow, I'm gonna fly to Hawaii, for the HICSS-41 conference in the Big Island. I will give a talk about something I did almost one year ago: building a test suite for web application scanner. This work has been done at NIST for the Web Application Security Scanner project at SAMATE.

It's kinda old work, but still, talking about levels of defense for modeling the different possible defense mechanisms that we can find a web application. This allow to test the web apps scanner with more realistic flaws and see how they behave with some few protections...

Anyway, I think I will have good time there, and also good talks to fellow...

The Web Application Security Consortium is pleased to announce the first results of the Script Mapping project! At this stage in the project we were able to cover most of the test cases for Internet Explorer 7, Firefox 2 and Safari 3.

The results can be found on the project page: http://www.webappsec.org/projects/scriptmapping/

Project Description:

The purpose of the Script Mapping Project is to come up with an exhaustive list of vectors to execute script within a web page without the explicit use of <script> tags. This data can be useful when testing poorly implemented Cross-site Scripting blacklist filters, for those wishing to build an html white list system, as well as other uses.

WASC is actively seeking volunteers from various sections of the community including penetration testers, security researchers, and developers to contribute to this project.

If you would like to be involved with the project or if you have comments about the results, test cases etc., please contact me.

I love iGoogle. I have a couple of widgets and my RSS feeds in it... That's actually the problem, I had too much feeds, so I decided to create my own "planet" in order to have just one feed with all security blogs I'm reading.

You can reach it here if you have the same taste as mine: http://rgaucher.info/planet

I know that planet-security is pretty much doing the same, but I don't like the interface of this website and it doesn't have all the feeds I'm following...

Previously, on the python version of Grabber, I used a BFS crawler. Good to scan all the code (as long as the parsers are not that dumb). The problem with these crawlers is that it's totally inefficient: all the problems are not everywhere.

Starting with this assumption, I tried to start rating what is actually important and what are the evidence that a page may be important for a security testing point of view. So, the architecture of the crawler is simply based on a priority queue and the priority is for now based on obvious reasoning which may be wrong: The script I prefer testing, is the one that is in POST, where the action is in HTTPS (and so on for the rest...) which gives something like that:

  priority <- 30
  If Form Then
    priority <- 10
    If Method = Post Then
      priority <- 5
  else if Anchor Then
    If Get Variables Then // To Understand: index.php?foo=plop, compared to index.php
      priority <- 20
  If HTTPS Communication for {Method action or Anchor URL} Then
      priority /= 2

This is a fairly incomplete work and kinda dumb, but at least it's unbiased for a set of URL.

In my previous blog post, I talked briefly about PHP-Ast/Oracle a PHP source code static analysis framework. I am developing it in order to play with source code and security. The goal of that framework is to be able to perform different type of operations on a PHP source code. I am releasing this tool as it is because I think people may be interested with this... Anyway, I learned a lot doing this.

PHP-Ast/Oracle is developed in C++ and the tool has been developed mainly for:

• Implementing source code (security) metrics
• Gathering information on the source code (data-flow variable paths, etc.) by building the model
• Creating personal source code obfuscations (currently only pattern based is supported, no data-type)
• Source to source converter (e.g. PHP to C++ code for static analysis purposes)

How it works

The source code repository is divided in 2 parts:

• php-ast is the converter from PHP to XML
• php-oracle is the actual engine

php-oracle get a XML file as input which is the output of php-ast. In the SVN there are some python scripts I used in order to combine the 2 tools (they may be outdated i.e. doesn't work with the current php-oracle).

How I think you could use php-oracle

I do not attend to make a clean build with an executable etc. I just provide source code. I decided to give only the source code because I don't want to spend too much time on creating a clean software, it's only research oriented stuff. Furthermore, there is not much documentation in the source code (advantages of being alone to develop such a tool) and then, only really interested people will download this! I can then help them if they have some question about how it works etc.

Getting the source code

You can download the source here: php-ast-oracle.zip

And the trac repository has more documentation about what the framework actually does: http://trac2.assembla.com/php-ast

Development

The tool is in perpetual development, I don't want to create a real software from that, but I think people can use it to perform security analysis, compute stuff, make code transformation and so on.

I've been working a couple of months on a project named php-ast/oracle. I am opening the source of the project today because I think that people may be interested in such a code. Roughly, php-ast/oracle is able to get/transform information on a php source code, I used it for: creating real obfuscations (control-flow, data-flow), implementing security metrics, writing a converter from php to c++ for static analysis purpose and some other stuff such as variables flow etc.. You can have more information here: http://trac2.assembla.com/php-ast. I may post about this project later don't have much time now...

But this news is only for releasing a script I used a lot this last weeks; a PHP preprocessor. I've been using this preprocessor in order to clean the crappy PHP code we can found in the wild... in order to use php-ast/oracle correctly for calculating security metrics and so on.

The preprocessor is actually doing 3 things:

• Simplifying the strings (keeping only the php variables in the strings -- really important for keeping the AST small with SQL queries and so on, because the strings could be evaluated in PHP, the AST would need to tokenize the strings)
• Removing comments and HTML
• Resolving the file inclusions (not for dynamic variable inclusion of course, but it's working with define names and static names)

The preprocessor is available here: preproc.zip

Grabber was a nice project. The main goal for me was to learn stuff around web application security/scanners; I didn't really know much before I started this project. But now that I've been playing with web apps scanners for more than 10months, I need to create a new one and go deeper in heuristics, browser integration and AI.

Grabber was in fact more a spider+fuzzer than something else... Not a good web apps scanner at all. Thinking of the analysis engine... It's something kinda stupid, no JavaScript execution, just simple heuristics for parsing and levenstein distances ;)

Anyway, I decided to start over this project. It's not gonna be a bunch of python scripts anymore, I am gonna use Qt/C++ extensively. The idea if this project is to be pen-testers oriented and open, I want to create a kind of wrapper around WebKit (especially using QtWebKit), a spider as core utilities and after, using plugins. The plugins should be either in C++ or JavaScript (QtScript actually). So far, we are 3 guys thinking of this project: we didn't start yet but we are open to every contribution; the project will of course be free and GPL'd.

I just post this in order to get some comments or suggestions about what a web apps scanner should do... Feel free to comment/mail...

Talking about web application security scanners , we all have the same problem: False Positive. It's a fact that cannot exactly be solve by the testing methodology itself (since it relies on pattern detection). So, the idea I started talking about on #webappsec today is a common format for exchanging information between tools.

Ideally, this would work like this:

1. Tool A is scanning a website.
2. It exports some information a given format: out-tool-a.xml
3. Tool B is able to understand out-tool-a.xml and take this as an input
4. Tool B would then be able to verify the results/false-positive of Tool A by scanning with the information in the out-tool-a.xml

I really think that would be helpful somehow, at least for open-source tools. I'm gonna try to implement this for the next release of Grabber.

Last Tuesday, I went to the Verify conference to give a talk about Web application scanners evaluation: what we are actually doing at NIST. I'm gonna make a simple entry reviewing what I actually talked about. The slides are here.

First of all, the evaluation was made with a test suite I made. The choices for the test suite are kinda simple, I wanted something really close to a real website. So I decided to use a real website (not a couple of test cases). The website contains multiple seeded vulnerabilities from different kinds (XSS, SQLi, RFi, CSRF, etc.). The website is actually configurable in a sense of vulnerability: you can choose what vulnerabilities will be in the website or not (let's say, I only want to have XSS vulnerabilities). Moreover, in order to see the web apps scanners capabilities, we can select a type of defense for the current protection: the level of defense.

Level of defenses

Programmers are different. They have different background, knowledge and approach to solve security problems. The filters we can see in wild web applications are not equivalents, some are good, some are just bad and we have the full shade of effectiveness. So, in order to test web apps scanner with different difficulties (for them) we implemented different level of protection around the vulnerabilities: the level of defenses.

A simple example: SQL Injection

• Level 0: No protection
• Level 1: Typecasting (in order to convert integer, boolean, double, strings, dates etc.). This protection will limit the SQL Injection on SQL native number types (integers will be converted as integer: 1' OR 1=1-- will be converted into 1).
• Level 2: Escaping the meta-characters. We are protecting about quote injection, etc.
• Level 3: Hiding the MySQL errors, we will now have Blind SQL Injections.
• Level 4: Restricted user management.
• Level 5: Using prepared statements.

Since the level of defenses will be use in combination, the order is important. (combination: level 2 = level 2(level 1(level 0))). So, using these level of defenses we are able to select the difficulty that the tool will have to break the vulnerabilities. For the results, if you are looking at the slides, in the detection rate slide, you'll see that there is not result for the level 2 which means that no tools were able to find vulnerabilities in the level of defense 2.

Attack Surface Coverage

Another point I have been working on is the attack surface coverage. A webapps scanner is not a simple piece of software which launches attacks! The crawling/parsing step is actually really important maybe the most important since it will try to understand the application. The attack surface of the test application is the places where the user has a direct interaction, means no algorithms etc. just inputs handling, error messages etc.

Here is an example of attack surface coverage check points (with numbers) for a login function:

(1) Touch the file [login.php]
if ( all fields are set ) then
	(2) All fields are set [login.php]
	Boolean goodCredentials = checkThisUser(fields)
	if ( goodCredentials ) then
		(3) Credentials are correct; the User is now log in [login.php]
		registerCurrentUser()
	else
		if ( available login test > 0 ) then
			(4) Login information incorrect [login.php]
			displayErrorLogin()
			available login test -= 1
		else
			(5) Too much try with wrong credential [login.php]
			displayErrorLogin()
			askUserToSolveCAPTCHA()
		endif
	endif
endif

Basically, we would like the scanner to use the normal behavior paths and also the abnormals (errors etc.) in order to find vulnerabilities there such as Information Leakage etc. Just a note about the attack surface coverage rate: this number cannot be interpreted alone. You need to use this with the detection rate and the false positive rate. In the slides you can see that the tool A as a 25% attack surface coverage of the application, but this is also the tool with best findings and no false positive. This means that the tool were able to find 33% of vulnerabilities (best results from all the 4 scanner we tested) in 25% of the application which can be considered as accurate compared to the others.

The attack surface coverage may have an important impact, depending on what type of testing you are doing with your webapps scanner. If you want a tool to run at the end, doing a full assessement, then you will need a tool which as a very good coverage (since you only rely on that). But if you are looking for a tool which is fully integrated in your testing process (testing == quality and security) then, I think it's better to have an accurate tool which will cover a lower surface, but the tool will cover the important points.

Conclusions

This is actually hard to make a real strong conclusion about the results given in the slides. The test application is a real simple website (banking application) and is far from a real company website; this is a huge confounding factor. Another problem is that I did the evaluation one vulnerability at the time (and one level of defense at the time). This prevent a couple of real life behaviors...