I <3 Bots!
Subscribe to the RSS feed

Keyword - source code scanners

Entries feed - Comments feed

Tuesday, June 30 2009

NIST Static Analysis Tool Exposition special publication released

The NIST SAMATE project conducted the first Static Analysis Tool Exposition (SATE) in 2008 to advance research in static analysis tools that find security defects in source code. The main goals of SATE were to enable empirical research based on large test sets and to encourage improvement and speed adoption of tools. The exposition was planned to be an annual event.

SATE 2008 was one of my last project at NIST. I really enjoyed working on this project from the beginning, it was challenging especially because we had to create so many artifacts to make the tool reporting the weaknesses the same way, integrate them all together and provide ways for assessors to make meaningful reviews.

In a nutshell, we selected 6 different open-source programs (3 en C, 3 in Java) and made tool vendors running their tool on these test cases. Tool vendors were allowed to customize their tool if their tool provide such capability. Fortify was the only vendor who created a custom rule (to help the tool with a validation routine for MVNForum). Our goal was then to combine the results all together and analyze: provide information on the correctness of the tool.

If you are interested, you can download the SATE data and the NIST SATE Special Publication.

Thanks to all the SAMATE team for this effort, and especially Vadim Okun and Paul E. Black.

For more information, you can reach the SATE page at NIST.

Saturday, February 21 2009

SHA-3 reference implementations buffer overflows

Fortify just posted a nice blog post about the audit they did on several reference implementation that compete for being the next NIST SHA-3.

They do not release much information on their findings: only one is described. I would have really like to see how powerful was the analysis (if it was) to find these problems.

It could be nice too to see other tool vendors, such as Grammatech, Klocwork, Coverity, etc. to do the same, and then, start another competition ;)

I'd really like to emphasize the conclusions in the Fortify's blog post:

Reference implementations don't disappear, they serve as a starting point for future implementations or are used directly. A bug in the RSA reference implementation was responsible for vulnerabilities in OpenSSL and two seperate SSH implementations. They can also be used to design hardware implementations, using buffer sizes to decide how much silicon should be used.

The other consideration is speed, which will be a factor in the choice of algorithm. The fix for the MD6 buffer issues was to double the size of a buffer, which could degrade the performance. On the other hand, memory leaks could slow an implementation. A correct implementation is an accurate implementation.

Sunday, August 10 2008

Why the "line of code" is indeed a good metric

When I first learned about source code metrics, I was amazed about people using the line of code for doing comparison with software. It was for me a lack of imagination.

At the beginning of the week, I started a small and fast experiment: extracting metrics from the SATE 2008 test cases. This experiment focuses on function-wise properties and therefore, I have to extract for each functions a couple of metrics:

  • McCabe's cyclomatic complexity which computes the code complexity, this is indeed a good metric to estimate the difficulty that a human will have to understand a given piece of code (very important for security related problems)
  • Line of Code
  • Line of Comments
  • Number of local variables
  • Number of parameters (which represents the coercion between the function and the whole program)
  • Number of function call
  • Number of function that are ``sources''
  • Number of function that are ``sinks''
  • Number of C standards functions (obviously, only for C test cases)

At first the the line of code was implemented cause it's an easy one to compute and it also gives an important value if we want to normalize the other metrics. We also decided to introduce the number of ``source/sinks'' for studying input validation weaknesses later on...

Anyway, after running some statistics on the output results, I was amazed by observing that the Pearson correlation coefficient between McCabe and Line of Code was never less than 0.90 (which could be compare to 90% as a correlation rate) (but I have to say that there is huge limitations in the parsers we are using for extracting information, for instance, the C is not pre-processed etc.). This result is only valid for C test cases, actually, the average of observed correlation in Java test case is around 0.60...

Of course further statistical analysis will be necessary to conclude anything on this subject, but if we were unlucky with the test cases selection, this may have been a source of the problem, but I don't think we were. Actually, this seems quite logical to think that these metrics a related, the longer the code is, the more complex in term of tests, loops etc. it can be, there is indeed more chance that a longer code contains more cycles :)

Oh well, I'll keep writing about especially since I expect to get results pretty soon...

Tuesday, June 10 2008

My talk at SAW: Automated Evaluation of source code analyzer output

It has been some time since I haven't post on my blog... well, I've been busy especially with the end of SATE, and oh well! had vacation :)

Anyway, at the next Static Analysis Workshop this Thursday, we're gonna talk about the SATE experiment and the observations/results we could get from this. I am then gonna talk about a tool I wrote in order to probe if a reported weakness is a false-positive: this is the Automated Evaluation.

The main idea of the Automated Evaluation, is to get some information on the source code and, under some assumptions, try to make a conclusion on the correctness of the piece of code. Behind all the reasoning from that particular tool, my approach had to be radically different than a classical SCA otherwise this would have been like creating a new SCA and this would have been obviously useless. The context of this automated evaluation is limited to the buffer overflows and this can only work for proving false-positive only!

So basically, I am reading the source code from the reported sink to the possibles sources and grabbing the actions that possibly affect the variable which have a role in the code.

These actions are like:

  • Allocation of a destination buffer
  • Computing the size of the source buffer(s)
  • Test for NULL
  • Test that involves the size of the buffers...
  • ... and some others

Then, once these actions are detected, the tool increments a global score of false-positiveness to this reported weakness. We then only have to set a threshold in order to know what correctness we want to have; this is really tied to the source code and how the program is developed.

Even though this evaluation method is not perfect, this was adapted to the C test cases we had in SATE 2008 since the global code quality was good. We can even say that the software were well written; it was then okay to make some assumption on the code such as:

  • If the size of the destination buffer is computed with the size of the source buffer, the size is good (basically: no off-by-one)

Also, the tool itself needs some information on the source code such since it uses regular expression to match the "actions"...



Here we are for a quick explanation and here are the slides: SAW: Automated Evaluation of SCA output

Friday, May 16 2008

Yet another study on code quality: A Tale of Four Kernels

If like me you are interested in code quality and some general conclusion that one can draw based on code quality studies, I really recommend to read this paper: A Tale of Four Kernels by Diomidis Spinellis, ICSE '08: Proceedings of the 30th International Conference on Software Engineering

I just want to quote a part of the conclusion by the author

Therefore, the most we can read from the overall balance of marks is that open source development approaches do not produce software of markedly higher quality than proprietary software development.

The only problem with this statement is that it is based on the fact that the metrics he used were not weighted for their importance for the "Code Quality" (if this means something). Therefore, the comparison between the Windows research kernel and Linux seems a little bit awkward to me. Anyway, this is a very interesting paper about code quality, and lots of interesting ideas from the author of CScout.

Wednesday, May 14 2008

Static Analysis Tool Exposition is over

Yeah, that's sad and also a relief: SATE is over. We actually released today the last stage of the evaluation (basically, the evaluation with some correction based on comments from the participants). Even though I would have prefer to have more feedback from participants on our evaluation, especially to increase its quality, I still think SATE is a good thing and will be an interesting resource for lost of researchers. This is, as far as I know, the only exhaustive resource on the subject (wild source code + weaknesses).

What do I want to do, see next? Since we have accumulated lots of data with the tool reports (raw weaknesses), the evaluations (I really want to thank MITRE's guys, especially Steve Christey and Bob Schmeichel for their help), I'm looking forward to do data analysis and trying to extract some limited results on it.

Anyway, this was overall a good experience, I actually did my first real code review mostly on lighttpd, dspace, mvnform and naim, I think I know way more on how detecting vulnerabilities, I also have been asking myself about how to rate vulnerabilities such as Cross-Site Scripting (hopefully, I will release the little document I wrote about it), I learned so much about how people are writing code trying to understand the design, the code etc. in the applications.

Also, hopefully, I will be able to release the website I developed to handle the weaknesses from different tools. It is, I think, interesting if you are working with more than one assessor. You can send evaluation, comments, merging the weaknesses etc. with a web interface. Even though it needs improvements (it has been done in less than 2 weeks) I think this would be an interesting piece of software for people who are dealing with tons of weaknesses. Another interesting point is that we (at NIST) may open that website for everybody in order to make new evaluation in order to increase the quality of the data we currently have.

Oh well, it seems like a journey is really close to its end, it was such a good time sometimes, and some other time such consuming work. We've been dealing with fifty thousands of weaknesses, dozen of tool reports, and almost tens of test cases... I will keep you posted about the next decision we are gonna make with SATE and hope that lots of people will find in this "exposition" the most they could get.

Friday, February 29 2008

NIST SATE step 3 completed: test cases information release

This evening at work, with Vadim, we were exhausted after days of work but we were smiling. Smiling and happy because we knew that the step 3 of SATE was pretty much done. The step 3 is when all the participants are sending their output to us. Even if we know that we will have hard time to come up with the master reference list for each test cases what we selected for SATE 2008, we know that this is interesting data for the SwA community and especially SCA studies.

Today, we can finally tell which test cases were selected by us for SATE 2008. First of all, we have 2 different tracks: C language and Java language. For the java track, we decided to look more into web applications. We then have:

And for the C track we selected:

  • Nagios: host, service and network monitoring with web interface (using CGI)
  • Lighttpd: web server
  • Naim: console instant messenger

You may have lots of comments on why these and I am totally ready to answer your questions. Just to let you know, during the selection phase, we reviewed 50+ different applications. For each applications, we had to scan them using tools, doing some manual review and our main goal is to find at least one exploitable vulnerability. Concerning the type of test cases themselves, the constrain is to have real exploitable vulnerabilities and they must be real applications which means basically, not test cases that we have in our SRD.

Just as reminder, the next important dates for SATE 2008 are:

  • April 15, we are distributing to the participants our master reference list, the list of real weaknesses found by the participants
  • June, comparison of all the participants results, the participants get all the reports submitted at SATE 2008
  • December, all the data and reports are public

Monday, February 25 2008

Code review: facilitate the SCA output analysis

This post is not exactly a follow up of a previous post called Code review tools: the missing link (so far), But since I will have to perform a lot of code review in the next couple of weeks and also tool output analysis, I was looking for some tool to help me, to facilitate my job. I've been asking people for links, tips etc. but nothing really convinced me. I am looking for a tool which is basically able to smartly index the source code I am reviewing, which means that I want to be able to look at the variables, where they are declared, affected and used... I also want to see the call graphs of functions and this, mostly to probe the correctness of tool output.

After a couple of hours looking at specialized tools, I was not able to find something good and free (No, I don't call cscope good!). Yes, there are a couple of commercial ones, especially the ones shipped with the commercial source code analyzers and well, they're not perfect at all!

So, this morning, I was like frustrated when I actually thought of using a tool I used a lot, but for a quite different utilization: Doxygen. You may know this documentation tool, but may not know all it is capable of.

As a documentation generation tool, it is really powerful and mostly based on specially formated comments that the developers seed in the source code. But the tool is also generating a bunch of structure related information such as classes relations, function calls graphs etc. As I don't want to generate a documentation of the code I'm reviewing, I don't mind not to have the well formated comments. I am asking this tool to generate me the structural information and facilitate the navigation from function to function.

I made a small example of the report generated by Doxygen using the configuration I made for getting all the information I wanted (only one page since the documentation and the pictures etc. are kinda big...). In order to generate the configuration I wanted, I made a tiny python script ozone.py since the DoxyWizard is not really convenient for that. Also, I will add a process to pre-compile the JSP files since Doxygen doesn't understand the JSP syntax and the option to use the Doxygen search engine (PHP script that use and file with indexed tags).

This is the first step of that script, as you may see by looking at the source code, I am also generating the XML files, this is because the XML generated Doxygen documentation contains a lot of interesting information that I may use later... Also, while looking at the Doxygen source code, I thought that it could be possible to integrate many more static analysis such as computing metrics, etc. Anyway, so many other things to do than thinking about that right now!

Tuesday, February 19 2008

Code review tools: the missing link (so far)

First of all, I do not consider myself as a pen-tester so maybe you will find these ideas irrelevant, stupid or useless... I have been doing some pen-testing though, whether it was for some friends, for fun (yeah, it's good to learn like that) or for profit (well, it was kinda part of my job for SATE 2008) so I'm not that n00b but I am not a pen-tester. I am not an expert in pen-testing and code review. But when I do some, at work, I have the chance to be able to use commercial tools — I say it's a chance because there is a real benefit of using such tools. In fact, tools are good, way better than me, they can find thousands of vulnerabilities in minutes... I cannot; I need way more time. But here is a little feedback vendors can have from me, utilizing the tools.

The tools are amazing to find some defects, saying that something doesn't look good to them and giving you a stack of 42 function calls. Eh! that's part of the job to examine this bunch of function in order to see why the tool reported this as a vulnerability. So, examining the functions means looking how the data will be transformed/transported from a point to one another. And I cannot tell you the pain it is to do that for the dozens of reported vulnerabilities where the correctness of the tool is not obvious (at least for me).

While talking about that with Vadim today, I thought of a tool that would be awesome for a code reviewer in order to facilitate the “correctness tests”. The idea is really simple and maybe the tool already exists — if so, please give me a link! — but what if you had a kinda debugger where you were able to select the point where you want to start the dynamic evaluation of a piece of code (the Entry Point) and the point where you want to finish and see the result (the Break Point). What is the difference with a typical debugger? The possibility to do such in relation with the source code. In the interface of the source code analyzer, I would be able to select the entry point I want to start my dynamic analysis and the break point. I would launch the dynamic evaluation which would go to the state of the entry point (maybe by asking how to go there... there is often multiple paths to go to one branch of the code), then I would do the modification I want (trying to bypass some filters for example with some weird strings) and the dynamic engine would run the piece of code until the Break Point; then look at the result.

What I just described is a really narrow view of such combination of static/dynamic analysis, by doing a step-by-step modification of the values. We could have information of the privilege state of the current user for a web application, would be able to replay easily a la web apps scanners, etc.

I know that building such a tool is doable. Hard but definitely doable. So far, the toughest point I saw is to be able to arrive at a given state of the program. You would need to do a binary coverage and looking at the branches to take, recording these and mapping the records with the source code. Once you're done with it, you're ready for modifying the parameters, and to look at the results. Yes, the main difference with a debugger is to come in a given state referenced by a function call. But wouldn't this help you to figure out the correctness of a given piece of code?

Thursday, February 14 2008

SATE ready to go + weaknesses walker + Shmoo + 100

Tomorrow will start SATE 2008: the registered participants will be able to get the test cases associated to the tracks they want to participate in. They will have until the 29th of February to send the report of the tools. We are all pretty excited here before the start. It was a real rush for finding the test cases that we think are good for such an event...

Anyway, just a news to release a python script which is definitely SATE oriented. The idea is only to convert the output of some free tools into the SATE XML format. The script is handling Flawfinder, ITS4 and RATS. It can also look at the NVD for the product and the version in order to retrieve the known vulnerabilities.

You can download the script weaknesses walker as a zip file or just the python script (you will need wwwCall for the NVD scrapping part; wwwCall is also included in the zip).

Example how to use ww with flawfinder:

./ww.py --tool flawfinder --file myproject.out.xml --format sate /home/romain/myproject

or for the NVD scrapper:

./ww.py --vdb winamp 5.2 --file winamp_5.2.nvd.xml

For the next version of ww, I may add the possiblity to play with the SATE XML format itself, such as merging the results of different tools with comparison of report or even just the report of multiple tools...

Also, if you are coming downtown DC this weekend for ShmooCon or even BlackHat DC 2008, if you wanna have a beer just drop me a mail. I wasn't able to find a ticket for Shmoo so will not go, but I will meet with dre and marcin from ts/sci security... so if you are around, just tell me I would be happy to meet more sec. people

The last thing is that this post is my number 100!

Tuesday, February 5 2008

NIST Static Analysis Tool Exposition: No, this is not a competition!

I've was happy yesterday when I learned that Fortify will participate to the Static Analysis Tool Exposition (SATE) we are currently organizing. And even more when I saw this morning Brian Chess blogging about SATE.

We've been working on SATE since our last Static Analysis Summit and, helped with a couple of existing exposition already existing at NIST such as TREC etc. for the guidelines, the rules and so on. But even so, we had some example, we had three difficult tasks:

  1. Make people agree on the fact that it is not a competition
  2. Make vendors participating (if you are a vendor, reading this please, subscribe for participating at SATE)
  3. Choosing the test cases

The last point is not solved yet, and even, none of them can be considered as solved since not everybody is participating to the 2008 exposition (which has 2 tracks: C and Java), but we've been seeking for good test cases in C and Java. Good test cases... means not too big, not too small and having exploitable vulnerabilities. By the way, if any of the readers of this blog have some idea of Java or C test cases that would be good test cases, please, send me links, ideas or whatever :)

Anyway, SATE is on his way, I hope more tool makers will sign up for participating at this experiment.

Maybe another point, due to my usual blogging on web security and web apps security scanners, if SATE is a success as we expect it to be, we may open new tracks for... web application security scanners and I would love to have special tracks for security metrics (I want to show up!! :p)

Wednesday, January 30 2008

Definition parsing: first step done

Since I started to work on my static analyzer using php-ast/oracle, I realized that looking for vulnerabilities need a lot of hard coded/database entries. This is really sad, since, in order to get something correct you would need a huge knowledge database. So I started thinking of generalization of vulnerabilities and way to express it. It's tough. Really.

The most realistic (if I can say so) idea I had is to actually handle vulnerabilities definition using a given taxonomy. I still need a lot of knowledge, especially on the language (PHP) I'm analyzing, especially the output functions, global variable, filters, resources etc. but the big advantage with rules is that you can generalize the definition.

Anyway, I started dealing with natural language, will try to make this fitting into my model in order to communicate with the future static analyzer engine of php-oracle... and thanks to the AIMA project, I was able to get some fast results on the processing:

# source definition:
unvalidated input go to sink in html context
# parse tree:
2 possiblities
##
  02NP[('Adjective', 'unvalidated'), ('Noun', 'input')][]
      23VP[('Verb', 'go')][]
        45NP[('Noun', 'sink')][]
       ('Preposition', 'to')
      35PP[]
     
    25VP[]
      68NP[('Name', 'html'), ('Noun', 'context')][]
     ('Preposition', 'in')
    58PP[]
   
  28VP[]

08S[]
##
  02NP[('Adjective', 'unvalidated'), ('Noun', 'input')][]
    23VP[('Verb', 'go')][]
        45NP[('Noun', 'sink')][]
          68NP[('Name', 'html'), ('Noun', 'context')][]
         ('Preposition', 'in')
        58PP[]
       
      48NP[]
     ('Preposition', 'to')
    38PP[]
   
  28VP[]
 
08S[]

And the taxonomy I used is the following (which needs to be extended to handle more than "input validation"):

IV = Grammar('InputValidation',
	Rules(
		S = 'NP VP | S Conjunction S',
		NP = 'Pronoun | Noun | Article Noun | Adjective Noun | NP PP | NP RelClause | Name Noun',
		VP = 'Verb | VP NP | VP Adjective | VP PP',
		PP = 'Preposition NP',
		RelClause = 'That VP'
	),
	Lexicon(
		Noun = "input | output | privilege | context | header | user | sink | file",
		Verb = "is | go | write | print",
		Adjective = "validated | unvalidated | asynchronous",
		Pronoun = "me | you | i | it",
		Name = "html | database | http | sql | ldap",
		Article = "the | a | an",
		Preposition = "to | in | on",
		Conjunction = "and | or | but | not",
		That = "that"
	))

Now, I only have to finish my model of a vulnerability (I do not think about building something really general, but a model that can handle injection flaws, privilege, communication would be awesome). Once this is finish, lots of things would be possible such as generating attacks directly from the definition (this would be more like a generalized attack generator) and vulns. checkers for the source code analyzer.

I know this is a kinda tough project and I really have lots of other things to do, but I really want to give this a try... just to see where it goes...

Tuesday, January 22 2008

PHP Source Code Analyzer

Months ago, I was talking about and doing some small tests with the php source code security analyzer that I was able to find on the web.

I was able to quickly test the new Fortify SCA 5.0 which is handling PHP application now. I can tell you that I am really exciting about this tool. First of all, it beats from far all the tools I've tested previously (for PHP), which is fair since it's a commercial tool.

But what I'm really excited about now is that I will be able to make more tests on my test suites, compare with my security metrics & basic security analyzer, looking at the behavior of SCA tools when the source code is obfuscated, and so on. You're on the good track Fortify, now, open an API and I will be able to make an hybrid tool...

Since I also have some plan of testing real PHP applications with both testing approaches (static/dynamic), I'd like to see the difference of application coverage, vulnerability finding and false-positive rates (yeah, the last one is obvious, but still interesting).

I'm also glad to see that vendors are taking PHP as a serious language and not only for script kiddies.

Wednesday, December 5 2007

Static Analysis Framework: PHP-Ast/Oracle

In my previous blog post, I talked briefly about PHP-Ast/Oracle a PHP source code static analysis framework. I am developing it in order to play with source code and security. The goal of that framework is to be able to perform different type of operations on a PHP source code. I am releasing this tool as it is because I think people may be interested with this... Anyway, I learned a lot doing this.

PHP-Ast/Oracle is developed in C++ and the tool has been developed mainly for:

How it works

The source code repository is divided in 2 parts:

  • php-ast is the converter from PHP to XML
  • php-oracle is the actual engine

php-oracle get a XML file as input which is the output of php-ast. In the SVN there are some python scripts I used in order to combine the 2 tools (they may be outdated i.e. doesn't work with the current php-oracle).

How I think you could use php-oracle

I do not attend to make a clean build with an executable etc. I just provide source code. I decided to give only the source code because I don't want to spend too much time on creating a clean software, it's only research oriented stuff. Furthermore, there is not much documentation in the source code (advantages of being alone to develop such a tool) and then, only really interested people will download this! I can then help them if they have some question about how it works etc.

Getting the source code

You can download the source here: php-ast-oracle.zip

And the trac repository has more documentation about what the framework actually does: http://trac2.assembla.com/php-ast

Development

The tool is in perpetual development, I don't want to create a real software from that, but I think people can use it to perform security analysis, compute stuff, make code transformation and so on.

Sunday, December 2 2007

Yet another study oriented release

I've been working a couple of months on a project named php-ast/oracle. I am opening the source of the project today because I think that people may be interested in such a code. Roughly, php-ast/oracle is able to get/transform information on a php source code, I used it for: creating real obfuscations (control-flow, data-flow), implementing security metrics, writing a converter from php to c++ for static analysis purpose and some other stuff such as variables flow etc.. You can have more information here: http://trac2.assembla.com/php-ast. I may post about this project later don't have much time now...

But this news is only for releasing a script I used a lot this last weeks; a PHP preprocessor. I've been using this preprocessor in order to clean the crappy PHP code we can found in the wild... in order to use php-ast/oracle correctly for calculating security metrics and so on.

The preprocessor is actually doing 3 things:

  • Simplifying the strings (keeping only the php variables in the strings -- really important for keeping the AST small with SQL queries and so on, because the strings could be evaluated in PHP, the AST would need to tokenize the strings)
  • Removing comments and HTML
  • Resolving the file inclusions (not for dynamic variable inclusion of course, but it's working with define names and static names)

The preprocessor is available here: preproc.zip

Tuesday, October 16 2007

Stuck at data-flow? Do box-modeling!

Since yesterday, I'm working on a data-flow problem. I need to model a function and I should do all the data-flow process. Well, that's kinda long if I have to do that on all functions and especially I will never use much of the information I would generate by analyzing the tree associated to the function (local variables etc.). So what the point of doing that? None.

I was stuck at this point, didn't find a good way to model a function (entry parameters, global calls etc.) so I thought of reasoning as a crystal ball. I can see what it is, but it's kinda blurry :) I am now modeling a function as inputs and outputs, only in terms of functions and global variables interaction. By this, I should be able to see the possible interaction of the given function on the system. Hope it's gonna work well!

Wednesday, October 10 2007

Working around security metrics...

I'm not gonna write a long entry about Security Metrics, but since I've been working on this for a couple of weeks now, I have some thoughts. Evaluating the security of a source code is actually pretty hard. Even if I'm sure there is a lot of source code security metrics out there, it's often (I guess) hard to compute. Basically, you will need to know lots of things about the source code then, you need an engine working on the AST , data-flow etc.

This is what I've done for a couple of months, an engine which is working on XML AST, generated by yaxx (this is the same engine that I use to do source code modifications, obfuscations, etc.).

With Vadim Okun, we had the idea of computing the "size" of the security in a source code. The idea is pretty simple and we are aware that this is limited to implementation flaws and not design flaws for now. The "size" of the security is the number of inputs going to sinks.

The inputs have to be taken in the large sense, these are in fact all the variable that are derivate from direct inputs. Here is a simple example of the variable diffusion:

$a = $_GET['foo'];
$b = htmlentities($a);
echo $b;

We are here counting $a and $b since $b is a modification of $a which is a direct input. We are using the same methodologies for all possible modification (concatenation, cast, etc.).

Once we know these variables, we are counting the ones that are going to sinks. The sinks are a list of function such as 'echo', 'mysql_query', 'fopen', and so on. Our list of sinks is directly coming from the PHP-SAT project. In the previous example, the metric result is 1 since there is only one sink 'echo' where a derivate input is going to.

And here we are, this is a fairly simple (in the idea, not the implementation) way to evaluate the possible security problems that you can have in your source code. We are going to try and evaluate this metric on different open source project (wordpress, joomla, mediawiki etc.). I'm sure this is really incomplete: first because we are only counting the security problems that are coming from inputs but also because it really depends on the programmer (his style of programming).

An other example is available here: smetric.pdf

Next Improvements

For the revised version, the first add would be to count the output validation problems. But for that purpose, I need a stronger data-flow analysis which would analyze in function definitions (not done yet). Then, I will be able to trace everything coming from supposed secure sources (databases, resources, local files, etc.) to sinks. Maybe the weight of such flows would be different than the first one (input to sink)...

Tuesday, September 4 2007

Source Code Obfuscation

Source Code Obfuscation is actually a powerful tool for testers. Whether you use it to obfuscate your bytecode (Java, .NET etc.) or increasing the code complexity of your current source code.

Working at SAMATE we are also playing, tweaking, testing, stressing source code analyzers. And now you see the relation. I'm writing a source code obfuscater in order to increase the complexity of our test cases and see if the tools are still doing well.

Thus, I was able (with good documentation, and yaxx) to create one. It currently only add control flow complexity (and of course renaming classes, functions and variables).

Some words on obfuscation

You may have heard about obfuscation in a sense of making the code unreadable for users. This is not what I'm interested in. I want to modify the actually source code, adding some information in it, some tests... I need the outputs of the original program and the obfuscated one to be the same, otherwise we cannot consider the source code as being equivalent.

So for example if I do:

if (var == 0) { 
  echo 0;
}

I will have the same behavior with this source code:

x = some_value;
if (var == 0 or x*x < 0) {
  echo 0;
}

Even though they have the same output, the second one is more complicated since it adds one other test.

Test case wise example

To see what it exactly does given this original source code

<?php
	$b = 0;
	$c = "Salut";
	$len = strlen($c);

	function fct($a) {
		return $a . "_1\n";
	}
	
	class T {
		function foo() {
			echo "test\n";
		}
	}
	echo fct(0);
	echo T::foo();
?>

...and by choosing one of my control flow obfuscating pattern

<?php
	class rand_class_name {
		function rand_func_name_2($rand_name_1) {
			return $rand_name_1 + 1;
		}
	}

	function rand_func_name_1($rand_name_2) {
		return $rand_name_2 + 1;
	}
	
	if (rand_func_name_1(0) > 0 && rand_class_name::rand_func_name_2(0)) {
		$enter_the_new_statement;
	}
?>

To have this result:

<?php
	function HXvE5Plwxp0RSoQM ( $ZMfP98Az96Rq67j6 ) {
		return $ZMfP98Az96Rq67j6 + 1 ;
	}
	class TF03COvMuzXRQcCK {
		function Ltghf3a0McCI8RaZ ( $V309os5vQo15ak9b ) {
			return $V309os5vQo15ak9b + 1 ;
		}
	}
	$b = 0 ;
	$c = "Salut" ;
	$len = strlen ( $c ) ;
	function fct ( $a ) {
		return $a . "_1\n" ;
	}
	class T {
		function foo ( ) {
			echo "test\n" ;
		}
	}
	if ( HXvE5Plwxp0RSoQM ( 0 ) > 0 && TF03COvMuzXRQcCK :: Ltghf3a0McCI8RaZ ( 0 ) ) {
		echo fct ( 0 ) ;
	}
	if ( HXvE5Plwxp0RSoQM ( 0 ) > 0 && TF03COvMuzXRQcCK :: Ltghf3a0McCI8RaZ ( 0 ) ) {
		echo T :: foo ( ) ;
	}

?>

How it actually works

First of all, the engine only works on Abstract Syntax Tree (AST) in order to do powerful manipulation and code refactoring. The idea is to take a couple of transformation patterns (the second source code is in fact a complicated one), and fitting this patterns with the original source code.

The patterns are meta code. You can see that they are in PHP using some names such as $rand_name_1 etc. this means that the engine will generate one unique name for each of them and replace it before the actual refactoring.

Select what I want to obfuscate is not a real problem, but for now I only selected the top statements and will apply the whole modifications to each of them.

A little schema explaining a little how it works is available here: schema_obfuscation.png

What's next

The applied control flow obfuscating pattern is on of the many I do have for now (many more to come), and I guess this is kinda promising, lots of interesting studies should come now.

Currently the tools is only for PHP but I should make it general by using my own AST nodes names and then be able to do code transformation on C, C++, Java etc.

There is no release of the tool (written in C++) right now, I will wait until it's more than correct and clean. I also need to do data obfuscation (using indirections etc.). The program will of course be public and free for everybody when it's gonna be ready.

Wednesday, August 29 2007

I now understand why it's difficult!

Okay, I know for the halting problem etc. Some theoretical stuff... But now that I'm working on one, I have to say:

Damn! That so complicated to do a source code scanner!

The dataflow is a real pain in the ass, and we know that it's impossible to have a real and full dataflow. But well, we need to do some. The dataflow is more complicated theoretically but what about the control flow? No really easier! I mean... that's easier but there are so many things to understand, so many patterns to recognize in order to build the model of the source code... And I'm not even talking about inter procedural stuff, multi-file source code etc.

So, I'd like to apologize to "I don't remember who are these people" but some source code scanners are good :) Well... for the moment! I'm really waiting for to see more high-tech stuff and AI in these kind of programs...

Anyway, I'm currently building a core engine working on a AST tree generated by yaxx (XML version). I have two short terms targets:

  • Real Obfuscation (from one source code to an equivalent with a different control flow... yes, not only rename the variables, functions, classes etc.)
  • A variable tracer (tool for pen-tester: $_GET['foo'] -> ($foo <- htmlentities()) -> echo or this kind of stack...)

Wednesday, June 20 2007

PHP Source Code Security Scanners: Pixy

I already talked about source code scanners for PHP, and even run a simple test between SWAAT and PHP-SAT. Today, a new toy has been released: Pixy, so I decided to make it pass the test. The first test is really basic, having a quite small php source code with a bunch of possible faults: tests.php

So, you find the output of the tool here: out.pixy.result.txt

I first have to say that it's normal that the tool doesn't catch the header injection stuff, os command injection etc. it doesn't claim to do that. Pixy claims to find the Cross-Site Scripting and the SQL Injection. On that point, I would say pretty good job guy!
The tool catch all the possible Cross-Site Scripting in the echo functions, doesn't warn for the persistent XSS (line 34, the bad html injection would be inserted into the SQL database, if there is no output validation, there are Persistent XSS).

Even better on the SQL Injection where it found every thing I tagged as true-positive.

To conclude, I will definitely keep an eye on this tool which looks promising to me, I will also continue working on the PHP-SAT security configuration in order to make a solid vulnerability disclosure system.

Thursday, May 24 2007

PHP Source Code Security Scanners basic test

For quite a long time now, I've been playing with lots of different black-box tools: commercial or not, mine or not. Months ago, I developed Crystal, a plugin for Grabber which does the link between the black-box engine in Grabber and a PHP Source Code Security Analyzer: PHP-SAT . At the time, it was the only advanced PHP SCSA I could find on the web, so I used it without really testings I admit.
That's for the story, few days ago, on #webappsec (irc.freenode.org), Larholm told me about SWAAT a new (at least, for me) PHP SCSA (and not only PHP actually). At the time, I didn't have time to try it; but today, I took the time to compare PHP-SAT and SWAAT with a test which can be view as a quite-exhaustive-basic-flaw-checker (it means that there is maybe 6 different vulnerabilities with variants and false positive/true positive check implementation).
You can see the PHP test file here: tests.php

The result of the two runs can be find here: php-sat-test-output.phps and swaat-output.html
How to read the reports:

  • SWAAT: HTML file with table for each type of vulnerabilities, it will report multiple lines (each line is a vulnerability). If there is a /* fase */ in that line, then, this is a false positive.
  • PHP-SAT: PHP-SAT takes the PHP source code and transform it by adding some information. For the vulnerability report you will have to look for the Malicious Code Vulnerability (MCV). Other report are more quality oriented.


I will not spend time to explain the difference of the tools but the tools don't really have the same goal (even if we can use them for the same utilization). Well, with the default configuration of both tools, SWAAT is really better! But as for many Source Code Security Analyzers, the configuration is really important, so I would mitigate my conclusion on these tools, I really need to dive into the configuration of that two tools and redo the tests.

I <3 Bots!