ModSecurity Trustwave
This blog has moved! Please update your
bookmarks to http://blog.spiderlabs.com/modsecurity/.

ModSecurity Blog: August 2006

ModSecurity Cookie and Link Protection Patch

A significant event occurred on the mod-security-users mailing list in July: a large code contribution was made by Daniel Fernndez Bleda and Carles Bonamusa Prez from Internet Security Auditors. The patch, made against ModSecurity 1.9.4, adds cookie and link protection using hashing and encryption. The patch is now available for download from the ModSecurity web site. Please note that the code is not yet production ready and that you should use it for testing purposes only. I expect to merge this code into the official code base soon (targeting the next stable release after 2.0.0).

ModSecurity Performance Tip

I was asked recently to investigate performance of an ModSecurity installation in order to see if there's room for improvement. This particular installation is used to defend against blog comment spam. It has a large number of simple rules, all regular expressions, that worked against the Referer header and the request parameters. I used the opportunity to figure out an answer to the question that has been bugging me for years: is a configuration with many simple regular expressions faster than a configuration with a single complex regular expression? I'll spare you the details of my tests, here the verdict:

A single rule with a complex regular expression is significantly faster than multiple rules with simple regular expressions.

So instead of:

SecFilterSelective VAR KEYWORD1
SecFilterSelective VAR KEYWORD2
SecFilterSelective VAR KEYWORD3

from now on you want to write:

SecFilterSelective VAR "(KEYWORD1|KEYWORD2|KEYWORD3)"

This is true for ModSecurity 1.9.x and for the last released version of ModSecurity 2.x. I am yet to investigate if there's any room for optimisation in the ModSecurity code, of the difference comes from the overhead in the regular expression library.

(Update 18 August 2006) My post to the mod-security-users mailing lists sparked an interesting discussion. Ryan rightfully pointed out that it is not a good idea to consolidate unrelated rules together as that increases your maintenance costs and prevents you from assigning unique IDs to individual rules. We then went on to discuss potential ways of making the consolidated regular expressions more readable and come up with the idea of using multiple lines together with PCRE-style comments (which only work with Apache 2.x):

SecFilterSelective VAR "(\
KEYWORD1(?# a comment)|\
KEYWORD2(?# a comment)|\
KEYWORD3(?# a comment))"

Apache Reverse Proxy Memory Consumption Observations

Last week I spent some time stress-testing Apache 2.2.3 configured to work as a reverse proxy. I discovered (actually, re-discovered would be more accurate) two issues worth sharing.

  1. Memory consumption of an Apache process will steadily increase as the number of processed requests rises. This is very easy to see if you send thousands of requests per second, with each request going to the same process. This has to be either a memory leak or a memory fragmentation issue. To deal with this you need to recycle processes before they become too large (and cause the operating system to start swapping). The MaxRequestsPerChild directive is meant to help with this. By setting its value to something other than zero (which means "unlimited") you are telling Apache to shut down every process that goes over the limit. No problems there. Except that it's where the second problem comes in.

  2. The MaxRequestsPerChild directive does not work as the name suggests. Apache does not count requests - it counts *connections*. This creates a problem if you have persistent connections enabled in your configuration - you don't know how many requests will come over a connection. It is probably safe to assume the number will not be large in most cases but you won't know if someone will try to abuse this problem and force a large number of requests over a single connection (e.g. by using a specially programmed script). To be on the safe side you need to divide your ideal MaxRequestsPerChild value with the MaxKeepAliveRequests value. This will prevent the Apache processes from growing too large. But there's a side effect - Apache will now recycle its worker processes more often. As your final step you need to make sure there are enough idle processes around (using MinSpareServers) to jump in as soon as an active process goes down. Yo need to have a few of these processes because there is a performance penalty associated with the creation of a new process and because Apache creates new processes at a rate of one every second.

ModSecurity 1.9.x Performance Testing

You can tell that I am too busy when I take almost three months to blog about something interesting and useful to a wider audience. This is one of those occasions. Earlier this year Adrian Asher, Head of Security at Betfair, kindly invited me to visit their performance testing lab. Consequently in May I spent one day testing the performance of ModSecurity 1.9.x working in the reverse proxy mode.

When you are using ModSecurity embedded performance is usually measured in milliseconds, as the time needed for ModSecurity to do its work. The most relevant information here is the relative speed. If the application you are protecting spends over 100 milliseconds processing every dynamic request, a millisecond here or there won't make a difference. Those deploying ModSecurity in the reverse proxy mode, on the other hand, usually wish to know how many requests can be processed every second. They need to know if the reverse proxy setup is going to be fast enough for their needs.

Testing of a reverse proxy setup requires one to execute a large number of transactions per second, hopefully simulating real-life conditions. In addition to the reverse proxy box, you need to simulate a bunch of clients and a bunch of web servers. This can be problematic because, even with modest hardware, one cannot easily saturate the reverse proxy box using only one client machine. In addition, you typically need more than one web server machine. That's why I jumped at the opportunity to visit Betfair - they have specialised testing equipment, namely an Avalanche and Reflector pair. This (expensive) equipment will take care of the client and web servers part, allowing you to focus on the box that sits in the middle.

1_9_performance-small.pngThe reverse proxy box we used was a Sun v20Z machine (single AMD Opteron 244 running at 1.8 GHz, 1 GB RAM, Gigabit network) running Red Hat Enterprise Linux (kernel 2.6.9-11 EL). One day sounds like a lot of time but it really isn't for things like this. Although we worked hard throughout the day we only managed to test a dozen of different (software) configurations and no tweaking of the machine or the operating system was performed. You can see the results on the attached graph (red = Apache alone, yellow = ModSecurity with 25 rules, green = ModSecurity with approximately 180 rules - the entire certified rules set). So although it is certainly not comprehensive this test is probably close enough to what you would experience in a typical deployment. Some things to note:

  • ModSecurity performance continues to not be a problem. (Unless you shoot yourself in the foot by deploying thousands of rules, that is.)
  • The CPU was the bottleneck in all our tests. Hence simply increasing the CPU power would yield better results.
  • The overhead of ModSecurity by itself is very low. Almost all of time ModSecurity uses is spent processing regular expressions, meaning you are in full control over the performance.
  • Apache w/ModSecurity running with a generous rule set can push through at least 1500 requests per second with latency under one millisecond running on modest hardware.

One thing I noticed during my tests is that most of the rules are actually very simple patterns yet they are executed through the regular expression engine. One of the things I will do very soon is introduce other pattern matching algorithms to ModSecurity 2. That will probably help us increase the performance soon. More information coming soon!

Calendar

November 2010
Sun Mon Tue Wed Thu Fri Sat
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30

Feeds

Atom Feed

Search

Categories

Recent Entries

Archives