ModSecurity Blog

ModSecurity 2.6 will likely be the last branch before ModSecurity 3.  The 2.6 branch will concentrate on polishing up the current 2.5 feature set, performance, ease of use, supporting arbitrary character sets, and better documentation.  I'll be posting 2.6 development releases periodically for users to test and comment on.  So that you know what is planned, I am publishing the roadmap for 2.6 below.

• Enhancements
  
  • Enhance persistent storage:
    • Allow relative changes to counters to be atomic.
    • Optimize storage and retrieval.
  • Enhance audit log sanitization:
    • Allow partial data sanitization.
    • Allow the RESPONSE_BODY to be sanitized.
  • Enhance external auditing/alerting (mlogc):
    • Optimize data queuing to lower RAM usage.
    • Allow sensor metrics to be sent to the console.
    • Add connection throttling which can be dictated by the console.
  • Allow for more flexibility when writing complex rules:
    • Add the ability to determine which targets previously matched.
    • Straighten out how non-disruptive actions work with chained rules.
• Performance
  • Add a high performance IP address/network matching operator capable of large lists.
  • Further tune the detection engine.
  • Enhance the detection engine cache with faster lookups.
  • Expose more performance metrics through the audit log.
• Ease of Use
  • Enhance the build process:
    
    • Allow static linking of dependencies on UNIX like OSes.
    • Allow better support for non-gcc compilers.
  • Allow for fully automate updates of the Core Rules (and others).
• Arbitrary Character Sets
  • Introduce decoding and validating of various character sets.
  • Allow for enhancing and expanding decoding in future versions.
• Documentation
  • Write more/better examples.
  • Enhance ModSecurity internals documentation.
  • Better document the different modes of operation.

Quite a few people have asked about the performance differences between using the regular expression (@rx) operator and using the phrase match (@pm or @pmFromFile) operator.  Lately, I have been working on better methods of gathering performance statistics and want to share my findings.

The phrase match operator was added to enhance performance of matching a larger number of static phrases.  For instance, you may want to look for a list of spam phrases in ARGS:

With the @rx Operator and Simple OR Expression:

SecRule ARGS \
  "@rx ambien|cyclen|cyclobenzaprine|paxil|phendimetrazine|phentamine|phentermine|viagra|viagara" \
  "phase:2,deny,status:403,log,alertlog,t:lowercase,msg:'Medical Spam Detected'"

With the @rx Operator and an Enhanced Expression (see Optimizing Regular Expressions):

SecRule ARGS \
  "@rx (?:p(?:hen(?:t(?:er|a)m|dimetraz)ine|axil)|cycl(?:obenzaprine|en)|viaga?ra|ambien)" \
  "phase:2,deny,status:403,log,alertlog,t:lowercase,msg:'Medical Spam Detected'"

With the @pm Operator:

SecRule ARGS \
  "@pm ambien cyclen cyclobenzaprine paxil phendimetrazine phentamine phentermine viagra viagara" \
  "phase:2,deny,status:403,log,alertlog,t:lowercase,msg:'Medical Spam Detected'"

To compare the performance of each of these, I used a utility I build for unit testing to execute the operators 10,000 times and took the average execution time.  I generated a randomized list of 1,000 phrases between 2 and 8 characters in length.  The following chart compares the processing time in milliseconds of each of the above operator types using from 0 to 200 phrases from the randomized phrase list.  Each operator uses the same set of phrases.  Note that this is the processing time of only the operator (no overhead of transformations, alerts or other aspects of executing the rule).

As you can see, there is quite a difference in performance.  The basic @rx operator performance decreases linearly as more phrases are added.  While the optimized @rx operator performance does come close to leveling out, it is still slower than the @pm operator and the rule itself is quite difficult to read and maintain.  In contrast, the @pm operator uses a constant, extremely low amount of processing time while the rule is easy to read and maintain.

While the @pm operator performs well, it is not very flexible and thus has limited use cases.  The operator can only use static phrases (no patterns) and cannot currently be anchored (meaning it will match a partial target and/or a partial phrase in the target).  If you need patterns or need to anchor matches to the beginning and/or end of the target or word boundaries, then you still must use the @rx operator.  In this case an optimized regular expression is the way to go if you need the rule to perform well, which is why these are used in the Core Rules.

In my previous post, in which I was commenting on the OWASP AppSec agenda, I forgot to mention the party. What was I thinking?! Breach Security is throwing a cocktail party on May 20th, which is the last training day of the conference. Instead of just inviting you to come and have a good time, I've decided to rename the party into a ModSecurity user gathering. There, now you have an official reason to come. That, and the fact I will be giving free ModSecurity advice while cocktails last.

So, if you are coming to the OWASP AppSec conference in Ghent, or if you live in or near Ghent, do come to see us. The party will be at:

Vintage Wine Bar
Onderbergen 35, 9000 Ghent Belgium
Phone: 35, 0032 9 223 51 32
www.vintagewinebar.be

We will begin at 6.30pm and end at 8.30pm. Full directions here.

Update (8 May 2008): Please send us an email if you're coming.

Judging from the list of talks alone, it looks like OWASP AppSec Europe in Belgium is going to be a great conference, especially if you are interested in web application firewalls and ModSecurity:

1. I will be giving a talk on web application firewall evaluation (Evaluation Criteria for Web Application Firewalls), which will be based on the Web Application Firewall Evaluation Criteria (WAFEC) project.
2. Christian Folini will be discussing the REMO project (Positive ModSecurity rulesets / Input validation).
3. Alexander Meisel will be talking about how to best use a web application firewall (Best Practices Guide: Web Application Firewalls), which is based on the document of the same name (available in German, as PDF).
4. Mario Heiderich will be promoting the PHPIDS project (PHPIDS Monitoring attack surface activity).

Ofer Shezaf, the Core Rules guru, and Ryan Barnett, the ModSecurity Community Manager will be there (Ofer will be giving his talk about web hacking trends: Trends in Web Hacking Incidents: What's Hot in 2008), as will be Christian Bockermann (it is rumoured), who has been working on some very interesting software related to ModSecurity.

Finally, Ryan is going to be teaching a two-day ModSecurity training course, which will cover a lot of ground, starting from the basics and into the advanced stuff. This is a great-value course, and I urge you to register if you are a ModSecurity user. You will not only find out about stuff you never knew existed in ModSecurity, but we will also give a thorough overview of various web application security issues.

In the recent survey, many people expressed a desire to meet with other ModSecurity users. Our community is large, but it's very diverse and spread geographically, and probably not yet large enough for regular local meetings. It strikes me that OWASP conferences may be a great opportunity for us to meet twice a year—once in Europe, and then the second time in the US. If you will be coming to the conference in Belgium and you are a ModSecurity user, please send me an email. With enough people interested, we may be able to organise a meeting.

If you care about the PCI standard, you should head over to my personal blog, where I have published a summary of the clarifications made by the PCI Council regarding Requirement 6.6 (code reviews and application firewalls).

I've just released an update to ModSecurity Community Console, our free audit log aggregation solution with support for up to 3 ModSecurity sensors. The focus of this release is the support for part K of the ModSecurity audit log format (list of rules that matched in the transaction being recorded), which was added to ModSecurity in version in 2.5. I also used the opportunity to update the documentation, change the Console to work over SSL by default, and improve performance (by dropping a large number of indexes that were just slowing everything down).

We are excited to announce that a ModSecurity 2-day training class has been added to the upcoming OWASP AppSec Europe Conference set for May 19-20 in Belgium.

We are extremely excited that OWASP has added this class to their training offerings as this allows ModSecurity users a chance to for live training, labs and discussions. The public ModSecurity mailing list is great, however nothing can beat a live environment where users can test out new features of ModSecurity 2.5 and have hands-on challenging labs where they can utilize their new ModSecurity virtual patching-fu to try and fix issues in demo buggy web applications :)

It should be a blast and we are looking forward to meeting many ModSecurity users face-to-face.

FYI - for those ModSecurity users who can't make the trip to this OWASP conference, don't despair!  Keep an eye out for an announcement soon about a live 2-day Mod training offering in the US this summer.

I've had a pleasure of participating in ApacheCon Europe in Amsterdam this week. Paradoxically, although I've been involved with the Apache web server for years, this was my first ApacheCon conference ever. Meeting the people I've been exchanging emails with for years was priceless. My presentation (Web Intrusion Detection with ModSecurity, available for download now) was well received, judging from the comments I have received afterwards, and from what both Rich and Nick have written in their blogs.

We've just completed a move to the new blogging platform. Not only is our blog now visually integrated with the rest of the web site, but we've enabled comments and trackbacks, which should make our blogging from now on much more interesting. The old posts are going to remain where they are. The old feed has been updated to that of the new blog, so if you are a subscriber you don't need to do anything.

A data model is the foundation of web application monitoring and, thus, key to successful utilisation of web application firewalls. We don't get to design the model; we can only deduct it from the information provided to us from the underlying technology. What we can do is build on it, and, for that reason, it is very important to understand what we have to work with.

An ideal model is one that helps structure the information available to us, allows us to enrich it with additional pieces of data and generally helps us raise events based on the information it contains.

The major parts of a web application monitoring data model are as follows:

• Connection - corresponds to one TCP connection.
• Request - corresponds to one HTTP request.
• Response - corresponds to one HTTP response.
• IP Address - the IP address of the client, retrieved from the TCP connection.
• Session - application session.
• User - authenticated user; in most cases this translates to the application user, but some sites still use HTTP authentication, and some might use both.
• Site - perhaps more accurately called Protection domain, or Application. None of these terms is perfect, but I generally prefer to use Site. In our model, Site refers either to the functionality behind an entire domain name (e.g. www.example.com), or only a subset of one (www.example.com/forums/).
• Country - the country where the request originates.
• City - the city where the request originates.
• Custom - any number of custom attributes. For example, you might want to have different policies for different departments within your organisation. To achieve this, you will map client IP addresses to department names, which you will then use to determine policies.

Most of the components are easy to construct, mapping from the structures used in programming, but there are a few places where the technology does not support the view, or where what we are given is not what we want to see:

• Some work is needed to be able to distinguish sessions. There are different session identifier techniques to consider (e.g. in the URI, in a parameter, in a cookie). While there is a number of platforms that have standardised session management, there is also a large number of applications using their own schemes, so in general some custom work will be needed.
• More so in the case of user identification. Building on session identification one needs to identify a successful login event in the traffic in order to determine the session username.
• The IP address may not be accurate. It may be that of an intermediary, and not of the client itself. Such cases can sometimes be identified (as is the case with HTTP proxies) , but not always (e.g. if a transparent HTTP proxy is used). The problem is that, unless you control the proxy, you can only rely on the IP address you got from the TCP stack; the information extracted from HTTP requests headers is not to be trusted.

Note: This post is part of the Web Application Firewall Concepts series.