We are starting a new blog post series here on the ModSecurity site called "Advanced Feature of the Week" where we will be highlighting many of ModSecurity's really cool capabilities.  These are the features that seldom used or fully understood by the average ModSecurity user however can provide detection of sophisticated attacks if used properly.  It is our goal with these blog posts to help shed light on these unique features and to provide some real-world, in-the-trenches gotchas for successful usage of these features.

This blog post series will have the following major topic sections -

1) ModSecurity Reference Manual Information

Provide reference manual data.

2) Use Within the OWASP Core Rule Set (CRS)

Outline if/when/how the CRS is utilizing this feature.

3) So What?

Will provide some context as to why you as a user should even care about this capability.  What advanced attack/vulnerability is this attempting to catch.

This week's feature is on the use of the @validateByteRange operator.

SecRule ARGS:text "@validateByteRange 10, 13, 32-126"

ASCII Byte Range Chart

Use of @validateByteRange in the OWASP ModSecurity CRS (from the end of the modsecurity_crs_20_protocol_violations.conf file) -

#
# Restrict type of characters sent
# NOTE In order to be broad and support localized applications this rule
# only validates that NULL Is not used.
#
# The strict policy version also validates that protocol and application 
# generated fields are limited to printable ASCII. 
#
# -=[ Rule Logic ]=-
# This rule uses the @validateByteRange operator to look for Nul Bytes.
# If you set Paranoid Mode - it will check if your application use the range 32-126 for parameters.
#
# -=[ References ]=-
# http://i-technica.com/whitestuff/asciichart.html
#

SecRule ARGS|ARGS_NAMES|REQUEST_HEADERS|!REQUEST_HEADERS:Referer "@validateByteRange 1-255" \
 "phase:2,rev:'2.0.8',pass,nolog,auditlog,msg:'Invalid character in request',id:'960901',tag:'PROTOCOL_VIOLATION/EVASION',tag:'WASCTC/WASC-28',tag:'OWASP_TOP_10/A1',tag:'OWASP_AppSensor/CIE3',tag:'PCI/6.5.2',severity:'4',t:none,t:urlDecodeUni,setvar:'tx.msg=%{rule.msg}',tag:'http://i-technica.com/whitestuff/asciichart.html',setvar:tx.anomaly_score=+%{tx.notice_anomaly_score},setvar:tx.protocol_violation_score=+%{tx.notice_anomaly_score},setvar:tx.%{rule.id}-PROTOCOL_VIOLATION/EVASION-%{matched_var_name}=%{matched_var}"

SecRule TX:PARANOID_MODE "@eq 1" "chain,phase:2,rev:'2.0.8',pass,nolog,auditlog,msg:'Invalid character in request',id:'960018',tag:'PROTOCOL_VIOLATION/EVASION',tag:'WASCTC/WASC-28',tag:'OWASP_TOP_10/A1',tag:'OWASP_AppSensor/CIE3',tag:'PCI/6.5.2',severity:'4',t:none,t:urlDecodeUni,tag:'http://i-technica.com/whitestuff/asciichart.html'"
 SecRule REQUEST_URI|REQUEST_BODY|REQUEST_HEADERS_NAMES|REQUEST_HEADERS|!REQUEST_HEADERS:Referer|TX:HPP_DATA \
 "@validateByteRange 32-126" \
 "t:none,t:urlDecodeUni,setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+%{tx.notice_anomaly_score},setvar:tx.protocol_violation_score=+%{tx.notice_anomaly_score},setvar:tx.%{rule.id}-PROTOCOL_VIOLATION/EVASION-%{matched_var_name}=%{matched_var}"

%u3008scr%u0131pt%u3009%u212fval(%uFF07al%u212Frt(%22XSS%22)%u02C8)%u2329/scr%u0131pt%u2A

〈scrıpt〉ℯval(＇alℯrt(”XSS”)ˈ)〈/scrıpt〉

<script>eval('alert("XSS")')</script>

So, this brings us to today's advanced ModSecurity feature - @validateByteRange.  By restricting the allowed character byte ranges, you can help to identify when unexpected character code points are used.  If the payload above is sent, you should receive an alert message similar to the following -

Message: Found 3 byte(s) in REQUEST_URI outside range: 32-126. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_20_protocol_violations.conf"] [line "251"] [id "960018"] [rev "2.0.6"] [msg "Invalid character in request"] [severity "WARNING"] [tag "PROTOCOL_VIOLATION/EVASION"] [tag "WASCTC/WASC-28"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE3"] [tag "PCI/6.5.2"] [tag "http://i-technica.com/whitestuff/asciichart.html"]

ModSecurity Community,

We will be hosting a ModSecurity happy hour during Black Hat USA. It is open to anyone who contributes, uses or wants to learn more about the project. You'll also get a chance to meet the members of Trustwave's SpiderLabs team that will be supporting the ModSecurity project moving forward. 

• What: ModSecurity Happy Hour
• Where: munchbar @ Caesar's Palace LV (next to Sports Bar & Pure Night Club)
• When: Wednesday, July 28th, 2010 - 4:00-6:00 PM

There will be appetizers, drinks, and Nintendo Wii action!

Look forward to meeting you next week!

Senior Vice President, SpiderLabs

Trustwave

Just wanted to let everyone know that if you are headed to Blackhat USA 2010 this summer in
Las Vegas, we have just added a 1-day workshop on the day before the Briefings start -

In the workshop, we will be mainly discussing the "Virtual Patching" concept of using a
WAF (ModSecurity in this case) and we will use the OWASP WebGoat app as the target.  In
the workshop, we will talk virtual patching theory and then have hands-on labs where we
will show how to use Mod to virtually patch the various WebGoat lessons.  As a side note -
we will also have a section on the new CRS v2.0 when discussing negative security models. 
So, if you want to come and dive into the deep-end of the pool and have fun using some of
ModSecurity's advanced features (such as Lua and Content Injection) then sign-up now!

Brian Rectanus and I hope to see you all in Vegas!!!  :)

More Than A Blocking Device

Unfortunately, most people have pigeon-holed WAFs as only "Attack Blocking Devices" and that is but one use-case option.  This Blog post will highlight another interesting use-case/benefit of a web application firewall - capturing full HTTP transactional logging when web applications generate error conditions.  Due to the fact that a WAF is constantly monitoring live communications between clients and web applications, it is able to identify if or when a web application generates errors.  This may of course be caused by an actual attack in the reqest payload, however I can speak from personal experience in working with both Breach customers and the ModSecurity users in stating that many times applications bomb out due to either a back-end communication problem or from abnormal, non-malicious client input.

Why Are These Errors Occuring?

A typical scenario is that a normal client is interacting with web application and they receive a 500 level responst status code from the web application and they have no idea why.  If this were to happen to a large number of clients and/or if these were reported to the site owners, the next step would be for administrators to try and trouble-shoot and figure out what happened to cause the failure.  The sad reality is that web application exception handling and logging mechanisms are sorely lacking.  How is an administrator supposed to conduct analysis to identify what happened when all they have are terse or cryptic error message such as the following Apache error_log entry - 

[Sat Jan 11 08:52:42 2009] [notice] child pid 24906 exit signal Segmentation fault (11)

Review of the access_log data shows the following -

202.99.11.91 - [11/Jan/2009:08:52:42 +0200] "POST /service/soap/GetItemRequest HTTP/1.1" 500 21364 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.13) Gecko/20080325 Fedora/2.0.0.13-1.fc8 Firefox/2.0.0.13"

This request seems to match up with the time stamp of the error_log entry, however unfortunately, the request uses the POST request method which means that there was data passed in the request body that is not logged by the Common Log Format (CLF).  Good luck figuring out what caused the problem with this data... 

Detailed server-side error and debug logging mechanisms are commonly disabled due to performance/disk space reasons, in favor of allowing the web application to include detailed stack dump data within the error pages presented to clients.  This is not an ideal situation as non-malicious clients probably won't know what to do with these technical details, while malicious clients may leverage the data to fine tune an attack.

Capturing Critical Transaction Data

The audit logging capabilties of web application firewalls allow them to address the two deficiencies outlined above: 1) Capturing the full request data, and 2) Capturing the full application response.  By inspecting the full transactional data, it can be determined if the inbound request was malicious or not and any detailed application error message can be reviewed.

Shortening Trouble-shooting Time

In my previous life, I was the head of operational security for a government client's web applications.  The approach that I found successful when introducing a web application firewall to operational personnel was to demonstrate to them how the auditing capabilities could help them to expedite their trouble-shooting steps.  Believe me, when your job requirements are tied to system uptime, this can be a life saver.

Helping QA/Security Testing 

If the cause of the application error was traced to some abnormality in the client's request, then it is a good idea to utilize that same request data as a part of future QA unit tests to ensure that the updated software correctly handles it. 

The web application firewall audit logs that capture "real" attacks lauched at your site are also very useful for security testing future software.  If you have ever been responsible for QA/Security testing of web applications, then you are familiar with the with the challenges of developing a comprehensive set of attack or abuse test cases.  These audit logs can help your organization create a repository of attack data.

Data Sharing Is The Key

They key take away from this post is that web application firewalls gather a wealth of information that could be utilized by other business units.  Therefore, it is critical that processes are setup to share this information.

Back in May, at AppSec OWASP in Ghent, I listened to Alexander Meisel (who was presenting on behalf of OWASP Germany) talk about best practices for web application firewall deployment. The interesting talk was backed by a larger document, which was only available in German at the time (I don't read German). The translated version is now available, and I am happy to say that it is as interesting as I thought it would be.

As you may be aware, I am leading the Web Application Firewall Evaluation Criteria (WAFEC) project, where we outline the questions that need to be asked when evaluating WAFs. WAFEC is quite technical, and I've always felt that we've skipped one step, addressing the technical questions too soon. Best Practices: Use of Web Application Firewalls addresses some of those things that people ask before they go to the evaluation stage. Here are some of topics covered by this document:

1. Roles that need to be established internally in order to support WAF deployment.
2. Suitability of applications for protection via web application firewalls
3. A mapping of WAF features against common security issues (what WAFs can and cannot do).

Overall, it's a very refreshing read and a step in the right direction.

Download:

• Best Practices: Use of Web Application Firewalls (PDF, 155KB)

My list of web application firewall use cases continues to involve. I've decided to shuffle things somewhat:

1. I am going to remove the "Network building blocks" use case because that is really a feature of reverse proxies. If a WAF happens to be implemented on top of a reverse proxy that it will also inherit all of its benefits (and downsides, I should mention). In retrospective, I thought the two (reverse proxies and WAFs) should really stay separate.
2. I am going to add "Learning" as a new use case. As I was looking at the previous list of use cases it stroke me that I was failing to convey the ability of web application firewalls to understand what is going on, and this is what learning does.

Note: This post is part of the Web Application Firewall Concepts series.

In case you missed it, Breach Security has teamed up with WhiteHat Security so that their Sentinel scanning service will automatically create custom ModSecurity rules for certain classes of vulnerabilities that they find in your web applications.  This works with both open source ModSecurity installations and with the commercial M1100 appliance.  If your initial reaction to this is not "Wow, that is cool!" then you probably have never been in the operational security position of having to protect public web applications.  In order to paint a better picture of why this is a pretty slick integration, let me provide you with some background.

As I mentioned in my previous post - What's the Score of the Game - I feel that one of areas where organizations are failing, with regards to web application security, is that there is a lack of communication between the following three groups: Development teams (who are running source code reviews), InfoSec teams (who are running vulnerability scans) and Operational Security teams (who are running web application firewalls).  These three teams each have unique perspectives on the vulnerabilities of the webapps and they should share their data with each other.

Speaking from my own personal experience, I used to lead an operational security team for a federal government customer.  I was charged with defending the public web applications and had built some home-grown ModSecurity WAFs to allow me to implement virtual patches for identified vulnerabilities while the development teams tried to address the root causes.  Unfortunately, much of my time was spent simply tracking down information about the vulnerability.  Either the vulnerability scanning team did not always provide OpSec with the results or the development teams didn't want to provide details about their "Ugly Baby".  So, I would get an initial statement that application X has an SQL Injection issue but with no actionable details (what host, url and parameter).

When I did track the technical information down, the next step was to analyze the details to see if it provided enough information for me to create an appropriate filter.  This was hit and miss, especially if the vulnerability scans were not tuned or if the secure code review consultant didn't understand how to abstract out and explain how a remote client could exploit the issue.  The point is that I spent a fair amount of time in the research phase.

When I did get enough information, I then had to create some ModSecurity rules and run through some testing to ensure that it functioned as expected and did not deny any legitimate traffic.  I could then deploy the virtual patch in production in a logging-only mode until we could schedule a re-scan.  At that point I could switch it into a blocking mode.

When considering the whole "Time to Fix" concept, the process I was going through was much faster then the actual source code fixing route, however it was still manually intensive and took a fair amount of time.  This is where I believe that the real value of the Sentinel + ModSecurity integration shows by automatically creating these custom ModSecurity virtual patches, we are solving two big problems -

1. Shrinking the time to fix - the process is expedited as the WAF analyst does not need to manually research, create and test the virtual patch, and
2. Increased confidence in blocking - The virtual patch is a targeted negative security filter that will not block legitimate traffic.

One other added benefit is that many organizations do not necessarily have technical staff with the required skillset to properly create ModSecurity virtual patches.  With this integration, you don't have to have a ModSecurity guru on staff to create the rules.  It will very interesting as Whitehat Security starts to track the "Time to Fix" metrics of their clients and to see how the customers who have ModSecurity installed fair against those that are using traditional code change processes!

If you care about the PCI standard, you should head over to my personal blog, where I have published a summary of the clarifications made by the PCI Council regarding Requirement 6.6 (code reviews and application firewalls).

A data model is the foundation of web application monitoring and, thus, key to successful utilisation of web application firewalls. We don't get to design the model; we can only deduct it from the information provided to us from the underlying technology. What we can do is build on it, and, for that reason, it is very important to understand what we have to work with.

An ideal model is one that helps structure the information available to us, allows us to enrich it with additional pieces of data and generally helps us raise events based on the information it contains.

The major parts of a web application monitoring data model are as follows:

• Connection - corresponds to one TCP connection.
• Request - corresponds to one HTTP request.
• Response - corresponds to one HTTP response.
• IP Address - the IP address of the client, retrieved from the TCP connection.
• Session - application session.
• User - authenticated user; in most cases this translates to the application user, but some sites still use HTTP authentication, and some might use both.
• Site - perhaps more accurately called Protection domain, or Application. None of these terms is perfect, but I generally prefer to use Site. In our model, Site refers either to the functionality behind an entire domain name (e.g. www.example.com), or only a subset of one (www.example.com/forums/).
• Country - the country where the request originates.
• City - the city where the request originates.
• Custom - any number of custom attributes. For example, you might want to have different policies for different departments within your organisation. To achieve this, you will map client IP addresses to department names, which you will then use to determine policies.

Most of the components are easy to construct, mapping from the structures used in programming, but there are a few places where the technology does not support the view, or where what we are given is not what we want to see:

• Some work is needed to be able to distinguish sessions. There are different session identifier techniques to consider (e.g. in the URI, in a parameter, in a cookie). While there is a number of platforms that have standardised session management, there is also a large number of applications using their own schemes, so in general some custom work will be needed.
• More so in the case of user identification. Building on session identification one needs to identify a successful login event in the traffic in order to determine the session username.
• The IP address may not be accurate. It may be that of an intermediary, and not of the client itself. Such cases can sometimes be identified (as is the case with HTTP proxies) , but not always (e.g. if a transparent HTTP proxy is used). The problem is that, unless you control the proxy, you can only rely on the IP address you got from the TCP stack; the information extracted from HTTP requests headers is not to be trusted.

Note: This post is part of the Web Application Firewall Concepts series.

After some deliberation, I have decided to add Continuous security assessment as a standalone item on my web application firewall use cases list. Although some could argue the functionality is already covered in the Web intrusion detection and prevention section, it is not obvious—you have to understand this field in order to know it is there.

Continuous security assessment is not specific to web application firewalls—it's been used for the network layer for years—but web application firewalls are more useful for web applications (than IDS tools are for network applications), simply because there's essentially one data format to analyse (if you can call a bunch of loosely related specifications used by web applications "one" data format). With web applications, you get to see clear picture of application input and output. Therefore, with proper parsing and analysis in place, it's almost like having a direct communication channel with the application itself.

The focus here is not on the attacks and other suspicious client behaviour, which comes out of the stock IDS/IPS functionality, but on everything else:

• application defects,
  
• information leakage,
  
• configuration errors,
  
• change control
  

and so on. The advantage is that you can detect some very subtle issues, only possible because of the depth of analysis.

Just as an extreme example, there are quite a few web applications out there where SQL injection (less often) and XSS (surprisingly common) exist by design—their clients are allowed and expected to send such low-level stuff and have it executed by the server. These types of problem can be detected early and with little effort, and because assessment never stops, you get the alert as soon as they are manifested.

Note: This post is part of the Web Application Firewall Concepts series.