ModSecurity Blog: Web Application Firewalls

Just wanted to let everyone know that if you are headed to Blackhat USA 2010 this summer in
Las Vegas, we have just added a 1-day workshop on the day before the Briefings start -

http://www.blackhat.com/html/bh-us-10/training/bh-us-10-training_RB-WAFVirtPatch.html

In the workshop, we will be mainly discussing the "Virtual Patching" concept of using a
WAF (ModSecurity in this case) and we will use the OWASP WebGoat app as the target.  In
the workshop, we will talk virtual patching theory and then have hands-on labs where we
will show how to use Mod to virtually patch the various WebGoat lessons.  As a side note -
we will also have a section on the new CRS v2.0 when discussing negative security models. 
So, if you want to come and dive into the deep-end of the pool and have fun using some of
ModSecurity's advanced features (such as Lua and Content Injection) then sign-up now!

Brian Rectanus and I hope to see you all in Vegas!!!  :)

More Than A Blocking Device

Unfortunately, most people have pigeon-holed WAFs as only "Attack Blocking Devices" and that is but one use-case option.  This Blog post will highlight another interesting use-case/benefit of a web application firewall - capturing full HTTP transactional logging when web applications generate error conditions.  Due to the fact that a WAF is constantly monitoring live communications between clients and web applications, it is able to identify if or when a web application generates errors.  This may of course be caused by an actual attack in the reqest payload, however I can speak from personal experience in working with both Breach customers and the ModSecurity users in stating that many times applications bomb out due to either a back-end communication problem or from abnormal, non-malicious client input.

Why Are These Errors Occuring?

A typical scenario is that a normal client is interacting with web application and they receive a 500 level responst status code from the web application and they have no idea why.  If this were to happen to a large number of clients and/or if these were reported to the site owners, the next step would be for administrators to try and trouble-shoot and figure out what happened to cause the failure.  The sad reality is that web application exception handling and logging mechanisms are sorely lacking.  How is an administrator supposed to conduct analysis to identify what happened when all they have are terse or cryptic error message such as the following Apache error_log entry - 

[Sat Jan 11 08:52:42 2009] [notice] child pid 24906 exit signal Segmentation fault (11)

Review of the access_log data shows the following -

202.99.11.91 - [11/Jan/2009:08:52:42 +0200] "POST /service/soap/GetItemRequest HTTP/1.1" 500 21364 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.13) Gecko/20080325 Fedora/2.0.0.13-1.fc8 Firefox/2.0.0.13"

This request seems to match up with the time stamp of the error_log entry, however unfortunately, the request uses the POST request method which means that there was data passed in the request body that is not logged by the Common Log Format (CLF).  Good luck figuring out what caused the problem with this data... 

Detailed server-side error and debug logging mechanisms are commonly disabled due to performance/disk space reasons, in favor of allowing the web application to include detailed stack dump data within the error pages presented to clients.  This is not an ideal situation as non-malicious clients probably won't know what to do with these technical details, while malicious clients may leverage the data to fine tune an attack.

Capturing Critical Transaction Data

The audit logging capabilties of web application firewalls allow them to address the two deficiencies outlined above: 1) Capturing the full request data, and 2) Capturing the full application response.  By inspecting the full transactional data, it can be determined if the inbound request was malicious or not and any detailed application error message can be reviewed.

Shortening Trouble-shooting Time

In my previous life, I was the head of operational security for a government client's web applications.  The approach that I found successful when introducing a web application firewall to operational personnel was to demonstrate to them how the auditing capabilities could help them to expedite their trouble-shooting steps.  Believe me, when your job requirements are tied to system uptime, this can be a life saver.

Helping QA/Security Testing 

If the cause of the application error was traced to some abnormality in the client's request, then it is a good idea to utilize that same request data as a part of future QA unit tests to ensure that the updated software correctly handles it. 

The web application firewall audit logs that capture "real" attacks lauched at your site are also very useful for security testing future software.  If you have ever been responsible for QA/Security testing of web applications, then you are familiar with the with the challenges of developing a comprehensive set of attack or abuse test cases.  These audit logs can help your organization create a repository of attack data.

Data Sharing Is The Key

They key take away from this post is that web application firewalls gather a wealth of information that could be utilized by other business units.  Therefore, it is critical that processes are setup to share this information.

Back in May, at AppSec OWASP in Ghent, I listened to Alexander Meisel (who was presenting on behalf of OWASP Germany) talk about best practices for web application firewall deployment. The interesting talk was backed by a larger document, which was only available in German at the time (I don't read German). The translated version is now available, and I am happy to say that it is as interesting as I thought it would be.

As you may be aware, I am leading the Web Application Firewall Evaluation Criteria (WAFEC) project, where we outline the questions that need to be asked when evaluating WAFs. WAFEC is quite technical, and I've always felt that we've skipped one step, addressing the technical questions too soon. Best Practices: Use of Web Application Firewalls addresses some of those things that people ask before they go to the evaluation stage. Here are some of topics covered by this document:

1. Roles that need to be established internally in order to support WAF deployment.
2. Suitability of applications for protection via web application firewalls
3. A mapping of WAF features against common security issues (what WAFs can and cannot do).

Overall, it's a very refreshing read and a step in the right direction.

Download:

• Best Practices: Use of Web Application Firewalls (PDF, 155KB)

My list of web application firewall use cases continues to involve. I've decided to shuffle things somewhat:

1. I am going to remove the "Network building blocks" use case because that is really a feature of reverse proxies. If a WAF happens to be implemented on top of a reverse proxy that it will also inherit all of its benefits (and downsides, I should mention). In retrospective, I thought the two (reverse proxies and WAFs) should really stay separate.
2. I am going to add "Learning" as a new use case. As I was looking at the previous list of use cases it stroke me that I was failing to convey the ability of web application firewalls to understand what is going on, and this is what learning does.

Note: This post is part of the Web Application Firewall Concepts series.

In case you missed it, Breach Security has teamed up with WhiteHat Security so that their Sentinel scanning service will automatically create custom ModSecurity rules for certain classes of vulnerabilities that they find in your web applications.  This works with both open source ModSecurity installations and with the commercial M1100 appliance.  If your initial reaction to this is not "Wow, that is cool!" then you probably have never been in the operational security position of having to protect public web applications.  In order to paint a better picture of why this is a pretty slick integration, let me provide you with some background.

As I mentioned in my previous post - What's the Score of the Game - I feel that one of areas where organizations are failing, with regards to web application security, is that there is a lack of communication between the following three groups: Development teams (who are running source code reviews), InfoSec teams (who are running vulnerability scans) and Operational Security teams (who are running web application firewalls).  These three teams each have unique perspectives on the vulnerabilities of the webapps and they should share their data with each other.

Speaking from my own personal experience, I used to lead an operational security team for a federal government customer.  I was charged with defending the public web applications and had built some home-grown ModSecurity WAFs to allow me to implement virtual patches for identified vulnerabilities while the development teams tried to address the root causes.  Unfortunately, much of my time was spent simply tracking down information about the vulnerability.  Either the vulnerability scanning team did not always provide OpSec with the results or the development teams didn't want to provide details about their "Ugly Baby".  So, I would get an initial statement that application X has an SQL Injection issue but with no actionable details (what host, url and parameter).

When I did track the technical information down, the next step was to analyze the details to see if it provided enough information for me to create an appropriate filter.  This was hit and miss, especially if the vulnerability scans were not tuned or if the secure code review consultant didn't understand how to abstract out and explain how a remote client could exploit the issue.  The point is that I spent a fair amount of time in the research phase.

When I did get enough information, I then had to create some ModSecurity rules and run through some testing to ensure that it functioned as expected and did not deny any legitimate traffic.  I could then deploy the virtual patch in production in a logging-only mode until we could schedule a re-scan.  At that point I could switch it into a blocking mode.

When considering the whole "Time to Fix" concept, the process I was going through was much faster then the actual source code fixing route, however it was still manually intensive and took a fair amount of time.  This is where I believe that the real value of the Sentinel + ModSecurity integration shows by automatically creating these custom ModSecurity virtual patches, we are solving two big problems -

1. Shrinking the time to fix - the process is expedited as the WAF analyst does not need to manually research, create and test the virtual patch, and
2. Increased confidence in blocking - The virtual patch is a targeted negative security filter that will not block legitimate traffic.

One other added benefit is that many organizations do not necessarily have technical staff with the required skillset to properly create ModSecurity virtual patches.  With this integration, you don't have to have a ModSecurity guru on staff to create the rules.  It will very interesting as Whitehat Security starts to track the "Time to Fix" metrics of their clients and to see how the customers who have ModSecurity installed fair against those that are using traditional code change processes!

If you care about the PCI standard, you should head over to my personal blog, where I have published a summary of the clarifications made by the PCI Council regarding Requirement 6.6 (code reviews and application firewalls).

A data model is the foundation of web application monitoring and, thus, key to successful utilisation of web application firewalls. We don't get to design the model; we can only deduct it from the information provided to us from the underlying technology. What we can do is build on it, and, for that reason, it is very important to understand what we have to work with.

An ideal model is one that helps structure the information available to us, allows us to enrich it with additional pieces of data and generally helps us raise events based on the information it contains.

The major parts of a web application monitoring data model are as follows:

• Connection - corresponds to one TCP connection.
• Request - corresponds to one HTTP request.
• Response - corresponds to one HTTP response.
• IP Address - the IP address of the client, retrieved from the TCP connection.
• Session - application session.
• User - authenticated user; in most cases this translates to the application user, but some sites still use HTTP authentication, and some might use both.
• Site - perhaps more accurately called Protection domain, or Application. None of these terms is perfect, but I generally prefer to use Site. In our model, Site refers either to the functionality behind an entire domain name (e.g. www.example.com), or only a subset of one (www.example.com/forums/).
• Country - the country where the request originates.
• City - the city where the request originates.
• Custom - any number of custom attributes. For example, you might want to have different policies for different departments within your organisation. To achieve this, you will map client IP addresses to department names, which you will then use to determine policies.

Most of the components are easy to construct, mapping from the structures used in programming, but there are a few places where the technology does not support the view, or where what we are given is not what we want to see:

• Some work is needed to be able to distinguish sessions. There are different session identifier techniques to consider (e.g. in the URI, in a parameter, in a cookie). While there is a number of platforms that have standardised session management, there is also a large number of applications using their own schemes, so in general some custom work will be needed.
• More so in the case of user identification. Building on session identification one needs to identify a successful login event in the traffic in order to determine the session username.
• The IP address may not be accurate. It may be that of an intermediary, and not of the client itself. Such cases can sometimes be identified (as is the case with HTTP proxies) , but not always (e.g. if a transparent HTTP proxy is used). The problem is that, unless you control the proxy, you can only rely on the IP address you got from the TCP stack; the information extracted from HTTP requests headers is not to be trusted.

Note: This post is part of the Web Application Firewall Concepts series.

After some deliberation, I have decided to add Continuous security assessment as a standalone item on my web application firewall use cases list. Although some could argue the functionality is already covered in the Web intrusion detection and prevention section, it is not obvious—you have to understand this field in order to know it is there.

Continuous security assessment is not specific to web application firewalls—it's been used for the network layer for years—but web application firewalls are more useful for web applications (than IDS tools are for network applications), simply because there's essentially one data format to analyse (if you can call a bunch of loosely related specifications used by web applications "one" data format). With web applications, you get to see clear picture of application input and output. Therefore, with proper parsing and analysis in place, it's almost like having a direct communication channel with the application itself.

The focus here is not on the attacks and other suspicious client behaviour, which comes out of the stock IDS/IPS functionality, but on everything else:

• application defects,
  
• information leakage,
  
• configuration errors,
  
• change control
  

and so on. The advantage is that you can detect some very subtle issues, only possible because of the depth of analysis.

Just as an extreme example, there are quite a few web applications out there where SQL injection (less often) and XSS (surprisingly common) exist by design—their clients are allowed and expected to send such low-level stuff and have it executed by the server. These types of problem can be detected early and with little effort, and because assessment never stops, you get the alert as soon as they are manifested.

Note: This post is part of the Web Application Firewall Concepts series.

There are many reasons to use a web application firewall. Most people tend to focus on prevention and blocking when the term is brought up, but that is just one of the possible uses. Three years ago, almost to the day, I wrote this post to argue how one needs a WAF to serve as a part of the overall defence strategy. My opinion remains unchanged today, but I have since expanded the list of use cases for web application firewalls. Here they are:

1. Web intrusion detection and prevention. Applying the principles of external traffic monitoring (IDS) and prevention (IPS) to HTTP and the related technologies, which are used to build web applications. Through your WAF you will look for signs of generic web application attacks (negative security model), or deploy a learning engine to construct a model of your site and reject all invalid traffic, not just attacks (also known as positive security model).
2. Continuous security assessment. The idea with this case is to emphasize the fact web application firewalls actually understand web applications pretty well. Armed with this knowledge, they can do more than detect attacks; they can observe from signs of weaknesses, information leaks, configuration errors, and similar problems before an attempt to exploit them is made.
3. Virtual (or just-in-time) patching. When you need to deal with a specific problem in your web site, which exists either in your code or in the code of the product you are relying on. The focus in this use case is on writing custom rules to deal with custom issues.
4. HTTP traffic logging and monitoring. Do you know what is actually going on in your web applications? Who are your users and how are they using your systems? Are you being attacked and how?
5. Learning. My original list did not include learning as a WAF feature but, in retrospective, I realised I was failing to convey a very important functionality and differentiator over intrusion detection and prevention systems. The point is that WAFs understand web applications, so in some cases they are able to move away from intrusion detection (as in traffic monitoring) to a default deny security model.
6. Web application hardening. If you deploy your WAF as a reverse proxy then you can get it to modify the traffic stream to fix some of the design faults of your application or the HTTP protocol.

I will expand on each use case in my future posts.

Update (10 July 2008): I've decided to remove the "Network building blocks" use case from the list. While I still believe this is an important use case, I now think the usefulness should be attributed to reverse proxies. And, if a WAF happens to be implemented as a reverse proxy, that it will include the advantages and the disadvantages of that model.

Note: This post is part of the Web Application Firewall Concepts series.

I went through all my ModSecurity Blog posts yesterday, partly to admire myself for blogging consistently for almost 5 years and partly to understand what is that I talked about during this time. While I knew that most of my posts were pretty technical (after all, I did start my new blog to focus on other things) imagine my surprise when I realised I didn't properly cover the one thing this blog is supposed to cover: web application firewalls!  The emphasize is on the word "properly": I provided a great deal of technical information but not enough content that would explain why one would deploy a web application firewall and how. This stuff had went into my conference talks and the Web Application Firewall Evaluation Criteria project, but I forgot to discuss the topics here. In an effort to fix this I am starting a series of blog posts called Web Application Firewall Concepts. Each post will be reasonably brief and cover one aspect of the technology, and I will continually update this post to serve as a table of contents. Posts in this series:

1. Use Cases
   1. Web intrusion detection and prevention
   2. Continuous Security Assessment
   3. Virtual (or just-in-time) patching
   4. HTTP traffic logging and monitoring
   5. Learning
   6. Web application hardening
2. Deployment models
   1. Inline
   2. Out of line
   3. Embedded
3. Data Model
   1. Model construction
   2. Persisting information across requests
   3. Distinguishing sessions
   4. Distinguishing users
4. Analysis Model
   1. Negative security
   2. Positive security
   3. Anomaly scoring
   4. Learning
   5. Evasion
   6. Impedance mismatch
5. Traffic logging
6. Special protection techniques
   1. Cookie protection
   2. Cross-Site Request Forgery
   3. Brute force attacks
   4. Denial of Service attacks
   5. PDF UXSS protection