ModSecurity Blog: Web Application Firewalls

If you care about the PCI standard, you should head over to my personal blog, where I have published a summary of the clarifications made by the PCI Council regarding Requirement 6.6 (code reviews and application firewalls).

A data model is the foundation of web application monitoring and, thus, key to successful utilisation of web application firewalls. We don't get to design the model; we can only deduct it from the information provided to us from the underlying technology. What we can do is build on it, and, for that reason, it is very important to understand what we have to work with.

An ideal model is one that helps structure the information available to us, allows us to enrich it with additional pieces of data and generally helps us raise events based on the information it contains.

The major parts of a web application monitoring data model are as follows:

• Connection - corresponds to one TCP connection.
• Request - corresponds to one HTTP request.
• Response - corresponds to one HTTP response.
• IP Address - the IP address of the client, retrieved from the TCP connection.
• Session - application session.
• User - authenticated user; in most cases this translates to the application user, but some sites still use HTTP authentication, and some might use both.
• Site - perhaps more accurately called Protection domain, or Application. None of these terms is perfect, but I generally prefer to use Site. In our model, Site refers either to the functionality behind an entire domain name (e.g. www.example.com), or only a subset of one (www.example.com/forums/).
• Country - the country where the request originates.
• City - the city where the request originates.
• Custom - any number of custom attributes. For example, you might want to have different policies for different departments within your organisation. To achieve this, you will map client IP addresses to department names, which you will then use to determine policies.

Most of the components are easy to construct, mapping from the structures used in programming, but there are a few places where the technology does not support the view, or where what we are given is not what we want to see:

• Some work is needed to be able to distinguish sessions. There are different session identifier techniques to consider (e.g. in the URI, in a parameter, in a cookie). While there is a number of platforms that have standardised session management, there is also a large number of applications using their own schemes, so in general some custom work will be needed.
• More so in the case of user identification. Building on session identification one needs to identify a successful login event in the traffic in order to determine the session username.
• The IP address may not be accurate. It may be that of an intermediary, and not of the client itself. Such cases can sometimes be identified (as is the case with HTTP proxies) , but not always (e.g. if a transparent HTTP proxy is used). The problem is that, unless you control the proxy, you can only rely on the IP address you got from the TCP stack; the information extracted from HTTP requests headers is not to be trusted.

Note: This post is part of the Web Application Firewall Concepts series.

After some deliberation, I have decided to add Continuous security assessment as a standalone item on my web application firewall use cases list. Although some could argue the functionality is already covered in the Web intrusion detection and prevention section, it is not obvious—you have to understand this field in order to know it is there.

Continuous security assessment is not specific to web application firewalls—it's been used for the network layer for years—but web application firewalls are more useful for web applications (than IDS tools are for network applications), simply because there's essentially one data format to analyse (if you can call a bunch of loosely related specifications used by web applications "one" data format). With web applications, you get to see clear picture of application input and output. Therefore, with proper parsing and analysis in place, it's almost like having a direct communication channel with the application itself.

The focus here is not on the attacks and other suspicious client behaviour, which comes out of the stock IDS/IPS functionality, but on everything else:

• application defects,
  
• information leakage,
  
• configuration errors,
  
• change control
  

and so on. The advantage is that you can detect some very subtle issues, only possible because of the depth of analysis.

Just as an extreme example, there are quite a few web applications out there where SQL injection (less often) and XSS (surprisingly common) exist by design—their clients are allowed and expected to send such low-level stuff and have it executed by the server. These types of problem can be detected early and with little effort, and because assessment never stops, you get the alert as soon as they are manifested.

Note: This post is part of the Web Application Firewall Concepts series.

There are many reasons to use a web application firewall. Most people tend to focus on prevention and blocking when the term is brought up, but that is just one of the possible uses. Three years ago, almost to the day, I wrote this post to argue how one needs a WAF to serve as a part of the overall defence strategy. My opinion remains unchanged today, but I have since expanded the list of use cases for web application firewalls. Here they are:

1. Web intrusion detection and prevention. Applying the principles of external traffic monitoring (IDS) and prevention (IPS) to HTTP and the related technologies, which are used to build web applications. Through your WAF you will look for signs of generic web application attacks (negative security model), or deploy a learning engine to construct a model of your site and reject all invalid traffic, not just attacks (also known as positive security model).
2. Continuous security assessment. The idea with this case is to emphasize the fact web application firewalls actually understand web applications pretty well. Armed with this knowledge, they can do more than detect attacks; they can observe from signs of weaknesses, information leaks, configuration errors, and similar problems before an attempt to exploit them is made.
3. Virtual (or just-in-time) patching. When you need to deal with a specific problem in your web site, which exists either in your code or in the code of the product you are relying on. The focus in this use case is on writing custom rules to deal with custom issues.
4. HTTP traffic logging and monitoring. Do you know what is actually going on in your web applications? Who are your users and how are they using your systems? Are you being attacked and how?
5. Network building blocks. This use case is not often a primary motivator for WAF deployment, but if you're already deploying a reverse proxy to serve as a HTTP switch/router, making the device security-aware is the way to go.
6. Web application hardening. If you deploy your WAF as a reverse proxy then you can get it to modify the traffic stream to fix some of the design faults of your application or the HTTP protocol.

I will expand on each use case in my future posts.

Note: This post is part of the Web Application Firewall Concepts series.

I went through all my ModSecurity Blog posts yesterday, partly to admire myself for blogging consistently for almost 5 years and partly to understand what is that I talked about during this time. While I knew that most of my posts were pretty technical (after all, I did start my new blog to focus on other things) imagine my surprise when I realised I didn't properly cover the one thing this blog is supposed to cover: web application firewalls!  The emphasize is on the word "properly": I provided a great deal of technical information but not enough content that would explain why one would deploy a web application firewall and how. This stuff had went into my conference talks and the Web Application Firewall Evaluation Criteria project, but I forgot to discuss the topics here. In an effort to fix this I am starting a series of blog posts called Web Application Firewall Concepts. Each post will be reasonably brief and cover one aspect of the technology, and I will continually update this post to serve as a table of contents. Posts in this series:

1. Use Cases
   1. Web intrusion detection and prevention
   2. Continuous Security Assessment
   3. Virtual (or just-in-time) patching
   4. HTTP traffic logging and monitoring
   5. Network building blocks
   6. Web application hardening
2. Deployment models
   1. Inline
   2. Out of line
   3. Embedded
3. Data Model
   1. Model construction
   2. Persisting information across requests
   3. Distinguishing sessions
   4. Distinguishing users
4. Analysis Model
   1. Negative security
   2. Positive security
   3. Anomaly scoring
   4. Learning
   5. Evasion
   6. Impedance mismatch
5. Traffic logging
6. Special protection techniques
   1. Cookie protection
   2. Cross-Site Request Forgery
   3. Brute force attacks
   4. Denial of Service attacks
   5. PDF UXSS protection

One of the challenges facing organizations that need to increase the security of their web applications is to concretely provide appropriate "Return On Investment" (ROI) for procurement justification. Organizations can only allocate a finite amount of budget towards security efforts therefore security managers need to be able to justify any commercial services, tools and appliances they want to deploy. As most people who have worked in information security for an extended period of time know, producing tangible ROI for security efforts that address business driver needs is both quite challenging and critically important.

The challenge for security managers is to not focus on the technical intricacies of the latest complex web application vulnerability or attack. C-level Executives do not have the time, and in most instances the desire, to know the nuances of an HTTP Request Smuggling attack. That is what they are paying you for! Security managers need to function as a type of liaison where they can take data from the Subject Matter Experts (SMEs) and then translate that into a business value that is important to the C-level Executive.

One, almost guaranteed, pain point to most executives are vulnerability scan reports that are presented by auditors. The auditors are usually being brought in from and reporting to a higher-level third party (be it OMB in the Government or PCI for Retail). Executives like to see "clean vulnerability scan reports." While this will certainly not guarantee that your web application is 100% secure, it can certainly help to prove the counter-argument. And to make matters worse, nothing is more frustrating to upper Management than auditor reports list repeat vulnerabilities that either never go away or pull the "Houdini" trick (they disappear for awhile only to suddenly reappear). Sidebar - see Jeremiah Grossman's Blog post for examples of this phenomenon. These situations are usually attributed to breakdowns in the Software Development Life Cycle (SDLC) where code updates are too time consuming or the change control processes are poor.

This is one of the best examples of where a Web Application Firewall can prove its ROI.

At Breach Security, we receive many inbound calls from prospects who are interested in WAF technology but are lacking that "Big Stick" that helps convince upper management to actually make the purchase. The best scenario we have found is to suggest a Before and After; comparison of a vulnerability scan report while they are testing the WAF on their network. The idea is to deploy the WAF in block mode and then initiate a rescan of a protected site. The difference in the reduction of findings is an immediate, quantitative ROI.

Here is a real example. One of our current customers followed this exact roadmap and this is a summary (slightly edited to remove sensitive information) of the information they sent back to us:

Our WAF is installed and running. I have tested its impact on www.example.com and it is operating very admirably. This morning I had the vulnerability scanning team run an on-demand scan to test the efficacy of the appliance, and I was very impressed with the results. Our previous metrics for www.example.com in the last scan were 64 vulnerabilities, across all outside IP addresses (including www.example.com, example2.com, example3.com, etc.) and with the Breach appliance in place, the metric from today's scan was 5 vulnerabilities, with details:

- High vulnerabilities dropped from 38 to 0

- Medium vulnerabilities dropped from 12 to 0

- 1 low vulnerability remains due to simply running a web server (we will eliminate this via exception)

- 1 low vulnerability due to a file/folder naming convention that is typical and attracts unwanted attacks (will be eliminated via rule addition)

Bear in mind that I have applied the appliance with a basic (almost strictly out-of-the-box) configuration and ruleset to protect only www.example.com (192.168.1.100 in the report), and the 35 warnings that remain are for the other websites, and would similarly disappear when protected by the appliance. In my opinion, this was a very successful test that indicates the effectiveness of the appliance.

So, looking at the report after the WAF was in place, the www.example.com web site removed 38 high and 12 medium vulnerabilities and left only 2 low ones (which are really just informational notices). That is pretty darn good and that was just with the default, generic detection ModSecurity rule set! Hopefully this information has helped to provide a possible use-case testing scenario to show tangible ROI of a WAF.

In a future post, I will discuss how custom WAF rule sets can be implemented to address more complex vulnerability issues identified not by a scanner but by actual people who have performed a web assessment/pentest.

Some time ago I decided to start a new blog, a place where I would be able to address the topics that are not ModSecurity specific. I felt the ModSecurity Blog has its purpose and a happy audience; I didn't want for it to lose the focus. Today I made my first proper post at this new blog:

There is a long-running tradition in the web application firewall space; every year we say: "This year is going to be the one when web application firewalls take off!" So far, every year turned out to be a bit of a disappointment in this respect. This year feels different, and I am not saying this because it's a tradition to do so. Recent months have seen a steady and significant rise in the interest in and the recognition of web application firewalls. But why is it taking so long?

To read more please continue to "Tide is turning for web application firewalls".

There have been many heated debates in web security circles around the wording of the PCI 1.1 Requirement 6.6 section and it all centers around the semantics of the word "either". The Requirement 6.6 states the following:

"6.6 Ensure that all web-facing applications are protected against known attacks by applying either of the following methods:

• Having all custom application code reviewed for common vulnerabilities by an organization that specializes in application security
• Installing an application layer firewall in front of web-facing applications.

Note: This method is considered a best practice until June 30, 2008, after which it becomes a requirement."

The word either in this context indirectly implies that these two option are functionally equivalent and therefore the user could choose either one and receive the same level of security benefit. Well, as you might guess, this sparked a vast amount of debate from webappsec folks as to the amount of security protection that is gained from a source code review vs. using a web application firewall. To make the waters even murkier, you even had web vulnerability scanner/service vendors asking the PCI counsel if users could run these tools vs. conducting an actual source code review. So now users who were attempting to become compliance with PCI section 6.6 weren't sure what they needed to do or what tool or process was the "best" approach...

I believe that an important issue that is being forgotten when people think of "source code review" is that the actual review is only one portion of the overall process. Most people do not factor in the remediation portion of the process. I could probably be convinced that a manual source code review vs. review with a source code analysis tool vs. running web vuln scanner could yield roughly similar results - they identify what the problems are. What about fixing them?

This is the core issue in 6.6 - to implement some sort of remediation to prevent successful web attacks. Why do I believe this? Well, if you refer to the PCI DSS Security Audit Procedures document it outlines how a PCI auditor will evalute each requirement to confirm whether or not the organization is in compliance. Section 6.6 - Testing Procedures - states the following:

"6.6 For web-based applications, ensure that one of the following methods are in place as follows:

• Verify that custom application code is periodically reviewed by an organization that specializes in application security; that all coding vulnerabilities were corrected; and that the application was re-evaluated after the corrections"

As you can see, the goal of this section is to show that not only were vulnerabilities identified but that they were also fixed. So whether or not the vulnerabilities were identified by source code review or a scanner does not seem to be the main issue from PCI but rather was the vulnerability actually fixed??? It is the process of actually remediating the vulnerabilities that is taking entirely too long for organizations, if it happens at all. I mean, how many times does an Authorized Scanning Vendor (ASV) find the exact same vulns showing up in scan after scan? They are quickly showing the customer what/where the problems are but they just can't fix them for a variety of reasons:

• Regression Testing Time; Any source code changes require extensive regression testing in numerous environments which may delay deployment in production by many weeks or even months.
• Fixing Custom Code is Cost Prohibitive; In-house web assessment identifies vulnerabilities in your custom coded web application, however it is too expensive to recode the application.
• Legacy Code/Breaking Functionality; Due to support or business requirements, legacy application code can not be patched as prior installs broke functionality. There may even be licensing issues where the vendor will not allow for changes in the code.
• Outsourced Code; Outsourced applications that would require a new project to fix.
• Certification and Accredidation is a Pain; For government organizations, the C&A process is very time consuming and any changes to the source code would require a new one.

Whatever the reason, current SDLC processes for quickly fixing vulnerabilities found in production are lacking. This brings us to the web application firewall. If you look at the 2nd party of the 6.6 testing procedures, it states this for WAFs –

"Verify that an application-layer firewall is in place in front of web-facing applications to detect and prevent web-based attacks."

Notice that the WAF has to be in block mode! This again, supports the idea of remediation. So, just because an organization deploys a WAF is not enough to comply with requirement 6.6. You need to be blocking attacks (mainly SQL Injection and XSS as they are the only 2 that are considered HIGH severity). It is for these reasons that I believe that the "intent" of 6.6 is geared towards remediation efforts and not just identification tasks.

Virtual Patching is a policy for a web application firewall (in this case ModSecurity) that is able to identify attempts to exploit a specific Website vulnerability. ModSecurity analyzes transactions and intercepts attacks in transit, so malicious traffic never reaches the target Website. The end result is that even if a vulnerability still exists within the application’s source code, the virtual patch will protect against clients attempting to exploit it.

Virtual Patching is an extremely valuable technique that can be used to provide immediate protection against identified vulnerabilities. The trick here, however, is that you first must identify them! You can’t really create a patch if you don’t know what the problem is. There are six main processes that may yield vulnerability information that you can then take action on by virtually patching them:

• Vendor contact (e.g. pre-warning)
• Public disclosure
• Bug report to the development team
• Vulnerability assessment (internal or external)
• Code review
• Security incident

All of these scenarios are somewhat similar in that they all provide vulnerability information in reports of some sort. The only exception is that last one, a Security Incident. This is a unique situation in that there are no ifs, ands or buts involved in the discussions as to whether or not you need to respond to this issue. Any sort of lead time that you may have been counting on for a normal patching process, source code fix or system outage is suddenly thrown out the windows as proper Incident Response steps require you to act immediately. This is where Virtual Patching can prove to be invaluable.

Want a real life example?

In case you missed it, the United Nations (UN) website was recently defaced by a defacement trio known as "KEREM125 M0STED AND GSY." They defaced the site by adding html text to the speeches page. An archived screen shot is located here - notice the text under the "Latest speeches" window. And then here on the specific speech page.

While the details of the specific attack vector can not be 100% confirmed, it is suspected that the attackers used an SQL Injection vulnerability. A software developer named Giorgio Maone chronicled this incident on his Blog site. Maone partly deduced that SQL Injection was the likely attack vector by the missing apostrophe/single-quote in the word "dont" in the defacement text. Single-quotes are normally a key component of creating proper SQL query syntax and it is assumed that attempting to include it in the text would have complicated the SQL Injection attack. Maone also showed that the "statID" parameter for the statments_full.asp page is the most likely candidate for the attack as this URL – "http://www.un.org/apps/news/infocus/sgspeeches/statments_full.asp?statID=105'" – reveals the following DB error message:

ADODB.Recordset.1 error '80004005' 
 
SQLState: 37000
Native Error Code: 8180
SQLState: 37000
Native Error Code: 105
[MERANT][ODBC SQL Server Driver][SQL Server]Unclosed quotation mark before the character string ''.
[MERANT][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared.
 
/apps/news/infocus/sgspeeches/statments_full.asp, line 26 

UN Incident Response Issues

Once the defacement was identified, the main UN site was taken offline and the following message was presented to clients:

This site will be temporarily unavailable due to scheduled maintenance.

After a number of hours, the site came back online. Unfortunately, vulnerability had not been patched as the same error message could be generated. It seemed that there was some sort of basic filter in place that attempted to filter out the single-quote character, however this is not a sufficient fix as there are SQL Injection queries that do not rely on this character. There is also the possibility of bypassing this filter by using the char() function. A short time later, the entire site was offline and presented clients with this message:

The UN website is undergoing urgent maintenance and is currently unavailable. Please check back in a short while.

When the site did come back online the Speeches section was not available and the same old vulnerability was still present...

What was happening?

We can only speculate at this point as to what was happening behind closed doors in the UN Incident Response team meetings, however they obviously had difficulty with addressing the standard "Eradication Phase" of the issue. When the choice is either being totally offline while waiting for the source code to be fixed vs. putting the site back online and monitoring the logs more closely for issues, the latter will always win out.

How a Virtual Patch could have helped

Looking at the URL again, we can narrow down the issue to the statments_full.asp application and specifically to the statID parameter. Looking at the normal, expected values associated with the statID parameter you can see that the data should only be digits. The following ModSecurity Virtual Patch could have been used to fix this issue by implementing a positive security ruleset:

<Location /apps/news/infocus/sgspeeches/statments_full.asp>
SecRule &ARGS "!@eq 1"
SecRule ARGS_NAMES "!^statid$"
SecRule ARGS:statID "!^\d{1,3}$"
</Location>

This rule uses the normal Apache Location directive as a container for the ModSecurity rules. Inside this location, we are enforcing the following three rules:

• The statments_full.asp page will only accept 1 argument
• The name of the argument must be statID (note that the rule uses all lowercase as there is certain transformation functions being applied to normalize traffic)
• The value of statID can only be 1 to 3 numeric digits

If this rule were in place, the example URL provided above would have been denied with this alert message:

[Wed Jun 13 01:06:37 2007] [error] [client 192.168.15.1] ModSecurity: Access denied with code 403 (phase 2). Match of "rx ^\\\\d{1,3}$" against "ARGS:statID" required. [file "/usr/local/apache/conf/rules/modsecurity_crs_15_customrules.conf"] [line "4"] [hostname "www.un.org"] [uri "/apps/news/infocus/sgspeeches/statments_full.asp?statID=105'"] [unique_id "lCFILsCoD4QAABWcDp4AAAAD"]

This Virtual Patch would have provided instant protection against this issue until the actual source code could have been update or fixed.

In many ways vulnerability remediation is like a Track and Field race and the firing of the starters pistol is the public vulnerability announcement. The goal of the race is to be the first one to either exploit or patch a vulnerability. The participants in the race may include; 1) Organizations running the vulnerable application, 2) Attackers looking to exploit the vulnerabilty manually, or 3) The odds on favorite to win the race - an automated worm program. Oraganizations looking to mitigate or patch their systems are the long-shots to win this race. Let's look at a breakdown of the challenges that organizations face:

Not Hearing the Starter's Pistol

Unfortunately, many organizations don't realize that they are even in a race! This can be attributed poor monitoring of vulnerability alerts. If you aren't signed up on your Vendor's mail-list or you don't have someone checking out US-CERT or the SANS Internet Storm Center (ISC) daily then you are immediately giving the attackers a 50 yard lead in this race...

What Do You Mean We Don't Have The Baton?

If you are running in a relay race, you need to have a baton to pass to each memeber of your team. In this case, the baton is the vendor's security patch. You might be ready, willing and able to start the patching process, however if the vendor doesn't release the patch, you can't really start the race then can you?

Not Getting A Clean Handoff

Each leg of the relay could be though of as a step in the patching process such as installation on a test host, then pushing the patch out to development, then regression testing and finally out to production. As each phase completes its tasks, it then needs to notify the next group and "hand off the baton" so they can move forward with testing. If this doesn't happen, then the patch will never make it to the finish line - which is when the patches are applied to production hosts. I can't tell you how many times I have seen customers who have patches that make through one or two phases but then just seem to fall off the priority list.

Getting Disqualified

In a relay race, if you step outsite of your lane, then can be disqualified. Similarly, if a security patch ever causes any sort of disruption to normal service then the patch is usually not applied. If there are problems during regression testing, then odds are that the security patch will not make it to the finish line. In the end, functionality will always trump security.

Let's Not Pull A Hamstring

Many organizations want to minimize being disqualified so they take a rather slow, methodical approach to the race and decide just to walk it. These are the organizations who only have quarterly downtime for patching. These companies may get a ribbon for participation but they will never win the race.

Don't Have A Lane To Run In

What happens if you are not able to apply any patches at all to your web application? Two valid scenarios may be companies who have outsourced the development of their web application and/or who are using an older version of a COTS product where the vendor is no longer providing patches. What options are left for these companies to compete in this race?

Evening The Odds

So, where does that leave us then? Is there anything that organizations can do the even the playing field in this race? The answer is yes. Virtual Patching can help by providing immediate mitigations to the vulnerability. If an organization were to implement a Virtual Patch on a web application firewall, this will act as a stop-gap measure to prevent remote exploitation of the vulnerability until the actual patch is applied. Using the relay race analogy again, this would be like forcing the attackers to run a steeplechase type of race where there are water pits and 10 ft. tall hurdles in their lane while you are allowed to run a normal race without any obstacles. In this type of scenario, you have a much better chance of beating the attackers to the finish line and protecting your web applications.

Virtual Patching Webcast

If you would like to know more about Virtual Patching, Ivan Ristic and I will be jointly presenting a webcast on this topic very soon -

Date: Wednesday, August 8, 2007
Time: 8AM, Pacific DT
Registration Link