I am excited to announce that the OWASP ModSecurity Core Rule Set (CRS) has completed its official review and has been promoted to a Release Quality Project!  I want to thank both Ivan Ristic and Leonardo Cavallari Militelli who served as project reviewers.

While this is certainly exciting news and will help to further promote ModSecurity and the Core Rule Set, there is still much work left to be done on the project.  We will be expanding the CRS documentation, re-organizing the rules to more accurately categorize the quality of the rules (with relation to false positive potential), as well as, cross-referencing rules with the OWASP AppSensor project.

Keep your eyes open as we are just getting started with the CRS.

We are starting a new blog post series here on the ModSecurity site called "Advanced Feature of the Week" where we will be highlighting many of ModSecurity's really cool capabilities.  These are the features that seldom used or fully understood by the average ModSecurity user however can provide detection of sophisticated attacks if used properly.  It is our goal with these blog posts to help shed light on these unique features and to provide some real-world, in-the-trenches gotchas for successful usage of these features.

This blog post series will have the following major topic sections -

1) ModSecurity Reference Manual Information

Provide reference manual data.

2) Use Within the OWASP Core Rule Set (CRS)

Outline if/when/how the CRS is utilizing this feature.

3) So What?

Will provide some context as to why you as a user should even care about this capability.  What advanced attack/vulnerability is this attempting to catch.

This week's feature is on the use of the @validateByteRange operator.

SecRule ARGS:text "@validateByteRange 10, 13, 32-126"

ASCII Byte Range Chart

Use of @validateByteRange in the OWASP ModSecurity CRS (from the end of the modsecurity_crs_20_protocol_violations.conf file) -

#
# Restrict type of characters sent
# NOTE In order to be broad and support localized applications this rule
# only validates that NULL Is not used.
#
# The strict policy version also validates that protocol and application 
# generated fields are limited to printable ASCII. 
#
# -=[ Rule Logic ]=-
# This rule uses the @validateByteRange operator to look for Nul Bytes.
# If you set Paranoid Mode - it will check if your application use the range 32-126 for parameters.
#
# -=[ References ]=-
# http://i-technica.com/whitestuff/asciichart.html
#

SecRule ARGS|ARGS_NAMES|REQUEST_HEADERS|!REQUEST_HEADERS:Referer "@validateByteRange 1-255" \
 "phase:2,rev:'2.0.8',pass,nolog,auditlog,msg:'Invalid character in request',id:'960901',tag:'PROTOCOL_VIOLATION/EVASION',tag:'WASCTC/WASC-28',tag:'OWASP_TOP_10/A1',tag:'OWASP_AppSensor/CIE3',tag:'PCI/6.5.2',severity:'4',t:none,t:urlDecodeUni,setvar:'tx.msg=%{rule.msg}',tag:'http://i-technica.com/whitestuff/asciichart.html',setvar:tx.anomaly_score=+%{tx.notice_anomaly_score},setvar:tx.protocol_violation_score=+%{tx.notice_anomaly_score},setvar:tx.%{rule.id}-PROTOCOL_VIOLATION/EVASION-%{matched_var_name}=%{matched_var}"

SecRule TX:PARANOID_MODE "@eq 1" "chain,phase:2,rev:'2.0.8',pass,nolog,auditlog,msg:'Invalid character in request',id:'960018',tag:'PROTOCOL_VIOLATION/EVASION',tag:'WASCTC/WASC-28',tag:'OWASP_TOP_10/A1',tag:'OWASP_AppSensor/CIE3',tag:'PCI/6.5.2',severity:'4',t:none,t:urlDecodeUni,tag:'http://i-technica.com/whitestuff/asciichart.html'"
 SecRule REQUEST_URI|REQUEST_BODY|REQUEST_HEADERS_NAMES|REQUEST_HEADERS|!REQUEST_HEADERS:Referer|TX:HPP_DATA \
 "@validateByteRange 32-126" \
 "t:none,t:urlDecodeUni,setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+%{tx.notice_anomaly_score},setvar:tx.protocol_violation_score=+%{tx.notice_anomaly_score},setvar:tx.%{rule.id}-PROTOCOL_VIOLATION/EVASION-%{matched_var_name}=%{matched_var}"

%u3008scr%u0131pt%u3009%u212fval(%uFF07al%u212Frt(%22XSS%22)%u02C8)%u2329/scr%u0131pt%u2A

〈scrıpt〉ℯval(＇alℯrt(”XSS”)ˈ)〈/scrıpt〉

<script>eval('alert("XSS")')</script>

So, this brings us to today's advanced ModSecurity feature - @validateByteRange.  By restricting the allowed character byte ranges, you can help to identify when unexpected character code points are used.  If the payload above is sent, you should receive an alert message similar to the following -

Message: Found 3 byte(s) in REQUEST_URI outside range: 32-126. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_20_protocol_violations.conf"] [line "251"] [id "960018"] [rev "2.0.6"] [msg "Invalid character in request"] [severity "WARNING"] [tag "PROTOCOL_VIOLATION/EVASION"] [tag "WASCTC/WASC-28"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE3"] [tag "PCI/6.5.2"] [tag "http://i-technica.com/whitestuff/asciichart.html"]

I presented on the OWASP ModSecurity Core Rule Set (CRS) Project yesterday here at the AppSec DC conf.  The reception was good and there were a lot of great questions.  For those interested, the PPT has been posted to the project site.  It gives a good overview of the CRS - http://www.owasp.org/index.php/Category:OWASP_ModSecurity_Core_Rule_Set_Project#tab=Presentations_and_Whitepapers

Good news, Blackhat has introduced the concept of Breakout Briefings and I just got confirmation for a ModSecurity User Group meeting! 

So, if you are attending Blackhat, come to the Genoa room, Third Floor, at 10am on July 30th (the 2nd day of BH).  This information should be presented in the Blackhat guide.

At the briefing, I will present the new ModSecurity Core Rule Set v2.0 (that we are going to be releasing at Blackhat).  After the CRS briefing, we will open it up to discussion topics for ModSecurity, as well as, any WASC honeypot topics.

In a previous post I showed how you can use both ModSecurity and Apache together to identify/modify SessionIDs that are missing the HTTPOnly flag.  I received some feedback where people were asking how to accomplish the same thing but for the "Secure" cookie flag which instructs the browser to *only* send the SessionID back over an SSL connection. 

If you are only interested in addressing the missing "Secure" cookie flag, then you can simply take the example from the previous post and edit it slightly to swap out "httponly" with "secure".  If, however, you want to try and address both of these issues together, then you will need to change the rule set approach a bit so that it works correctly.  This is because there are now three different scenarios you have to account for -

• Missing HTTPOnly flag
• Missing Secure flag (if the SessionID is being sent over an SSL connection)
• Missing both HTTPOnly and Secure flags

With this in mind, here is an updated rule set that will handle both missing HTTPOnly and Secure cooking flags.

#

# First we want to capture Set-Cookie SessionID data for later inspection

SecRule RESPONSE_HEADERS:/Set-Cookie2?/ "(?i:(j?sessionid|(php)?sessid|(asp|jserv|jw)?session[-_]?(id)?|cf(id|token)|sid))" "phase:3,t:none,pass,nolog,setvar:tx.sessionid=%{matched_var}"

 

#

# We now check the saved SessionID data for the HTTPOnly flag and set an Apache

# ENV variable if it is missing.

SecRule TX:SESSIONID "!(?i:\;? ?httponly;?)" "phase:3,t:none,setenv:httponly_cookie=%{matched_var},pass,log,auditlog,msg:'AppDefect: Missing HttpOnly Cookie Flag.'"

 

#

# Next we check the saved SessionID data for the Secure flag (if this is an SSL session)

# and set an Apache ENV variable if it is missing.

SecRule SERVER_PORT "@streq 443" "chain,phase:3,t:none,pass,log,auditlog,msg:'AppDefect: Missing Secure Cookie Flag.'"

 SecRule TX:SESSIONID "!(?i:\;? ?secure;?)" "t:none,setenv:secure_cookie=%{matched_var}"

 

#

# The final check is to see if BOTH of the HTTPOnly and Secure cookie flags are missing

# and set an Apache ENV variable if they are missing.

SecRule TX:SESSIONID "!(?i:\;? ?httponly;?)" "chain,phase:3,t:none,pass,log,auditlog,msg:'AppDefect: Missing HttpOnly and Secure Cookie Flag.'"

 SecRule SERVER_PORT "@streq 443" "chain,t:none"

  SecRule TX:SESSIONID "!(?i:\;? ?secure;?)" "t:none,setenv:secure_httponly_cookie=%{matched_var}"

 

#

# This last section executes the Apache Header command to

# add the appropriate Cookie flags 

Header set Set-Cookie "%{httponly_cookie}e; HTTPOnly" env=httponly_cookie

Header set Set-Cookie "%{secure_cookie}e; Secure" env=secure_cookie

Header set Set-Cookie "%{secure_httponly_cookie}e; Secure; HTTPOnly" env=secure_httponly_cookie

These rules will both alert and fix these cookie issues.  You may want to switch the actions to "nolog" so that you are not flooded with alerts.

There are three aspects of the ModSecurity Rule Language we are not very happy with. One comes from a wrong design decision (my own), with further two from constraints of working within the framework of Apache. All three break the principle of the intuitive action being the expected one. I am going to document them here and explain how we are planning to mitigate them in future versions:

1. In a chain starter rule, disruptive actions are processed when the chain matches, but non-disruptive actions are processed when the rule matches. In other words, it is only the disruptive actions that are treated differently in chains, all other action types behave as they would in standalone rules. Have a look at the following:
   
   

   SecRule T1 K1 chain,log,block,setvar:tx.counter=+1
   SecRule T2 K2

   In the example above the counter will be incremented if the first rule matches even if the chain doesn't. The blocking action, although defined with the same rule, would only be processed if both the first rule and the second rule match.
   
   In retrospective, disruptive actions for chains should have been placed with the last rule in a chain, not with the first one. If it is possible to move to that mechanism in the next major version while preserving compatibility with existing configurations we will do that.
   
   
2. SecDefaultAction is valid only for the configuration context in which it is used and is not inherited in child contexts. Configuration contexts are an Apache feature and they come with limitations, one of which is causing this problem.

   SecDefaultAction log,deny
   SecRule T1 K1
   <Location /some/other/path>
   SecRule T2 K2
   </Location>

   In the above example, the first rule blocks, but the second one just uses the ModSecurity defaults and only warns and lets requests through.
   
   In the next major version of ModSecurity (v3) we will handle our configuration ourselves and this problem will probably go away. In fact, the SecDefaultAction directive might be made obsolete in the next major version because we don't like it much. In retrospective, it was a wrong choice too. It is good practice to write rules to be self-contained. That way they will be easier to understand and maintain, and you don't risk configuration errors due to something being changed in the configuration elsewhere.
   
   
3. Configuration contexts other than <VirtualHost> cannot hold phase 1 rules. Again, this is a limitation of the current implementation that relies on Apache for configuration functionality.
   
   Short term (e.g. 2.6), we are planning to see if we can detect phase 1 rules in places where they cannot be run and respond with an configuration error. The problem will go away once we start handling our own configuration.

Using SecRuleRemoveById to handle false positives

The SecRuleRemoveById directive is most often used when ModSecurity users are trying to deal with a false postive situation. Used on its own, it is a global directive that will disable a rule that was specified before it based on its rule id number. While users can technically take this approach and just use SecRuleRemoveById on its own, we caution against this. Just because a rule triggered a false positive match does not mean that the only recourse is to disable the rule entirely! Remember, the rule was created to address a specific security issue so every effort should be made to only disable a rule or make an exception in certain cases.

Limitations of SecRuleRemoveById

The problem is that SecRuleRemoveByID is somewhat limited in its capabilities for selectively disabling rules. One of the common methods of attempting to selectively disable a Mod rule is to nest the SecRuleRemoveById directive inside of an Apache scope location (such as Location) like this -

<Location /path/to/foo.php>
SecRuleRemoveById 950009
</Location>

There currently aren't many other options for using SecRuleRemoveById to disable a rule other than triggering on URI location as shown above. A similar issue was identified with other global directives and was addressed in ModSecurity 2.0 by making it possible to update these settings on a per rule basis by using the "ctl:" action. In future versions of ModSecurit we will implement a "ctl:RemoveById" action to handle this. In the meantime, however, what else can a user do to selectively disable rules bases on arbitrary request data?

Using Transactional Variables (TX)

The approach that am going to discuss is meant as an example only and its usage should be fully considered prior to implementation. I believe that the TX variable is not currently being widely used by ModSecurity users. This may be caused by two main reasons - 1) The Core Rules don't use them, and 2) We don't have proper "use-case" documentation showing how you might use it more effectively. It is with the later issue that I hope this post will help.

Transaction variables are really cool and Ivan explained their general usage in a SecurityFocus interview. Here are the relevant sections -

The addition of custom variables in ModSecurity v2.0 (along with a number of related improvements) marks a shift toward providing a generic tool that you can use in almost any way you like. Variables can be created using variable expansion or regular expression sub-expressions. Special supports exists for counters, which can be created, incremented, and decremented. They can also be configured to expire or decrease in value over time. With all these changes ModSecurity essentially now provides a very simple programming language designed to deal with HTTP. The ModSecurity Rule Language simply grew organically over time within the constraints of the Apache configuration file.

In practical terms, the addition of variables allows you to move from the "all-or-nothing" type of rules (where a rule can only issue warnings or reject transactions) to a more sensible anomaly-based approach. This increases your options substantially. The all-or-nothing approach works well when you want to prevent exploitation of known problems or enforce positive security, but it does not work equally well for negative security style detection. For the latter it is much better to establish a per-transaction anomaly score and have a multitude of rules that will contribute to it. Then, at the end of your rule set, you can simply test the anomaly score and decide what to do with the transaction: reject it if the score is too large or just issue a warning for a significant but not too large value.

What I am about to show is an implementation of this concept.

Enabling/Disabling rules using TX variables

The first step in this process is to update your your modsecurity_crs_15_customrules.conf file to specify which rules will be active. If you aren't familiar with the the modsecurity_crs_15_customrules.conf file and its usage, please see this prior Blog post. The following two entries use the SecAction directive to set two different TX variables -

# Set the enforce variable to 0 to disable and 1 to enable
# Rule ID 950002 is for "System Command Access"
SecAction "phase:1,pass,nolog,setvar:tx.ruleid_950002_enforced=1, \
setvar:tx.ruleid_950002_matched=0"

As the comment text indicates, you can quickly toggle whether or not this rule is active by changing the tx.ruleid_950002_enforced variable to 0. With this directive, every request will have these two TX variables initially set. If you have ever seen any of those nature shows on television where the researchers capture an animal, tag it and then release it back into the wild, we are essentially doing the same thing. We are just "tagging" the current request with some data that will be updated and/or evaluate by later rules.

Altering the Core Rules

The next step in this process is to update the individual Core Rules files to edit the rules so that instead of applying a disruptive action (such as deny), they will only set a new TX variable upon a match. The idea is to decouple the evaluation of the attack pattern in the transaction from the disruptive action application (which will happen in the next step). Here is an example from the modsecurity_crs_40_generic_attacks.conf file for the command access rule -

#
# Command access
#
SecRule REQUEST_FILENAME "\b(?:n(?:map|et|c)|w(?:guest|sh)|cmd(?:32)?|telnet|rcmd|ftp)\.exe\b" \
        "capture,t:htmlEntityDecode,t:lowercase,log,pass,id:'12345',msg:'System Command Access. \
Matched signature <%{TX.0}>',setvar:tx.ruleid_950002_matched=1"

Now, if an inbound request matched this rule, then the tx variable called "ruleid_950002_matched" will be set to "1". This updates the original setting of this variable from the SecAction in the modsecurity_crs_15_customrules.conf file. This rule will also log the detection of this rule to the error_log file.

Evaluating the TX variables for blocking

The next step is to add a new rule to your modsecurity_crs_60_customrules.conf file to actually implement the blocking aspect of this process -

SecRule TX:RULEID_950002_ENFORCED "@eq 1" "chain,t:none,ctl:auditLogParts=+E,deny,log, \
auditlog,status:501,msg:'System Command Access. Matched signature <%{TX.0}>',id:'950002',severity:'2'"
SecRule TX:RULEID_950002_MATCHED "@eq 1"

The above example is a chained rule set where the first line checks to see if this rule should even be evaluated. If the tx value is set to 1 (meaning yes we are evaluating this rule) then it will go to the 2nd part of the chained rule and check to see if the matched TX value is 1 (meaning that the inbound request matched the RegEx check from the modsecurity_crs_40_generic_attacks.conf file). If both of these TX values return true then the entire chained rule matches and the actions on the 1st line is triggered and the request is denied. Here is what the short error_log message would look like -

[Sat Jun 23 18:04:54 2007] [error] [client 192.168.1.103] ModSecurity: Access denied with code 501 (phase 2). \
Operator EQ match: 1. [id "950002"] [msg "System Command Access. Matched signature "] [severity "CRITICAL"] \
[hostname "www.example.com"] [uri "/bin/ftp.exe"] [unique_id "@D6NJMCoD4QAABSNAoMAAAAA"]

What does this approach do for you?

At this point, you may be asking "Ok, how are these rules any different from the Core Rules? Didn't you just make the rules more complex?" It is true that functionally speaking, these new rules work exactly the same as the current Core Rule ID 950002. If client sent a request with one of those OS commands in them then it would be blocked by either rule set.

Advantages of this approach

The advantage of using this approach is that you now have extended flexibility to decided under what circumstances a rule will be evaluate or by which an exception can be made to a rule.

1) You could disable rules in phase:1. With the current approach of SecRuleRemoveById being used inside Apache scope directives, you could only run within phase:2 or beyond. With this approach, you could easily create a rule that runs in phase:1 and evaluates some variable (perhaps a remote IP or something) and then just sets "setvar:tx.ruleid_950002_enforced=0" and it will disable that rule.

2) Besides just deciding on whether or not the rule itself will be evaluated, you could also selectively decide if an inbound request matches the rule or not. Let's say that you keep having a false positive on rule ID 950002 when a client uses a specific web client (user-agent string). You could then easily add a rule to your modsecurity_crs_60_customrules.conf file to check for this user-agent string value and then use set "setvar:tx.ruleid_950002_matched=0" to set the TX variable back to 0 even if the rule had matched in the modsecurity_crs_40_generic_attacks.conf file :) Here is an example rule you would place in the *60* file before the blocking rule -

SecRule REQUEST_HEADERS:User-Agent "^Browser_1234$" \
"phase:2,log,t:none,id:'123456',setvar:tx.ruleid_950002_matched=0"

As you can see, using this approach you have much more flexibility to determine when and where you want to implement an exception to a rule and you can then use "setvar" to easily change the TX variables. This provides you with many more options than using a global directive.

Disadvantages of this approach

1) This approach pretty much goes against the recommendations that I have been promoting previously about trying to limit editing of the Core Rules themselves.

2) This approach also introduces more directives than would normally be present in your configurations. As we have stated in many previous posts, the more rules that you have the higher the impact on performance. This means that for those users who are concerned with performance may not want to use this approach.

Remember, however, that I said that the purpose of this post is simply to present an alternative approach to evaluating requests and to show a use-case example of using TX variables.

NIST has released a new guide on securing Web Services. It is a pretty good read for anyone who is planning to run WS, specifically Appendix A which lists Common WS Attack categories such as:

• Reconnaissance Attacks
• Privilege Escalation Attacks
• Attacks on Confidentiality
• Attacks on Integrity
• Denial of Service Attacks
• Command Injection
• Malicious Code Attacks

Protecting Web Services with ModSecurity

If you compile ModSecurity 2.x with XML support (with libxml2) and activate the libxml2.so file in httpd.conf, you can gain some protection for your WS traffic. While ModSecurity can not prevent every WS attack category listed above, it can certainly help to prevent a large number of the common HTTP attacks that now simply riding in the XML payloads.

XML Support in the Core Rules

Version 1.4 build 2 of the Core Rules introduced support for inspecting the the XML payloads of Web Services transactions. You can identify this by the inclusion of the XML:/* data in the variable listing. An example rule is listed below:

# Email Injection
SecRule REQUEST_FILENAME|ARGS|ARGS_NAMES|REQUEST_HEADERS|XML:/* "[\n\r]\s*(?:to|bcc|cc)\s*:.*?\@" \
        "t:none,t:lowercase,t:urlDecode,capture,ctl:auditLogParts=+E,log,auditlog,msg:'Email Injection Attack. Matched signature <%{TX.0}>',,id:'950019',severity:'2'"

Since the Core Rules offers generic detection and does not tie specific attack payloads to specific parameters, the XML:/* variable is somewhat similar to the REQUEST_BODY payload in that ModSecurity will treat it as one large piece of data. This results in ModSecurity searching the entire XML payload looking for rule matches. For those ModSecurity users who are familiar with the 1.9.x branch, this is similar to the SecFilter rule processing where it performs a wider search for attacks as it does not know exactly where the input vectors are located. The side-effect is that there may be a performance hit if you WS XML payloads are large. If this is the case in your environment, they you will want to create some custom XML rules.

Custom XML Rules

ModSecurity can also be used to create custom rules for your WS application. Not only will this make the protection stronger and lowering the false positive rate, but you will also gain a performance boost when you specify full XPath locations in the variable list vs. the generic XML:/* variable that the Core Rules utilizes. We have created a use-case document entitled Securing Web Services with ModSecurity2 that will help to provide you with some examples of how to setup custom WS rules. Taking the previous Core Rule example, if we customize it for our WS application that is running at "/axis/getBalance.jws" and has one input parameter called "id", then the new rule would look something like this -

<Location /axis/getBalance.jws>
SecRule XML:/soap:Envelope/soap:Body/q1:getInput/id/text() "[\n\r]\s*(?:to|bcc|cc)\s*:.*?\@" \
"t:none,t:lowercase,t:urlDecode,capture,ctl:auditLogParts=+E,log,auditlog,xmlns:soap=http:// \
schemas.xmlsoap.org/soap/envelope/,xmlns:q1=http://DefaultNamespace,msg:'Email Injection Attack. Matched signature <%{TX.0}>',id:'950019',severity:'2'"
</Location>

Notice the bolded portions of the ruleset where we have updated the XML variable to include a full XPath to our "id" input parameter and we also specified two xmlns actions to help ModSecurity to appropriately parse the payload.