This week's feature is the effective use of Transformation functions.

This excerpt is taken from the updated Reference Manual section of Ivan Ristic's book ModSecurity Handbook.

Transformation functions are used to alter input data before it is used in matching (i.e., operator execution). The input data is never modified, actually—whenever you request a trans- formation function to be used, ModSecurity will create a copy of the data, transform it, and then run the operator against the result.

Note

There are no default transformation functions, as there were in the first generation of ModSecurity (1.x).

In the following example, the request parameter values are converted to lowercase before matching:

SecRule ARGS "xp_cmdshell" "t:lowercase"

In most cases, the order in which transformations are performed is very important. In the following example, a series of transformation functions is performed to counter evasion. Per- forming the transformations in any other order would allow a skillful attacker to evade detection:

SecRule ARGS "(asfunction|javascript|vbscript|data|mocha|livescript):" \ 

"t:none,t:htmlEntityDecode,t:lowercase,t:removeNulls,t:removeWhitespace"

Warning

It is currently possible to use SecDefaultAction to specify a default list of transfor- mation functions, which will be applied to all rules that follow the SecDefaultAction directive. However, this practice is not recommended, because it means that mistakes are very easy to make. It is recommended that you always specify the transformation functions that are needed by a particular rule, starting the list with t:none (which clears the possibly inherited transformation functions).

SecRule REQUEST_FILENAME|ARGS_NAMES|ARGS|XML:/* "\bunion\b.{1,100}?\bselect\b" \
 "phase:2,rev:'2.0.8',capture,t:none,t:urlDecodeUni,t:htmlEntityDecode,t:lowercase,t:replaceComments,t:compressWhiteSpace,ctl:auditLogParts=+E,pass,nolog,auditlog,msg:'SQL Injection Attack',id:'959047',tag:'WEB_ATTACK/SQL_INJECTION',tag:'WASCTC/WASC-19',tag:'OWASP_TOP_10/A1',tag:'OWASP_AppSensor/CIE1',tag:'PCI/6.5.2',logdata:'%{TX.0}',severity:'2',setvar:'tx.msg=%{rule.msg}',setvar:tx.sql_injection_score=+%{tx.critical_anomaly_score},setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},setvar:tx.%{rule.id}-WEB_ATTACK/SQL_INJECTION-%{matched_var_name}=%{tx.0}"

delete from -- lowercase

DELETE FROM -- upper-case

deLeTe fRoM -- mixed-case

Delete From -- more than 1 space between keywords

DELETE\tFROM -- \t represents a TAB character

DELETE/* random data */ FROM -- use of SQL Comments

http://localhost/vulnerable_app.php?foo=1'%20UniOn%09(/*blah%20blah%20blah*/SeLeCt%20%20%20%20%20%20%20'1','2',PASSword%20from%20USERS)%20--%20-a

[31/Aug/2010:11:34:17 --0400] [localhost/sid#10080d708][rid#1025d2aa0][/vulnerable_app.php][9] T (0) urlDecodeUni: "1' UniOn\t(/*blah blah blah*/SeLeCt '1','2',PASSword from USERS) -- -a"
[31/Aug/2010:11:34:17 --0400] [localhost/sid#10080d708][rid#1025d2aa0][/vulnerable_app.php][9] T (0) htmlEntityDecode: "1' UniOn\t(/*blah blah blah*/SeLeCt '1','2',PASSword from USERS) -- -a"
[31/Aug/2010:11:34:17 --0400] [localhost/sid#10080d708][rid#1025d2aa0][/vulnerable_app.php][9] T (0) lowercase: "1' union\t(/*blah blah blah*/select '1','2',password from users) -- -a"
[31/Aug/2010:11:34:17 --0400] [localhost/sid#10080d708][rid#1025d2aa0][/vulnerable_app.php][9] T (0) replaceComments: "1' union\t( select '1','2',password from users) -- -a"
[31/Aug/2010:11:34:17 --0400] [localhost/sid#10080d708][rid#1025d2aa0][/vulnerable_app.php][9] T (0) compressWhitespace: "1' union ( select '1','2',password from users) -- -a"
[31/Aug/2010:11:34:17 --0400] [localhost/sid#10080d708][rid#1025d2aa0][/vulnerable_app.php][4] Transformation completed in 36 usec.
[31/Aug/2010:11:34:17 --0400] [localhost/sid#10080d708][rid#1025d2aa0][/vulnerable_app.php][4] Executing operator "rx" with param "\\bunion\\b.{1,100}?\\bselect\\b" against ARGS:foo.
[31/Aug/2010:11:34:17 --0400] [localhost/sid#10080d708][rid#1025d2aa0][/vulnerable_app.php][9] Target value: "1' union ( select '1','2',password from users) -- -a"

http://localhost/vulnerable_app.php?foo=%2f*&bar=%E2%80%98+UNION+SELECT+*+FROM+user+%26%23x2f*

[31/Aug/2010:11:56:15 --0400] [localhost/sid#10080d708][rid#1025d2aa0][/vulnerable_app.php][9] T (0) urlDecodeUni: "/vulnerable_app.php?foo=/*&bar=\xe2\x80\x98 UNION SELECT * FROM user /*"
[31/Aug/2010:11:56:15 --0400] [localhost/sid#10080d708][rid#1025d2aa0][/vulnerable_app.php][9] T (0) htmlEntityDecode: "/vulnerable_app.php?foo=/*&bar=\xe2\x80\x98 UNION SELECT * FROM user /*"
[31/Aug/2010:11:56:15 --0400] [localhost/sid#10080d708][rid#1025d2aa0][/vulnerable_app.php][9] T (0) lowercase: "/vulnerable_app.php?foo=/*&bar=\xe2\x80\x98 union select * from user /*"
[31/Aug/2010:11:56:15 --0400] [localhost/sid#10080d708][rid#1025d2aa0][/vulnerable_app.php][9] T (0) replaceComments: "/vulnerable_app.php?foo= "
[31/Aug/2010:11:56:15 --0400] [localhost/sid#10080d708][rid#1025d2aa0][/vulnerable_app.php][9] T (0) compressWhitespace: "/vulnerable_app.php?foo= "

[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][4] Executing operator "rx" with param "\\bunion\\b.{1,100}?\\bselect\\b" against REQUEST_URI_RAW.
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][9] Target value: "/vulnerable_app.php?foo=%2f*&bar=%E2%80%98+UNION+SELECT+*+FROM+user+%26%23x2f*"
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][4] Operator completed in 1 usec.
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][9] T (0) urlDecodeUni: "/vulnerable_app.php?foo=/*&bar=\xe2\x80\x98 UNION SELECT * FROM user /*"
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][4] Transformation completed in 42 usec.
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][4] Executing operator "rx" with param "\\bunion\\b.{1,100}?\\bselect\\b" against REQUEST_URI_RAW.
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][9] Target value: "/vulnerable_app.php?foo=/*&bar=\xe2\x80\x98 UNION SELECT * FROM user /*"
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][4] Operator completed in 1 usec.
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][9] T (0) htmlEntityDecode: "/vulnerable_app.php?foo=/*&bar=\xe2\x80\x98 UNION SELECT * FROM user /*"
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][4] Transformation completed in 78 usec.
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][4] Executing operator "rx" with param "\\bunion\\b.{1,100}?\\bselect\\b" against REQUEST_URI_RAW.
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][9] Target value: "/vulnerable_app.php?foo=/*&bar=\xe2\x80\x98 UNION SELECT * FROM user /*"[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][4] Operator completed in 0 us
ec.
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][9] T (0) lowercase: "/vulnerable_app.php?foo=/*&bar=\xe2\x80\x98 union select * from user /*"
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][4] Transformation completed in 109 usec.
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][4] Executing operator "rx" with param "\\bunion\\b.{1,100}?\\bselect\\b" against REQUEST_URI_RAW.
[31/Aug/2010:16:22:45 --0400] [localhost/sid#10080d708][rid#1040020a0][/vulnerable_app.php][9] Target value: "/vulnerable_app.php?foo=/*&bar=\xe2\x80\x98 union select * from user /*"

Transformation Function Tips

Here are a few recommended tips for using transformation functions.

Start with t:none

Due to the fact that transformation functions pipelines are cumulative, it is possible that you could unintentionally inherit transformation functions from a previous SecDefaultAction.  It is therefore good practice to start each transformation function pipeline with "t:none" as this will clear out any existing ones.

Choose the right transformations

There is no "one size fits all" magic transformation function pipeline.  You need to analyze what type of attack you are targeting and then identify the methods in which attackers may try evade detection.  For instance, the transformation pipelines for detecting Cross-site Scripting (XSS) is much different then what you would use for SQL Injection.

Beware of performance

The larger the payload is, the longer it will take to complete the transformation pipeline.  This is not that big of a concern for inbound request data as they are generally small in size.  Where you can run into higher latency hits is when you are attempting to inspect outbound data (RESPONSE_BODY variable).  It is for this reason that you should try to limit the use of transformations when inspecting RESPONSE_BODY and instead specify mixed-case within your regular expression.

I am excited to announce that the OWASP ModSecurity Core Rule Set (CRS) has completed its official review and has been promoted to a Release Quality Project!  I want to thank both Ivan Ristic and Leonardo Cavallari Militelli who served as project reviewers.

While this is certainly exciting news and will help to further promote ModSecurity and the Core Rule Set, there is still much work left to be done on the project.  We will be expanding the CRS documentation, re-organizing the rules to more accurately categorize the quality of the rules (with relation to false positive potential), as well as, cross-referencing rules with the OWASP AppSensor project.

Keep your eyes open as we are just getting started with the CRS.

We are starting a new blog post series here on the ModSecurity site called "Advanced Feature of the Week" where we will be highlighting many of ModSecurity's really cool capabilities.  These are the features that seldom used or fully understood by the average ModSecurity user however can provide detection of sophisticated attacks if used properly.  It is our goal with these blog posts to help shed light on these unique features and to provide some real-world, in-the-trenches gotchas for successful usage of these features.

This blog post series will have the following major topic sections -

1) ModSecurity Reference Manual Information

Provide reference manual data.

2) Use Within the OWASP Core Rule Set (CRS)

Outline if/when/how the CRS is utilizing this feature.

3) So What?

Will provide some context as to why you as a user should even care about this capability.  What advanced attack/vulnerability is this attempting to catch.

This week's feature is on the use of the @validateByteRange operator.

SecRule ARGS:text "@validateByteRange 10, 13, 32-126"

ASCII Byte Range Chart

Use of @validateByteRange in the OWASP ModSecurity CRS (from the end of the modsecurity_crs_20_protocol_violations.conf file) -

#
# Restrict type of characters sent
# NOTE In order to be broad and support localized applications this rule
# only validates that NULL Is not used.
#
# The strict policy version also validates that protocol and application 
# generated fields are limited to printable ASCII. 
#
# -=[ Rule Logic ]=-
# This rule uses the @validateByteRange operator to look for Nul Bytes.
# If you set Paranoid Mode - it will check if your application use the range 32-126 for parameters.
#
# -=[ References ]=-
# http://i-technica.com/whitestuff/asciichart.html
#

SecRule ARGS|ARGS_NAMES|REQUEST_HEADERS|!REQUEST_HEADERS:Referer "@validateByteRange 1-255" \
 "phase:2,rev:'2.0.8',pass,nolog,auditlog,msg:'Invalid character in request',id:'960901',tag:'PROTOCOL_VIOLATION/EVASION',tag:'WASCTC/WASC-28',tag:'OWASP_TOP_10/A1',tag:'OWASP_AppSensor/CIE3',tag:'PCI/6.5.2',severity:'4',t:none,t:urlDecodeUni,setvar:'tx.msg=%{rule.msg}',tag:'http://i-technica.com/whitestuff/asciichart.html',setvar:tx.anomaly_score=+%{tx.notice_anomaly_score},setvar:tx.protocol_violation_score=+%{tx.notice_anomaly_score},setvar:tx.%{rule.id}-PROTOCOL_VIOLATION/EVASION-%{matched_var_name}=%{matched_var}"

SecRule TX:PARANOID_MODE "@eq 1" "chain,phase:2,rev:'2.0.8',pass,nolog,auditlog,msg:'Invalid character in request',id:'960018',tag:'PROTOCOL_VIOLATION/EVASION',tag:'WASCTC/WASC-28',tag:'OWASP_TOP_10/A1',tag:'OWASP_AppSensor/CIE3',tag:'PCI/6.5.2',severity:'4',t:none,t:urlDecodeUni,tag:'http://i-technica.com/whitestuff/asciichart.html'"
 SecRule REQUEST_URI|REQUEST_BODY|REQUEST_HEADERS_NAMES|REQUEST_HEADERS|!REQUEST_HEADERS:Referer|TX:HPP_DATA \
 "@validateByteRange 32-126" \
 "t:none,t:urlDecodeUni,setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+%{tx.notice_anomaly_score},setvar:tx.protocol_violation_score=+%{tx.notice_anomaly_score},setvar:tx.%{rule.id}-PROTOCOL_VIOLATION/EVASION-%{matched_var_name}=%{matched_var}"

%u3008scr%u0131pt%u3009%u212fval(%uFF07al%u212Frt(%22XSS%22)%u02C8)%u2329/scr%u0131pt%u2A

〈scrıpt〉ℯval(＇alℯrt(”XSS”)ˈ)〈/scrıpt〉

<script>eval('alert("XSS")')</script>

So, this brings us to today's advanced ModSecurity feature - @validateByteRange.  By restricting the allowed character byte ranges, you can help to identify when unexpected character code points are used.  If the payload above is sent, you should receive an alert message similar to the following -

Message: Found 3 byte(s) in REQUEST_URI outside range: 32-126. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_20_protocol_violations.conf"] [line "251"] [id "960018"] [rev "2.0.6"] [msg "Invalid character in request"] [severity "WARNING"] [tag "PROTOCOL_VIOLATION/EVASION"] [tag "WASCTC/WASC-28"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE3"] [tag "PCI/6.5.2"] [tag "http://i-technica.com/whitestuff/asciichart.html"]

Since Black Hat and DEFCON we have been busying building teams and aligning objectives over here at Trustwave's SpiderLabs. We are committed to driving innovation into the development of ModSecurity for the future.

Here are are few things that we having going on right now:

• Website Updates - We are in the process of refreshing the website to reflect its new home within SpiderLabs but also clean out any dusty content and replace it with fresh new material. Look for phases of changes over the next several weeks.
• Recruiting - We are actively looking to hire full-time developers for ModSecurity. Those developers will be part of the SpiderLabs research team. If you or anyone you know is interested, please let us know.
• Internal Development Team - We have formed a team of people within SpiderLabs that will be actively contributing to ModSecurity. This team is comprised of many very experienced individuals including many OWASP, Black Hat and DEFCON researchers/speakers in the area of web application security. This team is excited to breath new life into this project.
• End User Survey - In the next few weeks, we will be sending out a survey to the ModSecurity mailing lists to ask for feedback on features to include in the next release.
• Blog Updates - Each week we'll be writing about an Advance Feature in ModSecurity that the community and end users may not be aware of. The idea is here is that with increased usage of these advanced, we'll get better feedback on how to improve the product.
• Twitter - We have just linked this blog to the ModSecurity Twitter account. Follow us!

As many of you already know, about a month ago Trustwave acquired Breach Security, the then owner and sponsor of the ModSecurity project. Trustwave's Spider Labs will now be leading the project. Within Spider Labs and the highly capable people there, ModSecurity will be able to thrive.

And so, it is with mixed feelings that I also announce that I am stepping down from the project. I have enjoyed working on ModSecurity with Breach for the past 3 1/2 years, but it is time for me to move on to another exciting project (more on that later). While I may be stepping down as a leader, I am not leaving, just changing roles. I plan to stay an active member of the community and to contribute where I can. I expect to see a lot of innovation from ModSecurity in the near future and I look forward to see where ModSecurity will go with all its new resources.

Cheers to ModSecurity's new home in Spider Labs and to a successful future!

P.S. -- I'll be around at Black Hat/DEFCON next week in Vegas if anyone wants to hook up.

ModSecurity Community,

We will be hosting a ModSecurity happy hour during Black Hat USA. It is open to anyone who contributes, uses or wants to learn more about the project. You'll also get a chance to meet the members of Trustwave's SpiderLabs team that will be supporting the ModSecurity project moving forward. 

• What: ModSecurity Happy Hour
• Where: munchbar @ Caesar's Palace LV (next to Sports Bar & Pure Night Club)
• When: Wednesday, July 28th, 2010 - 4:00-6:00 PM

There will be appetizers, drinks, and Nintendo Wii action!

Look forward to meeting you next week!

Senior Vice President, SpiderLabs

Trustwave

There was a recent blog article stating that ModSecurity can be bypassed by adding invalid characters to Base64 encoded data. Well, this is somewhat correct, but I am not sure I'd call it a bypass. It is really "Impedance Mismatch" as it depends on the decoder you are using in your app. PHP's decoder is ignoring characters (RFC-2045) and ModSecurity is doing what Apache does for HTTP Basic Auth and not allowing the extra characters (RFC-4648)

The article's example is roughly this:

1) Take an attack string: <script>alert(1)</script>
2) Base64 encode it to: PHNjcmlwdD5hbGVydCgxKTwvc2NyaXB0Pg==
3) Now add an illegal character: P.HNjcmlwdD5hbGVydCgxKTwvc2NyaXB0Pg==
4) Notice that most decoders will not work, but PHP's will (act surprised)

PHP will apparently just skip the invalid characters (RFC-2045) and so something like this (article's example, not mine) will of course fail:

SecRule ARGS:b64 "alert" "t:base64decode,log,deny,status:501"

The Base64 decoder in ModSecurity is based off the RFC-4648 implementation of Base64. There are many other variants. Well, as it turns out it is important to know a bit more about your platform on which your app is based and the above trivial rule is just not going to cut it.

For PHP and possibly others you will need to go a little further and validate the character set first using a positive rule. Something like this is going to be required for the article's example:

SecRule ARGS:b64 "!^[A-Za-z0-9\+/]*={0,2}$" \
  "phase:2,t:none,log,deny,status:403,msg:'Invalid Base64 Encoding'"
SecRule ARGS:b64 "alert" \
  "phase:2,t:none,t:base64decode,log,deny,status:403,msg:'Badness in b64'"

And now you get some better coverage:

# For PHNjcmlwdD5hbGVydCgxKTwvc2NyaXB0Pg==
ModSecurity: Access denied with code 403 (phase 2). Pattern match "alert" 
at ARGS:b64. [file "test.conf"] [line "3"] [msg "Badness in b64"] 
[hostname "myhost"] [uri "/foo"] [unique_id "S8-4-X8AAQEAACGOJcoAAAAA"]

# For P.HNjcmlwdD5hbGVydCgxKTwvc2NyaXB0Pg==
ModSecurity: Access denied with code 403 (phase 2). Match of 
"rx ^[A-Za-z0-9\\\\+/]*={0,2}$" against "ARGS:b64" required. 
[file "test.conf"] [line "1"] [msg "Invalid Base64 Encoding"] 
[hostname "myhost"] [uri "/foo"] [unique_id "S8-5BX8AAQEAACJSKBYAAA@i"]

Though I am picking on PHP a bit here, this may be true in many other areas if you have decoders/parsers that accept out-of-the-norm data. You really do have to know your apps to write targeted rules like the example in this article. You cannot detect encodings like Base64 generically and I would not expect to find such a rule as this in a generic rule set such as ModSecurity's CRS.

Edited: Added details on which RFCs I was referring to and removed blame on PHP after further investigation as it really is just an issue with multiple variants of base64.

Just wanted to let everyone know that if you are headed to Blackhat USA 2010 this summer in
Las Vegas, we have just added a 1-day workshop on the day before the Briefings start -

In the workshop, we will be mainly discussing the "Virtual Patching" concept of using a
WAF (ModSecurity in this case) and we will use the OWASP WebGoat app as the target.  In
the workshop, we will talk virtual patching theory and then have hands-on labs where we
will show how to use Mod to virtually patch the various WebGoat lessons.  As a side note -
we will also have a section on the new CRS v2.0 when discussing negative security models. 
So, if you want to come and dive into the deep-end of the pool and have fun using some of
ModSecurity's advanced features (such as Lua and Content Injection) then sign-up now!

Brian Rectanus and I hope to see you all in Vegas!!!  :)

I presented on the OWASP ModSecurity Core Rule Set (CRS) Project yesterday here at the AppSec DC conf.  The reception was good and there were a lot of great questions.  For those interested, the PPT has been posted to the project site.  It gives a good overview of the CRS - http://www.owasp.org/index.php/Category:OWASP_ModSecurity_Core_Rule_Set_Project#tab=Presentations_and_Whitepapers