ModSecurity Trustwave
This blog has moved! Please update your
bookmarks to

ModSecurity Blog: Web Security

Detecting Malice with ModSecurity: GeoLocation Data

I would like to introduce a new blog series entitled - Detecting Malice with ModSecurity and will be used as an alternative to the Advanced Topic of the Week blog posts.  The the idea is to take the outstanding web application intrusion/fraud detection concepts introduced by my friend Robert "Rsnake" Hansen in his book Detecting Malice and provide real-life implementation examples using ModSecurity.  Rsnake has given me the green light to provide actual snippets of text/quotes directly from his book!  The format will be that Rsnake will setup the underlying issues and methods to identify bad behavior and then I will present some practical implementations using ModSecurity.

In this installment of Detecting Malice with ModSecurity, we will discuss how to use Geolocation data.

Geolocation section from Detecting Malice

IP addresses aren’t quite like street addresses, but they’re sometimes close. There are a number of projects that correlate, with varying success, IP address information to geographic locations. Some very large retailers help these databases by tying in IP address information collected along with address information given by their users during registration. Others tie in ISP information and geographic information with the IP address – a far less precise method in practice. Yet others still use ping time and hops to get extremely accurate measurement estimates of physical location.

The following information is typically available in GeoIP databases:

 Country

 Region (for some countries)

 City

 Organization (typically from the WHOIS database)


 Connection speed

 Domain name

There are lots of reasons you may be interested in the physical location of your users. Advertisers are always on the lookout for ways to make their ads more interesting and meaningful (because that increases their click-through rate), and serving local ads is a good way to achieve that. Relevance is key, and that can be optimized by targeting the results to a user’s location.

Geographic information can also give you clues to the user’s disposition. If you are running a website in the United States and someone from another country is accessing your site, it may tell you quite a bit about the possible intentions of that user. That becomes more interesting if the site is disliked in principle by the population of other countries, or doesn’t do business with that country in particular. For example, many sites are totally blocked by the Chinese’s system of firewalls because the Chinese government believes certain words or thoughts are disruptive to the government’s goals. If someone from China IP space visits your site despite you being blocked by the Chinese government’s firewalls, there is a high likelihood the traffic is state sponsored.

Similarly, if you know that a large percentage of your fraud comes from one location in a particular high crime area of a certain city, it might be useful to know that a user is accessing your site from a cyber café in that same high crime area. Perhaps you can put a hold on that shipment until someone can call the user and verify the recipient. Also if you know that fraud tends to happen in certain types of socio- economic geographies, it could help you heuristically to determine the weighted score of that user. This sort of information may not be available to you in your logs, but could easily be mined through external sources of relevant crime information. Companies like ESRI heavily focus on mapping demographics, for instance.

In terms of history, it doesn’t hurt to look back into time and see if the IP address of the user has ever connected to you before. That information combined with their status with the website can be useful information. Another thing to consider is the type of connection used. If you know the IP space belongs to a DSL provider, it may point to a normal user. If it points to a small hosting provider, and in particular a host that is also a web server, it is highly likely that it has been compromised. Knowing if the IP belongs to a datacenter or a traditional broadband or modem IP range is useful information.

Knowing the physical location can help in other ways as well. If your website does no business in certain states or with certain countries due to legislative issues, it may be useful to know that someone is attempting to access your site from those states. This is often true with online casinos, which cannot do business with people in the United States where the practice is prohibited. Alerting the user of that fact can help reduce the potential legal exposure of doing business.

One concept utilizing physical location is called, “defense condition.” Normally, you allow traffic from everywhere, but if you believe that you're being attacked you can switch to yellow (the meaning of which is specific to the websites in question but, for example, can mean that your website blocks suspicious traffic coming from certain countries) or red (your websites block any traffic from certain countries). While this may seem like a good idea, in some cases this can actually be used against a website to get a company to block other legitimate users, so the use of this concept is cautioned against.

Utilities like MaxMind’s geoipaddress lookup can give you high level information about the location of most IP addresses for free. There are other products that can give you a lot more information about the specific location of the IP address in question, which can be extremely helpful if you are concerned about certain traffic origins.

$ geoiplookup.exe GeoIP Country Edition: BR, Brazil

There has been quite a bit of research into this concept of bad geographic origins. For instance McAfee published a study citing Hong Kong (.hk), China (.cn) and the Philippines (.ph) as the three most likely top level domains to pose a security risk to their visitors. While these kinds of studies are interesting, it’s important not to simply cast blame on an entire country, but rather think about why this is the case – which can give you far more granular and useful information. It should be noted that this study talks about the danger that websites pose to internet surfers, not to websites in general, but there may be a number of ways in which your website can contact or include portions of other sites, so it still may be applicable.

Using GeoLocation Data in ModSecurity

ModSecurity supports using geolocation data through the integration with the free MaxMind GeoLite Country or GeoLite City databases. The first step is to download the database of your choice and put it somewhere on the local filesystem where ModSecurity can use it (for example in the same directory as the Core Rule Set).

Before you can use the GeoIP data, you first have to use the SecGeoLookupDb directive:


Description: Defines the path to the geographical database file.

Syntax: SecGeoLookupDb /path/to/db

Example Usage: SecGeoLookupDb /usr/local/geo/data/GeoLiteCity.dat

Processing Phase: N/A

Scope: Any

Version: 2.5.0

Dependencies/Notes: Check out for free database files.

Once the SecGeoLookupDb directive is active, you then need to use a rule similar to the following which will pass the client's IP address (REMOTE_ADDR) to the @geoLookup operator:

SecGeoLookupDb /path/to/apache/conf/base_rules/GeoLiteCity.dat
SecRule REMOTE_ADDR "@geoLookup" "phase:1,t:none,pass,nolog"

When this rule is executed, the GEO variable collection data will be populated with the data extracted from the GeoIP DB.  Here is an example debug log section:

[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][4] Recipe: Invoking rule 1009c47d8; [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_15_customrules.conf"] [line "30"].
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][5] Rule 1009c47d8: SecRule "REMOTE_ADDR" "@geoLookup " "phase:1,t:none,nolog,pass"[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][4] Transformation completed in 1 usec.
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][4] Executing operator "geoLookup" with param "" against REMOTE_ADDR.
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] Target value: ""
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: Looking up "".
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: Using address "" (0x77c0e786).
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: rec="\x77\x31\x31\x00\x53\x65\x6f\x75\x6c\x00\x00\xb0\x32\x21\x2d\xd8\x2e\x00\x00\x00\x00\xa6\x45\x37\x00\x41\x75\x63\x6b\x6c\x61\x6e\x64\x00\x00\x25\xd7\x15\x13\x22\x36\x00\x00\x00\x00\xa6\x46\x35\x00\x4e"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: country="\x77"[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: region="\x31\x31\x00"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: city="\x53\x65\x6f\x75\x6c\x00"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: postal_code="\x00"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: latitude="\xb0\x32\x21"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: longitude="\x2d\xd8\x2e"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO: dma/area="\x00\x00\x00"
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][9] GEO:{country_code=KR, country_code3=KOR, country_name=Korea, Republic of, country_continent=AS, region=11, city=Seoul, postal_code=, latitude=37.566399, longitude=126.999702, dma_code=0, area_code=0}
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][4] Operator completed in 85333 usec.
[27/Oct/2010:09:49:51 --0400] [localhost/sid#10080d708][rid#1038060a0][/foo.php][4] Warning. Geo lookup for "" succeeded. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_15_customrules.conf"] [line "30"]

From this point on, you can use the geographic information from the GEO collection in your rules.

GeoLocation Usage Strategies

Allowed GEO Locations

If your clients only come from some specific geographic regions, then you could create a rule such as the following which would block clients who are not coming from US-based IP addresses:

SecRule GEO:COUNTRY_CODE3 "!@streq USA" "phase:1,t:none,log,deny,msg:'Client IP not from USA'"

DefCon Level - Static

The vast majority of organizations are global and would not want to implement this type of restriction for day-to-day operations.  This type of rule is useful, however, when the organization wants to initiate a more restrictive "Defense Condition Level" when under a direct attack.  You could make the previous rule conditional and only used when under an increased DefCon Level:  

SecAction "phase:1,t:none,nolog,pass,setvar:tx.defcon_level=1"
SecGeoLookupDb /path/to/apache/conf/base_rules/GeoLiteCity.dat
SecRule REMOTE_ADDR "@geoLookup" "phase:1,t:none,pass,nolog"
SecRule GEO:COUNTRY_CODE3 "!@streq USA" "chain,phase:1,t:none,log,deny,msg:'Client IP not from USA'"     
SecRule TX:DEFCON_LEVEL "@streq 1"

DefCon Level - Dynamic

While this example works, one of the big challenges in effectively using the concept of DefCon Levels with ModSecurity rules is the fact that you would have to initiate an Apache restart/reload in order for these DefCon Level variable data to be used.  This becomes a real scaling challenge if you are protecting a large server farm.  Under these types of scenarios, you can actually utilize the concept of dynamic rules which are activated based on web requests from authorized sources.  

Imagine you are running a SOC and you identify that your web sites are under attack and you want to initiate the new DefCon Level restriction rules to only allow requests from specific geographic regions.  Rather then needing to push out new Apache/ModSecurity configs to all web servers and initiate restarts, you could alternatively initiate a web request from an authorized web client system from within the SOC.  This client would have a specific IP address, User-Agent String and make a request to a specific URL with a specific parameter payload.  When this request is received by ModSecurity, the rules would then set a new DefCon level variable in a persistent Global collection.  When this happens, the GeoIP restriction rules would become active.  Once the threat has passed and the DefCon Level has been restored, a 2nd request can be sent out to all web servers to remove the DefCon Level variable and return the rules to their normal processing.  Again, the advantage of this approach is that it is a much faster method to toggle on/off a subset of rules vs having to update Apache/ModSecurity configs and initiate restarts.

Here are some example rules that implement this basic concept:

SecGeoLookupDb /path/to/apache/conf/base_rules/GeoLiteCity.dat
SecAction "phase:1.t:none,nolog,pass,setsid:global" 
SecRule REQUEST_FILENAME "/defcon_control" "chain,phase:1,t:none,log,msg:'DefCon Level 1 - GeoIP Restrictions Enabled.'"
    SecRule ARGS:DEFCON_LEVEL "@streq 1" "chain"
        SecRule REMOTE_ADDR "^192\.168\.1\.101$" "setvar:session.defcon_level=1" 
SecRule REQUEST_FILENAME "/defcon_control" "chain,phase:1,t:none,log,msg:'DefCon Level 1 - GeoIP Restrictions Disabled.'" 
    SecRule ARGS:DEFCON_LEVEL "5" "chain"
         SecRule REMOTE_ADDR "^192\.168\.1\.101$" "setvar:session.defcon_level=5" 
SecRule REMOTE_ADDR "@geoLookup" "phase:1,t:none,pass,nolog" 
SecRule GEO:COUNTRY_CODE3 "!@streq USA" "chain,phase:1,t:none,log,deny,msg:'Client IP not from USA'" 
    SecRule SESSION:DEFCON_LEVEL "@streq 1"

With these rules in place, the following request from source IP would enable the DefCon Level GeoIP restriction rules:

GET /defcon_control?defcon_level=1 HTTP/1.1

Assigning Fraud/Risk Scores 

As Rsnake discussed, there are many different fraud detection resources that have assigned general risk scores to certain geographic regions.  For example, in the Clear Commerce whitepaper entitled Fraud Prevention Guide, they list the top 12 High Risk Countries:

  • Ukraine
  • Indonesia
  • Yugoslavia
  • Lithuania
  • Egypt
  • Romania
  • Bulgaria
  • Turkey
  • Russia
  • Pakistan
  • Malaysia
  • Israel

If you wanted to use this general information and assign an increased anomaly score to requests originating from these countries, you could use a rule similar to the following:

SecRule GEO:COUNTRY_CODE "@pm UA ID YU LT EG RO BG TR RU PK MY IL" "phase:1,t:none,log,pass,msg:'High Risk Fraud Location',setvar:tx.fraud_score=+10"

This rule does not block the request but it does increase the "TX:FRAUD_SCORE" variable data.  The TX data can be increased by other rules as well, such as those doing IP Reputation checks with @rbl, and then it can be evaluated at the end of the request phase when a decisions is made after correlating all detection data. 

Per-User GeoIP Data

Another Fraud related concept that can use GeoIP data is to track the Country/City data used by each user.  You can save this data in a per-user persistent collection and simply raise the fraud_score if the data changes.  This concept can be implemented either across multiple user sessions or for possible session hijacking detection during the course of one session.

Here are some example rules that will save the GEO Country Code data in a persistent User collection.  It will count the number of times that user has connected from that same country and once it is gone over the defined threshold (10), it will then issue alerts when the use logs in from a different country.

SecGeoLookupDb /path/to/apache/conf/base_rules/GeoLiteCity.dat 
SecAction "phase:1.t:none,nolog,pass,setuid:%{args.username}" SecRule REMOTE_ADDR "@geoLookup" "phase:1,t:none,nolog,pass" 
SecRule &USER:GEO_COUNTRY_CODE "@eq 0" "phase:1,t:none,nolog,pass,setvar:user.geo_country_code=%{geo.country_code},setvar:user.geo_country_code_counter=+1" 
SecRule GEO:COUNTRY_CODE "@streq %{user.geo_country_code}" "phase:1,t:none,nolog,pass,setvar:user.geo_country_code_counter=+1" 
SecRule USER:GEO_COUNTRY_CODE_COUNTER "@gt 10" "chain,phase:1,t:none,log,pass,msg:'Geo Country Code Change for User.',logdata:'Username: %{userid}, Expected Country Code: %{user.geo_country_code}, Current Country Code: %{geo.country_code}'"
    SecRule GEO:COUNTRY_CODE "!@streq %{user.geo_country_code}" "setvar:tx.fraud_score=+10"

The final example shows how you can identify if the GEO Country Code data changes during the course of a Session (using JSESSIONID as the example).  The key is to save the GEO:COUNTRY_CODE data when the application issues a Set-Cookie SessionID and then validate it on subsequent requests.

SecGeoLookupDb /path/to/apache/conf/base_rules/GeoLiteCity.dat
SecRule REMOTE_ADDR "@geoLookup" "phase:1,t:none,nolog,pass"
SecRule REQUEST_COOKIES:JSESSIONID ".*" "chain,phase:1,t:none,pass,nolog,auditlog,msg:'Geo Country Code Change During Session.',setsid:%{request_cookies.jsessionid}"
    SecRule GEO:COUNTRY_CODE "!@streq %{session.geo_country_code}"
SecRule RESPONSE_HEADERS:/Set-Cookie2?/ "(?:jsessionid=([^\s]+)\;\s?)" "phase:3,t:none,pass,nolog,capture,setsid:%{TX.2},setvar:session.geo_country_code=%{geo.country_code}"

Hopefully the data presented in this blog post will help users to be able to better utilize GeoIP data.

Advanced Topic of the Week: Request Header Tagging

Request Header Tagging

Wouldn't it be cool if your WAF could share its data with the application it is protecting?  This concept is similar to anti-SPAM SMTP apps that will add additional mime headers to emails providing the SPAM detection analysis information. The CRS is attempting to mimic this concept at the HTTP layer by adding additional request headers that provide insight into any ModSecurity events that may have triggered during processing. The advantage of this approach is that it allows a WAF to be in a detection-only mode while still providing attack data to the destination application server. The recieving app server may then inspect the WAF request headers and make a determination whether or not to process the transaction. This concept is valuable in distributed web environments and hosting architectures where a determination to block may only be appropriate at the destination app server.

This concept has actually been discussed as part of the OWASP AppSensor Project and we have added a new Detection Point for it entitled - RP2: External User Behavior




Suspicious External User Behavior




External (to the application) devices and systems (e.g. host and network IDS, file integrity monitoring, disk usage monitoring, anti-malware service, IPS, network firewall, web application firewall, web server logging, XML gateway, database firewall, SIEM) detect anomalous behavior by the user (e.g. session and/or IP address).

This information can be used by the application to contribute to its knowleage about a potential attacker. In some cases, the information could be detected by the application itself (e.g. XSS pattern black listing), but may be more effectively identified by the external device, or is not known to the application normally (e.g. requests for missing resources that the web server sees, but does not pass onto the application).


The greater the knowledge a device or system has about the application, the greater confidence can be given to evidence of suspicious behaviour. Therefore, for example, attempted SQL injection detexcted by a web application firewall (WAF) might be given greater weight than information from a network firewall about the IP address.

The power of AppSensor is its accuracy and low false positive rate, and the usage of external data should be carefully assessed to ensure it does not contribute to a higher false positive rate.


Example 1: An IDS has detected suspicious activity by a particular IP address, and this is used to temporarily tighten the attack detection thresholds for requests from all users in the same IP address range.

Example 2: An application is using the ModSecurity web application firewall with the Core Rule Set, and utilises the anomaly score data passed forward in the X-WAF-Events and X-WAF-Score HTTP headers (optional rules in modsecurity_crs_49_header_tagging.conf) to adjust the level of application logging for each user.

Example 3: Information from an instance of PHPIDS suggests request data may be malicious.


[Java] [.Net] [PHP]


This rule set file will take all of the TX attack variable data and populate Apache ENV variables that Apache can then use to add X-WAF-Event request header data to the request.

Example showing the consolidated X-WAF-Events and X-WAF-Score data -

GET /path/to/foo.php?test=1%27%20or%20%272%27=%272%27;-- HTTP/1.1
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv: Gecko/20091109 Ubuntu/9.10 (karmic) Firefox/3.5.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
X-WAF-Events: TX: / 999935-Detects common comment types-WEB_ATTACK/INJECTION-ARGS:test, TX:999923-Detects JavaScript location/document property access and window access obfuscation-WEB_ATTACK/INJECTION-REQUEST_URI_RAW, TX:950001- WEB_ATTACK/SQL_INJECTION-ARGS:test
X-WAF-Score: Total=48; sqli=2; xss=
Connection: Keep-Alive


Advanced Topic of the Week: Identifying Improper Output Handling (XSS Flaws)

A Topic Presents Itself

As I sat down this morning thinking about an appropriate topic for this week's blog post, I quickly found my subject - XSS flaws.  The Twitter-verse exploded this morning with an outbreak of an XSS worm that is using the onmouseover event to propagate itself (technically I believe that this is a CSRF attack as it is forcing the user to send a Twitter update that they didn't intend to send... but I digress.).  I quickly added this incident into the WASC Web Hacking Incident Database.

How Prevalent is XSS?

Just how big of a problem is XSS?  Most web application security statistics projects have it listed right at the top -

OWASP Top 10 Web Application Security Risks for 2010

WASC Web Application Security Statistics 

WASC Web Hacking Incident Database

How Can You Protect Yourself?

First of all, make sure that all of your developers read, then reread, and then really sit down and take notes and seriously get a game plan together to address the issues raised in the OWASP XSS Prevention Cheatsheet document.  Make sure you understand this section on untrusted data:

Untrusted data is most often data that comes from the HTTP request, in the form of URL parameters, form fields, headers, or cookies. But data that comes from databases, web services, and other sources is frequently untrusted from a security perspective. That is, it might not have been perfectly validated. The OWASP Code Review Guide has a decent list of methods that return untrusted data in various languages, but you should be careful about your own methods as well.

Untrusted data should always be treated as though it contains an attack. That means you should not send it anywhere without taking steps to make sure that any attacks are detected and neutralized. As applications get more and more interconnected, the likelihood of a buried attack being decoded or executed by a downstream interpreter increases rapidly.

Traditionally, input validation has been the preferred approach for handling untrusted data. However, input validation is not a great solution for injection attacks. First, input validation is typically done when the data is received, before the destination is known. That means that we don't know which characters might be significant in the target interpreter. Second, and possibly even more importantly, applications must allow potentially harmful characters in. For example, should poor Mr. O'Malley be prevented from registering in the database simply because SQL considers ' a special character?

The OWASP ModSecurity Core Rule Set (CRS) contains many rules to try and identify malicious XSS payloads for example in the modsecurity_crs_41_xss_attacks.conf file.  The XSS payload used in the Twitter attacks today have been posted to PasteBin and if you send it to the CRS Demo you will see that it is detected by many different detection methods.

While input validation and looking for generic attack payloads is useful for identifying attack attempts, it is not the same as identifying the actual underlying application weakness that is exploited by XSS attacks and that is Improper Output Handling.  Essentially, what you need is Dynamic Taint Propagation detection where you can track where unstrusted user data is insecurely used within your application.  For XSS you need to identify where user supplied data is echoed back to clients in an unescaped form.

Dynamic Taint Propagation with ModSecurity

In CRS v2.0.7, we added an optional ruleset called modsecurity_crs_55_application_defects.conf which, among other things, implements an experimental dynamic taint propagation ruleset to identity if/when/where an application does not properly output encode/escape user supplied data.

# XSS Detection - Missing Output Encoding
SecAction "phase:1,nolog,pass,initcol:global=xss_list"

# Identifies Reflected XSS
# If malicious input (with Meta-Characters) is echoed back in the reply non-encoded.
SecRule &ARGS "@gt 0" "chain,phase:4,t:none,log,auditlog,deny,status:403,id:'1',msg:'Potentially Malicious Meta-Characters in User Data Not Properly Output Encoded.',logdata:'%{tx.inbound_meta-characters}'"
 	SecRule ARGS "([\'\"\(\)\;<>#])" "chain,t:none"
		SecRule MATCHED_VAR "^.{15,}$" "chain,t:none,setvar:tx.inbound_meta-characters=%{matched_var}"
  			SecRule RESPONSE_BODY "@contains %{tx.inbound_meta-characters}" "ctl:auditLogParts=+E"

# Check to see if TX XSS Data is already in the GLOBAL list.  If it is - expire it.
SecRule GLOBAL:'/XSS_LIST_.*/' "@streq %{tx.inbound_meta-characters}" "phase:4,t:none,nolog,pass,skip:1"
SecRule TX:INBOUND_META-CHARACTERS ".*" "phase:4,t:none,nolog,pass,setvar:global.xss_list_%{time_epoch}=%{matched_var}"
# Identifies Stored XSS
# If malicious input (with Meta-Characters) is echoed back on any page non-encoded.
SecRule GLOBAL:'/XSS_LIST_.*/' "@within %{response_body}" "phase:4,t:none,log,auditlog,pass,msg:'Potentially Malicious Meta-Characters in User Data Not Properly Output Encoded',tag:'WEB_ATTACK/XSS'"

Keep in mind - this ruleset's goal is not to try and identify malicious payloads but rather to indirectly identify possible stored/reflected XSS attack surface points by flagging when an application resources does not properly output escape user supplied data.  An advantage to this approach is that we don't have to be as concerned with evasion issues as they rules are not looking for malicious payloads.  Instead, they will monitor as normal users interact with the application and will alert if the application does not correctly handle the data.

Using this type of application weakness detection, it is possible to identify the underlying issues in the application and get a plan together to address them rather than perpetually playing "Whack-a-Mole" with inbound attacks.

Advanced Topic of the Week: Validating SessionIDs

This week's topic discusses how to validate application SessionIDs submitted by clients.

Reference Manual

Initializing the SESSION collection with the setsid action.


Description: Special-purpose action that initialises the SESSION collection.

Action Group: Non-disruptive


# Initialise session variables using the session cookie value 


On first invocation of this action the collection will be empty (not taking the predefined variables into account - see initcol for more information). On subsequent invocations the contents of the collection (session, in this case) will be retrieved from storage. After initialisation takes place the variable SESSIONID will be available for use in the subsequent rules.This action understands each application maintains its own set of sessions. It will utilise the current web application ID to create a session namespace.


This variable is a collection, available only after setsid is executed. Example: the following example shows how to initialize a SESSION collection with setsid, how to use setvar to increase the session.score values, how to set the session.blocked variable and finally how to deny the connection based on the session:blocked value.

SecRule REQUEST_URI "^/cgi-bin/foo" \
SecRule SESSION:SCORE "@gt 50" "pass,log,setvar:session.blocked=1"
SecRule SESSION:BLOCKED "@eq 1" "log,deny,status:403"

OWASP ModSecurity CRS

The OWASP ModSecurity CRS includes an example ruleset that will validate SessionIDs as part of the optional_rules/modsecurity_crs_43_csrf_protection.conf file.  

# This rule will identify the outbound Set-Cookie SessionID data and capture it in a setsid
SecRule RESPONSE_HEADERS:/Set-Cookie2?/ "(?i:(j?sessionid|(php)?sessid|(asp|jserv|jw)?session[-_]?(id)?|cf(id|token)|sid)=([^\s]+)\;\s?)" "chain,phase:3,t:none,pass,nolog,capture,setsid:%{TX.6},setvar:session.sessionid=%{TX.6},setvar:session.valid=1"
    SecRule SESSION:SESSIONID "(.*)" "t:none,t:sha1,t:hexEncode,capture,setvar:session.csrf_token=%{TX.1}"

# This rule file will identify outbound Set-Cookie/Set-Cookie2 response headers and
# then initiate the proper ModSecurity session persistent collection (setsid).
# The rules in this file are required if you plan to run other checks such as
# Session Hijacking, Missing HTTPOnly flag, etc...

# This rule set will identify subsequent SessionIDs being submitted by clients in
# Request Headers. First we check that the SessionID submitted is a valid one

SecRule REQUEST_COOKIES:'/(j?sessionid|(php)?sessid|(asp|jserv|jw)?session[-_]?(id)?|cf(id|token)|sid)/' ".*" "chain,phase:1,t:none,pass,nolog,auditlog,msg:'Invalid SessionID Submitted.',setsid:%{matched_var},setvar:tx.sessionid=%{matched_var},skipAfter:END_SESSION_STARTUP"
    SecRule SESSION:VALID "!@eq 1" "t:none"

SecAction "phase:1,t:none,nolog,pass,setuid:%{session.username},setvar:session.sessionid=%{tx.sessionid}"


Further description of this ruleset is discussed below.

So What?

Why should you try and validate SessionIDs sent by clients?  I am reminded of President Ronald Reagan's famous saying "Trust, But Verify."  Put it this way, just because a client sends a SessionID within the Request Header Cookie data does not mean that it is a valid one.  There are numerous attacks where malicious clients will send in bogus SessionIDs with the most prevalent one being Credential/Session Prediction where an attacker manipulates the SessionID data in hopes that they can achieve a Session Hijacking attack.  Another attack scenario that was just recently identified is the 'Padding Oracle' Crypto Attack against ASP.Net applications where the attacker sends in bogus encrypted SessionID.  Based on the errors sent back out my the application they may be able to identify the crypto keys which would allow them to possibly decrypt sniffed cookie data or forge tickets.

The key concept with mitigating these types of SessionID attacks is to simply ask yourself - Do you trust the client or the application?  The critical security moment in identifying many application session attacks is to initiate session collection data only when the application hands out a SessionID in a Set-Cookie response header.  If you use the ModSecurity setsid action at this point, you can then also set a new variable within the collection to mark the SessionID as "valid" meaning that it is one that application actually handed out to a client.  With this Session collection data saved, you can then use validation rules that will extract out any SessionID data submitted by client within Request Header Cookie data and verify if the "valid" variable has been sent.  If this variable is not present, then you know that the client is sending a bogus SessionID.

WASC WHID Bi-Annual Report for 2010

The Web Hacking Incident Database (WHID) is a project dedicated to maintaining a record of web application-related security incidents. WHID’s purpose is to serve as a tool for raising awareness of web application security problems and to provide information for statistical analysis of web application security incidents. Unlike other resources covering web site security – which focus on the technical aspect of the incident – the WHID focuses on the impact of the attack. Trustwave's SpiderLabs is a WHID project contributor.

Report Summary Findings

An analysis of the Web hacking incidents from the first half of 2010 performed by Trustwave’s SpiderLabs Security Research team shows the following trends and findings:

  • A steep rise in attacks against the financial vertical market is occurring in 2010, and is currently the no. 3 targeted vertical at 12 percent. This is mainly a result of cybercriminals targeting small to medium businesses’ (SMBs) online banking accounts.
  • Corresponding to cybercriminals targeting online bank accounts, the use of Banking Trojans (which results in stolen authentication credentials) made the largest jump for attack methods (Banking Trojans + Stolen Credentials).
  • Application downtime, often due to denial of service attacks, is a rising outcome.
  • Organizations have not implemented proper Web application logging mechanisms and thus are unable to conduct proper incident response to identify and correct vulnerabilities. This resulted in the no. 1 “unknown” attack category.

WHID Top 10 Risks for 2010

As part of the WHID analysis, here is a current Top 10 listing of the application weaknesses that are actively being exploited (with example attack method mapping in parentheses). Hopefully this data can be used by organizations to re-prioritize their remediation efforts.

WHID Top 10 for 2010


Improper Output Handling (XSS and Planting of Malware)


Insufficient Anti-Automation (Brute Force and DoS)


Improper Input Handling (SQL Injection)


Insufficient Authentication (Stolen Credentials/Banking Trojans)


Application Misconfiguration (Detailed error messages)


Insufficient Process Validation (CSRF and DNS Hijacking)


Insufficient Authorization (Predictable Resource Location/Forceful Browsing)


Abuse of Functionality (CSRF/Click-Fraud)


Insufficient Password Recovery (Brute Force)


Improper Filesystem Permissions (info Leakages)

Download the full report and Join the live Trustwave Webinar Sept. 16th: Web Hacking Incidents Revealed: Trends, Stats and How to Defend (registration required).

What's up @ ModSecurity?

Since Black Hat and DEFCON we have been busying building teams and aligning objectives over here at Trustwave's SpiderLabs. We are committed to driving innovation into the development of ModSecurity for the future.

Here are are few things that we having going on right now:

  • Website Updates - We are in the process of refreshing the website to reflect its new home within SpiderLabs but also clean out any dusty content and replace it with fresh new material. Look for phases of changes over the next several weeks.
  • Recruiting - We are actively looking to hire full-time developers for ModSecurity. Those developers will be part of the SpiderLabs research team. If you or anyone you know is interested, please let us know.
  • Internal Development Team - We have formed a team of people within SpiderLabs that will be actively contributing to ModSecurity. This team is comprised of many very experienced individuals including many OWASP, Black Hat and DEFCON researchers/speakers in the area of web application security. This team is excited to breath new life into this project.
  • End User Survey - In the next few weeks, we will be sending out a survey to the ModSecurity mailing lists to ask for feedback on features to include in the next release.
  • Blog Updates - Each week we'll be writing about an Advance Feature in ModSecurity that the community and end users may not be aware of. The idea is here is that with increased usage of these advanced, we'll get better feedback on how to improve the product.
  • Twitter - We have just linked this blog to the ModSecurity Twitter account. Follow us!

Helping Protect Cookies with HTTPOnly Flag

If you are unfamiliar with what the HTTPOnly cookie flag is or why your web apps should use it, please refer to the following resources -

The bottom line is this - while this cookie option flag does absolutely nothing to prevent XSS attacks, it does significanly help to prevent the #1 XSS attack goal which is stealing SessionIDs.  While HTTPOnly is not a "silver bullet" by any means, the potential ROI of implement it is quite large.  Notice I said "potential" as in order to provide the intended protections, two key players have to work together -

  • Web Applications - whose job it is to append the "HTTPOnly" flag onto all Set-Cookie response headers for SessionIDs, and
  • Web Browsers - whose job it is to identify and enforce the security restrictions on the cookie data so that javascript can not access the contents.

The current challenges to realizing the security benefit of the HTTPOnly flag is that universal adoption in both web apps and browsers is still not there yet.  For example, depending on your web app platform, you may not have an easy mechanism to implementing this feature.  For example - in Java you could following the example provided here on the OWASP site -, however this doesn't work well for the JSESSIONID as it is added by the framework.  Jim Manico has been fighting the good fight to try and get Apache Tomcat developers to implement his patch to add in HTTPOnly support -  The point is that with so many different web app development platforms, it isn't going to be easy to find support for this within every web app that you have to secure...

As for browsers - they too have sporadic, non-consistent adoption of HTTPOnly.  It was for this reason that the OWASP Intrinsic Security group has started an RFC Spec for HTTPOnly -  Hopefully this group will get some traction with the various browser developers.

So, at this point you might be asking yourself - Ryan, that is interesting news and all, but why is this being posted on the ModSecurity site?  What can a web application firewall do to help with this issue?  I would then in turn reply - Great question, I am glad that you asked. ;)

One of my pet peevs with the web application security space is the stigma that is associated with a WAF.  Most everyone only focuses in on the negative security and blocking of attacks aspects of the typical WAF deployment and they fail to realize that WAFs are a highly specialized tool for HTTP.  Depending on your circumstances, you may not ever intend to do blocking.  There are many other use-cases for WAFs and how they can help, in this case as a tactical response tool to help address an underlying vulnerability  In this case, we could monitor when back-end/protected web applications are handing out SessionIDs that are missing the HTTPOnly flag.  This could raise an alert that would notify the proper personnel that they should see if editing the web language code is possible to add this feature in.  A rule to do this with ModSecurity would look like this -

# Identifies SessiondIDs without HTTPOnly flag
SecRule RESPONSE_HEADERS:/Set-Cookie2?/ "!(?i:\;? ?httponly;?)" "chain,phase:3,t:none,pass,log,auditlog,msg:'AppDefect: Missing HttpOnly Cookie Flag.'"
  SecRule MATCHED_VAR "(?i:(j?sessionid|(php)?sessid|(asp|jserv|jw)?session[-_]?(id)?|cf(id|token)|sid))" "t:none"

While this rule is pretty useful for identifying and alerting of the issue, many organizations would like to take the next step and try and fix the issue.  If the web application does not have a way to add in the HTTPOnly cookie flag option internally, you can actually leverage ModSecurity+Apache for this purpose.  ModSecurity has the ability to set environmental data that Apache reads/acts upon.  In this case, we can modify our previous rule slightly to use the "setenv" action and then we add an additional Apache "header" directive that will actually overwrite the data with new Set-Cookie data that includes the HTTPOnly flag -

# Identifies SessiondIDs without HTTPOnly flag and sets the "http_cookie" ENV
# Token for Apache to read
SecRule RESPONSE_HEADERS:/Set-Cookie2?/ "!(?i:\;? ?httponly;?)" "chain,phase:3,t:none,pass,nolog"
  SecRule MATCHED_VAR "(?i:(j?sessionid|(php)?sessid|(asp|jserv|jw)?session[-_]?(id)?|cf(id|token)|sid))" "t:none,setenv:http_cookie=%{matched_var}"

# Now we use the Apache Header directive to set the new data

Header set Set-Cookie "%{http_cookie}e; HTTPOnly" env=http_cookie

The end result of this ruleset is that ModSecurity+Apache can transparently add on the HTTPOnly cookie flag on the fly to any Set-Cookie data that you define.  Thanks goes to Brian Rectanus from Breach for working with me to get the Header directive syntax correct.

One note of warning - make sure that you understand how the web application is handling setting SessionIDs meaning if they are created server-side vs. client-side (in javascript).  This rule set will work fine if the SessionIDs are generated server-side.  If they are created client-side, however, this will disrupt session management.

Hopefully the data presented here will help people who would like to have the security benefit of this flag however are running into challenges with implementing it within the app.

Microsoft and Oracle Helping "Time-to-Fix" Problems

Before I talk to the title of this post, I have to provide a little back story. I have had an ongoing DRAFT blog post whose subject was basically a rant against many vendors who were unwilling to offer vulnerability details. Every now and then I would review and update it a bit, but I never got to the point of actually posting it. I figured it wouldn't do much good in the grand scheme of things and the mere act of updating it provided adequate cathartic relief that a public post was not required.  There has been some recent developments, however that has allowed me to dust off my post and to put a "kudos" spin on it :)

I have long been a proponent of providing options for people to mitigate identified vulnerabilities.  We all realize that the traditional software patching process takes way too long to complete and push out into production when considering that the time it takes for the bad guys to create worm-able exploit code is usually measured in days.  When you combine this with most vendor's vulnerability disclosure policies (which is essentially not to disclose any details), then it is obvious that the bad guys have a distinct advantage in this particular arms race...

Ideally, all vulnerability researchers would work with the vendor and they would jointly release details with patches and then customers would immediately implement them on production hosts.  Unfortunately, reality is much different.  Researchers often have their own agendas and decided to release vulnerability details on their own.  In these cases, the end users have no mitigation options provided by the vendor and are thus exposed to attacks.  For those situations where the researchers and the vendor work together, then the end user at least has a fix that they can apply.  The problem is that the standard time-to-fix for organizations to test and install patches is usually a couple months.  So, the vendor has pushed the pig over the fence onto the customer and essentially takes a "it's now your problem now" approach.

What would be useful would be some technical details on the vulnerabilities that are addressed within the patches.  Let's take a look at Oracle's position on public disclosure.  The fact that this is Oracle is irrelevant as many vendors share this same view and that is that they don't want to disclose any technical details of a vulnerability BEFORE patches are released.  I really can't fault them for this stance as they want to ensure that they have patches ready.  What I am focusing on here is when they have a patch set ready, they should provide enough technical details about the vulnerability so that an organization can implement some other mitigation options until the actual patches are installed.  Unfortunately, the vendors position is that they didn't want to release the details as to prevent the bad guys from obtaining the info.  What they are missing, however, is that both the good guys (Sourcefire, iDefense, etc...) and the bad guys are reverse engineering the vendors patches to uncover the details about the vulnerability.  The only people who don't have any details are the end users.

So the point is that Pandora is already out of the box when vendors release patches.  What they should do then is to give technical details for security folks to implement some defenses (for IDSs/IPSs).  A great example of this is when bleeding edge/emerging threats folks would create Snort signatures so that an organization can identify if someone is attempting to exploit a flaw.

Now, the whole point of this post is to highlight that I have been fighting the good fight with many vendors to try and get them to see the light on the value of either releasing technical details on web-based vulnerabilities so that end users can create virtual patches with a web application firewall, or even better, for the vendor to release some virtual patches themselves (using the ModSecurity rules language).  Well, we haven't achieved the latter one yet but we are seeing signs that both Oracle and Microsoft are starting to address the former.  Specifically, Oracle/BEA recently released details about a WebLogic plug-in for Apache and in the mitigation section they actually mentioned the use of ModSecurity to address the problem!  That is a huge step and something that I am extremely excited about.  Then just within the last week we saw the announcement of Microsoft's Active Protections Program (MAPP).  Here is the short overview -

The Microsoft Active Protections Program (MAPP) is a new program that will provide vulnerability information to security software providers in advance of Microsoft Corp.’s monthly security update release. By receiving vulnerability information earlier, security software providers can give customers potential improvements to provide security protection features, such as third-party intrusion detection systems, intrusion prevention systems or security software signatures.

This is certainly an interesting initiative and may help organizations to receive more timely mitigation options to help protect themselves until the official patches are deployed.

Overall, I have have say GREAT job Oracle and Microsoft for truly helping your customers to close their time-to-fix windows.   

Enough With Default Allow Revision 2

A revised version (but still a draft) of the Enough With Default Allow in Web Applications! paper is now available for download. (My previous post on this topic is here.) The major changes in this version include:

  1. Decided to use a flat model of resources, rather than a hierarchical one, after realising the nested approach would make models very difficult to read for any non-trivial application. Also, we wanted to support the virtual patching case, which doesn't work with nesting very well.
  2. Behaviours can now specify character encodings, which is very important in order to properly parse parameters.
  3. We've allowed for a per-model data dictionary, which would allow parameter types to be defined once and reused throughout the model.
  4. Many clarifications and small fixes throughout.

Update (4 Aug 2008): Updated links to point to the final version (spell-checked, reviewed and branded) of the paper.

Enough with Default Allow in Web Applications!

The title of this blog post is also the title of a research paper we are currently working on. Although the paper is still in draft form, we've decided to circulate it widely (download here) because we believe a public exposure in this early stage would benefit it significantly. Also, as you will see from the paper, for the proposed concept to succeed it must have support from many diverse groups of users. What do we want to achieve? Let's look at the summary:

The default allow deployment model, which is commonly used to implement
and deploy web applications, is the cause of numerous security problems. We propose
a method of modelling web applications in a platform-agnostic way to adopt
a default deny model instead, removing several classes of vulnerability altogether
and significantly reducing the attack surface of many others. Our approach is best
adopted during development, but can be nearly as efficient as an afterthought, or
when used at deployment time.

Our main problem is with these three things:

  1. HTTP (in a wider sense, taken to mean all specifications needed to develop and deploy web applications) has grown to be quite complex, but although applications generally use a very small subset they are still expected to support every single feature.
  2. Many applications are treated as simple collections of files (if it's on the filesystem it's part of application), and this is creating all sorts of issues.
  3. Applications do not specify their external interfaces. This is really a consequence of the above two problems. Applications cannot specify external interfaces because they are all implicit.

The bottom line is that we have a chance to create a beautifully positioned protection layer (between web servers and applications), which would not only increase security overall, but turn applications into verifiable components with external contracts that can be enforced.

We propose a use of a platform-independent format to document what applications are willing to accept from the outside world, with the following use cases envisioned:

  1. Creation of full application models, which reduce application attack surface. Such models can be created by application developers (which is preferred) or by application users (which, we expect, could happen with very popular and/or open source applications).
  2. Creation of partial application models for use in virtual patching.
  3. Automated creation of application models through traffic analysis.

In addition to the paper itself, we are planning to release an open source profiling tool (which I will announce next week) to help with the third use case and automate the creation of positive security models (also known as the learning feature of web application firewalls).

Download Enough With Default Allow in Web Applications!

Update (4 Aug 2008): Changed links to point to the final version (reviewed, spell-checked and branded) of the paper. The follow up post is here.


November 2010
Sun Mon Tue Wed Thu Fri Sat
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30


Atom Feed



Recent Entries