ModSecurity Blog: Apache

A very interesting research paper titled "Apache Prefork MPM Vulnerabilities" was released a few days ago, as you can see in the corresponding Bugtraq post. The paper describes, in detail, the dangers of allowing third-parties to run code under the same account as the Apache web server. This normally happens when dynamic content is produced using Apache modules (e.g. PHP) or when CGI scripts are configured to run without suEXEC. This topic itself is not new. You will find several articles on runtime process infection following this Google search link. I warn about this problem throughout my book and especially in Chapter 6, which is dedicated to those situations when more than one party is using the one Apache installation. However, it is one thing to know that something is possible and another to demonstrate, step by step, how it is done. Another interesting finding resulting from this paper is that it is possible to send a SIGUSR1 signal, as root, to any process on the system instead of just to Apache children processes. This is an issue that will have to be fixed in one of the future versions of Apache.

This problem with running code as the same identity as the web server is well understood (and has been for years) among the advanced Apache users. The solution is to always execute CGI scripts through suEXEC and to never allow third parties access to any of the modules. The real problem is that, as with any other product, there are few people who understand Apache inside out (and they can protect themselves) but there also those who are using the technology but do not have the luxury of learning everything there is about it (and there are many legitimate reasons for that).

The solution is obvious. Apache must be safe out of the box! We should dispense with the idea of running things in the same process. Process isolation facilities (either suEXEC or something else) should be installed and running by default on all installations. We can and should make provisions for those who know what they are doing to shoot themselves in the foot, of course. But the only reason to attempt to run things in the same process is performance and I suspect, in this day and age, virtually all users will be happy with the performance of their web server doing things in a secure manner.

Last week I spent some time stress-testing Apache 2.2.3 configured to work as a reverse proxy. I discovered (actually, re-discovered would be more accurate) two issues worth sharing.

1. Memory consumption of an Apache process will steadily increase as the number of processed requests rises. This is very easy to see if you send thousands of requests per second, with each request going to the same process. This has to be either a memory leak or a memory fragmentation issue. To deal with this you need to recycle processes before they become too large (and cause the operating system to start swapping). The MaxRequestsPerChild directive is meant to help with this. By setting its value to something other than zero (which means "unlimited") you are telling Apache to shut down every process that goes over the limit. No problems there. Except that it's where the second problem comes in.

2. The MaxRequestsPerChild directive does not work as the name suggests. Apache does not count requests - it counts *connections*. This creates a problem if you have persistent connections enabled in your configuration - you don't know how many requests will come over a connection. It is probably safe to assume the number will not be large in most cases but you won't know if someone will try to abuse this problem and force a large number of requests over a single connection (e.g. by using a specially programmed script). To be on the safe side you need to divide your ideal MaxRequestsPerChild value with the MaxKeepAliveRequests value. This will prevent the Apache processes from growing too large. But there's a side effect - Apache will now recycle its worker processes more often. As your final step you need to make sure there are enough idle processes around (using MinSpareServers) to jump in as soon as an active process goes down. Yo need to have a few of these processes because there is a performance penalty associated with the creation of a new process and because Apache creates new processes at a rate of one every second.
   

My book was translated to Japanese and published by O'Reilly Japan! This is, apparently, old news, as they did it back in 2005, but I only found out about it from the three-montly royalties statement I received in April.

While we are on the subject of writing, I am starting to get the itch again. There are two or three topics I would like to explore further. Topics such as web application firewalls and ModSecurity, web application security, and application security patterns. On the other hand, I have a few compelling reasons against writing another book:

• It takes a lot of time (time better spent building Thinking Stone into a stronger business).
  
• It's lonesome (but this can be dealth with by finding someone to co-author the book with me.
  
• My hands and arms haven't fully recovered from the first book. (This one is the most compelling reason of all - I barely managed to finish Apache Security in the first place. If you are using keyboard extensively make sure to read about RSI and always keep Workrave active.)
  

I have been involved with Apache programming for several years now. During this time I've been following the main Apache development list and the module programming one. This is how I got to meet Nick Kew, probably the most helpful person on these two lists. (Perhaps on other lists too, but I only follow these two.) Rumours that Nick is writing a book (spread by the author himself) have been circulating for many months now. I am happy to say this is now official; Nick's book, Apache Programming (I am not sure if this is the official title or not) will be published by Prentice Hall in their Open Source Series. Nick has been kind to invite me to help him as a technical reviewer. This is great news for the Apache community! Apache is a fantastic web server but its growth is being slowed down by the lack of proper documentation for programmers. I only wish I had this book a couple of years back when I was starting with ModSecurity!

Yury Zaytsev wrote to me recently to tell me about his experiences in jailing Apache on Windows. Although, strictly speaking, Windows does not have the chroot system call or an equivalent it is still possible to do a pretty good job restricting its access to the system, as Yury demonstrates. From his email:

All you need is to make a local user, say, called "Apache" (you may even set him a password, don't think that makes any sense, but anyway) and deny him local and network login via group policies. Then you need to explicitly deny this user any access to the local drives (deny just everything: dir listing, read, write, modify etc), that's done via Properties - Security. Now any process spawned with "Apache"'s rights won't be able even to LIST the directories.

Now you've got to grant it the read/list folders access to the Apache Software folder (done via folder properties -> security) and write access to the PID file and log files (hopefully it doesn't need anything else).

And the last thing to do is to edit Apache's system service: you should change it's privileges via My computer - Manage workstation - Services - Apache from System service to "Apache" user (it might prompt you for the password if you've set any).

Reboot, check that Apache process is running as Apache user via the Task Manager, make sure everything is working fine and you're done.

This has also an important positive impact on the scripts security: now even if one manages to hack your poorly coded PHP/Perl script, since PHP/Perl is run via SAPI/mod_perl it couldn't list folders above Apache's root and even change any files you haven't allowed it explicitly inside Apache's root.

As you may see from above, my method is a complete rip off the Unix chroot (and chmod, he-he) technology. It's primitive and efficient (..er how efficient a Windows server can be comparing to Unixes?) Anyway it really saved my butt several times the script kiddies managed to exploit vulnerable PHP scripts.

I was recently involved with a project where we needed to configure an Apache server that was intended to run multiple web sites/applications. It's a pretty common assignment. To ensure the setup is secure I decided to start by creating a separate user account for each application. This allowed me to correctly configure file permissions to allow Apache to serve the static files directly. To take care of the dynamic content, I configured suEXEC to execute each application's scripts under its own account. (In case you are wondering, this particular server is fast enough to run the scripts as CGIs. But if process creation becomes a bottleneck we can always seamlessly switch to FastCGI to avoid the performance penalty. Nothing to worry about, then.)

SuEXEC is a great tool but I'd love it to be capable of jailing (via the chroot system call) the binaries it executes. However, this feature is not present in the stock version. Having been responsible for the internal chroot feature of ModSecurity, I think I have a pretty good idea of why this is the case: unless you know what you're doing it's pretty easy to break applications with chroot. And if that happens you are going to ask for help... from those that created the feature, right? Of course! As it turns out, chrooting is notoriously difficult to debug remotely and that's why the developers would much rather not deal with it.

But, if do you know your way around feel free to use my suexec chroot patch, which I have just added to the Apache Tools project. But, please, don't write to me if it's not working as you are expecting :)

A new beta version of the Apache web server has been released. This release is important because it is a beta version in the 2.1.x development branch, meaning Apache is one step closer to an official 2.2 version. The most important changes are the refactoring of the authentication/authorisation backend, faster and better caching support, and the much improved mod_proxy with support for load balancing.

I have made the PHP chapter from Apache Security available for free download. When we made the decision to set the installation and configuration chaper free, several months ago, I did not realise this chapter only told one half of the story. Most people need to configure Apache *and* PHP. This is now fixed, and the two chapters together make a valuable resource. My long-term plans are to convert both chapters to DocBook, keem them up-to-date, and publish them as PDF and as HTML. But not yet--the content is still very fresh!

My book, Apache Security, is finally out, after a year and a half of hard labour. I began talking to the publisher in the summer of 2003, and began research shortly after. I began writing in March 2004 and finished in December 2004. O'Reilly had it in stock on March 1st 2005, but it only become widely available in late March.

The work itself was great fun. It is a great privilege to be able to explore the boundaries of your own knowledge in detail. I did have periods of despair, of course. But I was prepared for them from reading blog entries of other book writers. I knew how difficult it was going to be. The biggest challenge I had was deciding what to cover, and what to leave out. It was never going to be a book for absolute beginners (officially, it's an "intermediate to advanced" type of book) but I didn't want to write a book that would be understood only by a few people who are already Apache experts. On an another level, it was also not possible to look at the security of the Apache web server in isolation. A book that pretends to provide "everything you need to know to secure your Apache web server" must delve into topics such as networks security, host security, and web application security. As it turned out I had to deal with these questions every single day. It was a struggle to keep the book from growing too much. Initially, the book was supposed to be around 280 pages long. In the end, it grew to over 400 pages.

As my work progressed I began to think more and more about the process. The traditional book writing process restricts the author to his experience, the experience of his immediate peers, and the experience of the technical reviewers. While this may work in some, or even many cases, I came to believe that a piece of technical writing can achieve its best only through the collaboration process with the readers. Now that the book is out, this is exactly what I am looking forward to.

The first step, the book's web site, is already completed. In the following months I plan to put more material from the book online, start adding fresh content, and generally try to engage the public by offering them the stuff they are interested in. The way I see it, my work has just began.

The Apache Security cover will feature a horse, I am happy to say. I knew all along my book was going to be an animal book but the identity of the animal was not known until fairly recently. Now that the animal is known and the tagline ("The Complete Guide to Securing Your Apache Web Server") is sorted I decided to publish the cover for everyone to see. The official launch date is less than two weeks from now so I'll save my "it was a difficult job" speech for then.

A beta chapter, Installation and configuration, is also available for download. A part of me wanted to advertise my knowledge by giving away one of the more exciting chapters. But since properly installing and configuring Apache is very important, in the end I decided to give away the less exciting but probably more useful chapter.

As a matter of fact, I signed the contract in March, two months ago. The fact I am announcing it just now speaks for itself how busy I am. Obviously, the fact that I am writing a book is very exciting. This is something I wanted to do for a long time. I think my wish has to do with the way my brain works. I hate keeping stuff in my head and I adore that magnificient feeling of putting what I have in my head to paper - freeing the mind to deal with new things.

The way I see it, this book is going to be one long checklist with plenty of very interesting information. No, no, do not take that literally. It is only the appendix that will be a checklist. The rest will consist of the information required to maintain a successful and secure Web presence.

There is a really nice article on ONLamp: Profiling LAMP Applications with Apache's Blackbox Logs. Most Apache users accept the default logging configuration and often never realise exactly what is going on. This works when things are going well, but not so much when a problem arises and needs to be troubleshooted. In the article, Chris Josephes explains how to make the most of the Apache logging facilities.

It is surprising how little people talk about logging, given that data is essential for forensic analysis. One of the reasons I initially created mod_security is to allow me to see the contents of POST requests.

Syhunt, a security tool company from Brazil, have released a free Apache configuration hardening utility. The utility feeds on Apache or PHP configuration files, and gives warnings and suggestions how to make the configuration more secure. I especially like the fact that they advise people to install and use mod_security :)

I thought a post to mark the end of the year would be in order. It has been a very good year for ModSecurity - it's gone from just an idea to a stable and useful product. The toughest part was not to figure out what functionality to add, but to bend the Apache into supporting it.

After the big marketing push in October (with the SecurityFocus article and the Slashdot link) our popularity rose significantly. We now have over 1500 downloads every month.

But if you think 2003 was good, wait to see what will happen in 2004. I have many things planned. Some of them will be the improvements to the existing software, but many will be completely new products. I wish you a very happy 2004 :)

Building on the multipart/form-data support I added to mod_security the other day, today I added two new configuration directives to support file interception. Using SecUploadDir you can tell mod_security to store files (works on per-directory configuration so you can have any number of different folders), and by setting SecUploadKeepFiles to On you can tell it not to erase after it's done with them. Pretty nice. The next step, a hook to execute external programs to verify uploaded files, is coming soon.

Over the weekend I worked on adding the multipart/form-data support to mod_security. As a result, the Apache 1.x version in CVS now supports it. It still needs to be polished, of course, but this feature opens a door for other interesting features, such as intercepting file uploads. For every uploaded file you will be able to invoke a script to verify it (grep, anti-virus, whatever...), or to store the file for analysis later. This is the last large chunk of functionality I wanted to have before labeling mod_security v2.0 and I am glad it is getting closer.

I feel like I've reached a new milestone with mod_security. First of all, it is important to note that since Monday my busy period is officially over and I can now spend more time working on mod_security. Second, in the last two weeks I published a new version (1.7), mod_security made it to Slashdot (reaching a wide audience), and my reverse proxy article was published on SecurityFocus (reaching a wide focused audience). Consequently, the number of downloads and visits increased tenfold.

I have also had a vulnerability. Not a big one, but big enough to let me know that I must be very careful from now on: no more late night coding without a plan and more work on testing and quality control.

These events are marking the end of the second phase in the life of mod_security. We are slowly entering a third phase - maturity. I've decided to enter maturity with a 2.0 version number. The next version, 1.8, will carry some must-have enhancements, and it will become a solid 2.0 version after a lot of polishing. Yes, polishing is important. I have been running for a while now, it is time to slow down to a walk and think more about the features we already have.

The new version of the script to convert Snort rules into mod_security rules is now available (from the same page as before). I initially forgot to escape characters that have a meaning in regular expressions and some rules were preventing Apache from starting.

I have also added these files to the CVS so they will make it into the official distribution starting with v1.8.

The last change before the 1.7 release is now in the CVS. I have refactored the code dealing with rule processing, and added three new actions: allow, skipnext, and chain. Allow enables you to stop rules processing on a single rule match, and let the request through. With skipnext you can skip one or more rules. Finally, the chain action is used to chain several filters together, essentially a logical AND. The final rule in the chain will be processed only if all rules before it match. These rules make mod_security much more fun to play with.

So, let's say that you want to restrict administration access to an application but you can't do it with standard Apache directives because the admin shares the same login panel as other users. No problem:

SecFilterSelective ARG_username admin chain
SecFilterSelective REMOTE_ADDR "!^YOUR_IP_ADDRESS_HERE$"

Now you can analyse cookies using new selective filtering variables (COOKIE_name, COOKIE_NAMES, COOKIE_VALUES). Even before this change it was possible to look at cookies (as cookies are just HTTP headers) but the functionality was limited. ModSecurity now parses cookies for you.

Let's say you wanted to prevent XSS attacks via the PHP session cookie; this filter would make sure the cookie is in order:

SecFilterSelective COOKIE_PHPSESSID "^[0-9a-z]+$"

COOKIE_NAMES and COOKIE_VALUES will examine all cookie names and values, respectively.

There is a new feature available in the CVS, and it allows you to mask your web server and instruct it to pretend to be something else. Normally, to do this sort of thing you would have to change Apache source code and recompile, now you can do it with a single configuration directive. Just type:

SecServerSignature "Microsoft-IIS/5.0"

I just came across an excellent web security book - HackNotes(tm) Web Security Pocket Reference. It is very refreshing; short but with a great deal of information and practical examples. You will find it very useful even if you think you knww everything there is to know about web security.

The new output filtering functions are now in CVS. I implemented this feature for Apache 2 first because of two reasons. First, this version supports the notion of input/output filters - making filtering work is simply a case of using the module API. In addition to that, I've been using mod_security as part of a reverse proxy for some time now and I really, really wanted it to be able to filter output.

As for Apache 1.x, well, I have a pretty good idea how I would be able to implement output filtering even without an API support. It is tricky, and I am not sure whether it will be portable (to Windows) but I'll give it a try.

To use output filtering, first turn it on by typing:

SecFilterOutput On

You are then free to apply filters against the output using a new selective filtering variable "OUTPUT". LIke this:

SecFilterSelective OUTPUT "some rude word"

I've just committed the Unicode validation feature to the CVS. It is a very good thing to have if the application or the operating system support and/or understand Unicode. Most importantly, this feature will protect from attacks where an ASCII character is encoded with more than one byte thus avoiding detection. In addition to this, ModSecurity checks that there is sufficient number of bytes available, and that all bits in all bytes have correct values. For a detailed description of the Unicode attack have a look at the OWASP guide.

I've just added a new feature to mod_security (CVS, both versions) that allows you to achieve a better control of what gets filtered. Up until now mod_security looked at every single request. Since most static resources (e.g. images) are not vulnerable it is safe to assume that we don't need to look at those types of requests. And, the number of image requests versus (dynamic) page requests is much bigger the savings in CPU cycles are probably quite big.

I've had quite a lot of "fun" with PHP CLI scripts the other day. As you perhaps know, there is an "exec" feature built into mod_security that allows you to execute some external binary in response to a filter match. This was working properly with Perl scripts but not with PHP scripts.

It seems that the problem lies in PHP itself. Apparently, it is designed to work as a CGI engine and as a command line binary at the same time, and uses environment variables to figure out how it was run. So, when it sees a bunch of web-releated environment variables it concludes that it should behave like a CGI script. That alone would not be a problem but then certain security restrictions kick in and script execution just stops.

Finally, after reading the source code for PHP the problem was resolved by simulating REDIRECT_STATUS and PATH_TRANSLATED environment variables from mod_security. Thanks to Shane Lahey who discovered this problem and exchanged countless emails with me until we nailed it down.

I've added a new (and experimental) feature to mod_security (CVS and Apache 1.x only at the moment) that greatly simplifies the process of chrooting in most cases.

Essentially, the chroot call is made from Apache itself, at the very end of the initialisation process. The beauty of it is that Apache performs everything it needs (shared libraries, log files) before the chroot call and that allows you to put only data files into the jail.

I've written a short article here:
http://www.modsecurity.org/documentation/apache-internal-chroot.html

and the link in CVS is (again, only Apache 1.x):
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/mod-security/mod_security/apache1/mod_security.c?rev=1.4

What I have described works perfectly for me but I am interested to hear other opinions (or experiences). If you are interested please give it a try and let me know how you feel.

I just fixed a small bug in the URL decoding routine. Apparently, I forgot to add code to convert '+' characters into spaces. It is a great comfort to use regression testing while development. So, this time, before making any changes to the source code I created a test (#31) to test the bug. Then I fixed, and run tests again. Now I know that the bug is fixed, but I also know that the functionality I had before is still there.

With module functioning well on Unix-based platforms I decided to start with the Windows port. The job was straightforward (I only tried with the Apache 1.x version): after creating the makefile and getting all the switches right I was rewarded with a file mod_security.dll.

That's when the trouble started - I couldn't open (from the module) any files or write to them. After spending a lot of time debuggung, I found that there seems to exist a problem with ap_popenf (portable Apache function that opens files) - it returns invalid file handles. After replacing the call to ap_popenf with a call to open, everything worked just fine. This is probably something specific to the version of Windows I have (Windows Me) otherwise other people would have reported it by now (I've found only one such report using Google, with no answer).

So, with the exception of the ap_popenf problem, the module seems to work quite well on Windows, passing all regression tests. The code is in the CVS if you want to give it a try.