ModSecurity Trustwave
This blog has moved! Please update your
bookmarks to http://blog.spiderlabs.com/modsecurity/.

ModSecurity Blog: Apache

Apache Process Infection

A very interesting research paper titled "Apache Prefork MPM Vulnerabilities" was released a few days ago, as you can see in the corresponding Bugtraq post. The paper describes, in detail, the dangers of allowing third-parties to run code under the same account as the Apache web server. This normally happens when dynamic content is produced using Apache modules (e.g. PHP) or when CGI scripts are configured to run without suEXEC. This topic itself is not new. You will find several articles on runtime process infection following this Google search link. I warn about this problem throughout my book and especially in Chapter 6, which is dedicated to those situations when more than one party is using the one Apache installation. However, it is one thing to know that something is possible and another to demonstrate, step by step, how it is done. Another interesting finding resulting from this paper is that it is possible to send a SIGUSR1 signal, as root, to any process on the system instead of just to Apache children processes. This is an issue that will have to be fixed in one of the future versions of Apache.

This problem with running code as the same identity as the web server is well understood (and has been for years) among the advanced Apache users. The solution is to always execute CGI scripts through suEXEC and to never allow third parties access to any of the modules. The real problem is that, as with any other product, there are few people who understand Apache inside out (and they can protect themselves) but there also those who are using the technology but do not have the luxury of learning everything there is about it (and there are many legitimate reasons for that).

The solution is obvious. Apache must be safe out of the box! We should dispense with the idea of running things in the same process. Process isolation facilities (either suEXEC or something else) should be installed and running by default on all installations. We can and should make provisions for those who know what they are doing to shoot themselves in the foot, of course. But the only reason to attempt to run things in the same process is performance and I suspect, in this day and age, virtually all users will be happy with the performance of their web server doing things in a secure manner.

Apache Reverse Proxy Memory Consumption Observations

Last week I spent some time stress-testing Apache 2.2.3 configured to work as a reverse proxy. I discovered (actually, re-discovered would be more accurate) two issues worth sharing.

  1. Memory consumption of an Apache process will steadily increase as the number of processed requests rises. This is very easy to see if you send thousands of requests per second, with each request going to the same process. This has to be either a memory leak or a memory fragmentation issue. To deal with this you need to recycle processes before they become too large (and cause the operating system to start swapping). The MaxRequestsPerChild directive is meant to help with this. By setting its value to something other than zero (which means "unlimited") you are telling Apache to shut down every process that goes over the limit. No problems there. Except that it's where the second problem comes in.

  2. The MaxRequestsPerChild directive does not work as the name suggests. Apache does not count requests - it counts *connections*. This creates a problem if you have persistent connections enabled in your configuration - you don't know how many requests will come over a connection. It is probably safe to assume the number will not be large in most cases but you won't know if someone will try to abuse this problem and force a large number of requests over a single connection (e.g. by using a specially programmed script). To be on the safe side you need to divide your ideal MaxRequestsPerChild value with the MaxKeepAliveRequests value. This will prevent the Apache processes from growing too large. But there's a side effect - Apache will now recycle its worker processes more often. As your final step you need to make sure there are enough idle processes around (using MinSpareServers) to jump in as soon as an active process goes down. Yo need to have a few of these processes because there is a performance penalty associated with the creation of a new process and because Apache creates new processes at a rate of one every second.

Apache Security in Japanese!

Apache Security in Japanese cover page

My book was translated to Japanese and published by O'Reilly Japan! This is, apparently, old news, as they did it back in 2005, but I only found out about it from the three-montly royalties statement I received in April.

While we are on the subject of writing, I am starting to get the itch again. There are two or three topics I would like to explore further. Topics such as web application firewalls and ModSecurity, web application security, and application security patterns. On the other hand, I have a few compelling reasons against writing another book:

  • It takes a lot of time (time better spent building Thinking Stone into a stronger business).
  • It's lonesome (but this can be dealth with by finding someone to co-author the book with me.
  • My hands and arms haven't fully recovered from the first book. (This one is the most compelling reason of all - I barely managed to finish Apache Security in the first place. If you are using keyboard extensively make sure to read about RSI and always keep Workrave active.)

Apache Programming Book On The Way!

I have been involved with Apache programming for several years now. During this time I've been following the main Apache development list and the module programming one. This is how I got to meet Nick Kew, probably the most helpful person on these two lists. (Perhaps on other lists too, but I only follow these two.) Rumours that Nick is writing a book (spread by the author himself) have been circulating for many months now. I am happy to say this is now official; Nick's book, Apache Programming (I am not sure if this is the official title or not) will be published by Prentice Hall in their Open Source Series. Nick has been kind to invite me to help him as a technical reviewer. This is great news for the Apache community! Apache is a fantastic web server but its growth is being slowed down by the lack of proper documentation for programmers. I only wish I had this book a couple of years back when I was starting with ModSecurity!

Jailing Apache On Windows

Yury Zaytsev wrote to me recently to tell me about his experiences in jailing Apache on Windows. Although, strictly speaking, Windows does not have the chroot system call or an equivalent it is still possible to do a pretty good job restricting its access to the system, as Yury demonstrates. From his email:

All you need is to make a local user, say, called "Apache" (you may even set him a password, don't think that makes any sense, but anyway) and deny him local and network login via group policies. Then you need to explicitly deny this user any access to the local drives (deny just everything: dir listing, read, write, modify etc), that's done via Properties - Security. Now any process spawned with "Apache"'s rights won't be able even to LIST the directories.

Now you've got to grant it the read/list folders access to the Apache Software folder (done via folder properties -> security) and write access to the PID file and log files (hopefully it doesn't need anything else).

And the last thing to do is to edit Apache's system service: you should change it's privileges via My computer - Manage workstation - Services - Apache from System service to "Apache" user (it might prompt you for the password if you've set any).

Reboot, check that Apache process is running as Apache user via the Task Manager, make sure everything is working fine and you're done.

This has also an important positive impact on the scripts security: now even if one manages to hack your poorly coded PHP/Perl script, since PHP/Perl is run via SAPI/mod_perl it couldn't list folders above Apache's root and even change any files you haven't allowed it explicitly inside Apache's root.

As you may see from above, my method is a complete rip off the Unix chroot (and chmod, he-he) technology. It's primitive and efficient (..er how efficient a Windows server can be comparing to Unixes?) Anyway it really saved my butt several times the script kiddies managed to exploit vulnerable PHP scripts.

Apache suEXEC chroot patch

I was recently involved with a project where we needed to configure an Apache server that was intended to run multiple web sites/applications. It's a pretty common assignment. To ensure the setup is secure I decided to start by creating a separate user account for each application. This allowed me to correctly configure file permissions to allow Apache to serve the static files directly. To take care of the dynamic content, I configured suEXEC to execute each application's scripts under its own account. (In case you are wondering, this particular server is fast enough to run the scripts as CGIs. But if process creation becomes a bottleneck we can always seamlessly switch to FastCGI to avoid the performance penalty. Nothing to worry about, then.)

SuEXEC is a great tool but I'd love it to be capable of jailing (via the chroot system call) the binaries it executes. However, this feature is not present in the stock version. Having been responsible for the internal chroot feature of ModSecurity, I think I have a pretty good idea of why this is the case: unless you know what you're doing it's pretty easy to break applications with chroot. And if that happens you are going to ask for help... from those that created the feature, right? Of course! As it turns out, chrooting is notoriously difficult to debug remotely and that's why the developers would much rather not deal with it.

But, if do you know your way around feel free to use my suexec chroot patch, which I have just added to the Apache Tools project. But, please, don't write to me if it's not working as you are expecting :)

Apache 2.1.7 beta released

A new beta version of the Apache web server has been released. This release is important because it is a beta version in the 2.1.x development branch, meaning Apache is one step closer to an official 2.2 version. The most important changes are the refactoring of the authentication/authorisation backend, faster and better caching support, and the much improved mod_proxy with support for load balancing.

PHP chapter from Apache Security available for download

I have made the PHP chapter from Apache Security available for free download. When we made the decision to set the installation and configuration chaper free, several months ago, I did not realise this chapter only told one half of the story. Most people need to configure Apache *and* PHP. This is now fixed, and the two chapters together make a valuable resource. My long-term plans are to convert both chapters to DocBook, keem them up-to-date, and publish them as PDF and as HTML. But not yet--the content is still very fresh!

The public life of Apache Security begins


My book, Apache Security, is finally out, after a year and a half of hard labour. I began talking to the publisher in the summer of 2003, and began research shortly after. I began writing in March 2004 and finished in December 2004. O'Reilly had it in stock on March 1st 2005, but it only become widely available in late March.

The work itself was great fun. It is a great privilege to be able to explore the boundaries of your own knowledge in detail. I did have periods of despair, of course. But I was prepared for them from reading blog entries of other book writers. I knew how difficult it was going to be. The biggest challenge I had was deciding what to cover, and what to leave out. It was never going to be a book for absolute beginners (officially, it's an "intermediate to advanced" type of book) but I didn't want to write a book that would be understood only by a few people who are already Apache experts. On an another level, it was also not possible to look at the security of the Apache web server in isolation. A book that pretends to provide "everything you need to know to secure your Apache web server" must delve into topics such as networks security, host security, and web application security. As it turned out I had to deal with these questions every single day. It was a struggle to keep the book from growing too much. Initially, the book was supposed to be around 280 pages long. In the end, it grew to over 400 pages.

As my work progressed I began to think more and more about the process. The traditional book writing process restricts the author to his experience, the experience of his immediate peers, and the experience of the technical reviewers. While this may work in some, or even many cases, I came to believe that a piece of technical writing can achieve its best only through the collaboration process with the readers. Now that the book is out, this is exactly what I am looking forward to.

The first step, the book's web site, is already completed. In the following months I plan to put more material from the book online, start adding fresh content, and generally try to engage the public by offering them the stuff they are interested in. The way I see it, my work has just began.

Apache Security cover and beta chapter available!

Apache Security cover The Apache Security cover will feature a horse, I am happy to say. I knew all along my book was going to be an animal book but the identity of the animal was not known until fairly recently. Now that the animal is known and the tagline ("The Complete Guide to Securing Your Apache Web Server") is sorted I decided to publish the cover for everyone to see. The official launch date is less than two weeks from now so I'll save my "it was a difficult job" speech for then.

A beta chapter, Installation and configuration, is also available for download. A part of me wanted to advertise my knowledge by giving away one of the more exciting chapters. But since properly installing and configuring Apache is very important, in the end I decided to give away the less exciting but probably more useful chapter.