Apache.org details recent security breach
The infrastructure team of The Apache Software Foundation released an incident report regarding the recent security breach that affected Apache.org.
It’s important to note that at no time were any Apache Software Foundation code repositories, downloads, or users put at risk by this intrusion. The Apache infrastructure team posted their analysis of the situation so that the community get get a better understanding of what happened.
The incident report
The server that hosted the apachecon.com (dv35.apachecon.com) website was compromised. The machine was running CentOS, and the attackers may have used the recent local root exploits patched in RHSA-2009-1222 to escalate their privileges on this machine which was fully compromised. This included gaining root and destroying most of the logs. This is why it was difficult to confirm the details about what exactly happened with this machine.
While this machine is owned by the ApacheCon conference production company and not by the Apache Software Foundation, members of the ASF infrastructure team had accounts on this machine, including one used to create backups.
Once the attackers gained access to apachecon.com, they attempted to use the passwords they had to login to the production webservers. They succeeded in getting access to people.apache.org using the SSH key of the backup account (minotaur.apache.org). Fortunately, this account was used to create backups from the ApacheCon host and it was an unprivileged user.
minotaur.apache.org acts as the staging machine for the Apache mirror network. It’s a primary shell account server which provides various services to Apache developers. However, none of the version control data is kept on this machine so there was no risk to Apache source code getting compromised.
Once the attackers had shell access, they added CGI scripts to the document root folders of several Apache websites. A scheduled rsync process copied these scripts to the production web server (eos.apache.org) and they became externally visible. These scripts were used to obtain remote shells, with information sent using HTTP POST commands.
To mitigate the problem, immediately after the CGI scripts have been discovered, the Apache infrastructure team shut down all servers that could have been affected: people.apache.org, as well as the EU and US website servers.
The potentially compromised servers were brought up in single user mode, using out of band access. At this point, it was clear that the EU webserver, aurora.apache.org, was not affected. The CGI scripts that were rsync’d to that machine have never been run.
After aurora.apache.org was brought up, the Apache team determined that the route of the breach was the backup system from dv35.apachecon.com so they took the logs from the server and shut it down. Analysis was resumed on the backup and US servers until all traces of a compromise have been removed and the machines were brought back online.
To learn more about what the Apache infrastructure team is doing to improve security in the future, read this blog post.