Thanks, “bobocat” (sorry for calling you bobcat earlier:), that’s a good idea. I’ve already chmod’ed lots of stuff back, but enough remains so that ls -alct|less makes it clear everything happened 2012-02-15 23:00 (late this past Wednesday, way past lakerat’s 9/2/11), at which time it looks like only “omalonzo” and I were logged on. Kind of suggests I did it myself, but what easy-to-make typo would do that? I can understand an easy typo affecting one directory, e.g., you want rm abc.* but accidentally put a space between the dot and asterisk. But I can’t quite see an unnoticed typo sufficient to affect every directory and subdirectory above the pwd. Too many extra chars required to go unnoticed. But I’d actually feel much better if I stupidly did it myself, rather than a dreamhost mistake or external hack, either of which would represent ongoing and hard-to-calculate risks. So I’ll tentatively accept the “shot myself in the foot” theory unless some additional info suggests otherwise. Thanks, guys.

No luck guys – I’ve been doing a lot of typing since then, chmod’ing stuff back the way it’s supposed to be, and Wednesday’s history is long gone. Should have thought to look then, or at least saved a copy, but it hadn’t occurred to me I might be to blame.

And an interesting side note that just maybe I’m not to blame… I have two public cgi’s that let people put math expressions in html using img tags, e.g., to see one of them, click http://www.forkosh.com/mathtex.cgi?\frac1{\sqrt{a^2+b^2}} New images are cached, and those two cache directories have 100,000 and 300,000 images, respectively, in them. All cached images created prior to 2012-02-15 23:00 were chmod’ed 755 at that time, and all created afterwards are 644 like they’re supposed to be. But trying to chmod them all back to 644, just gives “Argument list too long” when bash tries to expand the asterisk. So, arguably, I couldn’t have done it myself from the shell. And I’d definitely have noticed if anything I typed tried to do it. So it’s still somewhat of a mystery (at least to me).

interesting. not definitive, but interesting. they could all be chmoded with the find command, or xargs, without giving an error. you’d have to really mess up to do that by accident though. i have a hard time doing it on purpose on the first go!

I think the new/old files might be a red herring though. i’m a not completely sure, but I don’t think chmod changes the umask, so while all files may have had their perms changed, that doesn’t mean new ones won’t have those some perms if the umask is set. I think that’s how it works…

Anyway, one peculiar thing is the time. The odds of mistyping and chmoding everything exactly on the hour is slim. Seems like something a cronjob would do… You don’t have a cronjob set that only runs once a year on Feb 15 @ 2300 do you? Or some long enough time that you would have forgotten the last time it ran?

Yes – somebody else’s (with root privileges) runaway cron job; that’s my new best (and only) guess. Very astute of you to interpret the time that way (I’d noticed it but hadn’t made anything of it). I have no cron jobs of my own, and occasionally do use find to actually find files, but never to -exec anything. So, like you said, I’d have had to “really mess up” to accidentally do anything with the observed effect, which is why I didn’t feel your earlier chmod -R conjecture was very likely. Does dreamhost have any reason to go around chmod’ing bunches of files, like maybe under logs/, and maybe messed up somehow? Do they routinely run any kind of jobs at 11:00pm PST?

Yes – somebody else’s (with root privileges) runaway cron job; that’s my new best (and only) guess. Very astute of you to interpret the time that way (I’d noticed it but hadn’t made anything of it). I have no cron jobs of my own, and occasionally do use find to actually find files, but never to -exec anything. So, like you said, I’d have had to “really mess up” to accidentally do anything with the observed effect, which is why I didn’t feel your earlier chmod -R conjecture was very likely. Does dreamhost have any reason to go around chmod’ing bunches of files, like maybe under logs/, and maybe messed up somehow? Do they routinely run any kind of jobs at 11:00pm PST?

You were logged in at that time, though, right? So we can’t rule that out, especially if the IP address at that time belongs to you.

I don’t think DH’s cronjobs work on the user’s dir because all the logs are actually links to files in another dir. I guess they could do something like that in an emergency if an attack left a bunch of files as 777, but I would hope that they would do that conditionally, not touching files that had not been attacked. Again, though, it’s just speculation.

Going with Occam’s razor, if you were logged in at the time of the change, I think the likelihood is that somehow it’s connected to something you or one of your apps did. Given the time, it could have been an automated process. For example, I know wordpress sets up its own cronjobs. If you have something like that running in the same user, a rouge plugin could have chmoded everything for that user.

Yes, I was logged in at that time, so have to agree (especially since nobody else has replied to report a similar experience) the circumstantial evidence best suggests me. I want my lawyer. And it almost certainly looks like me logged on – I have verizon dsl and, from last, whois 98.109.229.53 is verizon’s (and my modem’s currently assigned 98.109.234.92).

But, no, don’t have wordpress or any third-party plugins or any automated processes. I do have various of my own cgi’s that are documented online and intentionally publicly accessible, but they know nothing about chmod-like stuff. And umask, from your earlier reply, doesn’t seem to be any kind of issue, either, as far as I can tell. And assuming no typo within reason would have that effect, I just see no conceivable way it happened. Hence the quandary: must be me, can’t be me. I’m stumped.[hr]

LakeRat:

On the surface the permissions changed to 755 sounds like one of the things that’s been happening lately in the rash of hacked sites.

Most suspicious thing I could find was a core file from one of my cgi’s (but timestamped 2012-02-16 14:28, apparently after the incident). From it I gathered REMOTE_ADDR=94.142.134.155, apparently someplace in Latvia. So I added a deny from 94.142.134. but have no idea what, if any, good that might do.

Yes, I was logged in at that time, so have to agree (especially since nobody else has replied to report a similar experience) the circumstantial evidence best suggests me. I want my lawyer.

It’s an interesting puzzle though. Obviously you are the type of person who knows what he’s doing, so the odds of this sort of massive mistake are slim. The odds of DH making a mistake… well, let’s not think about that too much but in general I’d say the odds are low.

forkosh:

I do have various of my own cgi’s that are documented online and intentionally publicly accessible, but they know nothing about chmod-like stuff.
[…]
Hence the quandary: must be me, can’t be me. I’m stumped.[hr]
Most suspicious thing I could find was a core file from one of my cgi’s (but timestamped 2012-02-16 14:28, apparently after the incident). From it I gathered REMOTE_ADDR=94.142.134.155, apparently someplace in Latvia. So I added a deny from 94.142.134. but have no idea what, if any, good that might do.

I assume you’ve checked the logs for that time period? Was there any unusual activity? One possible solution may be that one of your scripts has a heretofore unknown bug which has somehow been exploited? Your diff shows there are no new files or changes, so perhaps the exploit is only minimally useful? Of course, chmoding to 777 would have been more useful if it was due to an exploit…

a little awk magic might help. it should be around the chmod time, and I would guess it’s going to be a crafted set of parameters in the url if it happened that way. You could get i list of unique request for the hour before and after:

That proved much more useful than my messing around. Actually, I was surprised to find .2012-02-15 since I only keep default 3 days, but it was there so I copied it for safekeeping. Your awk emitted 11,338 lines for that time range. I grep -v’ed it for mathtex and mimetex (the two math cgi’s), which boiled it down to … 54 lines.

What really stands out is that I had a long-forgotten phpBB2 bulletin board for people with questions about those math cgi’s, but disabled it (rm’d the posting and registration scripts) several years ago, after spammers starting relentlessly registering and posting garbage (somehow they got past the gif image with mangled letters they have to enter before registering). But I left the bb there for reading so I could point people sending me email questions to already-written answers.

And included among those remaining 54 lines are
/phpBB2/posting.php?mode=newtopic&f=2&sid=959d73b7b03903c6030f7cdaa13450e8
/viewforum.php?f=2&sid=959d73b7b03903c6030f7cdaa13450e8
/posting.php?mode=newtopic&f=2&sid=959d73b7b03903c6030f7cdaa13450e8
/phpBB2/viewforum.php?f=2

posting shouldn’t even be there. Anyway, I tarballed the entire directory tree and rm r-'ed it.
I also grepped the original log for that sid=, finding six strings of the form

That 404 suggests to me he got nowhere, but I don’t know enough to be completely sure about much of anything regarding this kind of stuff. I added 1.161.141. (apparently some asia-pacific company, Chunghwa Telecom) to .htaccess for good measure. Can’t think of anything else to do about it, assuming it was even anything malicious in the first place.

Well, there’s suspicious activity in a php-based app that may be outdated (you didn’t mention that). Given that the time is close, I would suspect maybe there was an exploit around that time. I would look very carefully at the logs around 23:00 and see what you can dig up. And at minimum, update that app and lock it down!

It’s been an interesting search… good to hone the skills a bit

ps: 4m hits per week? 300k+ inodes. On a shared server!? Your site would never work on most other shared hosts I’ve looked at. DH’s generous resource limits, especially for highly efficient sites as yours appears to be, is what keeps me here.

Yeah, I’ve rm -r’ed the entire app. I’d forgotten all about it, despite that its home directory must have scrolled past my eyes many times. And some other guy (some russian ip)…
95.182.40.147 - - [15/Feb/2012:23:00:07 -0800] “POST /phpBB3/includes/index.php HTTP/1.1” 200 4040 “http://forkosh.com/phpBB2/includes/index.php” “Opera/9.80 (Windows NT 6.1; U; ru) Presto/2.10.229 Version/11.61”
…was messing with it at exactly 23:00. The 200 for index.php was expected and presumably benign. But, in any case, the app’s gone for good, now. And so’s 95.182.

DH’s limits are indeed generous. I moved the site from my new york city isp, panix.com, whose limits are way insufficient. (I still forward email there, though, since their shell email and procmail filtering provide much better control.) By the way, for the record, note that my cgi’s are totally free, and the source is all gpl’ed – I don’t make any money off anything I have on the web (except, hopefully, my resume:).

Thanks for the heads-up about those errors. Fixed now. Somehow the cache directory itself had been chmod’ed 644. I wonder if that was a side effect of when I tried chmod 644 * from >>inside<< the directory, and bash complained that it couldn’t expand the *. Maybe there’s some weird bug with that side effect when it fails like that.

I was curious why a file named index.php would be requested from a directory named “includes” as the convention is that files that are included would not be requested by a web browser. In the latest version of phpBB there is no index.php in that directory - a blank index.htm is found instead.

If I were you I would check the /phpBB3/includes/index.php file for backdoor code that might have been added to it. The fact that the status code is 200 simply means the web server had no problems with the request; it says nothing about whether or not such request was malicious in nature and it should not be presumed to be benign without considering why this file would be requested and what code would be executed. Any files being requested that should not be are a red flag.

[quote=“Atropos7, post:24, topic:57034”]I was curious why a file named index.php …[/quote] The entire phpBB directory, and all subdirectories, have been rm -r’ed, and I even deleted the associated mysql databases from the “panel”. I think that particular problem’s probably all gone. I hadn’t been using that application in years, anyway.

(earlier, to bobocat) Thanks for the heads-up …

I forgot to separately thank you for all the help tracking down this … whatever it is/was/will-be. Assuming, of course, it has been tracked down; who knows?. And assuming it was a hack by that 95.182.40.147 guy, is there a list anywhere of ip addresses that dreamhost users have already found annoying? I added deny from 95.182. to .htaccess after the fact, but would have preemptively done that had I had a n earlier clue.

And assuming it was a hack by that 95.182.40.147 guy, is there a list anywhere of ip addresses that dreamhost users have already found annoying? I added deny from 95.182. to .htaccess after the fact, but would have preemptively done that had I had a n earlier clue.

I think the general consensus is that IP-based blocking is basically worthless unless you are blocking a known spam provider or suchlike. Hackers will generally hide behind proxies and it’s trivial to switch to a new IP address. You could block all IPs beginning with 95.182, but assuming that block has been assigned to an ISP, you’d be blocking 65,535 other, likely legitimate, users.

Perhaps a beginner’s question, but what should Wordpress folders and files be? I was told 755 was safe, but I’ve been hacked, so clearly that’s not the case.

I believe you can set it to 700 and it should still work. 755 is generally standard and unless you have some evidence to show that your dirs were set to 755 and were breached because of those settings, I’d be hesitant to accept that 755 is not safe. It’s most likely that the code was injected via insecure programming in some theme or plugin or app, then that new code was used to do nasty stuff to your account. The dir settings likely had nothing to do with it.