Multisite photo upload no longer working

Description

Immediately after upgrading from 3.3.1 to 3.3.2, photo/media uploads stopped working. When a photo is uploaded, the request to the server completes and a row is inserted in the database (Gallery shows the photo) but the file is not written to the /wp-content/blogs.dir/[blognum]/files/ directory structure. This happens with both plupload and the browser uploader. Doesn't matter which web browser is used.

Change History (11)

This sounds like a pure coincidence to me, Did you skip to a new month as well?
Ie. Previously it was storing it in 2012/04/ but now it's storing it in 2012/05/ Does the directory actually exist, can PHP create that folder? check the permissions on that folder too.

I'm pretty sure this isn't a bug, but will leave it open for you to confirm the above.

I can confirm that the directory exists. So WordPress is creating the directory but not moving the file. You mention permissions, and I'm seeing that the new directories being created by WordPress are owned by 'root', not the web server user.

On a semi-related note, this may be a different manifestation of another bug plaguing us. When a new blog is created, a directory with the blog ID is supposed to be created in /wp-content/blogs.dir/. What actually gets created in that directory is a symlink like this:

I have to manually delete the self-referencing (broken) symlink, correctly create a directory and a subdirectory called 'files', set the correct permissions, and then everything functions normally after that.

I should mention that this problem is intermittent (about half of newly created blogs encounter the symlink issue). And we've tried the same setup on more than one server with different OSes and thoroughly tested the hardware, so we've ruled out hardware as the problem.

I know our plugins inside and out - most are custom-built, the rest have been vetted by myself. None of them have code that creates symlinks either.

So I seem to have fixed the issue by changing the permissions of the directories back to the web server user but that doesn't mean the underlying problem has been fixed. However, I don't see how the directory got created as 'root' in the first place by WordPress.

Some more information: I was just informed that the problem was only happening for some blogs, not all. Which is consistent with what we've been seeing all along - intermittent problems - random WSODs, random self-referencing symlinks during blog creation, and now the random inability to create directories as the web server owner.

and I'm seeing that the new directories being created by WordPress are owned by 'root', not the web server user.

Unless your web server is running as root (bad idea) & PHP is running as root (even worse idea!) then it's not possible for PHP to change the ownership to root, without a system-level permission exploit.. which I can assure you WordPress doesn't utilise :)

This sort of problem is common when server migrations are done and permissions are not reset properly, or if someone manually uploads files.

What actually gets created in that directory is a symlink like this:

WordPress itself uses mkdir(), You could use symlink() or link() and end up with a result like that, but once again, WordPress doesn't do that.

I don't think any of the problems you are seeing are caused by WordPress, or can be fixed by WordPress.
It seems that either
a) You have a plugin doing nasty stuff on the server/install
b) You have a local script on the server running as root to manage something and it's getting in the way - it might be a cronjob or backup process even
c) You're having bad interactions between PHP and the OS/Filesystem or the Filesystem is having troubles - If it's a remote filesystem, it could also be network issues. Metadata might be being corrupted/written wrong causing new directories to be stored as symlinks, or ownership details changed..

I'd seriously do some stand-alone testing with a php file (ie. mkdir(), file_put_contents() to write files, etc) and see if you can reproduce any similar problems that way, and/or from the command line, because you've got some problem there.. and it 'aint WordPress.

I'm closing the ticket as invalid, as there's nothing here that suggests it could potentially be a core WordPress bug, You can still reply without re-opening it, but I can't say you'll get much support here past what I've already said. You might like to try the WordPress support forums: ​http://wordpress.org/support/ or potentially a hosting forum such as "Web Hosting Talk" or similar (for example) as I know plenty of people who manage their own server drop by there.

a) All plugins are monitored closely. I can assure you that the plugins are clean. I track changes like a hawk and vet all plugins.

b) Hmm. Interesting. We do have a number of cron scripts running every few minutes. Without them, the site won't function though. Backups are performed regularly via rsync.

c) The file system is local.

The command-line scripts we use, including cron, have no problems - all operations complete successfully except for when we have occasional database table corruption. There's a lot of self-healing code in place to recover from numerous failure conditions, which makes it even more difficult to track down issues.