This bug messes up file histories if users are not aware of this bug. E.g. when using the Rotatebot to reupload a rotated version of a image (needed due to the sloppy implementation of the EXIF rotation at MediaWiki). Please do not include too much bugs in MediaWiki - otherwise it gets usuable (even with workarounds like Rotatebot). ;-)

This bug messes up file histories if users are not aware of this bug. E.g. when
using the Rotatebot to reupload a rotated version of a image (needed due to the
sloppy implementation of the EXIF rotation at MediaWiki). Please do not include
too much bugs in MediaWiki - otherwise it gets usuable (even with workarounds
like Rotatebot). ;-)

How prevalent is old thumbnails not updating in your opinion (like very rough estimate - is it 1 out of 100 000 images, or 1 out of every 10 type deal).

This bug messes up file histories if users are not aware of this bug. E.g. when
using the Rotatebot to reupload a rotated version of a image (needed due to the
sloppy implementation of the EXIF rotation at MediaWiki).

(In regards to the comment about the "sloppy" implementation of rotations)
If there are issues, Please report them in bugzilla.

But the issue I believe you are talking about presenting is images being rotated in third party image editing programs (for example: paint.net, gimp, photoshop, etc) and the program subsequently not updating the images metadata ([[Exif]] data) to reflect the rotational change in the file, which is what MediaWiki reads to detect how the image should be displayed.

Although this is a issue, There isn't really anything much more on the MediaWiki end that can be done to combat this I believe.

>Please do not include too much bugs in MediaWiki
lol, we try to avoid that.

Just wanted to warn just in case... ;-)

(In reply to comment #9)

(In regards to the comment about the "sloppy" implementation of rotations)
If there are issues, Please report them in bugzilla.

there: Bug 31504

But the issue I believe you are talking about presenting is images being
rotated in third party image editing programs (for example: paint.net, gimp,
photoshop, etc) and the program subsequently not updating the images metadata
([[Exif]] data) to reflect the rotational change in the file, which is what
MediaWiki reads to detect how the image should be displayed.

(In reply to comment #8)
> How prevalent is old thumbnails not updating in your opinion (like very rough
> estimate - is it 1 out of 100 000 images, or 1 out of every 10 type deal).
Well, you can click throughhttps://commons.wikimedia.org/wiki/User:Rotatebot/Log
30 checked - nonupdating 4 (but just the 800px thumbs):

I just had a conversation with Saibo on irc. Apparently these thumbs are only wrong if you're browsing from europe. Us north american folks can simulate by putting
91.198.174.234 upload.wikimedia.org
in our etc/hosts.

So my first guess is that HTCP purge packets aren't getting to european squids, but it gets weirder. If you mangle the url of the thumb so that it should be a cache miss (aka put ?foo at the end of the thumb url) I still get the wrong thumb. My understanding is that an image squid cache miss should come from some server full of image thumbs, so this should be the same regardless from where you're accessing. (But I could be mis-interpreting something, I'm not super familar with how wmf works its servers in different places of the world set up).

Anyways, here is the HTTP headers of the request that returned the wrong thumb.

Aaron Schulz and I did a fair amount of work trying to repro and investigate this last night. We didn't think to do the /etc/hosts trick, but one of our theories was a Europe-only problem, so that makes sense. We *did* repro a thumbnailing problem, though it was only temporary and easily fixed with ?action=purge on the image description page. I'm not sure we actually reproduced the problem that's causing the most grief, but we did at least prove it's possible (if not as easy) to see the problem on one of the North American squid servers.

One theory we had: maybe it's a race condition, where the thumb is requested after the image is updated, but before the thumb is generated, since the "Age:" http header for the image was clearly younger than the upstream thumbnail should have been.

Mark Bergsma is reporting that the HTCP purge daemon hasn't been running for a while, and will only restart manually, which is the likely cause. He's manually restarted this (so the symptoms should go away...please test), but let's not close this until he's confirmed that the daemon can be restarted automatically.

I just tested on the djvu file listed at bug 32388. Earlier that file had a broken thumb (When viewed from europe) that action=purging on the image desc page wasn't fixing. I just viewed it again, and thumb was still broken, but i purged, and this time the thumb got purged.

Additionally Saibo said on irc he hasn't seen any new instances recently

Therefore I'm calling this fixed.

p.s.
One thing I am rather confused about is how the wrong thumb was being shown on squid cache *misses* even if it was just htcp that was broken. (Perhaps there's just more caching layers that work in complicated ways I'm not familiar with)

[[:File:White Storks Migrating Northwards Over Bental Mountain DSC00695.JPG]] and [[:File:Introduction_to_the_RepRap_Self_Replicating_3D_Printer.ogv]] could be fixed now by a purge (which didn't help before - at least at the video and I am sure someone tried to purge the Storks, too).

[[:File:White Storks Migrating Northwards Over Bental Mountain DSC00695.JPG]]
and [[:File:Introduction_to_the_RepRap_Self_Replicating_3D_Printer.ogv]] could
be fixed now by a purge (which didn't help before - at least at the video and I
am sure someone tried to purge the Storks, too).

Umm, comparing md5sum's of different size thumbnails isn't going to tell you very much. Even if they were both correct thumbnails, the md5sum would be different. (However, I can confirm by visual inspection that the wrong thumb was returned for the 75px size).

This seems to have a different root cause then the other issues reported here, as it doesn't differ between europe/non-europe.

PS: There must be a much bigger problem. All thumbnails are loading very, very slow. Some are not loading at all (timeout). For example, when going to http://commons.wikimedia.org/wiki/Special:NewFiles it takes about a minute to load all thumbnails. A few thumbnails load and are displayed, then all transfers are blocked, then a few more thumbnails load, transfer is stopped again and so on. This does not happen on other web pages, so it's not my network.

I re-uploaded the file [[commons:File:Weisswasser0078.png]] but all previously generated thumbnails show the old image. From what I know this problem exists for at least a year. Isn't it possible to fix it?

PS: I had an idea. Maybe the "invalidate all thumbnails" process is not triggered because of the upload method used or because of some broken JavaScript in one of the upload methods? I'm using the basic upload form.

This is an other bug but I have to explain it a bit. When I click "re-upload this image" an way to complicated and therefor useless upload form shows up, made of a dozen input elements. This doesn't make any sense because most of the information entered there will be lost. The comment will be cropped. Sometimes the basic upload form shows up when I try to re-upload an image but most of the time I have to add "&uploadformstyle=basic" to the URL to force it.

(In reply to comment #8)
> How prevalent is old thumbnails not updating in your opinion (like very rough
> estimate - is it 1 out of 100 000 images, or 1 out of every 10 type deal).
Well, you can click throughhttps://commons.wikimedia.org/wiki/User:Rotatebot/Log
30 checked - nonupdating 4 (but just the 800px thumbs):

I do not upload many files to commons but every time I do it I came across this issue. Today this file (out of four) does not refresh after I uploaded a new version (the old version contains thin white lines between the colors). I tried to add "?action=purge" to both the description page and the thumbnail.

I do not upload many files to commons but every time I do it I came across this
issue. Today this file (out of four) does not refresh after I uploaded a new
version (the old version contains thin white lines between the colors). I tried
to add "?action=purge" to both the description page and the thumbnail.

Yes, because it was finally refreshed and displays the latest version now. Yesterday it showed the old version for hours.

I uploaded several new versions and most (but nor all) of the thumbnails are refreshed immediately. Some showed the old version but I was able to force a refresh by adding "?action=purge" to the thumb URL and the description page. This single thumb always showed the old version no matter what I tried. I consider this a bug. As said, this happens every few uploads. It's not a rare bug.

So just to clarify, the bug is not that it never gets purged, the bug is that
in some cases it takes several hours to get purged?

Seems so. Does not make a big difference (I know, it may be an important difference for the developers). This bug is wasting our time for many, many months now. It makes people re-upload files again and again. Nobody knows why a thumb is not refreshed and why it is "magically" refreshed the next day.

Also this is no new information. I don't see anybody complaining about a thumb that is still wrong after weeks. It seems in all cases it is refreshed after a while. No idea how long it took in average. I had better things to do.

Almost every time I re-upload an image I get old garbage for some thumbnail sizes. At least this is how it feels. I think I posted dozens of examples for over a year now. See bug #28613. Purge never does anything. Some hours or days later the bad thumbnails are "magically" fixed.

The "magic" fixing might just be the files getting rotated out of squid/varnish via LRU or expiry (cache time depends on last-modified date). I doubt the actual purge actions were just delayed for that long.

Either this is systemic (certain file purge actions always give bogus purge urls to cache) or there is some intermittent network problem.

Either this is systemic (certain file purge actions always give bogus purge
urls to cache) or there is some intermittent network problem.

If it was an intermittent network problem then after purging several times one of them would succeed. Maybe an squid ignores all purges (under certain load conditions?) and when a thumbnail gets assigned to it, it never gets out? (until expiry)

Or maybe if the network is overloaded all the purges gets lost in some hour frame...

is showing old garbage for a total of 8 days now! Time to call Guinness? Did somebody turned the "purge" feature off finally? Is the foundation out of money and stopped the thumbnail servers? Are the "Wiki Love Monuments" mass uploads to blame?

The "magic" fixing might just be the files getting rotated out of squid/varnish
via LRU or expiry (cache time depends on last-modified date). I doubt the
actual purge actions were just delayed for that long.

Either this is systemic (certain file purge actions always give bogus purge
urls to cache) or there is some intermittent network problem.

Aaron: Any idea how to investigate further in order to track this down?

(In reply to comment #40)
> Either this is systemic (certain file purge actions always give bogus purge
> urls to cache) or there is some intermittent network problem.

If it was an intermittent network problem then after purging several times one
of them would succeed. Maybe an squid ignores all purges (under certain load
conditions?) and when a thumbnail gets assigned to it, it never gets out?
(until expiry)

Or maybe if the network is overloaded all the purges gets lost in some hour
frame...

An intermittent problem could become permanent. See bug 41130.

(In reply to comment #45)

Aaron: Any idea how to investigate further in order to track this down?

Add Comment

Text is available under the Creative Commons Attribution-ShareAlike 3.0 License (CC-BY-SA); code is available under the GNU General Public License (GPL) or other appropriate open source licenses. By using this site, you agree to the Terms of Use and Privacy Policy. · Wikimedia Foundation · Privacy Policy · Terms of Use · Disclaimer · CC-BY-SA · GPL