Note per IRC conversation -- will also need to do a generic cleanup pass on non-Commons wikis that have local audio or video uploads. This shouldn't eat too much human time, just needs to be set up and looped.

There's some spare CPU still, so adding in the 360p.ogv and 480p.ogv re-run jobs, seeing how it goes... (still throttled; currently the high-prio queue is about 5 deep and the low-prio queue is at ~9000, down from ~9800 yesterday)

The 160p/240p.ogv 'missing' jobs are getting closer to done; going ahead and adding the 360p/480p.ogv 'all' jobs back in to the mix.

The queue runner count update (https://gerrit.wikimedia.org/r/337230) didn't get pushed out today due to the puppet SWAT being canceled due to an unrelated flurry of errors in production; will try to poke folks to push it out later.

Queue runner count has now been pushed up, so we should get through the rest of the files faster. Might still bump it slightly farther up, I'm seeing usage around 80%, though we're trying to avoid overcommitting CPU either.

It's hard to measure progress exactly without counts in the reqeueueTranscodes.php reporting (I should add that!) but looking at the totals and the progress through the sort order (oggs by name, then webms by name due to the way the index on media type is set up) I think it's about 2/3 through the 160p.ogv missing fill-ins. 240p.ogv and the 160/240p webms are continuing.

Re-running of 360p and 480p ogvs has been churning along, but I'm going to pause those jobs again and see if the low-prio queue falls faster. The throttle on requeueTranscodes.php keeps the queue from growing, but doesn't do much to allow it to shrink...

See grafana graph of the queues -- the response times are ggrreeaatt on the high-prio queue but it's been holding steady at 3-days-ish delay on the low-prio queue the last few days:

Fill-in jobs for 160p and 240p ogv and webm appear to have completed a while ago. Will re-start the 360p and 480p re-runs after adding one more option to avoid accidentally re-running new transcodes unnecessarily.