Database errors on most Wikipedias

At 10:57 UTC, the master database server for s3 (the cluster that holds most of our wikis) had a full disk and stopped writing. For this reason it was no longer possible to edit these wikis. The larger wikis live on separate clusters and were not affected. After switching to another master database, all wikis are back up and editable as of 12:02 UTC. A few edits that were made during the incident may have been lost.

At 10:57 UTC, the master database server for s3 (the cluster that holds most of our wikis) had a full disk and stopped writing. For this reason it was no longer possible to edit these wikis. The larger wikis live on separate clusters and were not affected.

After switching to another master database, all wikis are back up and editable as of 12:02 UTC. A few edits that were made during the incident may have been lost.

catrope
There seems to be the misconception out there that a wiki being on s3 somehow means it has a lower “priority” or that s3 somehow is structurally less stable than the other DB clusters. None of these things are true. If I wrote anything suggesting otherwise, I apologize for not being clearer.

It’s ok, Roan, no biggie – just when you posted yesterday in the -tech channel and on here, iirc you made a reference to small wikis, which gave me the impression that the ones I mentioned (meta, en.wikinews) were considered “small” in relation to others and therefore less important –

I administer a wikipedia which is in *my* eyes, small, tpi.wikipedia and my comparison was thinking that you were labelling things like en.wn small in comparison to others on s3.

I think you also need to consider migrating the English Wikinews off there too. I notice en.ikipedia and en.wikiquote aren’t on there, nor is commons. And what could possibly be up with having meta.wikimedia on s3? I think there’s a priority review needed there.

There seems to be the misconception out there that a wiki being on s3 somehow means it has a lower “priority” or that s3 somehow is structurally less stable than the other DB clusters. None of these things are true. If I wrote anything suggesting otherwise, I apologize for not being clearer.

About 97% of all wikis are on s3. Only wikis that are very large (where “large” is strictly in terms of the amount of data in its database and the volume of queries sent to it) are on other clusters (s1, s2, s4-s6) which contain one or a few large wikis.

The way wikis are distributed over these 6 clusters also doesn’t indicate any kind of priority. That would suggest that some wikis are deliberately made more stable than others, which is not true.

Moreover, the s3 cluster is not structurally less stable than the other clusters, it just happened to be the cluster whose master’s disk filled up yesterday. It could’ve happened to any other cluster, although in that case only a few (large) wikis would’ve been affected.

I think you also need to consider migrating the English Wikinews off there too. I notice en.ikipedia and en.wikiquote aren’t on there, nor is commons. And what could possibly be up with having meta.wikimedia on s3? I think there’s a priority review needed there.

Archives

Work at Wikimedia

Work with the foundation that supports Wikipedia and its sister projects around the world.
Apply and join us

Wikimedia Foundation

The Wikimedia Foundation, Inc is a nonprofit charitable organization dedicated to encouraging the growth, development and distribution of free, multilingual content, and to providing the full content of these wiki-based projects to the public free of charge. Get Involved | Log In

Wikimedia Projects

The Wikimedia Foundation operates some of the largest collaboratively edited reference projects in the world.

Wikimedia Movement Affiliates

The Wikimedia projects have an international scope, and the Wikimedia movement has already made a significant impact throughout the world. To continue this success on an organizational level, Wikimedia is building an international network of associated organizations.