September WMF Engineering Update

The Wikimedia Foundation Engineering staff has grown quite a bit over the past year, which has made it a lot harder for everyone to keep track of what we’re all working on. In an effort to make things a little clearer, we plan to report monthly on all of our active efforts, and maintain information pages on all of our active projects. Note that this isn’t (yet) a complete list of everything that the Wikimedia Foundation engineering team is up to, but we plan to make this increasingly comprehensive and more organized as we get better at putting together these reports. Here is a full list of projects.

You’ll see that each of these areas has a program manager assigned to the area. That’s the person who is responsible for coordinating the activity in that area, and someone from whom you can expect to get more detailed updates. More below the fold…

Status: We currently use Solaris/ZFS as the file system for media storage. Due to the rollout of our media-related projects (see “Multimedia tools” below) which have the potential to increase the load on our media storage infrastructure, we’re currently evaluating whether we are going to stay on ZFS, as well as what sort of infrastructure we need to implement in concert with whatever file system implementation we choose. This project will try to look at the whole strategy to design / implement a solution that will scale sufficiently for the next couple of years of projected growth.

Program manager: Danese

Monitoring – Enhancing both ops and public monitoring to a) notice potential outages sooner, b) increase transparency to the community, c) support progress tracking required in the 5-year plan.

Status: We use Nagios for systems/load monitoring, but we haven’t taken the time to tune its alert throwing to be really useful to us. We need to increase tooling to better monitor performance metrics such as page load time in target markets (such as India).

Program manager: Danese

Content Quality Tools

Status: We’re in the beginning phase of this project, figuring out requirements and generally determining the scope of our near-term and long-term efforts in this area. We are currently working on a pilot rating system which will be available as part of the Public Policy pilot program in late September.

Program manager: Alolita

Pending changes enwiki trial Pending Changes is a new review feature recently deployed to en.wikipedia.org, which allows changes made by anonymous and new users be reviewed before they appear as the primary version of an article.

Status: The official trial period has ended, with a straw poll now underway. Nimish Gautam and Devin Finzer put together some helpful statistics that we hope helps everyone how the feature performs on a per-article basis. Howie has provided some additional analysis to help interpret the numbers. Chad Horohoe has done some work on diagnosing and fixing some lingering performance issues with the feature as his schedule allows. Aaron Schulz is helping out when and where he can as he tackles other obligations, generally advising the rest of us on many aspects of the system.

Add media wizard – The Add-media wizard is a gadget to facilitate the insertion of media files into wiki pages. Its development is supported by Kaltura.

Status: This tool was originally released as a gadget on a test server. It’s currently being adapted to run as an officially supported extension and be better integrated into MediaWiki. This effort will be assisted by the deployment of the Resource Loader (see below).

Program manager: Alolita

MediaWiki Infrastructure

Resource loader – The resource loader aims to improve the load times for JavaScript and CSS components on any wiki page.

Status: Trevor and Roan are busy implementing this feature, with hopeful completion sometime in the next month or so.

Program manager: Alolita

Central Notice – CentralNotice is a banner system used for global messaging across Wikimedia projects.

Status: We’re revamping the CentralNotice extension to make it easier to add, manage, and test new banners and campaigns. We’re also looking into including new functionality like geo-location and tightly coupling in analytics to improve our decision making. We’re not only looking to make this tool more usable for fundraising but also simple and broad enough to benefit the Wikimedia community as a whole.
Ryan Kaldari recently finished up our first phase of making input simpler. The interface has gotten a huge face lift and is quickly approaching the discussed mockups at http://meta.wikimedia.org/wiki/CentralNotice_upgrades . We’ve tried to focus on not overwhelming our users and have chosen to collapse, hide and/or remove certain components so that banner input is simpler.
In our second phase, we’ll be looking for volunteers to help test the new geo-location functionality. We’re also working on a better testing infrastructure for our CentralNotice banners.

Program manager: Tomasz

Analytics Revamp – Incorporate an analytics solution that can grow and answer the questions that the Wikimedia movement has.

Status: We are evaluating several possible analytics frameworks such as Open Web Analytics as a supplement or even replacement for our homegrown system(s), based on therecommendations from the Strategy task force. We plan to make a decision soon about the system we will use for (at least) this year’s fundraiser, but with an eye toward deploying a more generally useful system.

Program managers: RobLa & Tomasz

Software Quality Infrastructure

Status: Markus Glaser wrote the original set of tests using the Selenium web application testing framework. Ryan Lane has set up a cluster of machines dedicated to Selenium testing. Priyanka Dhanda and Ryan have been working to refine the requirements and generally get us to the point where we can build out a large suite of automated tests for MediaWiki, and Mark Hershberger is starting to figure out how to drive automated runs of Selenium and PHPUnit tests using CruiseControl. A small group has started to meet regularly to plan a more coordinated push in this area. Ping any one of us on IRC or the mailing list if you’re interested in chipping in!

Status: This has turned out to be a very successful year for us. Though not all projects were finished completely as specified, all were completed to a sufficient degree that we felt very comfortable passing all of the students. While there’s no guarantee that everything here will get beyond the proof-of-concept stage (though at least a couple already are), there’s a lot of promising work to look forward to.

Status: We’re currently figuring out the general practices that don’t involve new tools (such as this blog post, and the wiki pages), as well as figuring out what tools will help us work best together with each other and the larger community. We’re also working to figure out what ways our existing tools (such as Bugzilla) can be configured to make it clear the order that we plan to tackle tasks clear and obvious to anyone who wants to find out.

Program manager: RobLa

If you read this far, thanks for sticking with us! We hope you found this useful. Please let us know what we can do to make this more useful for you.

Hurray, says the guy responsible for news reporting on such topics for the Signpost. Would it be too much to ask to have less but more often? (Or more and more often :P ) A step in the right direction, nevertheless.

Recent Posts

The Wikimedia Foundation, Inc. Is a nonprofit charitable organization dedicated to encouraging the growth, development and distribution of free, multilingual content, and to providing the full content of these wiki-based projects to the public free of charge.

The Wikimedia projects have an international scope, and the Wikimedia movement has already made a significant impact throughout the world. To continue this success on an organizational level, Wikimedia is building an international network of associated organizations.