Inventing as we go: building a visual editor for MediaWiki

We at the Wikimedia Foundation, in conjunction with Wikia, are building the VisualEditor for Wikipedia and all other sites based on the MediaWiki software. This is taking us some time, and we often get asked what’s involved in this, just how hard it is, and why it’s taking us so long, so we thought we’d explain some of the intricacies in the most challenging software project the Wikimedia Foundation has ever worked on.

What are we trying to do?

We are creating software that will let users load, edit and save Wikipedia articles visually, bypassing the existing system that requires our users learn “wikitext,” a complex markup code. Instead, the articles they’re editing will look the same as when they’re reading them, and any changes they make will be obvious in their effects before they press save — just like writing a document in a word processor.

Haven’t people done this before?

Yes and no. There are lots of visual text editors out there, and even a few open source ones that can edit Web pages quite well, but they are insufficient for our needs for a number of reasons.

One criterion that is hugely important to us is that our editor should work with lots of languages. This is not just a matter of supporting certain right-to-left languages, or a few based on ideograms, but being able to use any and all of the 290 languages we currently provide. We also want to be able to do so seamlessly in the same documents to support our multi-lingual communities. Some of the languages we are committed to working with have very little software support, and we are often one of the few sources of written material for them, or at least, one of the largest.

Another issue is that wikitext has grown over the past 12 years to have a large number of rich and complicated features that are not just “simplified ways of writing HTML.” Though we originally intended many of these to be used only occasionally, they have often evolved to be at the core of how MediaWiki pages are now written by many Wikimedia users and more widely.

These advanced features include content transclusion (pulling in a “live” copy of one page or part of it into another), templates (transclusion with parts defined at the second page rather than the source), and parser functions (pages that show different things depending on hundreds of potential options, like the day of the week or whether another page exists). Attempting to retrofit this into an existing editor would have been exceptionally difficult, and more work than starting from scratch.

How VisualEditor and Parsoid work together (click to enlarge)

What is the parser?

Finally, we need to not just edit pages, but to read and save existing wiki pages in the old wikitext format, and continue to work with it in parallel to the new editor. We can’t throw away the huge amount of work our communities have done over the past 12 years, so we need to re-write the “translator.” This means making a two-way “parser” — a bit of software that converts wikitext into HTML and back again. Until now, we have only had a one-way parser; the second stage, converting back from what people want to write to wikitext, has had to be done by our editors in their heads.

This means that we’ve had to have an entirely separate project – the Parsoid – that can translate in both directions: from wikitext to Web pages and also back again. This is not remotely an easy task; you have to be very attentive in replacing a parser, as it’s hugely important that we don’t break anything. The old parser, and the wikitext “language” it interprets, just grew organically as people had ideas, and no one really designed it. There are nearly two billion versions of pages in Wikimedia’s wikis alone, and this lack of design means that there are a huge number of little rules we need Parsoid to follow to avoid “dirty-diffs” — issues where the wikitext would be broken by people using the VisualEditor.

Explaining the HTML+RDFa content model used by Parsoid (click to enlarge)

We use automated testing to “round-trip” 100,000 randomly chosen pages from the English Wikipedia (as a reasonable sample of wiki content in the Latin alphabet): taking the wikitext, converting it to HTML format and back to wikitext, and comparing the result. This helps us by identifying many issues to fix so that using Parsoid does not cause articles to break. Right now we get about 80 percent of these articles to round-trip without any differences at all (up from 65 percent in October), an additional 18 percent round-trip with only minor differences (whitespace, quote style etc.), and the remaining 2 percent of pages have differences that still need fixing (down from about 15 percent in October).

Learn more

As you can see, to make a visual editor that our users need is a large amount of work. No existing technologies could do what we wanted to, so we have needed to work very hard to make sure that we deliver this. We look forward over the next few months to showing off how editors will be able to use VisualEditor and Parsoid for a much better experience, free of learning any complicated codes, letting them focus on the content and not the tool they use to write it.

If you’re interested, we have a brief presentation about how Parsoid and VisualEditor work, and what they will look like on Wikipedia.

I feel somewhat responsible for putting James Forrester on the hot seat concerning this article as I was the one who asked him to take out time from his busy schedule to explain some of the challenges faced by the team concerning the project and manage expectations somewhat concerning the release.

I want to thank you for your questions and concerns. They are very important, and realize if I’m not answering them satisfactory, it’s mostly-likely due to a failing on my part, not as some slight to you. If, on the other extreme, my answers seem too overwrought, please realize that I want to avoid being too glib by answering all your questions with “all your answers can be found on a wiki…somewhere.” ;-)

According to the time table First English Wikipedia deployments and iteration should start now, by July 2013 Visual Editor should be enabled by default for (almost) every wikipedia/mediawiki instance. Any updates?

The planned release for the VisualEditor on English Wikipedia will be this Tuesday, December 11th, in line with the 2012-13 goals document. The purpose of the article was to note the challenges — some foreseen, others not — that mitigate the scope of the release in order to hit this deadline. In particular the project will be:

opt-in only. This means you must be logged-in, and turn the visual editor on to edit visually.

be missing many important features: categories, templates, tables, references

have some small rendering errors when editing (those areas shouldn’t be available)

take longer to load a page for editing, or, in some cases, not allow editing at all (the new parser, because it lacks a cache and is not yet optimized, is slower than the current one)

does not support IE or Opera.

As for the July 2013 deadline. As deadlines are further out it is harder to be optimistic about this. Given that goal was written in May of this year, it is difficult to predict with certainty at the moment. Some of this will be a function of the parsoid fixing rendering bugs and porting to C++ for speed (both faster language, and the ability to integrate directly with mediawiki instead of through the API). Most of it will be related to challenges in the visual editor:

overcoming browser weaknesses (for instance, IE does not currently allow editable areas to be

closed out from editing within an area that is editable which is the reason it is currently supported)

how many bugs are found in the current visual editor
the time it takes to extend support for complex content (references, templates, images, etc)

As with all things engineering, especially in an area as untested as the VisualEditor and the Parsoid, it is hard to make accurate estimations. My magic-8 ball told me to “concentrate and ask again” (in a few months). ;-)

The 2011/2012 targets were, “Develop Visual Editor. First opt-in user-facing production usage by December 2011, and first small wiki default deployment by June 2012.”

In 2011-12, the targets were developed before the challenges James mentioned were even realized. This was before James and I were working at the Foundation, but it’s actually because the 2012-13 goals were developed with some understanding of these challenges that it looks like the team been able to hit a majority of deliverables so far. They’ve done a great job!

You say that this is a project undertaken in conjunction with Wikia. What part or proportion of the work is done by Wikia? What are the financial arrangements between Wikia and Wikimedia Foundation for that development work? How many staff are assigned to this in Wikia and in WMF?

The part of the work done by the Wikia team is currently the “rendering, selection and input (ve.ce)” area of the “module stack” architecture in James’s article. This consists of two full time engineers, provided to the foundation by Wikia.

There are currently no financial arrangements between Wikia and the Wikimedia Foundation for development work (though sometimes the team works at the respective institutions, so I wouldn’t say that they’re above spotting each other a visual editor t-shirt, snack, or two). Instead there is a commitment by them to provide those engineers for such “core work.” Recently, the VisualEditor has undergone a major refactoring in which all parts of this system is extensible through an API. This means in the near future, it would be possible that Wikia will devote additional engineering resources on areas that are important to them, but not to us — an example might be extending the VE to allow YouTube video embeds. That would be taken up by engineers outside of the core team.

The staff directly assigned to the VisualEditor on our end consists of four full time engineers and one part time engineer on the rest of the visual editor stack — Toolbar and inspectors (ve.ui) and Linear model and transaction systems (ve.dm) — three full time engineers on the parsoid, plus one full time product manager. Note: one of the full time engineering positions has not been filled as of this moment. I believe this sum (7+2 FTEs + 1 PTE + 1 PM) make this easily the largest single project resourced in the Foundation.

Note I have not counted the tremendous support from Operations and Platform that is provided, nor the work volunteered by other members of other teams at the WMF (I believe three engineers within Features alone have helped out at various times), nor the work done by our volunteer development community, including one summer Google Summer of Code student who has made the first forays into the Collaboration Server.

One small thing to note. This project has concentrated most of our best front-end engineering talent at the Foundation, as well as two of the top engineers at Wikia, furthermore the Parsoid in particular takes a unique engineering knowledge not typically found in the web world. From time to time, that talent needs to be made available to various other areas of our respective organizations to pay down technical debt, so occasional hiccups in effort have occurred, and will continue to occur. (As an example of this: one of my engineers once prefaced a request by saying, “I know this messes with the ‘dream team’ you’ve have over there in the Visual Editor, but…”) :-D

I don’t like that Wikipedia has such a close relationship with Wikia.

These concerns are noted. Right now, our goals and Wikia’s align with respect to the VisualEditor so it makes sense to work together. Since all the software produced by the teams are open-sourced and freely available, the licensing nature of the software is designed to keep this relationship from being corrupted by either party, as well as allow that anyone in the world sufficiently interested and motivated to contribute to the project.

As an example, our goals and Kaltura’s and Google’s align with respect to WebM support in the TimedMediaHandler. Should we look the gift-horse in the mouth, simply because they might be commercial or commercially-driven entities? IANAL, but I think the open-source license chosen creates a legally binding contract to allow the acceptance of such a contribution without ad hominem on the contributor.

Last year, around this time, the Wikia team, working on FCKedit integration, started experimenting in contentEditable. As browser support for the contentEditable attribute increased, the work they did progressed further and faster than Foundation work on the it’s own edit surface. It made sense to combine our efforts and the VisualEditor project is better for it.

This division of effort is unlikely to be a problem in the short term. In the longer term differences in our goals and Wikia’s may appear when the likely solution is to be implemented in ui.ce part of the VisualEditor. Under those conditions, the WMF might have to devote resources to fix them on our end. This is not true for InternetExplorer support, but I can imagine this being the case on Opera or some mobile browsers, as the Foundation mandate is likely to be much broader than that of Wikia’s. Alas, not every free (as in speech) software can similarly be developed free (as in beer)!

I would be interested to know how it’ll deal with Wikipedia’s existing microformats.

The engineers could speak directly to this, but I believe since Microformats are already in HTML, these are preserved by the parsoid. Right now, where the microformats are usually used (infoboxes) are not currently editable by the VisualEditor. This will have to be dealt with when the VE is extended to support it.

I was disappointed, again, that there was no discussion of creating a Table namespace, as exists with some other wikis (I saw one demonstrated years ago).

Again, the engineers should speak directly to this. Tables are a challenge because the templates that create them do not generate self-closing nodes (see the parsoid diagram: the tokens are generated into a HTML5DOM). However, the parsoid seems to support tables already even if VisualEditor does not at the moment.

I am not familiar with the Table namespace, but I believe the idea there was to abstract tables for transclusion in its own namespace. If such a thing existed, then it would make the parsoid much easier to have been written, but not affect the challenges already presented to the VisualEditor. Since the parsoid already has overcome this hurdle, I don’t believe it makes a difference at this point.

I was disappointed, again, that there was no discussion of creating a Table namespace, as exists with some other wikis (I saw one demonstrated years ago). That would significantly simplify the overall problem of a WYSIWYG editor, because a specialized table editor could be developed as a separate module.

The 2011/2012 targets were, “Develop Visual Editor. First opt-in user-facing production usage by December 2011, and first small wiki default deployment by June 2012.”

Those dates passed, and the new 2012/2013 targets then were: “Launch a new default environment for Wikipedia projects that does not require markup. Limited English Wikipedia release for real-world editing in December 2012. Deployed to majority of Wikimedia wikis and ready for default usage by July 2013.”

December 2012 is upon us. Would it be correct to assume that the limited English Wikipedia release will now now take place this month? If so, do you have any idea when it will be?

You say that this is a project undertaken in conjunction with Wikia. What part or proportion of the work is done by Wikia? What are the financial arrangements between Wikia and Wikimedia Foundation for that development work? How many staff are assigned to this in Wikia and in WMF?

Recent Posts

The Wikimedia Foundation, Inc. Is a nonprofit charitable organization dedicated to encouraging the growth, development and distribution of free, multilingual content, and to providing the full content of these wiki-based projects to the public free of charge.

The Wikimedia projects have an international scope, and the Wikimedia movement has already made a significant impact throughout the world. To continue this success on an organizational level, Wikimedia is building an international network of associated organizations.