Bob Ippolito on Erlang and MochiWeb

Recorded at:

Bio Bob Ippolito is CTO and cofounder of Mochi Media, Inc. Bob started using Erlang in 2006 to prototype an ad server, and it quickly became the de facto platform for building servers at Mochi. He is also an experienced Python user and frequent open source contributor.

Sponsored Content

The Erlang Factory is an event that focuses on Erlang - the computer language that was designed to support distributed, fault-tolerant, soft-realtime applications with requirements for high availability and high concurrency. The main part of the Factory is the conference - a two-day collection of focused subject tracks with an enormous opportunity to meet the best minds in Erlang and network with experts in all its uses and applications.

I’m Bob Ippolito, CTO and co-founder of Mochi Media. Mochi Media is a platform for game developers and also game publishers where essentially we provide a whole bunch of services and focus on the server-side and scale so that they don’t have to. They build a game and we take care of the rest.

That’s somewhat of a difficult question because there are many components of Mochi Media. The first component that we built was an analytics component for any Flash developer and then we built an advertising platform for game developers and with the advertising platform we built several other supporting platforms, such as a distribution platform that helped get their games out to other publishers on the internet. We built a deeper analytics platform specific to games, we built a high-score platform, we built a virtual currency platform, a social network integration platform and we’ve just been building these platforms to help game developers build new things on top of technology that would have been difficult for them to build themselves. We also have some other supporting businesses: we run a game portal - Mochi Games - and we’re working also as a game publisher to get games produced.

The analytics are up to the game developer. We provide just the infrastructure to track all the data and do reporting on the data, but the specifics of what we report on that’s really up to what the game developer is tracking. Some of the basics are: how many times has your game been played, what countries is it being played in, which websites is it being played on and how long are users spending playing that game. Other metrics are very much game-specific and it’s up to the game developer to implement this.

Virtual currency is a very popular way for game developers to monetize their games because it allows them to directly interact with the consumers, where advertising is very much indirect. Because it’s a technology that interacts with millions of consumers and game developers are primarily client-side developers, we built the platform that takes care of it for them. We do the payment processing from the consumers and we do the inventory management for the games. Many of the games using the platform literally have no server-side of their own. They are simply using our APIs and we’re taking care of all of the purchasing, fraud management - everything is taken care of by us.

The first thing that we built for game developers was the ad work. It’s a very simple way for them to simply copy a couple of lines of code into their game and we can show advertisement dynamically targeted directly to that consumer for the game.

MochiWeb is one of our open-source projects. Early on, when I was building the ad server because we were very small back then, I was trying to find a platform that was efficient because in the past we had built all of our software in Python. Python is a great language, but it doesn’t have any real concurrency support built into the language, it can’t even take advantage of multiple cores in a useful way without having separate processes. I benchmarked several other languages both for how easy it was for me to use it and also how well it performed in networking. Early on I tried out Erlang and the Yaws web server and they benchmarked really well and it was easy for me to learn. At that point I decided that all of our networking servers, if they had to deal with high concurrency would be written in Erlang.

But after some time, I found the Yaws web server to be rather complicated and the final straw was that at the time there was a memory leak in the Yaws web server, so I decided to write our own in-house web server called MochiWeb, because we’re Mochi Media and we have open-source projects that start with Mochi. So I decided to write this web server and my goals were to have something that was very easy to understand and also very simple because Yaws does many things like WebDAV and who know what and I just wanted something that was barebones that just did HTTP because all of our applications are very simple APIs. We don’t need all of the fancy templates or any of that and I didn’t want to have to audit all that code in Yaws.

I think one of the things that Python is best at is the typical web development. We use a lot of Django and Pylons and other web frameworks and they are a lot more mature in Python than they are in some of the more niche languages, like Erlang. Specifically, we use it for the very traditional relational database, HTML templates - the things that everybody else uses languages like Python and Ruby for. We also use a lot of Python in our data processing pipeline. So every time an ad is served or a game is played, we collect all this data and the data becomes very large files on disk and we use Python to process all that data and put it into our data warehouse.

We use Erlang for all of the high-performance API servers basically. Erlang doesn’t actually speak HTML to anything, it speaks low-level JSON or maybe it will send back one by one pixels. We even use it to generate SWF files on the fly - the Flash binary format. Erlang is really an ideal language for this because it supports binaries so well. You can build binaries a bit at a time, if you need to and unfortunately the SWF file format requires you to do that. Writing that code was just so much easier in Erlang than it would have been in other language.

Mochi Media was acquired in January of 2010 by Shanda Games and that’s been quite an interesting experience. First of all it’s the first company that I started and sold, so that was a huge learning experience for me, but also the most interesting part is that it’s a Chinese company. Now Mochi Media is kind of split between San Francisco and Shanghai. The cool part there is we have basically people working around the clock because the time zones are so different and it’s been a real learning experience to not only build a good team in San Francisco but also to help build a team in Shanghai and have them interoperate. We’ve made a ton of progress on that since we were acquired. I even spent seven months there myself. In addition to simply just starting another office out there, we’re also expanding all of our offerings and products to the Chinese market specifically, which is really cool because there is a ton of people in China and it’s a completely separate market from the US.

It’s actually quite interesting because it’s had a lot of adoption and I was really surprised by that. Some of the people that I’ve heard using MochiWeb for one I think the biggest installation is probably Facebook. Apparently they are using MochiWeb in their Facebook chat application. There are many other users of MochiWeb either directly, such as (I believe) CouchDB uses it as its REST interface, but also indirectly. There are several frameworks that provide higher level web server features on top of MochiWeb, such as the WebMachine framework by Basho. That framework is what they use for the Riak database for their REST service. I think MochiWeb might be used for all of the databases written in Erlang at the moment.

Yes. And it really surprised me that MochiWeb had such adoption because I think it’s one of my least documented open-source projects, but there was just such a market at the time for a very simple web server for Erlang, unlike the built in inets server or the Yaws server.