Archive

Hello there! Today I want to introduce you my interview to John Koening. John is a PhD student at University of Minnesota in the fields of distributed and real time simulations. John is also working in the game studio he founded. Currently they are developing the game The Electric Adventures of Watt which has some Erlang in it.

Learning something more about Ymir

Paolo – Hello John and welcome to my blog! Can you please introduce yourself to our readers please?

John – Hi Paolo, thanks for having me. My name is John Koenig and I am a PhD student at the University of Minnesota (UMN) studying distributed, real-time simulation. I am going into the third year of my PhD program preparing for my written and oral defenses. I have been a regular Erlang-user for about 6 years.

Prior to, and inter-mixed with my time at UMN I worked at Cray Inc. Most recently I was contracted under Cray’s Chapel team where I worked on several language improvements in the area of portability. A majority of my time at Cray was spent as part of their Custom Engineering initiative. Together, we engineered unique super-computing platforms and software stacks for various customers.

In 2010, I founded a game studio, Called Shot LLC, with my good friends Gabriel Brockman and William Block. We are currently in the first round of funding for our flagship game title: The Electric Adventures of Watt.

Paolo – This is a common question I ask during my interviews: how did you start using Erlang? What are the features of Erlang that made you learn it?

John – I was first introduced to Erlang while pursuing my undergraduate degree at University of Wisconsin – Eau Claire (UWEC), I think it was around 2006. As part of a Programming Languages course we were tasked with picking a new language and implementing a solution to a sufficiently interesting problem which applied to the language’s domain. At the time, I was big into Plan 9 and distributed software in general so I chose Erlang and implemented a distributed prime number sieve.

Being more of an applied school, UWEC had me spending most of my time programming C and C++ and I remember being really impressed with how Erlang modeled processes and inter-process communication directly in the language. Once I got past Erlang’s syntax learning curve and my newness to functional programming, I found myself able to express distributed solutions very naturally in Erlang. After that, I was hooked. I picked up Joe’s book, Programming Erlang, and started keeping up with the Erlang community online.

I first started using Erlang professionally at Basho in early 2008 when I was brought on as a Reliability Engineer. I had thought, coming out of UWEC, that I knew Erlang fairly well, but I grew considerably during my six months at Basho. Justin and his development team are incredibly talented and being around that level of skill and enthusiasm was highly contagious. I remember that time fondly.

Paolo – Would you like to introduce and describe in a few lines what Ymir is? Where will Ymir be used?

John – Ymir is an open-source (GPL), cross-platform, distributed 3D game engine written in Erlang.

With the number of cores available to gamers on the rise, it is Ymir’s purpose to break games out of the traditional, single-core-dominate game-loop and, in doing so, achieve faster, larger simulations which grow in proportion to the number of available cores.

Paolo – Why did you decide to use Erlang for Ymir? Was there any other candidate language at the beginning of the project?

John – Ymir grew out of a desire to create a multi-player RPG that got away from the traditional client/server model. Myself and a few friends enjoyed online RPGs but didn’t enjoy the MMORPG scene. We were interested the approach of Neverwinter Nights 2, however, which featured smaller worlds developed and hosted by members of the community. Hosting of these worlds could get terribly expensive, as the worlds were hosted on a single server and, as their player base grew, admins of these worlds would be required to either co-locate their server or pay for expensive home internet access with sufficient upload speed. I set out to change this, wanting instead to see a game capable of simulating a world in a more peer-to-peer fashion. Namely, a game engine capable of utilizing the additional computational power and bandwidth present as players login to enjoy the simulated world.

I didn’t consider anything other than Erlang for this task. Along with OTP, Erlang is still the best language for distributed development as it allows me to focus more on the high-level challenges of distributed real-time simulation and less on the gritty details of implementing my own task-queues, inter-process communication, etc. This choice was further cemented when I proved that communication to port drivers, with minimal trickery, was sufficiently fast to support online rendering.

Paolo – Reading your paper I spotted many words often used among Erlang developers: scalable, soft-thread, message passing and minimal amount of synchronization. Would you like to discuss the meaning of each term with respect to Erlang and Ymir?

John – Game engines are traditionally frame-centric. Their primary goal is to compute and render frames as quickly as possible. Two aspects make this approach difficult when scaling over multiple cores: first, computing a frame is recursively dependent on the frames which came before it and, second, traditional spatial data structures used in collision detection require all game entities to be synchronized in order to function.

Ymir takes an object-centric approach and aggregates frames as quickly as possible. Game objects (entities) are represented as Erlang processes (soft-threads) and each entity is responsible for simulating itself locally. Discrete events (e.g. collisions, user-input) are modeled as messages which occur at a specific point in simulation time. As an optimization we allow entities to exist at various points in simulation time and to resolve events in the recent past by application of timewarp. Entities proceed through their local simulations, streaming updates to their physical state to relevant renderers. To enforce fairness, a sense of global time is defined as the minimum of all entity simulation times. This introduces a small amount of global synchronization as entities “vote”on the value of this global time through various shared ETS-based counters.

To break away from traditional spatial data-structures, Ymir applies map/reduce to spatial reasoning in order to achieve scalable collision detection. When simulating forward in time, entities volumetrically hash their physical extents against a fixed cube to various buckets (also soft-threads) and aggregate contacts which result from writing their latest physical states into each selected bucket. The act of mapping to spatial buckets is analogous to selecting nearest-neighbors (broadphase) and the buckets themselves compute points of contact for each pair of entities overlapping within its given volume of simulation space. In short, Ymir serializes only those objects which are sufficiently close together while permitting objects sufficiently separated in space to simulate unimpeded.

Using map/reduce in this fashion allows Ymir to scale-out over many cores very well. Currently, we are able to realize ~11x speedup in overall simulation time on 16 cores and sustained frame rates of ~500 fps. Ymir’s performance is dependent on many factors, however, chief among these is the degrees of freedom between entities. As entities are serialized based on spatial proximity, scenes where all entities exist in persistent contact are currently unable to obtain such lofty speedups. I am currently expanding our methods to better model persistent contact which will help Ymir obtain better speedups in these scenarios. Furthermore, using map/reduce to compute contacts works well locally or over low latency networks but as we scale up to many machines connected with higher latency other approaches will be needed. I am currently investigating network overlays between entities which capitalize on spatial assumptions present in game simulations.

Paolo – Do you have any partial results about Ymir our readers can take a look at? What kind of tests do you do on Ymir?

John – We are actively maintaining performance results on Ymir’s indiedb page, and as time permits I will be documenting Ymir more completely on our development blog.

This video showcases the three testing scenarios we used to gather our preliminary results. All three scenes are rendered offline using Ymir’s built-in support for Mitsuba. Parallel is rather boring to watch, but provides a best case performance. Cylinders features a stack of spheres falling onto an static array of cylinders and is more representative of the types of rigid body interactions one might see in an interactive game. Last, is Bounce in which spheres move randomly within fixed scene boundaries. Bounce is currently being used to measure Ymir’s performance as it relates to scene density.

Paolo – Is there any way our readers can contribute to the development of Ymir? Is there any fund-raising? Can other developers join the project?

John – Glad you asked, yes! We are currently on indiegogo seeking funding for The Electric Adventures of Watt which will be powered by Ymir. In supporting The Electric Adventures of Watt, contributors will be directly helping us mature Ymir into its first public release.

We will be advertising the public repositories for Ymir concurrently with its first official release. In the meantime, if developers are interested in working on Ymir, please, don’t hesitate to get in touch with me: john calledshot.org.

Paolo – You are a PhD student at the University of Minnesota and your studies are mainly focused on parallel and distributed real time simulation. Do you think Erlang could be widely used in these fields?

John – Without a doubt, that is what is what Erlang was designed for. There is even a precedent for using Erlang server-side for games: MuchDifferent and SMASH. I feel that as CPUs continue to have more cores and affordable CPU accelerators (i.e. parallela) become available, game developers will turn to solutions like Ymir to grow their games. Scalable, real-time simulation is not an easy undertaking and savvy developers will be looking for the right tools for the job.

As many Erlang enthusiasts know, there are at times substantial resistance to using languages that are outside of other developer’s comfort zones. This is especially true for academia. I can’t even count the number of times I have had to defend Erlang to my lab mates at UMN. That said, we are not expecting that all developers wishing to use Ymir will embrace Erlang, and we have on our roadmap to develop front-ends for languages most game developers will find familiar: C/C++/Lua.

Paolo – Did you find any help in the Erlang community? Did any Erlang developer give you feedback or support online during your development?

John – Several times I got frustrated with the performance of Mnesia and ETS for Ymir’s collision detection and I turned to the Erlang IRC channel for support and guidance. The Erlang community has been nothing but insightful and supportive every time I have turned to them. Although for the life of me I cannot remember that handles of those who offered help, I owe the Erlang community thanks.

We also owe special thanks to you, Paolo, for featuring Ymir on your blog, Peer Stritzinger for helping us reach out to the Erlang community, and to your readers.

The game my flatmates and I play the most is for sureMonster Hunter Tri for WII. In this game you control an hunter which goes around, kills/captures monsters and so on. In the world of Monster Hunter is quite common to find (as in any other good game) some sort of “gather spot”, which are nothing more than places where you can find herbs to be picked up or rocks to be mined. A couple of days ago, while my friends where playing I started emacs and I started implementing a super simple gather spot.

I decided that my gather spot should respect these rules:

gather spot should be a non registered process

gather spot should be a gen_server process

gather spot can handle messages from players asking for a drop (if you don’t know what a drop is check outthis)

gather spot should have different drops according to its type (e.g. rock/herb)

gather spot should have a random number of drops

gather spot should have different probabilities of drop (a friend of mine lost his mind trying to get “Eternal strife” rust shards)

gather spot should stop when it has nothing more to drop

In this post I will neither tell you how I managed the location of the gather spot in the XY-plane, nor how I configured a supervision tree for them; I want only to show you how simple is to implement something working in Erlang. You may notice that in my code I didn’t register with a name the process, this is due to the fact that I may have a huge number of this kind of process in a single Erlang node. Here follow resource.erl and helper.hrl…I will not give you explanation about these two files since they are quite simple.

As you can see, the code above is fairly simple and respect all the rules I introduced above. You may notice that resource type, drops number and actual drop are random. The important thing here is that the gen_server which represents the resource will exit in normal way when no more drops are present in its state. Let’s see helper.hrl, which will give you some better insight:

Now, some may argue that this code is far from perfect…well I agree. In fact this is just a starting point…a lot of stuff can be improved, therefore why don’t you start from this code and do something on you own? How can you improve this code? Here some ideas:

how would you place in a random way a resource spot in a given x-y or x-y-x point?

Hi there! Today I would like to propose you my interview to Christian Flodihn. Christian is working at DemonWare, which is a famous company in the Erlang world. Christian came to my attention for his document “Next Generation MMO Architecture“.

The interview

Paolo – Hi Christian, would you like to introduce yourself to our readers? Christian – I was born in Sweden 1981, played a lot of games on my Nintendo as a kid. When I was 17 I started to play MUD’s (Multi User Dungeons) and started to program LPC when I was 19. Between 2003 and 2006 I went to University and since 2010, I been living in Ireland working as Software Engineer.

Paolo – Why did you start coding and why did you start using Erlang? Christian – Computers have been my interest since I was 14 and I have been programming since I was 19. I started program for a MUD and since then I have been hooked. My first experience of Erlang happened during University, we got a 2 week crash course. However, I did not start using Erlang myself until started to research the best language to write scalable servers in.

Paolo – You are currently working at DemonWare, which is a subsidiary of Activision Blizzard. Can you give us some information about the company and its specialization? Christian – We do online services, such as match making, leader boards, storage etc for Activision games. Our largest title is Modern Warfare 3. The company uses both C++/Python and Erlang where it is best suited, which aligns very well how I like software to be developed.

Paolo – You are also the CEO and founder of Epic Online Entertainment, the company behind project Abydos Online. Can you tell us something more about your company and the project Abydos? Christian – Actually, it is not a company but a registered business name, however, with the new directions we take this your, a real company will likely be formed but with a different name.

Paolo – Why did you started designing an MMOG? What languages did you try for this task and which ones proved to be more efficient? Christian – I wrote an MMO server prototype in Stackless Python, however I figured out it would be very hard and time consuming to scale, so I started research different languages and found out that Erlang was the best choice by far in relation to scalability and distribution.

Paolo – What are the main features of Abydos? How many concurrent players does the implementation support? Christian – I did some stress tests on a machine with 4 cores and 8 GB ram, the server handles about 10,000 to 16,000 players in the same area (a plane of of 1×1 kilometer). If you got real server hardware, imagine something like 32 cores and 64 GB ram it would probably handle much more, specially if you let 4 or 8 of those run the same area.

Paolo – I often hear people say that MMOG development requires a huge amount of skills and time. Do you agree on this? Christian– In general yes, it will require huge amount of skill. The most time consuming however, is the content, we are somewhat going around that problem by letting some of the players create their own areas, similar to what MUD’s usually do.

Paolo – How does erlang help in this sense?Christian– Erlang helps us by providing mechanism distribution, for example something simple as running its own name server on the OS to help find nodes in a network and the built in RPC/messaging functionality saves us a lot of time.

Paolo – I had the chance to read “Next Generation MMO Architecture”, which you wrote in January 2010 at University of Göteborg. I found this document really interestig. Can you suggest to our readers other books, articles or tutorials to improve knowledge on MMOs? Christian– I am afraid not. I been trying to find some good articles myself, most of them focuses on using peer to peer to “solve” the scalability issue, which gives you just bizarre security issues instead. Another popular subject are algorithms for (AOI) Area of Interest which by nature scales very badly and can be replaced by AOR (Area of Responsibility) in almost all cases.The closest I found to my own article was Michal Slaski’s “A Virtual World Distributed Server developed in Erlang as a Tool for analysing Needs of Massively Multiplayer Online Game Servers”, however it is very short, only 3 pages and only deals with about 1000 players. I guess keeping all their game data in a database could be the thing that kept their numbers low.

Paolo – How to play Abydos? Should people ask somewhere for a beta test key? Christian – We have a tester/newsletter signup button in our menu on the www.abydosonline.com webpage.

Paolo – Is there a way for interested developers or designers to collaborate to Abydos? Christian– Yes, you can just fire of a mail to recr@abydosonline.com. You can also visit our web site and click on recruitment in the menu.