Entity Systems are the Future of MMOs Part 5

It’s been a long time since my last post on this topic. Last year, I stopped working for a big MMO publisher, and since then I’ve been having fun doing MMO Consultancy (helping other teams write their games), and iPhone development (learning how to design and write great iPhone apps).

Previously, I posed some questions and said I’d answer them later:

how do you define the archetypes for your entities?

how do you instantiate multiple new entities from a single archetype?

how do you STORE in-memory entities so that they can be re-instantiated later on?

Let’s answer those first.

A quick warning…

I’m going to write this post using Relational terminology. This is deliberate, for several reasons:

It’s the most-correct practical way of describing runtime Entities

It’s fairly trivial to see how to implement this using static and dynamic arrays – the reverse is not so obvious

If you’re working on MMO’s, you should be using SQL for your persistence / back-end – which means you should already be thinking in Relations.

…and a quick introduction to Relational

If you know literally nothing about Relational data, RDBMS’s, and/or SQL (which is true of most game programmers, sadly), then here’s the idiot’s guide:

Everything is stored either in arrays, or in 2-dimensional arrays (“arrays-of-arrays”)

The index into the array is explicitly given a name, some text ending in “_id”; but it’s still just an array-index: an increasing list of integers starting with 0, 1, 2, 3 … etc

Since you can’t have Dictionaries / HashMaps, you have to use 3 arrays-of-arrays to simulate one Dictionary. This is very very typical, and so obvious you should be able to understand it easily when you see it below. I only do it twice in this whole blog post.

Where I say “table, with N columns”, I mean “a variable-length array, with each element containing another array: a fixed-size array of N items”

Where I say “row”, I mean “one of the fixed-size arrays of N items”

Rather than index the fixed-size arrays by integer from 0…N, we give a unique name (“column name”) to each index. It makes writing code much much clearer. Since the arrays are fixed-size, and we know all these column names before we write the program, this is no problem.

Beyond that … well, go google “SQL Tutorial” – most of them are just 1 page long, and take no more than 5 minutes to read through.

How do you store all your data? Part 2: Runtime Entities + Components (“Objects”)

We’re doing part 2 first, because it’s the bit most of us think of first. When I go onto part 1 later, you’ll see why it’s “theoretically” the first part (and I called it “1”), even though when you write your game, you’ll probably write it second.

Table 3: all components

(yes, I’m starting at 3. You’ll see why later ;))

Table 3: components

component_id

official name

human-readable description

table-name

There are N additional tables, one for each row in the Components table. Each row has a unique value of “table-name”, telling you which table to look at for this component. This is optional: you could instead use an algorithmic name based on some criteria like the official_name, or the component_id – but if you ever change the name of a component, or delete one and re-use the id, you’ll get problems.

Table 4: all entities / entity names

Table 4 : entities

entity_id

human-readable label FOR DEBUGGING ONLY

(really,you should only have 1 column in this table – but the second column is really useful when debuggin your own ES implementation itself!)

…which combines with:…

Table 5: entity/component mapping

Table 5 : entity_components

entity_id

component_id

component_data_id

…to tell you which components are in which entity.

Technically, you could decide not to bother with Table 4; just look up the “unique values of entity_id from table 5″ whenever you want to deal with Table 4. But there are performance advantages for it – and you get to avoid some multi-threading issues (e.g. when creating a new entity, just create a blank entity in the entity table first, and that fast atomic action “reserves” an entity_id; without Table 4, you have to create ALL the components inside a Synchronized block of code, which is not good practice for MT code).

Tables 6,7,8…N+5: data for each component for each Entity

Table N+5 : component_data_table_N

component_data_id

[1..M columns, one column for each piece of data in your component]

These N tables store all the live, runtime, data for all the entity/component pairs.

…and this is absolutely fine, so long as you remember ONE important thing: the above code is NOT inside a method “because you wanted it in an OOP class”. It’s inside a method “because you didn’t want to type it out every time you have a place in your code where you instantiate tanks”.

i.e. IT IS NOT OOP CODE! (the use of “methods” or “functions” is an idea that predates OOP by decades – it is coincidence that OOP *also* uses methods).

Or, in other words, if you do the above:

NEVER put the above code into a Class on its own; especially NEVER NEVER split the above code into multiple methods, and use OOP inheritance to nest the calls to “createComponet” etc.

But … it means that when you decide to split one Component into 2 Components, you’ll have to go through the source code for EVERY kind of game-object in your game, and change the source, then re-compile.

A neater way to handle this is to extend the ES to not only define “the components in each entity” but also “templates for creating new entities of a given human-readable type”. I previously referred to these templates as “assemblages” to avoid using the confusing term “template” which means many things already in OOP programming…

Table 1: all Assemblages

Table 2: assemblage/component mapping

Table 2 : assemblage_components

assemblage_id

component_id

This table is cut-down version of Table 5 (entity/component mapping). This table provides the “template” for instantiating a new Entity: you pick an assemblage_id, find out all the component_id’s that exist for it, and then create a new Entity and instantiate one of each of those components and add it to the entity.

Table 3: all components

Table 3: components

component_id

official name

human-readable description

table-name

This is the same table from earlier (hence the silly numbering, just to make sure you noticed ;)) – it MUST be the same data, for obvious reasons.

Things to note

DataForEntity( (entity-id) ) – fast lookup

If you know the entity-id, you may only need one table lookup to get the data for an entire component (Table5 is highly cacheable – it’s small, doesn’t change, and has fixed-size rows).

Splitting Table 5 for performance or parallelization

When your SQL DB is too slow and you want to split to multiple DB servers, OR you’re not using SQL (doing it all in RAM) and want to fit inside your CPU cache, then you’ll split table 5 usually into N sub-tables, where N = number of unique component_id’s.

Why?

Because you run one System at a time, and each System needs all the components with the same component_id – but none of the components without that id.

Isolation

The entire data for any given system is fully isolated into its own table. It’s easy to print to screen (for debugging), serialize (for saving / bug reports), parallelize (different components on different physical DB servers)

Metadata for editing your Assemblages and Entities

a.k.a. “Programmer/Designers: take note…”

It can be tempting to add extra columns to the Entity and Assemblage tables. Really, you shouldn’t be doing this. If you feel tempted to do that, add the extra data as more COMPONENTS – even if the data is NOTHING to do with your game (e.g. “name_of_designer_who_wrote_this_assemblage”).

Here’s a great feature of Entity Systems: it is (literally) trivial for the game to “remove” un-needed information at startup. If, for instance, you have vast amounts of metadata on each entity (e.g. “name of author”, “time of creation”, “what I had for lunch on the day when I wrote this entity”) – then it can all be included and AUTOMATICALLY be stripped-out at runtime. It can even be included in the application, but “not loaded” at startup – so you get the benefits of keeping all the debug data hanging around, with no performance overhead.

You can’t do that with OOP: you can get some *similar* benefits by doing C-Header-File Voodoo, and writing lots of proprietary code … but … so much is dependent upon your header files that unless you really know what you’re doing you probably shouldn’t go there.

Another great example is Subversion / Git / CVS / etc metadata: you can attach to each Entity the full Subversion metadata for that Entity, by creating a “SubversionInformation” System / Component. Then at runtime, if something crashes, load up the SubversionInformation system, and include it in the crash log. Of course, the Components for the SubversionInformation system aren’t actually loader yet – because the system wasn’t used inside the main game. No problem – now you’ve started the system (in your crash-handler code), it’ll pull in its own data from disk, attach it to whatever entities are in-memory, and all works beautifully.

Wrapping up…

I wanted to cover other things – like transmitting all this stuff over the network (and maybe cover how to do so both fast and efficiently) – but I realise now that this post is going to be long enough as it is.

Next time, I hope to talk about that (binary serialization / loading), and editors (how do you make it easy to edit / design your own game?).

Did this post help you?

You can say “thank you” by giving me your email address, and letting me contact you next time I make a game of my own:

Post navigation

154 thoughts on “Entity Systems are the Future of MMOs Part 5”

Hello I’m just beginning to write an entity system and I have a couple areas that’re really bugging me as far as the design goes.

The game I’m creating this for is a minecraft-style game. In fact for the purposes of the discussion you can just picture an es-based minecraft.

If you are at all familiar with Minecraft-style games, wherein there is an infinite procedural world, you’ve got to be familiar with the concept of “chunks.” So what I’m wondering/having trouble with is where chunks, and more basically-speaking game loading, come into play as far as the systems go. Of course all of the systems will have to be instantiated by some sort of manager. Do I do the chunk loading/unloading there? This, to me, implies a third letter in the ES (M-Manager) acronym, which doesn’t seem right. Also, later in dev it will be necessary to load multiple “worlds”(/scenes) at the same time. My first thought is to handle this by and allowing their chunks to be simulated as-needed based on if certain player-owned entities are within them, using a chunkComponent/chunkSystem structure. The chunkSystem would ensure that the live memory entity/component database contains only the entities within a certain distance of any chunkComponents (determined by client hardware capability)… But this seems really slow

Any advice from anybody here would be appreciated but I sincerely hope it’s not too late (and that my post is not too poorly worded) to hear from you on this, Adam.

I am creating a RTS game and want to use entity-system, but there are some questions confuse me.

Question 1: How to implement space-partition with entitiy-system.

In “component-based OOP”: component is responsible for processing logic and using message passing to coupling module.

For instance, “SpacePartition” is used to improve query performance in AI.
Entity have a PositionComponent which could SetPosition and SendSyncMessage.
SpacePartitionManager subscribe PositionMoveMessage on MessageBus and process entity’s move in SpacePartition.

Since using the message passing, SpacePartitionManager don’t care who changed position.
(NB: entity’s position may be operated by physics, navigation, or other user customized behaviors which like “teleport”)

In “entitiy-system”, we could not use message passing in component (OOP concept!), component are DATA only!

So we must make SpacePartitionManager become a component (SpacePartitionComponent, it has only one entity in world) which include all entity’s space partition info,
and call SpacePartitionComponent.SpacePartition.Move in all “moving-systems”. (which like: PhysicsSystem, FlyNavigationSystem, CrawlNavigationSystem, VehicleNavigationSystem, TeleportSystem, WanderSystem, etc)
That’s very inconvenient, because it means all “moving-systems” will depend on SpacePartitionComponent, but in concept “moving-systems” shouldn’t depend on SpacePartitionComponent. (SpacePartitionComponent only a optimization)

Question 2: Can I use OOP to implement system in ES?

case 1: Can I use message passing(sync callback or async queue) in the systems? (Not entity and component!)
For instance, when DamageSystem check a unit die, send a message to global message bus and other system (eg: a trigger system or any other system) will handle it.

case 2: Can I use inheritance in system?
For instance, I have many similar systems which maybe like FlyNavigationSystem, CrawlNavigationSystem, VehicleNavigationSystem, etc.
They have similar protected intefaces and common codes, do I have the necessary to create a common NavigationSystem which can be derived by XXNavigationSystem?

Question 3: How to avoid Query? (especially in OOP language)

I have read some ES source code which like artemis/ash, they resolve this problem by using aspect/node. (very great idea!)
But if your system not only iterate self but also check other entity component?
For instance, Usually a AI Unit need check other entities in radius (maybe use space partition optimize).
Therefore, for each UnitAIComponent iterated, we need get all other entities in raidus and query components on it.
But other entities maybe have a wide variety of components (IdleComponent, WalkComponent, AttackComponent, ItemInventoryComponent).
We only query compoenent…. (eg: EntitiyManager.EntitiyHasCompoenent(entityIDInRadius))

In your posts, entity is a GUID, component type has a GUID.
What about component instance id? Should we handle it?

Question 5: Can I use ScriptComponent?
In your posts, Component are only data not code! But if think of code as a set of string and state context in database?
For instance, ScriptSystem parse and execute ScriptComponent bytecode. Data in ScriptComponent will be changed by ScriptSystem. Disallow other system modify any data in ScriptComponent.
The advantage of ScriptComponent is
(1) You could write OOP code in component
(2) we can modify logic in database (ScriptComponent field) on the fly?

Any advice would be very appreciated.
Thank you for your kind reading my post while you are very busy.