The need for experienced multiplayer programmers in South Africa

In some genres adding multiplayer to a game can double, or possibly even quadruple sales. Some games cannot exist AT ALL without multiplayer (like DOTA or Slither.io or Agar.io), and many games would be severely diminished without multiplayer (like Hearthstone or Magicka or Rocket League).

There's a couple reasons for this:

* Some players only play certain types of games with their friends, and some players play games almost exclusively with friends and barely play games by themselves. Often the audience for the same game without multiplayer is much smaller than the audience for the same game with multiplayer.* An invitation to play a game with a friend is a much stronger recommendation than one to play the game by yourself.* There's a much greater incentive for players to recommend that their friends play a game when it means they can play together.* When your friend is playing a game there is a window of opportunity to play with them, and it means you have to buy the game soon, rather than planning on playing it later and maybe forgetting about it.

But programming a game to allow for multiplayer, especially if it is realtime and has a lot of interaction, adds a LOT of extra work, and a lot of that work is debugging (which not everyone enjoys).

On a game like Broforce, it was enough work to keep one programmer occupied nearly full time. But on a team of 10 people, that one extra person is an 11% increase in costs for possibly double the sales.

In a case where multiplayer strongly benefits the game, and the game is likely to cover its costs, even in the case where the team is just 2 people, adding a third person to do multiplayer programming makes a lot of business sense (a 50% increase in costs for double the sales).

But here is the problem.

Multiplayer programming is DIFFICULT. And there's not a lot of programmers in South Africa that I'm aware of that have done much more than getting a couple actors to sync their positions over a network. While syncing gameplay in Broforce or Genital Jousting over a Network is in principle a similar task using the same skills, syncing these games is an order of magnitude more complex than syncing the positions of actors in a tutorial FPS.

And I'm not sure there's many people in South Africa who even want to do this kind of work. Obviously it'd be way better to work with someone who is enjoying themselves doing the work rather than working with someone who is networking the game begrudgingly. In fact, if no one enjoys the task, it might be better to not add multiplayer despite the business logic, because a demoralized team can do more damage than doubling the sales can make back.

If this is a task that programmers enjoy, what I'm not seeing are the programmers who have work like this in their portfolio.

In the coming year, probably 2018, Free Lives is probably going to want to hire someone who can help us with networking games. But I don't know how to evaluate someone's skills if all they've done is the bare minimum. I've followed a Unity tutorial myself and made functionally networked FPS with a few game logic additions, but I wouldn't hire myself for this.

I have a couple questions then:

* Are people who enjoy network programming out there?* Or is this a task that maybe should be shared between programmers, like every programmer is responsible for networking their own code.* If someone says they can do network programming, how should we evaluate them, in the case where they have nothing in their portfolio near the complexity of the projects we need them to collaborate on?* How do we know if someone enjoys network programming if their portfolio doesn't demonstrate this?* Should we try hold network coding jams or something? Maybe something like the SA Game Jam but for networked games? Or are there other ways of encouraging this (baring in mind that this skillset is useful to a lot of potential projects).

Comments

I myself do enjoy working on multiplayer but I have found that doing Survival game multiplayer is too difficult as the Unity documentation doesn't really explain some rare scenarios. The single biggest problem I have found with Unity is the authority on who can do what. If that was fixed I think multiplayer would be a dream to work on.

Developing a deathmatch game is quite easy but the moment you bring it items such as a chest that 3 guys need access to it becomes a mission with assigning someone authority over that object without letting the "main" character that was spawned do it. (Confusing explanation)

Or is this a task that maybe should be shared between programmers, like every programmer is responsible for networking their own code.

Probably easier to have one guy really know in dept multiplayer stuff and work on the framework and just have the other programmers implement the basic stuff and use already written scripts for sync etc

If someone says they can do network programming, how should we evaluate them, in the case where they have nothing in their portfolio near the complexity of the projects we need them to collaborate on?

Most likely review his code that he did on small network games and try understand his thought process. Most important part of programming I've seen is how easy a person can learn.

How do we know if someone enjoys network programming if their portfolio doesn't demonstrate this?

Very difficult question because most people I've worked with love the idea of multiplayer games but when it comes to the coding of it, then they lose all hope because starting out with multiplayer (like UNET in Unity) is quite a difficult task in the beginning but I think if someone was able to get past the stage where he was struggling to implement multiplayer and actually have some small sample of games that he got where people run around shooting each other then it says a lot about how willing he was to get it to work and might demonstrate that he enjoyed doing it, just maybe needs more experience to truly enjoy it.

Should we try hold network coding jams or something? Maybe something like the SA Game Jam but for networked games? Or are there other ways of encouraging this (baring in mind that this skillset is useful to a lot of potential projects).

Only if it is a month long jam or more where the challenge is to challenge EVERYONE to do it regardless if they have done it before. Almost like forcing people to try something new.

Hope there is some truth to my post as it is my personal experience! Good luck hiring :)

Yeah there are. I believe @Ramperkash quite enjoyed working on the networking in a prototype or two we did ages ago. I'm currently working on a game with a very heavy multiplayer component (it's an MMO), and while the multiplayer side of the project isn't my favourite part, there are others in the team that seem to really enjoy it (they're not in SA tho). So yeah, there are ppl that enjoy this, but it's definitely not as commonplace as a more general game developer, especially in SA.

* Or is this a task that maybe should be shared between programmers, like every programmer is responsible for networking their own code.

I think it's important to have at least one person that is an expert on the subject. They don't have to do all of the netcode, but they should be able to debug the most difficult problems and manage others that might not be as skilled as them. So for games with a larger multiplayer component, it feels like a distinctly senior role because of how it combines so many different aspects.

* If someone says they can do network programming, how should we evaluate them, in the case where they have nothing in their portfolio near the complexity of the projects we need them to collaborate on?* How do we know if someone enjoys network programming if their portfolio doesn't demonstrate this?

This is a difficult question. On one hand, I think a networking expert will probably tend to be more senior and therefore undergo a slightly different hiring process to a more junior dev. On the other hand, you might be looking to skill up someone in this.

When I started at SF they gave me a verbal interview question that required knowledge of networking-related race conditions and some of the common pitfalls; I think that's probably a good way to get a gut-feel for someone's skills, and then just have a probation period where they have to show this.

Requiring some basic demonstration of this in their portfolio is not unreasonable, and listening to them describe that piece of work, with all the issues they ran into, is another good indicator.

A lot of this relies of have someone on the team that is an expert on this, or very close, and wants to make a judgement call on a potential hire's ability. If that's an issue (or you just want more info) you could outsource that part of the hiring process to others in the local industry.

* Should we try hold network coding jams or something? Maybe something like the SA Game Jam but for networked games? Or are there other ways of encouraging this (baring in mind that this skillset is useful to a lot of potential projects).

You could hold a networking jam. You could also just explicitly post a job position for someone that has these skills. Even if you don't get an expert in that, it might help signal that you're looking for that and lead to more potential hires that are keen to level up in that department.

I haven't done any Unity network stuff, but in general yes, I like network programming.

Not totally your question, but my opinion on why more people aren't talking about this (and maybe why you aren't seeing it in portfolios):I'd LOVE to add online multiplayer to VALA, but besides the technical bits, there is a high chance it'd be a waste of effort. And I imagine a lot of people are in the same situation.

You guys have a really big following (and Devolver behind you), so can add online and not have to worry too much (this is an assumption obviously) about whether there would be enough people to make online good.But for my game, and probably others, I have no idea if people are even going to buy the game, so spending months adding online just doesn't make sense.

And that means that if I get to my next game, I still won't have any experience doing online (technical and the selling/running part), so probably wouldn't try then either. Now repeat that cycle for the rest of my career.

Multiplayer games are wonderful due to the social aspects they bring to gameplay and I would love to see more prototype attempts produced locally that implement it.

A game jam using it as theme would be awesome, though I'd be surprised if there was a big turnout, since it is a fairly specialised/advanced thing.

I think locally most of us are still struggling with getting basic gameplay loops and coherent game design elements in place, never-mind multiplayer features.

I did attempt a small prototype awhile back and found it to be surprisingly easy to get very basic network functionality going (UNET) as an inexperienced dev. It was rewarding seeing the game sync across a LAN for the first time but I wouldn't go as far as saying I enjoyed it more than so many other aspects of game dev which, personally, is more gratifying to implement.

I find it interesting that you value a persons enjoyment of performing a task. It is not always considered by employers.

It's interesting thinking about this from a business development perspective the way @EvanGreenwood mentioned. The London based studio I work for is currently adding multiplayer to a game that been in early access for 2+ years. Even in a competitive game market like the UK, finding the kind of talent described (experience on large multiplayer projects) is difficult because you're competing with the AAA industry for talent.

Considering the state of the Networking middle ware out there, I'd almost consider it an engine programming role. Unity's built in stuff is bare bones and does nothing to help you solve hardcore problems like latency masking or authority. Are there people who enjoy low level plumbing behind the scenes engine style programming? Certainly. Does that talent exist in South Africa? Maybe. I dunno. Sounds like the kind of role someone would have to grow into.

Personally I would find a online multiplayer workshop much more useful than a game jam. Maybe have an online jam follow the workshop to put the theories learned into practice. But I agree that the jam would need to be longer than just a weekend.

Since not everyone uses Unity (though a lot of people do), it would be nice to chat to someone who has done it before about the broad concepts that can be applied regardless of engine, and then afterwards focus on engine specific tips and tricks for Unity or Unreal.

I have so many questions to ask, specifically about roll back net code. And then there are the questions I don't even know exists yet. Even if we just open up a forum thread to educate designers, so we can have a more meaningful discussion with a network professionals other than: "I want cool net code in my game, please!".

@Pieter While I think there's a lot of engine specific stuff, the large bulk of the work in networking a game isn't going to be spent in engine specific code, but trying to get the logic working (like the example @Zaphire gave).

For example, when we had Broforce ported to PS4 we handed them all the source code and they swapped out the low level network functions that worked on with Steam's API to network functions that worked with the PS4's API. The logic remained the same. The way messages are sent through Unity to other machines isn't the difficult part. The difficult part is keeping the programs running on different computers with different inputs affecting different agents at differing amounts of lg displaying the same game. And so the really challenging part is going to be unique for every game.

Engine specific experience isn't all that important in my view. In my view it's the experience that allows developers to solve the game specific problems that is really valuable.

There's a lot of tutorials about just getting networking working (in Unity, or other platforms). I'm not sure there's many people in South Africa who could lead a workshop that goes any deeper than those tutorials. Richard Pieterse of Free Lives could (though I don't think we've been making the kind of games where roll back net code is a relative concern).

The way we solved lack of multiplayer experience the first time at Free Lives was do several jams where we built multiplayer games, and eventually Richard grew to be our expert. We started doing this long before we began Broforce (because of the chicken-egg problem @roguecode mentioned).

I'd love to do a jam sometime. Perhaps a longer form jam like a few people suggested. Like I said, I'm not as sure about a useful workshop. The first steps in learning to program networked games feels to me like a solved space, it's the kind of problems that come after the initial setup that I see few people solving. I think expecting people to do a few tutorials on their own isn't unreasonable.