I've been developing a first person shooter/massive multiplayer online roleplaying game for my small business, and was wondering if it would be feasible to use peer-to-peer technology to communicate between players, without the use of an intermediary server (as our company does not have enough funds for a high-speed connection to run a game server on). How would such Peer-to-peer technology be best implemented in such a game? Here's a few of the options I've been considering:
a) Divide the game into segments, and have each segment be played on a separate P2P "mesh" (currently named XRevolution)
b) Have one mesh for all portions of the game, and place all of the players on a single grid
c) Some other P2P solution
I've already used option A (considering only one area of the game is done) on a small scale (4 computers), but am wondering how well that would scale when thousands of computers would potentially be bound to the same mesh.
Here's a list of possible concerns with P2P technology

During penetration testing, it was identified that a client could spoof packets to lie about the state of the game to other peers, allowing people to cheat. This is identified as "medium" risk, as it could be prevented to a certain extent through various security routines that would be integrated within the game (such as encryption, movement prediction, double-checking state with other peers, AI, etc.)

Lag was originally a problem on slower networks, but now movement prediction and other techniques are used to reduce the effect of lag, and disconnect peers whose lag is too high. However, a problem which remains is the amount of time it would take on a larger-scale network to find peers (other players), and establish connections. Due to the amount of time it would take to connect to so many peers, solution A would seem optimal to solve this particular problem

A problem with solution A, would be in-game chat between meshes (as no two peers in different meshes can communicate with each other, except for the purpose of relaying), however, this problem could be potentially solved by using something in-between solutions A and B (such as creating a separate mesh for inter-mesh communication, which would be specific to each player or something)

What's the best way to do something like this? Solution A, or B, something in-between the two, or an entirely different solution altogether. Due to our budget, using a server to facilitate communication between players is completely out of the question, so it would be nice to use a P2P solution, otherwise the project would have to be abandoned.

Be careful regarding the cheating potential. It's not just packet spoofing; usually in peer-to-peer games, all peers have information about all other peers (or at least, nearby peers), such as their location. This means information that should not be available to the player is sitting locally on their machine - and is thus always obtainable with enough hacking. If you attempt to store no information locally, you hit back on the problem you've already mentioned, of actively manipulating the network messages - only now it's harder to detect / avoid. Also not storing it locally will lead to lag.
–
OakMay 28 '11 at 20:37

Which is why, as of now, it's still a medium security concern for our application, as it still presents a risk; however, it does not allow for privilege escalation, remote code execution, and is not likely to expose personal information about the player, and is therefore not regarded as "high" concern.
–
IDWMasterMay 28 '11 at 20:41

Also; by design, each peer will need to cache the information locally, in order to reduce lag as you've mentioned, and this tactic is already implemented in the first multiplayer level in the game (each level contains its own executable code, and the game engine itself is only responsible for rendering the graphics, collision checking, audio/video playback, and a few physics calculations)
–
IDWMasterMay 28 '11 at 20:43

1

We have a different definition of high concern :) I agree that RCE isn't a risk, all I'm saying is to be careful. Cheating can absolutely ruin an otherwise great online game, unfortunately, and P2P is highly susceptible (though I still believe the future is in P2P!)
–
OakMay 28 '11 at 20:45

The question is not what the problems are, but what the best ways around the problem are.
–
IDWMasterMay 28 '11 at 20:49

2 Answers
2

One idea, borrowed from distributed communication models, is to use a gossip-like approach. Gossip protocols are mostly concerned with scalability and reliability. The nice thing, however, is that they all use a very low bandwidth, which is crucial for P2P systems. The cost is latency, but read on, it might be solvable.

The main idea of gossip protocols is that each client sends a message to a small number of peers - let's say log(n) - and whenever a client receives a message it sends it against to log(n) peers. This creates redundancy (a client can receive a message more than once) which is good for reliability but bad for you - but on the other hand, this can lead to very fast broadcasts.

The question is, how to choose those log(n) clients in a way that will

Guarantee all clients will eventually get the message in X time.

Minimize the travel time of messages between two clients that are close to each other in the game world.

Point (2) above is - in my opinion - the key here. As long as not all peers can affect each other immediately in the game world, having high latency between them is less of an issue. A FPS in a small arena is a poor candidate here, but a massively-multiplayer online game is a great candidate.

So, how do you choose which peers will be connected to which? What you want is to build a small world network (also called small world graph). You can find some algorithms used to create it (see Harari Graph), as well as algorithms that will allow you to change it on-the-fly according to user data (I recall something involving darknet which can change links according to location). I wouldn't just use a plain Harari Graph - instead, you're probably better off with a variation that takes super nodes into consideration, I'm just not familiar with any.

I think using these kinds of protocols is a better approach than the less-scalable (and more discrete - what happens in edge boundaries?) local mesh concept.

Great idea! Implemented this in the engine just now and it works great, even on the load simulations!
–
IDWMasterMay 29 '11 at 1:51

@IDWMaster cool, and if you don't mind me asking - how did you end up structuring the graph?
–
OakMay 29 '11 at 7:01

I used the same proprietary technology which we used for developing our distributed filesystem. We're primarily a networking company, just getting into gaming, so we've already worked on things like this. I already have a patent on a protocol which does this, bypasses firewalls, and creates at least 2 backups of each file on our system as soon as it is updated. I'll explain more about how this works in the next comment, as I'm running out of room for this one.
–
IDWMasterMay 29 '11 at 13:15

How it works is each router in the network (which is guaranteed to be reachable from any peer) stores data about any peer which connects to it. The peer will then send a query to the router, asking it for information about other peers that tried to connect to it, and also if it is bound to any of the same services as it. If it is bound to one or more of the same services, it establishes an XStream to the router. Then it connects to any peers it deems necessary (which it found by looking at information from the router) in order to find more information about the state of the system.
–
IDWMasterMay 29 '11 at 13:19

Both peers end up caching information about each other, and provide it to other peers when necessary. We use patented algorithms to perform this procedure, as well as the protocol which is "patent pending" right now. There's obviously a huge number of optimizations, and cached data is deleted once it is detected that it's no longer needed, and/or the peer is running low on resources needed to play the game at a high speed. There's also various "access controls" on the system which identify which peer "owns" the data, and information like this can be used to determine which peer can get it
–
IDWMasterMay 29 '11 at 13:22

One of the major problems about peer-to-peer is connectivity. I mean, really- a lot of customers just won't be able to connect to each other. Symmetric NAT, firewalls, and all this kind of thing is something that your customer will not want to deal with, the average customer just can't. Lag is also going to be an incredible problem as the number of players speeds up, and you're going to significantly increase your minimum hardware requirements as every player is running a server, effectively.

I would say, fundamentally, there's a reason that most FPS games are hosted on servers, and it's client-server works far better. If you can't afford to host your own servers, then get the users to host them- but don't go for BitTorrent-style mass peer-to-peer. That just won't work.

Develop as a client/server architecture, allow players to host both dedicated and non-dedicated, is the best way you can go. Most FPS games don't have significant server quantities provided by the developer themselves- it's nearly all third party servers. If you can't afford to host even a few servers for yourself, then it's time to seriously re-consider if you have genuinely considered the budgetary requirements for developing a first-person shooter.

Edit:

Ok, so hang on, let me get this straight. You can't afford to host a server for yourself, but you still want to have centralized server worlds, a'la an MMORPG. User hardware and user software configurations are not up to that kind of strain. I think that what you want is fundamentally impossible.

The P2P technology that we're using does force players with the capability and enough bandwidth to act as "routers" for computers that cannot connect directly P2P. the protocol is implemented in such a way that it is always possible for two computers to connect to each other, as long as they are able to both reach one of the routers.
–
IDWMasterMay 28 '11 at 20:44

Keep in mind, you can't have the "lobby" style server/client relationship in a MMORPG as you can in an FPS, and this game is a combination of the two concepts, so the "lobby"/"dedicated server" idea is out of the question.
–
IDWMasterMay 28 '11 at 20:45

2

@IDWMaster: At the cost of how much ping? How much system resources on the part of the "router"? How many players are going to be stuck going through the same router? This kind of idea just isn't appreciating the scale of the problem that you'll face.
–
DeadMGMay 28 '11 at 20:46

We've already ran simulations on the effect of several computers connecting through the same router, and have developed methods and limitations to prevent this from causing problems, which are implemented in the protocol that we are using. Only a certain number of clients will be able to connect to the same router, and routers will be randomly selected (to provide an even distribution) using a pseudo-random number generator.
–
IDWMasterMay 28 '11 at 20:49

1

@IDWMaster: So you're just going to hope that there's enough routers, placed in the right places, with enough power?
–
DeadMGMay 28 '11 at 20:49