So, I'm working on another game (surprise). The prototype single player version can be found here: http://www.cokeandcode.com/asd. Its essentially a simple dungeon crawl using the SNES RPG style graphics, randomly generated dungeons. The game play is intended to be like nethack/rogue apart from free form movement and not turn based (although at this rate that might change soon).

I'm having some problems coming up with a networking model to support this, I've considered:

1) Position Based

Just try and sync the players positions and state using game updates. Interpolate positions to smooth out missing bits of positional info and speed up movement where actors/entities/things might be lagging behind the expected. This works pretty well for FPS type games where you can't see alot of entities all the time, everyone is moving all the time and you can't judge the positions of things too well. However, for my 2D top down (sort of) game its not working out too well since you see alot of the corrections going on due to the viewpoint.

2) Deterministic Simulation

Having just had a discussion on #lwjgl, I realise I could atttempt to make everything deterministic at all ends. Make the player interaction command based, introducing enough delay between player action and avatar reaction that everyone runs the same simulation on all machines. This seems like it'd work pretty well but I'd end up having to change the style of the game from a keyboard dungeon crawl into a more sort of point and click affair. Not too keen, but this will be last hope

Having just had a discussion on #lwjgl, I realise I could atttempt to make everything deterministic at all ends. Make the player interaction command based, introducing enough delay between player action and avatar reaction that everyone runs the same simulation on all machines. This seems like it'd work pretty well but I'd end up having to change the style of the game from a keyboard dungeon crawl into a more sort of point and click affair. Not too keen, but this will be last hope

I'm toying with this idea myself for my current game. Making a game fully deterministic is usually something of a pain, but with Java handling most (all?) of the fp math issues and other obscure ways code become non-deterministic then i think it wouldn't be too tricky.

Problems with this (compared to client-server):- Not secure (this i can live with)- No late joining (may or may not be important)- One person's bad connection messes with the entire game for everyone else (so if over the internet this limits the number of players to ~4, maybe 8 if everyone's got a good connection).

Just try and sync the players positions and state using game updates. Interpolate positions to smooth out missing bits of positional info and speed up movement where actors/entities/things might be lagging behind the expected. This works pretty well for FPS type games where you can't see alot of entities all the time, everyone is moving all the time and you can't judge the positions of things too well. However, for my 2D top down (sort of) game its not working out too well since you see alot of the corrections going on due to the viewpoint.

What kind of artifacts are you seeing? Is it your speed-up thats noticeable? If so you may have to accept the idea that your going to have to carry a higher error term along and not cheat the speed, or at least not by much.

Quote

2) Deterministic Simulation

Having just had a discussion on #lwjgl, I realise I could atttempt to make everything deterministic at all ends. Make the player interaction command based, introducing enough delay between player action and avatar reaction that everyone runs the same simulation on all machines. This seems like it'd work pretty well but I'd end up having to change the style of the game from a keyboard dungeon crawl into a more sort of point and click affair. Not too keen, but this will be last hope

The problem with countign on dterminism is that you will have to eithr:

(a) Go to at least a semi-lock-step solution where the state is not advanced until player inpu thas been recieved by all players This means you run loop is

OR

(b) Deal with trying to regularly "resynch" the games and ensure that any synch drift cannot have an imapct on game logic (not real easy.)

The issue with lockstep is that your execution loop would look something like this;

(i) Poitn and click(ii) Wait until everyone elses packet for this time step is received(III) Then calculate and update.(iv) rinse and repeeat

It will make any latnecy issues visible as a pause btw input and response

Quote

Any suggestions or points to articles on other possible strategies?

I thin kyour already on the right path, you just need to work a bit on your latency hiding...

Kev

Quote

Got a question about Java and game programming? Just new to the Java Game Development Community? Try my FAQ. Its likely you'll learn something!

I've got the same issue with SharpShooter Arena, although being a FPS, it isn't as noticeable. Currently I'm sending position, velocity, rotation updates at 5Hz. However that means if you take your finger off the forward key just after Tx'ing an update, the remote client doesn't find out you've stopped moving for 200mS and overshoots. I thought of three 'solutions':

i) The local player continues to move until the next update is sent, even though you've let go of the key. This is only really noticeable when you try to stop at a precise location.ii) The local player stops immediately, the remote player jumps to the correct location on the next update. This was noticeable even in my FPS. Players tended to lurch.iii) The local player stops immediately, the remote player has it's velocity corrected to close on the correct location over a short period of time. This helps smooth out the lurches. However on stopping players tend to halt, then take a few steps backwards.

Currently I'm using a combination of i) and iii). I found I still needed iii) due to variable network latancy causing errors resulting in stuttering movement. Sometimes you get that stop and step back effect though. I'm also synchronising the client clocks and introducing 60mS forced lag, which keeps everything very close to in-sync without having to go for a lock-step approach. Incidentally I'm doing all my collision detection 60ms in the past as well and then feeding the correction back to the separately modelled current position.

However I guess that i) doesn't really hack it for 2 player as you need to change direction really fast to go down those narrow alleyways. Perhaps you could additionally send key-up key-down events across the network as they arise. If this was implemented in conjunction with around 60ms forced lag, it might be acceptable. You still need to send regular position updates to cater for when the player movement is effected by collisions.

Maybe first try implementing 60ms lag in your single player game and see if you can live with it.

I've got the same issue with SharpShooter Arena, although being a FPS, it isn't as noticeable. Currently I'm sending position, velocity, rotation updates at 5Hz. However that means if you take your finger off the forward key just after Tx'ing an update, the remote client doesn't find out you've stopped moving for 200mS and overshoots. I thought of three 'solutions':

Simplest solution I see would be just to send an "off" packet.

When the user relases the key send one more packet with final position and velocity 0.

Should solve your problem, yes?

Got a question about Java and game programming? Just new to the Java Game Development Community? Try my FAQ. Its likely you'll learn something!

Thats what I've basically got working at the moment, i) and iii) and I see the same things, the remove player gets to their location then rolls back a bit. Less noticeable I think in a FPS, but for a game of my sort just looks terrible.

If you're gameplay suits it (or, in my case, you're willing to make a few gameplay compromises) lockstep can work rather well. Particularly for hard to predict cases (typically tight interaction between multiple people). FPS' get away with a lot because the physical movement of the bullets gives you a perfect oppertunity to hide/fake (such as displaying blood splats from 'hits' but not actually taking any health off).

What I'm aiming for is a full Gigawing/Ikaruga bullet hell shooter but networked. Given that this requires 100+ moving bullets and pixel-perfect collision there's no way a normal client-server would ever work, even with full prediction. Instead i'm loosening up the lockstep system (much like the AoE method) and instead having the client's control each ship directly (which is more akin to a distributed simulation).

As with most distributed simulation this comes with one big drawback: cheating is now trivially easy as the client is completely trusted for reporting their own collisions. However I'm willing to accept this as:1. Since the game will be co-op, cheating against an 'opponent' is silly and infantile2. For high scores etc. I'll be recording input and playing it back to give a (largely) foolproof method of validation. Given that I'll already need a deterministic simulation this is practically for-free.

Thats what I've basically got working at the moment, i) and iii) and I see the same things, the remove player gets to their location then rolls back a bit. Less noticeable I think in a FPS, but for a game of my sort just looks terrible.

I'm going to move to a lock-step, 1500 archers type system.

Kev

Hmm. I suppose another solution might be to lag your motion slightly in time.. enough to cover that end shift? Another solution often used to cover this is to have a small amount of "acceleration/decelleration' in motion such that you dont have those hard edge cases at start and stop. If you look at GetAMped they do this, I think. They also have a cute "skid" that happens ocassioanlly. My guess is that they are doing soem kind of send/ack thing for motion and when the ack doesnt come abck fast enoyhugh, they skid to hide it.

Lockstep has it own set of problems. Cheif among them are that (a) everyone's performance is only as good the worst performing player and (b) varyinbg latency can make resposiveness uneven and (c) any bad latency spikes stall the game.

You can cover (b) at the cost of a constant control lag by latency-buffering . The others are pretty much hard artifacts of the technique.

Anyway do what works

Got a question about Java and game programming? Just new to the Java Game Development Community? Try my FAQ. Its likely you'll learn something!

I'm starting to consider use cases for this. I'm assuming that I'm only setting commands for some step in the future.

Do these sound right?

- I'm going to support late game entry by pausing the game at a specified frame, transferring state from one of the clients (it shouldn't matter which) to the new client then let the game continue to run.

- I was considering a standard evil type monster. If I send the command [zombie onto player X] and make that the monster move towards the player until in combat range and once in range attack, since my simulations are synhcronised I can assume that the zombie reaction to players will be the same on every machine. Hence the simulation stay synchronous even though some processing is being done all the client? (assuming of course I have a synchronised source of random numbers)

- I'm going to support late game entry by pausing the game at a specified frame, transferring state from one of the clients (it shouldn't matter which) to the new client then let the game continue to run.

Do it in parallel, split between the clients. Remember that upload b/w is these days 4-16 times less than d/l - so the transfer from client-to-client will go at snail's pace. Just carve up the data into e.g. "4 clients, each gets 1/4 of the map to send to the new player". Will go much quicker.

Quote

Hence the simulation stay synchronous even though some processing is being done all the client? (assuming of course I have a synchronised source of random numbers)

Yes. With caveats...

For instance, you are OK not to have all random data identical - what you want is that code which needs to share the same seeded deterministic string does share it, and code that doesn't does not. e.g. the "random bird sounds" mentioned in 1500 archers - no reason for that to be taking out of the main random stream. i.e. you want to do it OOP

The real problem is when you have code that executes a random() call or not depedning upon the output of a random event, and that random() call is a critical one. "random event" can be as simple as a GFX card driver bug causing a section of code in your game to execute slower than it should do, inducing randomness.

So, for instance, multiple threads == death by a thousand painful cuts. You ahve to ensure there is no such thing as a random event in your engine except for your actual output from calls to the random() (or, in java, nextInt() etc) method. This requires care: even something as innocuous as using a GUI rendering thread to service mouse-clicks in parallel with running paint logic is violating that. The pain is tracking down all these sources and finding out how they managed to (eventually) influence a call to random() to be later or not at all on one client compared to the others.

- I'm going to support late game entry by pausing the game at a specified frame, transferring state from one of the clients (it shouldn't matter which) to the new client then let the game continue to run.

That should be possible, but sounds quite tricky. I suppose it depends how much you've got going on in your game world. One possible snag - random numbers (again). Obviously you can't just share the original seeds with the new client. Easiest I suspect would be to re-seed everyone's random number generators with some newly-agreed values.

why not synch randmizer seeds around the clients ?I agree with Adam, I thinks it's quite hard to achieve that with multi-thread

Yes, that was the original assumption - there would be some common source of random numbers.

Quote

That should be possible, but sounds quite tricky. I suppose it depends how much you've got going on in your game world. One possible snag - random numbers (again). Obviously you can't just share the original seeds with the new client. Easiest I suspect would be to re-seed everyone's random number generators with some newly-agreed values.

- I was considering a standard evil type monster. If I send the command [zombie onto player X] and make that the monster move towards the player until in combat range and once in range attack, since my simulations are synhcronised I can assume that the zombie reaction to players will be the same on every machine. Hence the simulation stay synchronous even though some processing is being done all the client? (assuming of course I have a synchronised source of random numbers)

Hmm. Maybe Immissing a few things here...

If youa rent tightrly synchronizing the position of all players, then wont the Zombie AI potentially be afced with different situatiosn on different machines:

eg on machien one its a direct line to the player. On machien 2 we need to go around an obstacle...

Got a question about Java and game programming? Just new to the Java Game Development Community? Try my FAQ. Its likely you'll learn something!

If youa rent tightrly synchronizing the position of all players, then wont the Zombie AI potentially be afced with different situatiosn on different machines:

eg on machien one its a direct line to the player. On machien 2 we need to go around an obstacle...

I think I can read that

Na, the thing you're misisng is that I am intended to do synchonised simulations. I must have forgotten that ever so important detail. I've never done it before and it seemed like it might be quite interesting to work with.

If youa rent tightrly synchronizing the position of all players, then wont the Zombie AI potentially be afced with different situatiosn on different machines:

eg on machien one its a direct line to the player. On machien 2 we need to go around an obstacle...

I think I can read that

Na, the thing you're misisng is that I am intended to do synchonised simulations. I must have forgotten that ever so important detail. I've never done it before and it seemed like it might be quite interesting to work with.

Kev

Okay. Im a bit confused. I thought you said you were releasing the restrictions on local movement to get responsiveness for the player.....

If this is true lock-step then I agree its a non-issue.

Got a question about Java and game programming? Just new to the Java Game Development Community? Try my FAQ. Its likely you'll learn something!

I'm getting a little worried about the term lock-step actually. I can't find a good description anywhere through google so I'm just going to have assume its what I'm doing

1) Client sends command to move Bob to position X2) Server recieves command - timestamps it to be scheduled some configurable time in the future3) Server forwards the command on to everyone connected4) Clients (including the original one) recieve the command and add it to their scheduled list5) As a client's game progresses (in a deterministic manner (1)) they process commands recieved from the server which modifies the out come of the game.

For my personal game:

- This means the player's actions will be delayed slightly, but I'm turning it into a point and click dungeon crawl so it probably doesn't matter too much (ala Diablo?)- If a client recieves a command which is in their game's past, IMO - they've got ahead of time, now invalid. Either resynch with other clients or kill the client off.- If a client recieves a command which is in their game's future but not very far maybe we'll slow the running the game down a bit.

So the clients might be seeing different times in the simulation (by a bit) but they won't actually see different simulations at all.

Is this lock step?

Will this work? Do I need to send constant update messages to keep the games from running off into the future.

Kev

(1) In a deterministic manner - common seeded random number generator, strictfp, common rules governing everything.

Looks about right to me. Although you'll need to do frame-based rather than animation based simulation (to get it deterministic) so instead of timestamps you'll be counting frames.

A 'pure' lockstep you basically stall until you've got the inputs from all players for each frame. Obviously thats not practical over the internet, so you just add a buffer (say, 10 frames long). Raw controller data gets sent straight to other players and buffered. If your buffer is big enough you've always got everyones data ready for each frame. If not you stall until you get the next lot of input. Because you're waiting for the other player's input you don't have to explicitly slow down or speed up any of the simulations as it keeps itself in check.

Data from the past shouldn't happen - the only possiblility would be that it's a duplicate of data you've already got. If it's data you havn't got then you'll be waiting for it (and it'll be 'present' data). Data from the future going in the appropriate place in the buffer for use in a few frames or so.

AoE's method basically says instead of syncing per-frame you sync per 'turn' which is game-defined. They're only buffering one turn ahead, because ideally a turn is long enough not to cause a stall. Game logic within the turns stays nice and deterministic. Ideally you adjust the turn length based on the connection quality so you're never stalling but at the same time keeping the response nice and snappy.

Yeah FWIU I think I would call AOE's solution "turn based" rather then lock-step myself.

Your scheme si kidn of a hybrid. Its actually closest in some ways to lock-step plus latency buffering, but your sort of future-buffering rather then input buffering.

I need to thin ka bit about what the advantages/disadvantages are.

One qustion I have is, what do you do if someoen gets a really bad latency spike and misses getting a scheduled move unti lafter that time? Do you stall their game? If so, what "time" are future mvoes made at when they hit the sevrer?

Seems like getting thsi right coudl get kidna complex...

Got a question about Java and game programming? Just new to the Java Game Development Community? Try my FAQ. Its likely you'll learn something!

Isn't it impossible to receive commands from other clients that are too far in the past or future?

I thought that a client couldn't move past the next clock tick until it had received the command set from every other client for that clock tick. Clients cannot therefore fall behind or get ahead of other clients.

i.e.Tick = 0Client sends command set for interval 0->1 to every other playerClient waits until it has received command set from every other playerClient updates game stateTick = 1

On top of that you build an interpolator for intermediate states. If this is fully deterministic all clients will do the same thing.

The drawback I believe is that the player inputs can't be processed until you have a full intervals worth, which is sent to the server. Surely this means that the game still lags by one lockstep interval. i.e. I don't think it solves the problem which caused Kev to consider lock-step in the first place. I think it does solve the issue of how to synchronise AI players without using a huge amount of network bandwidth.

I think there's some articles about lockstep in one of the GEMs series. #3 I think (not checked)

One qustion I have is, what do you do if someoen gets a really bad latency spike and misses getting a scheduled move unti lafter that time? Do you stall their game? If so, what "time" are future mvoes made at when they hit the sevrer?

You'd have to. But then as they're no longer sending current commands everyone else eventually reaches the end of their buffer and ends up waiting themselves for the spiked player to catch up.

It messes with your head, but lockstep is surprisingly robust with just a few simple rules (assuming you can keep it deterministic).

What Alan described is what I understood to be pure lockstep, you've basically got to have confirmation that everyone has reached a particular frame before continuing. I don't really want to do this.

If one player is lagging behind I'd like it lag their commands getting distributed but not the game. If you've got a terrible ping then your command response will be terrible (up to a point when you actually run too slowly and start recieving commands for the past and hence have lagged out).

As to huge latency spikes this is what I meant by this:

Quote

- If a client recieves a command which is in their game's past, IMO - they've got ahead of time, now invalid. Either resynch with other clients or kill the client off.

If a players connection get so bad that they're game progresses past where commands they haven't recieved yet are meant to be scheduled they're completely scuppered. Either the whole game has to be stalled while this player gets synched up or they get disconnected and thrown away

I'm not hugey keen on the not having full lock step the more I think about it, but I'm hoping I can deal with it as it goes.

If one player is lagging behind I'd like it lag their commands getting distributed but not the game. If you've got a terrible ping then your command response will be terrible (up to a point when you actually run too slowly and start recieving commands for the past and hence have lagged out).

Now you're getting tricky.

Typically you'd just lag everyone's commands by a suitable amount to cover everyone's worst case latency. Now if you assume everyone's got a good connection then 200ms ping is about the worst you'll get. That's only a tenth of a second before commands get executed, which is probably unnoticable (more so if you do something to hide it, like verbal acknoledgement of orders straight away). But one bad connection does turn into command lag for everyone.

I suppose you could make each person work on individual 'turns' of different lengths. People with good connections get nice small turns and those with bad get delayed more. The problem is that the person with a bad connection still needs to receive and ack the commands from the good connection people so i don't think that's going to be too effective.

I'd suggest trying with a single command lag for everyone and seeing how it behaves. I suspect it'll be more responsive than you think as long as you stay below about 8 players (how many players were you aiming for?)

- The lockstep interval is small (say 20mS)- keyboard commands are scheduled for a future step about 10 to 100mS in the future depending on the worst client latency.- keyboard commands are sent on key down/up rather than every frame.- clients assume that all commands for a given step will have reached them before that step is due to be processed. This is necessary to remove the requirement for each client to transmit state every lockstep interval, which would clog up the network.

Problem - If a client lags so badly (lag spike) such that his/her transmitted commands are not received by one or more of the other players before the aaplicable time step is processed then the game must reset to the last commonly agreed state. There must be a protocol to achieve this tricky resynchronisation.

To make this work all players must have forced lag. Once you have this and send key up/down events as they arise, rather than at 5Hz or 10Hz intervals as in the non-lockstep technique, then the overshoot problem should largely go away (except during a lag spike) even without lockstep. Of course without lockstep, the clients tend to drift, so a periodic game state must be sent in addition to key states, to ensure everything stays in step. Thus in my view, the main benefit of lockstep is still reduced network bandwidth when there are a lot of non-player objects flying about. This may well be a key benefit if you are intending to use the Game Garden API without swamping their server with position updates.

I'd suggest trying with a single command lag for everyone and seeing how it behaves. I suspect it'll be more responsive than you think as long as you stay below about 8 players (how many players were you aiming for?)

The spec is for 4 players currently.

Quote

- The lockstep interval is small (say 20mS)- keyboard commands are scheduled for a future step about 10 to 100mS in the future depending on the worst client latency.- keyboard commands are sent on key down/up rather than every frame.- clients assume that all commands for a given step will have reached them before that step is due to be processed. This is necessary to remove the requirement for each client to transmit state every lockstep interval, which would clog up the network.

No keyboard controls - all point and click. The commands would be scheduled at minimum 100 ms in the future (and thats after its reached the server). Clients assume that all commands reach them before they need to be actioned, should they recieve a command in the past then thats the interesting part:

a) Kill the client - they've got a laggy connection so lets get rid of them b) Attempt to resync the client to the others by pausing everyone (tricky business)c) Possibly rewind the game state (undo on each command passed around) to get back to the correct state then replay.d) Maybe one copy of the game state could be maintained on the client which is updated when a recieved command is action. If they ever get ahead of the game and recieve an old command you simply jump back to the stored state, replay the game up until the point you want at top speed, then apply the new command.

b) Attempt to resync the client to the others by pausing everyone (tricky business)

It's really not that tricky - in fact it happens near-automatically. As I said earlier you just make sure you only advance when you've got *everyones* input. This may mean one person stalls waiting for a 'complete set'. The other peers are still sending this by default, so it eventually gets fixed up. Meanwhile the other players can advance *only* as far as they themselves have a complete set of commands. As one person is delayed this automatically makes everyone else wait for the simulations to match up again and commands are again arriving and leaving in sync.

Bonus points for freezing the logic but keeping the graphics/audio still running so small stalls are less noticable and ugly.

Quote

c) Possibly rewind the game state (undo on each command passed around) to get back to the correct state then replay.d) Maybe one copy of the game state could be maintained on the client which is updated when a recieved command is action. If they ever get ahead of the game and recieve an old command you simply jump back to the stored state, replay the game up until the point you want at top speed, then apply the new command.

Trailing State Syncronisation. Bleeding edge stuff, but basically an extrapolation of what you've just described. Snags are of course that you effectively double your CPU and memory use of your simulation (which may be acceptable depending on how big and complex it is), and you can have nasty time warps.

If one player is lagging behind I'd like it lag their commands getting distributed but not the game. If you've got a terrible ping then your command response will be terrible (up to a point when you actually run too slowly and start recieving commands for the past and hence have lagged out).

Now you're getting tricky.

Typically you'd just lag everyone's commands by a suitable amount to cover everyone's worst case latency. Now if you assume everyone's got a good connection then 200ms ping is about the worst you'll get.

Now this is getting pretty close to the latency bufering technique I know. if so, its actually much simpler to implement.

The only thing latency buffering adds is a queue for all the input. Eahc slot in the queue is 1 complete set of everyone's input. The buffer conceptuially starts with a full set of null entries so you dont start the game unti lthe first non-null entry has hit the head of the queue. Input and queue writing is doen ona seperate thread from queue reading so yo uare never sitting waiting for inptu unelss the queue fully empties.

Got a question about Java and game programming? Just new to the Java Game Development Community? Try my FAQ. Its likely you'll learn something!

DukeNukem3D on TEN used about 200ms latency buffering at 7 packets per second communications and played pretty darn well

In general, its actually preferrable to have a pedictable latency pause, even a longer one, then unpredictable shorter ones. This has been proved in psych tests. HUmans quickly elanr to factor in predictable latencies but unpredictable oens drive them nuts.

Got a question about Java and game programming? Just new to the Java Game Development Community? Try my FAQ. Its likely you'll learn something!

java-gaming.org is not responsible for the content posted by its members, including references to external websites,
and other references that may or may not have a relation with our primarily
gaming and game production oriented community.
inquiries and complaints can be sent via email to the info‑account of the
company managing the website of java‑gaming.org