While loading extra huge maps, the loading method throws out of memory exception where a new instance of map tile is created. I'd like to have whole map processed at least on server app (and on client if possible). How should I solve this problem?

UPD: the question here is how to make the game stop crashing when there's still free memory for it to use. As for splitting map in chunks, it's a good approach, but not what I want in my case.

UPD2: At first I went and assigned a texture to every new instance of tile class and that's what took so much memory (also loading time). Now it takes about four times less space. Thanks to everybody, now I can run huge maps without thinking of breaking them up in chunks just yet.

UPD3: After rewriting the code to see if arrays of tile properties work faster or consume less memory (than said properties in their respective tiles as object properties), I found out that not only it took me a lot of time to try that, it didn't bring any performance improvement, and made the game terribly difficult to debug.

20 million tiles at 4 bytes per tile is only about 80MB - so it sounds like your tiles are not so small. We can't magically give you more memory so we have to work out how to make your data smaller. So, show us what is in these tiles.
–
KylotanJul 17 '12 at 11:45

What does the loading method do? Post code, do you use the Content pipeline? Does it load textures to memory or just metadata? What metadata? XNA content types generally reference unmanaged DirectX stuff so the managed objects are very small, but on the unmanaged side there could be loads of stuff you don't see unless you're running a DirectX profiler as well. stackoverflow.com/questions/3050188/…
–
Oskar DuvebornJul 17 '12 at 15:10

2

If the system throws an out of memory exception, then there isn't enough free memory to use, despite what you might think. There won't be some secret line of code we can give you to enable extra memory. The content of each tile and the way you're allocating them is important.
–
KylotanJul 17 '12 at 15:30

3

What makes you think you have free memory? Are you running a 32-bit process inside a 64-bit OS?
–
Dalin SeivewrightJul 17 '12 at 18:23

6 Answers
6

This strongly suggests that you're not representing your tiles correctly, as this would mean that each tile is ~80 bytes in size.

What you need to understand is that there needs to be a separation between the gameplay concept of a tile, and the visual tile that the user sees. These two concepts are not the same thing.

Take Terraria for example. The smallest Terraria world takes up 4200x1200 tiles, which is 5 million tiles. Now, how much memory does it take to represent that world?

Well, each tile has a foreground layer, a background layer (the background walls), a "wire layer" where wires go, and a "furniture layer" where furniture items go. How much memory does each tile take up? Again, we're just talking conceptually, not visually.

A foreground tile could easily be stored in an unsigned short. There aren't more than 65536 foreground tile types, so there's no point in using more memory than this. The background tiles could easily be in an unsigned byte, as there are fewer than 256 different types of background tiles. The wire layer is purely binary: either a tile has a wire in it or it does not. So that's one bit per tile. And the furniture layer could again be an unsigned byte, depending on how many possible different pieces of furniture there are.

You should never have this representation stored as arrays of C# classes. They should be arrays of integers or something similar. A C# struct would work as well.

Now, when it comes time to draw part of a map (note the emphasis), Terraria needs to convert these conceptual tiles into actual tiles. Each tile needs to actually pick a foreground image, background image, an optional furniture image, and have a wire image. This is where XNA comes in with its various sprite sheets and such.

What you need to do is to convert the visible part of your conceptual map into actual XNA sprite sheet tiles. You should not be trying to convert the entire thing at once. Each tile you store should just be an index saying that "I'm tile type X," where X is an integer. You use that integer index to fetch which sprite you use to display it. And you use XNA's sprite sheets to make this faster than just drawing individual quads.

Now the visible region of tiles needs to be broken up into various chunks, so that you're not constantly building sprite sheets whenever the camera moves. So you might have 64x64 chunks of the world as sprite sheets. Whichever 64x64 chunks of the world are visible from the player's current camera position are the chunks you draw. Any other chunks don't even have sprite sheets; if a chunk falls off the screen, you throw that sheet out (note: you don't really delete it; you keep it around and respecify it for a new chunk that may become visible later).

I'd like to have whole map processed at least on server app (and on client if possible).

Your server does not need to know or care about the visual representation of tiles. All it needs to care about is the conceptual representation. The user adds a tile here, so it changes that tile index.

Split the terrain into regions or chunks. Then only load the chunks that are visible to the players and unload the ones that are not. You can think of it like a conveyor belt, where you're loading chunks at one end and unloading them at the other as the player moves along. Always staying ahead of the player.

You can also use tricks like instancing. Where if all the tiles of one type just have one instance in memory and are drawn multiple times in all the locations where it's needed.

Problem is such approach is good for client app, but not as good for server. When some tiles in the world go off and a chain reaction begins (and that happens a lot) the whole map should be in memory for that and it's a problem when it throws out of memory.
–
user1306322Jul 17 '12 at 3:48

6

What does each tile contain? How much memory is used per tile? Even at 10 bytes each you're only at 190 megabytes. The server shouldn't be loading the texture or any other information that's not needed. If you need a simulation for 20 million tiles, you need to strip away as much as possible from those tiles to form a simulation set that only has the information required for your chain reactions.
–
Byte56♦Jul 17 '12 at 3:53

Byte56's answer is good, and I started writing this as a comment to that but it got too long and contains some tips that might be more helpful in an answer.

The approach is absolutely just as good for server as it is for a client. In fact it's probably more appropriate, given that the server will be asked to care about far more areas than a client will. The server has a different idea about what needs to be loaded than a client would (it cares about all the regions that all the connected clients care about), but it is equally capable of managing a working set of regions.

Even if you could fit such a gigantic map (and you most likely can't) in memory, you don't want to. There is absolutely zero point in having data loaded which you don't immediately have to use, it's inefficient and slow. Even if you could fit it in memory, you most likely wouldn't be able to process it all in any sensible length of time.

I suspect your objection to having certain regions be unloaded is because your world update is iterating over all the tiles and processing them? That's certainly valid, but it doesn't preclude having only certain parts loaded. Instead, when a region is loaded, apply any updates that were missed, to bring it up to date. That's far more efficient processing-wise as well (concentrating a big burst of effort on a small area of memory). If there are any processing effects which might cross region boundaries (e.g. a lava flow or a water flow which will carry over into another region), then bind those two regions together so that when one is loaded, so is the other. Ideally those situations would be minimised, otherwise you'll find yourself quickly back at the 'all regions must be loaded at all times' case.

use spritesheets, not an image for each tile/object, and just load what you will need on that map;

split the map in smaller parts, say chunk maps of 500 x 200 tiles if your map is 4 times this size, or something you can haldle;

load this chunk in memory and paint only the visible parts of the map (say, the visible tiles plus 1 tile for each direction the player is moving) in a offscreen buffer;

clear the viewport and copy the pixels from the buffer over (this is called double buffer and smooths the moviment);

when you char moves, track the bouds of the map and load the next chunk when it gets near it and append to the current one;

once the player is entirelly in this new map, you can unload the last one.

You could also use a single big map in this way if it is not SO big and use the logic presented.

Off course the size of your textures have to be balanced and the resolution pays a great price in this. Try working at a lower resolution and, if you need, make a high res pack to load if a config setting is set to.

You will have to loop only the relevant parts of this map each time and render only what is needed. This way you will improve the performance a lot.

A server can process a huge map faster because it don't have to render (one of the slowest operations) the game, so the logic for it can be the use off a bigger chunks or even the entire map, but process only the logic around players, like in a 2 times the player viewport of area around the player.

An advice here is that I'm by no ways an expert in game dev and my game was made for my graduation's final project (which I had the best score), so I may be not so right in every point, but I've researched the web and sites like http://www.gamasutra.com and the XNA creators site creators.xna.com (at this time http://create.msdn.com) to gather knowledge and skills and it worked fine for me.

I have 8 Gb ram on win7 64bit and the app crashes when it reaches 1.5Gb. So there's really no point in buying more ram unless I'm missing something.
–
user1306322Jul 17 '12 at 14:49

1

@user1306322: If you have 20 million tiles, and it takes up about ~1.5GB of memory, that means your tiles are approximately 80 bytes per tile. What exactly are you storing in these things?
–
Nicol BolasJul 17 '12 at 16:54

@NicolBolas as I said, a few ints, bytes and a texture2d. Also they don't all take 1.5GB, the app crashes when it reaches this memory cost.
–
user1306322Jul 17 '12 at 18:12

2

@user1306322: That's still far more than it needs to be. How big is a texture2d? Does each tile have a unique one, or do they share textures? Also, where have you said this? You certainly didn't put it in your question, which is where the information should be. Explain your specific circumstances better, please.
–
Nicol BolasJul 17 '12 at 18:49

@user1306322: Frankly, I don't quite get why my answer was downvoted. I've enumerated three remedies to out-of-memory situations - of course, that doesn't mean that all of them necessarily apply. But I still think they are correct. I should probably have written "extend your memory" instead of "buy" to include the case of virtual machines, though. Am I being downvoted because my answer was concise instead of verbose? I think they points are straight-forward enough to be understood without further explanation?!
–
ThomasJul 17 '12 at 23:57