Game development, technology, and Japan.

I use GameObject.Find() in Unity for things like enabling or fading in/out a menu or to grab an object reference via code to store for later. (I usually prefer doing things in code rather than drag and dropping references using the Unity Editor when I can)

A problem is GameObject.Find() won’t locate inactive gameobjects which causes me problems because I tend to have inactive object trees in a scene that are just turned on/off when they are being used, like a GUI menu for example. It’s just kind of my programming style to do things that way.

I couldn’t find a clean full snippet for this online that used scene.GetRootGameObjects, so figured I’d post one.

The RTsoft PUBG squad both online and IRL. Akiko, June, Cosmo. Pic taken by Seth

A true story from PUBG

“I think I hear footsteps outside” Akiko whispered. I was confident we’d be safe, at least for a while, in the mountain shack we’d found. Our blessed respite from the cruel world of PUBG was about to be shattered.

“Stay here, I’ll check it out”. I opened the door and creeped around the outside of the building. “They think they can come here and threaten… ” I didn’t get a chance to finish my thought as it was unceremoniously interrupted by a shotgun blast to the back of my head.

Akiko screamed as she watched through the window. I fell to my knees and tried in vain to crawl back to the door. He stood over me, gun in hand, preparing a second shot to end my suffering.

But the shot didn’t come. He’d noticed movement inside the house. The bastard turned his attention toward my wife and there was nothing I could do about it.

In a panic, hands shaking, Akiko burst from the cabin firing wildly. But alas, her bullets did not meet their intended target.

The cutthroat returned fire and brutally put her down. I collapse only inches from her sprawled body. She died trying to save me.

Why PUBG is good

PUBG (and the survival/battle royale rules that Brendan Greene and others have developed and tweaked) breaks with tradition in a lot of ways:

There is no story (other than your own)

There is no voice acting (other than the occasional grunt)

There are no cut scenes

There is no text chat

Name labels are not drawn over enemies

A single round can last up to 35 minutes

Matchmaker ignores skill/ratings and just puts everybody together

It can be unfair. It’s not designed to be fair

You are dumped into a large open world with random loot and vehicle placements. It supports varied play styles, you can rambo it up and shoot everyone, or be stealthy and win without firing a shot. There really isn’t a wrong way to play.

A big part of the allure is the variety of situations that can occur due to randomness. No two games are alike. The scavenging aspect is a form of slot machine gambling (the good kind, not to be confused with money sucking loot crates), will you find that 8x scope in that bathroom or just another pair of shoes?

If you can find the right pieces for your gun, you can sort of create a matching set that gives you an advantage. Looting more houses gives you more lottery tickets to scratch.

In some ways it takes inspiration from games like FTL or Weird Worlds: Return to Infinite Space in that random loot drives a game that lasts less than an hour. This combined with solid FPS gunplay and huge worlds (the “can’t find other players” problem has been neatly solved by an ever-shrinking playfield) present an amazing experience.

Loot crate controversy!

Jim Sterling is doing the Lord’s work by calling out the recent crate madness. I don’t think people should support premium games that give a clear round-winning advantage to those who spend more, play a different game instead.

I’ve never bought a loot crate in any game. PUBG’s cosmetic crates don’t bother me. I just sell the ones I naturally earn through gameplay via the Steam store. I’ve made $40+ US doing that so hey, it paid for the game. <shrug>

PUBG Bugs and technical considerations

Nothing is perfect. Cheating is rampant. To give you an idea, over 1.5 MILLION accounts have been banned from PUBG. (that’s $45M in purchased copies, it’s insane)

I suspect the recent rubber-banding issues were from new anti-hack security, the more accurately you want to check and verify player actions, the slower the server gets. (I have a lot of experience with this…)

I play on the KR/JPN servers and latency is often an issue. PUBG does not give us any in-game tools to clearly check our latency which is a bummer because it DOES matter when resolving “who shot first”.

The future

People were ready for a game mode that cut out the fluff and just presented the meat. As usual, after a hit like this, over saturation will occur and soon enough, we’ll be ready for the next thing…

If you’ve looked at my recent Unity-related posts and downloaded the projects, you might have noticed I have .bat files like CreateAndUploadWebGLBuild.bat in there to cleanly create final versions easily.

Great. But if you run this .bat file while you are working on the game with the Unity editor, you’ll get this error:

Aborting batchmode due to failure:
Fatal Error! It looks like another Unity instance is running with this project open.
Multiple Unity instances cannot open the same project.

Ugh. WebGL builds are especially are incredible slow, so this is a big productivity waster if you’re doing a lot of WebGL testing. (If you work at a big company and waiting for builds is your favorite time to make coffee and catch up on reddit, well, close this page right now and hope your boss never reads this!)

Cloud Build?

Maybe you could use Unity’s Cloud Build but there are some down sides:

Cost $9 a month?

Requires that you commit each change to a cvs such as svn, perforce, or git

I doubt it can do custom post build commands such as code signing, building a final installer, or rscying files to a linux server and restarting the process. I guess you could do those things yourself when the build is done, but heck, why not just handle the build yourself from the start.

Requires that all your assets are also on cvs (?)

The DIY way

So let’s do it old school with … yep, you guessed it… even more .bat files! The secret is very simple, Unity will allow you to build in the background if the project directory is different.

You just need to copy your entire project to a temporary folder, then run Unity.exe with parms to do a headless build like normal.

So when you are in a good place with your project and would like to start a background build, hit Ctrl-S to save, then run your “CopyToTempDirAndBuild.bat” file.

After a couple seconds the initial copy is done and it’s safe to continue working while the build is happening in the temp directory – any changes will not be in the temp directory, so your build in progress won’t be affected. So you can keep working away, without ever closing your main unity editor window.

It’s not especially tricky to do, but here are some .bat files to look at as an example that could be tweaked.

To copy a directory tree to a temp dir: (it assumes the .bat is run from the directory that’s going to be copied)

Note: You may wonder why I’m being repetitive and using “Temp” everywhere instead of including it in a variable. It’s because you NEVER, NEVER use things like rmdir with only a variable if you can help it, because at some point, that variable is going to be set incorrectly. It might be .. or / or something and you’ll delete your whole hard drive. Safety first.

Another note: %cd% is a DOS trick to get the current full directory

CloneExclusionList.txt contains dirs we don’t want to waste time copying:

Temp\
Library\
build\win
build\linux
build\web\Build

BuildAndUploadWebGLInClonedDir.bat

:this sets some info about the project, for example, it causes %APP_NAME% to hold our main directory name
call app_info_setup.bat
:Setting no pause causes our .bat files to not do pause commands when they are done
set NO_PAUSE=1
:First, let's customize the directory name we're going to close to, by adding a postfix to make it unique
SET CLONE_DIR_POSTFIX=WebGL
:Now let's actually make it, we'll pass in the postfix as a parm
call CloneToTempDir.bat %CLONE_DIR_POSTFIX%
:Move to build dir
cd ../%APP_NAME%Temp%CLONE_DIR_POSTFIX%
:Do the actual build
call BuildWebGL.bat
call UploadWebGLRsync.bat
:Move back out of it
cd ..
:Delete the temp dir we were just using
rmdir %APP_NAME%Temp%CLONE_DIR_POSTFIX% /S /Q
pause

This calls CloneToTempDir.bat with the parm “WebGL” which gets appended to the <AppName>Temp dir.

It then “calls” (this means run another .bat, and come back when it’s done) .bats to create the webgl build and also upload it to the website.

It then destroys the temp directory completely, a good idea because Unity will mark it as the last project and you don’t want to accidently work on that directory later.

Parallel Builds

If you’ve got 16 threads sitting around like I do, it might make sense to build MORE THAN ONE version at a time. (now you see why I use a custom temp dir name for each build)

Apparently, Unity doesn’t care how many simultaneous builds you’re doing on a single computer, as long as its license is valid. (I’m using a pro license)

The key to running parallel builds is to use the “start” command instead of “call”. This means “run this, but instead of waiting, just continue running the rest of this .bat file”.

So, including the Unity editor you have open, when this is run you’ll have THREE instances of Unity running on the same computer at once. It all works fine!

Continuous integration as a background operation on the same computer you develop on

If you add a “goto :start” at the bottom of your .bat and a “:start” label at the top, you can “clone and build” non-stop all day. I don’t see this as very useful as it’s going to break all the time as we’re not doing controlled commits with a cvs but I thought I’d throw that there.

So you thought my post on networking a few days ago was boring? Get ready for database fun!

Normally I would just write about what worked, but in this case I’d rather trek through the foggy world of missteps I went through yesterday to make the various errors searchable, maybe my confusion and pain can be of help to some poor future googler.

Why do you want to directly talk to mySQL from a Unity app?

The first thing you normally see when searching online for help with integrating SQL and Unity is “you idiot, it’s too dangerous to ever connect directly to your server database!”

Well, for the CLIENT APP, that’s correct, it’s a horrible idea because anybody can steal your logon credentials and ruin your database. (I love you Team Meat, but … yeah)

Instead, you need a gateway of some kind (for high scores, this is usually a .php page on a website) that can do things like limit score additions from the same day from the same IP address and other types of ‘hardening’.

But for a game server that needs to access lots of data, a direct connection to a database from your Unity-based dedicated server is absolutely vital.

Step 1: Getting the MySQL Connector/Net files

I downloaded the files for “.NET & Mono” and drag and dropped the 4.5 branch files into my Unity project. (spoiler, this was dumb, I only needed one)

Note: Managed .dll files do NOT have to be dragged into any specific folder like C++ plugins do, anything inside of your Assets folder is ok, even sub-folders.

Step 2: Activating Unity’s 4.5 .Net support

From inside of Unity, I clicked on one of the newly added .dll files. Near the bottom of the property window (it’s easy to miss) it gave this message:

“Plugin targets .NET 4.x and is marked as compatible with Editor, Editor can only use assemblies targeting .NET 3.5 or lower, please unselect Editor as compatible platform.”

Step 3: Noticing I added the wrong files

I added ‘using MySql.Data;‘ to a .cs as a test. It compiled, but when I ran the game I got a “TypeLoadException” without any details. I installed Unity 2017.3.01f (beta) because I heard it gave better description of these kinds of errors, and it did:

TypeLoadException: Could not load type of field 'MySql.Data.Entity.MySqlMigrationCodeGenerator+<>c:<>9__4_0' (1) due to: Could not load file or assembly 'EntityFramework, Version=5.0.0.0,'

EntityFramework? Ohhh, so that zip had multiple versions of MySQL Connector and this one wants EntityFramework, whatever that is. I removed them all (and the .xml files too) and re-added MySql.Data.dll.

Note: You probably don’t need to update to a beta version like I did; this isn’t recommended usually. Obviously I just like to live dangerously.

Great, no errors and I can do a ‘using MySql.Data;‘ just fine.

Step 4: Adding support for System.Data

I added ‘using System.Data;‘ as we’ll need that too. Uh oh, got an error there. So I dragged and dropped <unity install dir>\Editor\Data\MonoBleedingEdge\lib\mono\4.6-api\System.Data.dll into my project. (I’m attempting to use the version that comes with Unity that matches the .NET version I’m using)

Well, it now compiles, but now I’m seeing ‘loading script assembly “assets/plugins/system.data.dll’ failed!’ error in the Unity log. Is this the wrong version?

The error goes away if I use <Unity dir>\Editor\Data\MonoBleedingEdge\lib\mono\2.0-api\System.Data.dll instead. Fine, I’ll use that. (spoiler, this causes problems later)

I wrote some test code using MySQL Connector, it works fine in the Unity Editor! I’m able to create/destroy/modify SQL databases. No warnings in Unity either! But uh oh, now I’m seeing this in Visual Studio:

warning MSB3277: Found conflicts between different versions of the same dependent assembly that could not be resolved.".

I guess Visual Studio is set to use a newer version and having the old System.Data.dll referenced is bad?

Not only that, but when I build a Win64 stand-alone version, when running it, I got this error:

NullReferenceException: Object reference not set to an instance of an object
at MySql.Data.MySqlClient.MySqlPoolManager.GetKey (MySql.Data.MySqlClient.MySqlConnectionStringBuilder settings) [0x00012] in <861c41359f7d4496a8fdf269ae744290>:0
at MySql.Data.MySqlClient.MySqlPoolManager.GetPool (MySql.Data.MySqlClient.MySqlConnectionStringBuilder settings) [0x00000] in <861c41359f7d4496a8fdf269ae744290>:0
at MySql.Data.MySqlClient.MySqlConnection.Open () [0x0016d] in <861c41359f7d4496a8fdf269ae744290>:0
at RTSqlManager.Init (System.String serverIP, System.String userName, System.String userPassword, System.String databaseName) [0x00083] in F:\Unity\SQLTest\Assets\RT\MySql\RTSqlManager.cs:51
at SQLTest.Start () [0x00030] in F:\Unity\SQLTest\Assets\_Script\SQLTest.cs:64

Uh… ok, I guess I used the wrong version of the .dll or something, I can see it’s being included but … when I look around, I can see other .DLL’s are coming from directory is actually from <Unity dir>\Editor\Data\MonoBleedingEdge\lib\mono\unityjit and not where I took them from. Using the System.Data.dll from this “unityjit” dir fixed the Visual Studio warning AND fixed the standalone versions (Win and Linux at least, that’s all I tested).

It works! So is it fast?

Hallelujah! Everything runs ok, both local in Windows (to test this, you need to setup a MySQL server on Windows) and on linux. (I’m actually using MariaDB, a MySQL fork, that works fine too)

Speed seems comparable to using the C++ version. As is my way, I wrote a Unity app with built-in Telnet server to test speeds, here are the results on a dedicated linux server:

Final thoughts

Well, I probably should have created the table with ENGINE=MEMORY as I suspect the update record test was hampered by IO writeback (this system doesn’t have an SSD), but overall I’m happy with these speeds.

I wonder how the speeds would compare with SQLite? Is the fact that I’m using a beta version of Unity change/break anything I did? I don’t know, but it works fine now so whatever.

In addition to writing my own junk, I regularly play with Unity and Unreal and am constantly trying to see how they can fit my developing style – mostly I need them because it’s becoming a losing battle to try to maintain your own 3D engine if you also want to actually design and release games too.

What’s LLAPI, anyway?

LLAPI stands for Low Level Application Programming Interface. It’s what Unity’s HLAPI (High level) API is built on. I guess both of those can be called part of UNet. (Unity networking)

Why not just use the HLAPI like a normal person?

That might be ok for some projects, especially round-based networked games without too many players.

When you are targeting MMO level network performance where a server may have to track stats of millions of players and simulate a world for weeks instead of minutes, the first rule is:

Stay as low level as you reasonably can.

I don’t want someone else’s net code to even think about touching my gameobjects, I just want it to deliver reliable/unreliable packets that I write/read straight bytes to. That’s it. I want to handle things like prediction, dead reckoning, and variable serialization, myself!

What’s the deal with UDP, TCP, and WebSockets?

Both UDP and TCP are internet protocols built on top of IP.

TCP is a bidirectional socket based connection that insures your data is correct and sent in the right order.

It’s like calling Jeff, he picks up the phone and you can both talk until one of you puts the phone down. Oh, and after you say something, you make sure Jeff understood you correctly each time, so that eats up a bit of time.

UDP is is a stateless connection where single packets get sent. You write your message on a rock and throw it at Jeff’s window. While it gets there fast, it’s possible you’ll miss and Jeff will never see it. If dozens of people are throwing rocks, he needs to examine each one carefully to figure out which pile to put it with so the text is attributed to the right thrower. He might read the second rock you threw before the first one.

WebSockets are basically TCP with a special handshake that’s done over HTTP first. To continue this already questionable analogy, it’s as if you had to call Jeff’s mom for permission, and then she have you his cell number to call.

In theory WebSocket performance could be similar to TCP but in practice they are much slower and unreliable, I don’t really know why but hey, that’s what we get. I suspect this will improve later.

For games, UDP is nice because you can send some of your packets fast with no verification that it was received, and others (more important packets, like dying) with verification which sort of makes those packets work like TCP.

That said, many most games probably would be ok with a straight TCP steam as well and having a connected state can make things easier, like easily applying SSL to everything and not worrying about unordered packets.

Why use WebSockets instead of UDP or TCP? Just use UDP or TCP for everything, moran

BECAUSE I CAN’T! Browsers won’t allow the Unity plugin to run anymore, they just allow straight javascript/WebGL these days. It hurts. I mean, I wrote a cool multiplayer space taxi test for Unity using TCP sockets and now nobody can even play it.

What does LLAPI use?

It can read from both sockets (UDP) and WebSockets at the same time and route them so your game can let them play together. But how fast is it, and does it work?

This brings us to the second rule of netcode:

Use stress tests without game code to notice glaring problems early and more clearly

Which finally brings us to the point of this post. NetTest is a little utility I wrote that can:

Run as a client, server, or both

If you connect as “Normal client” you can all chat with eachother

Run with a GUI or as headless (meaning no GUI, which is what you want for a server usually)

Can open “Stress clients” for testing, a single instance can open 100+ clients sockets

When stress mode is turned on, the stress clients will each send one random sentence per second. The server is set to re-broadcast it to ALL clients connected, so this generates a lot of traffic. (300 stress clients cause the server to generate 90K+ packets per second for example)

Everything is setup to push performance. Things that might mess with the readings like Unity’s packet merging or modifying values based on dropped packets is disabled. All processing is done as fast as possible, no throttling or consideration to sleep cycles.

Keep in mind a real game server is going to also being doing tons of other things, this is doing almost nothing CPU wise except processing packets. Testing things separately like this make it easier to see issues and know what the baseline is.

ABOUT TEST RESULTS

These tests are presented ‘as is’, do your own tests using my Unity project if you really want exact info and to check the settings.

My windows machine is a i7-5960X, the remote linux box is similar and hosted on a different continent, I get a 200 ping to it.

All packets in NetTest are being sent over the ‘reliable’ channel. No tests involve p2p networking, only client<>server.

TEST 1: 300 CLIENT IDLE (LOCALHOST, WINDOWS BOX)

Ok, here I’m curious how many bytes LLAPI is going to use with clients doing nothing.

Server is set to localhost. I’ve got 4 NetTest’s running – the one on the bottom right has had “Start local host” clicked, and the three others each have added 100 stress clients.

Results:

Around 30 bytes per stay alive packet per client. (23KB total over 7.74 seconds, sent 2.2 packets per client over 8 seconds, so roughly what I would expect for the 4000 ms ping timeout setting) Are stay alives sent over the reliable channel? Hmm.

Adding and removing test clients is a blocking operation and gets progressibly slower as more are added – why does it get slower and cause my whole system to act a bit weird?

While closing the instances looks normal, the instances are actually spending an additional 30 seconds or so to close sockets that were opened

Side note: Having a client ping the host (at localhost) takes about 2 ms. (normally this would be bad, but given that it can’t check until the next frame, then once more to get the answer, this seems to match the framerate decent enough)

I’m not going to worry about the socket slowness, this may be related to a windows 10 resource thing. I’ll launch with less clients in future tests to help though. The server itself has zero speed/blocking issues with actually connecting/disconnecting though, so that’s good.

TEST 2: 200 CLIENT STRESS (LOCALHOST SERVER, WINDOWS BOX)

Ok, now we’re getting serious. I have 5 instances – 4 have 50 stress clients, 1 is the server. (I could have run the server on one of the stress client instances, but meh)

First I enabled 50 clients in “stress mode” – this should generate 10,000 packets (50*200) per second. The server gets 50 lines of text from clients, then broadcasts each one to each of the 200 clients. Huh, stats show it only sending around 6,000 packets per second, not 10,000. However, we can see all the lines of text are being delivered.

That seemed fine so I upped it to all 200 clients doing the stress test – this should cause the server to send 40,000 packets a second (and receive 200 from the clients).

The stats show only 24,549 packets and 5.3 MB per second being sent. About 15% is unity packet overhead, seems like a lot but ok. Server FPS is good, slowest frame was 0.02 (20 ms) so not bad.

Is unity combining packets even though I set config.MaxCombinedReliableMessageCount = 1; ? Oops, I bet that needs to be 0 to do no combining.

I also notice 4,346 per second packets being received by the server. 200 for my messages, and I assume the rest are part of the “reliable guarantee” protocol they are doing for UDP where the clients need to confirm they received things ok. Oh, and UNet’s keep alives too.

In the screenshot you can see an error appearing of “An abnormal situation has occurred: the PlayerLoop internal function has been called recursively… contact customer support”. Uhh… I don’t know what that’s about. Did I screw up something with threads? It didn’t happen on the server, but one of the four client instances.

I let it run at this rate for a while – a few clients were dropped by the server for no reason I could see. Not entirely stable at these numbers I guess, but possible it would be if I ran it on multiple computers, it’s a lot of data and ports.

TEST 2: 100 CLIENTS STRESS (REMOTE SERVER, HEADLESS LINUX)

As nice as it is to be able to turn on both the server (localhost) and the client directly from the same app in the Unity editor to test stuff, eventually you’ll probably want to get serious and host the server on linux in a data center somewhere. This test let’s us do that.

I’ve changed the destination address to the remote IP. (You can click the GameLogic object to edit them, I was too lazy to make a real GUI way))

I have a .bat file that builds the Unity project (on Windows) and copies it to the remote server, then triggers a restart. It’s run on Linux like this:

Linux NetTest: (using telnet) it’s reporting it’s sending 1.22 MB per second (only 3,458 packets per second, yeah, definitely some packet merging happening) The total stats went negative, I ran this a while, uh… I guess I need to be using larger numbers, probably rolled over.

Linux: Server is running at 1100 FPS, so zero problems with speed. You’re probably thinking “Hey, it’s headless, what do you mean frames per second? What frames?”, ok, fine, it’s really measuring “update()’s per second”, but you know what I mean. Slowest frame out of the 90 second sample time was 0.0137 (13 ms)

I don’t have it in the screenshot, but top shows the server is not too power hungry, its process seems to use 3% (no load) to 15% (spamming tons of packets). Keep in mind we aren’t actually DOING anything game related yet, but for just the networking, that isn’t bad

I tried 200X200 (4x the total bandwidth & packets, so 5 MB a second) but the outgoing packet queue started to fill up and I started dropping connections.

TEST 3: WebGL (REMOTE SERVER, HEADLESS LINUX)

I’m not going to bother making pics and somehow this post already got way out of control, but here is what seemed to be the case when I played with this earlier:

No WebSocket stats are included in the stuff UNet reports – “the function called has not been supported for web sockets communication” when using things like NetworkTransport.GetOutgoingFullBytesCountForHost on it

WebGL/WebSockets be slow. Real slow. If pushed the bandwidth to over 18KB per second (per Chrome tab), packets started to get backed up. I don’t know if things are getting throttled by the browser or what. The server eventually freaks out when the send queue gets too big and starts throwing resource errors.

I suspect there could be issues with a naughty WebSocket filling up the queue and not sending acknowledges of receives, your game probably needs to carefully monitor packets sent/received to disconnect people quick if they abuse the connection

Mixing WebSocket and normal Sockets (UDP) clients work fine, that’s a very handy thing to be able to do

Final thoughts

All in all, I think directly using LLAPI in Unity 2017.1+ is promising. Previously, I assumed I’d need to write my own C++ server to get decent socket performance (that’s how I did the Unity multiplayer space taxi demo) but now I don’t think so, if you’re careful about C#’s memory stuff.

Issues/comments:

How is it ok that Unity’s text widget/scroller can only show like 100 lines of text? (65,535 vert buffer errors if you have too much text, even though most of it is off screen) Maybe I’m doing something wrong

Unity’s linux builds don’t seem to cleanly shutdown if you do a pkill/sigterm. OnApplicationQuit() does not get run, this is very bad for linux servers that could restart processes for reboots or upgrades. (if it’s a dedicated server you can work around it, but still!)

WebGL builds can’t do GetHostAddresses() so I had to type the real IP in for it to connect (I should try switching to the experimental 4.6 net libs)

To my great surprise, my telnet host contined to run and send answers even after stopping the game in the editor. How could that game object still exist?! Threads and references are confusing, I fixed it by properly closing the telnet host threads on exit.

The Unity linux headless build worked perfectly on my CentOS 7.4 dedicated server, didn’t have to change a single thing

It’s weird that you’ll get CRC errors if PingTimeout/AcksType/DisconnectTimeout don’t perfectly match on the client and server. I wonder why there is not option to just let the server set it on the client during the initial handshake as you want to be able to tweak those things easily server-side. I guess the client could grab that data via HTTP or something before the real connect, but meh.

NetTest download

Note: I left in my .bat files (don’t mock the lowly batch file!), thought they may be useful to somebody. They allow you to build everything for linux/win/webl, copy to the remote servers and restart the remote server with a single click. Nothing will work without tweaking (they assume you have ssh, rsync etc working from a dos prompt) but you could probably figure it out.

Source is released under the “do whatever, but be cool and give me an attribution if you use it in something big” license.