Unreal has had static code analysis passes not too long ago. It’s surprising that SA didn’t identify this issue. It’s also unlikely that programmer code reviews would have detected something as tiny as a missing ampersand.

Just another reminder of how difficult it is to identify performance problems.

It was the summer before 4th grade. Jimmy Carter was president. Apple Computer had just incorporated, and the Atari 2600 video game console had been released, but personal computers were still a novelty. I wouldn’t use a PC for many more years.

I was a voracious reader who loved anything involving knights or pirates. I spent hours building castles and sailing ships with Legos. I watched Star Trek episodes, of course, but science fiction wasn’t really that compelling. The world of Star Trek was too obviously fake.

We were visiting my aunt and uncle in Eugene, Oregon. On the 4th of July, we all went to the movie theater for a matinee. It was a new movie everybody was talking about. The theater was packed. We split up to find seats. I sat next to my mom on the left hand side near the front.

Ten words in blue appeared: “A long time ago in a galaxy far, far away….”

Then that opening brass fanfare and the now famous title crawl began. I was transfixed. Mesmerized. Spellbound.

In those five minutes, my life changed. I had never seen anything like this. Even watching that opening sequence 38 years later, I’m still hypnotized. I challenge you to find a better movie opening.

To a nine year old in 1977, the world of Star Wars was absolutely real. Watch the droids in that opening scene on the Rebel Cruiser. C3PO’s shiny skin is scuffed and marred. R2D2 is downright dirty. All those little details convinced me — and millions of others — that Luke and Leia and Ben and Han actually existed in some galaxy far away.

That summer, I started reading science fiction. I started building X-Wings and Star Destroyers and TIE Fighters. My bedroom was plastered with photos and drawings and newspaper clippings of anything and everything Star Wars. And when we finally got our first PC, I started programming (text-based) Star Wars simulations.

Which ultimately led me to programming jobs, then video game jobs, then Xbox, and now virtual reality.

George Lucas gets a bad rap these days, but I for one thank him and the entire original Star Wars production team. You transported this fan into a life and career he never would have foreseen on July 3, 1977.

One of the challenges in game development is pre-computing static lighting. Emulating photons bouncing around in a virtual environment can require legions of compute cycles. This process is called baking. The end result is a lightmap, a compact data structure that represents the way static objects are lit.

The Unreal Engine suite provides software that makes baking lightmaps easy and fast. There are three components involved:

Unreal Lightmass: the app that does the actual lighting computations. It’s designed to run on many different PCs simultaneously.

Swarm Agent: a background app that runs on any PC that has been designated to bake lighting. The Swarm Agent invokes Lightmass to do bakes.

Swarm Coordinator: an app that runs on a single always-on PC that receives requests for lighting builds and determines which Swarm Agents are available to participate in the bake.

To build lighting, anybody running the Unreal Editor can kick off a bake by selecting Build Lighting in the editor. This results in a number of Swarm Agents across the local network participating in the build. The number is a function of the size and quality of the bake, and also depends on which Agents have free CPU cycles. Each selected Swarm Agents receives Lightmass along with enough data to run its portion of the bake. The combined results are ultimately assembled back into a complete lightmap.

By running Swarm Agents on many computers on your local network, lighting builds can be performed very quickly. What might take hours on a single machine takes only a few minutes with Swarm.

The documentation on Swarm and Lightmass is sparse. The best reference is a single Unreal AnswerHub post.

Where I work at HBO, only a subset of employees use Unreal. However, we want every machine on our network to participate in lighting builds — without the hassle of installing Unreal. That goal turned into an interesting adventure. Although Epic recommends copying the Engine\Binaries\DotNET folder to install Swarm, that alone is insufficient.

Without rehashing what the existing documentation already covers, I’m including everything I’ve learned about Swarm as of Unreal version 4.9. This information is useful to anybody setting up Swarm on your network or wanting to understand how to best configure Swarm.

Swarm Coordinator

The Coordinator keeps track of all of the Swarm Agents on the network and the status of each. You can also use the Coordinator to restart Swarm Agent instances.

SwarmCoordinator.exe is designed to run on a single machine on your network. We run it on a low-end always-on Windows box. We also run Swarm Agent on the same machine.

Swarm Coordinator does not require that Unreal Engine be installed. I didn’t do the original setup of the Coordinator on our network, but based on empirical evidence, it requires the same files as the Swarm Agent (see below), plus the SwarmCoordinator.exe.config file.

We place a shortcut to SwarmCoordinator.exe in the Windows Startup folder so that the Coordinator always begins at system start.

In theory you could have multiple Coordinators on the network, each managing an independent set of machines/Agents. We haven’t tested this theory.

Unreal Lightmass

Lightmass is the workhorse application that bakes lighting. It is automatically distributed to Swarm Agents on a per bake basis.

Do not install UnrealLightmass.exe directly; it’s automatically copied and run whenever a bake is kicked off.

Because Lightmass is completely independent of Swarm, Swarm Agents can run different versions of Lightmass. That means you can have teams running different versions of Unreal and using different versions of Swarm Agents to bake lighting. Nice.

Lightmass will utilize multiple CPU cores (you can configure how many) and peg the CPU on most machines. However, because (by default) it is run at below normal priority, it doesn’t normally impact the productivity of anybody actually using the machine.

Lightmass is RAM intensive. It is not uncommon for it to use 1GB or more RAM depending on the bake size/quality.

Lightmass does not utilize the GPU (as of 4.9).

Depending on the size/quality of your bakes, Lightmass can require significant hard drive space. For instance, for our current small project, Lightmass uses many hundreds of MB per lighting build. See notes on configuring the Swarm cache for more details.

Lightmass source code is located at Engine\Source\Editor\UnrealEd\Private\Lightmass and Engine\Source\Programs\UnrealLightmass

The version of UnrealLightmass.exe that an Agent receives is dependent on processor architecture. There are 32 and 64-bit versions. Epic recommends only using Swarm/Lightmass on 64-bit systems for best performance, and we follow this advice.

Lightmass requires the following be installed in order to run. If you’re installing Swarm Agents on machines, you’ll need to be sure these components are also installed.

Swarm Agent is designed to run on multiple machines across your network, building lighting on demand.

SwarmAgent.exe runs on Windows XP and up. It does not run on MacOS or Linux (as of 4.9).

Swarm is built with C# using the .NET 4.0 framework

SwarmAgent.exe and its associated DLLs are 32-bit images

We place a shortcut to SwarmAgent.exe in the Windows Startup folder so that the Agent always begins at system start.

Swarm can technically run a variety of distributed tasks, but as of 4.9 it only runs lighting calculations by distributing and spawning Lightmass

There are two types of logical Swarm Agents. The same SwarmAgent.exe client supports both modes.

The Local agent is the agent that kicked off the lighting bake

Remote agents running on other network PCs are assigned to participate in a given lighting bake

After initial communication with the Coordinator, Swarm Agents appear to perform all Agent-to-Agent communication in a peer-to-peer fashion.

The Local agent distributes UnrealLightmass.exe and lighting information to each assigned Remote agent. The Local agent also participates in the bake (unless otherwise configured), collects the work from remote agents and merges the results.

Swarm will not execute bakes on machines where it detects that the CPU usage has been, on average, above 20% for the past 10 seconds (see CPUBusyThreshold in Agent.cs)

Swarm spawns the Lightmass process at BELOW_NORMAL_PRIORITY_CLASS on the Local agent and IDLE_PRIORITY_CLASS on Remote agents to reduce the CPU impact on busy machines. These values are configurable.

Swarm will not necessarily use all available non-busy agents. The number of agents appears to depend on the size and complexity of the lighting bake.

Swarm does not require that Unreal Engine be installed.

Swarm requires the following files be present in the same folder as SwarmAgent.exe. These files are found in the Engine/Binaries/DotNET folder.

Although Swarm itself doesn’t require DirectX v9 extensions or the C runtime, Lightmass does need these, so if you’re installing Swarm, be sure to install those components, too.

Swarm Agent runs as a Notification (or System Tray) app. It is designed to run continuously in the background.

To view the Swarm Agent interface, locate the app in the system tray (yellow and black S icon) and double click the icon.

SwarmAgent.exe does not respond to WM_CLOSE. To close it manually, you must choose File->Exit from the main menu. To close it programatically, you can run taskkill.exe SwarmAgent.exe.

Empirically (using Microsoft Message Analyzer) SwarmAgent and SwarmCoordinator use the following protocols and ports:

ICMP protocol for ping requests between agents

TCP protocol for all other communication, using ports: 8008, 8009, 54430, 56574, 56587 and 56589

Some ports are configurable. For instance, SwarmCoordinator.exe.config allow you to remap port 8009.

Swarm will not activate on sleeping PCs. I recommend that if you always want your Swarm Agents to be available, those PCs should not enter power-saving sleep states. Alternatively, you can use Windows Task Scheduler to wake machines at the appropriate times to run your bakes. See powercfg command line tool for details. We’ve found that not all machines have BIOS configurations that support powercfg, so your mileage may vary.

Until we explored the issue in detail, the most common Swarm problem was blocked communication between different Swarm agents. Firewalls are the culprits. Windows Firewall will block Swarm by default, even if the user chooses to allow Swarm to do outbound communications. Because Swarm communicates in a peer-to-peer fashion, it must also be configured to allow inbound communication from other agents.

Programmatically, the Windows Firewall can be configured to permit Swarm usage with the following commands. We also do IP filtering to make sure these firewall exceptions only happen on our local network — replace x.y with your actual leading IP octets.

This diagram shows a scenario where the PC in the bottom center has kicked off a Unreal lighting bake. Three PCs have Unreal installed, but all the PCs are running Swarm Agents. The top center PC is running the Swarm Coordinator. Five PCs were chosen to participate in the bake. The other PC was busy (compiling code, etc.). Communication between the local agent and the remote agents, represented by the arrows, happens in a peer-to-peer fashion.

Swarm Parameters

Many elements of how Swarm operates can be modified. The Settings tab in Swarm Agent allows you to change settings. Configuration information is stored in SwarmAgent.Options.xml and SwarmCoordinator.exe.config. Here are some of the most interesting elements, in order of importance.

CoordinatorRemotingHost: must match the Swarm Coordinator Computer name from Control Panel->System

CacheFolder: where Swarm caches job information. At install time, we set this to the equivalent of %temp%/SwarmCache

MaximumCacheSize: the maximum amount of HDD space that Swarm will consume in GB. We use 10 (the default).

MaximumJobsToKeep: the maximum number of cached jobs (and logs) to keep around. We use 5 (the default), but it’s extremely rare that we need to look at logs of old jobs, so I think 1 or 2 would be fine.

AllowedRemoteAgentNames: allows you to select a subset of Remote agents that you want to use. We always want to use all available, so we set this to *

AllowedRemoteAgentGroup: allows different PCs to participate in different Swarm groups. We currently have a single group with our team name.

AgentGroupName: this is the name of the group that the local agent belongs to. We use our team name.

LocalJobsDefaultProcessorCount and RemoteJobsDefaultProcessorCount (developer settings): the number of CPU cores used by Lightmass for local and remote bakes. These are set automatically to reasonable values, but can be reduced to lessen CPU impact at the cost of bake speed.

Swarm Installer

To make life easier for our extended team, I wrote a Windows installer that handles the complexity described above. The installer uses the WiX Toolset to generate a standard Windows MSI file. Notes about the installer:

The installer is a 64-bit package. This addresses: 1) Unreal’s recommendation that Lightmass run on 64-bit systems only, 2) a limitation of WiX that doesn’t allow 32-bit packages to include/install 64-bit binaries, plus 3) the fact that everybody in our office is running 64-bit Windows anyway🙂

The following DirectX v9 files must be installed: dxsetup.exe, dxupdate.cab, dsetup.dll, dsetup32.dll, Jun2010_d3dx9_43_x64.cab and Apr2007_XInput_x64.cab. You can grab these files from the DirectX SDK. The WiX toolset page has good instructions for installing DX extensions.

In addition to the normal installer-y stuff (copying files and updating the registry), the installer:

Without you having to do any extra work, the Windows Uninstaller automatically takes care of deleting any folders where it installed content. However, if an app writes to other folders that the installer doesn’t know about, it’s your responsibility to handle the removal of those folders.

In my case, the app that I was installing writes temporary files to a cache folder. The app could cache gigabytes of data, so it was key that the uninstall remove all those temporary files. Fortunately, WiX makes this easy. Here are the steps. First, we search the registry for the location of the cache folder and store it in a Property. We’ll use this if we’re uninstalling. Replace CompanyName and AppName with your company and app names.

The app I was installing communicates with other apps on our internal network, and it requires certain network ports be open for ICMP and TCP communication.

My first thought on solving this issue was to figure out where Windows Firewall Manager stores its firewall settings and then tell the firewall to read the new settings. Indeed, I found the location in the registry and I was able to successfully create the new settings. I was also able to kick the firewall using the commands “net stop “windows firewall”” followed immediately by “net start “windows firewall””. Everything seemed great.

But there were issues. For instance, during install, a little message showed up in the taskbar that the firewall was stopping. That’s confusing to the end user, especially because there’s no message that shows up saying the firewall is starting back up. Modifying the registry directly felt dirty. This was not a maintainable solution. The whole approach was a hack.

So I found a better way: the “netsh advfirewall” command. It’s available on Windows 7 and up.

Say you wanted to open ports 8008 and 8009 for inbound TCP traffic on the local network. Here’s the full command to add that rule:

Even better, the firewall automatically starts applying this rule without a restart. There are many other advfirewall parameters, but you can explore those on your own.

Running these commands in the context of the installer was a little more challenging. I needed to add multiple rules, and I needed to handle removing the rules on uninstall. Let’s start with the variables I used to simplify things:

The Quote variable handles the cases where we need to embed quotes in the command string. The IpRange variable locks this rule down to a specific set of IPs. Fill in your own values for w.x.y.z. Like many other advfirewall commands, it’s optional. FirewallAddRule has the meat of the command string, and then there are individual suffixes for the ICMP and TCP rules. The ICMP rule doesn’t accept ports. The FirewallRemoveRule ensures that all rules matching the given name will be removed on uninstall.

There are four custom actions that execute in sequence after the InstallFinalize stage. The first action establishes the command line for adding the ICMP rule using the variables defined above. CAQuietExec requires that the actual app command (netsh in this case) be quoted. The second action executes the command line established in the first command. Using multiple CAQuietExecs requires configuring the command line as a property with a matching ID in the following action. Changing firewall settings requires Windows elevated privilege, so we use deferred execution with no impersonation. The final two commands set the TCP rule in the same manner.

The funky notation ![CDATA[NOT REMOVE]] tells the installer that these commands should run whenever we’re not actually uninstalling the product (e.g. installs, reinstalls, upgrades, repairs).

We set up the command line to remove the firewall settings, then invoke that command line with elevated privileges. We only remove the firewall settings when the product is being uninstalled (not installed, reinstalled, or upgraded)

One additional note. I really wanted to lock down the firewall rule to a specific app, so I was excited that advfirewall has just the option I was looking for:

allowedprogram program=C:\MyApp\MyApp.exe name="My Application"

But unfortunately in my particular case the app I was installing is not actually the app that does all the LAN communication. My app spawns other apps that do the communication, and the locations and names of those apps is not necessarily known at install time. Hence we chose to lock down the firewall rules to specific (small) range of IPs, so that if this install package gets out in the wild it’s not unnecessarily introducing security holes.

The app I was installing writes to temporary storage, and the app stores a reference to the temporary storage folder in AppConfig.xml, which is included in the install manifest. AppConfig.xml contains a folder specifier like this:

<Options>
<CacheFolder>c:\cache</CacheFolder>
</Options>

The problem I faced was that I wanted to avoid using a hardcoded “c:\cache.” Not every user has a C drive or wants a cache folder at the root. And I didn’t want the user to have to configure this manually. The installer should default to a reasonable location.

Fortunately, WiX handles this sort of thing with aplomb. You can modify the contents of XML files at install time. Perfect!

Here’s what’s happening. You define your file component as normal. After you’ve specified the file information, use the util:XmlFile specifier. XmlFile File points at the actual file by using the # specifier, which resolves to the full install path of AppConfig.xml. Change the XML value of to “%temp%cache”, which resolves to the user’s temporary folder plus “cache,” e.g. “c:\users\pete\AppData\Local\Temp\cache” — just what we want.

To make this all work correctly, there are two more steps. First, ensure your WSX source references the utility extension where util:XmlFile is located.

Recently I put together a Windows Installer package (MSI file) for our team to use internally. The package installs a system tray app (sometimes called a notification app or notification area app). One common property of system tray apps is that they don’t actually close on exit — they simply return to the system tray and continue running in the background.

In order to properly uninstall this component, the app needs to be shut down by the installer/uninstaller. Typically, in WiX, you’d do this with the CloseApplication element, which sends a WM_CLOSE message to the system tray app, which it ignores completely. What to do?

First, we define a custom action that will execute the taskkill Windows system command. We ignore the return value because we don’t care if the app was actually running. Just in case the Windows system folder has embedded spaces, we wrap the command in quotes using the XML quote specifier. The /F flag requests a forceful termination, and the /IM flag requests that we terminate the specifically named process, which is again wrapped in quotes in case it has embedded spaces.

Next, we set up the custom action to execute in the first phase of the install process. This ensures that the system tray app is always shut down when the install/uninstall begins.

Ideally, we’d use the CAQuietExec special custom action to avoid a command screen from briefly flashing on the screen, but CAQuietExec must be run after the InstallInitialize phase, by which time the Windows Installer has already detected that the system tray app is running, and it’s too late for us to do anything about it.