Category Archives: Unit Testing

So, I’m kind of wondering if something like this exists. I currently have a set of tests that share common set up code which I only want to execute once for all of the tests, as well as some set up code that I want executed before each test. In addition, though, there are sets of tests that share test set up that I want to execute only once for that set of tests. Basically here’s what I want to happen

Parent Fixture Set Up
Child Fixture Set Up
Parent Set Up
Child Set Up
Test
Child Tear Down
Parent Tear Down
Parent Set Up
Child Set Up
Test
Child Tear Down
Parent Tear Down
(Etc…)
Child Fixture Tear Down
Child Fixture Set Up
(Etc…)
Child Fixture Tear Down
Parent Fixture Tear Town

There’s two ways I can see of doing this. One: through inheritance. Basically, the parent / child relationship is expressed through a base / derived relationship. NUnit may support this, but I haven’t tried it. I’m not really sure what happens when you provide two SetUp attributes for one class to NUnit (even if they are a base / derived relationship)

Second, and this is the method I would like to use, you could do it through a containment system. The parent contains the child, like so:

The only problem there is that in order to access, say, private data or methods of the parent class, you need to pass the parent class in as a parameter to the child. I’m pretty sure this is not supported by NUnit.

Does anyone know of an addin that might do this? Has anyone had a similar use case for any unit testing environment? How do you get around it?

So, I never really worked in an environment where unit testing was the norm. I’ve been trying to pick it up on my own for a while now and, in some respects, I think I’m doing a good job. I get the basic idea, but I’m not in tune will all of the tools and libraries that are actually available and heavily used out there in the real XP world.

So today I sort of stumbled upon and started looking at mock object libraries, mostly in Java (since I have a feeling that it’s the Java community that has the most robust and mature unit testing libraries), and I was impressed by the whole idea. It makes sense, so much so that I “implemented” it to test my generated code. But what I do a lot with a lot of custom code, logging and XML comparisons, they do with these mock objects, and they don’t need external files to do it. It’s all really, really cool.

So cool that, for a while today, I was thinking about replacing the XML logging system in my generated code tests with mock objects. But I still have an interesting problem. If I use mock objects, I can make the test code self contained, meaning that the generated tests can test the generated code and the pass / fail result tells me whether my generated code worked. But, this does not tell me whether or not my generated tests are testing everything I want them to, which results in a “testing the tests” problem. I’m not sure how to solve that one yet. For now I may just have to rely on my tests as they stand and try to figure out some way to make sure that I have at least key features tested in the unit tests.

In addition, to this, though, I was researching unit testing against databases (another thing I have to do frequently). Here, I got confused about best practices. Should I create a mock object for the database connection and test against that, or should I use something like DbUnitto connect to an actual database? My compromise, I think, is to do both. Test against the mock object to make sure that, from a unit perspective, the base library works and calls what I expect it to on the database. Then, use DbUnit to test against the actual target database platform (or platforms) to make sure that things get to where they’re supposed to go. This, of course, is a functional, not unit, test, but just as (if not more) necessary, since it actually tests what would happen in real world integration.

So, today just made me realize I have a lot to learn about unit testing. Any suggestions on where I can learn more?

So, I’ve started development on multiple virtual servers and I’ve found that VMWare actually makes things easier than I thought it would have, at least for the structure I discussed at the end of the last post. Some things I’ve found that make this setup a bit easier:

First, you can set VMWare to setup all virtual servers to be on their own virtual network shared by the host machine. This allows you to name all the servers by their use and not have them conflict between developer’s machines. So I can have a virtual machine named JBoss on my network and you can be developing against the “same” virtual machine named Jboss, but there won’t be a naming collision because both machines are on separate virtual intranets, hidden by our actual boxes. This is extremely useful, and because it’s NAT, your virtual machines can still access the internet. Double bonus.

Second, I’ve found that having VMWare split the disk into multiple 2 gig files and not having it allocate its entire disk space at once is a very good thing. Why? Because copying up a full disk (even a small disk) to a network is a slow process, and even though you suffer a performance drop on the actual server, it’s worth it in the long run in developer hours.

Third, although I haven’t implemented this yet, I would recommend setting up the “basic box” and then having a source control repository for anything anyone might need to change in terms of configuration files server deployments or source code locations. You may need to supply some sort or name translation system if you’re not running DNS on your local network (which, yeah, I’m not…) but having developers copy disk drive around constantly would probably become a huge pain. Better to only need to move drives around when something major on the box changes, like if you upgrade the OS, or upgrade certain specific pieces of software.

Lastly, don’t try to dynamically start / stop servers in your unit tests or even before your unit tests. It takes way too long, and even after the server is powered up, there’s no way for you to tell when it’s fully booted. Better to just leave it running on your build /test machine and have developers start it when they need it. That is, of course, assuming that your build machine is up to the task (ours currently isn’t, but I’m fixing that…)

Even with that done, there’s still a lot more to figure out here. I’m still wondering if this is the best way to go about this. I know most cross platform development, you just re-compile and re-run your tests either on a virtual machine or an actual machine of that platform and you’re done. But having one library that needs to be able to connect to multiple different types of servers in many different combinations, and unit testing all of them… I’m not sure how often this is a problem. Usually, I think (especially in middleware) some sort of architecture is just assumed, or you force the issue. Here, I’m trying to be as flexible as possible, even if it means having my build server have 4 different virtual servers, just to make sure all possible combinations of integration work.

The next big task at Orbus is to get our systems working with what I call “intermediary” servers: enterprise style message queues and things like that. Basically, they idea is that the game can sit and asynchronously pump metrics to the queue without sacrificing performance in the game, and the queue will catch up during times of low server load. These systems are also easy to run in parallel and in high availability modes, so if one messaging server does get overloaded (or dies), another is there to catch the slack. Right now, we're looking at JBoss, since it supports all of that in an LGPL license, which is nice.

Anyway, it’s time for me to install this server and start writing software against it. Of course, I’m planning on unit testing connections to it and making sure that everything actually works the way it’s supposed to. However, to do that, I’ll need to start up a JBoss server on my machine. This is on top of the three other servers currently running simply as multiplatform test beds. Additionally, our build / database machine is also starting to get overloaded with servers. IIS, CCNet, MySQL, MSSQL, soon JBoss, maybe eventually Postgres, MSMQ, and a small possibility of other Java Application Servers thrown in for good measure.

It’s starting to get ridiculous. Although the machines can handle it, I’d much rather see a situation where this wasn’t necessary, especially when it comes to my own development machine. But, it’s almost necessary to have each of those servers installed locally to develop against, as well as have another set up on the build machine to run the pre-deploy tests against. And, all of these systems have to be in sync. All of the developer’s databases need to match the database I developed against, and the test servers need to match mine before the tests are run. This should be done automatically, but for the time being it’s not (I’ll get around to it!).

So anyway, it’s a big fiasco. The way I see it, a perfect development / test server environment would have the following properties:

You have copies of all the local servers installed, but they are only running when you’re developing against them. Otherwise they’re disabled.

The build / test server can either have all servers running and installed, so long as they don’t conflict with one another.

All of these servers need the same names, usernames, and passwords, or be able to generate similar ones in the unit tests.

I’ve been thinking about virtualization as a possible way to overcome this problem. Basically, have each of the server collections running in a separate virtual server. So, have my database servers running on a virtual server on my box that I can shut off when I don’t need it. Then have JBoss running on a separate virtual server, and MSMQ on another, and have them named along the lines of “hostname_dbs”, and “hostname_jboss”. The idea here is that the names of the servers could be generated in the unit test, and the server images copied (with the host name changed) to all the developers pretty easily.

For now, I'll probably just install JBoss this way and see how it works. I'll let you know.

Shortly after posting this post about using dlls to run generated code to make sure it’s actually generated correctly, I realized that, unfortunately, that approach doesn’t work 90% of the time, and actually doesn’t work 100% of the time if you’re using C++, unless you find a work around to import objects. If you’re using STL (especially as parameters to functions) though, you’re kind of screwed when trying to implement your unit tests in C#.

So what to do? Again we have the same problems, but I’ve come up with a different approach. In the case of this generated code I can deem it “correct” if it conforms to the following properties:

It calls the correct functions in an external library with the correct parameters.

It puts data received from the external library into the correct place

I decided to test these two different ways, but that will show me not only that my generator created the correct code, but that it also created the correct unit tests for that code. Basically here was my end solution.

Code a Mock Object To Impersonate the External Library. In my case, the mock object is a mock connection to a database that always returns one row of data with fixed values. The fixed values are dependent on type, which makes it easier to track down if I’ve accidentally put the wrong data in an address. In addition, it logs ALL of the functions called to it, which I store for later use.

Generate Unit Tests using the same generator as the code generator. The unit tests initialize the objects using the mock connection. The objects then do what they’re supposed to do and then check to make sure they get the right data back. At the end of the test, the test outputs the calls to the mock connection object to XML (something coded into the mock object) to a predefined folder.

Code the C# Unit Tests. In this case, the unit tests generate code based off of a known value which has all of the features our generator supports. It then builds the generated code and its unit tests and runs them (actually as part of the post build step). If the unit tests fail, there’s something wrong in either the generated code or the generated tests. If they succeed, I go the extra mile to compare the output XML to files I’ve written (or at least verified) using Microsoft’s XMLDiffPatch library. This library was *also* how I checked that my Mock Object was outputing the XML correctly.

So why XML over just doing a diff on the code files? Again, all I care about are the calls to the external library. So long as they happen, I don’t care what the code internally looks like, and I actually may want (or need) to change the code to optimize things further on down the line, so comparisons against the code text is our of the questions. XML is nice because you can set the diff program to ignore whitespace and order, which I need.

So yeah, hopefully you find that helpful. By the way, I promise to go back to writing about design sometime in the future. I’m just finding a lot of this code stuff really exciting (and I can talk about it now, which is nice) so I figured… why not? Hopefully people are finding this interesting? I'd love to hear if anyone has any feedback on the shift in focus over the past few weeks.

Edit: This post was superseded by this one. Basically, I realized that this approach for C++ is way to hard and not worth the effort. So, if you're interested in testing generated code, especially in C++, I'd go read the other article. However, if you're curious as to how to build, load and test a C .dll, this would be the article for you.

So, as you may have noticed, I’m enamored with the concept of automated testing. In my opinion, it’s really the only way you can actually be sure your code is working as intended. You can’t rely on you code reading skills, and even if could, things that “makes sense” don’t always work. Now, I know unit testing doesn’t actually prove your code works (though there are ways to do that as well), but it does let you know that it works in at least the cases you’ve tested (in perpetuity if you run the tests after every build).

Recently, I ran into an interesting problem concerning unit testing . For our metrics suite, Orbus is creating a code generator, and supplying templates for various languages. The problem was how do you write a unit test (or functional test in this case) for a code generator? In my mind there are only a few options:

Take a test case and write out the code you want the generator to generate. Compare the generated code to the known value with a diff utility.

Run the generator once, and take this as the known value. Compare the generated code to the known value, again with a diff utility.

In a functional test, run the generator, compile the resulting code, then run it with various inputs, testing its outputs (much like a standard unit test).

In my mind, the first and second options are an exercise in futility. In reality, you don’t care about the text in the functions, you care about the fact that the generator produced functions that take certain parameters, and produce certain output. By trying to compare the text, you don’t actually get one of the main benefits of automated testing: discovering whether changes you made internal to the code (via refactoring or by adding additional functionality) have broken it. In this case, you want to be able to change the code generated inside the functions whenever you want, so long as the signatures remain the same and so long as the produce they proper output to external libraries. Basically, you want to treat your generated code as a black box.

The problem with the third option is that it’s really hard to do for some languages. Scripting languages, it’s pretty easy right? Just load up your trusty interpreter and feed it a few scripts that utilize your library. .NET languages have it pretty easy as well, since most of them have CodeDom objects that allow you to compile assemblies on the fly and load them. .NET’s reflection also allows you to check the number and types of parameters you’re loading, so no problems there. In reality we really only have a problem with native, compiled languages. For me, this was specifically a problem with native C++. Thankfully, I’ve actually “solved” the problem (at least for a C interface) in C# unit testing. Here are the steps our functional tests go through to test our generated code:

Generate the code to a temporary directory and build it into a .dll. The .dll I built exposes all of the API functions that someone could use from the generated code. This isn’t hard to do using preprocessor macros and the like, and we need to do it anyway so that potential clients can use our API in a .dll if they so choose. To aid in the building process, the generator also builds a .vcproj file (it’s just XML) for all of the generated files and then uses devenv /build to build it.

Load the library and do any initialization. Using LoadLibrary, you can get a handle to a loaded .dll pretty easily. Initialization is where things get hairy. In our API, we need to supply a connection object to a (static) initialization function. This object is really what generates the “output” of our library, so I wanted the connection object to be accessible to our C# unit tests. To do this, I created two mock objects in C++/CLI, one managed connection object and one unmanaged connection object that contained an auto_gcroot structure to the managed connection. I then pass the native class to the initialization routine, like so:

Now all calls into my libraries will use that connection object (it’s a singleton) so it’s a pretty easy way for me to check that everything is running nicely.

Use P/Invoke to call out to your system. Now, you could write typedefs out for all the functions you’ve generated and call them in C++/CLI using the same “GetProcAddress” from above, but since I want to do the actual tests in C#, it makes more sense to use the DllImport attribute from C# to import the methods you’re looking for. Furthermore, P/Invoke will do automatic marshaling of most of your types, and will automatically do name matching for you. In general, it's just an easier interface to use over GetProcAddress. Here’s an example:

This tests that my event generated method (LogEvent) told the connection object to execute the “LogEvent” procedure with the two parameters I supplied.

Now this may seem like a lot of work, but it’s worth it if you fell that you’re either going to be changing your generators a lot (or if you want the peace of mind of having these tests around) or if (like me) you’re going to be generating for a lot of different languages and platforms that can all compile to .dlls. With this library, I should be able to test any language that compiles to a native .dll no problem (so long as it exposes a C-like interface and doesn’t use any structures….)

Next, I’m going to work on grabbing C++ objects from the .dll instead of just a straight C interface and testing those. This stands to be a much more daunting task, since, unlike P/Invoke, I can’t rely on auto marshaling, and I can’t actually code the structures if I want them to be reusable. I’ll write up what I find.

Edit: The XSLT file works now (so far as I can tell) and has been updated on my site. The newest version uses the XSLT standard library to do some string manipulation to output a truncated name of the test that actually failed. That library, with instructions on how to install it, is available here. Note, my XSLT file assumes that at least the string.xsl file is in the same directory, however, you can edit it to point to the online version or a standard location if you want.

A little over a year ago, I attended a talk at GDC about unit testing given by the Games From Within guys. Now, I’d had training in unit testing before, but I hadn’t really been that interested in it until this particular talk at GDC. After attending that talk, I started doing unit testing on all of my personal projects, and (as anyone at my previous job could tell you at this point) I became totally convinced that unit testing was an absolute necessity in modern application programming, even in games. Since I was doing my development mostly in C++, I started using UnitTest++ for most everything I was developing. There were a lot of things I liked about UnitTest++:

It’s extremely lightweight. I can put both a debug and release library for it into source control for every project that uses it without feeling too bad about it.

It’s designed to be run after every build and integrate into Visual Studio’s error window, which is extremely helpful.

It’s very easy to debug. You just set your test project as the startup project and you’re good to go.

Now that I’ve gone back to doing development in other languages, I’m trying to force the UnitTest++ workflow onto other unit testing tools. So far, this means working with NUnit, the defacto standard for .NET unit testing. Thankfully, I’ve actually gotten NUnit’s workflow to match the one in UnitTest++, with the exception of being able to put NUnit into source control. For those of you that might be fighting similar fights, here’s what I did:

Create an XSLT for NUnit that will output nit test errors in Visual Studio Error Window readable form. I have such an XSLT file here. Note that this XSLT file may not be perfect, but hey, it’s about an hour’s worth of work.

Set up nunit-console to run after every build. In the properties for your test project, go into Build Events and in the post-build event command line window, add the command:

Note that your path to nunit may be different than mine. Also you may want to set the “Run the post-build event” option to “When the build updates the project output” so it doesn’t attempt to run even when nothing’s changed.

Setup NUnit for debugging. Again in the properties for your test project, go into the Debug pane. Select your start action to be “Start external program” and point it to nunit-console (you can use the NUnit gui here, I just don’t happen to like it). Under “Command line arguments” add the line:

/nologo YourAssembly.dll

Note, Visual Studio Macros don’t work in the command line arguments section for some unknown reason, so as much as you would like to, you can’t use $(TargetPath) like you normally would.

Now you can set your test project to be the startup project and test as normal, and have your tests run after every build, just like if you were using UnitTest++.

Who Is Jeff?

Hi, I'm Jeff, Programmer at WillowTree Apps, and hobbiest game programmer living in Charlottesville, VA. Previously, I worked for indie game company Fire Hose Games, game middleware start-up Orbus Gameworks and AAA game studio Bethesda Game Studios

This blog is where I ramble about most everything game related, including programming, tools, game design, and interactive narrative