Has it ever occurred to you, how wonderful would it be, if you could easily test all your complex subroutines written in an assembly language and targeting 6502 CPU by the means of creating authentic unit tests? Not only to create them easily, but also to run them... Run them fast! And to run them in an isolation, independently from a compiled program's environment and its side-effects or an accidental state of a processor? How great would it be to instantly provide your test runner with different execution contexts? And then to modify your tests whenever there is a need to tweak any subroutines without a fear of breaking things or spending significant amount of time on debugging, but just re-running your tests instead?

If your answer to an above set of question is a firm yes, consider this article being addressed to you specifically. All you dreamt of has now become possible thanks to a set of tools designed to conveniently unit test your 6502 assembly code. Any code that is expected to run successfully on a real hardware may now also be executed by CommTest. CommTest is the first project of its kind, a complete unit-testing framework created to verify correctness of compiled assembly programs targetting 6502 CPU, entirely written in Scala, and built on top of a very popular unit-testing framework ScalaTest.

CommTest is based on the CPU 6502 Simulator, a complete processor simulation library, natively written in Scala, whose main goal is to provide an easy-to-use tool for execution of any 6502-compiled code in a simulation-based environment. Everything is glued together by sbt-commtest, an sbt plugin specialised in running CommTest. sbt-commtest comes up bundled with a couple of additional helper tools, which let you not only compile your code and then run your tests. They also allow you to package your programs as executables, create virtual D64 disk images, and ultimately run them directly in a VICE emulator. Following a few simple rules established by sbt-commtest, compilation, testing, packaging, and running the project boils down to typing a single-word commands in sbt's interactive mode.

The most amazing thing about CommTest is that in spite of being written in Scala, you actually do not need to know how to write any Scala code yourself in order to use it to its full potential. CommTest defines its own very descriptive and well- documented DSL. Knowing Scala may of course be considered an advantage, because you can use its features virtually anywhere in your tests, but it is absolutely not a prerequisite allowing you to benefit from all of CommTest's features.

This all is probably quite an innovative approach to writing 6502 assembly code in general, never explored in such detail before, while being openly shared with the public at a same time. It may not be perfect, but we are here to learn from each other. If you stumble upon any urgently missing features or, even worse, encounter a bug in the toolset, please do not hesitate to drop me a message, create a pull request, or whatever else you may conceive as appropriate.

CommTest is meant to serve as a complete 6502 CPU unit-testing framework. It features a state of the art approach to writing tests you are likely to know from modern programming languages. Having it built on top of a native CPU 6502 Simulator allows you to literally do anything with your assembly source code. And I do truly mean anything.

A complete source code of all these tools is freely available on GitHub for you to explore all implementation details even further.

This tutorial will guide you step by step through the process of setting up a new project, writing and then executing your unit tests. Shall we begin?

Before proceeding further: it is assumed that a set of external tools has been installed in your system. Please make sure to have the following set of prerequisites available before advancing with this tutorial:

Everything listed above must be command line accessible (i.e. reachable via the PATH environment variable). Once verified that each command above is recognised by your shell process and functions without errors, you may continue reading.

A starting point is some code you would have already written and wanting to test. For the sake of this tutorial let's create a simple subroutine that subtracts 1 from a 16-bit number. Write the following lines into a file named subtract.src:

At the same let's create main.src program file that will later help us demonstrate a few extra features provided by sbt-commtest plugin. The following lines, when executed, will initialise a value to subtract, call subtract subroutine (included from subtract.src file), and eventually write the result (lo and hi bytes) to border and background colour registers ($d020 and $d021 respectively):

A trivial example, however it will serve us well in our encounters with exploring all available features of CommTest. Once well grounded in the framework, you will be able to extend the accumulated knowledge to deal with far more complex use cases.

sbt-commtest assumes a very specific structure of project directories. Therefore we must adhere to a convention and put all source code files into src/main/ and all unit tests into src/test/. Let's create this structure and then move both files just created to account for an expected layout of directories:

We are now ready to setup our new project in sbt. First, let's create a build configuration file named build.sbt with the following content:

mainProgram := "main.src"

scalaVersion := "2.12.4"

startAddress := 0x1000

sbt-commtest is very flexbile and allows you to configure more options than just this. Note you will always have to make sure to use Scala version 2.12. The other two values are only mandatory should you want to run your compiled executable in a VICE emulator.

It is out of scope of this tutorial to discuss all build properties. You may want to explore example build.sbt file included in sbt-commtest source code. You may even want to inspect plugin implementation to peer into all customisable settings (hint: look for baseSettings value, which defines a sequence of all configurable build tasks and settings, you may customise them all the way you want by overwriting their default definitions in your project's build.sbt file).

build.properties configuration file will ensure we are using a proper version of sbt build tool. Currently sbt-commtest plugin requires sbt version 0.13. Thus create a subdirectory named project:

$ mkdir -p project/

Then put the following content into a new project/build.properties file:

sbt.version=0.13.16

The very last setup point is telling sbt to use sbt-commtest plugin. This plugin will be enabled by default when loaded. A configuration file responsible for adding new plugins to a build definition is named plugins.sbt and should be placed in a project subdirectory as well (alongside a build.properties file). The only line you need to add in this file contains the following directive:

addSbtPlugin("com.github.pawelkrol" % "sbt-commtest" % "0.03")

Once build.sbt, project/build.properties, and project/plugins.sbt files have been created, we are ready to start sbt in an interactive mode:

$ sbt

An initial setup will take a moment, because sbt needs to download all dependencies. There are not too many though, so their resolution should only take a few seconds. It will take more if you have not used Scala compiler ever before. Once an interactive shell is started, sbt-commtest makes all build operations blazingly fast.

If you are very eager to see sbt-commtest in action as quick as possible, I do not want to hold you. Go ahead and compile the project by typing:

> compile

You will see an accustomed console output from dreamass. If you peer into target subdirectory now, you will notice four new files generated there:

main.log

main.prg

subtract.log

subtract.prg

Both .log files will be written with a complementary label log of addresses/values as they were defined in a source code. And .prg files are compiled binaries. The reason two files are generated from each .src file is that CommTest requires not only a compiled binary to run your tests, it also needs to know the meaning of all labels you defined in your code, because it allows referring to them directly in tests.

I hope you can already see the value in using sbt-commtest. With only three files and five lines of configuration data you are able to build and test your entire project. This is convenience in building C64 stuff brought to another dimension. Especially given the fact that you usually configure your new projects from scratch, you do not want a tedious and complicated setup process. This is true at least for me, when I want to write some new code, I do not want to care about anything else but writing this new code. sbt-commtest makes your life easier in a sense that it takes the whole burden of setting up everything manually away and does practically all the boring job in your stead.

All the setup has now been completed. How about writing some tests? After all this is the main reason you have got yourself interested in this article in a first place, right? Let's get to it!

At a first sight these examples might seem too obvious to be a worthwhile investment in testing at all. Keep in mind, however, that this is assembly language code, and it is going to be executed on an 8-bit CPU, and then nothing is trivial anymore. Will the lo/hi bytes get correctly decremented? With little programming skills you would now be able to quickly generate and test all 2^16 input/output possibilities. Nonetheless, we focus only on three elementary edge cases mentioned above.

How does this translate into an actual code? A short introduction to a structure of Scala test files will be necessary. First let's have a look at an empty test template file:

import com.github.pawelkrol.CommTest.FunSpec

class SubtractSpec extends FunSpec {

outputPrg = "target/subtract.prg"

labelLog = "target/subtract.log"

private val address = "address"

describe("subtract") {

// ...your tests go here...
}
}

Before moving on, let's create SubtractSpec.scala file in src/test/ subdirectory and populate it with an example content of an empty template presented above.

Let's tackle each line of a program at a time. First take a look at an import statement. In Scala code everything belongs to a package. FunSpec is a name of class scoped to a com.github.pawelkrol.CommTest package. An import statement makes FunSpec available in the current package, which practically means that we won't need to prefix FunSpec with its full namespace in the currently edited file. You could of course use fully qualified names everywhere in your Scala programs, this style is however not recommended, as it renders your code less readable.

The next line marks the beginning of a class definition. Every test suite is expected to extend FunSpec class from com.github.pawelkrol.CommTest package. Each class extending FunSpec is enabled to use CommTest's specialised DSL dedicated to accessing and manipulating memory and CPU state, but more on that later. An opening and a closing brace mark the beginning and the end of a class definition. The body of a class is where all tests belong.

There are just two mandatory properties that each test class has to define:

labelLog is an output file containing a label log corresponding to your compiled program

outputPrg is an output program file in a binary format created when compiling your source code

Remember a label log and a binary file generated during compilation process earlier? This is the place where we need to explicitly refer to them. Because src/main/subtract.src compiles to target/subtract.prg, while label log is written to target/subtract.log, this is exactly how we write them in our test code:

outputPrg = "target/subtract.prg"

labelLog = "target/subtract.log"

One important note here is that this is still subject to change, so make sure to keep an eye on a documentation. Future versions of sbt-commtest plugin are going to improve this aspect of setting up your tests in a way that instead of specifying targets explicitly you will only need to indicate a source code file under test (without an explicit and so obvious distinction between label log and compiled program).

As a side-note, it is also possible (not a very convenient solution though) to provide label log in a plain text instead of reading it from an external file:

labelLog = """
address = $110c
subtract = $1100
"""

This would not make much sense in our sophisticated test framework, however I wanted to plainly mention such possibility.

address is a mere constant. We assign a string with a label name to it in order to avoid repeating it all over the place. Should our original source code change, resulting in renaming of address label to something else once we decided that address was not the most fortunate name for our subtraction subroutine's input/output parameter, we would only need to change it in one place of our test file. This also saves us from some typing, as we no longer need to wrap address in quotation marks despite the fact that we continue to refer to it as a string label (Scala will substitute address constant for its actual value).

describe is a keyword with a special meaning. It takes two arguments: a string enclosed in parentheses, followed by a block of code. A block of code is where your tests go. A string, however, is somewhat magical. Although it does not have to be. You can write anything you want in there. But if you write the name of a subroutine under test, subtract in our case, you get an extra keyword that you can use in a following block of code: call. Saying call after specifying a subroutine label as an argument to an outside describe block will execute described subroutine. This is not only convenient, but also very descriptive. You say that you describe subtract, and then when you call it, you know the name of a subroutine called from its surrounding context.

Another feature of describe keyword is that it allows an unrestricted nesting of scopes. You are free to describe one aspect of your program in terms of some other preconditions. There may be many use cases, one can imagine something like below with an arbitrary number of nested scopes:

If you prefer you may use context keyword alternately with describe, for context is just an alias of describe.

Also worth noting is that tests do not leak memory and CPU state between each other. All tests are started as a clean slate, and their contextualised preconditions will be cascaded from outer to inner blocks during initialisation. This is a CommTest feature, and it was designed this way to help with test execution in a full isolation.

Now, your tests go here. A well-written test example should be self-contained and test only a single property of your subroutine. When I say property I am not necessarily referring to a single register or a processor flag, this may as well be a combination of several physical entities, but they should represent a single logical aspect of your subroutine's execution. Just like in an example below, we validate values of two distinct memory locations, however they logically form a single 16-bit number that was decremented after calling subtraction subroutine once.

If you doubt this is all real, and not a trickery, feel free to mess up with expected and/or input values, and you will instantly see test failures. Also, please note how well it reads (almost like a natural language!): when address is $0000 subtract results in address = $ffff.

Let's dissect the code into its individual pieces...

I have already mentioned that context is just an alias for describe. That means it can be used anywhere to create any number of arbitrarily nested scopes. We take advantage of this fact by stating our intentions unequivocally. You can do well without giving additional descriptions, their only purpose is to help you (a programmer) better understand what your tests were supposed to do after coming back to them after some time.

Every single test example is wrapped by an it statement. What follows is a simple description in the parentheses, and an actual code (wrapped by curly braces) that will be executed in the context of a particular scenario.

All our tests look fairly similar, and they all follow a common pattern of execution:

Our examples accomplish an above set of tasks by performing the following methods:

1. writeWordAt(address, 0x0000)

Here we begin with writing two bytes of data representing our input argument (a 16-bit value of $0000) into a memory address identified by an address label. writeWordAt is just a convenience method letting you write two bytes into a memory at one go. You could just as well use writeBytesAt(address, 0x00, 0x00) to produce an exactly same result.

2. call

call method will execute described subroutine. This is where all magic happens, i.e. a simulator boots up and runs a subroutine described in the current context. The moment the very last RTS operation is carried out program control is returned back from a simulator to a test engine, and we are able to check if we achieved expected results.

3. assert(readBytesAt(address, 2) === Seq(0xff, 0xff))

In any Scala program you can write assertions by invoking assert method and passing in a boolean expression. This is why we take advantage of what Scala programs can already accomplish. Because assert method takes a boolean expression as its argument, our expectations need to conform to this format. A recommended style is to use an equality operator === to compare each tested parameter against its expected value.

readBytesAt may be used to access memory location at a given address (or an address identified by a source code label) and fetch an indicated number of bytes starting from that address. In our example we read two bytes starting from address and compare them against a sequence of two bytes, each valued as $ff.

This is how our first test example when address = $0000, calling subtract once results in address = $ffff is realised in a Scala code utilising CommTest framework. The other two test cases can be derived by an analogy, only using different arguments.

These are the very basic features of CommTest. Before moving on to more advanced topics, let's take a step back and discuss additional features bundled with sbt-commtest plugin. I have mentioned that it facilitates seamless packaging and running of your programs. If you created main.src program file and configured build.sbt settings as described earlier in this article, you are ready to run the following command in sbt's interactive mode:

> package

It will use exomizer cruncher to create an executable program out of a compiled main.src file (as it was defined by mainProgram setting of your build.sbt) . Then it will create a D64 disk image using cc1541 tool and write crunched executable into it. So prepared D64 file is ready to be open in a VICE emulator. Go ahead and type run now:

> run

VICE emulator will get started, and it will run your program from an automatically attached D64 disk image file.

In the end, when you are done playing with a build tool, you can delete all files produced by a build process just by typing:

> clean

sbt-commtest may serve as a good companion not only when unit-testing your code, but also when running it directly in an actual system that is much closer to your ultimate target environment, which always will be a real Commodore 64 hardware. Doing it fast and easy way is possible just by typing run in an interactive shell. It is powerful, yet so simple.

Now that you know how to compile, test, package and run your code, we can take a closer look at the remaining features of CommTest framework that you may find useful while writing your unit tests.

In the next section the following features will be reviewed:

Accessing memory

before filters

Flags and registers

Calling subroutines

Assertions vs expectations

Shared examples

Relative addressing

An easy access to a computer's memory before and after execution of your program is a premier method of ascertaining your code to behave correctly. CommTest exposes a collection of methods to read from and write to a memory. All of them may be accessed either directly (via hexadecimal address of a target location) or indirectly (via a subroutine name listed in a label log file).

readByteAt and writeByteAt methods read/write a single byte value at a provided memory address:

before filters allow you to specify a common behaviour that is shared across all of your test examples in the same scope where before definition originally appears. For instance, if your division computation's divisor is accessible via divisor label and you want to share its assignment across several test examples, you may specify a separate context and execute common assignment once inside a before filter:

context("when the divisor is zero") {

before {
writeWordAt("divisor", 0x0000)
}

// ...test examples...
}

CommTest provides a direct read/write access to all processor registers and status flags. They can be referred via the following symbolic names from your tests:

Symbol

Description

AC

Accumulator register

XR

Index register X

YR

Index register Y

PC

Program counter

SR

Status register

SP

Stack pointer

BF

Break flag

CF

Carry flag

DF

Decimal flag

IF

Interrupt flag

OF

Overflow flag

SF

Sign flag

ZF

Zero flag

Each symbol may be used both in an assignment instruction as well as a return value:

AC = 0x80
CF = false

assert(SF === false)

If you ever need to debug your code and peek into the current value of any register/flag, you may simply print them out to a console:

println(XR)

Every subroutine name defined in a label log may be called and executed directly from within your test examples. It is as simple as calling a method named call and providing a subroutine name as an argument.

This is how to execute code residing at a specified memory location and return control to your test program as soon as a subroutine called executes RTS:

call("divide")

It has already been demonstrated that you may also avoid typing subroutine name each time you want to call it and leave out an argument if you specify a label as an argument to an outside describe block. From that point on each call will execute described subroutine:

describe("divide") {

it("computes division") {

call
// ...test assertions...
}
}

Under the hood CommTest is pure Scala, so you can write any Scala code you want to edit your tests. That means anything available in a Predef package, including (but not limited to) assert function, will work. You may bring them into play when writing your test examples:

assert(readByteAt("init") === 0x4c)
assert(AC === 0x00)

But there is a better way provided by CommTest's DSL: expectations. They are designed to read better and resemble natural language more than ordinary assertions. And they give you even more flexibility. They will let you automatically verify if a target value changes in an exact way you want it. Let's illustrate that with a few examples.

Suppose you write a test where your subroutine call is expected to change the value stored in an accumulator register from $00 to $80. This is how such a subtle clause would be shaped:

expect(call).toChange(AC).from(0x00).to(0x80)

What if you expect the value of an accumulator register to be just changed to $80, but you do not know (or care) about its original value (would it be $00 or anything else does not matter)? Just leave a from component out of an expression. You can write your expectation like this:

expect(call).toChange(AC).to(0x80)

OK, so what about a test that calls a subroutine that is not supposed to alter the value stored in an accumulator register no matter what the value actually is? Easy, just use a notToChange directive:

expect(call).notToChange(AC)

Shared examples give you an opportunity to implement a DRY (Don't Repeat Yourself) principle when writing your tests. They let you define your test examples just once and execute them multiple times from different contexts by simply including those test examples any number of times:

When referring to a specific memory address via label, it is possible to apply an arbitrary offset to the target address before fetching/storing an actual value at a memory address. This is as simple as adding a chosen offset next to a label.

Given that init points to $c000, the following lines will store $00 in $c000, and $01 in $c001:

writeByteAt("init", 0x00)
writeByteAt("init + 1", 0x01)

This item concludes our checkup of all advanced features of CommTest framework. Each reviewed component was illustrated with real code examples and was intended to help you out with faster implementation of your own tests.

CommTest's DSL is very robust, unleashing its full potential will let you write powerful and reliable unit-tests, and their consequence will be better maintainable and predictable code of your programs.

If any open questions remain after studying this in-depth tutorial, do not hesitate to contact me, I will be more than happy to talk them through. Good luck with all your future endeavours and remember not to let your code linger untested!