The second and third examples
test a post office protocol (POP3) client available from
CPAN. These two unit tests for
Mail::POP3Client indicate some design issues,
which are addressed in the Refactoring chapter.
The third example also demonstrates how to use
Test::MockObject, a CPAN module that makes it
easy to test those tricky paths through the code, such as, error
cases.

Testing Isn't Hard

One of the common complaints I've heard about testing is that it is
too hard for complex APIs, and the return on investment is therefore
too low. The problem of course is the more complex the API, the more
it needs to be tested in isolation. The rest of the chapter demonstrates
a few tricks that simplify testing complex APIs. What I've found,
however, the more testing I do, the easier it is to write tests
especially for complex APIs.

Testing is also infectious. As your suite grows, there are more
examples to learn from, and the harder it becomes to
not test. Your test infrastructure also evolves
to better match the language of your APIs. Once and only once applies
to test software, too. This is how Bivio::Test
came about. We were tired of repeating ourselves.
Bivio::Test lets us write subject matter
oriented programs, even for complex APIs.

Mail::POP3Client

The POP3 protocol[2]
is a common way for mail user agents to retrieve messages from mail
servers. As is often the case, there's a CPAN module available that
implements this protocol.

Mail::POP3Client[3]
has been around for a few years. The unit test shown below was
written in the spirit of test first programming. Some of the test
cases fail, and in Refactoring, we refactor
Mail::POP3Client to make it easier to fix some
of the defects found here.

This unit test shows how to test an interface that uses sockets to
connect to a server and has APIs that write files. This test touches
on a number of test and API design issues.

To minimize page flipping the test is broken into pieces, one part per
section. The first two sections discuss initialization and data
selection. In Validate Basic Assumptions First and the
next section, we test the server capabilities and authentication
mechanisms match our assumptions. We test basic message retrieval
starting in Distinguish Error Cases Uniquely followed
by retrieving to files. The List,
ListArray, and Uidl methods
are tested in Relate Results When You Need To.
Destructive tests (deletion) occur next after we have finished testing
retrieval and listing. We validate the accessors
(Host, Alive, etc.) in
Consistent APIs Ease Testing. The final test cases
cover failure injection.

Make Assumptions

To access a POP3 server, you need an account, password, and the name
of the host running the server. We made a number of assumptions to
simplify the test without compromising the quality of the test cases.
The POP3 server on the
local machine must have an account pop3test, and
it must support APOP, CRAM-MD5, CAPA, and UIDL.

The test that comes with Mail::POP3Client
provides a way of configuring the POP3 configuration via environment
variables. This makes it easy to run the test in a variety of
environments. The purpose of that test is to test the basic functions
on any machine. For a CPAN module, you need this to allow anybody to
run the test. A CPAN test can't make a lot of assumptions about the
execution environment.

In test-first programming, the most important step is writing the
test. Make all the assumptions you need to get the test written and
working. Do the simplest thing that could possibly work, and assume
you aren't going to need to write a portable test. If you decide to
release the code and test to CPAN, relax the test constraints after
your API works. Your first goal is to create the API which solves
your customer's problem.

The POP3 protocol uses a dot (.) to terminate multi-line responses.
To make sure
Mail::POP3Client handles dots correctly, we put
leading dots in the message body. The message should be retrieved in
its entirety, including the lines with dots. It's
important to test data dependencies like this.

The test only sends one message. This is sufficient to validate the
client implementation. Testing the server, however, would be much
more complex, and would require multiple clients, messages, and
message sizes.

The sleep(1) is used to give
sendmail time to deliver the message before the
test starts.

Validate Basic Assumptions First

The first case group validates some assumptions used in the rest of
the cases. It's important to put these first to aid debugging. If the entire test fails
catastrophically (due to a misconfigured server, for example), it's
much easier to diagnose the errors when the basic assumptions fail first.

Bivio::Test allows you to ignore the return
result of conformance cases by specifying undef.
The return value of Connect is not well-defined,
so it's unimportant to test it, and the test documents the way the
API works.

This case raises a design issue. Perl subroutines always return a
value. Connect does not have an explicit
return statement, which means it returns an
arbitrary value. Perl has no implicit void context
like C and Java do. It's always safe to put in an explicit
return; in subroutines
when you don't intend to return anything.
This helps ensure predictable behavior in any calling context, and
improves testability.

The second case tests the server supports CAPA (capabilities), UIDL
(unique identifiers), and CRAM (challenge/response authentication).
The capability list is unordered so we check the list for UIDL then
CRAM or the reverse. Bivio::Test allows us to
specify a Regexp instance
(qr//) as the expected value. The case passes if
the expected regular expression matches the actual return, which is
serialized by Data::Dumper.

Once we have validated the server's capabilities, we test the
authentication interface. Mail::POP3Client
defaults to AUTH_MODEBEST, but
we test each mode explictly here. The other cases test the default
mode. To be sure authentication was successful, we download
the body of the first message and compare it with the value we sent.
POP3 authentication implies authorization to access your messages.
We only know we are authorized if we can access the mail user's data.

In BEST mode the implementation tries all
authentication modes with PASS as the last
resort. We use knowledge of the implementation to validate that
PASS
is the last mode tried. The Message method
returns PASS failed, which gives the caller
information about which AUTH_MODE was used.

The test doesn't know the details of the conversation between the
server and client, so it assumes the implementation doesn't have
two defects (using PASS when it shouldn't and
returning incorrect Message values). We'll see
in Mock Objects how to address
this issue without such assumptions.

The authentication conformance cases are incomplete, because there
might be a defect in the authentication method selection logic. We'd
like know if we specify APOP that
Mail::POP3Client doesn't try
PASS first. The last case group in this section
attempts to test this, and uses the knowledge that
Message returns APOP failed
when APOP fails. Again, it's unlikely
Message will return the wrong error message.

The Body method returns the message body,
Head returns the message head, and
HeadAndBody returns the entire message. We
assume that 999 is a valid message number and that
there aren't 999 messages in the mailbox.

Body returns an empty array when a message is not
found. Should Body
return something else or die in the deviance
case? I think so. Otherwise, an empty message body is indistinguishable from
a message which isn't found. The deviance test identifies this design
issue. That's one reason why deviance tests are so important.

To workaround this problem, we clear the last error
Message saved in the
Mail::POP3Client instance before calling the
download method. We then validate that Message is
set (non-blank) after the call.

The test case turned out to be successful unexpectedly.
It detected a defect in
Message: You can't clear an existing
Message. This is a side-effect of the current
test, but a defect nonetheless. One advantage of validating the
results of every call is that you get bonuses like this without
trying.

When Body, Head, and
HeadAndBody are invoked in a scalar context, the
result is a single string, and undef is returned
on errors, which simplifies deviance testing. (Note that
Bivio::Test distinguishes undef from
[undef]. The former ignores the result, and the
latter expects a single-valued result of undef.)

Bivio::Test invokes methods in a list context
by default. Setting want_scalar forces a scalar
context. This feature was added to test non-bOP classes like
Mail::POP3Client.

In bOP, methods are invocation context insensitive. Context sensitive
returns like Body are
problematic.[4]
We use wantarray to ensure methods that return
lists behave identically in scalar and list contexts. In general, we
avoid list returns, and return array references instead.

BodyToFile and
HeadAndBodyToFile accept a file glob to write the
message parts. This API design is easily testable with the use of
IO::Scalar, an in-memory file object. It
avoids file naming and disk clean up issues.

We create the IO::Scalar instance in
compute_params, which
Bivio::Test calls before each method
invocation.
check_return validates that the method returned
true, and then calls actual_return to set the
return value to the contents of the IO::Scalar
instance. It's convenient to let Bivio::Test
perform the structural comparison for us.

We test an invalid message number and a closed file handle[5] in two
separate deviance cases. You shouldn't perturb two unrelated
parameters in the same deviance case, because you won't know which
parameter causes the error.

The second case uses a one-time compute_params
closure in place of a list of parameters. Idioms like this simplify
the programmer's job. Subject matter oriented programs use idioms to
eliminate repetitious boilerplate that obscures the subject matter.
At the same time, idioms create a barrier to understanding for
outsiders. The myriad Bivio::Test may seem
overwhelming at first. For the test-first programmer,
Bivio::Test clears away the clutter so you can
see the API in action.

Relate Results When You Need To

Uidl (Unique ID List), List,
and ListArray return lists of information about
messages. Uidl and
ListArray lists are indexed by message number
(starting at one, so the zeroeth element is always
undef). The values of these lists are the
message's unique ID and size, respectively. List
returns a list of unparsed lines with the zeroeth being the first
line.
All three methods also accept a single message number as a parameter, and
return the corresponding value. There's also a
scalar return case which I didn't include for brevity in the book.

The first case retrieves the entire list, and saves the value for
the first message. As a sanity check, we make sure the value is
non-zero (true). This is all we can guarantee about the value in all
three cases.

The second case requests the value for the first message from
the POP3 server, and validates this value agrees with the value saved from the
list case. The one-time check_return
closure defers the evaluation of $_SAVE until
after the list case sets it.

We cross-validate the results, because the expected values are
unpredictable. Unique IDs are server specific, and message sizes
include the head, which also is server specific. By relating two
results, we are ensuring two different execution paths end in the same
result. We assume the implementation is reasonable, and isn't trying
to trick the test. These are safe assumptions in XP, since the
programmers write both the test and implementation.

We put the destructive cases (Delete) near the
end. The prior tests all need a message in the mailbox. If we
tested delete first, we'd have to resend a message to test the
retrieval and list methods. The case ordering reduces test
length and complexity.

Note that we cannot guarantee anything about
Count except that is at least one. A prior test
run may have aborted prematurely and left another message in the test
mailbox. What we do know is that if we Delete
all messages from one to Count, the mailbox
should be empty. The second half of this case group tests this
behavior.

The empty mailbox case is important to test, too. By deleting all
messages and trying to login, we'll see how
Mail::POP3Client behaves in the this case.

Yet another reason to delete all messages is to reset the mailbox to a
known state, so the next test run starts with a clean slate. This
self-maintaining property is important for tests that access
persistent data. Rerun the entire test twice in a row, and the second run
should always be correct.

The POP3 protocol doesn't remove messages when
Delete is called. The messages are marked for
deletion, and the server deletes them on successful
Close. Reset clears
any deletion marks. We cross-validate the first
Count result with the second to verify
Reset does what it is supposed to do.

This section not only tests the accessors, but also documents the
State and Alive transitions
after calls to Connect and
Login.

There's a minor design issue to discuss. The accessor
Pass does not match its corresponding named parameter,
PASSWORD, like the Host and
User do. The lack
of uniformity makes using a map function for the
accessor tests cumbersome, so we didn't bother.

Also the non-uniform return values between
Alive and Close is clear.
While the empty list and zero (0) are both false in Perl, it makes
testing for exact results more difficult than it needs to be.

Inject Failures

The final (tada!) case group injects a failure before a normal
operation. Mail::POP3Client exports the socket
that it uses. This makes failure injection easy, because we simply
close the socket before the next call to POPStat.
Subsequent calls should fail.

We assume error handling is centralized in the implementation, so we
don't repeat all the previous tests with injected failures. That's a
big assumption, and for
Mail::POP3Client it isn't true. Rather than
adding more cases to this test, we'll revisit the issue of shared
error handling in Refactoring.

Failure injection is an important technique to test error handling.
It is in a different class from deviance testing, which tests the API.
Instead, we use extra-API entry points. It's like coming in through
the back door without knockin'. It ain't so polite but it's sometimes
necessary. It's also hard to do if there ain't no backdoor as there
is in Mail::POP3Client.

Mock Objects

Mock objects allow you to inject failures and
to test alternative execution paths by
creating doors where they don't normally exist.
Test::MockObject[6]
allows you to replace subroutines and methods on the fly
for any class or package. You can manipulate calls to and return
values from these faked entry points.

In BEST authentication mode,
Mail::POP3Client tries
APOP, CRAM-MD5, and
PASS. This test makes sure that if the server
doesn't support APOP that
CRAM-MD5 is used and PASS is
not used. Most POP3 servers always support
APOP and CRAM-MD5 and you
usually can't enable one without the other. Since
Mail::POP3Client always tries
APOP first, this test allows us to test
the CRAM-MD5 fallback logic without finding a
server that conforms to this unique case.

We use the Test::MockObject instance to
fake the IO::Socket::INET class, which
Mail::POP3Client uses to talk to the server.
The faking happens before Mail::POP3Client
imports the faked module so that the real
IO::Socket::INET doesn't load.

The first three methods mocked are: new,
autoflush, and connected.
The mock new returns $socket,
the mock object. We set autoflush to always returns true.
connected is set to return false, so
Mail::POP3Client doesn't try to close the
socket when its DESTROY is called.

We fake the return results of getline with
the server responses Mail::POP3Client expects
to see when it tries to connect and login. To reduce coupling between
the test and implementation, keep the list of mock routines short.
You can do this by trial and error, because
Test::MockObject lets you know when a routine
that isn't mocked has been called.

The mock print asserts that neither
APOP nor PASS is attempted
by Connect. By editing the lines as recommend by
the comments, you can inject failures to see that the test and
Mail::POP3Client works.

There's a lot more to Test::MockObject than I
can present here. It can make a seemingly impossible testing job
almost trivial.

Does It Work?

As noted, several of the Mail::POP3Client test
cases were successful, that is, they found defects in the
implementation. In Refactoring,
you'll see the fruits of this chapter's labor. We'll refactor the
implementation to make it easier to fix the defects the test
uncovered. We'll run the unit test after each refactoring to be sure
we didn't break anything.