Plack::Test and Great Test:: Interfaces

Plack is great. Most of it is wonderful. I'm less enamored with the interface of Plack::Test, however.

In writing the
Little Plack Book, I spent a couple of days writing tests for Plack and
applications at the PSGI level. Plack::Test occupies a strange
level in the ecosystem. It's incredibly useful for what it does in selecting
between a no-HTTP backend or any other PSGI-compatible handler, and it offers
some degree of abstraction for making requests and getting results, but it's
far too low level to write tests for complex applications.

When writing tests for a small Dancer application, I spent
more time getting the Dancer environment set up well for testing than I did
writing the tests. If I'd used Dancer::Test, I suspect
I'd have made more progress more quickly.

None of this surprises me—Catalyst and Dancer are one layer up the
stack from Plack. Most of the interesting things you can test about a Catalyst
or Dancer or Mojolicious or whatever application are properties expressed at
the framework and application layers, not the plumbing layer of Plack.

Plack::Test seems best for testing middleware and other
components which occupy the layer between PSGI and applications. Even so,
something about its interface kept bothering me as I wrote tests and prose
about the tests:

The pattern of a special test function which takes a block of code is
semi-common in the Test::* world; you can see it in Test::Exception and a
lesser extent in Test::Fatal. The best
example I've seen of this is Test::Routine, which
uses this block syntax to help organize tests into named groups. A disciplined
tester can use this to great effect to clarify what might otherwise become a
big ball of muddy tests.

I like that Plack::Test does the hard work of redirecting
requests to the app I want to test on whichever backend I want, so that I don't
have to worry about setting up a testing deployment target. That part's great.
The confusing part is:

my $cb = shift;
my $res = $cb->( GET '/some_url' );

test_psgi takes a PSGI application and an anonymous function as
its parameters, then invokes the anonymous function, passing a callback bound
tothe context of the application. Inside the anonymous function (the block, not
the callback), you invoke the callback and pass an HTTP::Request object
(constructed manually or with a helper such as HTTP::Request::Common)
and receive an HTTP::Response
object.

Put that way, it's a lot more confusing than it is, if you're comfortable
with the idea of first-class functions and closures and semi-fluent interfaces
in Perl 5.

Even so, my $res = $cb->( $req ) just looks
weird to me. It sticks out. It's visually different from all of the rest of the
code. Everything else outside the test is the semi-fluent interface of
anonymous subs or boring old method calls on objects.

In a discussion with Zbigniew
Łukasiak, I suggested that I'd want an interface more like:

You can see the influence of Test::Routine. I don't know
exactly how $app gets into the block, but this gives labeled
subtests and the concomitant organization, it obviates the need to create
HTTP::Request and HTTP::Response objects (or their
Plack:: equivalents) manually, and everything in the block uses
visually similar mechanisms.

The only hairy part is figuring out how to connect plack_test
to the app while not multiplying hidden global variables or disallowing
customization and decoration with other middleware. Compatibility with Plack::Builder is
important, but I don't especially want to pass in $app myself
manually.

Besides, to me the only obvious benefits over Test::WWW::Mechanize::PSGI
are the subtest groupings. Mech has a huge advantage of providing many useful
test assertion methods which grovel inside responses so I don't have to.

Maybe this is less my discomfort with one part of the
useful-at-the-appropriate-level Plack::Test and more a plea to
distinguish more clearly between various elements of Test::*
modules, as several distinct categories of behavior exist:

Setting up a testing environment

Organizing test assertions

Providing test assertions

We did a huge amount of work a long time ago with Test::Builder making
sure that everything which wanted to provide test assertions could interact
nicely, and we succeeded. Maybe it's time to consider ways to enable that
composition at other layers of the Test::* stack.

Tags:

4 Comments

It's incredibly useful for what it does in selecting between a no-HTTP backend or any other PSGI-compatible handler, and it offers some degree of abstraction for making requests and getting results, but it's far too low level to write tests for complex applications. (...) Plack::Test seems best for testing middleware and other components which occupy the layer between PSGI and applications.

It is correct that Plack::Test is targeted to write unit tests for PSGI app compatibility as well as writing PSGI middleware. And for higher level application testing, you're suggested to use interfaces like Test::WWW::Mechanize(::PSGI) as you pointed out in the rest.

Even so, my $res = $cb->( $req ) just looks weird to me.

It looks as weird as my $res = $app->($env); looks in the PSGI - the point is, or at least originally was, the Plack::test interface should be wrapped and presented as the Catalyst::Test/Dancer::Test interface as an application developers - such that app developers don't need to worry about which PSGI backend or handler it is using.

The other reason I chose the callback interface was to take the benefit of Test::TCP module, that expects the callbacks to be executed by forking the server and client code apart. You can argue that is an implementation detail, but at that time we wrote Plack::Test (in late 2009) that was the only (easy) way to achieve that.

That said, newer version of Test::TCP supports the OO interface that doesn't require two callbacks - only once for the server code which can be just a wrapper for the PSGI $app callback, and client caller code can be written in a synchronous looking code. There's even a ticket for that to put that back to Plack::Test, which I closed for some reason I don't remember - but it might be worth trying it again.

Zbigniew and I talked about the correspondence between the testing callback and the PSGI callback. Plack::Test's callback is very Plackish, and that's an appropriate design choice. It does the job and well, and I understand why you chose it.

Mixing of different interface styles in client code just seemed jarring. Maybe it was me trying to formulate the idea that Test::* modules provide different parts of a testing stack.