Sneek Peek at StoryTeller (i.e. my FitNesse killer for .Net)

I’ve gotten pinged 4-5 times in the past week about just where I’m at with StoryTeller development. I think I’m finally closing in on something that I could release at least as a preview. Just to prove that I’m still working on it, here’s a sneek preview of the tool. I’m tentatively doing another Elegant Code Cast with David Starr on StoryTeller soon, so just pile on with any questions.

Quick History

I’ve had a love/hate relationship with NFit and/or FitNesse for quite some time. In many ways, I still believe that “Executable Specifications” are the “Holy Grail” of Agile/Lean software development and that belief naturally led me to using first NFit and then FitNesse. However, I increasingly found FitNesse rather painful to use and impossible to transition to client teams. My frustration with FitNesse eventially led me to start a new OSS project called StoryTeller as a replacement for the test editing and management functionality of FitNesse, but keeping the underlying Fit engine. I coded on that for about 2 years with no more than middling enthusiasm and minimal success before I finally threw my hands up in the air over the FitNesse testing engine and junked the whole StoryTeller project.

The New StoryTeller

Late last year I started a complete ground up rewrite of StoryTeller with a totally different testing engine that we’re using to write acceptance tests and integration tests at Dovetail. I haven’t made an official release yet, but as of right now StoryTeller has:

A testing engine that is a superset of the basic FitNesse Fixture types (ColumnFixture, RowFixture, DoFixture, ArrayFixture, etc.). The engine supports a mix of flow-based and table-based testing.

Customizable type coercion

A command line runner for use in an automated build

A way to tag tests as either “Acceptance” or “Regression” for testing lifecycle (Regression test failures cause a build failure, Acceptance test failures are simply reported)

A User Interface tool that acts as a runner and an interactive tool for test editing

How does this relate to other tools?

NUnit/MbUnit/xUnit.Net — There is some overlap, but an xUnit test will never be terribly readable to non coders, and that’s what StoryTeller is striving for. I see xUnit tools being used for unit tests, StoryTeller for acceptance testing, and integration testing being a gray area where you might go either way on a case by case basis.

FitNesse — I obviously think StoryTeller is a far superior tool to FitNesse, or I’d just be using that as is. FitNesse is the closest analogue to StoryTeller.

Machine.Spec or Cucumber or RSpec — Kind of solves the same problems in very different ways, but StoryTeller is more geared towards integrated acceptance testing. I wouldn’t care to use StoryTeller for straight up TDD at the low level. I guess the short answer on this comparison is “I don’t know.”

Oslo — One of the frequent examples of Oslo I’ve seen is a DSL over the top of WatiN for web testing — and that’s great, but how about being able to quickly churn out little one off testing DSL’s?. In many ways StoryTeller is a tool for creating External DSL’s for testing your application code. That’s a direct overlap with Oslo (depending on which Oslo team member is speaking and to what audience), but at this time I feel like Oslo would require far more effort than StoryTeller in order to build a behavioral, runtime DSL to do testing. StoryTeller grammars allow you to quickly bridge the gap between how you want to express your test specifications and the system under test. Oslo would require you to parse and interpret those nasty, nasty AST’s to perform the actual functionality.

The Main Screen:

Woohoo! You can see all of the tests, you’ve got a status bar to show you what’s running, what’s queued up, and when the StoryTeller UI picked up the binaries (it does use a FileSystemWatcher like NUnit to watch for the system under test binary getting updated).

At the top, you have options to filter the tests shown in the test tree on the left:

Uh, and if that looks a lot like the ReSharper unit test runner, let’s say that imitation is the sincerest form of flattery.

You can also control the testing runtime like so:

I’m particularly proud of these features. Sometimes (most always), an integrated test can be slow to execute or Selenium can hang or who knows what can happen. StoryTeller will allow you to specify that tests should be aborted the first time they encounter an exception or a test failure instead of running the entire test as it normally does. You can also change the test timeout on the fly.

StoryTeller can show you the tests that are queued up to execute:

Yes, that screen is butt ugly so far, but it gives you the ability to track progress, clear tests out of the queue if you change your mind about running them, and abort the executing test.

Lastly, how about an actual test? Here’s the preview mode:

and the Results mode:

Like I said, this is just a sneak peek. I’ll have much more information out soon as I feel it’s becoming usable.

And no, I don’t think my problems with FitNesse are a team structure or a communication problem. Please don’t insult my intelligence (again) by changing the subject. The problem with FitNesse is mechanical.

About Jeremy Miller

Jeremy is the Chief Software Architect at Dovetail Software, the coolest ISV in Austin. Jeremy began his IT career writing "Shadow IT" applications to automate his engineering documentation, then wandered into software development because it looked like more fun. Jeremy is the author of the open source StructureMap tool for Dependency Injection with .Net, StoryTeller for supercharged acceptance testing in .Net, and one of the principal developers behind FubuMVC. Jeremy's thoughts on all things software can be found at The Shade Tree Developer at http://codebetter.com/jeremymiller.

Sounds great. Looking forward to seeing/using the implentation. Will it be backward compatible to our existing FitNesse tests? Probably not in first iteration but will you consider it? Be nice not to have to port our existing tests.

http://blog.troytuttle.com Troy Tuttle

Jeremy, Nice work on the project. I too like the goal of Fitnesse, but not the implementation. However, more recently, I’m not sure I like the separation of unit (developer) and user story tests. I don’t like the overlap.

Everyone that does the story test thing (where there are broad story tests vs. smaller developer/unit tests) have this notion that story tests have to be broad, and resigned to the fact that unit tests will be disconnected from user stories. Why don’t we try to do both in the same test suite?

Still open to be converted though. Maybe StoryTeller will be good enough to do that. 😉

http://michaelstechnical.blogspot.com Michael Hedgpeth

Promoting Fitnesse was probably one of the biggest mistakes in my “agile evangelism” career. I’m looking forward to promoting what you have here instead.

I also agree what you say about Oslo. They’re way too interested in data. It’s the behavior that’s interesting, at least with people like us.

Beth Brock

very interesting. I have the same issues with FitNesse, can’t wait to see more, and ditto on the actual test definition. thanks

Jonne Kats

Its starting to get shape, I would love to see some more about how the actual test definition takes place.

http://chadmyers.lostechies.com Chad Myers

@Jeremy: Our problems with Fitnesse were absolutely all about team structure and poor communication:

– We didn’t have a bunch of developers whose sole purpose was to write fixtures for the Business folks

– We weren’t able to effectively communicate to FitNesse (through code) what it is, exactly we needed it to do.

I always liked the idea of Fit/Fitnesse; I never managed to get it to actually run and do something. So… where’s the source, Luke? 😀

http://www.cornetdesign.com Cory Foy

You know, I never did understand why you wrote this. I really think that your issues with FitNesse are a team structure or a communication problem. Likely what we really need is just a FitNesse certification, and then you’d be able to use the tool correctly. We should totally now talk about certifications.

Oh, and your father smelt of elderberries. Which reminds me of this story I heard about blackberries, honey and biscuits…