I did a presentation recently at NYU for their Summer Launchpad programme where I talked about prototyping and agile development. There was one slide I touched upon in the talk that I wanted to expand upon here, and the topic was on the four goals every project should aspire to have. They are:

Features that have hypothesis

Hypotheses that can be easily validated

Code that is always production ready

Code that is easy to change

Why do I think this is important? Features that have hypotheses associated to them have gone through a thought process that believes something in the application will be improved. That may be experience, revenue or other performance indicators for an application. This will help focus the feature and enable suitable metrics to be extrapolated and used to falsify the hypothesis. Being easily validated means that the hypothesis and by the same nature, the feature is small. Small features mean the build-test-learn cycle can be shortened. Easily validated also means easily falsified, being that the team should not look for confirmation bias that the hypothesis was correct.

Code that is always production ready can imply some amount of housekeeping but I like to think it means going back to small features and small stories where each story is independent and adds value to the user. If that is the case, each commit can be a potential release candidate. Finally, code that is easy to change means two things. Firstly the code is designed so that any potential product change can rarely be a big technical change. This isn’t always attainable, but if features and stories are small no one story should change a codebase in a significant way. For me design also means evolutionary design through stories delivered combined with constant refactoring so that the codebase is always sensical.

Change also means confidence where confidence means tests. If a change is made to the codebase and no-tests fail that should mean that the code is production ready and if tests fail it means it is not production ready. If tests pass but the code is not production ready it either means that there is code coverage missing from the application.

All in all, these fairly simple goals have a lot going on in the background, but can push a project towards a heathy place.

Building a bookmarklet provides an interesting challenge. It involves interaction a website your application does not control where that site could be anything with any number of dependencies on CSS or Javascript libraries. The first choice to make is trying to work with that website and probably setting an !important on every CSS selector used and hope that there’s no namespace or versioning clashes with any Javascript included; or use an iframe.

Iframes seem to have fallen out of favour in recent times but the sandboxed nature of the content inside an iframe mean the worries of CSS and Javascript clashes are gone. However this is replaced with a communications overhead of communicating between the host document and the iframe. I wanted to touch on some of the things our team did on a recent project to try and make this fairly seamless.

Getting into the DOM

A bookmarklet is a small piece of Javascript that a user can drag onto the bookmark bar and upon pressing the link the Javascript will run. Because there is no guarentee what site will be loaded and if that will have any number of Javascript libraries included it’s best to use plain old Javascript to create an iframe element and append it to the body of the document. Our bookmarklet also needed to have some javascript if for nothing else but to be able to dismiss and remove the newly created elements. This can be done through the creation of a script element and appending to the body just like the iframe itself.

Once the iframe and script tags are appended they are treated the same as any other element. The content is loaded and the script is executed. The next step is getting the window to talk to the iframe. As a convienence the domain with protocol and port of the iframe is stored in a variable for later use.

The location is also sent to the remote server to load the iframe so that it can also store that location for passing messages to the host.

At this point this Javascript could also append a script tag to a version of libraries that may be required for it’s own application to run. It could also test for the existance of that library before hand so it doesn’t bring down an incompatible version. For example, bringing in jquery:

Sending a message

The postmessage method is available to communicate bewtween the window and the iframe. The host window with a reference to the iframe can call postmessage with a string as a message and as a security measure the target location. We had stored this during the loading of the elements as described above.

iframe.postmessage('hello', 'www.example.com');

That message won’t get anywhere unless the iframe is listening for the message event on the other end.

I’ve used the native Javascript above but really, once in the iframe itself the application has full control and could use JQuery or any other library at this point. Our application needs to listen to messages on both sides though so we needed the above to run in the host anyway.

RPC

Sending a message is all well and good but any non-trival application is going to have more than one function to run. We took influence from Remote Procedure Calls (RPC) to call functions within the host and remote sites. The message sent was stringified JSON with a very light ‘schema’ of the function to run and parameters to send to the function.

{
f: 'theFunction',
params: {
...
}
}

The recipient could then parse the string it knew to be JSON, extract the function to call and call it with any optional parameters also sent. This does create a binding between the host and the iframe but as the appliction controlled both sides we deemed it an acceptable risk. The function can be run from the window like so:

Dealing with namespaces

It was mentioned earlier that one of the goals of using an iframe was not to clobber any Javascript namespaces but we did end up including Javascript in the host and to avoid this we used an application namespace. However calling that as a property on the window would no longer work.

This is natually fairly crude, some defensive code could be added but this demonstrates the intent of the processing. Defense like making sure that function existed, or that ‘f’ existed in the data for the window could receive a message from another iframe.

Putting it all together

The bookmarket inserts two elements, an iframe and a script. The iframe has the source example.com?referer=bar.com. bar.com is inserted as a variable in the iframe Javascript code. The iframe inserted has an id of example_iframe

The iframe has an event listener which is the same code as the host, and runs the function example.world

var example.world = function(){
// hello, world
};

Wrapping up

This has shown some of the techniques for a ‘hello, world’ bookmark with two way communication between host and iframe that uses Javascript namespaces. This was enough to get our application off the ground as the two way communication acted as a solid base to build upon.

How to version an API has been a thoroughly discussed topic in the last several years regardless of protocol or approach, be that SOAP, REST or Hypermedia. Why contribute another post to the topic? My last post on avoiding breaking changes through better design led to conversations in the office and online about versioning.

In another post Nottingham adds that we shouldn’t use a custom header for versioning and touches on versioning through the Accept header but there is a fundamental thread here: try as hard as possible to not introduce breaking changes so that versioning isn’t a big issue.

One of the issues to contend with when considering versioning is often building APIs requires more thought up front that an agile developer might be used to. URIs, Data structure, meta-data and extensibility are important and would be best considered up front. Once those decisions have been made changes to the structure often result in breaking older versions of the API. Anyone at an early stage of building an API would do well to put some thought into the API design and setting some rules for future consistency.

Accept header

The Twitter comment above pointed me towards using the Accept header as part of content negotiation and I found a number of blog posts covering the subject as well as some companies using this approach.

RFC4288 section 3.2 outlines how a vendor, i.e. an application, can make use of customisable MIME types in the Accept header. Steve Klabnik shows how an application can make use of this to include a version number as part of the Accept header used for content negotiation.

Looking at some concrete examples through the well documented Github API shows how the Accept header can be used with their services:

Accept: application/vnd.github[.version].param[+json]

curl -v -H 'Accept:application/vnd.example.v1+json' localhost:3000

The vnd part is the vendor definition as outlined in RFC4288. Let’s take a look at a number of options to make use of this header.

Parsing the Accept header

Within an application, headers can be inspected from a HTTP request. Extracting the version number would require some regular expression matching, something along the lines of the following in an application_controller perhaps:

Version via a request parameter

The thing with the version in an Accept header is the URI is difficult to share. If I wanted to share the URI with version information with a college I would have to send instructions on what curl arguments to send to the server to get the right response. It’s can be as frustrating as trying to share a holiday on a website that renders pages based on what’s in the session for the individual. I would want to share a URI that someone can paste into a web browser address bar and see an appropriate response. Let’s compare the two approaches:

curl -v -H 'Accept:application/vnd.example.v1+json' localhost:3000

vs

curl -v localhost:3000/?version=v1

or

curl -v localhost:3000/?version=20130603

Version number as a date

The Foursquare API allows clients to send a version as a date in the format yyyymmdd which conveniently is an always increasing number. When a client starts to use the API they can use that days date and the response will always be in that format. I like this concept as it removes the burden of having to know what endpoint to use or what version to send in a header. The client uses a known point in time and the response will always match if that is sent. If nothing is sent then the latest version of the resource is returned.

With this in mind, I wrote a very small gem to put this into practice to demonstrate how I thought this could be achieved which boils down to the following. Given a number (i.e. the version sent in the request) and a list of numbers (i.e. known versions with some change in an application), find the previous closest number in the list from the version number given. For example, if the date 20130101 is passed and the list contains two dates of 20130101 and 20130601, then 20130101 is returned.

Robustness principle

Perhaps using the Accept header makes you a better denizen of the Internet, adhering to HATEAOS principles but I think using a version request parameter makes for a better Web experience. As a developer, I want to be able to put a URI into a browser and see a response rendered and for that I’d lean more towards the version parameter.

This approach can be generalised across other information sent from a client to a server. Clients like web browsers send information in every request, and these should honored as the defaults. The information sent includes what language, data and encoding formats the client would like. Using request parameters can offer overrides and support compatibility between different types of clients who want to use an API and coolbookmarkable URIs.

There are many blogs about how to expose an API for a Rails application and many times I look at this and am concerned about how these examples often leak the application design and the schema out through the API. When this leak occurs a change to the application internals can ripple out and break clients of an API, or force applications to namespace URI paths which I feel is unnecessary and ugly.

When the only consumer of application data models are the views within the same application then the object design can be fluid and malleable. Once an application exposes an API to more than one client, and especially if that client is on a different release cycle to the server, such as iPhone application, data models become rigid. Rails tends discouraged N-tier architecture to the benefit of development speed but APIs are contracts between a server and it’s client and can be difficult to change once they start being used.

Passing an object into the Rails JSON serialisation methods will work for a time, but relying on this will only get you so far. At some point a refactor will take place that will cause a breaking change. It could be something simple such as renaming a column, moving responsibilities from one class to another or adding extra meta-data to a response. Either way, adding this information into your model class starts to place more responsibilities into one place.

There are a few ways out of this potential issue. Let’s take a look at the classic blog application and its Post object. The Rails rendering engine will call as_json on an object if the request has sent the content-type of application\json to the server. Here we override the implementation from ActiveRecord to provide a stable, known version:

def as_json(options={})
{
author_id: author.id
title: title
}
end

A second option is to model the object explicitly and serialise the internal model into a public representation. We can duck-type the object to respond how ActiveRecord objects behave during a serialisation call. Although this can be seen as a step towards a N-tier architecture, it’s also a step towards service dependent abstraction:

The benefit of doing this is a separation of concerns between your data model and the data presentation. An application model doesn’t need to know how it’ll be represented by an API, command line interface or any other outside communication mechanism. If an application were tending more towards HATEOAS for instance this separation could help resolve hyperlinks relevant to the interface. You may lose some of the Rails respond_with goodness with this:

What’s the difference between this and putting the as_json method into Post directly? More control, separation of concerns with application modeling vs presentation and the big win is when breaking changes occur within the API. Now we can put version relevant information into new objects, or into the serialised class itself.

Through this we have versioning information in one place and through a request parameter of something like v=20130506 the application can handle multiple versions in one object. For me, this ultimately removes URIs like /v1/posts, but why is that important? The URI is an identifier which points to a resource and having v1 or v2 in the URI muddies the fact that the two identifiers are pointing to the same resource. Using a request parameter, much like pagination is handled, means we can ask for a representation of that resource rather than having to specify different resources. Then we can do away with needing controllers such as Api::V1::PostsController and just deal with Api::PostsController or even just PostsController and deal with the versioning within the object instead of the URI path.

Thanks to the benefits of Open Source software and working with great people, I’m pleased to announce some updates to both ElementalJS and to SimpleBDD.

ElementalJS

Thanks to Ian Zabel who made a performance improvement to ElementalJS after fighting a large DOM in Internet Explorer. Elemental will now load the behaviours much quicker if the document is passed as no filtering will take place. If another node is passed, filtering will be applied but the thought is that DOM will be much smaller so hopefully won’t hit this issue.

Extending ActiveRecord::Base leaks a powerful API throughout an application which can lead to tempting code which breaks good design. Take the classic blog example where you may want to retrieve the latest posts by a given author. You may have seen, or even written code that gets the dataset you need straight into the controller or view:

For me this is a design violation as well as breaking the “Law of Demeter”[Edit: Current Pivot Adam Berlin and former Pivot John Barker pointed out that chaining with the same object was not a Demeter violation]. The example above tells me structure of the schema that the calling class has no business knowing. It also makes testing using stubs ugly and encourages testing against the database directly. A test would have to chain three methods to stub a return value. It’s brittle, as in it’s susceptible to breaking due to changes outside of the class. For me it also fails from a narrative perspective in that it doesn’t succinctly reveal the intent of this part of the application.

If we were testing this and attempting to use stubs, we’d have to write something like the below. You can see how this is at best cumbersome, but also fragile.

Here’s what Bryan Helmkamp has to say on query objects in his excellent write up on fat ActiveRecord models. Bryan here rightfully points out that once in a single purpose object, they warrant little attention to unit testing. Now is the right time to use the database to ensure the right data set is being returned and that N+1 queries are not being performed. This means that database testing would only occur within the class actually hitting the database and not the rest of application which has a dependency on the database.

All of these techniques discussed serve to improve the design of an application by preventing leaking responsibilities from one class throughout the rest of the application. I’m also not saying that developers shouldn’t be using ActiveRecord or even Rails, but to use the tools responsibly.

REST principles by default is a fantastic convention within Rails applications. The documentation for how to route HTTP requests are comprehensive and give examples about photo resources within an application. If you’ve got photo and tag as first class resources of your application, Rails has you covered. But what if you are building an application with a focus on one type of resource, do you really want /resource_type as a prefix to all of your application paths? I certainly don’t and I’ll show you how to remove that without diverging from Rails core strenghts.

For better or worse, I’m always conscience of making sure applications I’m involved in have Cool URIs and sometimes that does mean fighting the Rails conventions. However Rails routing is very flexible and can provide me with the application paths that make me happy.

Take Twitter as an example. Every user has their username as a top level path, so instead of having /users/robb1e, they simply have /robb1e. When dealing with an application where there is one core resource it can make a lot of sense to strip the resource prefix. This can be achieved through scopes in the routing configuration.

Your::Application.routes.draw do
scope ":username" do
get '', to: 'users#show'
end
end

Gives you routes which look like

GET /:username(.:format) users#show

If you wanted to see the followers and followees of that user, you have two options. Return to the default resource or use HTTP verb contraints. I’ll show you both.

Your::Application.routes.draw do
scope ":username" do
get '', to: 'users#show'
get '/following', to: 'user#following'
get '/followers', to: 'user#followers'
end
end

This gives the paths

GET /:username/following(.:format) user#following
GET /:username/followers(.:format) user#followers

If you are trending into paths unknown, you always have the safety of tests to help you out. Both Rails and RSpec have ways to test your application routes.

One gotcha which using the default resource routing removes is clashing paths. If you decide to build an admin page and want to put that at /admin, that needs to be in the routes config before the scoped block and if a user has given themselves the name of admin then you may be in for some fun.

So the next time a need arises for an unconventional route, check the documentation, it’s probably possible although almost always warants thinking about.

I was pretty interested in refinements in Ruby 2.0, and after listening to the latest Ruby Rouges podcast where some serious doubts were raised about the viability of refinements I thought I’d build a little example of how I was thinking I could use it.

I failed first time out and I tried copying and pasting examples without success. After some time poking around I found a blog post about why this wasn’t working. Ultimately, refinements are sort of left in the language, but not fully supported and are marked as experimental.

Here’s what I wanted to achieve. Coming from Scala in my previous job, I thought I could use refinements as a proxy for implicit conversions. Here I refine the Fixnum class to allow it to respond to a to_currency message. When called it converts the Fixnum instance to a Currency instance.

A small number of projects here in New York have adopted my extremely simple behaviour driven development library, SimpleBDD, and I thought I’d share some of the emerging patterns those teams have developed while using it.

SimpleBDD, is a way of using a Gherkin like specification language and turning into a method call with Ruby. It takes the approach of tools like Cucumber but reduces it down the smallest set of features. Essentially taking a method call:

Given "a logged in user"

and calling an underlaying function:

a_logged_in_user

If that method is in the scope of the executing test, that method is executed, if the method isn’t in scope or doesn’t exist the standard method not found exception is raised. It enables a developer to produce Gherkin like specifications while staying in Ruby, using the test framework of choice.

I generally try and follow the advise of my colleague, Matt Parker, with his excellent post on steps as teleportation devises. We try and create a reusable and stateless domain specific language (DSL) for our tests and our steps call into the DSL and hold state pertinent to that particular test run.

At first on my current project we have had three separate areas for our request specs. We have the request spec itself which used SimpleBDD to describe the behavior of the application. We then had a ‘steps’ file which had the methods calls from the SimpleBDD and translated those into the reusable DSL for the application. The steps file was reused across all request specs and was becoming big pretty quickly.

Dirk, on another project which is also using SimpleBDD, skipped the ‘steps’ file and placed those methods straight into the rspec feature files underneath the scenario blocks. Then after some discussions with JT on where to keep the state our tests depended on, Brent and our team started using rspecs ‘let’ methods and the ‘steps’ within the scope of the feature block to keep the intention of the test in one place.

By also putting more responsibility onto the DSL, these methods are pretty dumb, leaving the test describing what the test is attempting to achieve through SimpleBDD method calls and the how through calling the DSL via the ‘step’ methods within the feature block in the same file.

A surprising amount of simple can get an application over a number of speed bumps. We’re going to look up and down the whole application stack and use stories to show what simple things people have done to build a sustainable system without re-architecting.

You’re gonna need a bigger boat

One of my favourite stories from a colleague was when they consulted for a previous company. The application was struggling to scale and the answer from the development team was vertical scaling. Buying bigger servers, or putting in more RAM would buy more time for the application to keep ticking over until next time. My diligent colleague when joining this team spent some time digging around trying to find bottlenecks in the application. They discovered that there was not a single index on the database and that pretty much every transaction was doing a full table scan to get the result. The previous answer to get more RAM was so that the entire database could fit within it, in an effort to increase the performance.

OK, so no indexes at all seems like an extreme case, but it can be easy to skip an index or over index. Small companies don’t tend to have database administrators and if a test isn’t failing and no-one is complaining in production it’s easy to see that this area can be skipped without realising. There are some tools out there like ‘Rails Best Practices’ which can help identify where indexes are missing. Some simple changes and checks and drastically improve performance and delay that re-architecting we’re all afraid (or excited depending who you are) to do.

Instrument, Instrument, Instrument

A different colleague started their previous job just as a major rewrite was nearing its completion. There was some stress as a move from ColdFusion to Ruby was not paying off the dividends the team had sold to the product owners; the application performance wasn’t good enough to go live with. Tests were green and no bugs were reported so no light was shed on the troubled areas of the application. By adding instrumentation using a tool such as NewRelic, slow processes and queries were found and refactored. Over the course of a week, working on and off the problems the performance was brought up to an acceptable level where the application could go live.

In a way, this was not a terrible position to be in. Performance is one of those things that it isn’t a problem until it is, and just before pushing a release out of the door seems like an ideal time to do some performance testing. The tool NewRelic itself can be used locally for this and can run as a hosted service against real production requests. On teams I’ve been involved in, I like to go through the slowest requests on and identify and fix any problem areas as a Friday afternoon chore. Instrumentation doesn’t have to come through a tool like NewRelic, it can be looking at logs of web request times and slow database queries, but taking some time to fix these can make some significant improvements.

Caching in

There are a number of caching techniques I’ve heard about and seen. Some have been effective, others have created more problems they had set out to resolve. First a cautionary tale.

Like scaling a database by putting the entire database into memory, caching can obscure underlying issues. On a recent project caching mechanisms were scattered through the code, where to cache, where to invalidate. This meant that when caching something, either through the application or even changing code, we couldn’t be sure if the caches would be affected. In one instance, our production environment was showing some strange behaviour that we could not replicate. After digging around, we found that the caches were being invalidated by the last update of the object being cached, but we had changed the template within the cache, leaving some objects being presented with the old template, and others with the newer one.

Phil Karlton’s quote “there are only two hard things in Computer Science: cache invalidation and naming things” springs to mind. The lesson here is caching can increase performance significantly but can hide issues. By caching the result of slow running code, are you hiding code that could be improved?

Rails 4 has tried to solve some of these issues by suggesting that applications generally only cache the results of rendered views. It also takes away the cache invalidation part by using the objects name, last updated time and the MD5 of the template being rendered as part of the keys. Using a caching system which automatically drops the least used cached entries should be sufficient to deal with this.

Caching out

Sometimes the responsibility of caching can be handed to another part of an applications’ infrastructure. This is exactly what we had done on some projects when I was working on the Guardian. The applications we were writing depended heavily on external services for data and these services, being good citizens of the web, had returned appropriate cache headers in the HTTP responses. For this given application, we didn’t want to model the data coming back, we merely wanted to transform the response and place in a HTML template. Using a HTTP caching proxy like Squid installed on the same server as the application making outbound calls meant we could rely on Squid to do the caching. There was the HTTP request out, but as this never left the server, it was a small hit.

201 Created

Donald Knuth said that “premature optimization is the root of all evil” but there are a series of small optimisations that can be worthwhile. When making a request to a web server or external service, an application is either changing or reading state, sometimes both at the same time. When reading back state at the same time in the same request as changing it, if an application performs only the work necessary for the response and puts the rest of the work in a background job of some implementation, the application can respond more quickly. RFC 2616 HTTP response codes 201 and 202 were made for this sort of operation.

The response code 201 is useful for letting the client know a resource has been created. In one of my first projects in telecommunications, we would send a 201 to indicate a phone call had been started. The client would request a call be made between two phone numbers, but we didn’t want the request to be tied up during the actual phone call and maybe returning only when the call was finished. A 201 with a location header for the client to get the status of a call was an idea choice. A resource (the call) had been created and had an address which the client could use.

A more web based example could be signing users up to a new website. If there are welcome emails to be sent and mailing lists to be joined it’s not in the applications interest to make the user wait while SMTP gateways respond and third party services give their OK to a request. If this is spun out into a thread or background job the application can return to the user and allow them to carry on. The less essential processes will happen, but the delay that occurs is acceptable and the user gets a more performant experience.

Perceived performance

Steve Saunders and the Performance Group at Yahoo have done great work with tools like YSlow and highlighting the issues with perceived performance. When discussing this issues, Saunders said “Optimize front-end performance first, that’s where 80% or more of the end-user response time is spent”. So much of the advice YSlow suggests is so simple to implement that I recommend just setting it up at the beginning of the project and checking the advice every now and then. Some of the suggestions are one off and can be done at the start of a project, others require some ongoing checking, but these techniques will give a faster experience for the end users.

An interesting example is where this has been taken to an interesting level has been the recent rewrite of the Guardian mobile website. The application focuses on the most important aspect: the content. Javascript has been stripped back to only what is required to load the content, there’s no jQuery in sight.

The main Guardian website with an empty cache downloaded 1.06mb with 212 requests. It took 2.5 seconds for the DOM to download it’s content and 4.3 seconds before ‘onload’ was fired. For the mobile version of the website, it downloaded 25k of data with 77 requests with the DOM content load event happening after 260ms and ‘onload’ being fired after 950ms. That’s a pretty big difference and sometimes that effort is warranted.

Another technique used by the new Guardian mobile website is conditional loading of content. Say on a sports team page, after the main content has been downloaded and the reader is celebrating or sobbing depending on their team news, asynchronous calls are made for extra content. In this case, fixture/schedule information, results and related content. This information isn’t required for the reader to achieve the main purpose of their visit to the page, but it might help keep them there. Using conditional loading, the page loads quickly, the reader can start reading the article without having to wait longer for a bigger DOM or synchronous loading of the extra content.

Real-time vs Near-time

One important I’ve personally learnt is when asked to build something in ‘real-time’, it to ask the product owner to define real-time. Real-time can mean something very different to a developer than to a product owner or user. For the longest time I equated real-time with ‘immediately’. When I discovered the actual requirement was ‘within a reasonable time but not necessarily immediately’, this changed many assumptions I’d made about the application.

A company that perform complex calculations to give a user a view to potential savings when switching to a different service provider, the product team had requested the price update in real-time. The developers knew that crunching the numbers can actually take a non-trivial amount of time, so they put the processing in a background job and used the HTTP meta-refresh element to refresh the page and ultimately hold the user in place, followed by a redirect to a page with the crunched numbers.

This may sound crude, but when considered against the time to build a better, more ‘real-time’ experience, it was an easy choice. Consider too that this pattern is often employed during a checkout experience, especially when booking something like a flight. After your credit card information is taken, you’re taking to a holding page until the payment is confirmed and a seat reserved.

Meanwhile, in another part of town, a team was content with itself after building a ‘real-time’ application that displayed government bond finances and gave 10 second updates. They relied on a message queue publish-subscribe architecture to get the very latest information to the users. When they went out to watch the application in the wild, they found that those people watching these bonds would take a look every few minutes in-between doing other tasks. It turns out that working on bonds is not the same as working on the stock exchange. The application had been over-engineered for a problem that didn’t exist. After realising this, they could correct their course and remove complexities from the application while remaining ‘near-time’ for their users.

Take a breath

We’ve looked at some simple changes that can bring big benefits and how user perception of performance is probably more important that server side performance. We’ve also seen how asking the right questions can lead to simplicity in itself.

Step two to a successful business: know the product, embrace the tools that show weaknesses and learn where best to invest your time.

Big companies with well established products and business practices often differ from smaller, younger, start-up companies with the cash flow available to prove business ideas. The startup companies will often have a “runway” or “burn-rate”, meaning they have a limited amount of cash to keep the lights on. Big companies have budgets, committees and risk departments. Those companies may take a risk on a new product, but if it fails the lights will stay on and people will more than likely still have a job. Established companies will also invest in existing products to strengthen their market position which may result in growth targets rather than a focusing on validating those targets.

Too big to fail

New products should always look for market validation regardless of the size of the company. Just look at how long the HP Touchpad (48 days) and the Microsoft Kin phone (49 days) lasted in the marketplace before being withdrawn. It’s reported that HP would go onto lose up to $300m after cutting the Touchpad and associated webOS product line along with job cuts. So much for market validation and their risk department. Another huge loss was RIMs Playbook, reported to have lost the company $1.5bn in a period when the company really needed a boost in the marketplace amid falling Blackberry sales.

Not since the dot com boom have startup companies had anywhere near that sort of money to take a product to market and while these examples are hardware based incurring the associated research and development costs there is a lesson about validation in there regardless of the size of the company. Software companies seem to fail in three categories: massive dot com blowouts (e.g. WebVan and pets.com); being squashed by a competitor (Friendster to MySpace to Facebook); and the ones you’ve never heard of as they failed before they made it.

That’s not to say that the companies that are using metrics and data to help make their decisions don’t push the process too far. The example of testing 40 shades of blue at Google is probably the best known. However, a little information can go a long way.

Agile and counting

How should a company with a limited cash flow proceed? That company needs to know that every feature built is going towards a larger goal, is required and ultimated validated by the users. How does a company know when a feature has been validated?

Each new feature should have some validation criteria, e.g. “Removing password confirmation should result in decreased number of users getting stuck and leaving at the registration stage”. This then needs to be measured, which means we need to have that measurement before the feature is written ideally.

There are a number of tools available to start gather metrics, Google Analytics is good for general visitor information but tools like KissMetrics have gathered support for more of data driven demands. There are also open source alternatives such as statsd and Cube if the project has the time to invest in their own tools. If the hypotheses are more visually driven, tools such as CrazyEgg have proved useful for validating calls to action and discoverability within an application.

As an example, something as simple as CrazyEgg can show where users of an application get confused. On a recent project we deployed CrazyEgg on the homepage and almost immediately we could see that users were clicking on headings like they were links. You could almost sense their frustration through the heatmap when a click didn’t change the page.

On another project we could see we were dropping a high number of users through a certain part of the application funnel. At this point we did some user testing and tried to focus on this area where users were dropping off. We found that the application, which had a “real-time” price for the service the user was getting a quote built in some assumptions for earlier selected defaults but without telling the user that had happened the users were getting put off by the high quote. We could hypothesise on what might increase the throughput of the funnel, make those changes and see which one worked best.

Tools and talent

There are some tools to help follow a feature through to validation. Tools such as Lean Launch Lab or more general tools such as Trello can help monitor the success of features and their validation.

So you have the tools, you have the talent. Now what? You need to know where the application currently stands with the metrics which drive the business. When a new feature is written it should include a hypothesis about what metrics will change when the metrics goes live. The feature gets built and goes live. How do you follow what happens next? Most tools and processes stop here, features have been accepted and is live, what more do we need? Those features should be monitored while the product team measures the effectiveness of the change. Did that feature make the hypothesised impact? If not, should the feature be re-evaluated or thrown out?

Step one to a successful business: know the product, understand the users and always be validating.

Recently I had posted about a few testing strategies that can be applied with RSpec. One of the patterns I mentioned was using something like NullDB to ensure your unit tests were not hitting the database. I had a few conversations about what I’d written, notably from my colleague Ian Lesperance. We discussed, and I conceded, that it’s preferable to have tests related to one class in one spec file. In particular I had split out the tests for the unit level and the integration with the database tests. So, here are my experiments on how I brought those tests back while keeping the same integrity of using a database for some test and forcing the null object pattern on other tests.

I had some issues having those tests in the same file, but with a little help from another colleague, JT Archie, we managed to figure it out.

Consider this rspec test:

describe Widget do
describe "#higest_selling", :db do
it "uses the 'highest_selling' scope" do
...
end
end
describe "#display_name" do
it 'concats the widget name and manufacturer' do
...
end
end
end

The ‘higest_selling’ method is a scope and has the ‘:db’ tag associated to the block, while the ‘display_name’ test has no tags applied. I wanted this to be the case, no tags means no database but if you want to hit the database, you need to explicitly call it out.

One trick you might have missed above was no longer needing to do ‘db: true’ in the RSpec tag. With the following setting in the spec helper, you can apply a symbol directly like ‘:db’.

config.treat_symbols_as_metadata_keys_with_true_values = true

Testing with NullDB

Using NullDB within the same file, we can use the ‘nullify’ and ‘restore’ helpers, but I found it worked best using the ‘around’ configuration. Using ‘before’ and ‘after’ I was having issues with changing the connection adapter during a transaction. This way, it appears to get around that issue.

We run the configuration block around each test that has the ‘type: :model’ tag. RSpec-Rails applies these automatically to any tests in the ‘spec/models’ directory. We look to see if the example has the ‘:db’ tag and if it does, we restore the default connection adapter, and run the example. If the example does not have the ‘:db’ tag applied, we apply the NullDB adapter, run the example and then restore the default adapter.

Testing using stubs

There are other options and with a sizable amount of help from JT, we created a simple way to achieve a similar outcome. Under ActiveRecord there are two methods which actually hit the database, ‘exec’ and ‘exec_query’. These methods can be stubbed out much like any method on any object in an application codebase.

In the ‘spec_helper’ file, we replace the NullDB configuration with the following. We again check for the ‘db’ tag and if it’s not there we stub ‘exec’ and ‘exec_query’.

config.around(:each, type: :model) do |example|
unless example.metadata[:db]
ActiveRecord::Base.connection.stub(:exec).
and_raise("You're not allowed to do that")
ActiveRecord::Base.connection.stub(:exec_query).
and_raise("You're not allowed to do that")
end
end

Testing using Nosql

We took this concept one step further and created a Gem that wasn’t RSpec specific. We couldn’t believe our luck when RubyGems showed there was no Gem called ‘nosql’, so with that problem solved we created the Nosql gem. When included in a test suite, any call to the database will raise an exception.

With the around configuration block Nosql is disabled and enabled accordingly.

All three of these options force unit tests to not hit the database. Database calls will either be ignored (NullDB), or will raise an error (Nosql). This should result in decreased execution time for tests as it will encourage the developer to stub out those interactions with the database.