This is the blog of David M. Raab, marketing technology consultant and analyst. Mr. Raab is Principal at Raab Associates Inc. The blog is named for the Customer Experience Matrix, a tool to visualize marketing and operational interactions between a company and its customers.

Tuesday, March 13, 2007

I had a long and interesting talk yesterday with Larry Epstein at SiteSpect, a vendor of Web site multivariate testing and targeting software. SiteSpect’s primary claim to fame is they manage such tests without inserting any page tags, unlike pretty much all other vendors in this space. Their trick, as I understand it, is to use a proxy server that inserts test changes and captures results between site visitors and a client’s Web server. Users control changes by defining conditions, such as words or values to replace in specified pages, which the system checks for as traffic streams by.

Even though defining complex changes can take a fair amount of technical expertise, users with appropriate skills can make it happen without modifying the underlying pages. This frees marketers from reliance on the technical team that manages the site. It also frees the process from Javascript (which is inside most page tags), which doesn’t always execute correctly and adds some time to page processing.

This is an intriguing approach, but I haven’t decided what I think of it. Tagging individual pages or even specific regions within each page is clearly work, but it’s by far the most widely used approach. This might mean that most users find it acceptable or it might be the reason relatively few people use such systems. (Or both.) There is also an argument that requiring tags on every page means you get incomplete results when someone occasionally leaves one out by mistake. But I think this applies more to site analytics than testing. With testing, the number of tags is limited and they should be inserted with surgical precision. Therefore, inadvertent error should not be an issue and the technical people should simply do the insertions as part of their job.

I’m kidding, of course. If there’s one thing I’ve learned from years of working with marketing systems, it’s that marketers never want to rely on technical people for anything—and the technical people heartily agree that marketers should do as much as possible for themselves. There are very sound, practical reasons for this that boil down to the time and effort required to accurately transfer requests from marketers to technologists. If the marketers can do the work themselves, these very substantial costs can be avoided.

This holds true even when significant technical skills are still required. Setting up complex marketing campaigns, for example, can be almost as much work in campaign management software as when programmers had to do it. Most companies with such software therefore end up with experts in their marketing departments to do the setup. The difference between programmers and these campaign management super users isn’t really so much their level of technical skill, as it is that the super users are part of the marketing department. This makes them both more familiar with marketers’ needs and more responsive to their requests.

Framing the issue this way puts SiteSpect’s case in a different light. Does SiteSpect really give marketers more control over testing and segmentation than other products? Compared with products where vendor professional services staff sets up the tests, the answer is yes. (Although relying on vendor staff may be more like relying on an internal super user than a corporate IT department.) But most of the testing products do provide marketing users with substantial capabilities once the initial tagging is complete. So I’d say the practical advantage for SiteSpect is relatively small.

But I’ll give the last word to SiteSpect. Larry told me they have picked up large new clients specifically because those companies did find working with tag-based testing systems too cumbersome. So perhaps there are advantages I haven’t seen, or perhaps there are particular situations where SiteSpect’s no-tag approach has special advantages.

4 comments:

As a SiteSpect user, I can speak to the difference between SiteSpect's system and page tagging. My site is a vertical search portal for online degrees and programs. The site is 100% dynamic, with all pages built using custom business logic, a relatively sophisticated content management system, and a range of page templates. We do extensive testing of page variations to optimize both user experience and conversion.

To perform testing on items on a specific page or page category using the Javascript approach, one of the following must be modified with page-specific tags: the codebase that implements the CMS, the page content itself, or, typically, the templates that organize the pages. This poses two problems.

First, those alterations impact the site itself - it requires change to set up the test and change again to tear it down, even if the result of the test is that no change is desired. An axiom of development (software and content alike) is that every change offers the chance to introduce error. So this approach carries implementation and rollback risk to the production site.

Second, we implement a development queue and follow a strict release process to control all changes to the codebase and page templates, as any error in those elements can impact performance across significant areas of the site. Testing using Javascript tags has to be inserted into this flow and managed in concert with the main flow of site development - leading to less flexibility and more rigid timelines.

Contrast this to the proxy approach. Tests are set up on the SiteSpect proxy using URL and page-level pattern matching to define pages and elements to change. When a user becomes part of an experiment, our production site serves unaltered pages to the proxy. The proxy alters each page as required per the experiment rules and forwards it to the user, tracking information in the process. The benefits are exactly the problems above. First, absolutely no change is made to our codebase, templates, or content. Second, test development and QA can proceed independent of site development, with checkpoints as appropriate to the level of change being made.

The potential disadvantage to the SiteSpect approach is loss of control over the network path between your server and your users. Although we were initially concerned about this, it has not proven a problem in our implementation to date.

Thanks Claude. Always good to hear from a happy user. I do wonder, though--is your quality control work on tests through SiteSpect really less rigorous than the same QC you'd do with changes via Javascript tags? Seems to me you have to do the same things either way. Do you have experience with products other than SiteSpect for this sort of thing?

The difference in schedule flexibility is not because QC is lighter for the experiments themselves, but because the experiments do not have to be worked into our main codebase release schedule (as page template changes would). We have a lot of active code projects going on at any given time, and they compete to get into each SW release cycle. Allowing UI A/B testing to proceed independently has been a big win for us.

I have not used other testing products, but did review a number before we settled on SiteSpect.