It is one thing to try and find a needle in a haystack, that's easy and is a task perfect for magnets, but finding a needle in a bunch of needles can be a trickier proposition. But this was one of the problems under discussion at the Splunk's SplunkLive! customer event in London last week.

Tech Pro Research

Splunk's analytics software helps businesses make sense of the data that is inside their systems and to use it in intelligent ways to answer 'what-if' questions.

At the event Aleem Cummins, the release manager at retailer John Lewis, offered a case study of when this sort of analytics can be useful. He explained that an issue had come up last year where thousands of customers were being denied an online service for no readily apparent reason. There was a fault but nobody could identify the fault, why it happened – as it did on a regular basis – let alone how to fix it.

Enter Splunk and, as Cummins explained, the issue was so frustrating he was willing to try anything that might fix it. "Within weeks of installing Splunk we absolutely had our money back," said Cummins. How difficult was the issue? "It wasn't a case of finding a needle in a haystack it was a case of finding a needle in a bunch of needles."

By using Splunk, he said, the retailer could, "drill down until we could get enough information to be able to re-create the problem and then once we could re-create the problem, we could then see what we had to do to fix it."

The retailer is also using the software in other areas of the business. As his colleague, Paul Adams, operational intelligence lead at John Lewis pointed out, the one issue with Splunk was that the original version did not support clustering. However, this was addressed in Splunk 6.1, which was released earlier this month, and had been available earlier than that in beta.

"So we could see the potential of what we could do but what we needed was a big event where we could use Splunk to really show the organisation what could be done, something like a Black Friday or a Cyber Monday or Christmas," said Adams. "For us it was Christmas last year."

Christmas 2013 went very smoothly for John Lewis and it was a particularly good for on-line sales. Cummins says that a substantial part of the credit for that should go to Splunk. "We could not have done it without the software," he said.

While Splunk has been very successful for John Lewis it is a piece of software not without competitors, especially the open source, Apache Hadoop. Hadoop’s take on the needle in a haystack issue is that the problem now is trying to find a needle when nobody has put it there in the first place.