Palantir: big data needs to get even more abstract(ions)

The modern computing era didn’t really start to take off until we figured out how to make operating systems that could remove a lot of the manual labor required to program early computers. According to Ari Gesher of Palantir Technologies, it’s time for a similar breakthrough in data to become more widely used.

Speaking at Structure Data Thursday, Gesher took attendees on a bit of a computer history lesson, recalling how computers once required their users to manually reconfigure the machine each time they wanted to run a new program. This took a fair amount of time and effort: “if you wanted to use a computer to solve a problem, most of the effort went into organizing the pieces of hardware instead of doing what you wanted to do.”

Advertisement

Operating systems brought “abstraction,” or a way to separate the busy work from the higher-level duties assigned to the computer. This is the foundation of modern computing, but it’s not widely used in the practice of data science.

In other words, the current state of data science is like “yak shaving,” a techie meme for a situation in which a bunch of tedious tasks that appear pointless actually solve a greater problem. “We need operating system abstractions for data problems,” Gesher said.

Palantir, a notoriously secretive data analysis firm that counts the federal government among the customers of its software, has built such abstractions into its system, Gesher said. This can allow organizations to “take teams of teams and have them all looking at the same data to tease out things you couldn’t do by looking in isolation.”