This Simple Data-Scraping Tool Could Change How Apps Are Made

Ryan Rowe and Pratap Ranade have spent the last 6 months building Kimono. Photo: Kimono Labs

The number of web pages on the internet is somewhere north of two billion, perhaps as many as double that. It’s a huge amount of raw information. By comparison, there are only roughly 10,000 web APIs–the virtual pipelines that let developers access, process, and repackage that data. In other words, to do anything new with the vast majority of the stuff on the web, you need to scrape it yourself. Even for the people who know how to do that, it’s tedious. Ryan Rowe and Pratap Ranade want to change that.

For the last five months, Rowe and Ranade have been building out Kimono, a web app that lets you slurp data from any website and turn it instantly into an API. Using a bookmarklet, you highlight the parts of a site you want to scrape and Kimono does the rest. Those with programming chops can take the code Kimono spits out bake it into their own apps; for the code illiterate, Kimono will automatically rework scraped data into a dynamic chart, list, or a simple web app. In essence, it’s a point and click toolkit for taking apart the web, with the aim of letting people build new things with it.

Excitement’s already bubbling around the potential. Kimono’s already raised money from big-name VCs like Ron Conway and its founders have had to turn down at least one offer for an early buy-out. The site’s already managing some 15,000 users–and it’s still in beta. But for Rowe and Ranade, things are just getting started.

Eliminating the Bottleneck

The idea for Kimono was born out of Rowe’s time as a developer at the design consultancy Frog, where he continually ran into the same frustrating problem. A designer would have an idea that revolved around web stuff of one sort or another, but they’d have to find a developer before they could even get a sense of how the idea might actually work. “Getting the data just to prove if these apps would be interesting or not took a huge amount of time, which sucked,” Rowe says.

“You have these situations where designers and analysts really want to do stuff with data but have no means to get it,” adds Ranade, whose most recent gig was at consulting firm McKinsey & Company. “We realized that there doesn’t need to be that bottleneck.”

To laypeople who don’t already think of the web in terms of streams, sources, or APIs, it can be hard to grasp Kimono’s potential. But early adopters are already using it for a striking variety of projects. When they noticed there was no official API for the recent Sochi Olympics, Rowe and Ranade used Kimono to create one themselves. Devs and designers took it from there, building elegant medal-tracking apps, dynamic maps that visualize when and where Olympians were born, and more.

Around the time the Kimono beta went live last month, Golan Levin, a pioneer of computational art and design, was introducing his students at Carnegie Mellon to the unglamorous first steps of any data viz project: acquiring, parsing, and cleaning data. He thought it’d be valuable to acquaint them with the process. While new tools like Temboo are making it easier than ever to work with official APIs for big-name sites, there traditionally haven’t been straightforward ways to get structured data off the majority of pages on the web. “Kimono came along and really changed that,” Levin says.

Levin himself is using Kimono to track real estate purchases in his home town of Pittsburgh. He also cited an upcoming meeting of civic-minded coders called the Pittsburgh Data Brigade, where he expected Kimono to see some use. “Pittsburgh’s information systems are so old and creaky that getting data out is really hard,” he explains. It’s a problem many municipalities face; they’re eager to open up their data but lack the means to actually open it up. Kimono could help bridge that gap.

A simple medal count app built with a Kimono API.

Democratizing Data Scraping

These use cases might sound esoteric, and in some senses, they are. But part of the ambition with Kimono is bringing data scraping to a wider audience. It’s about letting artists, historians, sociologists and more cull and combine content from various sources and present it in novel ways.

As an example, Ranade brings up Malcolm Gladwell’s theory about elite hockey players and how their success might be explained by where their birthdays fall in relation to Canada’s little league cutoff dates. A successful author like Gladwell can presumably tap a research assistant to trawl Wikipedia and collect the relevant data. A grad student probably cannot. With Kimono, however, she could amass a list of Wikipedia URLs, point Kimono to the “date of birth” and “place of birth” fields, and let it corral the data for her.

This sort of birthday/little league cutoff connection isn’t going to be made by a random developer, Ranade posts, but rather by a person who has “domain knowledge” in that field. “They might not be a programmer,” he says. “But if we gave a little bit of programming capability to that person, how could they look at the world in a different way?”

Looking at the World in a Different Way

In the short term, Rowe and Ranade plan to make money by charging users depending on how many APIs they use and how frequently they update (right now the service is in beta, and anyone can make however many APIs they want). They’ve already heard interest from a number of corporate clients, who see Kimono as a means to free the flow of data between departments and project teams without relying on an internal IT team to act as translator in-between.

But the duo is already thinking even bigger picture. To them, Kimono’s greatest potential comes out as we move from today’s mobile phones and their attendant apps to the next generation of wearable devices and the internet of things.

Kimono’s greatest potential comes out as we move to the next generation of wearables.

“Smartphones are only a transitional point,” Rowe says. “From there we go to smartwatches and Google Glass and other ways of interacting with data around you that don’t involve a screen. And to get from there to there to there you need to package up web data and make it consumable in all these different contexts. We’re trying to position Kimono to be the framework for that conversion.”

“When the killer apps finally start coming out for things like smartwatches and glasses, they’re not going to be made by the companies that have the most interesting data,” he continues. “They’re going to come from the developers and designers who are thinking about it a little bit differently.”

If we suspend our doubts for a moment and peer into this crystal ball, Kimono starts to look like something very big indeed. In the scenario Rowe lays out, it takes root as a sort of connective tissue for an entirely new class of interactions and experiences–something like a nervous system for the internet of things. You could imagine pointing Kimono at not just websites but other sorts of streams, making objects react to sound, say, or building applications that respond to live video feeds.

At that point, you’re well beyond the esoteric realm of web scraping. “The ability to turn a website into an API is a very powerful thing,” Rowe says. “Being able to turn anything into an API is epically powerful.”