Details, details, details

One of the most difficult features to implement is the import/add progress. It’s a small detail that shows up only when you add something new, or when you import an account. However, it takes a lot of engineering time and talent to do it right.

Let’s walk through the process of importing an account. First, we have to authenticate your credentials on a third-party site. Next, we have to find your friends on MySpace, Friendster, or Flickr. Many of our users have hundreds of friends on those social networks. We check each of your friends’ content and bring back their content to your Spokeo account.

Think about how long it takes to load your friends’ MySpace pages or Flickr album on your browser. On average it takes about 1 to 2 seconds. Now multiply that time by 100, and you can see that it takes 200 seconds, or more than 3 minutes to bring just 100 friends’ updates back to you. Now you’ll wonder, why Spokeo is so quick in bringing back your friends’ content (we take about 20 seconds as opposed to 200 seconds to bring 100 results back to you). Well, the secret is "parallelism".

It does take one computer 200 seconds to import 100 friends; after all, you can’t trick the math. However, what if you can divide out the tasks to 10 computers? Now each computer will only have to independently process 10 friends (hence "parallelism"), and the time can be reduced to 20 seconds. This sounds simple, but it is actually very hard to do. When given the command to talk to each other computers may talk simultaneously or fight for the talking time. You need a complicated management system to keep everything in sync.

Building and managing this complicated cluster of computers is a delicate process like making a watch, any little crash will cause the end result to malfunction.

Now with the time cut down from 200 seconds to 20 seconds (it can’t be cut down further because of scaling issues, but that is a blog for some other time), we feel like that is still not enough. Users are accustomed to Google’s millisecond speed, if a site appears inactive for 3 seconds, people will leave. This is why we have invested months of engineering time to build a progress report that tells you exactly what’s going on. Now our computers not only have to talk to each other, they also have to talk to our users. Needless to say, that made our job more challenging, but worthwhile.

Why did we invest months of time to refine such a detail? We feel it’s the details that makes the Spokeo experience better. Both Toyota Camry and Pontiac Grand Prix drive fine and cost about 20K, but Toyota has included higher quality materials and built a more intuitive design. We feel our users deserve the little details that make using Spokeo an easy to use service. While we concentrate on bringing you cool features, we will continue to refine the existing ones. As we’ve said over and over again, we see each feature not as a one-time add on, but as an ongoing tool that can be refined and made better. Our attention to detail is a commitment to our loyal users.