A year ago, Segment faced a major engineering challenge. They were in the midst of launching a new product designed to help companies personalize their own users’ experiences: Personas.

Personas needed to satisfy two very different query patterns. On the one hand, Segment had to provide fast, millisecond access to billions of rows for real-time user personalization. On the other, they also had to run aggregate computations across hundreds of millions of rows. And it had to work across the billions of new data points ingested every single day.

It was clear that using a single datastore just wouldn’t work for these two very different use cases crunching this amount of data. In this talk, Calvin French-Owen, CTO at Segment, will explain how Segment created its own Lambda Architecture, leveraging Cloud Bigtable for fast, random reads and BigQuery for crunching large analytics datasets. He’ll also discuss the decision framework used to choose these datastores, from cost analysis with tools like Dynamo, to the specific use cases served by both BigQuery and Cloud Bigtable.

The company now stores tens of terabytes in multiple systems, querying Cloud Bigtable with less than10ms latency, and scanning tens of gigabytes every second to run their batch computations. Their new product (Personas) has had significant success since its launch six months ago and has ingested 260 billion data points to date.

Google Cloud Platform, offered by Google, is a suite of cloud computing services that runs on the same infrastructure that Google uses internally for its end-user products, such as Google Search and YouTube. Alongside a set of management tools, it provides a series of modular cloud services including computing, data storage, data analytics and machine learning.