QuickSight is a web-based social media analysis tool. It essentially replaces the three steps marketers routinely need to perform:

1. Fetch, store, and cleanup data

2. Export the data into another tool -- typically Excel -- for analysis

3. Report the findings in yet another document, such as a presentation.

QuickSight handles all three of these. CitizenNet has built a really powerful server-side processing tool that fetches data (such as Facebook advertising stats, Tweets, Nielsen reporting, etc) and does the necessary ETL (extract, transform, and load) manipulations, and finally packages it together.

This data package is then presented to the UI (User Interface), which is what I lead the development of.

2) What were some of the goals? What was some of the inspiration of the design?

There were a few overarching goals:

1. Web-based. We didn’t want to have the user rely on any other tools or databases. This also leads to easy access and team collaboration for everyone.

2. Free-form. Lots of native analytics tools for marketers (like the tools built into ad buying platforms) are overblown dashboards that fit some people’s needs, some of the time -- leaving most people to search for the “Export” button just so they can do what they want. Its hard to anticipate how someone wants to explore a dataset, so we wanted something as free-form as a spreadsheet.

3. Dynamic. Even if you walk through all the typical steps, you still get something that is largely a static report. What if you are in a meeting and someone asks for just one small tweak to your report? You have to do it all over again! We wanted to be able to answer lots of questions very simply, and on the fly.

A lot of inspiration came from this tool from Square

They built a Javascript library called Crossfilter focused on handling large table operations in the browser, which we use extensively in QuickSight. I love how they make extensive use of histograms, which makes data analysis really quick.

Early Sketch of Using Crossfilter for Advertising data

Google Docs is also a great example of powerful tools that are simply done, and with great collaboration features.

2) So there are a number of tools, ranging from Excel to Tableau, and new companies focused on the marketing industry like Domo. What makes QuickSight different?

Probably the biggest difference is that CitizenNet hosts and manages all the data. When data sets gets large or complex, integrating data into these tools becomes a fulltime job all on its own. With QuickSight, you don't need a data architect to do the initial import, and it is automatically updated when more dimensions, metrics, and charts become available.

3) Tell us about some of the technologies you are utilizing in the front end.

Angular creates most of the UI that we interact with. It handles the drag and drop report creation, filtering, report viewing and remote data access that is needed for Crossfilter. Once the data arrives to the client, we pre-process it before throwing it to Crossfilter. Crossfilter adds fast filtering, and data aggregation to large multivariate datasets. This allows us to play with the data clientside in a fast way because we are working with a small partition of a dataset clientside, in memory. Lastly, dc.js (and the underlying d3.js) provides the drawing toolset for all the charts. DC is great because it is designed to work nicely with d3 and Crossfilter.

4) What were some interesting technical challenges you faced?

We talked earlier about our Crossfilter improvements being the most fun part of the project. While we have nearly a half dozen improvements, the three biggest are:

Advanced nested datasets

Custom calculated metrics

User-defined multi-dimension breakouts

With these 3 features, we could build complex pivot tables in Excel fashion, but this previously hasn’t been done in Crossfilter that I have found (in fact, a question regarding nested datasets is the highest voted question for Crossfilter on Stack Overflow).

A great example of using these three additions is in implementing tags, because a single tag can be applied to multiple dimension breakouts. However, the trouble comes in with aggregation.

This is best described in showing the CSV input you would provide to crossfilter. Let’s say you are General Motors, and you have run 3 campaigns, with three different tags.

As a Venn diagram, this would look like:

Keep in mind the “Grand Total” is a calculated metric, meaning it its a total of whatever you have displayed (not something static stored in the CSV). This means, of course, if you flatten this structure you would get an incorrect total:

From a technical aspect, this has to be done inside the Crossfilter library. We had to bucket each record in the CSV just like a pivot table. For each bucket, you can have nested child rows. And all this is recursive. Then you must sum up on every level, including the top most level. Then you let your view level of the code dictate how granular you display the data. Maybe you just show the grand total, or maybe it's a pivot table break out of all the levels.

Here’s a demo of the final result; note how the total is correct regardless of the settings.

There’s another neat trick here: we are allowing both AND and OR operations. So for example, “Video” AND “Car Model” would be just the Suburban Ads campaign:

But “Video” OR “Car Model” would actually be all three campaigns. To do this, the user would ‘conditionally’ exclude the branding campaign.

4) Anything else you want to conclude with?

This is just the tip of the iceberg -- there’s lots more new features coming, so stay tuned!