“We make some of the best known brands in the world, and those brands are used by 2 billion people every day.”

– Unilever

Few industries reach out to and impact the lives of more consumers every day than the Consumer and Retail Industries. Yet the industry’s transformation from Mass Marketing to 1:1 Marketing is mostly all hype with dramatically negative bottom line results. Let’s begin by considering the current state of the Consumer and Retail industries:

Despite billions of dollars spent on market research every year, big food Companies were slow to respond to changing consumer tastes from packaged and processed food to fresh and wholesome alternatives. Some 43% of US consumers are planning to eat less processed food in the next year according to Mintel’s American Lifestyle 2014 report.1 The bottom line impact? As per Fortune’s Special Report: The war on big food, “Major packaged-food companies lost $4 billion in market share alone last year, as shoppers swerved to fresh and organic alternatives.”

According to Ad Age, “Quite simply, big brands are losing one of their most valuable assets: consumer trust. And the fight to regain it will shape the industry for years to come.” 2

What Do Consumers Really Want?

In the past direct consumer interactions were mostly restricted to calls or complaints into the call center. Most retail and consumer companies were surprised when consumers began to reach out and communicate to them directly via digital and social media. Responding to individual consumers was not their expertise. In fact, consumer conversations were unheard of beyond regimented settings such as facilitated focus groups and primary research interviews.

Today’s digital consumer is much more vocal and diverse. What do they expect from their favorite brands? The four most common needs include:

1. Personalization: Investing in providing products and associated services and information that are specifically relevant to them as individuals. Practically speaking, this could mean a shift from what’s good for the company i.e. “Packaged and Processed” to what’s good for me “Fresh and Wholesome.”

2. Two way dialogues: Communicating with them on their terms, be that time, channel platform or message. Consumers seek a thoughtful dialogue with brands, where consumer opinions are solicited and their responses are acted upon

3. Omnichannel: Clear and transparent access, via multiple channels, with a single and consistent proposition and treatment strategy be it online or in-store

4. Permission-based marketing: Consumers dislike general and unsolicited push marketing, particularly through personal devices such as phones and tablets. They are also much less likely to respond to generic messages or mass media

Why Have Retail and Consumer Companies Failed at 1:1 Marketing?

Call it what you may – but ultimately the industry shift from mass marketing to 1:1 (or personalized, individual) marketing calls for a major transformation in how business is done today. There are three major challenges to address:

Challenge 1: How do you manage the volume, variability and ownership of data?

For a retail or consumer company gaining the right insights and responding to the billion consumers who buy your products each week with the right personalized messages is key. The challenge is volume.

Data on consumers comes from multiple external sources including consumer panels that may be owned by the syndicated data providers, retailers who own in-store loyalty data, social and digital data that you may need to buy from Facebook, government demographics, as well as internal research done across brand, consumer insights, and sales departments. The challenge is obtaining and integrating the consumer data.

Challenge 2: Who responds to the consumer?

Consumers expect not only a seamless experience but also a personalized response versus the generic mass-content they get from TV and cable advertising.

Responding via personalization to create a seamless shopping experience over any channel calls for the integration of multiple sources of heterogeneous data to derive insights and create more targeted responses.

Challenge 3: Existing IT systems just weren’t built to respond to a billion consumers live!

Retail systems largely utilize Relational Database Management Systems (RDBMSs) first developed in 1970s and driven by Oracle, IBM DB2, Microsoft. Using SQL (Structured Query Language) as the programming language for managing the data, structure is actually the primary challenge of using an RDBMS to store and manage content. Because in order for RDBMS to perform well, the data must first be mapped with a pre-defined schema, or set of constraints, that defines how it is structured and organized for analysis. Unfortunately for industry users of RDBMS, consumer data is not just vast and variable, it is also largely unstructured and does not fit neatly into rows and columns. This is a problem because today’s consumers need to be responded to individually and live at the point of purchase in-store or online. Take the case of Jen Dough and Joanne Deogh, both 30 years old and living in the same apartment building. Joanne is single while Jen is a married, new mother. If your loyalty database was not updated to capture Jen’s unique new status as a parent, these women are not likely to be offered personalized, differentiated promotions at the point of purchase – and you are not likely to succeed in your marketing efforts. Agility and the ability to encompass (and make sense of) unstructured data is a crucial aspect to any effective, enabling technology for today’s consumer data. Designing the perfect schema with rows and columns for a billion consumers with the goal of responding to them each as individuals just won’t work with RDBMS technology!

Designing a Consumer 360 Approach That Works

Ultimately consumer and retail companies need to listen, analyze, and engage with consumers 1:1 and live at the point of the transaction whether that’s online or in-store. We call that Consumer 360. Clearly traditional mainframe or RDBMSs lack the flexibility and scalability to support Consumer 360.

NoSQL Represents a Revolutionary Solution for Consumer 360NoSQL (Not Only Structured Query Language) technology represents a transformational change in perspective. Instead of getting the schema just right before doing anything else, NoSQL advocates loading up the data first and then seeing where the problems lie. This problem-oriented approach focuses on how the data will be used (queried) rather than how the data must be structured to fit within a traditional RDBMS. For Consumer 360, this shift means you would not have to spend a year trying to figure out the right data model and perfect schema to analyze and store data on a billion consumers. Instead, you can load the data, have it indexed automatically, and then search and query it for emerging trends and demand signals.

Why MarkLogic is the leading NoSQL Database for Consumer 360
In a nutshell, there are four key value differentiators MarkLogic brings to retail and consumer companies, including:

1. Enterprise Grade PerformanceFirstly, and perhaps most critically, we’re talking about consumer data in your database. This is all about handling huge volumes of data while ensuring transaction integrity and security. Most open-source vendors fail at just this crucial requirement. MarkLogic’s platform was designed to be enterprise-ready from day one – which means that you have all of government-grade security, high availability, disaster recovery and transactional consistency you need for your mission-critical consumer applications.

2. A Superior Operational Data WarehouseA Consumer 360 calls for real-time, operational capabilities. You need to recognize and respond correctly and accurately to Jen Dough the moment she enters your store or online. Making the right product and promotional recommendations to her is what will close the deal. Simply put, your data warehouse must be transactional and operational to enable websites, e-commerce and other applications in-store while at the same time enabling analytics and storage of terabytes and potentially petabytes of data.MarkLogic was the only NoSQL database appointed to Gartner Leaders Quadrant for Operational Database Management Systems. Also, in the 2014 Gartner Report: Critical Capabilities for Data Warehouse Database Management Systems, MarkLogic came out ahead of all other vendors in the customer rating for Operational Data Warehouses.

3. Search and Query CapabilitiesMarkLogic has a built-in search engine that no other database has. You can fine-tune search across structured or unstructured data—including full text, large binary and geospatial data—getting lightning fast results and advanced features such as alerting. From a consumer or business user’s perspective this makes your website and e-commerce business much more appealing because they can finally find the products they seek. MarkLogic’s semantic capabilities also enable storage and search for logical product linkages (e.g., ingredient linkages for dinner recipes or the right toner for my HP Printer). With semantics you can store and query billions of facts and relationships, and infer new facts – making your data and your database much more intelligent.

4. Application Services for Consumer AnalyticsFor a Consumer 360 to work you want to build the right applications on top of the data. MarkLogic’s integrated HTTP Server has full-featured Java and REST APIs, among many others, to make managing data and building two and three-tiered applications easy, giving your developers faster access in the language they’re most comfortable with using.

The consumer and retail industry are at a critical crossroads when it comes to listening, analyzing, and responding to consumer needs. Sure, part of the problem has been internal organizational structure and change management issues. But the biggest issue has been the lack of credible technology solutions to deal with the problem. MarkLogic represents a transformational approach to this technology challenge – and the business case has never been clearer.

At the Hollywood IT Summit on May 14, Michael Martin – SVP Product, Technology and Operations, NBC Entertainment Digital – spoke about the unique opportunity his team had to reinvent the way fans interacted with one of the network’s most famous and widely loved shows, Saturday Night Live.

To accompany the 40th Anniversary of Saturday Night Live, the network’s digital team wanted to do “something special” with all the great content the show has generated over the years. During that broadcast, which became the most successful live entertainment broadcast in the past 10 years, some of the best content from every era was featured, including the Bass-o-matic of the 70s, Wayne’s World of the 80s, Celebrity Jeopardy of the 90s and digital shorts from the 2000s.

This range of content mirrored what Martin and his team found when they researched the impact of the show: Everyone knew the show and was familiar with its great content across the years and most people even had a “golden era” of favorite shows. The problem was that, with the show’s history and consistency of quality, everyone had a different golden era! Also despite the show’s broad, multigenerational appeal, online viewership was almost always of the more recent shows. Even though much of the show’s library was online in one way or another, the audience just wasn’t able to reach the content they liked best. To bridge this gap, Martin and his team set out to build a unique digital experience. They envisioned an app that would not only “reconnect all the fans to all of the content,” but also would give them an entirely new experience consuming SNL content.

To get started, the team had to address one of its biggest challenges: The data that would drive the experience. Part of the reason the SNL library wasn’t getting a lot of traffic is that the metadata surrounding it didn’t shed light on the content itself. The most reliable data consisted of dates, titles and characters. But this data just wasn’t enough as the sketch titles were often deliberately bland to conceal the joke (e.g., CPR Class), some characters’ names were never known (anyone remember Dooneese Maharelle?) and show fans seldom knew when sketches were actually aired.

What Martin and his team needed to do was capture what we all actually do know about the show – every character, cast member, season, impersonation and sketch and also the characteristics of each item. This data taken together would make up the world of the show and help people get their favorite content. For instance, for Dooneese Maharelle it is important to know that she has “tiny hands” and to actually access her clips you would have to know that she was part of repeating series called The Lawrence Welk Show. The world of data about the show would also have to fit together so that the team could create new experiences. For instance, at the HITS conference Martin pointed out the famous example of Sarah Palin who is a character (played by Tina Fey), a guest (when she appeared on the 40th Anniversary special) and also a cast member (for her role impersonating Tina Fey impersonating Sarah Palin). They needed to be able to capture all these different concepts (character, guest, cast member) and how they fit together to let you find just that one sketch and do the same for all the different concepts and themes of the show.

In a traditional RDBMS, the method for capturing this type of data would be to create a record of all the clips with a tag for each item (Character = Dooneese Maharelle, Characteristic = tiny hands, Cast member = Kristen Wiig, etc.). And as each tag drives queries, just getting the Sarah Palin record straight so you could find the right clip would take quite a bit of data modeling! Also, with this approach, every clip would have its own mini world of data but it would never fit together. An alternative solution was needed.

So, the SNL team turned to smart content, the mix of metadata and semantics, as a means to model data. The world of the show is now described using a semantic ontology in which individual characters are all their own concepts. From Sarah Palin to Dooneese Maharelle, every character and all their specific characteristics are managed as separate semantic data in the same manner as cast members, seasons, themes and the many other types of data the team put together to describe the show. Each record has a metadata record, but instead of having lots of fields, this data has the core information (dates and titles) and then has semantic triples that represent where that clip belongs in the world of the show.

To put all this data into action, the NBC team used MarkLogic’s Enterprise NoSQL database. With MarkLogic, the ontologies are stored as semantic triples and the metadata records are stored as NoSQL documents with embedded triples that link to the ontologies. This combination of NoSQL and semantic data is used to power the searches and queries that drive the SNL 40th Anniversary app (now available for iPad and Android) and the online site.

This data and the ability to query is dynamically within MarkLogic also helped the team solve a key puzzle in the way they wanted users to experience the content. Instead of having the entry point be a search or browse, they wanted to the app to simply start to play video and let the users swipe back and forth through videos that the app recommends. To get the right video in front the user for that key first impression, the team needed a compelling predictive engine. The semantic data representing the world of the show gave the developers the data they needed to string together likely clips. But the team wanted to go even further, so in addition to all the data about the show, they also captured user watch event data logged as semantic triples (and captured anonymously or as part of user profiles and favorites). To create dynamic personalized recommendations, the team leveraged MarkLogic’s ability to query the semantic data, the user information and the metadata, maintain data integrity and with transactions and execute computational complexity at scale. The result is that the app creates a dynamic recommendation to each individual user and responds with new recommendations based on what the user was doing in their current session.

Wrapping up his discussion at the Hollywood IT Summit, Michael Martin talked about the success of SNL’s new app. Having been featured so far in over 1200 articles, the app has greatly increased fan engagement with the show by delivering over 100 million videos to viewers across the globe. And the users have also changed how they interact with the clips. According to Martin, there is no more “one and done.” Instead, the network is seeing that with the immersive watch experience, fans are consuming the content in longer sessions and are making a “meal” out of all the great SNL “snacks.”

Having changed the way users experience great content, what’s next for the NBC Digital Entertainment team? Only Martin and his team know for sure, but we’re confident it won’t be the same old app experience ever again.