You don’t have to be a management consultant to know that we are all besieged by information, but it’s the consultant’s job to come up with a snappy buzzword to describe the phenomenon, and they have settled upon the catchall term “Big Data.” Everything from the most trivial details of our personal lives to highly sensitive information at work is now stored and catalogued in bits and bytes; Big Data refers to the deployment of these vast troves of information to make businesses more efficient and responsive to clients and consumers.

From healthcare to finance to professional sports, data is being collected and analyzed like never before — but much of it goes on behind the scenes where the average person may not even notice. The retail sector, however, is different. By definition it interacts with average folks in a way that few other industries do, and retailers are interested in learning as much about their customers as they can. In the process, they are radically altering the buying experience for customers — both online and, increasingly, also in the world of bricks and mortar stores.

If you’ve done any shopping online recently – you’ve probably already seen Big Data in action. We’ve all experienced it: You go shopping for a pair of shoes online, put them in your virtual shopping cart, but then for some reason change our mind. Afterwards, seemingly every site you visit features an ad for that very pair of shoes at that same online store. The reason? Online retailers can give you a virtual identification number and track you as you go from site to site, and purchase targeted ads for products they already know you’re strongly interested in.

Based on a user’s behavior, sites like Amazon can present special offers or alert users of products they might not have otherwise been aware. According to a report issued last year by McKinsey Global Institute, Amazon has had tremendous success by using data it has collected to discover what additional products its users are likely to buy. As the study noted, “Amazon reported that 30 percent of sales were due to its recommendation engine.”

But the proliferation and maturation of information technology hasn’t only aided business. According to Paula Rosenblum, a retail analyst at RSR Associates, the ubiquity of Internet access and smartphones has given consumers more bargaining power than ever. “I don’t think people realize how much the consumer is in control these days,” she says. Consumers are no longer bound by geography — the handful of stores offering a specific product within driving radius — when searching for the best price or service. Even when they are physically shopping in a store, smartphones allow them to quickly and easily check for better prices elsewhere.

In the era of Big Data, it’s those physical stores that seemed destined to be left out in the cold. But the vast majority of purchases – somewhere around 90% — still occur in a traditional retail setting. And brick and mortar retailers are looking towards big data to help them stay relevant.

One company that hopes to give traditional retailers the kind of analytic tools available to ecommerce firms is RetailNext. The firm has developed a computer program that uses a store’s security cameras to give managers all kinds of information about how consumers interact with the store. Using this program, Retail Next can show exactly how many customers are in a given store at a time, which parts of the store they explore, which specific items customers spend a lot of time perusing — and which they do not. RetailNext can combine this information with other variables like staffing levels, weather, product assortment and placement to determine what does and doesn’t boost sales. Luxury retailer Mont Blanc has used RetailNext’s services to improve its staffing levels and its product arrangement within its stores, increasing same-store sales 20% in the process. Retailers like American Apparel and Family Dollar have also successfully utilized RetailNext’s services to improve the layout of their stores and increase same-store sales.

In fact, it’s possible that traditional retailers could one day have a better understanding of their customers than ecommerce firms do. That’s at least what Tim Callan, Chief Marketing Officer at RetailNext, argues. He says that for customers in physical stores the “decision making capability is infinite, while there are only so many things they can do online.” In other words, given the right tools, a retailer can glean much more about a shopper from watching her peruse a traditional retail aisle than he can watching her click through links on a webpage. His firm is working on computer programs that can accurately distinguish — through video cameras — whether a shopper is male or female, and believes in the future this sort of technology could interpret customers’ facial expressions and other gestures that will help retailers understand why someone did or did not buy a certain product.

Inevitably, as retailers use Big Data to learn more about their customers — especially through video — privacy concerns will be raised. But as Callan argues, “video surveillance has been around for thirty years, and people assume that it’s necessary for public security.” Indeed, consumers have come to take it for granted that a camera is watching their every move in a store.

On the other hand, it’s one thing to for consumers to assume that cameras are watching, and another thing to have your every movement picked apart by a computer for marketing reasons. Pam Dixon, Executive Director of the World Privacy Forum, thinks that these types of methods are a breach of consumer privacy expectations, and that companies who use these types of analytics need – at the very least — to be forthright about their practices. “I think it’s absolutely crucial for these companies to come clean with the public and disclose what is happening,” she says.

While privacy issues related to in-store use of video cameras has yet to cause much of an uproar, another component of retail companies’ collection of data – smartphone tracking – has been a hot-button issue. The Wall Street Journalconducted a study in 2010 of over 100 popular smartphone applications and found that applications were collecting information about, “location, unique serial-number-like identifiers for the phone, and personal details such as age and sex,” and sending that information to marketing companies.

Though privacy concerns about tracking users on smartphones has made many in the public uncomfortable, marketing intelligently through smartphones is believed to be a crucial next step for many retailers. Smartphones can bridge the gap between the online and offline worlds, because users always have their phones with them, even when they’re not browsing the Internet. Retailers can learn about a customer through their online shopping behavior and then offer them short-term discounts through a cell-phone when the consumer is near that stores brick-and-mortar location.

Unfortunately for consumers who are made uncomfortable by all this surveillance, there aren’t any federal laws that block this kind of behavior, and the federal regulators haven’t been entirely successful at forcing companies to be forthright about their practices. But for brick-and-mortar retailers who wish to serve their customers better by knowing them better, there has never been a more exciting time.

This is the fourth in a series of articles on the future of retailing. You can read the first three installments here, here, and here.