Effective data mining is all about connecting the dots. In the days after two men were arrested for going on a shooting spree in the Washington D.C. area, word got out that witnesses had spotted the snipers' car. What's more, police had previously run the car's license plates through their system several times. But authorities never made the connection, and the men were eventually arrested based on other information. And if the terrorist attacks of September 11, 2001, have taught law enforcement anything, it's that authorities know more than they think they know about potential criminalsbut they don't always know how to put the pieces together.

Nearly every organization has a database, and most have more than one. If organizations can piece together information from these various databases, they can catch more criminals, sell more products, or simply operate more efficiently.

"It's ironic. The more organizations know about themselves and their customers, the less they know what to do about it," says Usama Fayyad, CEO of data-mining company digiMine. "Everyone understands the value of data, so they store a lot of it. But they can't always get to their data. And when they get to it, they have trouble making sense of it."

Companies like digiMine, Autonomy, ClearForest, and iPhrase Technologies sell software that uses complex algorithms to look for relationships between data points that are either spread across multiple data warehouses or collected into one.

Government agencies can certainly benefit from data mining. In fact, Autonomy and other companies are supplying the U.S. Department of Homeland Security with software to share and analyze data. But retailers seem to use the technology best. Microsoft's Web site for its popular game console, the Microsoft Xbox, uses digiMine to study Web activity and map it against marketing data. The goal is to sell more products and tailor the site to individuals' tastes.

"The information we pull together is central to Xbox marketing," says Scott Pickle, Xbox online's site manager. "It's important for us to be able to segment users and understand their behavior in order to give them what they want."

Web logs of Xbox's site visitors are stored in a data warehouse that digiMine hosts, while customers' personal information is stored at Microsoft. Site managers in 27 countries, as well as Xbox marketing staff, can initiate a data query from a Web browser. digiMine then extracts the relevant data, such as players of the game Halo who have read the site's article on Star Wars: The Clone Wars and indicated an interest in buying any game within the last ten days by clicking through to one of Xbox online's retail partners. Once digiMine returns the information, Xbox marketers can map the data against its internal database to create targeted e-mail offers.

Some data-mining solutions are customer facing, offering friendly interfaces for self-service customer support and research. Charles Schwab uses data-mining technology from iPhrase to power its online search tool. As a simple example, someone can ask the system, "What is the difference between a stop order and a limit order?" Rather than just showing entire documents that describe types of orders, iPhrase mines data files and presents only the relevant information.

"Our customers normally have complex questions that a simple search engine can't answer," says Leigh Hood, Schwab's director of product development. "This solution actually does research by actively pulling together different information from various sources." The iPhrase solution saves Schwab $125,000 a month by letting customers conduct advanced research into companies and investment options without calling a Schwab representative.

In the end, it doesn't matter if J.Crew online is using digiMine to match potential buyers to the latest line of clothing or the FBI is using ClearForest software to identify terrorists, data mining works the same. By sifting through mountains of information and analyzing relationships, data-mining solutions help make sense out of an increasingly data-driven world.

Get Our Best Stories!

This newsletter may contain advertising, deals, or affiliate links. Subscribing to a newsletter indicates your consent to our Terms of Use and Privacy Policy. You may unsubscribe from the newsletters at any time.