Big Databases Are Big Challenges

As databases bulk up, tasks such as administering access and managing data integrity become more difficult

The size and workload of transaction-processing and decision-support databases have increased rapidly--doubling in some cases--in the last two years, according to a Winter Corp. survey of the operators of the world's largest, most-heavily used database systems. That continuing growth poses challenges for those charged with managing these behemoths.

At peak workloads, the U.S. Bureau of Customs and Border Protection's database processes 51,448 transactions per second, nearly twice its peak of 26,655 TPS in 2001. It has the distinction of being the hardest-working transactional database in the world, according to survey results released earlier this month.

"Transactions and queries are getting richer and more complex as time goes on," says Richard Winter, president of market research and consulting firm Winter Corp. Companies are making more use of data for cross-selling, and the number of transactions is increasing as more business is conducted in real time. Rising workloads are a symptom of success for decision-support data warehouses, Winter says, "because it means [businesses] are getting more value out of them."

The survey, conducted in September, also identified the largest decision-support and online transaction databases in the world. France Telecom topped the list of big decision-support databases with its 29.2-terabyte data warehouse, followed by AT&T and SBC Communications. Land Registry, a U.K. government agency, has the largest transaction-processing database with its 18.3-terabyte system, followed by telecom firm BT plc and United Parcel Service.

HARD-WORKING
DATABASES

Top
10 databases for online transaction processing, in transactions per second

Company

Peak workload (TPS)

Vendor

Bureau of Customs and Border Protection

51,448

Computer Associates

Anonymous

4,010

Microsoft

Internet Auction

3,634

Microsoft

Commander Communications

2,604

Microsoft

Scottish and Southern Energy

2,152

Microsoft

2001Outlet

1,204

Microsoft

Anonymous

1,200

Microsoft

Caixa EconÙmica Federal

1,133

Computer Associates

CheckFree

739

IBM

Land Registry

702

IBM

Data: Winter Corp.

As databases get larger and their workloads increase, managing them becomes more difficult. The job demands administering access for tens of thousands of users, maintaining data integrity, and managing the thousands of disk drives that actually store the data. Vendors have been scrambling to develop software to automate more of these chores, but Winter believes database growth may be outstripping those capabilities. "It's a huge management challenge," he says.

Winter, who provides database-consulting services to government organizations, including intelligence agencies, declined to say why the database at the Customs Bureau grew so quickly. But with increased scrutiny of people leaving and entering the country in the wake of the Sept. 11, 2001, attacks, the reasons may be obvious. The Customs Bureau, part of the U.S. Department of Homeland Security, has a transaction-processing system based on an Advantage CA-Datacom database from Computer Associates running on IBM eServer zSeries hardware.

Experian's decision-support system is based on an Oracle database running on Sun Microsystems Sun Fire servers running the Solaris operating system. Almost all of the decision-support databases with the largest peak workloads run on Unix systems, according to the survey. But six of the 10 transaction databases with the largest peak workloads run on Microsoft's SQL Server database (see chart). SQL Server was once considered fit only as a departmental database.

The survey provides a good snapshot of the current state of database usage, Winter says. But the survey is voluntary, and some companies known to have huge database systems--Wal-Mart Stores Inc. being one example--don't participate.

Among 688 respondents, 46% have deployed mobile apps, with an additional 24% planning to in the next year. Soon all apps will look like mobile apps – and it's past time for those with no plans to get cracking.