Answers.com (“Answers”) is the leading Q&A web-site. It is ranked among the top 20 web-sites in the U.S. and delivers millions of answers daily. Their data centers faced two challenges:

Keeping end-users’ query response time low. This challenge grew as data volume and site traffic increased.

Keeping replication performance high. Users are constantly updating data and slow replication from the Master to a read server could result in responses containing stale data.

According to Dan Marriott, Director of Production Operations, these challenges created a constant battle for backend resources to tune queries for performance.

To overcome these challenges, Answers maintained an innovative and scalable MySQL database system that allowed them to drop in servers to horizontally scale performance.

A typical Answers database cluster consisted of a master database server, which handled all writes and distributed the load of incoming user requests, plus around 20 read (or slave) database servers that replicated in real time from the master.

This design ensured their customers got best-in-class performance, but it saddled them with increased operating costs for power and cooling, rack space, and server administration, and was reaching the limits of the design’s performance capabilities. However, at the time the system was implemented, this solution was their only option.

The Solution

On a data storage specialist’s recommendation, Derek Balling, Answers’ Data Center Manager, visited the SanDisk booth at a conference. After just a few minutes discussion he realized “SanDisk’s Fusion ioMemory solution was perfect for what we did in our databases.”

Amping Up Replication
Derek got his first taste of what the Fusion ioMemory ioDrive® cards could do as he was configuring two test servers, one with a card and one without. The wait for the master to update the read server without the card took over six and a half hours. The wait for the server with the card was just twelve and a half minutes.

“We ended up doing it a couple times because we thought, ‘That’s just not possible,’” Derek said. “It was what we call the “Aha!” moment,“ Dan added.

Replication Update Time

With SanDisk

12.5 minutes

Without SanDisk

6.5 hours

31x

Faster

Empowering the Database
To test the systems, they ran complex queries against the test databases. A server without the card reached around 350 queries per second, at which point the database began locking up. Comparatively, a server equipped with the ioDrive card could run 3,500 queries per second, with no problems whatsoever. Improving their database capabilities to nearly nine times more queries per second improved its web application’s average response time by 30%, a benefit particularly valuable to their end users.

Query Performance Improvement

With SanDisk

3,500

Without SanDisk

350

10x

Faster

Improved query performance from 350 to 3,500 queries per second.

Slashing Disaster Recovery Time
Answers discovered an additional benefit as they were setting up a redundant data center located in Utah to its New Jersey site. Installing a 320GB ioDrive card into one of their key master database servers in Utah reduced the time it took to copy and uncompress the other data center’s backup from one hour and forty minutes to under 15 minutes. Combined with the greatly reduced “catch up” times of the read servers, this reduced the worst-case data center recovery time from eight hours to less than sixty minutes.

Scaling for Server Consolidation
While the benchmarks showed Answers just how much the cards could improve performance, they learned just how scalable their performance was when they deployed them in production. Dan initially deployed five ioDrive card-equipped servers in a cluster that previously had 19 slave database servers. This met Answers’s comfort level for high availability and growth requirements, and would accommodate server failures and taking servers offline for maintenance.

Derek and Dan soon saw they had more than ample application performance and, as a test, reduced the number of active servers in the cluster to two, “with no appreciable change in user experience. Though we did not try it, I’m fairly confident just one server could have handled the traffic of the site at that time,“ Derek said. “It was literally astonishing to see. The cards completely resolved our database replication and query response issues,” Dan stated.

Enhancing Reliability
The server consolidation improved more than the performance of its database layer, it also enhanced its reliability. With 75% less hardware, its database layer had far fewer mechanical failure points. Additionally, the ioDrive cards added no embedded processors or memory to processing, reducing the risk of soft errors and coding errors. Finally, SanDisk’s patent-pending self-healing “Flashback Protection,” with N+1 redundancy ensured they would never lose data due to NAND Flash failures.

Immediate and Continuing ROI
By repurposing 75% of their read servers, per cluster, Answers was able to cut capital expenditures in other areas of its data center, delivering an immediate ROI that surpassed their purchase price. On top of the capital cost returns, the server consolidation reduced operating costs for power, cooling, rack space, and server administration by 75%.

Dan declared, “The value of the cards is tremendous. By reducing the servers I needed from around twenty to five, I estimate my capex per cluster improved by the cost of about ten servers, plus I have significant headroom for future growth. Also important, I reduced my database layer’s operating costs by 75%.”

Above the capital and operating cost returns, Answers gained hours of valuable back-end database team time that they could devote to actively improving applications rather than tuning the database.

Easy Migration
The migration path to the ioDrive cards was simple. The cards drop into Answers’s existing blade system, allowing Derek to leverage the benefits of their horizontally scalable architecture.

To update a server, Derek takes it out of rotation, installs the new card, moves the data onto the card, and brings the server back up. The database comes online within minutes of the time it boots. The entire process takes around eight minutes per server.

System Before

20 x blade servers, 32-48GB RAM

OS: CentOS 5 - 64bit

Applications: MySQL 5.0.x

Disks: (2) 146GB Hard Drives (SAS/10k)

System After

Repurposed 15 blade servers

Added one ioDrive 320GB to each server

Performance per Rack Unit (Density)

With SanDisk

3,500

Without SanDisk

350

10x

Faster

Improved query performance from 350 to 3,500 queries per second = 10x, in the same rack space.

Summary

Implementing the Fusion ioMemory solutions gave Answers the following benefits:

Needless to say both Dan and Derek are thrilled, and are looking at deploying additional cards in a number of Answers’ other high I/O database environments.

Dan said, “Prior to purchasing the ioDrive cards, we were adding database servers to this cluster every two months or so. I estimate that two of the five ioDrive card- equipped servers can handle the same load that required around twenty of the non- equipped servers. The three other servers give us enough headroom that our traffic can double before we need to expand this cluster.”

About the Company

Answers Corporation (NASDAQ: ANSW) owns and operates Answers.com, the leading Q&A site, which includes WikiAnswers and ReferenceAnswers. The site supports English, French, Italian, German, Spanish, and Tagalog (Filipino). WikiAnswers is a community-generated social knowledge Q&A platform, leveraging wiki-based technologies. Through the contributions of its large and growing community, answers are improved and updated over time. The award-winning ReferenceAnswers includes content on millions of topics from over 250 licensed dictionaries and encyclopedias from leading publishers, including Houghton Mifflin, Barron’s and Encyclopedia Britannica.