Cycle Computing spins up 10,600 instances in Amazon's cloud

High performance cloud computing company Cycle Computing is no stranger to spinning up massive clusters of servers in Amazon's public cloud, but this week the company says it recently ran one of its largest jobs ever, one that used 10,598 multi-core instances.

Cycle Computing provisioned Amazon Web Services Elastic Compute Cloud (EC2) servers for a pharmaceutical client to simulate a drug test. It took two hours to configure and ran for nine hours, for a total cost of $4,362. If the infrastructure had been built by the company, Cycle estimates it would have taken a 12,000-square-foot data center and cost $44 million. Cycle says it's the biggest job the company has performed in terms of the number of virtual machine instances that have been used for a single run.

Cycle Computing's software provisions large amounts of cloud-based compute resources for HPC jobs. This particular run -- for a pharmaceutical company that Cycle would not name -- involved testing how millions of different compounds would interact with a protein that's commonly associated with a certain type of cancer.

Normally, running such a scientific experiment would be a hefty and costly job. Cycle estimates the task would take 341,700 hours to run on a single machine. Using Amazon's cloud and the combined power of roughly 10,600 virtual machine instances, Cycle finished the job in 11 hours total, the company explains in a blog post.

Cycle used Amazon's spot market to provision the resources, which are unreserved virtual machines in AWS's cloud. Unreserved instances are more expensive than on-demand instances, but provide more flexibility to the user. The Cycle software scheduled the virtual machines, scaled them to capacity and ran them at 99% CPU capacity. Cycle used open source tool Chef to configure the cluster.

Four types of instances in Amazon's cloud were used, with eight-core c1.xlarges and four-core m1.xlarges being the most common. The four-core instances come with 1690GB of storage, 15GB of memory and are optimized for high input/output processing.

It was a big job for Cycle, but not the company's biggest. Last year, Cycle ran a 50,000-core job that used 6,732 instances for computational chemistry company Schrodinger. That job had fewer instances but more compute cores.

Cycle Computing CEO Jason Stowe said the company used to alert Amazon as to when it would be running these massive cluster compute cycles, but now HPC jobs are becoming "pedestrian." Amazon even builds HPC clusters itself.

"We just handled 10,600 servers, and our software built the environment, secured it, scheduled data across, scaled it, and tracked everything for audit/reporting purposes," a blog post on Cycle's website reads. "Chef 11 handled configuration for all of them. But now we're ready to add zeros here, and so is our software."

Network World staff writer Brandon Butler covers cloud computing and social collaboration. He can be reached at BButler@nww.com and found on Twitter at @BButlerNWW.

Top Whitepapers

Cloud, big data and analytics, mobile, social, and IT security have impacted networking – bringing it almost to breaking point. This whitepaper looks at the future of networking to accommodate these evolving trends.

Zones

Featured Whitepapers

This infographic looks at the current backup and recovery practices of IT managers and CIOs as well as their predictions and plans to protect their valuable information for the future. Find out: How much and where data in organisations is growing the fastest, the top 5 back up challenges organisations are facing and what the road ahead holds for backup and recovery strategies.

Copyright 2015 IDG Communications. ABN 14 001 592 650. All rights reserved. Reproduction in whole or in part in any form or medium without express written permission of IDG Communications is prohibited.