OpenStack in Production: Accelerating Innovation at PayPal

OpenStack is playing a major part in PayPal’s vision by enabling a private cloud that helps the company’s developers quickly respond to its customers’ increasing demands and constantly changing needs, while developing a stable platform for customers to pay for their purchases.

At any given minute, more than $4,400 changes hands online over PayPal’s 113 million registered accounts around the world. The payments come via PCs or mobile devices in 190 different markets with 25 different currencies. For PayPal, this is the future of money – enabling customers to make payments anytime, anywhere and from any device.

“We want to deliver the future of money today. What that means is we want to enable our customers to make payments any time and any way,” said Saran Mandair, senior director of PayPal infrastructure engineering. “We are moving to the cloud powered by OpenStack to enable agility, availability and the innovation necessary to get the best products to our customers, faster than our competitors.”

Agility with Stability: OpenStack at PayPal

A key component of the selection criteria for PayPal’s cloud infrastructure was the ability to scale quickly without compromising availability. For its engineers and developers to be successful, the platform had to support PayPal’s requirement of 99.9999% availability. According to Mandair, “We needed agility without sacrificing availability. For our customers, that means they have a website that is up and running to conduct their business to process payments anywhere, anytime that they choose to use PayPal.”

To meet this demand, PayPal decided to build its private cloud infrastructure with OpenStack. OpenStack is the fastest growing open cloud community, building software to power public and private clouds for a growing number of organizations like CERN, eBay, HP, Intel, MercadoLibre and Rackspace. The software controls and automates pools of compute, storage and networking resources to turn standard hardware into a powerful cloud computing environment.

PayPal is using a broad portfolio of OpenStack technologies to support key areas of mobility, payments, credit, merchants and the web operations for more than 117 million active registered accounts. The first phase of its OpenStack-based private cloud was rolled out in December 2011 in the midst of the online holiday shopping season.

According to Anand Palanisamy, an architect in the PayPal infrastructure engineering team, the company started with an aggressive schedule and small, dedicated team of two developers to develop a prototype.

“We had a fairly detailed success criteria and check list within eight weeks,” he said. “We quickly saw it pay substantial dividends, and the initial prototype validated further investment in OpenStack.”

“Our OpenStack project uses the Fuel library by Mirantis, which has dramatically accelerated our OpenStack deployment with robust, production-grade architecture while giving us the flexibility to tailor OpenStack to our needs.”

Moving forward, Palanisamy said, they will organize teams split into two different aspects – OpenStack deployment and application development. OpenStack supports the development process for agile teams by providing a framework for interconnected modules that are isolated but can be put together fairly easily.

“We are focused on individual cases like networking, compute and storage, the primary capabilities of the OpenStack platform,” Palanisamy said. “All three areas have engineers engaged on the individual subjects rather than having one large cloud team to work all over the place. OpenStack enables us to operate smaller teams and be more selective in our deployment and development.”

As the project advances, the PayPal infrastructure team is targeting aggressive internal cloud milestones, covering 90 percent of some parts of their infrastructure in the next nine to 12 months with several thousand physical nodes.

In the meantime, Palanisamy said early results are validating their plans to continue down an aggressive path with OpenStack.

Open Source/Open Standards

As an open source technology, OpenStack provides PayPal with input and insights from other organizations that share their passion and focus. Mandair says he’s committed to open source technologies like OpenStack because he can benefit from others and give back to the community with PayPal’s own findings.

“One of the benefits we see in an open development model is leveraging the intellectual mindset of people who are outside of our company—across the world—who have this passion,” Mandair said. “We take advantage of an aggressive roadmap of what the community is contributing, while we put our code back in for others to consider and leverage if they want to.”

Palanisamy also values the role the open source community plays by encouraging vendors to develop products with open standards for compatibility with a variety of technologies in their computing environment. By developing products that are not just unique to PayPal, new innovations support and build the entire ecosystem.

“It also helps our vendors because they don’t have to grapple with multiple standards to meet different customer needs,” he said. “OpenStack as an open source product gives us the flexibility to choose the products that are right for our needs and frees us from proprietary settings. This validates that OpenStack is the right technology versus everything we’ve looked at so far.”

As for how OpenStack compares with other open source technologies, both Palanisamy and Mandair have been impressed with the support they’ve received from the OpenStack community. They say problems and questions have been addressed in less than two hours, which exceeds their experience with other cloud products.

For PayPal, OpenStack APIs have accelerated the company’s development process by making applications easier to deploy on common infrastructure. Having one consistent standard across their environment helps developers get up to speed faster without additional training or support.

“By leveraging the collective innovation of the OpenStack community, we can develop and grow our private cloud much quicker without having to reinvent anything,” said Mandair. “With this very active community, we help each other grow by avoiding similar mistakes and guiding each other along the way. That actually validates how much the community out there is helping to build products in a much quicker timeframe in a simple and elegant manner.”

In 2012, PayPal experienced dramatic growth in its registered users, new ventures with Discover and Softbank, along with new innovations to make it easier to for small businesses to make and accept mobile payments. As these and other milestones in PayPal’s future create new customer demand and greater payment volumes, Palanisamy and Mandair plan to leverage the cloud with OpenStack for the agility to address changes and a platform to help ensure availability.

“From what we are experiencing, OpenStack is ready for primetime,” Mandair said. “And that is encouraging us to stay on this path in a fairly aggressive manner, so we can leverage the benefits and the contribution from a community that is growing every time you look at the website.”