כל המידע על מחשוב ענן במקום אחד

Switching clouds - Spotify Story

Like moving house, packing up your data and shifting it from one cloud to another can be a stressful experience.

But not as fraught as you might think, says Jyrki Pulliainen, who as a software engineer at music streaming service Spotify has just overseen the move of more than 1.5 billion files from Amazon Web Services (AWS) to Google Cloud Storage.

While Spotify stumbled over a few gotchas when shifting the 3PB of data, the process was completed in the past couple of weeks and without major setbacks.

Amazon Web Services (AWS) Certification is fast becoming the must have certificates for any IT professional working with AWS. This course is designed to help you pass the AWS Certified Solutions Architect (CSA) - Associate Exam. Even if you have...

Training provided by Udemy

"The lesson learned here is that our private capacity points had IP addresses registered to Stockholm, so we were ping-ponging to Europe and ping-ponging back to US Central to fetch the files," he said, speaking at the Google Cloud Platform Next conference in London.

"So if you do these kind of moves, remember to check where your IP addresses are registered and change them accordingly."

3. Check how well your new cloud provider integrates with your CDN

Spotify uses three major content delivery network (CDN) providers, Fastly, Akamai and Verizon, to cache music locally for users across the globe.

While Spotify's CDN partners offer Google Cloud Storage (GCS) support 'out of the box', there were some roadblocks.

A negative for Spotify is that its CDN providers "don't really support" OAuth, the token-based authentication system that allows third-party services to access online accounts without knowing the user's password.

"We would have liked to have had OAuth support. We had to create artificial Google accounts for our CDN providers. It's a hassle for us, for our IT department, for our security and we get less fine-grained access control."

4. Check what you'll lose during the move

Each cloud provider has strengths and weaknesses, so you should expect some gaps. Check which features from your existing cloud provider are missing or inferior to your previous choice.

"Now that we've done the move, we are generally really happy but we have some wishes for the future.

"We would love to see cross-continental replication, like the fact we now run in a US Central regional bucket, that causes a slight latency for our European users.

"We ideally would like to have another bucket in Asia. At the moment, the GCS does not allow cross-continental replication.

"We're also seeing slightly lower cold read latencies than with S3, the good news is that Google has improvements upcoming and we've worked with them to shear off 30ms.

Pulliainen added that these latencies should be invisible to users, due to Spotify's technical architecture.

While moving this much data understandably took a couple of weeks, the actual process was straightforward, said Pulliainen, who described moving the 1.5bn files as nothing more than a matter of "clicking around a UI for a moment" and called the overall experience "super simple".

Out of the 1.5bn files, 144 had transfer failures, but Pulliainen said it took Google less than one day to track down those that were missing.

Image: LawrenLu/Getty Images/iStockphoto

Like moving house, packing up your data and shifting it from one cloud to another can be a stressful experience.

But not as fraught as you might think, says Jyrki Pulliainen, who as a software engineer at music streaming service Spotify has just overseen the move of more than 1.5 billion files from Amazon Web Services (AWS) to Google Cloud Storage.

While Spotify stumbled over a few gotchas when shifting the 3PB of data, the process was completed in the past couple of weeks and without major setbacks.

Amazon Web Services (AWS) Certification is fast becoming the must have certificates for any IT professional working with AWS. This course is designed to help you pass the AWS Certified Solutions Architect (CSA) - Associate Exam. Even if you have...

Training provided by Udemy

"The lesson learned here is that our private capacity points had IP addresses registered to Stockholm, so we were ping-ponging to Europe and ping-ponging back to US Central to fetch the files," he said, speaking at the Google Cloud Platform Next conference in London.

"So if you do these kind of moves, remember to check where your IP addresses are registered and change them accordingly."

3. Check how well your new cloud provider integrates with your CDN

Spotify uses three major content delivery network (CDN) providers, Fastly, Akamai and Verizon, to cache music locally for users across the globe.

While Spotify's CDN partners offer Google Cloud Storage (GCS) support 'out of the box', there were some roadblocks.

A negative for Spotify is that its CDN providers "don't really support" OAuth, the token-based authentication system that allows third-party services to access online accounts without knowing the user's password.

"We would have liked to have had OAuth support. We had to create artificial Google accounts for our CDN providers. It's a hassle for us, for our IT department, for our security and we get less fine-grained access control."

4. Check what you'll lose during the move

Each cloud provider has strengths and weaknesses, so you should expect some gaps. Check which features from your existing cloud provider are missing or inferior to your previous choice.

"Now that we've done the move, we are generally really happy but we have some wishes for the future.

"We would love to see cross-continental replication, like the fact we now run in a US Central regional bucket, that causes a slight latency for our European users.

"We ideally would like to have another bucket in Asia. At the moment, the GCS does not allow cross-continental replication.

"We're also seeing slightly lower cold read latencies than with S3, the good news is that Google has improvements upcoming and we've worked with them to shear off 30ms.

Pulliainen added that these latencies should be invisible to users, due to Spotify's technical architecture.

While moving this much data understandably took a couple of weeks, the actual process was straightforward, said Pulliainen, who described moving the 1.5bn files as nothing more than a matter of "clicking around a UI for a moment" and called the overall experience "super simple".

Out of the 1.5bn files, 144 had transfer failures, but Pulliainen said it took Google less than one day to track down those that were missing.