Cloud Analytics Proving Costly for Some

Alex Woodie

(world-of-vector/Shutterstock)

A great migration to the cloud is currently underway as companies look to get out of the business of hosting and supporting their own IT gear. The rapidly growing sophistication of big data analytics software on public cloud platforms like AWS and Azure is another big draw. But some companies that thought they’d save a bundle on their IT bills are coming to the painful reality that they were wrong.

Some of the bills that Amazon has sent to customers in recent months are awe-inspiring – but not in a good way (unless you hold shares of AMZM). According to a report of Lyft’s IPO filing in Business Insider, the car ride service is on the hook to pay AWS a minimum of $8 million per month for the next two-and-a-half years. If it doesn’t pay AWS at least $300 million by then, per its contract, it’ll have to cut a check for the difference.

Of course, Lyft is a digital native. It was born on the Web, and hosting applications on the public cloud is a smart business decision for digital natives, the conventional thinking goes. While that may or may not be true, the AWS bills of some non-digital-native companies have also grown quite high.

According to a recent story in The Information, Capital One’s AWS bill in 2018 came out to about $260 million, which was a 73% increase over the previous year. Adobe, Infor, and Intuit – all storied software firms, but not exactly what you’d call “digital natives” – also saw their AWS bills skyrocket by 64% to 93% from 2017 to 2018, according to the information The Information collected. Each of them paid AWS over $100 million last year (although Infor says its AWS bill is now on the downswing).

Is Cloud Worth It?

As the Great Cloud Migration continues to pick up steam, it’s worth asking the question: Are companies getting their money’s worth by building on the cloud?

It’s a tough question to answer, to be sure, and it’s likely true that the answer is different for each company. For some customers and some types of predictable workloads, the flexibility of cloud computing provides obviously benefits. However, for less predictable workloads – say, ad hoc analytics, for example — the cloud can actually cost you more money in the long run.

Priyank Patel, the co-founder and chief product officer of Arcadia Data, has seen demand for cloud computing explode among his customers. One third of the companies are already on the cloud in some fashion, and he expects that number to exceed 50% by the end of 2019.

“I think the market has shifted or moved towards data storage in the cloud, processing for machine learning, processing for SQL warehouses, and processing for BI in the cloud,” Patel tells Datanami. “You see where Cloudera is going. People want this shared data cloud kind of experience, especially the enterprises.”

In response to this clear market signal, Arcadia Data has worked to ensure that its software – a ROLAP layer that enables customers to use regular BI reporting tools atop data lakes containing petabytes of data — not only works well in public cloud infrastructures, but works well for customers too.

Arcadia started out in the Hadoop ecosystem, and still has many customers using its software to turbo-charge how BI tools like Tableau, Qlik, Microstrategy, and Microsoft PowerBI access on-premise data lakes running atop the Hadoop Distributed File System (HDFS). But the company also now runs in the cloud, and worked to ensure that the data access paths its customers take in the cloud are as streamlined as possible.

Part of that entails getting the Arcadia Data ROLAP server as close to the cloud object stores as it can. Patel says Arcadia is sitting right next to S3 in the case of AWS and Azure Data Lake Storage (ALDS) for Azure. It also currently offers (or plans to offer) that same level of support for Google Cloud and Snowflake.

Patel says this approach eliminates the need to move data within the cloud environment, which keeps costs down compared to how a customer would typically use a system like Amazon Redshift. “Today even in the cloud you have to take the data from S3 and move it to Redshift, then take the data from Redshift and move it into the Tableau Server eventually do the analysis there,” Patel says.

“Instead of duplicating the data into System A, B, or C just to get your Tableau reporting to work on top that, [we say] let’s go ahead and take that compute directly on top of the data store,” he continues. “What you’re doing is you’re eliminating the unnecessary storage of the data, and the unnecessary systems that are required to process the data.”

Cloud Lock In

The same holds true on Azure, he says. Arcadia recently worked with a customer to analyze the difference in cost of utilizing Arcadia’s more optimized approach versus using standard Azure components. The total came out to more than $1 million. “Even on the Azure stack, the play off multiple hops actually translates into actual dollars,” he says.

Would customers be making these wholesale migrations to the cloud if they knew they would likely be spending more money on the cloud than before? It’s an interesting question, and one that Patel has asked his customers.

“I talk to customers and say ‘Why did you do the cloud?'” he says. “They say, ‘Well it’s cheaper and more flexible.’ Cheaper, it is not. More flexible is absolutely true. The real data shows that cheaper is not true, even though the illusion of cheap might be there.”

Cloud is cheaper mainly with smaller workloads, where a cloud environment can be spun up and spun down very easily, he says. But costs can add up quickly for enterprises with large, demanding workloads. And what’s more, thanks to the high fees that public cloud providers charge for extracting data from their clouds, once the cloud has your big data, it can be prohibitively expensive to get it out.

“I never had anybody tell me they felt duped,” Patel says. “But after doing this analysis, they say it makes sense. It’s rational why I have to pay so much. Maybe the bills are subsidized in the first and second year of the cloud contract, because I have a sweet deal from the cloud provider. But the third year will hit me. And the third year will hit me at a place where I will not even be able to move out of it.”