Amazon is notoriously hush-hush about its internal operations

Usually Amazon Web Services, which many consider to be the leader in the infrastructure as a service (IaaS) cloud computing market, is pretty hush-hush about the internal workings of its massive cloud.

But recently an anonymous person who claims to be a former AWS engineer gave an in-depth, inside look at AWS's cloud during a question-and-answer session on Reddit. (Reddit moderators routinely authenticate the credentials of "I am a (IamA) ... "ask me anything" (AMA) participants.)

The engineer says he used to work on OS/software issues at the cloud division of Amazon.com, and he now gets paid more to do development outside of AWS. Reddit has verified his identity, but not provided his name. Given the breadth and depth of his responses in the post, he seems to know what he's talking about.

Amazon uses a lot of secret sauce

Amazon customizes both its hardware and software that make up its cloud. Most hardware nodes run one of a variety of flavors of "Amazon Linux," for example. The company's Amazon's Elastic Block Storage (EBS) service is also custom-built. "It's simply just a lot of commodity disks with 100% custom written software running the show. Keep things cheap that way. EBS volumes are true block devices underneath, they're not sitting in some filesystem."

Working on Amazon's cloud isn't all it's hyped up to be

The engineer says he and a friend were both disappointed in the "mundane" tasks they were doing at AWS. But, it did pay off. "Putting Amazon SDE (software development engineer) on your resume is quickly turning into the same thing as putting Google SDE on your resume. So, while you may feel it's boring, the real reward will be the job opportunities in the future." He did not make more than $100,000 working there, and he says the stock plan isn't great. Employees do get a 10% discount, up to $1,000 worth of purchases at Amazon.com. "The company's core values include 'frugality' and that's just the word 'cheap' wrapped up in a bow."

One questioner asked if AWS can support Level 4 data security, to which the engineer said yes, but at a cost. AWS recently released a new hardware security module (HSM), which is literally a piece of hardware that sits in its cloud to provide extra security for customers through encryption of Elastic Block Storage (EBS) volumes, and Amazon Machine Instances (AMI), two of the major offerings from AWS. This allows customers who are willing to spend the money to deploy these systems across their infrastructure to achieve HIPAA, SOX and ITAR compliances.

"The cloud" isn't really a new technology and is a lot of marketing hype

"The cloud, [is] an 'interesting' concept, because in a lot of ways it's still just a VPS [virtual private server]," the engineer says. "Sure it has some redundant features, but the definition of cloud is fairly dynamic and bent to fit whatever a company wants to sell as a 'cloud.'"

How Virtual Private Clouds (VPC) work

Amazon breaks up its hardware nodes into two major categories: "On-demand" and "Virtual private cloud" (VPC). The company has recently encouraged more use of the VPC option, making it the default setting when customers spin up virtual machines. VPCs have dedicated IP addresses for customers, he says, whereas on-demand instances have IP addresses slotted to those specific hardware nodes.

What to do if you're having problems with "noisy neighbors"

Because AWS is a multi-tenant cloud, some users may have problems with "noisy neighbors," or other customers eating up compute capacity on a shared server. The engineer says for $7,500 a month, Amazon will guarantee they will be on dedicated hardware, and not just have the IP addresses segmented through the VPC. Noisy neighbors are usually only a problem for larger customers though, he says. "If you spawn 20 instances, odds are, you'll be on 20 HNs, if you spawn 100 instances, odds are you'll see yourself share about 20-30% of the HNs. Though you won't actually see it."

Is AWS good for small-scale workloads?

Yes and no. AWS was designed for big workloads, but small workloads can work there as well, the engineer says in response to a question about hosting about a dozen Web servers on Amazon Elastic Compute Cloud (EC2). The more important question is not small or large though, it's how dynamic the workload is. AWS, and the cloud in general, thrive in being able to scale capacity to demand. "AWS is meant to scale with user load, [so] by definition you need a user load that is dynamic enough to justify putting the cost into developing a 'Cloud Architecture' for your app," he says.

On the competition for AWS from Microsoft Azure, Rackspace and Google

They're all basically on the same playing field now, in this engineer's opinion. "If it was a Venn diagram there would be major overlap. The places outside the overlap is what differentiates all the individual companies," he says. Competition in the cloud industry is good for consumers, and will continue to drive down costs and drive up quality.

He considers Google to be a serious threat to AWS, but not Azure. "Anyone who thinks there is a 'cloud war' currently and AWS is seeing any type of real challenge is just fooling themselves. Nobody can compete currently with the size of AWS, they were the first in and will be the last out. Google however will make it rain a bit, I've been using their cloud platform a little bit lately and I have to say. It's ... impressive."

Is EBS Optimized, a premium service for Elastic Block Storage, worth it?

EBSo adds a quality of service marker to packets, he says, and by doing so will "significantly smooth out your EBS experience under heavy load instantly ... EBS Optimized instances are totally worth it if you need hardcore disk i/o."

Disks are the biggest limiting factor for increased network power in the future

This is not related directly to AWS's cloud, but the engineer makes the point that read/write capabilities are not keeping up with network traffic speeds. "We live in a world where our CPUs, memory, networks and video are many times faster than our disk throughput. What good is gigabit Internet if you can only write to your disk at 250Mb/s?"

Network World senior writer Brandon Butler covers cloud computing and social collaboration. He can be reached at BButler@nww.com and found on Twitter at @BButlerNWW.

Senior Writer Brandon Butler covers the cloud computing industry for Network World by focusing on the advancements of major players in the industry, tracking end user deployments and keeping tabs on the hottest new startups. He contributes to NetworkWorld.com and is the author of the Cloud Chronicles blog. Email him at bbutler@nww.com and follow him on Twitter @BButlerNWW.