Posts Tagged 'Racks'

The highlight of any customer visit to a SoftLayer office is always the data center tour. The infrastructure in our data centers is the hardware platform on which many of our customers build and run their entire businesses, so it's not surprising that they'd want a first-hand look at what's happening inside the DC. Without exception, visitors to a SoftLayer data center pod are impressed when they walk out of a SoftLayer data center pod ... even if they've been in dozens of similar facilities in the past.

What about the customers who aren't able to visit us, though? We can post pictures, share stats, describe our architecture and show you diagrams of our facilities, but those mediums can't replace the experience of an actual data center tour. In the interest of bridging the "data center tour" gap for customers who might not be able to visit SoftLayer in person (or who want to show off their infrastructure), we decided to record a video data center tour.

If you've seen "professional" video data center tours in the past, you're probably positioning a pillow on top of your keyboard right now to protect your face if you fall asleep from boredom when you hear another baritone narrator voiceover and see CAD mock-ups of another "enterprise class" facility. Don't worry ... That's not how we roll:

Josh Daley — whose role as site manager of DAL05 made him the ideal tour guide — did a fantastic job, and I'm looking forward to feedback from our customers about whether this data center tour style is helpful and/or entertaining.

If you want to see more videos like this one, "Like" it, leave comments with ideas and questions, and share it wherever you share things (Facebook, Twitter, your refrigerator, etc.).

One of the founding principles of SoftLayer is automation. Automation has enabled this company to provide our customers with a world class experience, and it enables employees to provide excellent service. It allows us to quickly deploy a variety of solutions at the click of a button, and it guarantees consistency in the products that we deliver. Automation isn't the whole story, though. The human element plays a huge role in SoftLayer's success.

As a Site Manager for the corporate facility, I thought I could share a unique perspective when it comes to what that human element looks like, specifically through the lens of the Server Build Team's responsibilities. You recently heard how my colleague, Broc Chalker, became an SBT, and so I wanted take it a step further by providing a high-level breakdown of how the Server Build Team enables SoftLayer to keep up with the operational demands of a rapidly growing, global infrastructure provider.

The Server Build Team is responsible for filling all of the beautiful data center environments you see in pictures and videos of SoftLayer facilities. Every day, they are in the DC, building out new rows for inventory. It sounds pretty simple, but it's actually a pretty involved process. When it comes to prepping new rows, our primary focus is redundancy (for power, cooling and network). Each rack is powered by dual power sources, four switches in a stacked configuration (two public network, two private network), and an additional switch that provides KVM access to the server. To make it possible to fill the rack with servers, we also have to make sure it's organized well, and that takes a lot of time. Just watch the video of the Go Live Crew cabling a server rack in SJC01, and you can see how time- and labor-intensive the process is. And if there are any mistakes or if the cables don't look clean, we'll cut all the ties and start over again.

In addition to preparing servers for new orders, SBTs also handle hardware-related requests. This can involve anything from changing out components for a build, performing upgrades / maintenance on active servers, or even troubleshooting servers. Any one of these requests has to be treated with significant urgency and detail.

The responsibilities do not end there. Server Build Technicians also perform a walk of the facility twice per shift. During this walk, technicians check for visual alerts on the servers and do a general facility check of all SoftLayer pods. Note: Each data center facility features one or more pods or "server rooms," each built to the same specifications to support up to 5,000 servers.

The DAL05 facility has a total of four pods, and at the end of the build-out, we should be running 18,000-20,000 servers in this facility alone. Over the past year, we completed the build out of SR02 and SR03 (pod 2 and 3, respectively), and we're finishing the final pod (SR04) right now. We've spent countless hours building servers and monitoring operating system provisions when new orders roll in, and as our server count increases, our team has grown to continue providing the support our existing customers expect and deserve when it comes to upgrade requests and hardware-related support tickets.

To be successful, we have to stay ahead of the game from an operations perspective. The DAL05 crew is working hard to build out this facility's last pod (SR04), but for the sake of this blog post, I pulled everyone together for a quick photo op to introduce you to the team.

DAL05 Day / Evening Team and SBT Interns (with the remaining racks to build out in DAL05):

About a month ago, Kevin Hazard visited SoftLayer Amsterdam after a conference in London, and while he was here, I invited him on a data center tour. You saw a few glimpses of the data center in his "This is Different" video, but he turned the camera around on me to give a simpler "Data Center Tour" video to show off some of the key characteristics of the server room environment in AMS01.

Given the fact that nearly everything in the data center is the same, if you've ever seen a SoftLayer data center, this tour will seem very familiar. The configuration and architecture of all 13 of our data centers are identical, and with the exceptions of a few Dutch words on the walls, this tour could be given (and is frequently given to customers) in all of our facilities around the world:

As we were recording this video, I started thinking about all the similarities and differences between all the entrepreneurs I have worked with during my career — which coincidentally lines up well with Clayton's "Building. Business. SoftLayer." blog. I cut my technology teeth in Silicon Valley during the dot-com tsunami of the late 90's, and since then, I have collaborated on-location with entrepreneurs from the Netherlands, United Kingdom, Spain, Germany, Austria, Switzerland, France, Chile, Ukraine and Italy. While these cultures often vary widely with customs, manners, food and methods of business, I would have to say that entrepreneurs have far more similarities than they do differences.

At the peak of the dot-com boom, money was raining from the sky, and anyone with a decent PowerPoint presentation containing the word "Internet," could raise million dollars of dollars in a matter of days. After the bubble popped, funding all but dried up. Even real businesses with profitable business models couldn't raise a cent. My neighbor went from being worth over $10M on paper and keeping company with the Queen of the Netherlands to scrambling to pay the rent and fighting for a seat at the local coffee shop.

In my opinion, that's when the real magic happened: The creators just kept on creating. Despite all our friends making fun of us — telling us "the Internet thing" was dead — we kept building cool stuff and coming up with innovative products that pushed the limits of technology.

While entrepreneurs liked the idea of making tons of money and building a global company from a simple idea, money and fame are not the primary drivers of true entrepreneurs. They were really more interested in creating something that would impact peoples' everyday lives and disrupt tired industries ... Just look at SoftLayer. In 2005, "tired" would have been one of the nicest things you could have said about the hosting industry, and in response to that environment, our "Innovate or Die" mentality shot us to the front of the pack.

Entrepreneurs are a lot like our data centers ... They may look a little different from the outside, but they are exactly the same on the inside. Ask them how they'd change the world, and take note of the wild look in their eyes. Our growth is fueled by the passions of our customers, and as long as we have brilliant customers doing amazing things, you can expect to see more and more of these "new data center" tour videos in the coming months and years.

When I was chosen by Lance to manage a "special project," I knew I had my work cut out for me. My mission is to redefine how data centers are built, so before I get too far in the creation of my data center facility, I thought it would be a good idea to get a quick refresher about how SoftLayer builds and runs its data centers:

You can disregard the references to Lance "requiring" me to go on the tour ... and the formality of Jon having to sign a note that said I successfully completed the tour. The references to those were just for dramatic effect since Lance and I are pretty much eye-to-eye about everything that needs to happen for PHIL's DC to succeed, and we both thought it would be good to have signed documentation that I went on a tour.

As I mentioned at the end of the tour, I didn't find the tour to be a complete waste of time because I was able to observe some of the biggest hurdles in building and maintaining a data center. Is redundancy really necessary ... or is it just redundant? What the heck did Jon say in the UPS room? How fast does a person have to run on a single treadmill for that treadmill to power 40,000 servers?

While I try to source my own "PHIL's DC" rack identifiers, my adoring public can take time to go through this video with a fine-toothed comb to suggest any potential data center innovations that you think I may have overlooked. I already have the most innovative and efficient data center designed in my head, but I'll consider crediting you if you share an idea ... even if (and by if, I mean "when") I've already thought of it.

I've got a hot lead on some slightly used hardware, so the next time you see me, the PHIL's DC inventory area should be fully stocked and ready for my own truckday.

When we last talked, I broke down the differences between features and benefits. To recap: a feature is something prominent about a person, place or thing, while a benefit is a feature that is useful to you. In that blog, I discussed our customer portal and the automation within, so with this next installment, let's move into my favorite place: the data center ... Our pride and joy!

If you have not had a chance to visit a SoftLayer data center, you're missing out. The number one response I get when I begin a tour through any of our facilities is, "I have been through several data centers before, and they're pretty boring," or my favorite, "We don't have to go in, they all look the same." Then they get a glimpse at the SoftLayer facility through the window in our lobby:

What makes a SoftLayer DC so different and unique?

We deploy data centers in a pod concept. A pod, or server room, is a designed to be an identical installation of balanced power, cooling and redundant best-in-class equipment in under 10,000 square feet. It will support just about 5,000 dedicated servers, and each pod is built to the same specifications as every other pod. We use the same hardware vendor for servers, the majority of our internal network is powered by Cisco gear and edge equipment is now powered by Juniper. Even the paint on the walls matches up from pod to pod, city to city and now country to country. That's standardization!

That all sounds great, but what does that mean for you? How do all these things benefit you as the end user?

First of all, setting standards improves our efficiency in support and operations. We can pluck any of our technicians in DAL05 and drop him into SJC01, and he'll feel right at home despite the outside world looking a bit different. No facility quirks, no learning curve. In fact, the Go Live Crews in Singapore and Amsterdam are all experienced SoftLayer technicians from our US facilities, so they help us make sure all of the details are exactly alike.

Beyond the support aspect, having data centers in multiple cities around the world is a benefit within itself: You have the option to host your solution as close or as far away from you as you wish. Taking that a step further, disaster recovery becomes much easier with our unique network-within-a-network topology.

The third biggest benefit customers get from SoftLayer's data centers is the quality of the server chassis. Because we standardize our SuperMicro chassis in every facility, we're able to troubleshoot and resolve issues faster when a customer contacts us. Let's say the mainboard is having a problem, and your Linux server is in kernel panic. Instead of taking time to try and fix the part, I can hot-swap all the drives into an identical chassis and use the portal to automatically move all of your IP addresses and network configurations to a new location in the DC. The server boots right up and is back in service with minimal downtime.

Try to do that with "similar" hardware (not "identical"), and see where that gets you.

The last obvious customer benefit we'll talk about here is the data center's internal network performance. Powered by Cisco internal switches and Juniper routers on the edge, we can provide unmatched bandwidth capacity to our data centers as well as low latency links between servers. In one rack on the data center floor, you can see 80Gbps of bandwidth. Our automated, high-speed network allows us to provision a server anywhere in a pod and an additional server anywhere else in the same pod, and they will perform as if they are sitting right next to each other. That means you don't need to reserve space in the same rack for a server that you think you'll need in the future, so when your business grows, your infrastructure can grow seamlessly with you.

In the last installment of this little "SoftLayer Features and Benefits" series, we'll talk about the global network and learn why no one in the industry can match it.

Over the course of about a month, you were able to follow the build-out progress of SoftLayer's Singapore data center facility. Todd book-ended his coverage of the process with an early look on September 2 and the official "LIVE" announcement on October 3, and given the fantastic response from customers to those updates, we're going to keep them going from Amsterdam.

If you follow SoftLayer on Twitter or keep an eye on our Flickr account, the last time you saw the Amsterdam facility, it looked pretty empty. You might assume that with all the attention on Singapore, Amsterdam wasn't getting much attention, but you'd be wrong ... Folks have been working non-stop in Europe as well, and the facility looks beautiful:

It's pretty obvious with the racks you see pictured that our go-live team has been on the ground and working hard in the new facility. We shipped loads of gear across a different ocean to get it to Amsterdam, but things will probably look pretty familiar.

When Singapore went live on Monday, customers were ecstatic. We've already provisioned a few hundred servers in the new facility, and the chorus of users anxious about our European expansion has gotten louder as a result. As you can see, Amsterdam is coming along nicely, so you'll have a SoftLayer server in Amsterdam before you know it.

SoftLayer's growth internationally has been fueled by customer demand, so while we're working on Amsterdam, we'd love to hear where you'd like to see us next. Leave a comment with the country/region you think could best benefit from a local SoftLayer facility ... And if you agree with any of the ideas, be sure to post your agreement as well so we get an even clearer picture of customer demand.

It's basically a crescendo of applause in a crowd that starts with a single hand clap. A few seconds after that first clap, you hear the second. A slow rhythm takes shape. A few people join in. The rhythm is contagious, and it starts to spread through the crowd. As more people join in, the natural tendency is for the pace to speed up as the volume increases, and within about a minute, a single hand clap becomes a huge roar of applause. In the movie Rudy, one character starts a "slow clap" on the sideline of a football game, and the cheer ends up filling the entire stadium ... And that's the visual that comes to mind when I think about the upcoming "go live" date for our Singapore data center.

Start a slow clap in your mind and think of each successive milestone getting faster and exponentially louder applause:

From that press release, you learned that SoftLayer's Singapore data center will go live on October 3, 2011.

If you imagined correctly, the applause in your mind should be borderline deafening ... And I didn't even mention the fact that we enabled pre-orders on select servers in Singapore last week with a Triple Double special exclusively for servers in the new SNG01 facility.

AND I haven't said anything about the progress of our first European data center in Amsterdam. We already have a team of people there working to get that facility ready, and it's coming together just as quickly. Don't be surprised to see a few sneak peeks at the build-out process there in the next few weeks.

It's almost unfathomable that we're so close to the launch of our first facility outside the United States, and when you consider how quickly Amsterdam will come online after Singapore, you probably think you're taking crazy pills ... Or that we are. I don't want to take any of the wind out of the sales of our launch day, so I'm just going to share a few more glimpses into the data center.

On Monday, you can light your first server at the end of this Singaporean hallway:

All of the racks are powered:

The server rails are installed:

And we thought it might be a good idea to go ahead and install a few servers:

Last week, a few of the folks from marketing were invited to celebrate in the Truck Day festivities for Pod 2 in DAL05 (SR02.DAL05), and I jumped at the opportunity. I don't go anywhere without at least one camera on-hand to document and share what's going on with the SoftLayer community, and Truck Day wasn't an exception ... In fact, I had three different cameras going at all times!

The truck arrived at around 7 a.m. with a few dozen pallets of servers, and about forty employees from all around the company immediately jumped into action. As the pallets moved from the loading dock to the inventory room, people were unboxing servers and piling them on carts. When a cart was full, it was whisked to the data center and unloaded. The data center techs plugged in each of the servers to confirm its configuration and stacked it with matching configurations in designated areas around the data center. By the time one cart got back to the inventory room, another was on its way to the data center, so very little time was lost.

Back in 2007, SamF did a great job of explaining the process, so I won't reinvent the wheel. Instead, I'll let you see the activities as they were captured by the three cameras I toted along:

To give you an idea of how fast all of this was done, each the time lapse cameras set up in the data center and in the inventory room captured images every five seconds. When the video was compiled, the frame rate was set to 20 frames per second, so each second of time lapse video is the equivalent of 100 seconds of work. In a matter of just a few hours, we received, inventoried, racked, cabled and started selling around 750 servers in a brand new data center pod. Competitors: Be afraid. Be very afraid. :)

In the past three weeks, we brought three different data center pods online in three different parts of the country: On April 25, it was our first server room in San Jose (SJC01); on May 2, the second server room in DAL05; and on May 10, our second server room in WDC01. As far as I know, we don't have a new pod planned for next month, but given how quickly the operations team has been building data center space, I wouldn't be surprised to get a call asking me to come in a little early to help unload servers in a new data center next week.

A few weeks ago, SamF posted "Before They Were SoftLayer Data Centers," a virtual scrapbook from the San Jose data center construction process, and based on the surge of traffic we saw to the post, our customers loved it. It's incredible to see an open warehouse-looking space transformed into an enterprise data center environment, and there's more amazingness where that came from.

In addition to the pre-"Truck Day" pictures we posted on the blog and in the San Jose DC Construction album on Flickr, we trained a video camera on a row in the data center to capture the cabling process.

What's so interesting about plugging in cables?

Consider the fact that each of the network switches we use in a rack has at least 48 ports. Now consider that each rack has two public network switches, two private network switches and one out-of-band management network switch that need to be connected to every SoftLayer server in the rack. That's 240 pre-measured network cables that need to be labeled and routed to specific heights in each rack ... without getting tangled and knotted up (see: behind your TV or under your computer desk).

The cabling process is so precise that if a single cable is out of place, the zip-tie on an entire bundle will be cut, and the process is started from scratch. The process is time-consuming, but the results speak for themselves:

Without further ado, here's the SJ data center team in action. The video is playing at 20x normal speed, and given the amount of time it takes to complete the cabling process for each rack, we enlisted the help of Spongebob SquarePants in our use of the "Two Hours Later" cut:

In the past few weeks, we've fielded a few questions from our Twitter followers about temperatures in our data center and how CRAC units work. John mentioned in the "Building a Data Center" series that his next post would be about keeping the data center cool, so I'll try not to steal too much thunder from him by posting a basic CRAC unit explanation to answer those questions.

To record this video, we made the long walk (~2 minutes) downstairs to Pod 1 of SoftLayer's DAL05 facility to give you a first-hand look at the star of the show: the DC Computer Room Air Conditioning Unit. Because this was recorded on a "Truck Day" at SoftLayer, the pod was bustling with activity, so we found a "quiet" open area in a section of the pod that will soon be filled with new servers to record the video.

Due to the ambient noise in the data center, my explanation had to be "yelled," so please forgive the volume.