What happens in the Data Center, ends up here.

Category Archives: Isilon

Now that I ripped the band aid off (yes, I’m a ripper – I find it best to just get it over with!), I guess I should explain myself.

I worked at Heritage Auctions (HA.Com) for just shy of 10 years. I can’t say enough great things about the opportunity, the company, the people, and especially the owners there. Over the last 3 years of my illustrious career, I primarily focused my efforts as VP of IT on converging the data center. After much research, debate, and negotiation a vast majority of our infrastructure landed on VMware for virtualization, Cisco UCS for compute, and EMC for storage. And thus, the love affair began.

I’ve said it a thousand times, and I’ll say it again – thanks to great partners like Presidio, and a great friend in Tommy Trogden (@vTexan) – we were able to start the transformation of our data center into an amazing tool for the business. UCS made compute sexy. VMware is the leader in the hypervisor market for a reason. And, EMC storage provided drastically needed performance, scale, and availability like we’d never had before.

One of our investments was in EMC’s VMAXe – thanks to some cunning sales work by Aaron Skogsberg (@dweebiak) we ended up with the 16th one off the line after the gigantic enterprise customers grabbed theirs up. This VMAX was supposed to bring Tier1 to a new sector of customers, making it easier to use / manage / implement / install. Everything he promised was true, and I was in love (not just with Aaron, but with EMC). After being asked to be a reference customer, I got sucked into a whirlwind of events that included speaking to other prospective customers, doing roundtables, videos, speaking at Brian Gallagher’s ESD all hands!!!, two Chad’s Worlds, speaking at EMCWorld with Fidelma Russo, speaking at VMworld, and the list goes on. I even got elected to be part of EMCElect in 2013! I was in love with the EMC product, I was infatuated with the EMC culture, and I had made hundreds of great friends at EMC.

My staff joked constantly that I was going to leave any day, and I told them they were being foolish. I had a great job, at a great company, with no reason to leave. Why start over? Why take the chance? Besides, EMC has thousands of customers who love them as much as I do – I’ve seen them all pack into a Bruno Mars concert and come out wearing blinky schwag. I’d received plenty of calls from recruiters, etc asking me to put myself out there, and I’d always said no, until now. One day, I got a call asking me to interview at a manufacturer and I said no. A few days later, knowing that my (now manager) Trevor Starnes was looking for someone, I started that fateful conversation. The ‘ol “what if I….” and that was all she wrote. A few interviews and a resume (that I hadn’t written in 15 years) later, and I was in.

I’m excited, and scared, and excited. I haven’t felt more overwhelmed and useless in my life and I love it. I’ve got so much to do, and it’s just day 2.

This change isn’t a sprint, but rather one of those awesome hardcore Ultramarathons. I’m going to document my transition from VP of IT to PreSales Engineer and all of the things I learn between. From perfecting the art of chest-bumping, to crying at night after some VP of IT tells you that your beloved EMC sucks. As embarrassing as it is, coming from the Ops/IT world, I’ve already had to visit the local IT support numerous times to beg forgiveness as I’ve been locked out of every system I’ve tried to get into. I think I’m gunna have to bring him some Hypnotic Donuts tomorrow to make amends.

We recently finished the migration of 45TB worth of data from our existing and aging XIO Storage (Xiotech) and EMC Fiber Channel SANs (And an Apple DAS) over to a new Isilon NL-Series 5 node cluster we implemented in early September.

We migrated away from dedicated servers – a mix of Windows Server 2003, Windows Server 2008, Windows Server 2008 R2, and OS X Server (Snow Leopard) to the NAS functionality of the Isilon using a mix of SMB and NFS shares.

We started to hear grumblings a few weeks in that things were slower. How could that possibly be? Mac OS X users specifically were saying that data transfers of images for Photoshop and proofing were taking 10-20x longer than they were prior to the migration. We tested the issue on Windows based machines that the admin team uses on a daily basis and didn’t see the issue. Cluster was hardly working so it wasn’t a performance issue, etc.

Through troubleshooting, it turns out the issue was specific to OS X – any version of 10.X saw the same issue, so it wasn’t version specific. NFS would get the performance back, but we have yet to find the information we need to implement user authentication and locking to dedicated IP’s won’t work for us.

After a bit of Google searching, I decided I’d reach out to twitter. My use of twitter is really for technical education, business networking, and a few jokes here and there – so my friends list is primarily technical.

Essentially, if you are having a problem with your Mac OS X Clients having slow file copy on a Samba (SMB) share on an Isilon cluster:

Open a terminal session and:
sudo vi /etc/sysctl.conf

Then add the following line to the /etc/sysctl.confnet.inet.tcp.delayed_ack=0

Even while copying an existing set of 4GB worth of files on a machine, we could make the above change and start the same copy a second time, and the latter would finish before the first copy was even close to being done.

Problem Solved! Thanks to Social Media, and specifically the power of Twitter, we were able to solve a problem that was affecting 30+ users after a migration that was supposed to make their lives easier. I may not have chosen the appropriate keywords in my Google search, but thankfully people out there in the twitterverse already had.