Last year I did a presentation on NVMe for Beginners along with Craig Waters for vBrownBag at the Melbourne VMUG UserCon. It was a daunting experience as it was a new cohort to present in front of and NVMe is a topic I had no expertise in. It’s something I wanted to learn more about and I thought that doing a presentation on it would mean I’d have to pull my finger out and really get down to understanding it. Nothing like a bit of pressure to learn something :-). Thankfully with Craig I had someone that had been through the mill a few times when it came to presenting. His mentorship and guidance made the presentation so much easier, and it gave me the confidence to do the presentation on my own a few months later at a normal VMUG.

Unfortunately the presentation contained proprietary information so it cannot be circulated online but I’ll run through the premise of the presentation and hopefully provide a brief introduction to NVMe. If you want to get the best understanding possible about NVMe I cannot recommend enough that you take time to read J Metz’s article on Cisco blogs NVMe for Absolute Beginners. It’s a phenomenal breakdown on NVMe and it’s so well written that even I was able to comprehend it.

I’ll run through the presentation as much as possible here. So, why NVMe?

Earlier this week I got a great email from the NetApp United team to say I had been accepted for a second time into the team. This followed on from a similar email from the VMware vExperts the week before. Being included as a member of these advocacy/influencer groups has meant I’ve completed the trifecta along with last months Cisco Champions inclusion. I’ve spent the past number of years working with technologies from all three companies and it’s nice to be recognised as a contributor to the community.

Being part of these groups means a huge amount to me. They have all, in particular the vExpert group, been something I’ve aspired to being a part of over the past couple of years and I’m overjoyed about being accepted into each program. Each of the influencer programs is different in their approach but there’s some commonalities too. The common things among the programs are:
* Being able to network with peers across the globe.
* Having access to NDA information before it gets released to the general public.
* Ability to interact with engineers and in some instances leadership within those companies.
* Dedicated comms channels for easier interaction (Slack for vExpert and NetAppUnited and Spark for CiscoChampions)

ESXi upgrade preparation

With Cisco UCS you really need to make sure that your ESXi hosts are running the correct driver version. If you’re running NFS or FCoE storage into your ESXi hosts as either datastores or RDM disks then it’s critical that you have the right fnic and enic drivers. Even if you use the Cisco Custom image for ESXi upgrades the enic and fnic drivers may not be correct according to the compatibility matrix. I’ve had this issue in the past and I saw intermittent NFS datastores going offline for a Dev ESXi host and the resolution was to upgrade the enic driver which handles ethernet storage connectivity.

The best place to go is to VMware’s compatibility site for IO drivers which comes under the System/Servers. To find out which drivers you currently have you will need to check on the driver versions on the ESXi hosts. This can be done by following KB1027206. Using the values for the Vendor ID, Device ID, Sub-Vendor ID and Sub-Device ID it’s possible to pinpoint the interoperability with your respective hardware. In my case I have both VIC1340 and VIC1240 in the mix so I had to go through the process twice. Primarily you’ll be using the ‘ethtool -i’ command to find the driver version.

In this image you can see the version of enic drivers I’m running, 2.1.2.71 doesn’t match the firmware version that will be installed as part of the Cisco Custom ISO image. This shows that the enic driver version will need to be upgraded as part of the process.

During a recent upgrade from vCenter Server 5.5 Update 2d to vCenter Server 5.5 Update 3b it kept failing at the web client upgrade. After successfully upgrading Single-Sign On I proceeded with the upgrade of vSphere Web Client. I got the following error during the installation:

The update to 5.5 3b caused disk capacity to fill up and make the installation process unable to finish the upgrade. The SSO install worked but the WebClient fails with error 29702. The primary issue was that over 40GB of space on C drive was taken up with SSO upgrade. I searched for fixes and found the following link but before carrying out the task of removing the Java Components and re-installing again I wanted to check with support on the procedure.

Step 4: Following the reboot you can then begin the upgrade process once again and this time it will succeeed. Run the vCenter installer and from Custom Install select vCenter Single Sign-On. Click Next.

Step 5: Click Install

Step 6: The single sign-on components will begin to install, including components such as OpenSSL

One of the key components being installed is VMware JRE.

Step 7: If you get prompted to close some applications select “Close the applications and attempt to restart them”. Click Ok.

Click ok to the prompt to close apps automatically

Step 8: Click Finish to complete the Single Sign-On upgrade

Step 9: Click on vCenter Web Client to begin the next stage of the upgrade

Step 10: Click Yes to continue

Step 11: Click Accept License agreement and click Next

Step 12: Click Install to begin the web client installation

Step 13: Click Finish to complete the installation

Once you click Finish click Ok on the dialog to advise that the services will take a few minutes to restart

One of my goals for the day was to attend a number of the community sessions. I found the vBrownBag sessions conducted by Alastair Cooke (@demitassenz) to be the most informative and entertaining sessions of the day, along with those of Chris Wahl. The award for the funniest session of the day went to Simon Sharwood (@ssharwood) from the Register as part of the vBrownBag session. It wasn’t just entertaining but a great insight into how content is derived for the site.

I missed one of the sessions I had intended on getting to but here’s a break down of the sessions I did attend.

The annual Melbourne VMUG UserCon takes place this Thursday, 25th February. It’s also an important day for me as it’s my wedding anniversary. I know which one my wife is more interested in! But, for the IT community in Melbourne all eyes will be on the VMUG. This years event has moved location from the old Hilton on the Park to Crown on Southbank. I think this is a good move and makes the VMUG even more accessible than in previous years. Last years guest speakers were excellent with Chad Sakac, Vaughan Stewart and John Troyer and this year it’s been lifted another notch again. This year the enterprise IT giants include Scott Lowe (@scott_lowe), Keith Townsend (@CTOAdvisor), Brad Tompkins (@VMUG_CEO) and my own personal IT hero Chris Wahl (@ChrisWahl). There’s also going to be vBrownBag sessions being hosted by Alastair Cooke (@demitassenz). If you’ve been following twitter you’ll have seen that Scott’s been having issues with flights and has had United basically crap all over his plans. Hopefully things work out for him and he can make it on time to the Sydney VMUG on Tuesday 23rd but it looks like it’ll be a close call. I wish him safe travels from here on.

VMUGs are all about the community. It’s the primary reason it exists and we’re incredibly fortunate to have the organisers volunteer their time to put on such a great event. Melbourne has some of the finest at its helm and that has been recognised globally. If you haven’t attended before I’d highly recommend fitting it into your calendar. You’ll be glad you did and your employer will be glad you did too. It’s really worth getting to the keynotes at the UserCon as unlike keynotes at other events they are not strictly vendor focused and they can provide some real insight into your industry as a whole and even your career path. But the main focus should be the community speeches. Hearing from others out in the field about the trials and tribulations they’ve had with specific technology is where the real learning takes place. These contribute a shorter part of the agenda and it’s something I’d like to see more of in future events but I also appreciate that it’s hard to get speakers for such sessions. For me this year that’ll be my focus outside of the keynotes. There are a number of vendor based sessions as well throughout the day that delve into new technology .

There’s a lot of information and knowledge to be gleaned from this event. I’d also recommend working out your agenda before attending and have a ponder over what you’d like to get out of the event. The sessions I’m planning on attending are:

Early this year I decided to up the ante a bit on my level of blogging. While I had really started to take it a bit more seriously the year before I wanted to make a concerted effort this year. During the months running up to the end of 2014 the traffic on the blog had grown quite significantly from what it had previously been. This was at a point when I wasn’t putting out any content all that regularly so it came as a surprise and encouraged me to think about creating more content. Anthony Burke over at NetworkInferno, a great blog if you get some downtime to have a flick through, wrote an article earlier this year which completely summed up my reasons for doing a blog. It’s called VMUG, Community and you (me). In that post Anthony talks about his VMUG contribution, his blog, career and how other skills have developed. All thanks to taking an active part in the community.

For me, I basically use the blog as a means to share my thoughts and experiences and probably most importantly as a way to cure professional isolation, similar to Anthony. I also see it as a way to provide assistance to someone else who may face similar challenges. I’ve been lucky enough to have been dug out of some holes thanks to someone else taking the time to write up their experiences and fixes to problems and I feel it’s only right that I reciprocate. Maintaining a blog and setting myself challenges to produce x number of blog posts does not come naturally to me. Writing doesn’t come naturally to me. It’s something I’ve struggled with but I’ve found that writing blog posts has been a great way of forcing me to be more concise. Another upside, and this is invaluable really, is that it has helped me formulate my opinions and understanding of technology. Through researching topics to ensure that what I’m writing is accurate I’ve gained a far more in-depth understanding of the core concepts of a number of technologies and this has without doubt made me a better employee.

VMware Metro Storage Cluster (vMSC) allows vCenter to stretch across two data centers in geographically dispersed locations. In normal circumstances, in vSphere 5.5 and below at least, vCenter would be deployed in Link-Mode so two vCenters can be managed as one. However, with vMSC it’s possible to have one vCenter manage all resources across two sites and leverage the underlying stretch storage and networking infrastructures. I’ve done previous blogs on NetApp MetroCluster to describe how a stretched storage cluster is spread across two disparate data centers. I’d also recommend reading a previous post done on vMSC by Paul Meehan over on www.virtualizationsoftware.com. The idea behind this post is to provide the VMware view for the MetroCluster posts and to give a better idea on how MetroCluster storage links into virtualization environments.

The main benefit of a stretched cluster is that it enables workload and resource balancing across datacenters. This helps companies to reach almost zero RTO and RPOs and ensure uptime of critical systems as workloads can be migrated easing using vMotion and Storage vMotion. One thing to keep in mind regarding vMSC, it’s not really sold as a disaster recover solution but rather a disaster avoidance solution when linked with the underlying storage. Some of the other benefits of a stretched cluster are:

Workload mobility

Cross-site automated load balancing

Enhanced downtime avoidance

Disaster avoidance

System uptime and high availability

There are a number of storage vendors that provide the back-end storage required for a vMSC to work. I won’t go into the entire list but you can find out more on the VMware Compatibility Matrix site. The one that I have experience with is NetApp MetroCluster but I know of others from EMC and Hitachi at least. So what components make up a vMSC? It comes down to an extended layer 2 network across data centers so that vMotions can take place with ease and also a resilient storage platform connected to ESXi via VMFS or NFS datastores. VMware vCenter itself does need some configuration changes but it’s nothing outside the scope of what a regular VMware admin can implement. A view of what a vMSC looks like is below. The networking and storage components have been simplified.

I ran into an interesting problem that took a bit of digging around to both find the root cause and also to find the final fix. When running backups on Vmware 5.5 running on NetApp storage I could see some, but not all VMs, failing and throwing up the below errors in the event logs

The key alert here is Event ID 12289. It was also the most off-putting. It initially looked like a floppy drive issue but there was no floppy drive attached to the VM nor were there any floppy drivers installed on the VM. A look around the VMware community forums led me to this posting – https://communities.vmware.com/thread/309844?start=0&tstart=0 It was focused more on vSphere 4.1 however and most of the advice was around installing an older version of VMware Tools. Comment 27 was the jackpot winner. The System Reserved partition was causing the issue.

So what does the System Reserved partition do?

The System Reserved partition contains the Boot Manager and Boot Configuration data that are read on start up of the virtual machine. The VM boots from the boot loader n the System Reserved partition and then boots Windows from the System drive. It is also used as a location for the start up files for BitLocker Drive Encryption. If you need BitLocker then you’ll need to have a System Reserved partition. For Windows client OS’s then that’s a great feature to have but from a server OS perspective where BitLocker just isn’t used then it’s superfluous. The System Reserved partition is created by default on OS installation so there’s two options to remediate.

Remove the partition manually post installation

Remove the partition from your Windows OS templates

I won’t go into the details on how to remove the partition from your templates here but you can find more information over on mydigitallife.info which can be used. I ran through the steps myself to do this for all of our Windows templates following finding the root cause of the initial error.

As per one of the links mentioned in Comment 27 in the VMware communities post it’s possible to change the location of the boot files so that the partition can be removed. This information can be found over on geekshangout.com. However the steps didn’t include how to re-claim that partition so that there isn’t an unallocated disk partition sitting in front of the C drive (disk 0). While I haven’t tested backups in this configuration I wouldn’t be surprised if it cause other issues during backup. So below I’ve listed the steps to follow so you can successfully remove the partition as per the steps on geekshangout and then re-claim the space on gparted.