Right now, you can sign up for the VCAP6-DCV Design or the VCAP6-CMA Design. I’ll probably be going only for the former. VCAP6-DTM Design for Desktop and Mobility won’t be out until later in Feb.

You’re going to need to be either a VCP6 or a VCP5+VCAP5 to qualify to be approved.

Exam cost is $100 USD, and I’d doubt any discounts would be applicable (eg: VMUG Advantage, etc) – but I’ll try once I’m approved. First appoinments will be Feb 15 2016, and there’s no indication how long they’ll run. Access is First Come, First Served, so if you’re interested – sign up now!

Recently I’ve been having some issues in my home lab where hosts seem to stop responding. I’ve been busy, so had a tough time finding the time to verify if the hosts crashed, locked up, or if it was just a networking issue. So quite unfortunately, I’ve been rebooting them to bring them back and keeping it on my To Do list.

Until it started happening at work. We have a standalone box that is non clustered that has internal disks and functions as a NAS. So it’s “very important”, but “not critical” – or it would be HA, etc. And the same symptoms occurred there – where we of course HAD to troubleshoot it.

Tried the following:

* Disconnect the NIC’s and reconnect

* Shut/no-shut the ports on the switch

* Access via IPMI/IDRAC – works, could access DCUI to verify host was up. Could ping VM’s on the host, so verified those were working.

* Tried removing or adding other NIC’s via the DCUI to the vSwitch that has the management network. No go. Didn’t matter if the two onboard NIC’s, two add in, split across two controllers, one NIC only, etc.

Understandably we’re pretty sure we have a networking issue – but only on the one host at this time. The Dell R610 and R620 have no similar issues, but this is an in-house built Supermicro we use for Dev. So the issue appears related to the host, and having only one, it’s tough to troubleshoot.

ESXi NIC’s are not the same – the host has 82574L onboard and 82571EB 4 port. The C6100 I have is 82576 – and I haven’t confirmed it is having the same issue. But in any case, it doesn’t seem like an issue with a certain chipset or model of NIC.

After upgrade of ESXi 5.5 to 6.0 server loses every few days the network connection

Reach out to VMware support on this issue to see if it is related to a known bug in ESXi 6. The event logged in my experience with this same situation is "netdev_watchdog:3678: NETDEV WATCHDOG". I did not see this in your logs but the failure scenario I have experienced is the same as you described.

Okay, not great, but sounds like the same sort of issue. Someone then suggested that the bug is known. Okay. Check the release notes and:

New Network connectivity issues after upgrade from ESXi 5.x to ESXi 6.0After you upgrade from ESXi 5.x to ESXi 6.0, you might encounter the following issues The ESXi 6.0 host might randomly lose network connectivity The ESXi 6.0 host becomes non-responsive and unmanageable until reboot After reboot, the issue is temporarily resolved for a period of time, but occurs again after a random interval Transmit timeouts are often logged by the NETDEV WATCHDOG service in the ESXi host. You may see entries similar to the following in the/var/log/vmkernel.log file: cpu0:33245)WARNING: LinNet: netdev_watchdog:3678: NETDEV WATCHDOG: vmnic0: transmit timed out cpu0:33245)WARNING: at vmkdrivers/src_92/vmklinux_92/vmware/linux_net.c:3707/netdev_watchdog() (inside vmklinux) The issue can impact multiple network adapter types across multiple hardware vendors. The exact logging that occurs during a transmit timeout may vary from card to card.

That sounds similar. We’re still digging through logs and also waiting for it to occur again to catch it in the act.

If you’re seeing this in your vmkernel.log at the time of the disconnect it could be related to an issue that will one day be described at the below link (it is not live at this time). We see this after a random amount of time and nothing VMware technical support could do except reboot the host helped.

Also if you haven’t bumped into it before, there’s great links here for Upgrade Best Practices, Update Sequences, etc. DO give this a read.

Understandably, we won’t get this fix in TODAY. We do have a maintenance cycle coming up soon, where we’ll get vCenter Server 6.0U1 in and can go to ESXi v6.0U1a – and hopefully this will fix the network issue. If not, back to troubleshooting I guess. Fingers crossed.

I’ll post an update next week if we get the upgrade in and see a resolution to this.

When I do a review of a vSphere site, I typically start by looking to see if best practices are being followed – then look to see if any of the 3rd party bits are installed. This post picks on Dell environments a little, but the same general overview holds true of HP, or IBM/Lenovo, or Cisco, or…. Everyone has their own 3rd party integration bits to be aware of. Perhaps this is the part where everyone likes Hyper Converged because you don’t have to know about this stuff. But as an expert administering it, you should at least be aware of it, if not an expert.

I’m not going to into details as to how to install or integrate these components. I just wanted to make a cheat sheet for myself, and maybe remind some folks that regardless of your vendor, make sure you check for the extra’s – its part of why you’re not buying white boxes, so take advantage of it. Most if it is free!

You likely will not need this for a smaller installation, but it can help out if you need to standardize, by allowing you to configure and export/import things like BIOS/UEFI, Firmware, iDRAC, LCC, etc settings. Can’t hurt to have around.

There is no longer a need for a “SUU” – Systems Update Utility, as the Lifecycle Controller built into ever iDRAC, even the Express will allow you to do updates from that device. I recommend doing them via the network as it is significantly less hassle than going through the Dell Repository Builder, and downloading your copies to a USB/ISO/DVD media and doing it that way.

Now, the above covers what you’ll require for vSphere. What is NOT immediately obvious is the tools you may want to use in Windows. Even though you now have management capability on the hosts and can see things in vCenter, you’re still missing the ability to talk to devices and manage them from Windows – which is where I spend all of my actual time. Things like monitoring, control, management, etc, all are done from within Windows. So let’s go ahead and change that OS to “Windows Server 2012 R2 SP1” and get some additional tools:

Even though you’ve installed the OMSA VIB’s on ESXi, there is no actual web server there. So you’ll need to install the OMSA Web Server tools somewhere – could even be your workstation – and use that. You’ll then select “connect to remote node” and specify the target ESXi system and credentials.

If you’re managing many Dell systems and not just servers, you may want to go with OME if you do not have SCCM or similar. It’s a pretty good 3rd party SNMP/WMI monitoring solution as well. But will also allow you to handle remote updates of firmware, BIOS, settings, etc, on various systems – network, storage, client, thin client, etc.

The iDRAC is the same on every system it is only the licence that changes. To apply the Enterprise license, you’ll need the License Manager.

Hopefully this will help someone keep their Dell Environment up to date. Note that I have NOT called out any Dell Storage items such as MD3xxx, Equallogic PSxxxx, or Compellent SCxxxx products. If I tried to, the list would be significantly longer. Also worth noting is that some vendors _networking_ products have similar add ins, so don’t forget to look for those as well.

The high level explanation is that Veeam is using both TLS and SSLv3 components to connect to VMware – and VMware has disabled the SSL method in v6.0U1. There is a bug in how Veeam is auto-detecting SSL or TLS connectivity, causing this issue. Other VMware products are having similar issues talking to their own products, from what I understand.

Veeam has a KB2063 on the issue here: http://www.veeam.com/kb2063 You have two options – call in and request a private out of band hotfix from Veeam, or make changes on the VMware side.

I’ve whipped up a quick BASH script that seems to work in my testing. It will:

· see if the desired option exists and exits

· if the options exists, but is the opposite setting (true vs false, etc) it will flip the setting

· if the option does not exist, it will add it

TEST IT BEFORE YOU RUN IT IN YOUR ENVIRONMENT, I’m not responsible if it does wonky things to you.

Applying the changes does NOT require Maintenance Mode on the hosts, or any Veeam service restarting. You can simply “retry” the job on the Veeam server, and “It Just Works”

This will likely be resolved by end of September when Veeam releases the next update to Veeam B&R – or there may be a vSphere v6.0U1a released. Once the Veeam fix is released, it may be prudent to reverse or disable these changes on your hosts so you can use TLS vs SSL.

So I’m not 100% certain if the issues I’m having on my C6100 server are vSphere 6 related or not. But I have seen similar issues before in my lab, so it may be one of a few things.

After a recent upgrade, I noted that some of my VM’s seemed “slow” – which is hard to quantify. Then this morning I wake up to having internet but no DNS, so I know my DC is down. Hosts are up though. So I give them a hard boot, connect to the IPMI KVM, and watch the startup. To see “loading IPMI_SI_SRV…” and it just sitting there.

In the past, this seemed to be related to a failing SATA disk, and the solution was to pop it up – which helped temporarily until I replaced the disk outright. But these are new drives. Trying the same here did not work, though I only tried the spinning disks and not the SSD’s. Rather than mess around, I thought I’d find a way to see if I could disable IPMI at least to troubleshoot.

In a previous post, Got 10GbE working in the lab – first good results, I talked about getting 10GbE working with my Dell C6100 series. Recently, a commenter asked me if I had any pictures of the modifications I had to make to the rear panel to make these 10GBE cards work. As I have another C6100 I recently acquired (yes, I have a problem…), that needs the mods, it seems only prudent to share the steps I took in case it helps someone else.

First a little discussion about what you need:

Dell C6100 without the rear panel plate to be removed

Dell X53DF/TCK99 2 Port 10GbE Intel 82599 SFP+ Adapter

Dell HH4P1 PCI-E Bridge Card

You may find the Mezz card under either part number – it seems that the X53DF replaced the TCK99. Perhaps one is the P/N and one is the FRU or some such. But you NEED that little PCI-E bridge card. It is usually included, but pay special attention to the listing to ensure it does. What you DON’T really need, is the mesh back plate on the card – you can get it bare.

Shown above are the 2pt 10GbE SFP+ card in question, and also the 2pt 40GbE Infiniband card. Above them both is the small PCI-E bridge card.

You want to remove the two screws to remove the backing plate on the card. You won’t be needing it, and you can set it aside. The screws attach through the card and into the bracket, so once removed, reinsert the screws to the bracket to keep from losing them.

Here we can see the back panel of the C6100 sled. Ready to go for cutting.

You can place the factory rear plate over the back plate. Here you can see where you need to line it up and mark the cuts you’ll be doing. Note that of course the bracket will sit higher up on the unit, so you’ll have to adjust for your horizontal lines.

If we look to the left, we can see the source of the problem that causes us to have to do this work. The back panel here is not removable, and wraps around the left corner of the unit. In systems with the removable plate, this simply unscrews and panel attached to the card slots in. In the right hand side you can see the two screws that would attach the panel and card in that case.

Here’s largely what we get once we complete the cuts. Perhaps you’re better with a Dremel than I am. Note that the vertical cuts can be tough depending on the size of the cutting disk you have, as they may have interference from the bar to remove the sled.

You can now attach the PCI-E bridge card to the Mezz card, and slot it in. I found it easiest to come at about 20 degree angle and slot in the 2 ports into the cut outs, then drop the PCI-E bridge into the slot. When it’s all said and done, you’ll find it pretty secure and good to go.

That’s really about it. Not a whole lot to it, and if you have it all in hand, you’d figure it out pretty quick. This is largely to help show where my cut lines ended up compared tot he actual cuts and where adjustments could be made to make the cuts tighter if you wanted. Also, if you’re planning to order, but are not sure if it works or is possible, then this is going to help out quite a bit.

I know I’ve had both a lot of local VMUG members as well as forum members where I frequent, asking about when vSphere 6, vCenter 6, and ESXi 6 would be available as part of EVALExperience – as understandably, people are anxious to get their learning, labbing, and testing on.

Of course, if you’ve signed up with VMUG, you should be getting the e-mail I just received as well. I’m not certain if it would go to all VMUG Members, only those that are already EVALExperience subscribers, or what.

What is now included, as per the e-mail blast is:

NOW AVAILABLE! VMware recently announced the general availability of VMware vSphere 6, VMware Integrated OpenStack and VMware Virtual SAN 6 – the industry’s first unified platform for the hybrid cloud! EVALExperience will be releasing the new products and VMUG Advantage subscribers will be able to download the latest versions of:

vCenter Server Standard for vSphere 6

vSphere with Operations Management Enterprise Plus

vCloud Suite Standard

Virtual SAN 6

*New* Virtual SAN 6 All Flash Add-On

It is worth noting that the product download has been updated and upgraded. They do call out that the old product and keys will no longer be available. I can understand why, as part of this will be to help members stay current. But it would be nice if you could use the N-1 version for a year of transition, etc. Not everyone can cut over immediately and some people use their home labs to mirror the production environment at work so they can come home and try something they couldn’t at the office.

Some questions I’ve had asked, and the answers I’m aware of:

How many sockets are included? The package includes 6 sockets for 3 hosts.

Are the keys 365 days or dated expiry? I understand they’re dated expiry, so if you install a new lab 2 weeks before the end of your subscription, you’ll see 14 days left, not 364.

What about VSAN? There had previously been a glitch which gave only one host worth of licences – which clearly does not work. This has been corrected.

Just a friendly reminder, as a VMUG leader to look into the VMUG Advantage membership. As always, VMUG membership itself is free, come on down and attend a local meeting (the next Edmonton VMUG is June 16 and you can sign up here – http://www.vmug.com/p/cm/ld/fid=10777).

In addition, your VMUG Advantage subscriber benefits include:

FREE Access to VMworld 2015 Content

20% discount on VMware Certification Exams & Training Courses (If you have a $3500 course you need/want, plus a $225 exam, for $3725 total, spending $200 or so on a VMUG Advantage to make your costs $2800+$180=$2980 is a great way to get $745 off. This is the sell you should be giving your employer )

$100 discount on VMworld 2015 registration (This is the only “stackable” discount for VMworld. Pre-registration/early-bird ends on June 8th I believe)