Stop error 0x0000007a occurs on a virtual machine that is running on a Windows Server 2008 R2-based failover cluster with a cluster shared volume, and the state of the CSV is switched to redirected access

UPDATE: The hotfix for issue 3 has become available today, April 26th 2011 as announced on the TechNet forum here:

A hotfix is now available that addresses the Win2008 R2 service pack 1 issue with Validate on a 3+ node cluster. This is KB 2531907. The KB article and download link will be published shortly, in the mean time you can obtain this hotfix immediately free of charge by calling Microsoft support and referencing KB 2531907. Update 27/05/2011 Here is the link: http://support.microsoft.com/kb/2531907/en-us?sd=rss&spid=14134

Issue: When you try to install Windows Server 2008 R2 on a computer that has more than 64 logical processors, Windows Setup may stop responding in one of the following operations:

Initialization of Windows Setup

One of the two restarts that are required to complete Setup

Cause: This issue occurs because of an error in the Network Driver Interface Specification.This issue occurs because of an error in the Network Driver Interface Specification (NDIS) driver. When a computer has more than 64 logical processors, the NDIS driver does not correctly handle some operations. Therefore, the computer encounters stop responding issues and other system failures.

I don’t have any nodes under my care who have more than 64 logical processors so that’s why I guess But with ever more cores available you it’s bound to happen in the near future.

Update 2: To keep me busy this KB article was released within 24 hours of me posting this blog on a BSOD with CSV and redirected access for witch a hot fix is available

Stop error 0x0000007a occurs on a virtual machine that is running on a Windows Server 2008 R2-based failover cluster with a cluster shared volume, and the state of the CSV is switched to redirected access

Disclaimer: The Dilbert® Life series is a string of post on corporate culture from hell and dysfunctional organizations running wild. This can be quite shocking and sobering. A sense of humor will help when reading this. If you need to live in a sugar coated world were all is well and bliss and think all you do is close to godliness, stop reading right now and forget about the blog entries. It’s going to be dark. Pitch black at times actually, with a twist of humor, if you can laugh at yourself.

Have you ever worked in an office where no one ever takes responsibility and all communication is CC’d and BCC’d to an absurd number of people? There are corporate “cultures” that act very different from our own style. Size often has nothing to do with it. But this habit is just another symptom and indication of blatant management failure.

These are organizations where no one feels like they can make decisions or take actions without involving half of the company in some form of meeting or committee. CYA (Cover Your Ass) in action. One of the symptoms is the fact that just about anyone who has (or thinks they have) some important or urgent information sends all mail with some managers in CC or even BCC. Often the middle management acts the same way and before you know it more CC & BCC recipients are involved making the entire mail flow a mess and proving without any doubt that you’re in kinder garden. Decisions are postponed or never made. No one is going to take responsibility for a decision, that much is for sure. So when a decision is finally made it often by the wrong person, too late and probably not the best one. Basically you have a management structure where no one knows who’s responsible and is utterly dysfunctional

This is also a symptom of another issue: managers without authority. Yes, they are not very good at their area of expertise; they can’t delegate or organize and lack real people skills. These are often found in middle management where they can be used by the upper level. After all there needs to be someone between the hammer (management) and the anvil (employees). You see authority does not come from your rank or pay grade. It comes from what you know and can do and the support you get from other well respected managers or leaders. If you need to CC all your bosses and all bosses of the people you’re mailing that indicates that you’re a whining kid that can’t hack it. And no, simply not using CC or BCC anymore won’t solve that problem. I thought I’d mention this as they tend to think and act rather simplistic. We have a saying: “You salute the rank, not the man. You respect the man, not the rank.”

Anyway, the mail process is as most people in the mail are not involved, don’t care, don’t need and shouldn’t care and hopefully don’t want to care. Once it got so out of control I added some more people to the CC list and wrote sarcastically at the top of the mail body that we really should make an effort to senselessly involve as many people as we possibly could. Not very nice, I know. Shouldn’t do that, I know. Some got the message, some didn’t. Another solution is to ignore the mail. Really if so many people, including a bunch of managers above and way above me are in the CC list I would not have the arrogance to assume I have anything to say in the matter and thus I await their proposals or decisions.

The best employee a manager can have is Vanilla Ice. Really “…If there was a problem yo I’ll solve it …”, that’s what a good manager needs and wants. You see your boss has better things to do than micro manage the details of your incompetence. You know your end state, so all you need to do is figure out what you need and how you’ll achieve it. Results, that’s what your boss really needs, not details and moaning about how hard middle management is. I know shit flows down and gripes flow up but try to maintain a balance or you’ll find yourself holding a pink slip or being promoted to where you can do the least damage and annoy the least amount of people. I secretly think some people have that as a cunning plan .

But if you’re stuck with a couple of micro managers, do not despair. You can work around them, unless they surround you. In the latter case, break out and run! They deal with urgent and very pseudo important problems that are actually just details which are benign in nature and are not negatively affected by all that overzealous attention. So the trick is to keep it that way. You have to treat them like mushrooms: keep them in the dark and feed ‘m shit. As long as they don’t know any better and keep getting their “data” fix they are lovable. Whatever you do, don’t give ‘m real information or show them the real problems. Micromanagers really can’t handle them. Ambitious ones that get into the light and get gourmet food can become very dangerous. Both to themselves and the organization. Now you do need ‘m to keep ‘m involved and they need to sign and approve work and proposals. So give ‘m finished work, solutions that are ready to go. Forget about involving them in the details or the decision making process, they’ll just get lost. And guess what, this is like a good boss should work and act so we have a win-win situation for the entire company!

Now you know how to help prevent that e-mail becomes a burden instead of a useful tool. No CC or BCC unless really needed. Go practice it.

Great news from the educational & conference front. First of all, I’m attending BriForum in London, United Kingdom in May (http://briforum.com/Europe/index.html). That’s good news, normally we’d have to pop over the big pond to go to that one, so this is pretty neat. And timely, due to some prospecting I’m doing for Disaster Recovery, Business continuity, application aware storage in a virtualized environment It’s a good match and I hope to get in to some educational discussions about the challenges we all face. Some of the storage vendors we’re interested in are there as well so there is certainly some potential to make it a good experience.

And just recently confirmed that The Experts Conference is coming to Europe. TEC2011 Europe will be held in Frankfurt, Germany from October 17th to October 19th 2011. This conference is high quality and created to fill the needs of the most experienced users, which is one of the reasons I would like to attend. The more you learn & grown the more you bump into the next level of challenges and being able to learn form high level content and interact with experienced speakers and attendees who are dealing with the same issues can be very rewarding. Attendees of TechEd have a way to measure the level of the sessions, well, they are all supposed to be Level 400 only. Quest is hosting this, so they certainly should be able to round up the expertise. I’m going to make it to the new “track” at this conference and that’s “Virtualization & Cloud”. More information can be found here http://www.theexpertsconference.com/europe/2011/virtualization-cloud-training/overview/

Ever since Windows 2008 R2 SP1 became available people have been waiting for Windows Hyper-V Server R2 to catch up. The wait is over as last week Microsoft made it available on their website http://technet.microsoft.com/en-us/evalcenter/dd776191.aspx. That’s a nice package to have when it serves your needs and there ‘s little to argue about. Guidance on how to configure it and how to get remote management set up has been out for a while and is quite complete so that barrier shouldn’t stop your from using it where appropriate. If you’re staring out head over to José Barreto’s blog to get a head start and here’s some more information on the subject http://technet.microsoft.com/en-us/library/cc794756(WS.10).aspx and naturally there are some tools around to help out if needed and the Microsoft provided tools are not to you liking http://coreconfig.codeplex.com/. So there you go, now you have a free and very capable hyper visor available to the public that gives you high availability, Live Migration, Dynamic Memory, Remote FX and they even threw in their software iSCSI target 3.3 in to the free package so you can build a free iSCSI SAN supported by Microsoft. Live is good .

You install the Hyper-V role on a computer that is running Windows Server 2008 R2.

You run a virtual machine on the computer.

You use a network adapter on the virtual machine to access a network.

You establish many concurrent network connections, or there is heavy outgoing network traffic.

In this scenario, the network connection on the virtual machine may be lost. Additionally, the network adapter is disabled. Note You have to restart the virtual machine to recover from this issue.

We’ve seen this one on VM’s that have indeed a lot of outgoing traffic. In our environment the situation looks like this:

You can access the VM with Hyper-V Manager or SCVMM but not via RDP as all Network connectivity is lost. The status the guest NIS is always “Enabled” but there is no traffic/connectivity

You can try to disable the NIC but this tales a very long time and when you try to enable it again this never succeeds. Disconnecting the NIC form the virtual network and connecting it again doesn’t help either.

You need to shut down the host but this takes an extremely long time, so long you really can’t afford to wait if it ever succeeds. It seems to hang at shutting down with a “non whirling whirly”. So finally you’ll power off the VM and start it up again. Apart from entries related to having not connectivity the event logs are “clean” and there is no indication as to what happened.

Well this exact same issue is back with Windows 2008 R2 SP1. That’s the bad news. The good news is there is a hotfix for it already so you can fix it. You can read up on this issue in Knowledge Base article 2263829 and request the hotfix here. Instructions to get the hotfix are in there as well as a reference to the previous fixes for Windows 2008 R2 RTM.

Consider the following scenario:

You install the Hyper-V role on a computer that is running Windows Server 2008 R2 Service Pack 1 (SP1).

You run a virtual machine on the computer.

You use a network adapter on the virtual machine to access a network.

You establish many concurrent network connections. Or, there is heavy outgoing network traffic.

In this scenario, the network connection on the virtual machine may be lost. Additionally, the network adapter may be disabled. Notes

You must restart the virtual machine to recover from this issue.

This issue can also occur on versions of Windows Server 2008 R2 that do not have SP1 installed. To resolve the issue, apply the hotfix that is described in one of the following Microsoft Knowledge Base articles:

Oh yeah, people often seem confused as to where to install the hotfix. Does it go on the Hyper-V hosts or and/or on the guest? It’s a hyper visor bug in Hyper-V so it goes on the hosts. Have a nice weekend.

Disk with identifier 2sef8cdf has a Persistent Reservation on it. The disk might be part of some other cluster. Removing the disk from validation set

“Normally” you would expect such a warning if the LUN ever belonged to another cluster and it needs the old reservation cleared. To do that you would use following command on the node that throws the warning (where in this example the disk is disk 2 in disk manager/diskpart) and after making sure it is not in use anywhere else in the SAN

"cluster node clusternode1 /clearpr:2"

However this is not the cause here as were most others in this discussion. And I’m pretty no san software or MPIO software is putting a reservation on there either so what is this? A bug? Well yes, it has been confirmed by Microsoft support that is is indeed a bug an that is fix will be made available by April 18th2011 .

This was not a show stopper bug, but it could be one if you needed to add a host to a cluster and confirm all is well and supported. However if you’re certain you’ve done everything right you can choose not to run cluster validation.

I will update this blog with more information when the fix becomes available.

UPDATE: The hotfix has become available today, April 26th 2011 as announced on the TechNet forum here:

A hotfix is now available that addresses the Win2008 R2 service pack 1 issue with Validate on a 3+ node cluster. This is KB 2531907. The KB article and download link will be published shortly, in the mean time you can obtain this hotfix immediately free of charge by calling Microsoft support and referencing KB 2531907. Update 27/05/2011 Here is the link: http://support.microsoft.com/kb/2531907/en-us?sd=rss&spid=14134

Due to an issue with Blackberry devices Exchange Server 2010 SP1 Update Rollup 3 was withdrawn on March 14th. See my blog on this here.

The issue was identified and fixed so now there is a new release. My environment doe not use Blackberries so we were not impacted, We had already deployed RU3 so all we have to do is apply the current release over that one and we’re good to go.

It’s a good thing it’s back available since if bring back UDP support which is a much asked and need feature for Outlook 2003 and it also fixed the DAG copy GUI bug introduced with SP1 (September 2010) where you could see and thus mange the copies anymore with EMC.

As TechNet subscribers we had access to Windows Storage Server 2008 with Microsoft iSCSI Software Target 3.2 (also see Jose Barreto’s blog on this here). That was sweet but for one little issue. This SKU cannot be a Hyper-V Host. In order not to lose a physical host in the lab you could edit the MSI installer from the Windows Storage Server 2008 install media where you would delete the SKU check. Problem solved but not very legal so nobody ever did that . You can install Windows Storage Server in a VM for the lab I know but that becoming very SkyNet like … Virtual servers providing virtual storage for virtual servers … and while a good option to have I like to have a hardware host.

Bring Windows Storage Server 2008 R2 along and Microsoft decided that we could have the have the iSCSI Software Target 3.3 software without constraints, except that you needed a TechNet/MSDN subscription, to install on W2K8R2. This is the one I’m running in my labs at the moment installed on a Physical Windows Server 2008 R2 Enterprise edition that also is a Hyper-V host. This provides all my iSCSI storage to both physical and virtual clusters . I used it to test MelioFS with FileScaler recently with a 2 node virtual cluster.

Today, Jose Barreto blogged about the public release of iSCSI Software Target 3.3 for Windows Server 2008 R2. This is very good news as now everyone has access to an iSCSI target for labs, testing, POCs and even production. Thank you Microsoft . Now with some luck we could get some SMI-S support for it with SCVMM2012 ? Please?

If you need some help, Jose Barreto has a bunch of blog posts on configuring the iSCSI target, so I suggest you check out his site. As an added benefit, Microsoft iSCSI Software Target 3.3 setup & configuration is scriptable using PowerShell.

The cause is that on these versions of the operating systems the the required memory enlightenment is supported. After installing the hotfix (or SP1 in the case of Windows 2008 R2) memory addition enlightenment is available on these SKU and your good to go.

But recently I ran into an issue. One virtual machine guest on a a Windows 2008 R2 SP1 cluster that is running the Windows 2008 Standard edition without Hyper-V SKU, x86 the stand alone installer for Windows6.0-KB2230887-x86 throws following warning:

—————————

Windows Update Standalone Installer

—————————

The update does not apply to your system

—————————

OK

—————————

That’s a bit strange. As you can see the hotfix has not been installed. And why would I need the with Hyper-V SKU to use dynamic memory with this guest?

I installed the same hotfix without any issue on several Windows 2008 Standard Edition (the SKU with Hyper-V) guests. So what is going on here? What exactly is different about the without Hyper-V SKU? Note that Windows 2008 R2 no longer has such a SKU. Microsoft states that it is an SKU for people who don’t need virtualization. Well as they installed it as a guest virtual machine they chose this version and now what? No hotfix for dynamic memory as a guest?

There is a TechNet forum discussion going on here where they have this issue and I asked them to check what SKU it was happening on. Bingo! It’s on the Windows 2008 Standard without Hyper-V version!

I don’t expect any Hyper-V functionality or management tools but I do expect to get the hotfix to enable Dynamic Memory in guests with this SKU, just like we get any other update that is relevant to Hyper-V, for example upgraded Integration Services. I’m going to ping a Microsoft contact on this one and see if they can help with this. And in case you’re wondering SKU = Stock Keeping Unit.