Monday, June 30, 2014

IssueFor a customer the Dell MP (only the Scalable MP, not the Detailed MP…) is in place. However, many Dell servers aren’t discovered, even though all prereqs are in place AND operational. But no matter what, these servers weren’t do be discovered as Dell servers. Time for some troubleshooting.

This Discovery can be modified through the regular mechanism of setting Overrides. In this case I set the Discovery to run once per minute for a particular Dell server having these issues AND I enabled logging of this Discovery by setting the Parameter NameLog Level to 1:

Because of this the Discovery will create the log file ‘C:\Windows\Temp\Discovery_DellServer.log’. Soon when the Overrides were distributed to this server and processed by the Microsoft Monitoring Agent, the log file was created and neatly filled with information. And this is what came out of it:

So besides the hardware manufacturer Dell, there is also a manufacturer DELLOE… .

Time for a deeper investigation of the Discovery involved. I exported the MP to a plain XML file and used Notepad++ to open it. Soon I found the Discovery and it’s script, DellServerDiscovery.vbs. This is a huge script containing many functions, among them the function GetManufacturerName:

So this function runs this WMI query: Set wbemObjectSet = GetObjectSet(wbemServices, "select * from Win32_ComputerSystem", "Manufacturer", "Win32_ComputerSystem"). Since the log file contains the entry INFO: HardwareManufacturerName Name retrieved successfully, I know this function run just fine. However, the value it contains isn’t accepted… Back to the Discovery script.

And yes, here I found the pain: Instr is case sensitive:

So this causes the Discovery not to work for servers with DELLOE as hardware manufacturer…

Where does DELLOE come from?As is turns out, DELLOE is related to Dell servers running an OEM BIOS. Apparently this information is piped into WMI as DELLOE.

However, installing a Dell BIOS on servers like these isn’t the way to go since I don’t have any idea to what extend it will impact the operations of this server in a negative kind of way.

Solution?As this moment I am thinking about multiple ‘solutions’ or better, workarounds. Ranging from dirty to acceptable but involving a lot of work. First I have to discuss them with other people and look into them. When I know more, I’ll update this posting accordingly. For now these Dell servers running an OEM BIOS won’t be covered by SCOM .

Monday, June 16, 2014

For any SCOM 2012x environment there are some community efforts out there which are just awesome and add that little bit of extra which makes it far more easier to manage and/or enables deeper monitoring for a certain service/product.

Please feel free to comment with your own recommendations. I’ll update this posting accordingly. And PLEASE don’t be angry when you’re not named here. Just send in your comments and I’ll update this posting.

Tao YangNo, he’s a person so you can’t import him into your environment, even though SCOM would really benefit from it . However, the results of his hard work and dedication to the SCOM community are really awesome and impressive. Not only because they fill up many gaps but also of the level of quality and ongoing improvements. Therefore Tao has earned my deepest respect.

Stefan RothFormer MVP, SCOM guru and PowerShell Master. He shares his insights, ideas, code and tools with the community. His blog is always packed with really good information.

- MAST Tool. I already blogged about it. A must have for anyone using SCOM to send out Notifications.

Raphael BurriAnother Master of SCOM! Switzerland makes more besides watches and chocolate. Also HIGH quality MPs and this one comes for FREE! And yes, I am talking about the

- PKI Verification MP. Recently updated (totally rewritten from the ground up). My customers love this MP since it has saved them from embarrassing situations and loosing tons of money (like: website for customers down because of an expired SSL certificate).

System Center Central > SCC Health Check Reports MPWritten by Pete Zerger and Oskar Landman, two other SCOM Heroes. Even though the SCC Health Check Reports MPisn’t updated for a very long time, it works great with SCOM 2012x, up to R2! And it gives you a deep insight in SCOM itself and its health.

I know. There’s more. But my intention with this posting is to start a process where YOU (the reader of this blog) adds his/her comments, all about other great tooling, scripts and so on to be used in SCOM. And again, I’ll update this posting accordingly.

IssueThis one was hard to crack: it took me a whole day to identify the culprit, the underlying mechanisms, the why and the possible solutions/workarounds…

Because of Patch Tuesday some W7 based images had to be updated with the latest security updates. For some good reasons we don’t use Offline Services for OS Images. Instead we deploy the W7 image and install the software updates provided through ADR, all by a single Task Sequence.

So far so good. But when it was time to capture the newly build image, trouble started. For some reason the SCCM Client didn’t contact the SCCM MP. Even though the boundaries were okay, it just didn’t work. The SCCM servers involved looked upon the system as one without a SCCM Client, even though it was installed and running.

Time for additional checks. The SCCM Client on the newly build system was running but anything from okay.

The Client certificate wasn’t present:

Most components weren’t enabled so the list of Actions was very limited. And when Software Center was run, it showed the default values instead of the values set for this particular SCCM Site.

The log files PolicyAgent.log and LocationServices.log weren’t okay either. The first one contained serious errors last one missed crucial configuration information. And the log file ccmsetup-ccmeval.log was really alarming. Normally this file isn’t present at all! Also the last two entries in that log file told me something was seriously wrong:

So the SCCM Client was uninitialized!

Provisioning ModeAs it turns out the SCCM Client was in PROVISIONING MODE. Basically this leaves the SCCM Client fully installed and running but without an active configuration. So it won’t contact any SCCM MP, ever.

By the way this is the way the registry looks like when the SCCM Client is stuck in Provisioning Mode:

But what caused it? Time for a test. I ran the same build Task Sequence again, but now with the Install Software Updates Task disabled. And now the SCCM Client was fully initialized and functional. The SCCM MP was found by the SCCM Client and all policies came in and were processed. A self signed was made and loaded. Software Center was configured as intended.

So somewhere the cause of the SCCM Client ending up in Provisioning Mode was caused by the software updates present in the Patch Tuesday release of June 2014.

But before I continue some additional information about the why a SCCM Client is normally put into Provisioning Mode (and later removed from that mode) and why it’s a GOOD thing when running a Task Sequence.

The mechanism of SCCM Client during a Task SequenceWhen running a build Task Sequence the Task Setup Windows and Configuration Manager will install the SCCM Client. So this is a good thing.

However, when this SCCM Client becomes fully active while the Task Sequence is still running it might become a huge show blocker because it contacts the SCCM MP, downloads all kinds of policies and will process them. Changes are it will collide with the running Task Sequence resulting in a wrecked system, which is bad.

So therefore the SCCM Client is set into Provisioning Mode while the Task Sequence is running. At the end of it the SCCM Client will be removed from that mode, resulting in a fully functional SCCM Client.

However, in order to get there, the Task Sequence has to be AND stay in control of the whole build process. When something happens resulting in a (partially) broken or interrupted Task Sequence, changes are the SCCM Client won’t be removed from Provisioning Mode.

Time to move on to the cause…

The Big Internet and the BAD thing of it…Yes, I know what you’re thinking. Why not modify the mentioned registry keys (can even be done by the SAME Task Sequence(!)) and be done with it.

HOWEVER, that won’t fly. Simply because it won’t fix it for EVER. Every time a Scheduled Task is run on every system running the SCCM Client. This Task ‘Configuration Manager Health Evaluation’ will reconfigure the SCCM Client BACK into the Provisioning Mode.

Why? Well for this particular Scheduled Task the file MobileClient.TCF is used. And guess what? It contains (among many other entries) this entry:

So all the blog postings out there telling you to modify the registry keys in order to fix this issue are simply wrong! It will only fix it for some time, leaving you later with the same issue all over again.

There is also another blog posting telling you not only to modify the registry keys but also to fix the mentioned file. And yes, this will fix the issue for good. BUT… it’s still not a solution only a workaround at best and the real cause is still there, unaddressed and still wrecking havoc.

I for myself don’t like solutions like these because you’re not in control of the whole process.

The ROOT causeLike the title already mentions, the REAL cause of the SCCM Client being stuck in Provisioning Mode is a security update released in the Patch Tuesday cycle of June 2014.

The Security Update isn’t the real culprit but the way it works. Since this particular update requires MULTIPLE reboots. And a Task Sequence can handle ONE reboot for an update just fine. But a second reboot for that SAME update will cause the Task Sequence to FAIL and/or the SCCM Client being stuck in Provisioning Mode.

Even though the same KB article states to be updated with updates requiring multiple restarts, it’s not up to date:

So how to find it? Luckily the internet has GOOD information as well and soon I found this blog posting from the company Ardamis, all about an update released with the Patch Tuesday release of June 2014, requiring MULTIPLE reboots!

However, this isn’t only at play for x64 systems and x64 based update. But ALSO for the x86 version of it!!! See MS14-030.

Solutions/ workaroundsSo now we’ve located the culprit and the root cause of it all. But how to fix it? Let’s make a list of the available solutions:

No, I won’t use this ‘solution’ or workaroundModifying the registry keys.The pain: Is only a temporary fix but won’t address the root cause.

Nor will I use this ‘solution’Modifying the registry keys AND related file entries.The pain: Even though it fixes the SCCM Client being stuck in Provisioning Mode, it still doesn’t address the root cause NOR does it guarantee the related Task Sequence has run for a full 100% without ANY glitches…

Nope, this one isn’t a good one eitherExcluding the updates causing multiple restarts from the Task Sequences and run them through the normal channels, using the SCCM Client.The pain: I don’t want to TEST/EXAMINE ALL updates released monthly by Microsoft in order to see whether one of them or more, causes the system to reboot more than once.

Could be a possible workaroundExcluding ALL updates from the Task Sequences and run them through the normal channels, using the SCCM Client.This resolves this issue WITHOUT the pain mentioned in the third option. But it’s not a solution.

OS Image creation/updating is something which shouldn’t be done solely by SCCM 2102x But MDT 2013 instead.This is a whole different discussion but something I’ve heard from multiple highly respected sources who have far more DEEP SCCM knowledge and experience compared to me. At WMUG Netherlands we had the honor to have Johan Arwidmark presenting a webinar about this topic in English. This webinar is recorded and to be found here.

Tuesday, June 3, 2014

Last months a whole bunch of FREE ebooks has been released by Microsoft Press, all about the System Center family.

Here is an overview of all of these titles. These books are a great opportunity to learn more about these technologies even though some of these books contain a lot marketing mumbo jumbo. None the less, they are still a great way to start learning new stuff, paving the way for the bigger and better stuff out there .

IssueWhen following this TechNet article all about installing a SCOM 2012x Gateway Server, also by using the command prompt (header To Install the gateway server by using the Command Prompt window), you’ll certainly bump into the situation where the Gateway Server won’t install. Ever!

ProblemsThe log file only becomes a few KBs big and it will every time show the message that the installation failed without stating a clear reason why.

And double checking the earlier mentioned KB article will show you nothing since you followed that TechNet article to the smallest detail.

Widgets or dead wood?With SCOM 2012x new styled dashboards were released as well, based on widgets. Even though these widgets may have a sexy look & feel to them, they aren’t flawless.

Even in SCOM 2012 R2 UR#2 there are still issues like widgets which time out (the underlying SQL queries aren’t ‘spot on’ so the widget times out…), making those same widgets pretty useless in bigger environments.

Where as other widgets don’t allow scoping for a a particular set of data, like the Alert widgets making them far to generic for proper usage by scoped end user groups.

Back to basics?IMHO the widgets don’t live up to their promises yet, resulting in a quest for SCOM dashboards in the old SCOM 2007x style. These kind of dashboards are actually nothing but an aggregation of other already existing Views in the Console, enabling one to combine different Views to be presented on one single BIG screen.

The SCOM 2012x dashboard wizard won’t allow it…However, the biggest party crusher here is the wizard which enables you to build dashboards. In SCOM 2012x this wizard won’t allow you to build these old style SCOM 2007x dashboards. Instead you’re forced to use widgets…

The workaroundIn order to enable you to build SCOM 2007x style dashboards, I’ve built in my SCOM 2007 R2 CU#7 environment at home a simple MP containing most of the dashboards in SCOM 2007x style. None of these dashboards contains any view, only the placeholder enabling you to add the required SCOM Views.

Simply import this MP into your SCOM 2012x environment. Now in the Monitoring pane you’ll get the folder _SCOM 2007 Dashboards with the most used SCOM 2007x dashboard lay-outs:

The names of the Views are self explanatory. Simply select the dashboard you want to use, right click it, select Copy and paste it into the Unsealed Management Pack / View where you want to use it.

When the dashboard is pasted you can edit it by clicking Click to add a view in order to add an existing View to it.

Veeam

NiCE

Search This Blog

Didacticum

Pageviews last month

Visitors to this blog:

Why this blog?

On an almost daily basis I work with Azure, OMS & System Center related technologies. At the moment my main focus areas are Azure, OMS, SCOM & SCCM.

Because I bump into many challenges I decided to start this blog, which has two main purposes: to help YOU with mastering these products by covering the undocumented features and last, but not least, as my personal - but open to any one - knowledge base.

From January 2010 on I have been rewarded with the MVP award and until now this this status is prolonged every year.

MVP AWARD

Follow me on Twitter

Disclaimer

The information in this blog is provided 'AS IS' with no warranties and confers no rights. This blog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my own personal opinion. All code samples are provided 'AS IS' without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.