I recently had to make some changes to my scom test environment server. It’s a single virtual machine with scom, sql, ssrs, the works on it. Despite what one would think, it has performed well with about 30 agents. Recently I added about that many more and it pushed up the ram to the 4gb max that I had given it. When I added more I realized I had used 2008 x32 for the OS (oops) and was not able to see the extra ram. After some research I found that I could do an upgrade from 2008 standard to enterprise, which is what I did. Afterwards, scom wouldn’t run because the SDK service would not start. This is the event log message.

At first I really liked this MP, it knows a LOT about exchange and there was some serious effort put into making sure it grabs everything. After a while though, there are some things you need to be able to change, but can’t.

Take this alert for disk space, we want to change the % that it alerts on, well guess what you can change with the override?

That’s right, the only thing you can do is enable or disable the rule.. that’s it. And while we’re on the subject of disabling a rule, that’s not working for at least this one:

Here’s a few of the instances of this alert..

And if you look at the overrides, this rule is clearly disabled… but still alerting.

I could go on for days about SCOM and the URL monitoring and how it needs to be improved. Honestly.. it kinda sucks. So here I will attempt to describe what I think is wrong with it and how I work around it. The items in bold below are what I feel like are failures in the way this was designed.

To begin with, you will need to figure out what you need to monitor. In many cases it is simple enough to pull up the main page of a website and as long as it comes up, is in a reasonable timeframe, and is giving an HTTP status code of 200, you’re OK. This sort of monitoring is useful, but you can do so much more in order to get a lot more out of it. What I like to do is get the devs to code you up something special through some sort of bribery or blackmail. In our case what they did was define 5 business processes, for example “make a payment” and create a page that does the back end work of making that transaction but also the other end of the work which is cleaning up after itself. What you will get in the end isn’t exactly user experience, but it’s a good way to track the ongoing performance of a process relative to itself, and it’s a very good up/down indicator. Since we have dev environments as well, I have those on a development scom server, and I have the below web monitoring in place there as well in the first production like environment. This allows our QA folks to compare state and response time and see if the environment is working before they release code or start a test, but also they can see the impact of the new code by comparing response times from before and after the code release.

Once you have your URL’s, it’s time to get to work.

Create a web application monitor and give it your URL. The problem with those default settings is that by default you are only logging the transaction response time and not alerting on it. From an alert standpoint, there is no timeout for your web request, matter of fact, the only thing SCOM will tell you out of the box is just if it was eventually able to pull up a URL as long as it doesn’t have an HTTP response code > 400. This default setting is not useful!

To fix this, what you want to do is add response time criteria like this.

Because of a problem with the service level dashboard that I will explain later, I only put one HTTP request in each web application monitor. This brings me to a little UI weirdness here because you can also set response times in the “configure settings” for the specific URL pull like this.

I always leave this performance criteria blank because I can see the other one easier and get more out of it. This one here just seems redundant.

Seeing the data

Now once you gather some data you will want to, well, see what’s going on. In order to do this, create a new performance view in the monitoring console and scope it to “collected by specific rules”, and then you get to go manually pick your rules. This is where Microsoft fails again, because the list of rules is not searchable and they all have arbitrary names. For web requests I figured out they are called “Performance Collection: Transaction response time total for Name of web app monitor”. like this screenshot.

Now that you have done that, you will be able to see a nice blank performance chart with some stuff to check.

Now when we pick one, we get a pretty graph like this.

This brings me to my next issue with all of this.. it’s that the performance chart settings are user specific.. meaning I cannot create a view of any sort that contains performance information and have the counters checked already. No matter which ones I put in, and it doesn’t matter if you are using a performance view or even a dashboard view that contains a performance view, those have to be selected every time. This is a pain!

This also means that if you wanted to say, get fancy with a URL to a specific view, you cannot just create one of these and have folks click the link and end up at a pretty performance chart with the counters already checked. The fact that you cannot do this is a serious limitation with SCOM, IMO.

setting up alert parameters (what you cannot change)

You will likely have to play with the values a bit in order to get them not to false alert. And this brings me to my next problem with SCOM web monitoring, it’s that you cannot change anything about how it samples other than where it is from (what host) and how often it samples. What I would love to do is be able to say “only alert when two consecutive thresholds are exceeded”, but that’s not an option. We get a lot of failures at night during our backup window that cause a single transaction to go out of SLA, and we get alerts based on that. As a result, we have to set our thresholds for response time to the highest level it could possibly be so that we aren’t false alerted every night, but this makes it so high that the alerting becomes less useful during the daytime. As of now I do not have a workaround for this.

stopping duplicate alerts

When you do get your first alert you will see that two are sent.. one for the URL pull and one for the aggregate monitor on the web application monitor. This doesn’t really make sense to me why this would be set up this way at all, so let’s fix it.

Start by right clicking on one of the alerts and open the health explorer for it. Expand it out and you will see something like this.

Each of the red lines has an alert set up for it, and the lower one for the actual request rolls up into the web application one. In my mind the web application one is redundant, so I am going to disable it. Right click, choose “monitor properties”, go to alerting, and uncheck it.

Now you will receive one alert instead of two.

useful alert details

Of course the text of the alerts isn’t useful at all out of the box (it doesn’t tell you if the URL failed for time, SSL, http response, or anything). I am using this article as a basis for fixing this, but I don’t have it totally worked out yet. This will continue to require some further tweaking.

This post ended up being longer than I intended (there’s a lot to fix) so I am going to break it up into two parts and get the service level dashboard stuff into a 2nd post.

Performance data collection process was unable load SQL Server Authentication configuration information. Account for RunAs profile in workflow "Microsoft.SystemCenter.DataWarehouse.CollectPerformanceData", running for instance "INFMGT02.accessgeneral.com" with id:"{81890C12-35B3-7AEA-C0FF-3EFCA7486E97}" is not defined. Workflow will not be loaded. Please associate an account with the profile. Management group "Access"

OK guys WHICH profile? Come on.. how hard is this? I mean I can guess, and I have, and guess what? It has one associated.

In case you weren’t aware, for SCOM to work against a non domain machine, all manner of certificates is required between the RMS and the agents in order for this to work. Not only is it required, but you have to use the fairly archaic tools provided with certificates, oh, and you will need your own certificate authority too. This is such a complete and utter #FAIL that I don’t really know where to start. Mainly my issue is that it doesn’t need to be this hard.. if someone wants to see the CPU time on my webserver, then by all means, hack in, but damn if I care enough to go through this level of work for it. And that brings me to my second issue, the shit just doesn’t work. Sure you could say this is a “rush it out the door” kinda thing, but this happened back in 2007 and there have been plenty of releases including an R2 version, yet still this useless and archaic process is still in place.

So in short, the SCOM guys failed by over-complicating something that isn’t needed, and then making it 10 times more difficult than necessary. FAIL.

Since I’m a bit of a mediacenter advocate, I use the internets on the tv a lot. One of my issues with flash is that it never stays full screen if you click the mouse on another monitor. Today I ran across a how-to at lifehacker on how to fix this. It’s temporary (it will break when they upgrade versions) but it’s better than nothing.