Tasty Treats for System Center Operations Manager (by Tyson Paul)

SCOM Enhanced Email Notification Script Version 2.1

In September of 2010 Tao Yang wrote a fantastic Powershell script (v2.0 here) for command channel notifications in System Center Operations Manager. Here is my adaptation of the script.

NOTE: I’ve tested this thoroughly in my lab and I suggest you do the same before using in any other environments. This script is meant to be an example and therefore use it at your own risk. Tested with SCOM 2012, 2012 SP1, 2012 R2, 2016.

Version History

2017.9.28:v2.3.61Added support for execution policy bypass from command line. This is an alternative to modifying the global execution policy.

2017.8.30: v2.3.6 Improved network device details for CustomField9.

2016.5.17: v2.3.5 Fixed muli-subscriber failure problem. Delayed alert customfield updates until the very end of the script due to stale data issue (event ID 33333: “Request to update alert ignored due to invalid TimeModified”). Cleaned up a few unnecessary lines of code.

2015.5.14: v2.2.4 Added LastModified field. The script adds the “LastModified” time to the email body. However, the script still takes a few more seconds to finish and right before it finishes, it updates the alert status to “Notified” thereby modifying the alert. If you look in the Console at the LastModified time, it won’t precisely match what is shown in the email notification; the Console will show a timestamp that is slightly more recent.

2015.5.1: v2.2.1 Added support to include Parent name (IP address) and ‘sysName’ properties for network devices into CustomField9 of alert, also included in notification. Added Powershell code snippet to documentation for setting registry value to increase asynchronous process limit.

Knowledge Article:

Summary

The context information for these alerts is not always helpful. To see more detailed information about this alert log into the console for the management group.

Wiki

Note:

The context information provided in this notification is limited and is not always helpful. To see more detailed information about this alert, log into the appropriate SCOM console for the applicable SCOM management group and use the Health Explorer to find more details about the state change event(s) for the object.

Knowledge Article:

Summary

The context information for these alerts is not always helpful. To see more detailed information about this alert log into the console for the management group.

Wiki

Note:

The context information provided in this notification is limited and is not always helpful. To see more detailed information about this alert, log into the appropriate SCOM console for the applicable SCOM management group and use the Health Explorer to find more details about the state change event(s) for the object.

Directions:

In the script there are instructions at the top that describe a few of the settings (lines of code) that must be configured before use. Simply edit the “Settings” area within the script. Use Notepad or your favorite text editor. I like Powershell ISE or Notepad++ .
Configuration is very simple and there are examples for each setting within the script comments at the top.

Here’s what I recommend:

PERFORM THESE TEST PROCEDURES IN A LAB ENVIRONMENT (NON PRODUCTION):

Remove all management servers from your Notifications Resource Poolexcept foronesingle server. This will help you troubleshoot during testing if needed.

Open the Config.xml file and customize the necessary fields: smtp server, port, smtp sender, web console url, etc. See the example files also in the folder. This Config.xml file must be located in the same directory as the .SCOMEnhancedEmailNotification.ps1 script. I suggest you put these files in this path on your management servers: C:\SCOM_SCRIPTS\Notifications

Make sure that your default action account is a local administrator on your test mgmt server.

Make sure that your Powershell execution policy is not blocking the command channel execution.

View policy:Get-ExecutionPolicy

Or just run the following to be sure the policy is “unrestricted”. In an elevated Powershell console on the mgmt server, run this command:Set-ExecutionPolicy Unrestricted

Note: When using -ExecutionPolicy parameter notice the double quotes in the command above! If you use single quotes around the file name instead of double quotes your command channel won’t trigger and you won’t get any logging or events in the event log and you will lose hours troubleshooting this (like I did).

Subscribers:

Create a Test Subscriber with your email address. You can configure multiple subscribers, each with one or more recipients. The script will send to To, CC, and BCC. Notice the “TO” and “CC” preceding the addresses. If “TO”, “CC”, or “BCC” are omitted, the default becomes “TO”.
Note: the email address is in the “Name” field. With other channels the “Name” field is used for a friendly/display name. With the Command channel you use the actual email address here.

You should receive the email quickly if you configured everything correctly.

Copy the script folder to your other management servers that will be in the Notifications Resource Pool. It must be in the same path on each mgmt. server.

Put each management server alone in the Notifications Resource Pool, one at a time, and test each with the Synthetic Transaction procedure described above.

Troubleshooting:

If the script doesn’t appear to be running, open Task Manager, sort by Name, and watch for a new Powershell.exe process to spin up under the Management Server Action Account context/user . The script takes from 10-20 seconds to complete. If the new Powershell.exe process doesn’t open within 60 seconds or opens briefly and then disappears, then you have a problem.

Make sure the script itself is not being blocked from running. Right-click the script, select Properties. If you see a “Unblock” button, click it to make sure the script is not being blocked from execution.

The Powershell execution policy may be Restricted. Change this to Unrestricted. Set-ExecutionPolicy Unrestricted

There may be an error with the Channel configuration or path(s). Double check the path to the .ps1 script. Make sure the full file path is precise. I suggest navigating to the actual path where the script lives and copying the full path right from the File Explorer address/path field. This way there’s no chance of misspelling the path to the script. See command line parameter field example above.

Check the Notifications folder for any errors that might appear in the error log. Use the -Verbose and -WriteToEventLog command line parameter switches for additional Ops Manager event logging detail.

If needed, you can manually run the script. You must first set the required variables near the top of the script.

An additional method to gain visibility into any errors that might occur (before the script has a chance to write error output to the event log) is to append the line shown directly below at the end of the command line parameters:

This will output any errors that occur when the workflow attempts to launch the command channel.

Also, I’ve seen some weird things happen when copying/pasting from online sources into Powershell scripts. Make sure you are not copying some weirdly encoded characters. I’m pretty sure this blog doesn’t use any strange encoding but to keep your sanity, copy the code portions into Notepad++ and force the encoding to be UTF-8, then copy from there into the Console wizard.

*Please let me know if you have any suggestions to improve this article.

2014/10/17: Updated troubleshooting section. Also updated the script to add the Subscribers to the CF4 field on “New” alerts. Previously subscribers would be empty for new alerts.

2014/10/27: Updated troubleshooting section.

2014/10/29: Updated some sloppy grammar.

Disclaimer: Sample scripts in this guide are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

Join the conversation

Question for you – how does SCOM handle concurrency with this notification channel? In a hypothetical situation with 10 alerts at once, would we end up with 10 PowerShell.exe processes, or do you use any special threading or runspaces if you see powershell.exe as the command file?

Each instance of the command channel is separate. In the comment/instructions are at the top of the script there is information on how to increase the "asynchronous" process limit. I usually set mine to 20 on mgmt servers but this would depend on your server specs.

The commented line should/could probably be uncommented. The “If” block below it basically will determine if the “Path” of the alerting instance includes the FQDN of a server, if yes, will include the PathName in the subject. The commented line could be uncommented to act as a default in case the path does not include a FQDN. Both assignment statements include the AlertName although at the tail end of the subject. If you wanted the AlertName to be near the beginning of the subject so as to be more easily readable in a cramped email window/screen see the revised code block below.

I can't seem to get this script to work. If I put the external email address in as the one to use for error reporting, that message goes through. However, if I try to use this as an additional notification channel for an existing subscription, it doesn't work.

You haven't given me a ton of detail about your current configuration or what the error email contained so it's difficult to diagnose from here. Please have a look at the Troubleshooting section of the post. I just added

a few more suggestions.

If you still can't get it working, send me your script with your customized

settings (minus any email account password) and I'll look it over to see if I can spot anything that might need tweaking.

Thanks for the script, working very well! However I am having a weird issue with the emails being sent. It seems that the alert emails ALWAYS send a second alert, exactly the same as the first, about a minute or so after the first has been received. Since we are sending this to our servicedesk program this creates a double of every job logged by SCOM. Any idea why this might be happening?

My guess is that the you need to modify your subscription criteria to only send alerts for "New" and/or "Closed" alerts. When the script runs it changes the alert Resolution State to "Notified" (ID value: 2), which updates the alert. When alerts get updated in any way, they are flagged to be processed (again) by the notifications/subscriptions workflows. If the subscription criteria is already correct, then make sure that you actually created the Alert Resolution State "Notified" , ID: 2, in the Administration workspace , Settings->Alerts window in the Console. You should be seeing the Resolution State change to "Notified" in the Alert view.

I have set the subscriptions to only alert on new or closed alerts, not updated ones. And I have also created the ID:2 n(otified) as a customer resolution state in SCOM. However, the way I am using this is a little different. When a "new" alert comes in, the script does not kick off, we do not want to send EVERY alert that comes in, to servicedesk. Instead, I have set the command channel to run, when it notices a new job with the resolution state of 2 (notified) this way, I can keep scom open, right click and set the resolution state of the jobs that need to go through to the servicedesk manually. After I do this manually, it runs the command channel, runs the script, which AGAIN updates the alert status to 2 (notified) and the script runs again, does this sound right?

I may need to change the settings a little to fit with what I am trying to achieve. However I am unsure of how to do this.

You stated that your subscription is configured to notify on New or Closed but not updated. If the current ResolutionState of an alert is New or Closed and you update it in some way (other than changing the ResolutionState to something other than New/Closed), the alert will be flagged again internally for a subscription criteria check. If any subscription criteria matches, the corresponding channel will get kicked off, whatever that channel is (SMTP, Command, etc.). So I don't understand your first statement.

I think I understand what you are trying to accomplish. Let me state what I suspect you are after:

– When a New alert is created, do nothing.

– When the ResolutionState is manually changed to whatever the ID is for "pending Service Desk investigation" (ServiceDesk, PendingSvcDesk, etc.) trigger command channel as usual. (Alert is touched/updated internally, subscription check is triggered, and an Email gets sent)

Create an alert ID specifically for "pending Service Desk". Name it something like, "ServiceDesk" and give it an ID of say… 3.

Configure your subscription to trigger on these ResolutionStates ONLY: ServiceDesk, and Closed.

The script will only change ResolutionState from New to Notified IF the ResolutionState is currently New. In the case described above, the script will not trigger from New alerts. Once you change it manually from New to ServiceDesk, it will stay set to ServiceDesk until you change it again manually or until it auto-closes (if source=Monitor, not Rule). Does that make sense?

I look at alert notification for network and then some information is missin. For be able to see name of the device "Full pathname" need to be added, and port see port number "Description" in entinty properites. Description display portnumber like Ethernet1/31.

You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. The property 'CustomField10' cannot be found on this object. Verify that the property exists and can be set. The property 'CustomField9' cannot be found on this object. Verify that the property exists and can be set. The property 'CustomField8' cannot be found on this object. Verify that the property exists and can be set. The property 'CustomField7' cannot be found on this object. Verify that the property exists and can be set. The property 'CUstomField6' cannot be found on this object. Verify that the property exists and can be set. Cannot validate argument on parameter 'DisplayName'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again. Cannot bind argument to parameter 'Id' because it is null. The property 'CustomField3' cannot be found on this object. Verify that the property exists and can be set. The property 'CustomField2' cannot be found on this object. Verify that the property exists and can be set. The property 'CustomField1' cannot be found on this object. Verify that the property exists and can be set. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression.

what was the solution, I am having the same problem? 2016-12-02_16:11:47.732 alert.NetBiosComputername:
alert.PrincipalName:

Class.name:

TempFilePath: C:\Windows\TEMP\tmpC300.tmp Cannot bind argument to parameter ‘ResolutionStateCode’ because it is null. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. The property ‘CustomField10’ cannot be found on this object. Verify that the property exists and can be set. You cannot call a method on a null-valued expression. Cannot validate argument on parameter ‘Class’. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again. The property ‘CustomField9’ cannot be found on this object. Verify that the property exists and can be set. The property ‘CustomField8’ cannot be found on this object. Verify that the property exists and can be set. The property ‘CustomField7’ cannot be found on this object. Verify that the property exists and can be set. The property ‘CUstomField6’ cannot be found on this object. Verify that the property exists and can be set. Cannot validate argument on parameter ‘DisplayName’. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again. Cannot bind argument to parameter ‘Id’ because it is null. The property ‘CustomField3’ cannot be found on this object. Verify that the property exists and can be set. The property ‘CustomField2’ cannot be found on this object. Verify that the property exists and can be set. The property ‘CustomField1’ cannot be found on this object. Verify that the property exists and can be set. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. The user MDYNYCMAS\Sys_dev_scomaa does not have sufficient permission to perform the operation. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. The user MDYNYCMAS\Sys_dev_scomaa does not have sufficient permission to perform the operation.

Looks like your mgmt server action account account is having some permission issue accessing/updating the alert data. This might sound too simple but I’ve seen this before and a restart of your mgmt server (whichever one is the current notification server nominee) will probably solve it. If you don’t know which server is running the notification workflow, just reboot all of them, one at a time (those that are in the Notification pool).

This is an interesting problem. I can't imagine why that line in the Cleanup function is giving you grief. I'm curious which version of Powershell you have on your notification server. In the meantime, you could try to comment out the line# 988 where the Cleanup function is being called. Or comment out line 353 which is where the command is called to remove the session variables. Cleaning up the variables is not entirely necessary in Powershell but it's generally accepted as good coding etiquette and makes for a more consistent experience across a number of hosts.

@Matthew Raida: I see what you are trying to do here but this could be tricky because not every alert has a "path" ($alert.MonitoringObjectPath). I suppose you could simply TRY to get the path and, if it exists in the form of an FQDN, attempt to compare it to your custom objects and if there is a match then display the AD custom property/note of the matching custom object.

Excellent work on the script! It's great to have an option like this since it's not built in to SCOM (especially one that's been kept up to date with fixes and features added along the way).

That seems to be a commonality with System Center products. SCCM is the same way, except you have even less options as far as configurable email alerts go. And all these alerts from either system with little to no customization is in my opinion, pretty terrible.

I am getting the below errors, I am not able to find a solution for the below errors. Can anyone help me with the below errors with the above script.

2015-12-13_08:47:03.498 SUCCESS outlook.AFG.alfaraa.com "SCOM.Service" <SCOM.Service@alfaraa.com> C:ScriptsSCOMEnhancedEmailNotification.ps1 XXXXX alert.NetBiosComputername: , alert.PrincipalName: , Class.name: , TempFilePath: C:UsersScom.ServiceAppDataLocalTemp2tmpC260.tmp Cannot bind argument to parameter 'ResolutionStateCode' because it is null. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. Property 'CustomField10' cannot be found on this object; make sure it exists and is settable. You cannot call a method on a null-valued expression. Cannot validate argument on parameter 'Class'. The argument is null or empty. Supply an argument that is not null or empty and then try the command again. Property 'CustomField9' cannot be found on this object; make sure it exists and is settable. Property 'CustomField8' cannot be found on this object; make sure it exists and is settable. Property 'CustomField7' cannot be found on this object; make sure it exists and is settable. Property 'CUstomField6' cannot be found on this object; make sure it exists and is settable. Cannot validate argument on parameter 'DisplayName'. The argument is null or empty. Supply an argument that is not null or empty and then try the command again. Cannot bind argument to parameter 'Id' because it is null. Property 'CustomField3' cannot be found on this object; make sure it exists and is settable. Property 'CustomField2' cannot be found on this object; make sure it exists and is settable. Property 'CustomField1' cannot be found on this object; make sure it exists and is settable. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. Cannot bind parameter 'Id'. Cannot convert value "" to type "System.Guid". Error: "Unrecognized Guid format." You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. Cannot bind parameter 'Id'. Cannot convert value "" to type "System.Guid". Error: "Unrecognized Guid format."

I was able to get this working in our environment right away. Nicely tested and it’s rock solid. My question is if I wanted to call another script from this script where I would pass the AlertID to it as a parameter, and to pull information from the second script to include in the email, where in the logic should I do that? Ive been experimenting and I’m not sure I’m truly following the flow. Would you mind helping?
Thanks

@Sylvr317
You could do this anytime before building the html body. Then insert the data into the appropriate location within the body; I’m guessing that you would want to add a row to an existing table. However, it might make more sense to bake the code directly into the main script instead of calling a separate script. What info are you trying to add to the email?

The script just uses a bunch of high level descriptive stuff that you would want in the ticket. Jira is the ticketing tool and there are about 10 fields that we put information into. But most are hard coded. Right now I’m just wanting to grab the description and use that in the ticket summary. We are using REST logic to create the incident. It replies back with Incident ID, Key and URL of the ticket itself.
We want to put that URL into the alert email that goes out after the ticket is created. Right now we are focused on physical and VM servers for this subscription action, so we will be including the link to our Wiki page for whatever server or vm has generated the error.

Im still having issues, hope you can shed some light on this. I can run the PS script from the PS prompt (E:\SCOM_Scripts\Notifications\SCOMEnhancedEmailNotifications.ps1 -Description ‘MyDev’ -alertID ‘{xxx}’ -SubscriptionID ‘{xxx}’). I can not get it to run in a command prompt (C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe E:\SCOM_Scripts\Notifications\SCOMEnhancedEmailNotifications.ps1 -Description ‘MyDev’ -alertID ‘{xxx}’ -SubscriptionID ‘{xxx}’). This gives me is not recognized as the name of a cmdlet….. and it won’t run from scom channel. When running from SCOM there are no errors, or eventlog entries to help troubleshoot. SCOM 2012 r2 UR9

@Stevan:
Usually this means that your PowerShell Execution Policy is interfering. You can either
1) set your policy to Unrestricted
2)or set to Remotesigned, right click the ps1 file and select Unblock on Properties window.
Option 2 is probably the better approach.
Be careful when copying/pasting anything from this website because the characters get encoded and won’t work in your channel setup. Mostly the quotations cause problems. Very annoying, I know.

When testing in a Command prompt, start one piece at a time. Simply try to launch PowerShell, then exit it if successful. Then launch PowerShell and feed it the ps1 file only. Then move on to add more parameters/values until you have the full command line. See if you can isolate the piece that is faulty.

Hi, the script seems to have problems under SCOM2012r2 and no email is sent. I get a lot of errors when running from PS itself and when using the test the script fails on: Get-SCOMNotificationSubscription : Cannot validate argument on parameter ‘Id’. The argument is null or empty. Provide an argument that is not null or empty, and then
try the command again.
At C:\SCOM_Scripts\SCOMEnhancedEmailNotification.ps1:701 char:71
+ ForEach ($SubscriberName in ( (Get-SCOMNotificationSubscription -ID $Subscript …
+ ~~~~~~~~~~
+ CategoryInfo : InvalidData: (:) [Get-SCOMNotificationSubscription], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.SystemCenter.OperationsManagerV10.Commands.GetSCNotificationSubscriptionCmdlet

Please help me out, because looking at the email template we need something like it.

kind regards,
Jan

PS> I downloaded and installed the management pack but it seems there is only a xml file available.

Hi Jan,
The script most definitely works with SCOM2012 and later versions. I suspect that your command line is problematic in the Channel setup. Is it possible that you copy/pasted from the website/blog? Sometimes I see some strange problems when copying from web pages. It depends on the font/encoding. Sometimes the quotes (single and/or double) cause problems.
Here’s an example from my lab (also SCOM2012R2):
C:\SCOM_SCRIPTS\Notifications\SCOMEnhancedEmailNotification.ps1 -Description ‘MyDev’ -alertID ‘$Data/Context/DataItem/AlertId$’ -SubscriptionID ‘$MPElement$’

Try hand typing something similar (appropriate for your script Path and Description) into your Channel configuration and see if that helps.