Pages

Dec 14, 2010

UPDATE:pete zerger was kind enough to point out that sometimes i don't make sense, and i missed a very obvious point in the documentation. since the post itself is still useful, i didn't just scrap it. :) instead, i added an addendum.

if you've been working with opalis long enough, you might will find that there are moments when hacks are required to get you from one point to the next point. i've been experimenting a lot with nested workflows. it's like evolving from inline scripting to scripting with functions and/or subs.

i discovered that when using trigger policy to run a nested workflow, a bizarre thing happens. even if the nested workflow executes with an error, the status returned by the calling trigger policy object is "success". it didn't make sense at first until i realized that by all accounts, the trigger policy did execute successfully.

well, there's a problem with this. if it comes back as success, even though something failed, the policy will continue on down the path unless you tell it otherwise. enough of that. let's talk specifics about my scenario.

the workflow i created was designed to do one thing: usher alerts from opsmgr into tickets in remedy. since remedy is divided into many different operating queues, i had to consider how to create tickets into the correct queues. i decided to try it based on computer group membership.

in order to get the group membership, i had to query the opsmgr database. i decided to push that into a nested workflow so that it could be reused in other workflows at some later point. the information retrieved from the nested workflow would be the basis of information fed to a text file. the master workflow could reference the text file to search through for cross checking names.

now what would happen if the database failed to query, and the associated text file never filled with any data? if you're cross checking the alerts against an empty text file, chances are you will never have a match and as such no tickets generated.

but if opalis is returning a success on the nested workflow, how do you know the query is failing? that seems simple. if the published data returned from the nested workflow is empty, then obviously the query failed. too bad the link operators don't have any filters for stuff like "is empty" or "is not empty".

all isn't lost though. to get the effect that we want, we simply have to know what to look for. going to the nested workflow, we can use the query database object status as our criteria to branch appropriately. if successful, the publish policy data object writes the expected server list. if it runs into a warning or failure, we publish static text to a different publish policy data object in the form of "FAILED".

back in the master workflow, we can now use the link operator to cull out anything that tries to come through with "FAILED". if it matches the include filter, the policy processing stops.

addendum:

keep in mind that link operators do not have an "AND" operation. instead the filters are evaluated as "OR" expressions. however, the include/exclude tabs are separate so mixing and matching is a possibility, assuming you have the right content coming through.

in the opalis client user guide, the trigger policy object section has a table that states this description for the child policy status: the status that was returned by the child policy. it's important to clarify that the default behavior of a link operator coming from the trigger policy object is to set the filter to look for anything coming from trigger policy itself to "success".

if you're looking for the status coming from the child policy, you should change the link operator filter to look for something like this:

Dec 8, 2010

here is an interesting and rather complex problem i experienced with opalis. honestly, this thing has probably been going on since i built the environment. however, i have only begun working with nested workflows which really helped flush it out.

summary

essentially, when a master workflow launches a sub workflow, the policy spins in the background and never does anything. another indication of the problem is that the summary of policies indicate a high number of running instances. lastly, the problem appears to be environmental since only one opalis instance has the problem.

here is a view of the operator console with this problem.

troubleshooting

the first thing i prefer to do in scenarios like these is to try to recreate the problem. that wasn't difficult on the server having the problem. the operator console shows a high number of instances, sub-workflows hang indefinitely, etc. when i tried it in a different environment, replicating the same master/sub nested test, it worked fine. i decided it must be my policy and exported the test version into my broken opalis instance, ran it, and got the same result -- it hung.

i decided to look down another path into the running instances count to see if that was part of the problem. i began by restarting the opalis service, hoping instances that were trapped somewhere would free themselves and disappear. not a chance! next, i started looking through the process list and found policymodule.exe which appears to be the running process that a policy resides in. i figured with that many policy instances running, surely, i was way over the concurrent policy instance maximum. unfortunately, i only found two of these processes which matched up to the two running policies as seen in the screenshot above.

since that looked fine, i went back to sifting the logs (yeah, i mistakenly dismissed it the first time around).i found this statement in the logs (full snip below)[3] which had me scratching my head:

2010-12-07 15:12:38 [1640] 1 Opalis Event: Frequent DB errors

that information wasn't very telling but eventually a little further down in the logs, i found this error:

apparently, i had the wrong set of permissions. according to the administrator guide, the action server requires "part of database users group on the datastore computer". going by this guidance, the action server service account was granted db_datareader and db_datawriter.

this made perfect sense. all of my other environments were ones i set up simply to test. with this broken one, it was set up as a dev environment utilizing best practices such as using databases from people who know how to run them (not me). adding to that, using the minimal level of rights required is followed. by the log output, you can see i missed one of those such rights, however. i did not grant the "execute" permissions that the account required.

this seems to speak to the root of the problem but in my research found that it did not correct the running instances count that appears in the operator console. looking through the database views, i found dbo.policies_view (which details a lot more than what you see in the operator console, by the way). i opened this view and found "runningpolicyinstances" column which contained the exact counts i saw in the operator console:

the design of the dbo.policies_view indicated very clearly how this column was being constructed.

(SELECTCOUNT(*) AS Expr1 FROM dbo.POLICYINSTANCES WHERE (dbo.POLICIES.UniqueID = PolicyID) AND (TimeEnded ISNULL))

so basically, if the timeended value in the dbo.policyinstances table is null, it counts them up and displays it here. sure enough, the table contained a high number of rows with null values. i searched through all of the stored procedures to try to locate an entry that indicated inserting values into policyinstances but in almost every case, data was drawn from the table and rarely written to. to add to this problem, log purging doesn't delete entries where the timeended value is null. i suspect this is because it believes the policies haven't finished executing yet. thus, these counts would have never gone away on their end.

resolution

:: database ::

the problem with the database permissions can be fixed in one of two ways:

grant the account db_owner rights

grant the account execute rights on all stored procedures [1]

once either one of these is done, the error in the opalisactionservice logs stop generating. i would be cautious with this. first, granting db_owner is probably granting rights beyond what the account actually needs. second, if you choose to just grant rights to the stored procedures, it may not be the entire set of rights required. my testing has been limited so far, and thus i may find in doing so that i run into other issues [2].

:: operator console ::

correcting the summary count will require some changes to the policyinstances table. obviously going in and messing around with tables is not going to be supported by microsoft so proceed at your own risk. my environment is a testing environment so it makes sense for me. you may want to call microsoft support.

to begin, i shutdown all running policies so that any timeended date stamps would write in as necessary (allowing me to avoid unnecessarily changing data that didn't need modification). afterwards, i ran the following sql query to set the timeended value to the current timestamp.

grantexecuteon sp_insertevent to [myServiceAccount];grantexecuteon sp_GetLogEntriesForDelete_FilterByEntries to [myServiceAccount];grantexecuteon sp_GetLogEntriesForDelete_FilterByDays to [myServiceAccount];grantexecuteon sp_GetLogEntriesForDelete_FilterByEntriesAndDays to [myServiceAccount];grantexecuteon sp_CustomLogCleanup to [myServiceAccount];grantexecuteon sp_PublishPolicy to [myServiceAccount];grantexecuteon sp_UnpublishPolicyRequest to [myServiceAccount];grantexecuteon sp_UnpublishPolicy to [myServiceAccount];grantexecuteon sp_DeleteTreeData to [myServiceAccount];grantexecuteon sp_FindTreeInsertionPoint to [myServiceAccount];grantexecuteon sp_InsertTreeData to [myServiceAccount];grantexecuteon sp_MoveTreeBranch to [myServiceAccount];grantexecuteon sp_StopAllRequestsForPolicy to [myServiceAccount];grantexecuteon sp_StopAllRequests to [myServiceAccount];

Dec 7, 2010

a friend of mine pointed out this really cool cmdlet called convertfrom-csv today. using it, you can immediately create a PSCustomObject. pretty cool! as a practical example, you can dump out repadmin and use the object to work with data any way you see fit.

Dec 6, 2010

don't be alarmed if you find that while testing nested workflows, the testing console generates an error when it hits the "trigger policy" object. this is "by design" as the testing console is only designed to test a single policy instance.

Dec 2, 2010

let's say that a friend breaks the operator console in an opalis lab that you helped set up. if you run into this problem, this is the way to unbreak it. so to begin with, no matter what you attempt to do, your login fails to work. in the console, you receive this message: The username or password you have entered is not correct. Transaction failed.

don't be fooled by these messages since they provide very little value with what the actual problem is. instead, go look at the server.log in the \jboss\server\default\log directory. in it, you may find information a little more valuable like this message:

that message makes a little more sense to me. I asked about any recent changes and was told that the port of the sql server changed. to correct the problem, I modified the opalis-ds.xml (\jboss\server\default\deploy) file. it was as simple as adding the port value to the instance name. before editing it looked like this:

Nov 2, 2010

a conversation with my buddies kwan thean keong and alexandre verkinderen got me started on looking into this particularly finicky integration pack. it seems that during the transition of opalis to microsoft, some things were lost in translation. this is an attempt to restore some of that but only so far as I've tested -- which admittedly, is not much.

requirements

first of all, let's talk about what you're going to need. one of the things lost apparently were some release notes that provided the much needed requirements that would have saved many an admin some hair and frustration. as stated in the provided help file for this intpack:

System requirements, installation, license, removal and known issues information is published in the Release Notes for this Integration Pack.

ah, but fortunately, the details were captured in this blog post. here is a synopsis.

failure to install any of the prerequisites I mentioned above will cause a giant banner to flash upon your screen displaying the word "FAIL!". okay, seriously, if you miss them, the test connection properties will fail. that should be your first indication that something is wrong.

here we are at a most pleasing connection dialog for the AD intpack. I'm sure by now you're familiar with these dialogs if you've dealt with opalis at all. one thing to consider here is that you do not need to supply a server name unless absolutely required. you can simply supply your domain name. any client that understands srv records will know how to find the closest domain controller.

test the connection when you're done. that should get you started down the right path.

a little test

what's that you say? you encountered a problem even though your test connection was successful? well, as it turns out, so did i. let's take a look at a very simplified policy I was running through.

the second object should require no explanation. the custom start is simply supplying a parameter to reset user password named "username". the password itself was hardcoded to "password". I ran it, providing my test account as a username and received the following problem:

it's very difficult to read the context of the error summary like this, so I pasted it to notepad and included some artistic breaks for clarity. this is what we're working with in actuality.

The server is unwilling to process the request. (Exception from HRESULT: 0x80072035)

Please make sure you have Powershell and Quest Active Directory command installed on the Opalis Action Server

allow me to make the assertion that most likely the last line is not the problem you're encountering if you're to this point (unless your management server and action server are separated. it's wholly possible then.) the first thing I did was ran the commands through a powershell console to validate the opalis environment wasn't the issue. it wasn't.

if you recall, I hardcoded the password to "password". if you're operating in an environment with password complexity engaged, the quest cmdlets executed in this manner will not allow for overriding. to verify this, I changed the password to "#99bottles of BEER!%" and ran it through a powershell console and again through the opalis test console. the results are as follows:

success! keep in mind if you're using an object like generate random text that you're setting it properly so that it generates the right level of complexity for your organization.

Oct 27, 2010

for anyone who has been creating custom DDRs, this is old hat. for me, I just wanted to prove that it could be done in powershell. apparently no one has tried -- or at least web searching has led me to believe it. :)

in sccm 2007, the command to send the DDR to the site server was removed in the sccm sdk redistributable dlls. this isn't a tragedy. it simply means you have to copy the DDR to the <site server>\sms_xyz\inboxes\ddm.box folder yourself. I didn't include that bit in the script since this is just for fun. anyway, once the DDR is processed, this is what you would see:

go discover the world -- or at least your macs and linux machines? :) and you know what I'm thinking? yeah, that's right. get this into opalis. that'll be my next post.

other useful information I learned

$SMSDisc = New-Object -ComObject SMSResGen.SMSResGen.1

to start off on this adventure, I began navigating the sample script provided for creating a DDR [1]. the first thing I did was try to execute the command above. it just spit out this stuff:

it took me longer than I will ever admit that I did not have the necessary sccm sdk. after installing it, I tried again and got the same result. it took me longer than the part I will never admit that I had to register the dll. so to reiterate -- download the sdk, then register the dll. after installing the sdk, register smsrsgenctl.dll. it's in this path: <Program Files>\Microsoft System Center Configuration Manager 2007 SDK\Redistributables.

the second hurdle is how do you know which com object to use during the creation of the new object? well, other than the fact that they spell it out for you in the sample script --

Set newDDR=CreateObject("SMSResGen.SMSResGen.1")

-- there really is nothing that stands out. powershell provides some conventions on getting this stuff out, but in actuality, it's nothing more than sifting through the registered classes in the registry. I found something on from tobias weltner [2] that made this way easy. it's a real gem you may want to hang on to:

and there we find the com object we were looking for. now honestly, I'd have been goofing around for awhile trying to figure out which in the list matched. it was helpful though for knowing that I had the dll registered properly!

moving on, now that you have the object, you can bring out the methods.

Oct 16, 2010

The Atlanta Southeast Management User Group and System Center Virtual User Group invites you to attend the next SMUG meeting scheduled for October 11th, 2010 for a day of great presentations, discussions, and networking.

Because this is a hybrid user group meeting there are two ways to register:

DATE & TIME

“The nworks Management Pack provides continuous monitoring of the largest, most demanding virtual environments. It features a centrally managed, distributed architecture for horizontal "no limits" scalability and automatic failover and load balancing for high availability. Optimized, user-configurable data publication methods and use of consecutive sample monitors, optimized performance providers and other advanced features of System Center deliver maximum information with minimal overhead.”

PRESENTER BIOGRAPHIES

Aaron Nelson

Aaron Nelson is a Senior SQL Server Architect with over 10 years experience in architecture, business intelligence, development, and performance tuning of SQL Server. He has experience managing enterprise-wide data needs in both transactional and data warehouse environments. Aaron holds certifications for MCITP: Business Intelligence Developer, Database Administrator, Database Developer; as well as MCTS: Windows Server Virtualization, Configuration (meaning Hyper-V).

Scott Moss

Scott has been working in the IT industry for more than 14 years. The majority of his time served in IT has been at various Telecommunications institutions, as well as a 3 year work release program in the Lottery industry. The past 5 years he has been focused on Systems Monitoring using MOM 2005 and Operations Manager 2007. Scott was also awarded the Microsoft's MVP Award 2010 for Operations Manager. For the past two years he has been vice president of the Atlanta Southeast Management User Group and President of the System Center Virtual User Group. He is also a SystemCenterCentral.com blogger and forums contributor.

Marcus Oh

Marcus is a Lead Systems Administrator for a large telecommunications provider, running directory services and management infrastructure for ~30,000 systems. He has been a MVP for the last six years in System Center specializing in Configuration Manager and Operations Manager. Marcus has written numerous articles for technology websites as well as his own blog. He co-authored Professional SMS 2003, MOM 2005, and WSUS.

Denzil Ramsey

Denny is a Datacenter Technology Specialist focused on Microsoft Datacenter Virtualization and Management solutions. Denny has been with Microsoft for 5 years in several roles including Networking Technology Specialist, Exchange Premier Field Engineer and Technical Account Manager working with customers like Home Depot and Coca-Cola. Before joining Microsoft, Denny spent 7 years a Cox Communications where he managed the Windows Core Infrastructure team supporting Exchange, Active Directory, Systems Management and Windows Server infrastructure.

Brian Pavnick

Brian Pavnick is a Solutions Architect at Veeam Software who specializes in integrating Microsoft System Center technologies with VMWare's VI. Prior to Veeam, he has worked over 10 years as a Sr. Systems Administrator acquiring skills in operating, implementing, and project managing Microsoft Server Infrastructure Technologies. Throughout his career, he has specialized in Systems Management for Microsoft Server technologies. This includes OS deployment, patch management, system profiling, system and application monitoring, data security, and disaster recovery. Brian is a Microsoft Certified IT Professional in Enterprise Administration.

Brian Huneycutt

Brian is currently a developer (more a jack-of-all-trades really) on the Configuration Manager Sustained Engineering team. He started as a Support Engineer handling consumer desktop support issues at Microsoft in 1999. After having his fill of "No, there is no double right click, just a double left click" and "I'm sorry sir, I cannot help you with your ISP's login password issue" he escaped transitioned to the SMS team, where he quickly learned the value of the SMS logs. Once firmly entrenched in the SMS world, he worked through all levels of support, ending up as an Escalation Engineer ("What do you want to debug today?") prior to moving to the product team. These days he still partners with CSS (current acronym for Support) on problem investigations, along with development work on hotfixes, service packs, and generally anything that needs either fixing or a very long email. Those fleeting moments of free time are split between family, tooling around town in a little Triumph Spitfire, or making sawdust in the garage / workshop.

Ted Sendler

Ted is a Support Escalation Engineer on the System Center Support Team at Microsoft. He is primarily focused on issues dealing with MOM 2005, Operations Manager 2007, Service Manager 2010, System Center Essentials 2007 and 2010 , and Opalis.

Wally Mead

Wally has been with Microsoft for 17+ years. He started in the training group and helped develop the training course for the original release of SMS 1.0. He has then been involved with all versions of SMS, and now Configuration Manager, from developing and delivering training, or assisting customers on newsgroups and forums, or directly with them through the TAP program. He now is a Senior Program Manager in the Configuration Manager Product Group, responsible for community efforts, such as forums, managing the MVPs, presenting at conferences, and working with TAP customers.

- Unable to connect to the remote machine with the OpalisRemotingService - Cannot connect to the Escorter service on the remote machine - Cannot instantiate "RemoteEscorter" - _com_error "Class not registered" "" "-2147221164"

The user account specified for the Action Server does not have the "Log on as a service" user right granted. Please specify a user account with the proper rights or add the right to the account that was specified.

Oct 1, 2010

I really do love october. for two reasons: it really marks the autumn season which is my favorite time of year, and it's my renewal time with the microsoft mvp program.

anyway, I got the email today. I'm in the program for another year! (by the way, it's a fantastic program!)

Congratulations! We are pleased to present you with the 2010 Microsoft® MVP Award! This award is given to exceptional technical community leaders who actively share their high quality, real world expertise with others. We appreciate your outstanding contributions in System Center Operations Manager technical communities during the past year.

congratulations to all my fellow MVPs who were awarded this month as well.

Sep 30, 2010

The Atlanta Southeast Management User Group and System Center Virtual User Group invites you to attend the next SMUG meeting scheduled for October 11th, 2010 for a day of great presentations, discussions, and networking.

Because this is a hybrid user group meeting there are two ways to register:

DATE & TIME

“The nworks Management Pack provides continuous monitoring of the largest, most demanding virtual environments. It features a centrally managed, distributed architecture for horizontal "no limits" scalability and automatic failover and load balancing for high availability. Optimized, user-configurable data publication methods and use of consecutive sample monitors, optimized performance providers and other advanced features of System Center deliver maximum information with minimal overhead.”

THE AGENDA

Presenter

Agenda

Start

End

ATLSMUG

Opening and Introductions

9:50 AM

10:00 AM

Aaron Nelson

Uncovering Performance Gremlins in SQL Server

10:05 AM

11:00 AM

Scott Moss

Top 10 SCOM Hotfixes

11:05 AM

11:30 AM

Marcus Oh

Orchestrating Maintenance Mode with Opalis

11:30 AM

12:00 PM

Brian Pavnick

Connecting the Dots with nWorks MP for VMware

12:15 PM

1:00 PM

Denzil Ramsey

SCVMM Self Service Portal / vNext

1:05 PM

2:00 PM

Brian Huneycutt

Top 10 SCCM Hotfixes

2:05 PM

2:30 PM

Ted Sendler

OpsMgr R2 – Common Issues and Troubleshooting

2:35 PM

3:00 PM

Wally Mead

SCCM v.Next

3:00 PM

3:55 PM

PRESENTER BIOGRAPHIES

Aaron Nelson

Aaron Nelson is a Senior SQL Server Architect with over 10 years experience in architecture, business intelligence, development, and performance tuning of SQL Server. He has experience managing enterprise-wide data needs in both transactional and data warehouse environments. Aaron holds certifications for MCITP: Business Intelligence Developer, Database Administrator, Database Developer; as well as MCTS: Windows Server Virtualization, Configuration (meaning Hyper-V).

Scott Moss

Scott has been working in the IT industry for more than 14 years. The majority of his time served in IT has been at various Telecommunications institutions, as well as a 3 year work release program in the Lottery industry. The past 5 years he has been focused on Systems Monitoring using MOM 2005 and Operations Manager 2007. Scott was also awarded the Microsoft's MVP Award 2010 for Operations Manager. For the past two years he has been vice president of the Atlanta Southeast Management User Group and President of the System Center Virtual User Group. He is also a SystemCenterCentral.com blogger and forums contributor.

Marcus Oh

Marcus is a Lead Systems Administrator for a large telecommunications provider, running directory services and management infrastructure for ~30,000 systems. He has been a MVP for the last six years in System Center specializing in Configuration Manager and Operations Manager. Marcus has written numerous articles for technology websites as well as his own blog. He co-authored Professional SMS 2003, MOM 2005, and WSUS.

Denzil Ramsey

Denny is a Datacenter Technology Specialist focused on Microsoft Datacenter Virtualization and Management solutions. Denny has been with Microsoft for 5 years in several roles including Networking Technology Specialist, Exchange Premier Field Engineer and Technical Account Manager working with customers like Home Depot and Coca-Cola. Before joining Microsoft, Denny spent 7 years a Cox Communications where he managed the Windows Core Infrastructure team supporting Exchange, Active Directory, Systems Management and Windows Server infrastructure.

Brian Pavnick

Brian Pavnick is a Solutions Architect at Veeam Software who specializes in integrating Microsoft System Center technologies with VMWare's VI. Prior to Veeam, he has worked over 10 years as a Sr. Systems Administrator acquiring skills in operating, implementing, and project managing Microsoft Server Infrastructure Technologies. Throughout his career, he has specialized in Systems Management for Microsoft Server technologies. This includes OS deployment, patch management, system profiling, system and application monitoring, data security, and disaster recovery. Brian is a Microsoft Certified IT Professional in Enterprise Administration.

Brian Huneycutt

Brian is currently a developer (more a jack-of-all-trades really) on the Configuration Manager Sustained Engineering team. He started as a Support Engineer handling consumer desktop support issues at Microsoft in 1999. After having his fill of "No, there is no double right click, just a double left click" and "I'm sorry sir, I cannot help you with your ISP's login password issue" he escaped transitioned to the SMS team, where he quickly learned the value of the SMS logs. Once firmly entrenched in the SMS world, he worked through all levels of support, ending up as an Escalation Engineer ("What do you want to debug today?") prior to moving to the product team. These days he still partners with CSS (current acronym for Support) on problem investigations, along with development work on hotfixes, service packs, and generally anything that needs either fixing or a very long email. Those fleeting moments of free time are split between family, tooling around town in a little Triumph Spitfire, or making sawdust in the garage / workshop.

Ted Sendler

Ted is a Support Escalation Engineer on the System Center Support Team at Microsoft. He is primarily focused on issues dealing with MOM 2005, Operations Manager 2007, Service Manager 2010, System Center Essentials 2007 and 2010 , and Opalis.

Wally Mead

Wally has been with Microsoft for 17+ years. He started in the training group and helped develop the training course for the original release of SMS 1.0. He has then been involved with all versions of SMS, and now Configuration Manager, from developing and delivering training, or assisting customers on newsgroups and forums, or directly with them through the TAP program. He now is a Senior Program Manager in the Configuration Manager Product Group, responsible for community efforts, such as forums, managing the MVPs, presenting at conferences, and working with TAP customers.