Archive

I recently ran into a situation where a client has a group per server for Administrators, Remote Desktop Users, and hopefully, Service Accounts. This may or may not be the best way of dealing with this, but it does solve a need by moving user access to AD vs configuration on local servers. It’s a little easier to centralize and manage by administrators that may have access to AD but not the servers themselves (eg: HelpDesk users). The problem, as indicated below, is that setting the rights for the service account/groups has been getting done manually to the systems as they are built or needed. This has resulted in inconsistencies, as one might expect. So I found a way to standardize and bring it all “back up to code”, as it were.

PROBLEM:

You have a need to set a user or group to have “Log on as a Service” or “Log on as a Batch Job” rights. This can be done via the Local Security Policy (secpol.msc) or via GPO. However, there are two obvious issues with this:

1) Using SECPOL.MSC means you’re editing the local security policy. While this may be the only way to accomplish this, it is decentralized and uncertain to maintain.

2) Using the GPO method only allows you to set a particular set of user(s) or group(s) to the affected machines

However, if you have a need to set a 1:1 relationship with a dynamic name to the system, GPO’s and the Local Security Policy leave something to be desired. There is no functionality within the GPO to say “Apply GRP-%SERVERNAME%-SVC” to have this rights, and have it apply as needed – at least for the Logon As a Service right. Using other methods you can allocate to existing groups with existing rights, but you cannot either dynamically specify a group in THIS GPO location, affect the Local Security Policy, or set the rights for this local group.

REQUIREMENT:

Have each server/system have a group such as GRP-SERVER01-SVC group identifying service accounts. This would be a company policy scenario, and would ensure that administration and auditing of local group memberships was ONLY done via Active Directory, and could be done via delegated rights by users who may not have rights to login to the server.

Have the group apply only to the named server. Eg: GRP-SERVER01-SVC should have rights on SERVER01, but not SERVER02 or SERVER03

If possible, one should also be able to add to the local group a GRP-ALLSERVERS-SVC for a service account that might be globally allowed. Eg: DOMAIN\svcAutomation, DOMAIN\svcBackup, etc.

Centrally manageable

Automatic, dynamic, updates and standardizes over time.

OPTIONAL – also do similar for the pre-existing local groups of “Administrators” and “Remote Desktop Users” for a corresponding GRP-%COMPUTERNAME%-ADM and GRP-%COMPUTERNAME%-RDP as appropriate.

This MUST be run with the –u / -p switch to specify the user to use with the –h “highest privileges”. The –C must also be used to copy the batch file to the local system so it can run.

You will see entries in the log similar to:

Granting SeServiceLogonRight to Service Accounts on \\NW-ADCS1... successful
Granting SeServiceLogonRight to Service Accounts on \\NW-DC1... successful
Granting SeServiceLogonRight to Service Accounts on \\NW-DC2... successful

5) We now have a local group called “Service Accounts” and this local group has the rights “Logon as a Service”.

We can verify this by running “SECPOL.MSC” on one of the servers and checking the rights assignments:

Sure enough, the local “Service Accounts” group is listed.

6) We can now handle the remainder of this via normal GPO’s for Restricted Groups, using DYNAMIC naming.

Open the GPO editor and create a new GPO and name it something obvious such as “LOCAL_RESTRICTED_GROUPS”, and then edit it.

We will choose UPDATE for an action, as the group should already exist based on our previous work.

The group name will be “SERVICE ACCOUNTS”.

Click ADD to add members

This is where the magic comes in. If you press the “…” beside the NAME, you can search for the group/user based on a traditional ADUC type search. But we don’t want that. Instead, place your cursor in the NAME field. Press the F3 key:

We get a list of VARIABLES! We want to use ComputerName so that we can reference the group as GRP-%COMPUTERNAME%-SVC and each computer will get its own group. Click SELECT.

Note the variable shows %ComputerName% as expected. Modify that as needed to have the GRP- and -SVC prefix and suffix.

Click OK to close this window.

I’ve chosen to also add an -ADM and –RDP group for Administrators and Remote Desktop Users as this is another use case.

Close and save the GPO

9) Link your GPO appropriately:

Here I have a GROUPS-TEST OU and I have placed my NW-VEEAM01 server in this OU, along with the 3 associated groups. This will limit impact during testing.

10) On the system in question, check the current group memberships:

11) On the system in question, run a “gpupdate /force”

12) Again on the system in question, confirm the updated group membership:

There you have it. The ADM/RDP groups were easy as they not only pre-exist, but are pre-defined. The complication really was the “Service Accounts” group, which both does not pre-exist, and has no special rights by default or built in direct way of adding them via the command line.

The recommendation would be to run the SET_LOGONASSERVICE.BAT as part of the server build process/scripts, or have it pre-done in your deployment image/WIM/VM Template. Equally, a PSEXEC run against all servers in the domain could force set this group on a periodic basis to ensure the rights existed. Additional error checking could be built in to check if the command was successful, check if the domain group exists, create it if required, etc.

Some post comments:

Remember that the local account has a SID. If it is deleted, and recreated with the same name, that won’t be enough as the Log on as a Service right will be assigned to the old SID

As the batch file creates the account with a description and we didn’t tell the GPO to do so, it’ll create a new group if required, but with no description. This is your identifier that something is off, and hopefully that helps you troubleshoot.

Yesterday, I posted some details about MS14-068 and MS14-066 (https://vnetwise.wordpress.com/2014/11/19/cve-2014-6324-ms14-068-and-you/) and of course today, have had to do some investigating into a few sites that have a variety of patching systems. Some are using SCCM, some WSUS, some have policies and procedures, some don’t. But I noticed a potential ‘perfect storm’(?) of situations that could cause some of them grief – and it was more than just one.

Let me draw you a picture of what is a pretty common environment:

WSUS exists for updates, because that’s “the responsible thing to do”

WSUS was likely configured some time ago, and no one likes it because it’s not sexy or fancy, so it doesn’t get any love. Thus, it is probably running on Windows 2008 or 2008 R2.

Someone at some point *did* ensure that WSUS was upgraded or installed with WSUS 3.0 SP2

This all sounds pretty good, on the face of it. Now let’s introduce some real world into this environment….

Procedures state that you will install updates that are previous month or older – so you’re staying 30 days out, which is reasonable – let someone else go on Day0.

Those same procedures state that you will look at the list, and select the Critical and Security Updates from the last month, and approve them.

Nothing is stated for what to do about the current month’s patches – they are left as “unapproved” – but also not “declined”

Alright, so still pretty “common” and at face value, not that bad. A year or two goes by, and now you introduce Windows 2012 and Windows 2012 R2 to the mix. This itself is not a problem, but it’s where you start to see the cracks. Without even having to look at the environment, I know already the things I want to be looking for….

Because the current month’s updates are not being “Declined”, they’re showing up in the list as “missing”. If you have 10 updates, and 8 are approved and 2 are not, you will only ever possibly show 90% patched. The remaining two WSUS/WU knows are “available”, but “I don’t have them. You want to decline those so they only show up as 8 updates and 100% success. Otherwise, how do you know at a glance if the missing update is the approved one that SHOULD be there, or one from this month? Your reporting is bad. See: https://vnetwise.wordpress.com/2014/03/24/howto-tweaking-wsus-so-it-only-reports-on-updates-you-care-about/

Because the process counts on someone approving “last months” updates and not “all previous updates”, there’s almost certainly going to be some weird “gap” where there is a period of a few months that isn’t approved and isn’t installed for some reason. But the “assumption” is that they’re all healthy. Because the previous point doesn’t “decline” any updates, the reports for completion are untrustworthy – and/or never reviewed anyways.

Next, Windows 2012+ has been introduced. There’s a KB that is required to be installed on the WSUS server *and* rebuild of the WSUS package on the client to ensure compatibility. See MS KB2734608 (http://support.microsoft.com/kb/2734608). Because this is an “Update” and neither Critical nor Security, it is not applied to either the WSUS server or the clients.

In order for the Windows 2012/2012R2 WU/WSUS behavior to actually be changed, you need GPO’s that Windows 2012/2012R2 understands. In order for that to be true, you need 2012+ ADMX files in your GPO environment. Preferably in your GPO “Central Store” (again – https://vnetwise.wordpress.com/2014/03/20/howto-dealing-with-windows-2012-and-2012-r2-windows-update-behavior-and-the-3-day-delay/). But because Windows 2012 and 2012 R2 were likely “added to the domain” with no testing, studying, certification, or reading, this wasn’t done. Equally, even if it WAS done, most likely someone is still editing the GPO’s on the 2008/2008R2 based Domain Controller – which wipes out the ADMX based changes and replaces them with ADM files and the subset of options that they understand. You’ll never know this happened though, and even if you jump up and down and tell people not to do it, they will.

No one is ever doing a WSUS cleanup, so Expired, Superceded, etc updates are still present. Which isn’t helping anyone.

So to make that detail a little shorter:

Choosing Critical and Security Updates only is causing you to miss out on *required* updates. Stop being “fancy” – just select them all please.

Because you’re choosing “date ranges” of updates, you’re missing some from time to time. Stop being “fancy” – select “from TODAY-## to END”

If you introduce a new OS to your environment, you need to ensure your AD and GPO’s support them.

On top of the Updates and Update Rollups above that cause those issues, let’s take a quick look at some of the other things that are NOT considered Critical or Security Updates:

That’s just ONE Update Rollup. None of those look like ANYTHING I’d want to happen to my servers. </Sarcasm> So why WOULDN’T I want to install those? Yes, there may be features you’re not using. Perhaps you don’t use DeDuplication or DFS-R. Won’t it be fun later when you install those Roles/Features, and WSUS scans that server, and says “all good, nothing to update” for you? Tons of fun!

So, long story short – please stop being fancy. You’re introducing complexity and gaps into your environment, and actually making things harder. This means more work for you and your staff and co-workers. That likely don’t have enough time and resources as it is.

This is something I run into from time to time, and figured I’d make sure I wrote down in case anyone else is still seeing this.

On a fresh installation of Windows 2012 and 2012 R2, you’ll find that file downloads are disabled by default. This is not a debate about whether you should or should not be downloading files from the internet on a server OS, but instruction on how to change this behavior, should you wish.

The error you’ll see, typically if you’re going to download Google Chrome as a browser, looks like this:

For searchabilty reasons, the error is “Your current security settings do not allow this file to be downloaded”.

This has been something I’ve had to keep flipping back to my notes for, so I figured I’d jot it down. Various products, such as vCenter Server, require the installation of .NET 3.5 on Windows 2012/2012 R2. Doing so, however, is not as ‘simple’ as finding the download and installing it. This component is actually a Windows Feature, and must be installed via this method.

Above is a screenshot from the Add Windows Roles and Features wizard. Do NOT check the box for Application Server and click NEXT.

If you see the above screen, prompting for .NET 4.5, then in my first step above – you actually checked the APPLICATION SERVER role, which has .NET 4.5 as mandatory. As we don’t need it, you should go back and uncheck it.

Note that highlighted yellow bar. The reason you’re seeing this is that it’s informing you that the actual components for this are NOT cached on the local system. You need to click on the other circled link, to SPECIFY AN ALTERNATE SOURCE PATH

Here you can specify the path. In this case, I’ve double clicked and mounted the Windows 2012 R2 with Update ISO so it is mounted as the E: drive – thus, I can specify E:\SOURCES\SXS as the source folder. Click OK then click INSTALL on the previous screen when it returns.

Click FINISH. While it doesn’t force you to reboot, it might be wise to both check for any Windows updates related to .NET 3.5 and reboot as well.

Some additional comments:

Network UNC paths – I’ve had mixed success with unpacking the ISO to a network share and specifying the name. Usually I’ve been short on time, so couldn’t spend the time to troubleshoot. Suspected suspected culprits are:

DFS-N name spaces give it grief

Long (>64) character UNC paths cause issues

Spaces in the path can cause grief

The MACHINE account for the computer doing the installation doesn’t have rights to the share, even if the USER does.

Command line installation – you can perform all of the above with PowerShell using the command:

a) If you did not install .NET 3.5 as per the pre-requisites, you’ll see the following message:

Ensure you go and install .NET 3.5 via the Add-Features.

b) The first screen after the licence agreements, will ask for a licence key:

You’re able to skip this at this point, so click NEXT. We can add licences later if this PoC stays around.

c) Accept the licence agreements, and the default Destination Folder.

d) On the Database Options screen, you’ll need to make some choices.

As this is both small scale, and a PoC, we’ll use an SQL 2008 Express instance. This might work just fine long term for a small environment with a few hosts. Click NEXT.

e) The vCenter Server Service will ask how it will run.

Most sites opt to use a named service account. If you uncheck the USE WINDOWS LOCAL SYSTEM ACCOUNT option, you’ll find that you cannot change the Account Name – this is why we always run the installations logged in as the user who will be the service account. Note, however, this service account doesn’t yet have any rights to SSO or vCenter Inventory, so you may have to go back a step to fix/configure that, if you want to go forward with a named service account. It should be fine to utilize the Local System Account. Click NEXT.

f) Next, we have to choose if we’re using Linked Mode or not.

A proper enterprise deployment would likely consider a Linked Mode deployment. However, there are simplicities that come from having a standalone environment. I’ll be using a standalone vCenter. Click NEXT.

We’ll be configuring and confirming all AD group authentication after the fact, so we’ll accept the default. Click NEXT.

i) On the vCenter Inventory Service Information screen, accept the defaults:

Click NEXT.

j) Choose your Destination Folder:

Click NEXT.

k) When the Ready to Install the Program screen appears:

Click INSTALL.

Click FINISH when it is complete.

6) Launch the VMware vSphere Client installer. As this is still vSphere v5.5, we’re going to require the C# client for things like VUM and SRM if we go that far. We’re all familiar with this install, it’s “Next, Next, Finish.”

7) Launch the vSphere Update Manager installer

a) Accept the licence agreements

b) On the Support Information screen, leave the default checkbox for DOWNLOAD UPDATES:

Click NEXT.

c) On the vCenter Server Information screen, remember that we have not yet configured SSO for AD and/or a service account. So you will need to configure this using the administrator@vsphere.local account for the moment. Enter the password for this account:

Click NEXT.

d) VUM requires a database of its own. The default will install a separate instance, which is fine for our needs:

You would almost certainly want to share a database instance with vCenter and/or other infrastructure. Note that VUM still requires a 32bit DSN, which has some hoops to jump through to be created on a 64bit OS. Click NEXT.

e) VUM wants to know how it will be identified on the network. From the drop down choose the FQDN and leave the firewall ports default.

Click NEXT

f) Choose the destination folders:

Click NEXT

g) VUM by default, suggests 120GB free on the VUM drive:

Keep this in mind, as you’ll likely be expanding the VUM disk at some point. Click OK.

h) Click INSTALL to install:

Click FINISH when it is complete.

The next step will be to configure the vCenter server. Steps such as creating a datacenter, a cluster, adding hosts, etc. Also any modifications required for AD roles and assignments.

WSUS is a great built in tool for working with Windows Updates, but sometimes it takes a bit of effort to find the best way to use that tool. Here are a few things to help make the system run smoother.

You must validate the patches in advance, including a DEV and TEST domain or environment.

There isn’t enough time from “Patch Tuesday” to deploy in DEV, test for a week or two, deploy in TEST, test for a week or two, then approve for Production – which might only be two weeks from Patch Tuesday

To accommodate the above schedule, you then install “Current Month -1” for all updates. Thus in March, you would deploy and approve Dec/Jan/Feb updates, but NOT Mar.

This allows you to install in DEV the week after Feb Patch Tuesday. You can then install in TEST two weeks later, or about the beginning of Mar. TEST can then be run for 2-4 weeks depending on Quarterly Outage window, to validate and be certain of updates in Production.

It is acceptable for TEST and PRODUCTION to be out of sync for this period. There needs to be a balance between TEST/PRODUCTION being identical and being able to pre-validate updates.

1) Approving Updates

In the WSUS console, click on SERVER -> UPDATES -> ALL UPDATES, and then click in the main window.

Right click on the headers and choose to show “RELEASE DATE”. Sort by RELEASE DATE.

NOTE: In my example here I’m showing “APPROVAL=DECLINED”. You would be choosing “APPROVAL=UNAPPROVED” but currently I have none to use as an example.

Sort by the RELEASE DATE column. Remember that as we are in MAR of 2014, we do NOT want to select any ##/03/2014’s as they are “too new”. Select ALL PREVIOUS updates from “Month -1” or 02/2014 in this case. Right click and choose APPROVE.

You want to click on the parent level and choose APPROVED (which has already been done here, as indicated by the GREEN CHECK that is shaed out). Repeat this but then choose APPLY TO CHILDREN – if this is appropriate for all of your WSUS Groups. In this environment, WSUS is only used for Windows Server OS groups, and they’re grouped by “Automatic”, “Manual”, and “Primary/Secondary” groupings. As such, they all GET the same updates, it’s just to have different schedules and methods for installation. Click OK. A new dialog will pop up as it sets each update to APPROVED and will take some time to complete.

Until you perform this step, you will see the updates in reports showing computers that require the update, but they’re not allowed to install it. Thus, even if you go and perform a manual Windows Update check, you’ll never see the updates to be able to select them. A sample update report would look like:

The APPROVAL column for the update(s) would say “Not approved”. The STATUS column will know if the system has already downloaded and staged the update.

2) Declining Updates.

The above all seems well enough, except for the non-obvious results. For this month you’ll have Mar/2014 updates not approved and as the months go buy you’ll have downloaded the updates for Apr/May/Jun. Your reports are now going to show that your systems aren’t 100% compliant, even if you install all current updates available. You’ll spin your wheels trying to figure out why WSUS says you have 2 updates outstanding, but the Windows Update client says “no updates found”. This is because WSUS knows about the updates, and will indicate they’re available but not approved. So your system DOES require them, but you haven’t let them off the leash yet. So the report is in fact, valid. But what it’s really showing you is “next time you do updates, you’ll need to install these updates”. That’s great for the week AFTER quarterly outages, but it does nothing to help you DURING or just after the outage to measure success.

To fix this issue, what you want to do is DECLINE the updates.

Change the APPROVAL drop down to show “ANY EXCEPT DECLINED”, which will not show all previously declined updates. Sort by the RELEASE DATE column. Remember that as we are in MAR of 2014, we DO want to select ONLY ##/03/2014’s as they are “too new” to be Approved Select ALL updates from the last Approved date or 03/2014 in this case. Right click and choose APPROVE. (this is counter-intuitive)

Choose “NOT APPROVED” (still not intuitive – you’re going to want to try looking for a “DECLINE” option, and it’s not an option – you need “NOT APPROVED”) from the top level drop down. Then click again and choose APPLY TO CHILDREN. Then click OK.

Now when you pull reports on your system, you’ll actually see 100%:

You now want to keep performing updates on your servers until everything shows 100%. That will then be:

All KNOWN updates

Including APPROVED, which will actually allow installation of a KNOWN update

NOT including DECLINED, which will not show them as “needed” in your reports of % columns.

3) Each month between “now” and “Next Quarterly Update”

This will now make you fine for “Today” assuming today is “March 2014, after Patch Tuesday, but before April 2014, Patch Tuesday”. However, come April/May/June Patch Tuesday, new updates will get downloaded to the WSUS server. For your reports to remain accurate, you’ll need to come into WSUS and set all the new updates to “DECLINED”. Follow the same process you did in Step #2, only of course you’ll see more than just 03/2014 to select. Just select from the first date of ##/03/2014 and go to the bottom and repeat the DECLINE option.

4) NEXT Quarterly Update cycle:

Steps #1 and #2 above assume you have a net-new WSUS installation. If you’ve done this process before, then come Jun/2014 when you need to select Mar/Apr/May months for approval, you’re going to have all of Mar/Apr/May/June of 2014 set as “DECLINED”. You need to now set them to approved, as well as the now downloaded Apr/May.

Similar to Step #3, you’re now going to take all your Mar/Apr/May updates and set them to “APPROVED”. You’ll want to do this immediately following the May Patch Tuesday, as this will then let your reports be accurate to reflect the number of updates and systems required. You can now provide accurate details on how long and how many updates you will need to perform.

5) Just BEFORE NEXT Quarterly Update cycle:

Understandably, you’ll now show accurate reports for May 2014 and you’ll no longer show 100% up to date, as of course you are not. However, as soon as Jun 2014 rolls around, your numbers will be inflated again because of updates that are now known after June’s Patch Tuesday but are not approved. This will, as per Step 2, skew your numbers and prevent you from hitting 100% success in your maintenance window. So ensure you set then all June updates to “DECLINED”.

A general rule of thumb might be that following a Patch Tuesday you should:

Go in and APPROVE all previous month updates

Go in and DECLINE all current month updates

This would allow non-critical servers that are set to update automatically on some schedule, to keep up to date on a monthly basis vs waiting for quarterly. This provides two benefits:

1) You get the new updates tested (albeit in limited fashion) on existing servers up to 3 months prior to quarterly outages

2) There is far less load and number of systems to be manually or brute force updated during your maintenance window. Less load, means less IOPS on shared storage, which means updates perform quicker, which means you can do more/other maintenance in the same outage window.

WSUS is a great tool for automating and managing Windows Updates to various systems in a domain. However, it’s not really all that granular, which is a problem. While you could say “install all updates at 03:00 on Saturday”, you can’t say “and after rebooting, check again, because you’re still in the maintenance window”. You also can’t specify “do it RIGHT NOW, don’t wait for a random period” and there are some difficulties with “reboot when complete, don’t want 5-15 minutes, don’t wait 3 days, do it now”.

If you use these methods it might take you a bit of tweaking and fighting to make it work. Specifically if you’re having issues with Windows 2012/2012R2 systems, check: HOWTO: Dealing with Windows 2012 and 2012 R2 Windows Update Behavior and the 3 Day Delay.

This method can be pushed out to all systems via PSexec. Note though that there are some things to watch for:

· The GPO must be set to “4 – Download and Install Updates”. If it is set to “3 – Download and Notify” then all the “wuauclt /UpdateNow” in the world won’t make it do what it’s not allowed

· Except for maybe on Windows 2012/2012R2 systems, where it will think it’s in a maintenance window, and well, you said “UpdateNow”, so let’s do that.

· I’ve found it to be intermittent if the Day/Hour for the option to install in the GPO is not set near the time you’re pushing out. This doesn’t matter so much if you’re doing a scheduled restart such as “Sunday @ 03:00”. But if you have a manual maintenance window where you’re trying to brute force blast out and confirm all the updates that starts at Friday @ 20:00 – you should probably ensure that the GPO is set to the same, especially given that this batch file will refresh the GPupdate.

· As time goes on through that maintenance window, update the GPO time as well. They must go hand in hand.

What you’ll see is that it will schedule the installation for the next day. In the above example, C:\WINDOWS\WINDOWSUPDATE.LOG is showing that on 2014-03-20 at 2:20AM it says it is scheduling the installation to occur at March 21 2014 at 12:00AM. This is because the first line indicates the GPO setting is “Every day” @ “00:00”. So if anything, you’d like that to be “the next hour, of the same or following day”. Watch things like running Friday at 11:45PM and not changing your “Install Day” from Friday to Saturday to accommodate the 00:00 or 01:00 next time.

· There doesn’t seem to be any harm in pushing out the batch file to a system that’s already updating, other than it will restart the Windows Update service. Where possible though, you want to push it to systems that are not otherwise installing. I don’t yet have a method for knowing if a current update process is occurring. Perhaps if you took the “ping” process that is the timer, and made it a “start /wait” with a title, then looked to see if a process was running with that title, don’t run…. But this is as far as I’ve gotten for now.

· Periodically check the WSUS console for “Last Updated” and “Last Reported” to get an idea for what systems need checking. Also look at the % complete column to know which systems are done.