An Infrastructure Geek Floating in a Sea of UberCodershttps://blogs.technet.microsoft.com/lrobins
Various bits that float through my skull, usually related to securityMon, 24 Aug 2015 23:20:01 +0000en-UShourly1Best Practices for Securing Active Directoryhttps://blogs.technet.microsoft.com/lrobins/2013/08/16/best-practices-for-securing-active-directory/
https://blogs.technet.microsoft.com/lrobins/2013/08/16/best-practices-for-securing-active-directory/#commentsFri, 16 Aug 2013 08:29:58 +0000https://blogs.technet.microsoft.com/lrobins/2013/08/16/best-practices-for-securing-active-directory/Better Late Than Never...

Hello again, world's most sporadic blogger here. A while back, I posted here recommending that people who are interested in admin-free Active Directory stay tuned to this site. The reason for that post was that I'd just learned that we were going to write and publish a document that would include some of the information I'd originally intended to publish here. We published Best Practices for Securing Active Directory (BPSAD) in April of 2013, and if you haven't seen it already, please take a look. It's a long document, but we hope that the content is useful.

More to Follow...

Now that the BPSAD has been published and readers have had some time to digest it, I'm planning to produce a post now and then discussing some of the things that are in the document, some of the things that aren't in the document, and our reasons for each. I'd also like to provide related information such as sample approaches to some of the things we recommend in the document. If you're interested in any of those things, please check back here now and again. I can't promise that I'll be posting regularly or frequently, but I will be posting.

Thanks for reading, and I'll be back soon...

-Laura

]]>https://blogs.technet.microsoft.com/lrobins/2013/08/16/best-practices-for-securing-active-directory/feed/1I Have Not Fallen Off the Face of the Earth. Not Yet, Anyway.https://blogs.technet.microsoft.com/lrobins/2012/07/17/i-have-not-fallen-off-the-face-of-the-earth-not-yet-anyway/
https://blogs.technet.microsoft.com/lrobins/2012/07/17/i-have-not-fallen-off-the-face-of-the-earth-not-yet-anyway/#commentsTue, 17 Jul 2012 18:13:00 +0000https://blogs.technet.microsoft.com/lrobins/2012/07/17/i-have-not-fallen-off-the-face-of-the-earth-not-yet-anyway/Me, and My...Blog

As I've noted previously, I'm a terrible, terrible blogger. It's not that I don't have content for this blog, mind you. It's that I have a whole pile of it backed up, waiting to be polished and published. However, before I can post any of it, I have to do things like build virtual machines that I can use for demonstration purposes, take screenshots, polish text, dig up informational links, make sure that the things I suggest are properly vetted, do everything with appropriate (and usually multiple) versions of our operating systems and applications, etc., etc., ad nauseam. All of those things take a lot more time than I ever seem to have. However, I've now publicly mentioned this blog on multiple occasions, and I did so because it would force me to do something about the sad state of this little repository of mine.

A Little About Microsoft (Really Little), and a Little About One Small Piece of My Work this Year

Here at Microsoft, our fiscal year runs from July through June. At the beginning of each fiscal year, one of the things that we all do is to create our commitments. Commitments are descriptions of what we plan to do in a given year, how we plan to do it, and how we will define success of the endeavor. While we can modify our commitments as necessary throughout the year, I generally strive to establish my commitments with enough planning and foresight so that they're largely unchanged at the end of the year and all I have to do is to follow the roadmap that I set out at the beginning of the year (with a lot of organizational and managerial guidance, of course). One of the great things about creating effective commitments is that they allow us to not only state what we plan to do over the course of the year, but they also allow us to provide justification for allocation of appropriate time, resources, etc. that we need in order to complete the commitments.

Why am I telling you all this? Well, it's definitely not to divulge any secrets about Microsoft- this is not only public knowledge, we've actually published white papers about it. No secret stuff. I'm telling you all this because one of my commitments for this year is that I'm going to work to get some of that previously-mentioned "pile" polished and published. (How's that for alliteration? Did I ever mention my short attention span? Moving on...) Getting these posts published was actually part of another commitment last year, but I had defined different options for how to fulfill the commitment and this blog was one of them. I ended up delivering on the commitment via other mechanisms I'd listed as options, but this year, the commitment is actually going to specifically be about this blog, right here, with its five (about to be six) published posts in almost as many years. By publishing a post saying that I'm going to do this, I give myself some pretty strong incentive to finish what I started.

Getting to the Point

So if the "admin-free" posts have piqued your interest, only to disappoint you when their companions failed to appear, please stay tuned. Don't expect miracles- I'm simply not prolific enough to crank out a dozen posts in a month or two, but I do have some approaches in mind that should at least get me producing at a slightly faster pace than I have previously. And if you've never seen this blog and/or have no interest in the few posts that already exist in it, then I'm all the more impressed that you've read this far. Hopefully I can post something over the next year or so that DOES pique your interest.

I'm now receiving incoming Lync messages about a malfunctioning codec that is preventing me from completing some of the other work that's sitting on my plate right now, so I'll wrap this up. Watch this space, or maybe just come back in a year and read everything then.

Thanks for reading.

]]>https://blogs.technet.microsoft.com/lrobins/2012/07/17/i-have-not-fallen-off-the-face-of-the-earth-not-yet-anyway/feed/7"Admin Free" Active Directory and Windows, Part 2- Protected Accounts and Groups in Active Directoryhttps://blogs.technet.microsoft.com/lrobins/2011/06/23/admin-free-active-directory-and-windows-part-2-protected-accounts-and-groups-in-active-directory/
https://blogs.technet.microsoft.com/lrobins/2011/06/23/admin-free-active-directory-and-windows-part-2-protected-accounts-and-groups-in-active-directory/#commentsThu, 23 Jun 2011 13:19:00 +0000https://blogs.technet.microsoft.com/lrobins/2011/06/23/admin-free-active-directory-and-windows-part-2-protected-accounts-and-groups-in-active-directory/I am a terrible blogger when it comes to timeliness and consistency of post intervals. I admit it. All I can say is, it has been a busy summer. I actually have a half-dozen posts queued up for publication, but each needs to be scrubbed and fleshed out before I post them, so even though I may be slow to get them out, please know that there are definitely more in this series.

In the "Don't do as I do" category...

All screenshots and text examples provided in this post were captured on a Windows Server 2008 R2 virtual machine I built for testing purposes, and I did several things that you should never do (unless you're just trying to get a blog post finished and are going to shut down the virtual machine as soon as you're done). I logged onto the VM (a DC) as the built-in Administrator. I did the same thing with each of the test accounts. I used the same password for each of the accounts. In short, I did things that I would never do in a production environment, nor should you. Since I fully expect that the eagle-eyed among you will quickly notice that I've violated basic security principals in these screenshots, I thought I might as well 'fess up and get that out of the way so that we can focus on the important bits.

Without further ado, here goes...

Protected Accounts and Groups in Active Directory

Within Active Directory, a default set of highly-privileged accounts and groups are considered “protected” accounts and groups. With most objects in Active Directory, admins (or delegates) can change permissions on the objects, including changing permissions to allow themselves to change memberships of the groups, for example. However, with protected accounts and groups, the objects' permissions are set and enforced via an automatic process that ensures that the permissions on the objects remains consistent even if the objects are moved about the directory. Even if somebody manually changes the objects' permissions, this process ensures that they're returned to their "defaults" in short order.

AdminSDHolder and SDProp

In the System container of every Active Directory domain, an object called AdminSDHolder is automatically created (CN=AdminSDHolder,CN=System,DC=...). The purpose of the AdminSDHolder object is to provide "template" permissions for the protected accounts and groups in the domain. Every 60 minutes (by default), a process known as SDProp (Security Descriptor Propagator) runs on the domain controller that holds the domain’s PDC Emulator (PDCE) role. SDProp compares the permissions on the domain’s AdminSDHolder object with the permissions on the protected accounts and groups in the domain. If the permissions on any of the protected accounts and groups do not match the permissions on the AdminSDHolder object, the permissions on the protected accounts and groups are reset to match those of the domain’s AdminSDHolder object.

Additionally, permissions inheritance is disabled on protected groups/accounts, which means that even if the accounts and/or groups are moved to different locations in the directory, they do not inherit permissions from their new parent objects. Inheritance is also disabled on the AdminSDHolder object so that permission changes to the parent objects do not change the permissions of AdminSDHolder.

Changing the SDProp Interval

Except for purposes of testing, you shouldn't change the interval at which SDProp runs. However, if you're like me, you probably still want to know how to do it, "just in case". To change how often SDProp runs, on the PDCE in the domain, use regedit to navigate to HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters. Add a DWORD key called "AdminSDProtectFrequency" if the value does not already exist. The value of AdminSDProtectFrequency is set in seconds and valid values range from 60 to 7200 seconds (one minute to two hours). To return to the default 60-minute execution interval, just delete the key you created. And please, don't do this in production, because reducing the interval can increase LSASS processing overhead (read: spike your DC's utilization) and as far as I know, there's no reason to change the value in a production environment.

Manually running SDProp

In operating systems prior to Windows Server 2008, you can use the procedure found in KB article 251343 to force SDProp to run immediately. In Windows Server 2008 or 2008 R2, however, you can use LDP to execute the RunProtectAdminGroups task. To do this, launch ldp.exe, connect and then bind to your PDCE with appropriate credentials, and in the Browse menu, select Modify. In the resultant dialog box, type "RunProtectAdminGroupsTask" (without quotes) in the Attribute: field and "1" (without quotes) in the Values: field. Leave the Operation set to Add, click the Enter button to populate the Entry List field, and then click Run. (See screenshot below) This is useful if you are testing some of the concepts described in this post, so that you don't have to wait for SDProp to complete its scheduled run in order to see the results of your modifications. (And is yet another reason why you shouldn't need to change the SDProp interval.)

Default Protected Accounts and Groups in Active Directory by OS and SP Version

The following table lists the default protected accounts and groups in Active Directory by operating system version and service pack level:

Default Protected Accounts and Groups in Active Directory

Windows 2000 <SP4

Windows 2000 SP4 –Windows Server 2003 RTM

Windows Server 2003 SP1+

Windows Server 2008 – Windows Server 2008 R2

Administrators

Account Operators

Account Operators

Account Operators

Administrator

Administrator

Administrator

Administrators

Administrators

Administrators

Backup Operators

Backup Operators

Backup Operators

Cert Publishers

Domain Admins

Domain Admins

Domain Admins

Domain Admins

Domain Controllers

Domain Controllers

Domain Controllers

Enterprise Admins

Enterprise Admins

Enterprise Admins

Enterprise Admins

Krbtgt

Krbtgt

Krbtgt

Print Operators

Print Operators

Print Operators

Read-only Domain Controllers

Replicator

Replicator

Replicator

Schema Admins

Schema Admins

Schema Admins

Schema Admins

Server Operators

Server Operators

Server Operators

Note: DSHeuristics Attribute

An attribute on the AdminSDHolder object, DSHeuristics, allows limited customization (removal) of groups that are considered protected groups and are affected by AdminSDHolder and SDProp. This customization should be carefully considered if it is implemented, although there are valid circumstances in which modification of DSHeuristics on AdminSDHolder is useful. More information on modification of the DSHeuristics attribute on an AdminSDHolder object can be found in the Microsoft Support Article #817433.

The adminCount Attribute

When an account or group is flagged as a protected account/group, the value of the adminCount attribute on the object is set to a value of "1", which changes how security is applied to the account's object in AD. Objects whose adminCount attribute has a value of "1" will have their ACLs set to match those of the domain's AdminSDHolder attribute every time SDProp runs, regardless of whether the accounts are actually privileged accounts. Any account that has direct or transitive membership in any protected group (whether the membership is derived from security or distribution groups) is subject to this restricted security. If a user is a member of a group that is, in turn, a member of a protected group in AD, then that user object is flagged as a protected account.

Distribution Groups Nested in Protected Groups

When transitive membership in a protected group includes nested distribution groups (DGs), accounts that are members of nested distribution groups will not receive the protected group’s SID in their access tokens. However, distribution groups can be converted to security groups in Active Directory, which is why distribution groups are included in protected group member enumeration. Should a protected-group-nested distribution group ever be converted to a security group, the accounts that are members of the former distribution group will subsequently receive the parent protected group’s SID in their access tokens at next logon, but will already have been subject to AdminSDHolder and SDProp.

In summary, if you're a member of a DG that is protected, your account is flagged as a protected account, although that protected group's SID isn't included in your access token unless and until the nested DG is converted to a security group.

Take, for example, the user Jack NoAdmin shown below. Jack is a member of Domain Users and a group called "AdminSDHolderDistributionGroup" (I never claimed to be creative in my naming conventions.)

The AdminSDHolderDistributionGroup group is, you guessed it, a distribution group:

And the AdminSDHolderDistributionGroup group has a lone member, Jack NoAdmin:

However, the AdminSDHolderDistributionGroup group is also a member of the Administrators group in the NWTraders.msft domain:

When Jack NoAdmin logs on and runs whoami /groups, the Domain Admins group is not included in his access token (nor, of course, is the DG's SID since it's not a security group):

Nonetheless, Jack NoAdmin's account is flagged as a protected account:

Orphaned Protected Accounts

Suppose one day I'm perusing my directory and I notice that somebody has placed a distribution group into my Domain Admins group. Since this makes no sense to me, I remove that distribution group from the Domain Admins group. Alternately, suppose I am simply pruning the membership of my Domain Admins group because I'm working towards a "zero admins" Active Directory. What I may not realize is that my cleanup work is not finished at this point. When an account is removed from a protected group, it is no longer considered a protected account, but its adminCount attribute remains set to “1” if it is not manually changed. The result of this configuration is that the object’s ACLs are no longer updated by SDProp, but the object still does not inherit permissions from its parent object(s). Therefore, the object may reside in an OU to which permissions have been delegated, but the formerly-protected object will not inherit these delegated permissions. A script to locate and reset formerly-protected objects in the domain can be found in the Microsoft Support Article #817433.

AdminSDHolder Ownership

Most objects in Active Directory are owned by the domain’s built-in Administrators group; however, the AdminSDHolder object is, by default, owned by the domain’s Domain Admins group (this is a circumstance where Domain Admins do not derive their rights and permissions via membership in the Administrators group for the domain).

In versions of Windows prior to Windows Server 2008**, owners of an object can change permissions of the object, including granting themselves permissions that they did not originally have. The default permissions on a domain’s AdminSDHolder object prevent users who are members of Administrators or Enterprise Admins groups from changing the permissions for a domain’s AdminSDHolder object. However, members of the Administrators group for the domain can take ownership of the object and grant themselves additional permissions, which means that this protection is rudimentary and only protects the object against certain accidental modification by users who are not members of the Domain Admins group in the domain. Additionally, both the Administrators and Enterprise Admins (where applicable) groups have permission to change the attributes of the AdminSDHolder object.

**In another post, I'll talk about things like the OwnerRights SID in Windows Vista and later operating systems.

There are actually lots of reasons why the AdminSDHolder object and SDProp are important, but most of them aren't the focus of this post. This post is focused on one particular aspect of AdminSDHolder, which is this: we often encounter customer environments in which somebody has modified AdminSDHolder to "block" unauthorized changes to the most privileged groups in AD. In fairness, this kind of modification is something that is documented in an article entitled Modify the AdminSDHolder container. However, there's a really important line in that article, which is "This procedure will not completely prevent unauthorized changes to membership of administrative groups in an Active Directory domain." Somehow, this sentence doesn't seem to resonate with people, because time after time, we see modifications made to AdminSDHolder in an effort to increase security that actually does no such thing. At this point, I'll run through the modification we often see and why it doesn't work the way people intend it to work.

When we perform Active Directory Security Assessments (ADSAs), we run a series of PowerShell scripts that retrieve information about the forest we're assessing. These scripts do not require any special privileges in and of themselves, but in order to ensure that we're retrieving accurate information, we require an account that has read permissions to essentially everything in the directory that can be read by Enterprise Admins plus Domain Admins and Administrators in each domain. In an ideal world, there would be a built-in "Auditor" role in AD that would provide read access to the entire directory, plus read access to event logs, GPOs, etc., but as of this writing, there is no such thing. We always ask the customer if they have created such a role, but so far, none have, and as a result, we usually end up needing membership in the most privileged groups for the accounts that are used to run the scripts. In order to truly assess the security of the directory, we need to be able to read information that has been ACL'd to be available only to privileged accounts, such as cases in which attributes have been marked confidential or in which the organization has changed various "read" permissions in an attempt to prevent non-admins from reading directory information. It is here where we tend to run into a lot of creative modifications to AD that usually don't achieve the results their implementers intended.

We often see modifications made to the AdminSDHolder objects in an organization, usually because somebody is trying to allow, say, Domain Admins to modify objects, but prevent Administrators from modifying the object. A typical example of this is when we see customers modifying AdminSDHolder to attempt to prevent Administrators from adding themselves to the domain's Domain Admins group, and in the case of the root domain, the Enterprise Admins group. There are a few issues with this approach, however. The first is that there is usually a fundamental misunderstanding of what rights and privileges are granted to which of the three most powerful groups in AD. If you read Part 1 of this series, you know that many of the rights and permissions granted to Domain Admins and Enterprise Admins are actually the result of those groups being nested in their domain's Administrators group, the very group that is being treated as if it's "less" privileged. You also know that you should really consider these groups effectively equivalent from the perspective of potential privilege, since via various means, one may make itself a member of the others.

In these environments in which we see the organization attempting to "restrict" Administrators, what we also typically see is that they have smaller numbers of Enterprise Admins and Domain Admins, but lots of Administrators, because they're under the impression that Administrators aren't "as privileged" as Domain or Enterprise Admins. The default rights, permissions and scope of access associated with each of the three groups vary, but that does not mean that one group is more or less privileged than another- it just means that they have different default breadth of capability, not depth of capability. In short, believing that placing users into the Administrators group rather than the DA or EA groups buys a level of security is a misconception. It prevents somebody who is a member of the Administrators group for a domain from accidentally performing functions that are reserved (by default) for EA or DA members, but it doesn't prevent intentional attempts to bypass the default rights and permissions associated with the group- and to be clear, even in cases in which the bypass is intentional, it's not always malicious. A user who is a member of Administrators, but not Domain Admins, may encounter an access failure while trying to fix something that is broken and resolve the access failure by adding his/her account to the Domain Admins group. The action isn't malicious, but it is intentional, and it's trivial to perform.

The second misconception that we find in organizations that make these kinds of modifications is that they can modify AdminSDHolder in a fashion that will allow them to prevent Administrators from making themselves Domain Admins (or EAs), and that's the focus of this post. I now intend to show you the most common modifications we see made to the AdminSDHolder object, and to illustrate why none of them actually work.

Note: the second most common modifications we see in this area are attempts to "block" Enterprise Admins from accessing domains outside of the forest root domain. This is also a fruitless exercise, but isn't the focus of this particular post. I might perhaps write a separate post about this one, or I might not, but I hope you'll trust me when I say that what I wrote in Part 1 about how you should treat the three groups as effectively equivalent because each can become the others is true.

"Blocking" Administrators in Active Directory

Okay, so let's get down to brass tacks. In the scenario below, I've constructed a single-domain forest. (These principles still apply in multi-domain forests.) In this domain/forest, I have created three administrative users, each with membership in just one of the three most privileged groups in the domain. Well, sort of...

First, there's Jane Administrator. Jane is a member of Domain Users (default), Administrators and a third group that will come into play later in this scenario, a group I have created called "AdminDenyGroup".

Next, I've created Jim Enterprise Admin. Jim is a member of Domain Users, Enterprise Admins and the aforementioned AdminDenyGroup.

Finally, I have created Joe DomainAdmin, who is a member of Domain Users, Domain Admins and the AdminDenyGroup, which is not yet used for anything in this scenario.

At this point, you're probably saying, "wait, Joe and Jim aren't just members of those groups," and you're correct. In reality, all three of my users are members of the Administrators group in this domain, because of the default setting below, which is that EA and DA groups are nested into the domain's Administrators group.

If Jane, Jim and Joe log on, open a command prompt and type whoami /groups, here's what each of them sees:

Given the above, I'm using Jane Administrator for demonstration purposes since theoretically, she's the "least" privileged of the three users due to the fact that she's only in the Administrators group, but not the DA or EA groups.

In my scenario, I'm my company's primary AD administrator, and I want to prevent my three junior admins from making themselves members of the other privileged groups in the domain. First, I create a security group called AdminDenyGroup and I put Jane, Jim and Joe into that group. (Sometimes we see individuals added to these deny groups, sometimes we see the Administrators group added to these deny groups and sometimes it's a combination of the two. It doesn't much matter what permutation is used, the results are the same, which is that there is a subset of highly-privileged users who are "locked down", but still retain membership in one or more of the three most-privileged groups.)

Next, I modify the ACL on my domain's AdminSDHolder object to deny members of the AdminDenyGroup the ability to write to the members property on any of the protected groups in the domain. I do this using the DSACLs command-line utility, typing dsacls CN=AdminSDHolder,CN=System,DC=NWTraders,DC=msft /D NWTRADERS\AdminDenyGroup:WP;"member" into an administrative command prompt.

Adding "Deny" Entries to the AdminSDHolder DACL

In the DSACLs line above, I first invoke DSACLs, then provide the path to the object on which I want to view or modify ACLs. /D adds a "deny" access control entry (ACE) to the object and is followed by the name of the principal for which the deny ACE will be set. The WP entry represents "write property" and the "member" attribute is the property to modify.

Note: for full usage information for DSACLs, you can type "dsacls /?" at a command prompt. I'll also be providing some additional usage information later in this post.

Now that I've denied my junior administrators the ability to write to the membership of any of my domain's protected groups by modifying the AdminSDHolder ACLs, I force SDProp to run so that my changes are immediately propagated to all privileged groups in the domain. (As shown earlier in this post). Next, I look at the ACLs on the domain's AdminSDHolder object using ADSIEdit.msc and this is what I find:

AdminSDHolder ACL

I can see that my AdminDenyGroup has been added to the ACL with Special permissions, so I click the Advanced button to view the entry. This is what I see:

Special Permissions for AdminDenyGroup

No matter where I look on the AdminSDHolder object, I don't see what are the special permissions that I thought I had just set. However, if I look at the ACL on one of my protected groups, I can see that my ACE has actually been applied.

Domain Admins ACL

I point this out because it can be confusing to try to determine AdminSDHolder ACLs from a GUI unless you know where to look, which in some cases is actually on a different object. If, however, I use DSACLs to display the security on the AdminSDHolder object, I will actually see that there is a deny ACE for the AdminDenyGroup. If you're ever looking at one of your domains' AdminSDHolder objects and aren't able to see the ACEs for some of the entries, you should use DSACLs to display the security instead.

Now I feel that I've effectively stopped my junior admins from making themselves members of other protected groups and I head out for a long weekend. Over the weekend, Jane is troubleshooting an issue and receives an "access denied" message because she's not a Domain Admin. Jane fires up Active Directory Users and Computers and views the membership of the Domain Admins group, intending to add her account to the group. When she brings up the group's properties, she sees that the Add and Remove buttons are grayed out, which means she can't add herself to the Domain Admins group. Right?

Not so fast. Jane really needs to be a member of the Domain Admins group to fix the issue she's troubleshooting, so she locates her own account in the directory, right-clicks it and selects "Add to a group..."

Voila! Jane is now a Domain Admin and proceeds to fix whatever it was that she needed to fix.

I return to the office after my long weekend and I discover that Jane is now a member of the Domain Admins group in my domain. While I should probably just go talk to Jane to ask her why she made this change, instead I decide to get medieval on Jane and really block her ability to make herself a member of any other privileged groups besides Administrators. First, I spend some time with DSACLs and I look at the various permissions entries that I can set, shown below.

I can set generic permissions, but those are a little too broad for what I'm trying to accomplish, so I look at the specific permissions that I can set.

Specific PermissionsSD- DeleteDT- Delete an object and all of its childrenRC- Read security informationWD- Change security informationWO- Change owner informationLC- List the children of an objectCC- Create child objectDC- Delete a child objectWS- Write To Self (also known as Validated Write).

Note: there are 3 kinds of validated writes, and how the WS permission is applied will depend on the type of object to which it is applied (I don't have to specify this in DSACLs). Self-Membership is applied to Group objects and allows updating membership of a group in terms of adding/removing one's own account. Validated-DNS-Host-Name is applied to computer objects and allows systems to update their DNS names. Validated-SPN is also applied to computer objects and allows the computer to update its own SPN attribute.

Note: LO can be used to grant list access to a specific object if List Children (LC) is not granted to the parent as well. It can also be denied on specific objects to hide those objects if the user/group has LC on the parent. AD DS does NOT enforce this permission by default, it has to be configured to start checking for this permission.

Now that I've familiarized myself with the various specific permissions that I can deny my AdminDenyGroup, I type the following at an administrative command prompt:

This command adds deny ACEs to block the group from deleting protected objects, reading the ACLs on protected objects, changing the ACLs on protected objects, changing ownership of the protected objects or adding themselves to protected groups.

I run SDProp and pull up the permissions on the Domain Admins group.

Satisfied that I've finally thwarted my junior administrators, I head out of town for a three-day conference. While I'm out, Jane is performing routine maintenance and receives an "Access denied" message when she tries to perform a function that, by default, requires Domain Admins membership. Jane is perplexed, because she remembers that she added herself to the DAs group over the weekend, so she locates her account in AD, notices that she's no longer a member of Domain Admins (since I removed her) and tries to re-add herself via the same mechanism that she used over the weekend. This is what she sees:

Jane is curious about this new behavior, so she decides to take a look at the permissions on the Domain Admins group. When she brings up the group's properties and tries to view the Security tab, this is what she sees:

Fortunately, there is that "Advanced" button that Jane can click, so she does, and she sees this unusual dialog box:

Since it seems pretty straightforward, Jane selects her own name in the "Change owner to:" field and clicks "OK". She receives a message telling her that she needs to close and reopen the object's properties, so she does, and this is what she now sees:

Now that Jane owns the object, she clears the restrictions on the AdminDenyGroup, or, if she wants quicker results, just removes the ACE altogether.

Jane then adds herself to the Domain Admins group. The next time SDProp runs, the AdminSDHolder-defined permissions are reset back to what I had defined, but Jane is still a member of Domain Admins.

I return from my conference and I again find that Jane is a member of the Domain Admins group, but when I go to look at the ACLs on the group, nothing seems amiss, so I'm now completely confused.

Getting to the Point(s)

First, why was Jane able to take ownership of the Domain Admins group despite the fact that I had explicitly denied that permission to the AdminDenyGroup? Because Jane is a member of the Administrators group in the domain. One of the privileges held by the built-in Administrators group is the ability to take ownership of objects. Privileges trump permissions. (The difference between rights and privileges is fuzzy, even in some of our own documentation, so for brevity's sake, let's just go with "privileges" here.)

Second, at this point, you might be thinking of various creative approaches that I can take to stop Jane from doing what she did. Perhaps in addition to what I've done above, I can create a restricted groups policy and enforce the membership of my DA, EA and Administrators groups. Well, Jane can still do all of the things that she did above, and once she has that membership in Domain Admins, she can go delete or modify my restricted groups policy (the mechanics will depend on how my GPO security is configured as well as other factors, but one way or another, she can find a way to bypass that policy). Perhaps I decide that I'm going to give Jane a restricted set of administrative tools and deny her the ability to launch dsa.msc. Jane can just switch to a command line to do what she previously did via the GUI. Perhaps I can start setting all kinds of deny ACLs for that AdminDenyGroup, muck with default schema permissions, block .dlls, muck with rights and privileges...

And all I'll really accomplish by doing all of those things is to make a mess of my directory and in all likelihood, generate a slew of difficult-to-fix errors. Why should I "lock down" people rather than simply not giving them an excessive level of privilege in the first place? If I don't give it, I don't have to take it away. So why don't I look for ways that I can let Jane do what she needs to do without giving her wholesale privilege that I then try to "tweak"? You don't make a demigod by shackling a god; build up rather than tearing down.

Perhaps you're thinking that I should be implementing stringent auditing, as well. Well, I should be, but auditing is reactive, not proactive. Alerts come after the breach has already occurred, which brings me to what I discussed in the first post in this series. I've mentioned in a previous post what the team for which I work in Microsoft does, and that we've been heavily focused on APT and APT-like attacks for several years. We've seen varying levels of sophistication in these attacks, where the adversaries usually start with the simplest approach that gets them what they want, escalating their tactics only once they've been discovered and the target organization has started to kick in with some defenses. What we do see, however, is that once the attackers have gained privileged access to an environment, they act swiftly to obtain both data and additional access to the environment. The goal after the first intrusion is usually to find "legitimate" means to access the target organization, and to blend in with real users and systems so that it's hard to find the attackers. So while there may be excellent auditing and alerting in use within your company, the people targeting it may only need one of your privileged accounts for minutes in order to achieve what they want to achieve. No matter how quickly you respond to an alert that a privileged account has been breached, at that point, it's too late.

This is why I am a proponent of the "zero admins" approach to Active Directory (and it's not just AD to which this principle is applicable). If you have no accounts that actually have broad and deep privilege in your environment, then you have already exponentially increased the difficulty the attackers face in attempting to own your environment. You will never stop somebody who wants to attack your organization from doing so, but what you can do is to make it so hard for them to get to what they want that you're able to get them out before they've gotten what they've come for. This isn't a new idea- it's basic defense in depth, least privilege, etc. We've been telling this story for years. I'm just trying to tell it a little bit differently, I guess.

Thanks for reading a really long post. I hope it made some sense.

Until next time...

P.S. This post was actually published on 13 August 2011, but since I started it in June, it bears an old date. Sorry about that.

If You Haven't Been Hacked, You May Not Be Looking Closely Enough

Clearly, I am not the most conscientious blogger, as can be observed by the lack of any posting regularity here. This is in part due to the fact that for the past few years, the team on which I work has been busy helping compromised customers respond to a specific class of attacks known as Advanced Persistent Threat (APT) attacks. Because there is much debate in the security community about what is or is not an APT, one of our sibling teams has coined a more general term to describe a broader classification of attacks, which is Determined Human Adversaries (DHA). Regardless of whether we're talking about APTs or DHAs, however, the important part is this: compromise has become the norm rather than the exception. The report to which I just linked is consistent with what we've been seeing in our work- even companies who thought they were "bulletproof" have discovered that perception and reality can be quite different. Something that is specific to the attacks with which my team primarily deals is that these attacks focus not on destruction, but on exfiltration of an organization's intellectual property (IP). There are additional types of attackers out there, however, and their interests may be very different, whether they be monetary theft, denial of service, defacement or destruction of the computing environment. In the end, it doesn't really matter what motivates the attackers who may have targeted you- what matters is how easily they can penetrate and compromise your environment, and how deep and broad the compromise is.

So, now that that small bit of background is out of the way, on to the important bits. As I said, my team has been working with one compromised organization after another, and if there is a single factor that has been the tipping point between a breach that can be contained and a breach that takes over the entire environment, it is how tightly the company has locked down privileged accounts in their environments. This is by no means the only factor in the severity of a compromise, but in one customer after another, what we've seen is that it is via control of the most privileged accounts that attackers have succeeded in wholesale compromise rather than picking around the edges with piecemeal success.

Where to Begin?

Since the inception of Active Directory, I have been a proponent of having no Domain Admins (DAs), no Enterprise Admins (EAs) and no Built-in Administrators (BA)- that's right, zero admins. I said this long before I joined Microsoft as a full-time employee (FTE), and I haven't changed my mind in the years that I've been an FTE. It has always been possible to design and deploy an Active Directory implementation with privileges delegated to various groups and accounts and no use of the "canned" privileged AD groups that I just mentioned (DA/EA/Admins). However, it has also been fairly tedious to build and difficult to maintain without additional tooling.

In hopes of helping companies to significantly increase the security of their infrastructures, I am now attempting to post a series of posts on the subject of "admin free" Active Directory, and if I manage to keep this exercise rolling, I'll then attempt to address admin-free Windows, as well. There's obviously a lot more than just Windows at play in most environments, but Active Directory is ubiquitous, so that's where I'm starting. In this first post, I'm keeping it pretty simple and will provide some basic information about the most privileged groups in Active Directory, including the depth and breadth of their privilege. I have found that the differences between each of these groups are often misunderstood, and the first step in addressing the problem is understanding the problem. What I'm posting below is text that I have written and given to a number of customers in the past, usually as part of an Active Directory Security Assessment (ADSA), which is an assessment built by our team over the past several years.

Built-in Privileged Accounts and Groups

Active Directory (as a product) is designed in a manner that is intended to facilitate delegation of administration and the principle of least privilege in assigning rights and permissions. “Regular” users who have accounts in an Active Directory domain are, by default, able to read much of what is stored in the directory, but are able to change only a very limited set of data in the directory. Users who require additional privilege can be granted membership in various “privileged” groups that are built into the directory so that they may perform specific tasks related to their roles, but cannot perform tasks that are not relevant to their duties.

Within AD, there are three built-in groups that are the highest privilege groups in the directory- Enterprise Admins, Domain Admins and Administrators. The following describes the default configuration and capabilities of each of these groups:

Highest-Privilege Groups in Active Directory

Enterprise Admins

Enterprise Admins is a group that exists only in the forest root domain, and by default, it is a member of the Built-in Administrators group in all domains in the forest. The built-in Administrator account in the forest root domain is the only default member of the Enterprise Admins group. Enterprise Admins are granted rights and permissions that allow them to effect forest-wide changes, meaning changes that affect all domains in the forest, such as adding or removing domains, establishing forest trusts, or raising forest functional levels. In a properly designed and implemented delegation model, Enterprise Admin membership is required only when first constructing the forest or when making certain forest-wide changes.

Domain Admins

Each domain in a forest has its own Domain Admins group, which is a member of that domain’s Built-in Administrators group as well as a member of the local Administrators group on every machine that is joined to the domain. The only default member of the Domain Admins group for a domain is the built-in Administrator account for that domain. Domain Admins are “all-powerful” within their domains, while Enterprise Admins have forest-wide privilege. In a properly designed and implemented delegation model, Domain Admin membership should be required only in “break glass” scenarios, meaning situations in which an account with high levels of privilege on every machine in the domain is needed. While native Active Directory delegation mechanisms do allow delegation to the extent that it is possible to use Domain Admin accounts only in emergency scenarios, constructing an effective delegation model can be time-consuming and many organizations leverage third-party tools to expedite the process.

Administrators

The third group, Administrators, is the domain local group into which Domain Admins and Enterprise Admins are nested, and it is this group that is granted many of the direct rights and permissions in the directory and on domain controllers. However, the Administrators group for a domain does not have any privileges to member servers or to workstations- membership in the machines’ localAdministrators group is where local privilege is granted (and Domain Admins are members of all domain-joined machines’ local Administrators groups by default).

Note:

While these are the default configurations of these privileged groups, the reality is that a member of any one of the three groups may manipulate the directory to gain membership in any of the other groups. In some cases, it is trivial to achieve, while in others it is more difficult, but from the perspective of potential privilege, all three groups should be considered effectively equivalent.

Schema Admins

A fourth highly-privileged group, Schema Admins, exists only in the forest root domain (by default) and has only that domain’s built-in Administrator account as a default member, similar to the Enterprise Admins group. The Schema Admins group is intended to be populated only occasionally (when modification of the Active Directory schema is required), and temporarily.

Built-in and Default Groups in Active Directory

The table below provides some general information about Built-in and Default groups in Active Directory

Group

Description

Notes

Account Operators

Members of this group can create, modify, and delete accounts for users, groups, and computers located in the Users or Computers containers and organizational units in the domain, except the Domain Controllers organizational unit. Members of this group do not have permission to modify the Administrators or the Domain Admins groups, nor do they have permission to modify the accounts for members of those groups. Members of this group can log on locally to domain controllers in the domain and shut them down. Because this group has significant power in the domain, add users with caution.

Members of this group have full control of all domain controllers in the domain. By default, the Domain Admins and Enterprise Admins groups are members of the Administrators group. The Administrator account is also a default member. Because this group has full control in the domain, add users with caution.

Default user rights: Access this computer from the network; Adjust memory quotas for a process; Back up files and directories; Bypass traverse checking; Change the system time; Create a pagefile; Debug programs; Enable computer and user accounts to be trusted for delegation; Force a shutdown from a remote system; Increase scheduling priority; Load and unload device drivers; Allow log on locally; Manage auditing and security log; Modify firmware environment values; Profile single process; Profile system performance; Remove computer from docking station; Restore files and directories; Shut down the system; Take ownership of files or other objects.

Builtin Container

Domain Local Security Group

Backup Operators

Members of this group can back up and restore all files on domain controllers in the domain, regardless of their own individual permissions on those files. Backup Operators can also log on to domain controllers and shut them down. This group has no default members. Because this group has significant power on domain controllers, add users with caution.

Default user rights: Back up files and directories; Allow log on locally; Restore files and directories; Shut down the system.

Builtin Container

Domain Local Security Group

Cert Publishers

Members of this group are permitted to publish certificates for users and computers to the directory. Typically, certification authority (CA) servers are placed into this group in each domain to which they may publish certificates.

Users Container

Domain Local Security Group

Cryptographic Operators (Windows Vista SP1 and above)

FIPS 140-2 defines a “Crypto Officer” role, which is represented by the Cryptographic Operators group in Windows, first introduced in Windows Vista SP1. When the "System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing" security setting is configured in local or group policy objects, only members of the Cryptographic Operators group or the Administrators group can configure Cryptography Next Generation (CNG) settings by default. Specifically, Cryptographic Operators can edit the cryptographic settings in the IPsec policy of Windows Firewall with Advanced Security (WFAS)

Applicable only when FIPS-compliant encryption is enforced.

Builtin Container

Domain Local Security Group

Debugger Users

The presence of a Debugger Users group indicates that debugging tools have been installed on the system, whether via Visual Studio, SQL, Office or other applications that require and support a debugging environment. This group allows remote debugging access to remote machines. When this group exists at the domain level, it indicates that such an application has been installed on a domain controller.

By default, the only member of the Debugger Users group is the Administrator who installed the application. Additionally, the Debugger Users group is only granted Launch and Access permissions to the Machine Debug Manager.

This is neither a built-in nor a default group, but when present in Active Directory, is cause for further investigation.

DHCP Administrators

Members of the DHCP Administrators group can view and modify any settings on the DHCP server. DHCP Administrators can create and delete scopes, add reservations, change option values, create superscopes, or perform any other task required to administer the DHCP server, including export or import of the DHCP server configuration and database.

Members of the DHCP Administrators group do not have unlimited administrative rights. For example, if a DHCP server is also configured as a Domain Name System (DNS) server, a member of the DHCP Administrators group can view and modify the DHCP configuration but cannot modify DNS server configuration on the same computer.

Because members of the DHCP Administrators group have rights on the local computer only, DHCP Administrators cannot authorize or unauthorize DHCP servers in Active Directory Domain Services (AD DS). Only members of the Domain Admins group can perform this task by default.

Users Container

Domain Local Security Group

DHCP Users

Members of the DHCP Users group have read-only access to the server by using the DHCP Microsoft Management Console (MMC) snap-in, which allows them to view, but not to modify, server data, including DHCP server configuration, registry keys, DHCP log files, and the DHCP database. DHCP Users cannot create scopes, modify option values, create reservations or exclusion ranges, or modify the DHCP server configuration in any other way.

Users Container

Domain Local Security Group

DnsAdmins

Members of this group have administrative access to the DNS Server service. This group has no default members.

Users Container

Domain Local Security Group

DnsUpdateProxy

DNS clients who are permitted to perform dynamic updates on behalf of some other clients (such as DHCP servers performing registrations on behalf of clients that are incapable of performing dynamic DNS registrations).

Users Container

Global Security Group

Domain Admins

Members of this group have full control of the domain. By default, this group is a member of the Administrators group on all domain controllers, all domain workstations, and all domain member servers at the time they are joined to the domain. By default, the Administrator account is a member of this group. Because the group has full control in the domain, add users with caution.

Default user rights: Access this computer from the network; Adjust memory quotas for a process; Back up files and directories; Bypass traverse checking; Change the system time; Create a pagefile; Debug programs; Enable computer and user accounts to be trusted for delegation; Force a shutdown from a remote system; Increase scheduling priority; Load and unload device drivers; Allow log on locally; Manage auditing and security log; Modify firmware environment values; Profile single process; Profile system performance; Remove computer from docking station; Restore files and directories; Shut down the system; Take ownership of files or other objects.

Users Container

Global Security Group

Domain Computers

This group contains all workstations and servers joined to the domain. By default, any computer account created becomes a member of this group automatically.

Users Container

Global Security Group

Domain Controllers

This group contains all domain controllers in the domain.

Users Container

Global Security Group

Domain Guests

Contains all domain guests. By default, the only member of this group is the built-in Guest account for the domain, which is disabled by default and does not receive the Authenticated User SID in its access token if it is enabled and used for logon.

Users Container

Global Security Group

Domain Users

This group contains all domain users. By default, any user account created in the domain becomes a member of this group automatically. This group can be used to represent all users in the domain. For example, if you want all domain users to have access to a printer, you can assign permissions for the printer to this group (or add the Domain Users group to a local group, on the print server, that has permissions for the printer).

Users Container

Global Security Group

Enterprise Admins (exists only in forest root domain)

Members of this group have full control of all domains in the forest. By default, this group is a member of the Administrators group on all domain controllers in the forest. By default, the Administrator account is a member of this group. Because this group has full control of the forest, add users with caution.

Access this computer from the network; Adjust memory quotas for a process; Back up files and directories; Bypass traverse checking; Change the system time; Create a pagefile; Debug programs; Enable computer and user accounts to be trusted for delegation; Force shutdown from a remote system; Increase scheduling priority; Load and unload device drivers; Allow log on locally; Manage auditing and security log; Modify firmware environment values; Profile single process; Profile system performance; Remove computer from docking station; Restore files and directories; Shut down the system; Take ownership of files or other objects.

Users Container

Universal Security Group unless domain/forest is Windows 2000 mixed, in which case it is a Global Security Group

Event Log Readers (Windows Server 2008 or later)

Members of this group can read all event logs from local machines when a local machine group is used, and from domain controllers when the domain group is used. This group is introduced at the domain level in Windows Server 2008 and can be used either to grant users the ability to read event logs, or to grant machines to read event logs, in the case of event subscriptions.

By default, contains the Guest account for the domain (which is disabled by default) and the Domain Guests domain global group. Members of the Guests group have the same rights and permissions as Users do, with the exception of the built-in Guest account, which does not receive the Authenticated Users SID in its access token

Builtin Container

Domain Local Security Group

Incoming Forest Trust Builders (exists only in forest root domain)

Members of this group can create one-way, incoming forest trusts to the forest root domain. For example, members of this group residing in Forest A can create a one-way, incoming forest trust from Forest B. This one-way, incoming forest trust allows users in Forest A to access resources located in Forest B. Members of this group are granted the permission Create Inbound Forest Trust on the forest root domain. This group has no default members.

Builtin Container

Domain Local Security Group

Network Configuration Operators

Members of this group can make changes to TCP/IP settings and renew and release TCP/IP addresses on domain controllers in the domain. This group has no default members.

Builtin Container

Domain Local Security Group

Performance Log Users

Members of this group can manage performance counters, logs and alerts on domain controllers in the domain, locally and from remote clients without being a member of the Administrators group.

Builtin Container

Domain Local Security Group

Performance Monitor Users

Members of this group can access performance counter data on domain controllers locally and remotely without being members of the Administrators or Performance Log Users groups.

Builtin Container

Domain Local Security Group

Pre-Windows 2000 Compatible Access

Members of this group have read access on all users and groups in the domain. This group is provided for backward compatibility for computers running Windows NT 4.0 and earlier. By default, the special identity Everyone is a member of this group. Add users to this group only if they are running Windows NT 4.0 or earlier.

Members of this group can manage, create, share, and delete printers connected to domain controllers in the domain. They can also manage Active Directory printer objects in the domain. Members of this group can log on locally to domain controllers in the domain and shut them down. This group has no default members. Because members of this group can load and unload device drivers on all domain controllers in the domain, add users with caution.

Servers in this group are permitted access to the remote access properties of users

Users Container

Domain Local Security Group

Remote Desktop Users

Members of this group can remotely log on to domain controllers in the domain if granted the right to log on via Terminal Services. This group has no default members.

Builtin Container

Domain Local Security Group

Replicator

This group supports directory replication functions and is used by the File Replication service on domain controllers in the domain. This group has no default members. Do not add users to this group.

Builtin Container

Domain Local Security Group

Schema Admins (exists only in forest root domain)

Members of this group can modify the Active Directory schema. By default, the Administrator account is a member of this group. Because this group has significant power in the forest, add users with caution.

Users Container

Universal Security Group

Server Operators

On domain controllers, members of this group can log on interactively, create and delete shared resources, start and stop some services, back up and restore files, format the hard disk, and shut down the computer. This group has no default members. Because this group has significant power on domain controllers, add users with caution.

Default user rights: Back up files and directories; Change the system time; Force shutdown from a remote system; Allow log on locally; Restore files and directories; Shut down the system.

Builtin Container

Domain Local Security Group

Users

Members of this group can perform most common tasks, such as running applications, using local and network printers, and locking the server. By default, the Domain Users group, Authenticated Users, and Interactive are members of this group. Therefore, any user account created in the domain becomes a member of this group.

Builtin Container

Domain Local Security Group

I think that's enough for now. Stay tuned for further information, and rest assured, this isn't the only topic area of concern when we're talking about these kinds of attacks. If I manage to plow through my "admin free" series, the next series is going to talk about all those legacy systems and applications in your environment. If you could eliminate legacy systems and software and implement appropriate role-based access controls (RBAC) so that you have an admin-free environment, you could reduce your attack surface by a significant percentage.

As always, anything in this blog is my opinion, based on my experiences. You should not take what I write as being representative of Microsoft policy, recommendation or best practice, because I don't have the authority to define any of those things. What I do have, however, is experience and opinion, and nothing I say should contradict any Microsoft recommendations- I just build on top of them. And remember- I'm a security nerd, so you may read some of what I write and think that it's overkill. That's absolutely your prerogative.

So, it turns out that if you don't sign into the Zune marketplace for 30 days, all of your DRM'd content expires. I got a new 64 GB Zune HD recently and couldn't figure out why it was loading so slowly. I usually just sync my Zunes when I'm home, because my work laptop (which travels with me) runs Windows Server 2008 R2 and I never thought it was a big deal to just synch with one of my personal machines when I was home.

Despite the fact that I subscribe to the marketplace on a quarterly basis, it would seem that when you download music from the marketplace, it is *always* tagged with a 30-day expiration.

Long story short, I took my Zune HD on the road with me and installed Zune software on my WS08R2 machine (which is completely unsupported and something you should not do), and when I finally had a few minutes to dig into my collection, I found that the reason that the Zune was loading slowly was because all of my rights had expired and something wasn't working when it came to updating them. I tried various things to fix this, and finally found something that worked. I fired up Windows Media Player, which is configured to automatically download my usage rights, and I let it update my library. Voila! The Zune software now recognizes my rights to the songs and is churning away at putting them back into my library.

So, don't do what I did. Specifically, DON'T:

1. Fail to log in to the Zune marketplace for more than 30 days.

2. Install Zune on an unsupported platform

3. Use the Zune software to purge all of your subscription content and tell it to re-download everything.

If you have migrated or upgraded the sites on which you host your CA CRLs and delta CRLs to IIS 7, you may have noticed a (rather frustrating when you're experiencing it) new behavior. IIS 7 will, by default, reject requests containing double escape characters (for example, files containing a "+" sign in the name, such as delta CRLs). While this is a valid, standards-based security feature, the end result is that your clients cannot retrieve delta CRLs from an IIS 7-hosted site unless you change the configuration to allow the double escape character. Do not configure this change server-wide; configure it only on the site(s) hosting delta CRLs. To disable the double escape checking for your CRL site(s), you can use the example below (replace the site name with that of the site hosting your CRLs):

First, the warnings:

1. Sometimes I am a bit of a salmon, meaning that I have a tendency to swim upstream, metaphorically speaking.

More specifically, I like to take current thoughts around "best practices" and pick them apart to see if they actually make sense as a best practice. One of my favorite words is "specious". A specious argument is one that seems to make sense on the surface, but when actually evaluated, turns out not to make so much sense, after all.

2. Anything I say in this blog is purely my personal opinion and may not reflect the opinions of my employer, my colleagues, my mother or anybody else. They're just my thoughts, and there are no warranties associated with them. I also tend to have a sense of humor that may perplex some. If something comes across as obnoxious, it was probably intended to be humorous. Please don't take offense; I certainly don't intend any.

Now, the point of this post:

For some time now, I've been a proponent of using virtual machines to host offline CAs- with a lot of specific caveats about how to do it. I've started discussions on this subject in various forums (electronic or otherwise), including internally at Microsoft.

Some months back, somebody posted to an internal distribution list asking if it is a supported scenario to install an offline CA in a virtualized environment and are there a lot of companies doing such. That query ended up spurring a relatively lengthy debate between myself and a few others on the DL, and I think a summary of the discussion might be useful for those of you out there who are wondering whether or not you can build your offline CAs in virtualized environments.

I'll give you my thoughts as to when and how virtualized offline CAs may be appropriate, and then I'll give you the arguments against the idea that I've received in the past and my rebuttals to those arguments.

Should I Build My Offline CAs in Virtualized Environments?

Okay, so you're looking at building or updating your PKI and you'd like to put your offline CAs (root, policy, whatever) in virtual machines (AKA partitions in Windows Server 2008 Hyper-V). I am absolutely in favor of your doing this if you do the following:

1. Build your offline CA in a virtual machine that is hosted on a removable/external drive. If that is not an option, build it on a separate disk that you can reformat at the end of this process. Don't build this on one of your "regular" servers- build this whole shebang on a machine that you've built specifically for the purpose of building your offline CA, and don't connect the host machine to a network.

2. Use a network-attached HSM with a crossover cable connected to the host machine to generate and secure your CA's keys (most major HSM vendors support virtualized CAs these days).

3. Back up your keys and store them in a secure location. As in, a safe, not your desk drawer.

4. Shut down your offline CA once you've done what you need to do with it.

5. Burn at least two copies of your offline CA's virtual machine to removable media such as external hard drives.

6. DELETE the virtual machine from the host server. If you built the virtual machine on a removable or external drive, all you're deleting is the configuration information that your virtual machine host maintains about the virtual machine. If you did not build the virtual machine on a removable/external drive, perform a DoD wipe of the disk on which you built it. Even better is to just wipe the entire machine that you used as your host.

7. Place one copy of the virtual machine in your primary datacenter's safe along with a backup of its keys. Place the second copy in your disaster recovery datacenter's safe with a second backup of its keys. Ideally, courier a third copy of the virtual machine and backed-up keys to your company's lawyers with instructions that they must secure it in a safe and must notify your company at intervals when the CA needs to be brought up for maintenance, CRL publishing, certificate renewal, subordinate CA certificate issuance/renewal, etc.

8. At each CRL publication interval (or any other time at which you're required to spin up the CA), do the following things:

Fire up the virtualization software you use, [very, very, very] preferably on a freshly-built machine that has never been connected to a network. For example, grab a spare laptop, wipe it, install your virtualization server of choice (Windows Server 2008 Hyper-V rocks, btw).

Obtain all copies of your backed up virtual machine and keys.

Connect everything up with your primary copy of the virtual machine and keys and load the virtual machine into your virtualization host.

Bring up the offline CA. Since you have it up and running, even though you may have only brought it up to publish the CRL, take advantage of the fact that you have it up and running. Apply any patches that need to be applied to the CA. Apply any updates that need to be applied to your virtualization software. If your virtual disk format needs updating, update it. Publish your CRL. Renew certificates that need renewing. Backup your keys again, if necessary.

Assuming the above all went well, shut down the CA. Burn fresh copies of the virtual machine to each removable/external drive, fire them up to make sure they work, then shut them down and lock everything back up in the safes. Repeat this process whenever you need to bring up your offline CA(s). If you broke something when you were updating your primary copy of the virtual machine, start over with your secondary copy and don't do whatever you did that broke the first one. Then do the copy-and-lock-up thing.

The Really Important Warnings

Here are the key concepts in the above:

1. Don't build the virtual machines on a production server that is connected to your network. Build them on a disconnected machine that was built specifically for the purpose of temporarily hosting these virtual machines.

2. If you want to keep the host machine, put it in the safe with your removable drive containing the virtual machine. Do NOT just build all of this on a laptop and stick it in a safe “as is” and hope that when you fire it up again, everything works. Get the virtual machine off of that host machine's hard drive.

3. If you have a situation wherein you need to bring up the CA in an emergency, having a ready-to-run machine in the safe with the backed-up virtual machine and keys is very handy. However, every time you bring up that machine you still need to do the things above to remove the virtual machine from the parent, because you do not want to find out five years from now during an emergency CRL publication when you're scrambling to bring up your root CA that the hardware has failed. If you're going to store a machine in the safe with the virtual machine, every time you maintain the CA, you should also evaluate that host machine to determine if it should be replaced with a newer host machine.

4. Every time you bring up the CA, make sure that you update and re-burn the virtual machine.

5. Never leave the virtual machines sitting on any host that is accessible in any way other than opening the safe and pulling everything out (or calling your lawyers, telling them that you had a massive disaster and that you need them to courier their copy from their safe).

6. Do not sit alone at your desk building these things. Build them in accordance with your company's security policies, with more than one person sitting there during the entire process.

7. DOCUMENT everything, including the processes around the above, the instructions regarding when the CAs are brought up, instructions for your lawyers, etc.

In other words, treat that virtual machine as you would any crucial and highly secured piece of your infrastructure, albeit doing certain things virtually rather than physically.

Arguments and Rebuttals

On to the last part- here are some of the rebuttals I've received in the past when I've espoused virtualized offline CAs.

"A virtual machine isn't as secure as a physical server- somebody could just put it in his/her pocket and walk out with it."

No, somebody cannot do that, because you're going to follow my instructions and never have this thing sitting in an accessible location, nor will anybody ever be left alone with it. See why I insist upon multiple people being there for the build and maintenance processes?

"You can't just stick a laptop in a safe- it's going to get old and die when you least expect it, and then what?"

That's why I tell you not to build the virtual machine on a host and leave it there. The whole point of a virtual machine is that it's hardware agnostic. That means that you can build out a new host each time you crack open the safe rather than praying that the hardware didn't die while it was gathering dust. Besides, if you built the whole shebang on a physical server in a locked cage in your datacenter, how is that physical machine any less likely to die than the one in the safe? Take the hardware out of the equation.

"How are you going to guard it? Cage-locked servers are more secure."

Cage-locked servers in a secure datacenter are no more secure than something that is locked in a safe and completely physically untouchable. In fact, at a company where I worked and built a PKI with virtualized offline CAs in the past, one of the chief arguments in favor of virtualization was this- our VP of Operations could walk into our datacenters and do whatever he wanted to do to a physical server in the guise of "maintenance", and he wouldn't be challenged because, well, he was the VP of Operations and was a hands-on kind of guy. However, for him to retrieve a virtual machine from a safe in the secure datacenter not only required many more hoops for him to jump through, but would also raise much more suspicion if he tried to do it covertly. In fact, he simply couldn't just go to the safe, crack it open and start mucking with what was inside. Nuh-uh. No way. Not gonna happen without security guards tackling him to the floor, sorry.

"Virtualization software changes over time. What if your virtual machine is no longer compatible with your virtualization software?"

That's why you update the virtualization software and virtual machine format whenever you bring up the virtual machine. Additionally, when you stick the virtual machines in the safes, it's a good idea to also stick in a DVD containing a copy of the virtualization software that you used at the time you spun up the virtual machine. Worst case scenario, you end up having to load some old virtualization software on a sacrificial machine the next time you spin up the virtual machine. And then, of course, you're going to update everything, remember? Lather, rinse, repeat. This is a regular, albeit infrequent and stringently controlled, operational task. Treat it like such.

"A virtualized CA requires a netHSM, which is more expensive than a dedicated HSM."

You should be using a netHSM for your issuing CAs, anyway. Use that same netHSM to generate and back up your offline CAs' keys. But remember, don't leave them on the HSM. And whenever you need to use the netHSM (you really should have two for redundancy), connect it to the virtual machine host with a crossover cable- while it is not connected to your network. Alternately, waste spend your money on a dedicated machine with a dedicated HSM and stick that in the safe, then evaluate it every time you fire up your CA to determine if it needs updating. I don't recommend this, personally. It's unnecessary expense and increases your risk of hardware failure. So, let's see- use a device you're buying anyway (the netHSM), OR buy an additional device (the dedicated HSM) that you're going to use perhaps forty times, at most, over the next couple of decades. And pray that it doesn't die. Hm, better buy two of them, just in case. Or maybe three. Or just use the netHSM that you should already be buying anyway.

"Most virtualization software supports scripting APIs, so somebody could remotely attack the virtual machine via the host. Or it could get infected with a Trojan."

Uh, nope. Host never connects to network. Host never hosts virtual machines without multiple people sitting there watching each other. Host can't be infected via the network, Host can't be infected via human being unless multiple human beings are in collusion and don't value their jobs, and then what are you doing letting them handle such security-sensitive stuff in the first place?

"You rely on the host/parent security as well as the virtual machine security when you build an offline CA in this configuration, while in a dedicated machine scenario you rely on one layer only; you're increasing your attack surface."

This is one of those specious arguments I mentioned earlier. It sounds like it makes sense until you really think about it. Let's think about it.

Host is never connected to a network.

Host is built from scratch every time it's used (except in the emergency scenario where you might have that laptop that was sitting in the safe that you had to fire up for this emergency). Sounds to me like a heck of a lot more secure host than a physical server sitting in my datacenter where somebody has years to try to figure out a way to get at it (and where it's probably connected to my network, anyway).

"The procedures that have been taken to secure the host machine will have to be done every time you want to bring the offline CA online since the host hardware is usually reused."

I smell speciousness again. First, there's an assumption of hardware reuse, which I have pretty clearly recommended against. Second, how is securing your freshly-built, never-connected-to-the-network host any harder than securing a machine that you've left up and running and sitting in your datacenter connected to your network?

"I just don't like it. We've always said it's not a best practice to virtualize offline CAs."

Any argument that begins with "we've always done it that way" has instantly lost credibility with me. Times change. Hardware changes. Technologies change. Therefore, best practices change. Call me a Darwinist, but I'm of the opinion that dinosaurs are extinct for a reason.

Thanks for reading this very long post, and for those who are wondering- yes, we do support virtualized offline CAs, although I'd strongly recommend that you create and use them with Windows Server 2008 Hyper-V, because we can't fully support other vendors' virtualization products.

At some point, I'll probably write up another really long post around the things that I think you should always do from a policy and procedure perspective when you're designing and building a PKI.