Whem running an Exchange 2010 DAG over a WAN, you may run into some of the limitations of Microsoft FCS (Failover Cluster Service). This service defaults to fairly low timeouts for fast failover in LAN environments. In a WAN environment, where latency may be higher, and some packet loss may occur, you may need to tweak the timeouts for FCS. I advise to tweak most of the settings via the FCS admin tool. However, there are a few settings to tweak via the command line, and here are the maximum values you can configure to make it “less sensitive”:

Exchange 2010 DAGs use Windows Failover Clustering. By default, FCS has fairly low timeouts that are ideal for use in fast localised LAN environments

If you operate your Exchange 2010 DAGs over a WAN where issues such as latency and packet loss can occur, you may find that your email databases are failing over. By default, heartbeat frequency (subnet delay) is 1000ms for both local and remote subnets and when a node misses 5 heartbeats (subnet threshold) another nod within your DAG cluster will initiate a failover.

You can change these values to their maximums by issuing the commands below on a DAG mailbox server in a command box.

cluster /prop SameSubnetDelay=2000:DWORD

cluster /prop CrossSubnetDelay=4000:DWORD

cluster /prop CrossSubnetThreshold=10:DWORD

cluster /prop SameSubnetThreshold=10:DWORD

You can check that the properties have been applied by executing the following command on a DAG mailbox server in a command box.

cluster /prop

If you virtualise your Exchange 2010 mailbox servers, this may also assist in preventing failover when backing up your VMs using backup products that take snapshots of your VMs like Veeam Backup and Replication. Note that doing backups in this manner is NOT supported by Microsoft at this time.

February 3, 2011

We are currently migrating from a single server Exchange 2003 setup to multi server Exchange 2007 SP3 system. When using the EMC 2007 application, selecting the servers in either the Organisation or Server Configuration displayed an error which started as such;

Warning:
Extended protection has not been enabled. Install the operating system update specified in KB968389 onto server “servernamehere” and try again.

Now that KB does not relate to Windows 2008 R2, so it can’t be applied.

The fix is to add a registry entry that sets the RPC port for the Application Host Administration (AHADMIN) which is used by EMC, then allow that port through the Windows firewall.

To add the registry entry requires a change to the permissions of the registry key. Only the TrustedInstaller process has read\write permissions, so you need to take ownership of teh key in question, then change the permissions that would allow you to make the change. In my case, as I was a local admin on the server in question, I added the local Administrator group. Once the permissions are change you perform the following;

We currently use Veeam Backup and Replication 4.1.2 to replicate our VMWare vSphere 4.1 VM’s to our second data centre. We then backup those replicas to another location so that they can be backed up to tape. The Veeam replications take advantage of the Changed Block Tracking that is available in VMWare vSphere 4 and greater, which only backs up the blocks of the VM disks that have changed. This means we can back up a 500GB VM in around 15 minutes.

We also recently upgraded our production iSCSI SANs to newer Hitachi AMS 2100’s from our legacy Hitachi WMS100’s. Now rather than just throw out 2 x 14TB SANs (28 x 500GB SATA drives), I figured that I could use them as the replication repositories, leaving the newer AMS2100s to operate as the production VM Storage. So I reconfigured the WMS100s with two 5.4TB RAID 6 (12 disk + 2 Parity) arrays, each with 2 x 2TB (-64KB! Because of VMWare limitations) and 1 x 1.35TB LUNs. This gives us a total of 10.7TB of medium\low performing disk storage.

I had been replicating using Veeam to LUNs on the AMS2100s, so I needed a way to move the replicated data to the new storage, without affecting the CBT. If you use Storage VMotion, it rolls up the Replica Snapshot info so you lose the CBT, which will make the replication job fail and you will have to perform a new Full Replication of the VM.

Luckily, the Veeam tool gives us gives us the answer, at least in part. Here is the process;

1. In your Veeam client, ensure that the replica job is not currently running. If it is not, disable the replica job so that one does not fire off whilst you are undertaking this procedure.

2. In the vCenter, review the existing replica details, particularly, the VMs name (including whatever additions to the name you’ve used to identify it as a replica, Veeam by default tags _replica on the end of the VM name) the Host it resides on, and the datastore it currently resides on. Note that currently I believe that Veeam requires the replicas to have all of the vmdks of the VM in the one directory, but I need to confirm this. Either way, in our setup each VM has all of its vmdks in a single directory.

3. Note the Datastore that you want to move it to, ensuring that you have enough current space for the replica, plus allowing for growth depending on your growth patterns and number of replicas rollback points you keep.

4. Using the VMWare Datastore Browser tool, open the datastore that the replica is on.

5. Back in vCenter, Right Mouse Click on the VM Replica, and Remove it from Inventory.

6. Back to the Datastore Browser. If there is not a VeeamBackup directory already on your destination datastore, you can just move (or copy and paste) the VeeamBackup directory as this will relocate all subdirectories below it. However, if a VeeamBackup directory already exists on your destination, you need to move the replica VM folders themselves.

7. Wait… This move\copy can take some time, depending on your SAN load\performance etc. I moved a 1TB VM, and it took 24 hours, due to the LUN raid being created and formatted at the same time. Best to have the storage online when first creating the replication job!

8. Once the copy\move is complete, within the Datastore Browser Root\VeeamBackup\VMName(vm-number)\ find the vmname.vmx file, right mouse click on it and select Add to Inventory

9. Run through the Add to Inventory wizard, remembering to name the VM exactly the same as it was previously, e.g. VMName_replica

10. Once it is in the inventory, note the host that the VM is now residing on as if you are adding to a cluster, it will not end up on the same host as it was previously.

11. Back to the Veeam Backup and Replication console. Find the replication job, edit the properties, and change the target to the new host and datastore. Note that Veeam will alert to be sure that you have done the process above, but in a much briefer method.

12. Finish up the properties changes, and that is it. It will now replicate the VM to the new datastore, and it will maintain the index of Changed Block Tracking so a full replication is not required at next backup.

September 9, 2010

Last week, a colleague required a list of all active users from our Active Directory. This is relatively easy to do using the CSVDE tool available from Microsoft. The only caveat is that the AD needs to be kept up to date with the correct information. Garbage In, Garbage Out!

If you do a straight dump, using CSVDE –f c:\ADExport.csv you get everything, a list of all objects in your AD. Groups, Computers, Containers, foreignSecurityPrincipal, publicFolder, etc. EVERYTHING.

Note that the switch –f is to name the file and where you and it exported to.

So, you need to filter out the dump to show only users. Plus, we only want users, who are actually Users, so no Service Accounts, Builtin accounts, or Contacts. We need to filter on the following two AD attributes

objectClass=user ; if you only filter on this it dumps both user and

objectCategory=person ; if you only filter on this, it dumps users and

You can filter rows using a –r switch, and to fine tune filtering on both object type, it would be

Additionally, when you do the dump, you get all of the AD attributes, DN, cn, objectGUID, objectClass etc. all across the top of the csv file. Good for referring to if you don`t know what exact attributes you want, but makes the csv file very unwieldy with A LOT of unnecessary information for the task at hand.

If you have your Active Directory laid out in a sensible manner, you can also target the export to specific OUs and their subOUs. To do this, you use the –d filter, specifying the DN of the OU you need to target, as follows.

Now my colleague needed to know the users Distinguished Name, Display Name, account name, their internal phone extension number, their external telephone number, and their email address. Matching each of these up to the AD attributes can be done fairly simply by finding the relevant field in the straight dump you did previously.

In our case, we needed the following fields:

DN cn sAMAccountName mail telephoneNumber ipPhone

To do this, with use the –l (lowercase L) switch. CSVDE is supposed to be case insensitive, but this is what is listed in the help.

-l “sAMAccountName,cn,telephoneNumber,name,ipPhone,mail”

The order of filters is not important, as the dump will list the order that it comes out from AD. Additionally, DN will always be the first column.

September 2, 2010

Update 05/04/2011: This process has also been tested and works on Windows 2003 R2 x64!

Currently trying to get Quest MessageStats 6.8 installed onto a Windows 2008 R2 server. The catch is we are currently using Exchange 2003, and I hadn’t seen anything that would allow Exchange System Manager 2003 on a W2K8R2 server. So, I like a challenge…..

Disclaimer: I’m doing this on a server that is expendable, in a test lab. I’m not responsible if you destroy your production environment by following the instructions below.

Here is the process

Install the W2K8R2 RSAT ADDS tools and II6 compatibility. This will require a reboot, which is forced by the installation of the ADDS tools.

Download and install: Exchange System Manager for Windows Vista found here.Now this can be a bit tricky, as the ESM for Vista does an OS detection and stops you from installing it. Luckily, there is a way around it. Extract the .msi file from the download, open a command prompt as Administrator, change to the location of the .msi file and execute the following:

AD CS Advanced Lab Scenario

The following sections describe how you can set up a lab to evaluate more features of AD CS than in the basic lab setup.

Steps for Setting Up an Advanced Lab

To test additional features of AD CS in a lab environment, you will need five computers running Windows Server 2008 and one client computer running Windows Vista. The computers for this guide are named as follows:

TEST_DC1: This computer will be the domain controller for your test environment.

TEST_CA_ROOT1: This computer will host a stand-alone root CA for the test environment.

TEST_CA_ISSUE1: This enterprise CA will be subordinate to TEST_CA_ROOT1 and issue client certificates for the Online Responder and client computers.

TEST_NDES. This server will host the Network Device Enrollment Service that makes it possible to issue and manage certificates for routers and other network devices.

TEST_CLI1: This client computer running Windows Vista will autoenroll for certificates from TEST_CA_ISSUE1 and verify certificate status from TEST_ORS1.

To configure the advanced lab setup for AD CS, you need to complete the following prerequisite steps:

Set up a domain controller on TEST_DC1 for contoso.com, including some OUs to contain one or more users for TEST_CLI1, client computers in the domain, and for the servers hosting CAs and Online Responders.

Install Windows Server 2008 on the other servers in the test configuration and join them to the domain.

Install Windows Vista on TEST_CLI1, and join TEST_CLI1 to contoso.com.

After you have completed these preliminary setup procedures, you can begin to complete the following steps:

Step 1: Setting Up the Stand-Alone Root CA

A stand-alone root CA is the anchor of trust for the basic lab setup. It will be used to issue certificates to the subordinate issuing CA. Because it is critical to the security of the public key infrastructure (PKI), this CA is online in many PKIs only when needed to issue certificates to subordinate CAs.

To set up a stand-alone root CA

Log on to TEST_CA_ROOT1 as an administrator.

Start the Add Roles Wizard. On the Select Server Roles page, select the Active Directory Certificate Services check box, and then click Next two times.

On the Select Role Services page, select the Certification Authority check box, and then click Next.

On the Specify Setup Type page, click Standalone, and then click Next.

On the Specify CA Type page, click Root CA, and then click Next.

On the Set Up Private Key and Configure Cryptography for CA pages, you can configure optional settings, including cryptographic service providers. However, for basic testing purposes, accept the default values by clicking Next twice.

In the Common name for this CA box, type the common name of the CA, RootCA1, and then click Next.

On the Set the Certificate Validity Period page, accept the default validity duration for the root CA, and then click Next.

On the Configure Certificate Database page, accept the default values or specify other storage locations for the certificate database and the certificate database log, and then click Next.

After verifying the information on the Confirm Installation Options page, click Install.

Step 2: Setting Up the Enterprise Subordinate Issuing CA

Most organizations use at least one subordinate CA to protect the root CA from unnecessary exposure. An enterprise CA also allows you to use certificate templates and to use AD DS for enrollment and publishing certificates.

To set up an enterprise subordinate issuing CA

Log on to TEST_CA_ISSUE1 as a domain administrator.

Start the Add Roles Wizard. On the Select Server Roles page, select the Active Directory Certificate Services check box, and then click Next two times.

On the Select Role Services page, select the Certification Authority check box, and then click Next.

On the Specify Setup Type page, click Enterprise, and then click Next.

On the Specify CA Type page, click Subordinate CA, and then click Next.

On the Set Up Private Key and Configure Cryptography for CA pages, you can configure optional settings, including cryptographic service providers. However, for basic testing purposes, accept the default values by clicking Next twice.

On the Request Certificate page, browse to locate TEST_CA_ROOT1, or if, the root CA is not connected to the network, save the certificate request to a file so that it can be processed later. Click Next.

The subordinate CA setup will not be usable until it has been issued a root CA certificate and this certificate has been used to complete the installation of the subordinate CA.

In the Common name for this CA box, type the common name of the CA, TEST_CA_ISSUE1.

On the Set the Certificate Validity Period page, accept the default validity duration for the CA, and then click Next.

On the Configure Certificate Database page, accept the default values or specify other storage locations for the certificate database and the certificate database log, and then click Next.

After verifying the information on the Confirm Installation Options page, click Install.

Step 3: Installing and Configuring the Online Responder

An Online Responder can be installed on any computer running Windows Server 2008 Enterprise or Windows Server 2008 Datacenter. The certificate revocation data can come from a CA on a computer running Windows Server 2008, a CA on a computer running Windows Server 2003, or from a non-Microsoft CA. An Online Responder will typically not be installed on the same computer as a CA.

Note

IIS must also be installed on this computer before the Online Responder can be installed. As part of the setup process a virtual directory named OCSP is created in IIS and the Web proxy is registered as an Internet Server Application Programming Interface (ISAPI) extension.

To install the Online Responder service

Log on to TEST_ORS1 as an administrator.

Start the Add Roles Wizard. On the Select Server Roles page, select the Active Directory Certificate Services check box, and then click Next two times.

On the Select Role Services page, clear the Certification Authority check box, select the Online Responder check box, and then click Next.

You are prompted to install IIS and Windows Activation Service.

Click Add Required Role Services, and then click Next three times.

On the Confirm Installation Options page, click Install.

When the installation is complete, review the status page to verify that the installation was successful.

As with any certificate template, the OCSP Response Signing template must be configured with the enrollment permissions for Read, Enroll, Autoenroll, and Write before any certificates can be issued based on the template.

Click the Security tab. Under Group or user name, click Add, and type the name or browse to select the computer hosting the Online Responder service.

Click the computer name, TEST_ORS1, and in the Permissions dialog box, select the Read and Autoenroll check boxes.

While you have the Certificate Templates snap-in open, you can configure certificate templates for users and computers by substituting the desired templates in step 3, and repeating steps 4 through 7 to configure permissions for TEST_CLI1 and your test user accounts.

Step 5: Configuring the Authority Information Access Extension to Support the Online Responder

You need to configure the CAs to include the URL for the Online Responder as part of the authority information access extension of the issued certificate. This URL is used by the Online Responder client to validate the certificate status.

To configure the authority information access extension to support the Online Responder

Select the Include in the online certificate status protocol (OCSP) extension check box, and click OK.

Specify the locations from which users can obtain certificate revocation data; for this setup, the location is http://TEST_ORS1/ocsp.

In the console tree of the Certification Authority snap-in, right-click Certificate Templates, and then click New Certificate Templates to Issue.

In Enable Certificate Templates, select the OCSP Response Signing template and any other certificate templates that you configured previously, and then click OK.

Open Certificate Templates, and verify that the modified certificate templates appear in the list.

Step 6: Assigning the OCSP Response Signing Template to a CA

Once the templates are properly configured, the CA needs to be configured to issue that template.

To configure the CA to issue certificates based on the newly created OCSP Response Signing template

Open the Certification Authority snap-in.

Right-click Certificate Templates, and then click Certificate Template to Issue.

Select the OCSP Response Signing_2 template from the list of available templates, and then click OK.

Step 7: Enrolling for an OCSP Response Signing Certificate

Enrollment might not take place right away. Therefore, before you proceed to the next step, confirm that certificate enrollment has taken place so that a signing certificate exists on the computer, and verify that the permissions on the signing certificate allow the Online Responder to use it.

To verify that the signing certificate is properly configured

Start or restart TEST_ORS1 to enroll for the certificates.

Log on as a CA administrator.

Open the Certificates snap-in for the computer. Open the Personal certificate store for the computer, and then verify that it contains a certificate titled OCSP Response Signing_2.

Right-click this certificate, and then click Manage Private Keys.

Click the Security tab. In the User Group or user name dialog box, click Add to type in and add Network Service to the Group or user name list, and then click OK.

Click Network Service, and in the Permissions dialog box, select the Full Control check box. Click OK twice.

Step 8: Creating a Revocation Configuration

Creating a revocation configuration involves the following tasks:

Identify the CA certificate for the CA that supports the Online Responder.

Identify the CRL distribution point for the CA.

Select a signing certificate that will be used to sign revocation status responses.

Select a revocation provider, the component responsible for retrieving and caching the revocation information used by the Online Responder.

To create a revocation configuration

Log on to TEST_ORS1 as a domain administrator.

Open the Online Responder snap-in.

In the Actions pane, click Add Revocation Configuration to start the Add Revocation Configuration wizard, and then click Next.

On the Name the Revocation Configuration page, type a name for the revocation configuration, such as TEST_RC1, and then click Next.

On the Select CA Certificate Location page, click Select a certificate for an existing enterprise CA, and then click Next.

On the following page, the name of the CA, TEST_CA_ISSUE1, should appear in the Browse CA certificates published in Active Directory box.

If it appears, click the name of the CA that you want to associate with your revocation configuration, and then click Next.

If it does not appear, click Browser for a CA by Computer name and type the name of the computer hosting TEST_CA_ISSUE1 or click Browse to locate this computer. When you have located the computer, click Next.

Note

You can also select the CA certificate from the local certificate store or import it from removable media in step 5.

View the certificate and copy the CRL distribution point for the parent root CA, RootCA1. To do this:

Open the Certificate Services snap-in, and then select an issued certificate.

Double-click the certificate, and then click the Details tab.

Scroll down and select the CRL Distribution Points field.

Select and copy the URL for the CRL distribution point that you want to use.

On the Revocation Provider Properties page, click Add, enter the URL of the CRL distribution point, and then click OK.

Click Finish.

Using the Online Responder snap-in, select the revocation configuration, and then examine the status information to verify that it is functioning properly. You should also be able to examine the properties of the signing certificate to verify that the Online Responder is configured properly.

Step 9: Setting Up and Configuring the Network Device Enrollment Service

The Network Device Enrollment Service allows software on routers and other network devices running without domain credentials to obtain certificates.

The Network Device Enrollment Service operates as an ISAPI filter on IIS that performs the following functions:

SCEP was developed as an extension to existing HTTP, PKCS #10, PKCS #7, RFC 2459, and other standards to enable network device and application certificate enrollment with CAs. SCEP is identified and documented on the Internet Engineering Task Force Web site (http://go.microsoft.com/fwlink/?LinkId=71055).

Before you begin this procedure, create a user ndes_user1 and add this user to the IIS user group. Then, use the Certificate Templates snap-in to configure Read and Enroll permissions for this user on the IPSEC (Offline Request) certificate template.

To set up and configure the Network Device Enrollment Service

Log on to TEST_NDES as an enterprise administrator.

Start the Add Roles Wizard. On the Select Server Roles page, select the Active Directory Certificate Services check box, and then click Next two times.

On the Select Role Services page, clear the Certification Authority check box, and then select Network Device Enrollment Service.

You are prompted to install IIS and Windows Activation Service.

Click Add Required Role Services, and then click Next three times.

On the Confirm Installation Options page, click Install.

When the installation is complete, review the status page to verify that the installation was successful.

Because this is a new installation and there are no pending SCEP certificate requests, click Replace existing Registration Authority (RA) certificates, and then click Next.

When the Network Device Enrollment Service is installed on a computer where a registration authority already exists, the existing registration authority and any pending certificate requests are deleted.

On the Specify User Account page, click Select User, and type the user name ndes_user1 and password for this account, which the Network Device Enrollment Service will use to authorize certificate requests. Click OK, and then click Next.

On the Specify CA page, select either the CA name or Computer name check box, click Browse to locate the CA that will issue the Network Device Enrollment Service certificates, TEST_CA_ISSUE1, and then click Next.

On the Specify Registry Authority Information page, type ndes_1 in the RA name box. Under Country/region, select the check box for the country/region you are in, and then click Next.

On the Configure Cryptography page, accept the default values for the signature and encryption keys, and then click Next.

You can verify the setup steps described previously as you perform them.

After the installation is complete, you should verify that your advanced test setup is functioning properly.

To verify that the advanced AD CS test setup functions properly

On the CA, configure several certificate templates to autoenroll certificates for TEST_CLI1 and users on this computer.

When information about the new certificates has been published to AD DS, open a command prompt on the client computer and enter the following command to start certificate autoenrollment:

certutil -pulse

On the client computer, use the Certificates snap-in to verify that the certificates have been issued to the user and to the computer, as appropriate.

On the CA, use the Certification Authority snap-in to view and revoke one or more of the issued certificates by clicking Certification Authority (Computer)/CA name/Issued Certificates and selecting the certificate you want to revoke. On the Action menu, point to All Tasks, and then click Revoke Certificate. Select the reason for revoking the certificate, and click Yes.

In the Certification Authority snap-in, publish a new CRL by clicking Certification Authority (Computer)/CA name/Revoked Certificates in the console tree. Then, on the Action menu, point to All Tasks, and click Publish.

On the client computer, use the Certificates snap-in to export one of the issued certificates and save it as an X.509 file.

Open a command prompt, and enter the following command:

certutil –url <exportedcert.cer>

In the Verify and Retrieve dialog box, click OCSP (from AIA), and then click Retrieve. After the CRL is retrieved, the status will display Verified.

I’m forever finding things that will help me in my day to day job of designing and managing IT infrastructure, noting them, then forgetting about them. This doesn’t bode well for my plans for World Domination, and I also needed a way to share pictures, videos, links etc with my counterparts in the IT industry, and friends around the world. What better way than to get on the Web 2.0 bandwagon with a blog.

Where did the name come from? I’ve been known to exude a level of confidence from time to time, and over a few beers at a conference, this got turned into a Field of Awesome, and I became the International Man of Awesome.

The DRAC (Dell Remote Access Controller) is an interface card by Dell which provides out-of-band management. The controller has its own processor, memory, battery, network connection, and access to the system bus. Key features include power management, virtual media access and remote console, all available through a supported web browser. This gives system administrators the ability to configure a machine as if they were sitting at the local console (terminal).

The DRAC card has several services bound on its dedicated IP; here is the list of the default ports and their usage: