Tuesday, December 7, 2010

I recently had the dubious “pleasure” of having to install and integrate Websense with an ageing ISA 2006 Server. After lots of trials and tribulations (then it works, then it doesn’t, or it appears to work but doesn’t filter etc. etc.) I finally had to throw in the towel and call Websense support. Here is what I had to do to get it working:

Install Websense (with Filtering module) on the ISA server, making sure to select "the “ISA Integration” option during installation

They then had me set up the following Access Rule in ISA Server: "All outbound traffic" TO AND FROM "Local Host" and "All Protected Networks"

Only after all this did it work as expected. I have not had the opportunity to test this with ISA’s successor, Microsoft’s Threat Management Gateway, but I would imagine that the procedure should remain fairly similar.

Sunday, December 5, 2010

DPM is a terrific backup solution for the all-round Microsoft shop. In my humble opinion it’s also one of the best ways to back up Hyper-V Virtual Machines. It integrates beautifully with your SAN vendors hardware VSS providers, but if you don’t have the budget to splash out on the requisite SAN licenses, there is a way to do it in software.

The Challenge

When backing up VM’s residing on a CSV, you might run into the following DPM error message:

Failed to prepare a Cluster Shared Volume (CSV) for backup as another backup using the same CSV is in progress. (ID 32612 Details: Unknown error (0x8007173d) (0x8007173D))

The Solution

Run the DSConfig.ps1 script (included below) on any node in your CSV containing cluster

This will create a file called DataSourceGroups.xml in the same folder that you executed the DSConfig.ps1 script

This file needs to be updated every time you create and protect a VM using DPM

If a protection group has already been created for the virtual machines, execute the Modify Protection Group Wizard. If a protection group has not been created, create a new protection group and the job serialization described above will take effect.

This will allow you to get rid of the pesky alerts and streamline your backup process. The optimum solution would be to use a hardware VSS provider, but this is not always financially feasible, or if you have an iSCSI SAN.

Tuesday, September 28, 2010

I was recently approached by an organization who suffered through an IT Systems and Security Audit. Needless to say the audited highlighted a couple of holes in their system, and I was tasked with implementing the audit recommendations. Part of my scope of work included securing their HP Procurve switches, this post will outline a couple of best-practices when securing your network equipment. It is by no means exhaustive, but it should provide you with a good baseline. All instructions assume you're logged into the switch and at the (config)# prompt

Defaults Configurations and Secure Alternatives

Telnet vs SSH
Telnet is an antiquated and insecure protocol; it sends all traffic across the wire in clear text. This includes user
names and passwords. Any attacker may eavesdrop on a Telnet session and obtain credentials. Below are the commands detailing how to disable Telnet and and implement SSH.

crypto key generate ssh

ip ssh

no telnet-server

HTTP vs HTTPS
ProCurve switches and routers can, by default, be configured through the web interface. The HTTP
interface has the same limitations as Telnet. It is therefore recommended that the HTTPS interface be configured and the HTTP interface be disabled, like so:

crypto key generate cert 1024

web-management ssl

no web-management plaintext

TFTP vs SFTP & SCP
TFTP client and server should be disabled as they do not require any authentication. Secure File
Transfer Protocol (SFTP) and Secure Copy Protocol (SCP) are part of the SSH protocol suite. As such provide an encrypted session using public/private keys between client and server just like
SSH. You configure it like so:

crypto key generate ssh

ip ssh filetransfer

no tftp server

no tftp client

SNMPv1/2c vs. SNMPv3
SNMP version 2 is enabled by default. SNMPv2 community names are sent across the network as clear text (similiar to Telnet). SNMP version 3 overcomes these weaknesses via encryption. Here's how to configure it:

snmpv3 enable

snmpv3 only

If, for whatever reason, you cannot run SNMP v3, it is possible to run SNMP v2 in restricted mode, i.e. provide Read-Only access to your switch:

snmp-server community restricted

It is also possible to completely disable SNMP altogether:

no snmp-server enable

Authorised IP Managers
It’s possible to restrict IP access to the switch to certain management IP's only. This is configured via the following command:

ip authorized-manager <IP address> mask

Whilst not the be-all and end-all of security, the above should go a long way to ensure a reasonably secure base line for your HP Procurve network equipment.

Tuesday, September 14, 2010

Exchange ActiveSync (EAS for us smart people) is a pretty rocking way to get Smart phones hooked up to your Exchange Server. 9 times out of ten it "just works" if you follow the docs, but sometimes it doesn't. Otherwise I wouldn't writing about it now, would I?

I had a rather perplexing case where EAS only worked for some users. On the non-working users the only error message displayed was the rather unhelpful HTTP Error 500. The only suggestion the usually excellent www.testexchangeconnectivity.com could offer was an old article applicable to Exchange 2003.

Diagnosing

The next step was to scour the Application log on the Exchange Server, and lo and behold, the following error was logged:

Righto. So it would appear that to actually use EAS, information has to be written to the user account and to do this the Exchange Servers group needs to have permissions to the user object. Further investigation revealed that the only accounts giving the error was Admin accounts.

AdminSDHolder and SDPROP are the culprits

Because the failing accounts has domain admins rights their User Object permissions were being reset by AdminSDHolder, via a process called SDPROP. I quote "Each Active Directory domain has an object called AdminSDHolder, which resides in the System container of the domain. The AdminSDHolder object has a unique Access Control List (ACL), which is used to control the permissions of security principals that are members of built-in privileged Active Directory groups (what I like to call "protected" groups). Every hour, a background process runs on the domain controller that holds the PDC Emulator operations master role. It compares the ACL on all security principals (users, groups and computer accounts) that belong to protected groups against the ACL on the AdminSDHolder object. If the size or the binary string is different, the security descriptor on the object is overwritten by the security descriptor from the AdminSDHolder object.."

Resolution

Because SDPROP runs every hour, the fix needs to be applied and a EAS sync done before an hour has gone past. Here is how:

Launch the EMC from your Exchange CAS Server

Browse to Server Configuration and select your server

Go to the System Settings tab to see which Domain Controller is in use

Launch ADUC from the DC located in step 3 and make sure Advanced View is enabled

Right-click the user account and go to Properties

Click the Security tab, then click Advanced

Check the Include inheritable permission.... checkbox

OK out of all the dialog boxes

Sync your device

The sync should be successful, providing not more than an hour has passed between the sync and step 1. Whilst this has been a very difficult issue to nail down, there is a lesson in there somewhere...DON'T USE PRIVILEGED ACCOUNTS FOR DAY-TO-DAY USE!!!

Monday, August 30, 2010

When you upgrade to vSphere 4.1 your hosts might start disconnecting from your vCenter Server with the following error message: A general system error occurred: internal error: vmodl.fault.HostCommunication. Restarting the management agents does not resolve the error, nor does rebooting the host. This VMware KB points to name resolution issues, but that is not at fault here. The issue is vCenter Server cannot manage an ESX 4.1 host.

Workaround / Solution

Currently there are two solutions available:

Upgrade your vCenter Server to version 4.1. (Once you've upgraded you'll have to remove the hosts from your inventory and re-add it - simply reconnecting didn't work in my case)

Downgrade your ESX hosts to version 4

Strangely enough I could not find this documented anywhere on the VMware Knowledge Base, even though it seems to be a pretty widely reported problem.

I was tasked with upgrading a clients vSphere installation from vSphere 4.0 to 4.1. Due to various external factors the client couldn't make use of the vSphere Update Manager, so I had to do it old-school style from the command line. Here's how to do it:

Friday, August 20, 2010

Had a fun scenario recently where a customer (a Casino!) ran out of disk space on their SQL 2005 Server. This in turn caused all their slot / video poker / whatever gambling machines to stop working. Turns out old Bill Shakespeare had it all wrong, because it turns out that Hell Hath No Fury Like A Gambler Scorned!

Long story short - their SQL log file grew to gargantuan proportions, and yours truly had to whip them back into shape. Here's how it went down!

Run the following stored procedure: "use your_db_name"

Followed by: "exec sp_helpfile". This will return the physical names and attributes of files associated with your DB. Record the DB and log filenames, without the path and extension

Enter the folowwing commands

USE your_db_name

GO

BACKUP LOG your_db_name WITH TRUNCATE_ONLY

GO

DBCC SHRINKFILE (your_dblog_filename, 1)

GO

DBCC SHRINKFILE (your_db_filename, 1)

GO

exec sp_helpfile

Step 9 should output the same info has Step 2, you can now compare the filesizes to see if the process was succesfull.

You might get an error "Cannot shrink log file because all logical log files are in use". In that case you can follow the instruction here to resolve. I've detailed the steps below, if you're too lazy to follow the link.

Open SQL Enterprise Manager

Right-click on the database you want to shrink and click Properties

from the Data Properties go to Options.

Set the Recovery Model to Simple and click OK and try to shrink the database

Thursday, August 19, 2010

Nope, I didn't fudge up the title. Veeam Backup and Replication is a wonderful product, allowing you to replicate vSphere VMs to a offsite Disaster Recovery location. When disaster strikes, it's a pretty straight-forward process to fail over to your DR site. It's what I call a forehead procedure - you only have to hit the spacebar with your forehead. Thanks - I'll be here all night!

What's not so intuitive and well documented is using Veeam Backup to move VM's to a different location, for example a Server Room / Data Center relocation *and then commiting those changes*, i.e. not failing back to Production. The below steps assume we've replicated and failed over our VM's to our DR location already.

Ruan's Step By Step Guide on using Veeam Backup to relocate VM's

Delete all Veeam VM snapshots using the vSphere Snapshot Manager

By default your DR replica will be named "VMname_replica", rename it back to its original name, i.e. VMname

Remove the VM replica from the list of replicas in the Veeam Backup and Replication Console

Delete the Production -> DR replication job responsible for replication the VM in question. Recreate it to reflect the new Source and Target locations

Delete the .vrb file from the VM datastore, as we will no longer be using these restore points

Delete the replica.vrb and running.rbk files

Pat yourself on the back - you've just done the easiest VM move you'll ever do!

Thursday, July 29, 2010

I had a "interesting" experience recently, namely had a EVA 4400 overheating due to environmental issues (fancy-talk for Aircon failure). The client phoned me, complaining that half of their Hyper-V VM's are not running. Further investigation revealed that the CSV's were offline. Hmmm, this was getting serious. I logged into Command View and saw that most of my VDisks were faulted, this was due in no small matter to the fact that all the drives in one of my shelves were faulted.

Event Logs

I had a look at the relevant EVA logs and discovered the following relevant entries:

In retrospect it was a fairly simple sequence of events, as evidenced by the entries above. The Air Conditioner failed, which caused the temperature within the Drive Shelf to rise (this is the HSV300 controller referred to in the event log). To prevent damage to itself, the drive then switched itself off, which prompted the log entry about the physical drive disappearing.

We then started seeing volumes transitioning to the missing state, i.e. our VDisks went missing. Hardly surprising considering that the drives containing them switched themselves off.

Resolution

Restored Air Conditioning (goes without saying I guess)

Powered off the EVA and all attached disk shelves

Powered on disk shelves and waited for the Numeric ID LED's at the back to display the proper IDs.

Powered up the Controller

Lo and behold! All the previously failed physical disks came on-line, meaning that my missing VDisks also made a most welcome return

Unfortunately my Hyper-V Hosts still couldn't access the Vdisks, so I had to unpresent and re-present them via Command View. I assume the EVA assigned new WWN's to the LUNs.

I re-scanned for storage from the Disk Management MMC on the Hyper-V Hosts

Brought the Disks and CSV's online via cluster manager

Started up the VM's

Conclusion

This was quite a harrowing experience, obviously. What struck me as ridiculous is that HP does not have *ANY* thermal shutdown logic / capabilities on the EVA controller itself. It keeps on trucking till the drives themselves fail, causing a very ungraceful failure of the VDisks. There is also no guarantee that your drives and VDisks will come back online. In essence - if your EVA overheats there is a distinct possibility that you lose your Data. Caveat Emptor...

Friday, July 2, 2010

Namibia is a third world....errr....developing country. So apart from rampant (55%) unemployment it also means that we are bandwidth starved. This also means that some businesses are running on capped accounts, so attempts need to be made to conserve bandwidth. I've had to move WSUS servers to new hardware or VM's a couple of times, and needless to say it's a huge time-sink and waste of bandwidth to re-download all updates every time you move your WSUS server. Thus I've come up with a way to move WSUS without downloading tons of patches - steps are outlined below for your enjoyment.

Install WSUS on your new server, making sure to select the option to use the existing Windows Internal Database

During the Choose Upstream Server of the configuration wizard, be sure to select Synchronise from another Windows Server Update Services Server

Ensure that the This is a replica of the upstream server check box is selected. This ensures that existing approvals, settings, computers and groups are maintained

Complete this Wizard

Your new replica server will synchronise with your upstream server. This is what we're talking about - no re-downloading many GB's worth of patches! N.B. Wait for this process to complete before carrying on with step 6.

Open up a CMD Prompt and navigate to "C:\Program Files\Update Services 3.0 API Samples and Tools\WsusMigrate\WsusMigrationExport" folder on your old WSUS server

Run "wsusmigrationexport.exe WSUS_Settings.xml" to export the settings. This will backup your approvals and target groups to an XML file

Transfer the WSUS_Settings.xml created above to your new WSUS server

Again navigate to "C:\Program Files\Update Services 3.0 API Samples and Tools\WsusMigrate\WsusMigrationImport" folder (on the new WSUS server). Run "wsusmigrationimport.exe WSUS_Settings.xml All None"

Review and compare settings on your two WSUS servers, ensuring that they match each other

Update the relevant GPO's to ensure clients are pointing to the new WSUS server

One last point, I've noticed that if you are not using GPO's to assign your computers to Computer Groups all your clients will get stuck into the Unassigned Computers group. You'll have to manually sort them into the appropriate groups again. Bummer, but you really should be using Computer Groups....

Monday, June 21, 2010

You know when you do something and you think to yourself "Oi, I should really document this"? And then you go "Nahhh, this is easy peasy lemon squeezy - I won't forget" and a month later you have to repeat the task and wonder of wonders, you did forget. Well Goddammit - never will I have to wonder again how to create a LUN on a HP EVA 4400 SAN and use said LUN (or Vdisk in vHP's vVirtual vSpeak vWhere vEverything vStarts vWith a vStupid vV). vWhere...er I mean where was I. Ahh yes, lets create a vDisk...

Create a vDisk on the EVA SAN

Log into Command View (what a cool name, sounds like you can bomb some backwater country into the Stone Age with a mere misplaced mouse-click)

Expand your Virtual Disks Folder

Click Create vDisk

Name your disk, then give it a size and RAID level

Click Create vDisk

Present the vDisk to your Hosts

Navigate to your newly created vDisk and click the Presentation tab

Click Present and select all hosts to present the vDisk to. For some obscure reason every 4400 I've worked on would only allow me to present to four hosts / HBA's at a time, anything more and it errors out

Initialise and format the vDisk

Open the Disk Management MMC on one of your SAN-connected Hyper-V hosts

Right-click the Disk Management node and click Rescan on the context menu

Right-click on the newly appeared unallocated disk and select Bring Online

Friday, June 4, 2010

I find myself having to explain the concepts behind VLAN's a lot, so I've decided to do a blog post about it.

The Wikipedia article defines a VLAN as: "a group of hosts with a common set of requirements that communicate as if they were attached to the same broadcast domain, regardless of their physical location. A VLAN has the same attributes as a physical LAN, but it allows for end stations to be grouped together even if they are not located on the same network switch. Network reconfiguration can be done through software instead of physically relocating devices".

PROTOCOL CONCEPTS

We have regular "untagged" Ethernet frames. ("Frame" is the data-link (layer 2) term for what some protocols calls a "packet" or "datagram".) These are the same Ethernet frames from before VLANs were invented.

Then we have "tagged" frames. These take a regular Ethernet frame and add a marker -- the "tag". The tag itself contains the VLAN number (along with a priority code). Thus, a tagged frame is associated with a particular VLAN number.

Inside the frame, the tag is basically just a reserved Ethernet frame type, followed by the tag information, followed by the rest of the original frame (complete with its original frame type). So to network nodes which are *not* VLAN aware, a tagged frame looks just like a regular Ethernet frame, but with an "unknown" frame type.

For computers, a VLAN-aware network stack will usually turn each VLAN into a separate network interface, as if each VLAN was a different network card with a different cable. The "stock" network stack included with Win 2000/XP does *not* include VLAN capability, so they can only handle untagged frames. Tagged frames will generally be silently ignored. Third-party software (often included with high-end network cards) can add VLAN capability. I'm not sure about 2003/Vista/2008.

Switches which are not VLAN-aware, along with any repeaters, will treat tagged frames exactly like they do any other frame.

SWITCH CONCEPTS

Switches which are VLAN-aware generally let you control which ports are associated with which VLAN(s). The terminology, commands, etc., will vary with manufacturer. I'm most familiar with the HP ProCurve managed witches, so I'll speak mostly to that. The concepts should carry over to other manufacturers.

With HP, you can tell the switch that each VLAN is tagged, untagged, or not available for each port. Internally, frames are *always* associated a VLAN number. The configuration just tells the switch which VLANs go with which ports, and which VLAN number to use for untagged frames.

When the switch receives an untagged frame, it needs to know what VLAN to associate it with. If the port has an untagged VLAN associated it, the fame will go to that VLAN. If there is no untagged VLAN associated with the port, the frame is dropped. This is why a port can have (at most) one untagged VLAN associated with it -- how could the switch know which one to use otherwise?

When the switch receives a tagged frame, it simply checks to see if that VLAN is associated with the port. If so, it accepts it; if not, it drops it. This is basically just an access control mechanism

When the switch has a frame to *transmit*, it only considers ports which are associated with the frame's VLAN. If a port is configured as tagged for the frame's VLAN, it transmits the frame with the VLAN tag. If a port is configured as untagged for the VLAN, the frame is transmitted on that port without a VLAN tag.

Real-World Implementations

Okay, enough conceptual stuff, how do we use it? Below I'll describe a implementation I had to do for a client recently.

VLAN's were numbered and split up as follows -

1 – Reserved

2 – Main company private network

3 – Guest network

4 – ISP uplink

5 – IP telephony

I used separate VLANs for security, manageability, and robustness. I don't want guests to be able to probe/sniff/screw-up the corporate network. I didn't want the cable modem link (which is full of garbage traffic from other subscribers) on the main LAN. I want the phones to have higher priority over regular data, and to have their own DHCP server, so if the "data side" of the house goes down, they still have phones.

I don't, as a rule, use VLAN 1 because that's the default on most things. This also avoids confusion.

VLAN 2 is where most traffic is in this scenario. Most ports are configured as untagged for VLAN 2. The computers, printers, etc., plugged into those ports aren't even aware they are on a VLAN. All the employee workstations are like this, as are most servers. The syntax to configure this is easy. For example, if I want port 1 to be untagged for VLAN 2 like this:

vlan 2 untagged 1

VLAN 3 is for guests (visitors). For obvious reasons, I don't let them plug in to the main network. They get a separate IP subnet, a separate DHCP server, and bypass much of the network filtering. Again, all untagged, and connected nodes are not aware of the VLAN.

VLAN 4 was created because the client has a ADSL Internet feed they use for employee web browsing. Circumstances meant the modem had to go in a different building from the main server/network room. So the cable modem plugs into a particular switch port, which uses VLAN 4 to carry it to the firewall in a different building. Again, untagged, unaware.

VLAN 5 is for premises voice-over-IP. The phones include a built-in switch for "daisy chain", and are VLAN-aware. Switch ports connected to phones are tagged for VLAN 5 and untagged for VLAN 2, so you can plug a PC (not VLAN aware) into the phone. Traffic is kept separate, and voice traffic is given priority. The central phone equipment has an Ethernet port which we connect to one of the core switches, with that switch port untagged for VLAN 5. So the central equipment just thinks it is talking to a regular LAN.

For example, suppose port 20 was the central phone equipment, and port 7 had a VoIP phone. We would do:

vlan 2 untagged 7

vlan 5 tagged 7

vlan 5 untagged 20

For links between switches, each VLAN is configured as *tagged* for those ports, with *no* untagged VLAN. Those ports are only used to carry traffic between VLAN-aware switches. (Cisco would call this a "trunk port", IIRC.) The HP syntax for (e.g.) port 24 as a switch link would be:

vlan 2 tagged 24

vlan 3 tagged 24

vlan 4 tagged 24

vlan 5 tagged 24

Finally, our firewall, which *is* VLAN-aware, is configured as tagged for all VLANs on its port, with no untagged VLAN. Virtual network interfaces are configured in the firewall OS, and all the IP configuration and filtering rules are done with those virtual interfaces, not the physical interface. This also means our firewall only has one network cable plugged into it. Configures the same as a switch link, really.

Monday, May 31, 2010

Dammit - me and my lack of documentation. Worked with Hyper-V for the first time six months ago (I'm predominantly a VMware dude). I Had to add a node to a failover cluster this morning, and found myself wishing I had documented it better the first time around. Got it sorted after a bit of scratching around; as you'll see below I've also finally gotten round to documenting the process. The below assumes that you've hooked it presented appropriate the appropriate LUNs on the SAN etc. etc....

Install the OS (no sh!t Sherlock). Make sure you run the same edition on all nodes. Also installed any required MPIO software that may be required by your SAN.

Join the server to the same domain that your other Cluster nodes are members of

Enable and configure Windows Updates

Make sure that your network adapter naming convention is the same as on the other nodes. This will make management easier

Install the Hyper-V role, reboot when prompted

Make sure to have naming consistency in your Virtual network. In other words, exactly the same name and capitalization

Install the Failover Clustering feature, reboot when prompted

Physically connect the server to your SAN

Join the node to the cluster. This can be done via the Failover Cluster Management MMC: Right click your cluster name, select Add Node, enter the server name and click OK. The command line alternative is "cluster.exe /cluster:Cluster_Name /add /node:Node_Name"

This is a fairly basic task that needs to be done fairly frequently, but I've never documented it anywhere so it always takes me 10 minutes longer than what it should. The client where I had to do this today happens to run a HP EVA 4400 Fiber SAN, so I will include the instructions for the EVA side of things as well. Without further ado...

Expanding a existing LUN on an HP EVA 4400 SAN

Log into Command View (HP's delightfully named SAN configuration solution)

Navigate to the the Virtual Disks folder, more specifically to the Vdisk you want to expand

Click the Vdisk, and wait 10 seconds for the incredibly laggy Command View interface to catch up

Use Cluster Resource Manager to determine who owns the Disk Resource you wish to expand

Once identified, go to "Disk Management" on the owner

Right-click the Disk Management node, and select "Rescan Disks". The un-allocated space should show up now

Right-click the volume you wish to expand and select "Extend Volume". Click Next 2 times and finally Finish

Fire up SCVMM and shut down the VM whose VHD you want to expand

Click the "Hardware Configuration" tab, and select the Disk you want to expand

Check the "Expand this disk" checkbox and enter the desired size

Click OK

This should take care expanding the VHD, but this does not mean the underlying OS will automatically make use of the added capacity, no sirrreeee. In my case the VM was running Windows 2008 Server R2, and this is what I had to do....

Monday, March 29, 2010

Hi Kids, todays post is brought to you by the letter J. I previously mentioned a vSphere deployment I had to do, connecting to a EMC AX4-5i iSCSI SAN. Once I got the storage hooked up to my ESX hosts I of course wanted to enable jumbo frames.

Jumbo Frames, for the uninitiated heathens out there, is basically a Ethernet frame with a payload of more than 1500 bytes, up to a typical maximum of 9000 bytes per frame. Why would I want to do this? Performance, in a nutshell. The only requirement would be that your switch supports Jumbo Frames, which I believe most, if not all, mid to high-end kit does. So let's get down to business, shall we?

Switch ConfigurationI'm fairly proficient with Cisco and HP networking gear, so I'll give a quick rundown of the commands needed to enable jumbo frames on their kit:

Cisco: Go into conf t mode and enter the following command "system mtu jumbo 9000". Once that is done issue a "reload" command to reboot your switch

HP Procurve: On these babies Jumbo frames need to be enabled per VLAN. Execute the following command: "vlan # jumbo"

That takes care of the networking side of things, let's move on to the ESX...

Configure Jumbo frames in vSphere

Log on to your ESX host using your favorite SSH client

Change your chosen vSwitch MTU with the following command "esxcfg-vswitch -m 9000 vSwitch#" Replace vSwitch# with the name of the vSwitch you want to modify

Seeing as one cannot change an existing VMKernel port MTU, you will either have to remove and recreate your existing VMKernel port, or create a new one. To delete an existing VMKernel port, enter the following command: "esxcfg-vmknic -d -p VMKernelport"

I have thought of merging this post with the iSCSI Multipathing post, but seeing as not all environments can/will support end to end Jumbo frames, I have decided to keep them seperate for the time being.

Friday, March 26, 2010

Look, I love Exchange 2010. As a matter of fact I'm pretty sure Bono is busy installing it somewhere in a attempt to cure World Hunger. It's like caviar, dipped in celery salt and served from Uma Thurman's belly button...but geez Louise, it can be a frustrating beast at time as well, sometimes through no fault of it's own, as we'll see below.

There my Exchange 2010 was, humming along nicely, I already moved the mailboxes across, configured Autodiscover, Outlook Anywhere, ActiveSync, all those highly visible, nice to haves. I created a test account, tested all the remote services, and it worked beeeautifully! So with a flourish I sent a mail to the client, giving him the go-ahead to roll out the remote / mobile services to his users. Only to hear back from them 10 minutes later that bugger-all remote services works. Godammit. No OA, no AS, no OWA, nada, zip. After wiping the egg from my face, I started troubleshooting. First instinct was that I somehow fudged my ISA Publishing rules, but no, it was all good. I then headed off to the bestestExchange Connectivity Troubleshooter in the history of the world. I ran a test or two and it told me in no uncertain terms that:Attempting to ping RPC Endpoint 6001 (Exchange Information Store) on server exc.clientabc.local Failed to ping Endpoint Tell me more about this issue and how to resolve it Additional Details RPC_E_ACCESS_DENIED error (0x5) was thrown by the RPCRuntime

Now the interesting thing here was the fact that it was only doing this for existing users, newly created ones worked without a hitch. I had a antacid (for some reason troubleshooting permission issues on IIS gets to me) and got to work. Suffice to say that I compared IIS permissions, folder NTFS permissions, even user properties in ADSI Edit. I ensured the new users had exactly the same group membership as the existing ones, everything. Then my eye caught something in the security log, All existing users, when logging on via any remote services, threw a EventID 534 in the Security Log; The user has not been granted the requested logon type on this computer.

I was making progress! a Quick review of the existing users account properties revealed that users were only allowed to log onto their workstations. The new test accounts that I created had no such restriction of course. Once I removed these restrictions from the user accounts everything worked flawlessly!

P.S. I tried granulizing (is that even a word?) the logon restrictions by allowing the users to logon to the Exchange 2010 Server. This fixed the OWA, but OA and AS was still not working. The only way I could get OA / AS to work was to remove all logon restrictions.

As promised, I will outline some of the issues I had with a recent Exchange 2010 migration. After Installing the required UCC and intermediary certificates on both the Exchange and ISA 2006 SP1 servers, following the GoDaddy instructions here, I ran into some issues. Namely ActiveSync refused to work. I headed of the best Exchange Connectivity Troubleshooting site EVAR! This gave me the following detailed info:

After some serious troubleshooting, googling etc, I still didn't have the issue fixed. As a last resort I rebooted (thought those days were gone) the ISA server. Lo and behold it worked, but that's four hours of my life I'm not getting back anytime soon. What's weird about this is that it's the first time I've had to reboot a ISA server to get a certificate to "take". Oh well...

Thursday, March 25, 2010

There I was, setting up a small DR site for a client, hardware was a blinged up ProliantDL380, connecting to a sweet, sweet little EMC AX4-5i SAN. This being my first non-FC SAN, I obviously investigated the multipathing options. I was kind of (no, really!) taken aback when I discovered that it's not so straightforward to set up. What's a boy to do? I took a deep breath, bowed in the direction of the (not-so-sweet-anymore) AX4-5i SAN and unleashed the fury of my Google-Fu! Hee-Ya! About 3 hours later the fiendish ESX 4 and a AX4-5i had to yield to the fury of my touch typing - below I chronicle it for all eternity.

Setting up MultipathingThe DL380's I worked on here had an additional 4 port NIC installed, for a total of 8 1Gb ports. The 4 on-board ports I dedicated to the Service console, vmkerneletal, standard stuff. The 4 ports on the add-on NIC I used for iSCSI, perfect considering the AX4-5i also has 4 iSCSI ports. What I did then was set up a new vSwitch, added my 4 Nics to the vSwitch and then created 4 seperateVmKernel ports, so that we have a one-to-one mapping between the VMkernels and the NICs. Here's how to do that:

Now we create VMkernel ports for all our newly added adapters, like so: Go vSwitch - Properties and click the Ports tab and click Add. Select VMkernel, give it a nice label and a suitable IP address and click Finish. Create a VMkernel port for every adapter assigned to your iSCSIvSwitch.

Now the important bit, we're going to do the one-to-one VMkernel to NIC mapping: On the ports tab, select one of your newVMkernel port and click Edit.

Select the Nic Teaming Tab and check the Override vSwitchfailover order

Make sure only one adapter is active, move all the other NIC's down to the Unused Adapters section

Rinse and repeat for every VMkernel port on your vSwitch

Alas, we're not done yet - now I'm going to introduce you to my little friend esxcli. This fun little guy will enable us to connect our newly created VMkernel ports to the ESXiSCSI initiator

So, I recently completed a Exchange 2003 to Exchange 2010 migration for a mid-sized client. There were a (of course) couple of hiccups, which I will detail as soon as I get my stuff in order. With that said, here's what I learned about UCC certificates (more specifically GoDaddyUCC Certs).

THE BASICS:

This client had 2 separateDNSnamespaces, the AD DNS was clientabc.local, the external DNS was clientabc.com. Internally the Exchange server was called cabc-exc-001, externally it was mail.clientabc.com. So let's get down to the 3 commandments.

Any name by which your server will be accessed needs to be included on the certificate. In my case it was the following: mail.clientabc.com, cabc-exc-001.clientabc.local and finally clientabc.com

Make the common name the server's external DNS alias, eg. mail.clientabc.com

If you use the Autodiscover server (which you should, it RAWKS) you should add that to your UCC certificate. In my case: autodiscover.clientabc.com and autodiscover.clientabc.local

Generating the Certificate Request:

Fire up your EMC and click "Manage Databases" on the homepage

Click "Server Configuration", then click on "New Exchange Certificate" in the actions pane

You'll be prompted for a "Friendly Name". This is purely descriptive, so call it something descriptive.

On the "Domain Scope" dialog, do not select the "wildcard" option

Next up is the "Exchange Configuration" menu. Check the boxes for the services you plan to secure. The wizard will recommend names, ensure they're correct for your environment, keeping in mind our 3 commandments

On the next screen you'll be allowed to enter your Org info

Et viola! Click on the "Browse" button to save all hard work from above into a .req file

The contents of the .req file must now be submitted to your Certificate vendor of choice (I used Godaddy).

Once you've completed that you should be able to download your certificate. Once that is done it's on the next section.

About Me

About This Blog

This blog serves 2 purposes. Firstly, I want to share information with other IT pros about the technologies we work with and how to solve problems we often face. I work with technologies from the desktop to the data center, Active Directory, System Center, Exchange, Hyper-V, VMware, Networking and Storage.

Less altruistically, I use my blog as a reference. There's so much to learn and remember in our field that it's impossible to keep up. By blogging, I have a notebook that I can access from anywhere. It has made me look much smarter than I probably am on many occasions.