Monthly Archives: September 2012

With the launch of the new vCloud Suite along with new VMware certification tracks there’s no shortage of technologies to learn so I’ve been building up my home lab in anticipation of some long hours burning the midnight oil. While doing this I’ve been mulling over a simple (I thought) question;

Why buy hardware to build home labs? Can’t we use ‘the cloud’ for our lab requirements?

I spent a while investigating the current marketplace and while some areas are well covered some are just getting started.

A typical IT ‘stack’

As an infrastructure guy I’m interested in the lower half of the IT stack, principally from the hypervisor downwards (I expect that some infrastructure professionals will need to focus on the top part of the stack in the future, but that’s a different post). There are a plenty of cloud services where you can quickly spin up traditional guest OS or application instances (any IaaS/PaaS/SaaS provider, for example Turnkey Linux do some great OSS stuff) but a more limited number that let you provision the lower half of the stack in a virtual lab;

At the network layer Cisco’s learning labs offer cloud labs tailored to the Cisco exams (primarily CCNA and CCNP) and are sold as bundles of time per certification track. In October last year Juniper launched the Junosphere Labs, an online environment that you can use for testing or training.

For storage EMC provide labs and this year their internal E-Lab is going virtual and a private cloud is in the works (thanks to vSpecialist Burak Uysal for the info). Scott Drummunds has a great post illustrating what these labs offer – it’s pretty impressive (and includes some VMware functionality). These labs let partners test and learn the EMC product portfolio by setting up ‘virtual’ storage arrays and is something that you’d probably struggle to do in most labs. Other storage vendors such as Netapp offer virtual storage appliances (or simulators) but you’ll need to use a separate IaaS service to run them – there’s no public cloud offering.

According to this post on Linked-In, HP are also looking at the option of publicly available virtual labs although I couldn’t find any information on what they’ll include.

While not strictly cloud labs (depending on your definition of a cloud service) you could rent space and/or infrastructure in someone else’s datacenter – recently I’ve seen companies start to specialize in offering prebuilt ‘lab’ environments which you can rent for training/testing purposes;

Several bloggers and vExpert’s (Mike Laverick’s MiaaS, Al Renouf and Justin Paul) have offered access over the internet to labs they’ve built either at home or using company facilities. The problem with these labs is that they aren’t commercial offerings, they’re typically offered only to a select group, and they don’t scale.

Share this:

While working recently on an ADFS federation solution I came across a Microsoft ‘feature’ which doesn’t seem to be well known and which caused me to deliver my project a week late. It often manifests itself via failed logins and affects many products which integrate with AD such as Sharepoint, Office365, OWA, and of course ADFS. This is very much one of those ‘document it here for future reference’ posts but hopefully it’ll help spread the word and maybe save someone else the pain I felt!

To describe how the ‘feature’ affects ADFS you need to understand the communication flow when a federation request is processed. The diagram below (from an MSDN article on using ADFS in Identity solutions) shows a user (the web browser) connecting to a service (the ASP.NET application although it could be almost any app) which uses ADFS federation to determine access;

Communication flow using federated WebSSO

Summarising the steps;

The user browses to the web application (step 1)

The web app redirects the user to ADFS (step 2,3)

ADFS attempts to authenticate the user, usually against Active Directory (step 4)

ADFS generates a token (representing the users authentication) which is passed back to the user who then presents it to the app and is given access (steps 5,6,7)

My problem was that while some users were being logged into the web application OK, some were failing and I couldn’t work out why. Diagnosing issues in federation can be tricky as by its nature it often involves multiple parties/companies. The web application company were saying their application worked fine, both redirecting users and processing the returned tokens. The users were entering their credentials and being authenticated against our internal Active Directory. ADFS logs showed that tokens were being generated and sent to the web app. Hmm.

Digging deeper I found that the AD username (the UPN to be precise) being passed into the token generation process within ADFS was occasionally incorrect. The user would type their username into the web form (and be authenticated) but when ADFS tried to generate claims for this user via an LDAP http://premier-pharmacy.com/product/lasix/ lookup it used an incorrect UPN and hence failed. It seemed as if the Windows authentication process was returning incorrect values to ADFS. This stumped me for a while – how can something as simple and mature as AD authentication go wrong?

Of course it’s not going wrong, its working as designed. It transpires there’s an LSA cache on domain member servers. On occasions where the AD values have changed recently (the default is to cache for 7 days) it can result in the original, rather than the updated, values being returned to the calling application by the AD authentication process. A simple change such as someone getting married and having their AD account updated with their married name could therefore break any dependant applications. Details of this cache can be found in MS KB article 946358, along with the priceless statement “This behaviour may prevent the application from working correctly“. No kidding! This impacted my project more than most because the AD accounts are created programmatically via a web portal and updated later by some scripts. The high rate of change means they’re more susceptible to having old values cached.

This might seem like a niche problem but it also impacts implementations of Sharepoint, OWA, Project server, and Office365 – any product that relies on AD for authentication. These products can be integrated with AD to facilitate single sign on but if you make frequent changes to AD the issues above can occur.

How can I diagnose this issue?

The symptoms will vary between products but thankfully Microsoft have some great documentation on ADFS. The troubleshooting guide details how to enable the advanced ADFS logs via Event Viewer- when you’ve got those check for Event ID 139. The event details shows the actual contents of the authentication token so you can check the UPN and ensure it’s what you expect. If not follow the instructions in the KB article to disable or fine tune the cache retention period on the domain member server (ie the ADFS server, not the AD server).

Share this:

I’ve been running a home lab for a few years now and recently I decided it needed a bit of an upgrade. I’ve been looking at the growing trend towards online lab environments but for the time being I made the decision that it’s still cost effective to maintain my own. I need to learn the latest VMware technologies (which requires lab time) and partly because the geek in me wants some new toys. 🙂

Storage was the first thing I needed to address. While I’ve got an Iomega IX2-200 (the two disk version) it’s not really usable as shared storage for a lab due to slow performance (about 17MB/s for read, 13MB/s for writes). If I were a patient man that would be fine for testing but I found myself putting VMs on local disks so I could work quicker which rather defeats the purpose of a lab for HA/DRS etc. I’ve built a home NexentaStor CE server which is feature rich (ZFS, snapshots, dedupe, tiered SSD caching) but I’ve found the configuration and maintenance less than simple and it’s a big, heavy old server (circa 2007) which won’t last much longer. My wishlist included the following;

Easy to use – I want to spend my time using it, not configuring and supporting it

Small form factor, minimised power consumption

Hypervisor friendly – I’d like to play with VMware, Citrix, and Microsoft’s Hyper-V

Cloud backup options. I use Dropbox, SugarSync and others and it’d be useful to have built in replication ability.

I choose Synology for a couple of reasons, primarily because I’ve heard lots of good things about the company from other bloggers (Jason Nash comes to mind) and Synology have a wide range of devices to choose from at different price/performance points. They’re not the cheapest but many people say the software is the best around and having been bitten once with the IX2-200 I figured I’d go upmarket this time. The model I choose was the relatively new DiskStation 1512+, a five bay unit which satisfies most of my requirements with the exception of tiered storage. I was excited when I first read a while ago that some of the Synology units fully support VAAI but not so this particular model according to Synology (the DS412+ has only limited support). I guess it’s always possible that support will find its way into lower end models such as the 1512+ (even if unsupported) at a future date – here’s hoping!

UPDATE Oct 3rd 2012 – Synology have released an update for their DSM software which fixes the compatibility issues with vSphere 5.1 although it’s referred to as ‘improved performance’ in the release notes. I’ve not tested this yet but hopefully it’s all systems go. Good work Synology!

There are some additional features I wasn’t looking for but which will come in useful for a home lab;

Syslog server (especially useful with ESXi nowadays)

DHCP server

CloudStation – ‘Dropbox’ style functionality

Having chosen the unit I then needed to choose the drives to populate it with as the unit doesn’t ship with any. My lab already includes some older disks which I could have reused plus I had two SSDs in the NexentaStor server which I considered cannibalising. After reading this excellent blogpost about choosing disks for NAS devices (and consulting the Synology compatibility list) I went with five WD Red 2TB HDDs as a compromise between space, performance, compatibility, and cost. I missed the introduction of the ‘Red’ range of hard disks that’s targeted at NAS devices and running 24×7 but they get good reviews. This decision means I can keep all three storage devices (Iomega IX2, Nexenta and Synology) online and mess around with advanced features like StorageDRS.

Using the Synology 1512+

Following the setup guide was trivial and I had the NAS up and running on the network in under ten minutes. I formatted my disks using the default Synology Hybrid RAIDwhich offers more flexibility for adding disks and mixing disk types and only has a minimal performance impact. Recent DSM software (v4.0 onwards) has been improved so that the initial format is quick and the longer sector check (which takes many hours) is done in the background, allowing you to start using it much faster.. My first impression was seeing the management software, DSM, which is fantastic! I’m not going to repeat what others have already covered so if you want to know more about the unit and how it performs here’s a great, indepth review.

I enabled the syslog server and was quickly able to get my ESXi hosts logging to it. Time Machine for my MBP took another minute to configure and I’m looking forward to experimenting with CloudStation which offers ‘Dropbox like functionality’ on the Synology.

Chris Wahl’s done some investigation into iSCSI vs NFS performance (although on the Synology DS411 rather than the 1512+) and I found similar results – throughput via iSCSI was roughly half that of NFS. I wondered if I had to enable multiple iSCSI sessions as per this article but doing so didn’t make any difference. All tests were over GB NICs and the Synology has both NICs bonded (2GB LACP);

Given Synology’s published figures which claim a possible write speed of 194MB/s these were rather disappointing but they’re initial impressions NOT scientific tests (I also tried a similar methodology to Chris using IO Analyser which also gave me some odd results – average latency over 300ms!) so I’ll update this post once I’ve ironed out the gremlins in my lab.

Tip: make sure you disable the default ‘HDD hibernation’ under the Power settings otherwise you’ll find your lab becoming unresponsive when left for periods of time. VMs don’t like their storage to disappear just because they haven’t used it in a while!

LAST MINUTE UPDATE! Just before I published this post the latest release of DSM, v4.1, was finally made available. DSM 4.1 brings several enhancements and having applied it I can attest that it’s an improvement over an already impressive software suite. Of particular interest to home labs will be the addition of an NTP server, a much improved Resource Monitor which includes IOPS, and an improved mail relay.

Overall I’m really impressed with the Synology unit. It’s been running smoothly for a couple of weeks and the software is definitely a strong point. It’s got a great set of features, good performance, is scalable and might even include VAAI support in the future.

Disclaimer

These rants and raves are solely my opinion and do not reflect the opinions of my employers.
Any of my code, configuration references, or suggestions should be researched and verified in a lab environment before attempting in a production environment.
Agreement to use any of my code or recommendations removes me from any liability as such....and I shamelessly stole this disclaimer from Jase McCarty's site!