Tuesday, November 25, 2008

Pulling Teeth

It's an exceedingly common scenario these days to want to make a virtual machine out of some physical machine. In fact when it comes to reproducing an environment I can't think of a more thorough way (especially when there's multiple machines involved).

This often leaves someone tasked with finding a powerful, affordable, and flexible physical to virtual conversion utility. Now finding a P2V solution isn't really a challenging problem, but finding one that's easy to use is.

I would encourage anyone who's ever been frustrated with the P2V tools on the .VHD (Microsoft Virtual Server, Virtual PC) side to consider making a .VMDX instead (VMWare Workstation, Server, Fusion).

Novocain

The VMWare Converter is free, effortless to use, and makes hot clones (in addition to cold ones) with the same ease that Johnnie Cochran acquits his clients. With a hot clone you don't even need to shut down the OS or stop any services! You can perform a P2V transformation and experience zero down time...all for free (if you call being bound to a VM Ware product free). This utility is so handy it just may be the help you need to help you make a better vendor choice when it comes to virtualization...

Just walk through a wizard, click hit a few radio buttons, and you'll be creating a hot clone in no time.

Details

The VMWare converter will run on the following OS's:

Windows XP Professional

Windows 2003 Server

Windows 2000 Professional

Windows 2000 Server

Windows NT SP4+ (IE5 or higher required)

You're likely to get a much smoother ride with Windows XP and Windows 2003 as they both make use of the Volume Shadow Copy service, but supposedly you can make hot clones with the other OS's as well (I've only made hot clones on Windows XP and Windows 2003).

Like any task that could go horribly wrong, there's a couple of things you should check before selling yourself on hot clones, or at least placing any bets.

Don't just take my word for it, try the utility on some machine you don't like and create a clone (hot or cold). When you're done, just be sure post a YouTube video of you Office Space'ing the machine just like these guys did.

The clones should be made hot, the revenge should be served cold.

Note: Some employers take offense to employees taking their hardware into fields and striking company assets repeatedly with with sports equipment, regardless of whether or not a hot clone has been made.

Saturday, November 22, 2008

Uh, You're Testing That Where?

A while back one of our data analysts and a network admin stubbed out a sheltered domain within our LAN. The idea was that we would set up a sandboxed class C network where developers could test code and not worry about breaking anything on our public network.

While this might seem a little hardcore at first, remember that a lot of solutions these days are meant to be deployed to some reasonably complex IT environments (like this sample BizTalk topology on the right). As such these codes may be making calls to Active Directory, Exchange, sit underneath some type of proxy (ISA/Squid), and get deployed to some kind of web farm (think load balancer appliances, NLB clusters, etc...).

Needless to say these kinds of environments are dramatically different from what you find on your workstation. Depending on the type of application you're delivering and where it's getting deployed, simply testing on Windows XP and IIS 5 might not qualify as a realistic environment.

The Right Kind of Cheap

The good news is that with Virtual Server/PC being free (give or take a windows license), virtualization has never been cheaper or easier. We ended up using a bunch of hardware kicking around the office to provision these machines. Our little test subnet already has a wide variety of virtual instances running around in it, hosting everything from Exchange to AD, and even a full blown MOSS farm.

The bad news is that maintaining this virtual instance still costs time, and as you may have heard, time is money. This can be mitigated in part, by the ease of performing machine level backups on VMs, and other VM features like Undo Disks, Differencing Disks and saving state. Nevertheless it makes sense to budget some time to maintain this kind of environment, rarely does IT infrastructure maintain itself...

The Easy Value

Originally the thought was to set up some SharePoint farm and allow developers to stage their code in a farm environment prior to deploying at the clients. You'd be surprised at just how many deployment issues crop up just by deploying to a farm instead of a single machine.

The real value is that you give the solution a chance to break in the your farm first. I'd much rather start up a troubleshoot at my desk than in a client's server room.

As far as I'm concerned the farm has already paid for itself. It's already stopped me from deploying a MOSS update that I'm positive would have trashed one of our clients SharePoint farms. As some of you may know, there is no uninstall for most SharePoint Service Packs and Updates, you often have no choice but to continue forward...even if it's off a cliff.

By breaking things in our VM farm, not only do I have an easier time rolling back if I hit a wall, but I can also troubleshoot it with all the tools and brainpower that my fellow developers have to offer. The alternative is breaking it for the first time at some client location where I'm all alone and only have the tools I thought to bring with me. Some fights just shouldn't be fought alone (at least the first time around).

The Extra Value

This wasn't immediately apparent to me but a lot of developer benefit from this kind of sandbox for another reason. In addition to testing their code in a more complex environment, they actually get a chance to look under the hood and play with the many services that decorate typical IT environments. It's surprising how many of these services that your typical developer isn't familiar with (AD, DNS, ISA, IIS, MOM, etc...). Let me elaborate.

A ton of web developers have no idea how DNS works. This could also be said for HTTP, TCP and the web servers that host their applications.

This akin to a cab driver that knows how to drive, but doesn't know any traffic laws or anything about the car.

This isn't meant to be a roast for developers who haven't had the benefit of IT fundamentals. Some of these topics just aren't covered in a Computer Science degree. This is about enabling developers. Empowering them to discover what hidden gems exist in typical IT environments and how these existing IT assets can help deliver better solutions.

Trust me, there's a big difference between telling a developer what DNS is and simply having the guy create and A record and see that light switch on in his eyes. What's even better is letting him discover for himself what IT assets can do for him in a clean and controlled environment where mistakes are easy to rollback.

Imagine This

Don't get me wrong, there's always going to be a percentage of developers that are clueless when it comes to all things IT (starting with their workstation). That's where your job security comes from. But if you educate just a couple more bodies you'll start to notice the difference pretty quickly. There's a night and day difference between a dev who simply lobs a program over the wall and one who has a holistic understanding of the landing zone. These kinds of investments don't take long before they start to yield noticeable returns. There might even be a day where IT staff come to trust developers to not completely mangle production environments.

Sunday, November 16, 2008

Some Reasons

Maybe you're tired of putting your SharePoint web application in full trust and running web applications with huge security surface areas. Or maybe you've become tired of GAC'ing assemblies just so that they can run in full trust. Heck, maybe you've even read this article and decided to finally start handling Code Access Security in a more elegant way.

All of the above are good reasons to make a custom Code Access Security policy file, and in the following section we're going to do just that. If you're curious as to why you might be doing this in the first place, it might be a good idea to peruse common code access security issues described here.

Creating A Custom Policy File

The intent of the custom policy we're about to make is to allow yourcode and only your code to run in full trust regardless of where it is (doesn't have to be in the GAC). All other code in the application will be running as WSS_Minimal which means it will have a reduced set of privileges (see table). For example, codes running in WSS_Minimal can't access the SharePoint object model. Yourcode however will be running in full trust and will be able to do whatever it wants.

This is often desirable since we have no idea what web parts information workers or administrators will lob into our SharePoint application. It's a little presumptuous to assume that they're all safe and won't do anything malicious. Granting them full trust allows them to do things like write to sensitive areas of the disk, access the registry, etc... Users with high privileges (ie. administrators) get duped in to running malicious code all the time, that's one of the reasons Code Access Security exists in the first place. If you're curious about what the difference between code running in full trust and code running in WSS_Minimal is, refer to the table in the link above. Onward.

First make a copy of C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\CONFIG\WSS_Minimal.config and put it in your application folder, call it WSS_Custom.config. If you want to increase the privileges of other code you can start with WSS_Medium, this will allow other code to access the SharePoint API without having security exceptions thrown.

Strong name your assembly. We're not going to put in the GAC, but it still needs to be strong named so that it can be uniquely identified by the WSS_Custom.config policy file.

Open up the Visual Studio Command Prompt (or go find sn.exe) and extract the public key from your .snk file into another file. (It's worth noting that the Public Key is not the same as the Public Key Token that you might get from your assembly using Reflector).

sn -p YourStongNameFile.snk PublicKeyOnly.snk

You should see the output "Public key written to PublicKeyOnly.snk".

Now we print out that public key with the following command:

sn -tp PublicKeyOnly.snk

Copy that insanely long stream of numbers (your assembly's public key). Add the following line to your WSS_Custom.config just below the <CodeGroup class="FirstMatchCodeGroup"...>'s <IMembershipCondition> element.

If you were getting security exceptions before they should be gone now, at least all those coming from your assembly. Other code will continue to run in WSS_Minimal (or WSS_Medium if you used that .config as a template) and may throw security exceptions should they try to access APIs that they don't have the privileges to do so.

This is a lot more preferable than two popular alternatives:

Having the entire web application run in full trust (which gives all assemblies full trust).

Putting your assemblies in the GAC just so that they can run in full trust.

Thursday, November 13, 2008

Ugly Error

I've seen this error often enough that I think it deserves a brief entry. Essentially the symptoms are:

When you open up the Internet Information Services (IIS) Manager it takes a long time to load and when it finally does the console is blank. Running iisreset (Start->Run->iisreset) will temporarily fix the problem.

In the event viewer under Application you see error codes for 6398, 7076, and 6482. Specifically they look like:

Event Type: ErrorEvent Source: Windows SharePoint Services 3Event Category: TimerEvent ID: 6398Date: 11/13/2008Time: 1:34:47 PMUser: N/AComputer: [COMPUTER NAME]Description:The Execute method of job definition Microsoft.Office.Server.Administration.ApplicationServerAdministrationServiceJob (ID [GUID]) threw an exception. More information is included below.Attempted to read or write protected memory. This is often an indication that other memory is corrupt.For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

A Fix

Supposedly this behavior takes place when the Windows SharePoint Services Timer (OWSTimer.exe) has two threads that try to access IIS at the same time. There's currently a hot fix and description for this problem that patches IIS that. Ideally there'll be an IIS update that comes out in the future that isn't a hot fix. If you can bear it I would consider waiting.

Wednesday, November 12, 2008

Who Is This Handsome Helper?

If you've never heard of Robocopy (Robust File Copy) you're missing out. When it comes to reliably coping files over an unreliable network, there's no better tool. In fact Robocopy just recently saved my bacon, which is why I'm writing this in the first place.

Originally distributed with the Windows 2003 Resource Kit, Robocopy is now available on Windows XP, Vista and Windows 2008. It's worth mentioning that Robocopy isn't anything like RoboCop, they just sound the same, they both never quit and neither of them are very entertaining to watch (boo-yeah!).

Features

Relentless - By default if Robocopy fails (ie. your network connection cuts out) it will retry every 30 seconds until it's tried to copy the files 1 million times, if there's even a remote possibility of getting that file over the wire, it will get there. Given good uptime, odds are that your source machine will die from hardware failure before Robocopy has stopped trying to deliver your file. I dream of a world where USPS had this kind of stamina.

Resumable - You can throw down the /Z switch and files will pick up right where they left off. Don't believe me? Try the following; start copying a large file over the network (with Robocopy and the /Z flag) and then toy with Robocopy by unceremoniously pulling out the Ethernet cable. Robocopy will first throw an angry error and then relentlessly retry every 30 seconds until either your file has been copied or you've gone Office Space on the source machine (there's no switch to defend against that). Plug the cable back in and voila, the file automagically resumes. Trust me, this trick is great at cocktail parties and a smash hit with most women. It's worth mentioning that resumable copying does happen a little slower (30% slower in my tests).

Replication - Robocopy is exceedingly good at replicating/syncing directory structures. There's never been an easier way to mirror directory structures or file systems.

Others - There's a bunch of other features like decent logging, a User Interface if you don't like reading instructions, scripting, mirroring file ACLs, etc...

Faults

My only wish is that I could get this utility to call some code once it's done (for notification). Yes I could use a process or file watcher but I'd still like something a little cleaner. I feel like I'm being left in the dark. Maybe that's the very nature of fire and forget...I shouldn't need a notification.

Use It

So next time you're trying to move a lot of files over the wire consider using a utility that is specialized with this very same problem. I've seen (and written) a lot of code that tries to be smarter than it needs to be, none of it has had the effectiveness or cost of ownership of this precious little utility. I wish on you a buggy network, but great file copying.

Sunday, November 9, 2008

Problem Solving

I think most people would agree that problem solving comes in pretty handy when it comes to software development. Knowing what you can infer given a set of data or exactly what you need to provide a feature is key in coming up with a lean solution.

Puzzles

It was a Data Structures and Algorithms T.A. who first convinced me the value of puzzles when it comes to programming. We were trying to come up with a solution for Instant Insanity when a fellow student asked if there was really any value in knowing a solution for a children's game.

The T.A. then made the excellent point that when you brush away all the noise from most business problem you essentially end up with a simple (albeit potentially dull) puzzle. In fact being able to solve Instant Insanity puts you in a great position to solve a lot of constraint based scheduling problems, a family of concerns that many businesses (ie. airlines) can use to save millions of dollars a year. Being able to solve puzzles algorithmically is analogous to solving business problems in code.

Sharpening the Saw

So where do you go about getting good puzzles to solve? A coworker recently pointed me to a site called Project Euler (as in the mathematician). The site boasts a sizable collection of math based problems that are meant to be solved programatically. Once you come up with an answer you can submit it and get immediate feedback. The problems are ranked in terms of difficulty and some of them are guaranteed to get you thinking...in fact, that's the point!

Once you've solved a problem you also get access to a forum where other developers have solved the problem and discuss their solutions. For myself the merits have been three fold:

Once you come up with an answer, you can compare your approach to that of thousands of other developers and get some perspective on different approaches.

It's fun.

Consider the following problem (Problem 9). It's the 9th problem in the site's inventory of 216 problems. While finding the solution is far from impossible (in fact 19885 people have already solved it at time or writing) it will get definitely test your problem decomposition skills.

Tuesday, November 4, 2008

XSLT In SharePoint

Unfortunately for me, being a well rounded SharePoint developer involves a broad range in skills. Not only do you need to become familiar with the SharePoint object model and a litany of MOSS technologies (Excel Services, Search, BDC, IPFS). Knowledge of ASP.NET, Web Parts, Workflow, XML and XSLT are also near essential competencies when it comes to a rounded SharePoint tool belt. This is of course my opinion, fact...is another matter all together.

If I had to pick a weakest link out of the above, XSLT would probably be my Achilles heel. Lucky for me most of the changes I make to style sheets are mild in nature. They usually center around styling the output from some out of the box SharePoint web part. I'm not an all star, but when paired with a decent XSLT debugger I can usually muddle my way through.

Against The Grain

Today I was styling a BDC Web Part and having a difficult time getting the result I wanted. It got repetitive enough that I decided flailing at the problem probably wasn't going to improve the situation, and that pointing a debugger at the style sheet would probably save me time in the long run.

Here's the hitch though, the debugger of course takes the style sheet (which you have) and the source XML...which you don't. That particular ingredient is unavailable in these situations. So the options were two fold:

Reflect on the Web Part and try to figure out where it's getting the result set from and do the same.

Provide the Web Part a style sheet that simply produces the same XML which it gets applied to.

A Solution

The answer ended up being quite straight forward. There's an XSLT element by the name of <xsl:copy-of select="expression" /> which solves the problem quite nicely. The copy-of element makes a copy of not only the current node, but it also copies namespace nodes, child nodes and attributes. Because of this sweet little gaffer the solution ends up being quite terse (at least for XSLT):

At this point I'm positive there's some XSLT developer out there rolling his eyes at me for posting entry level XSLT on the web...but that's what newbs post. Newbish content :-). In fact my next post about XSLT isn't likely to get much better. If there's a simpler solution do let me know.

Sunday, November 2, 2008

Getting Into The Farm

A lot of developers first introduction to SharePoint is being asked to take an ASP.NET application that they're familiar with, and integrate/port it into a WSS/MOSS instance. These same developers are often thoroughly disappointed when it comes to the application developer experience for SharePoint. The prototypical ASP.NET developer is likely to miss a lot of the everyday ASP.NET developer comforts they've become used to writing typical ASP.NET applications. Common gripes are often along the lines of:

No designer surface for building user interfaces with code-behinds.

Lack of tools making it easy to deploy/maintain ASP.NET applications across the farm.

Reduced trust levels for the SharePoint web applications. If you read the last post, you'll remember that unlike ASP.NET web applications (which run in full trust), SharePoint web applications start out in WSS_Minimal which has far less CAS privileges.

Other Techniques

By no means is this the first post speaking to different ways of getting ASP.NET applications into SharePoint. In fact my favorite post by Chris Johnson compares and contrasts 4 different techniques not listed here. They include:

_layouts directory deployments.

Building custom Web Parts.

User Controls with the Smart Part web part.

Embedding ASP.NET pages directly into the content database.

This Technique

This techniques is a little different. It involves 6 steps.

Create a User Control that represents some functionality you'd like to use in SharePoint.

Deploy the user controls to the SharePoint web application folder along with assemblies.

Add Safe Control directive for you assemblies user controls to the web.config.

Create a Page Layout to dictate the layout/structure of a page instance.

Embed your user control into the page layout.

Create an instance page based off the page layout.

It's important to note that because this technique requires Page Layouts, it's only available to developers that are working with MOSS and have turned on the Office SharePoint Publishing Infrastructure feature for their given site collection, and the Office SharePoint Publishing feature for their given site. These two features are under the Site Settings->Site Collection Features and Site Settings->Site Features respectively. Or you can simply create a publishing site which starts with these two features turned on, your choice. Onward.

Create and Deploy the User Control

Open up Visual Studio 2005/2008 and create a Web Application Project. This is the kind of project that compiles all of it's contents into a single assembly (.dll). If you're running VS 2005 and you haven't already downloaded this project, you can get it here.

Add a new user control (HelloUser.ascx) with the following .ascx content.

Deploy the projects assemblies to C:\Inetpub\wwwroot\wss\VirtualDirectories\[ApplicationRoot]\Bin. Make a directory called PageLayoutUserControls in the [ApplicationRoot] and deploy the .ascx's to C:\Inetpub\wwwroot\wss\VirtualDirectories\[ApplicationRoot\PageLayoutUserControls.

Add the following lines to the web.config to safe control the assemblies and the .ascx's.

Create an instance of the page. Site Actions->Create Page->Pick the Page "(Page)Hello World" and give it a name->Create. You're done!

Costs and Benefits

I like this technique because it has a lot of the benefits from traditional ASP.NET and a lot of flexibility.

Developers can develop their user controls w/out SharePoint (unless they need to call against the SharePoint object model).

You get a design surface to develop these codes.

You can deploy/organize your user controls to any application directory want (unlike the Smart Part).

You can make use of the SharePoint object model and run in the SPContext of your given site.

It's very easy to create hybrid page layouts that include both your user controls, a couple of web part zones and come SharePoint publishing controls. This lets you keep a lot of the SharePoint functionality that works well for your project, and still let you customize pages using traditional ASP.NET development tools.

You can get rid of web parts should you be interested in cleaner markup. If you have some exotic layouts and don't want to fight with all the <table> markup this could be quite helpful.

There's still some overhead with this approach. This lets you leverage a lot of the stock SharePoint functionality but there's no such thing as a free lunch.

You'll potentially need to create a Page Layout and page instance for each page you want in your application. This can make deployment and change management more tricky. You'll need a build manager that is savvy with tools to manage the content database (stsadm, Content Deployment Wizard)

You'll need a MOSS license and access to SharePoint Designer. If you don't have these the Smart Part is your next best bet for this type of UserControl/deployment type development.

About Me

Tyler Holmes is a Solutions Architect working in Portland, Oregon. He lives mostly in the MS tech stack and is currently treading the waters of Communication/Collaboration and Business Intelligence with off the shelf/open source technologies.