Have you ever deployed an app into a Windows domain that uses Active Directory authentication, and sometimes it works and other times it doesn’t? This can be an especially annoying issue if you aren’t a domain admin and you can’t log into your domain controllers to examine their settings.

Fortunately, if you suspect that the problem may be the SSL Certificate the server is using when you connect to it there is something you can to do conclusively troubleshoot the issue. First let’s talk about the issue I ran into.

When your app wants to talk to Active Directory to validate credentials it can do so without using SSL if it connects to port 389. If your app is using that port then read on for curiosity’s sake only because this isn’t your problem. If your connection string to AD specifies port 636, you are using SSL. This will only work if the server you are connecting to hands you a certificate in the connection attempt that your machine trusts.

In a normal domain setup this isn’t a problem. The AD server should have a certificate issued by either a domain CA that all machines trust, or a third party cert that again all machines trust, this time because the CA is recognized by Windows automatically. These are certs issued by Verisign or something like that.

If the certificate you receive from the AD server is not trusted, your connection will fail, authentication will fail, and the smoking gun will be a very easy to read Windows Event Log entry like the one below.

That error log entry tells you very specifically what the issue was, but it leaves out some important information.

What server did I connect to that failed?

Connection strings often just specify the domain name not the specific server to connect to. If you have multiple domain controllers you could end up connecting to any of them for any given connection attempt.

What certificate did it give me that just failed?

Admins hate it when you just point fingers and say “your thing is broken” with no other details. It would be nice to be able to say to your domain admins “hey, I got this certificate from this issuer and my machine doesn’t trust it. Let’s work this out.”

To get that information, and to get to the point of this article, we are going to do a couple things. We are going to find all of the domain controllers on the network, and then connect to them one by one and grab the certificate out of the connection to examine it.

To find the domain controllers we can ask directory services to just give us the list.

That command would show you very clearly if a domain controller is trying to use a self signed certificate, and yeah, that happens to me a lot. You can also look at the issuer and see if you trust that issuer in your local machines Root certificate store.

Any time two Windows Servers need to communicate to support application data requests, there’s a good chance the Distributed Transaction Coordinator or MSDTC will be involved. When two servers are configured and functioning correctly, like most things, you won’t even notice this layer of coordination between the two exists.

The problem is that in modern VM based corporate environments, it is very common for these DTC’s to be unable to communicate with one another if a VM admin builds machines from templates that are not properly sysprep’ed.

The bug is that if two MSDTC’s have the same CID (a GUID identifier), they cannot communicate with one another. They both believe they should have the same name effectively, so that can’t talk with one another.

As you can imagine this can be an annoying bug to track down as some servers will have issues connecting to only some servers, and only if the communication goes through MSDTC.

Fortunately, with PowerShell and a little bit of registry foo, we can test conclusively for this issue, and it’s an easy one to fix. Below is the script to detect it.

The output will be a table showing you the list of servers and their MSDTC CID’s. Any two or more entries with the same CID will be unable to participate in distributed transactions with one another.

The immediate fix is to uninstall and reinstall the DTC on each of the affected machines or at least $numberOfAffectedMachines – 1 machines, to ensure they all have unique ID’s.

The long term fix is to have a chat with your VM admin about sysprepping machines so you don’t have to deal with this anymore.

This could easily be turned into a function that will detect and then remotely fix this, but since I don’t run into this problem on a regular basis, I’ll leave that task as an exercise for any readers that do.

The Problem

I love Pester, and I really want to get on the operational testing band wagon. But one of the perpetual issues involved with testing is how do I visualize my test metrics, and can I take action on failed tests automatically?

Enter Jenkins and Pester. You can express a lot of concepts in the form of Pester tests and Jenkins is more than happy to take Pester’s output and not only visualize it for you, but take down stream action based on failures and show you those results too.

The Tools

Jenkins

There are lots of great tutorials out there for installing Jenkins, but it’s a big subject that I’m not going to cover here. We’re going to use Jenkins for this, but I’ll assume you have an instance running already.

Pester

Installing pester is very easy, but I like to use the Install-Module CmdLet from PSGet and that needs to be installed too if it isn’t already.

That was easy. Don’t get complacent though, take a look at the directory the module was installed into. The problem here is that if you try to load the module from Jenkins right now, it won’t be able to find the module. To make it discover able for Jenkins and for other users we need to copy that module folder to the system wide module folders.

Notice that I copied the module to the SysWOW64 modules folder. This is because in your own testing you will mostly use 64bit PowerShell, but at the same time, most of you will have downloaded and installed 32bit Jenkins. Using 32Bit Jenkins means you must copy any modules you want to use into the SysWOW64 folder because it can’t see the 64 bit folder. We are going to run a check shortly to make sure the module will be available within Jenkins before we start to run into confusing errors.

Plugins

We are going to need three plugins to make this workflow happen. Go to Manage Jenkins > Manage Plugins and select the Available tab to select the plugins we need to install: Nunit plugin, Parameterized Trigger plugin, Copy Artifact plugin, and the PowerShell plugin, as depicted in the screenshots below.

Creating the First Project

Our first project is going to be called Test Permissions Job.

The only thing we are going to ask this job to do at first is confirm that the Pester module is going to be available to Jenkins as we expect. It’s worth discussing for just a second though how I intend to accomplish this. Below is a screenshot of the script I’m going to run in the project.

That’s not much right? Here’s the thing. Writing actual PowerShell code in that little text box would suck really really bad. I don’t want to do it. So instead all I do is use the automatic environment variables that Jenkins gives me to find the path to scripts that I will edit in PowerShell ISE instead.

Using this strategy buys me a lot of nice things. The config is nice, clean and easy to read. The scripts are easy to edit in the normal PowerShell ISE. Finally, if I want to make a change to the way the job behaves, I can just edit the script, hit save in the ISE, and execute the job again, without ever having to actually change the job configuration at all! It really makes your testing iterations go much much faster. I think you’ll like it.

If you’re working with a brand new project, the workspace won’t exist yet so you won’t be able to save anything there, but just execute the project with no configuration so it doesn’t do anything, and Jenkins will create the workspace folder for you. Using the the workspace folder is usually a good choice btw, instead of somewhere else on the drive so that if you set up slaves, the jobs will continue to run and be able to access the scripts they need. The workspace folder on the master is usually found at C:\Program Files (x86)\Jenkins\workspace\<job name>

Ok, moving on. The contents of the script are as follows:

try {
Get-Module -ListAvailable
}
catch {
throw $_
exit -1
}

And we are going to save it to the workspace folder:

Get-Module is a very simple CmdLet, but don’t forget to almost always put your code in try/catch blocks when scripting for Jenkins. The reason is that to execute PowerShell, Jenkins is calling Powershell.exe from cmd.exe. If you, the script author are not diligent about not only catching errors, but returning non zero return codes, then Jenkins doesn’t have any way of knowing that something went wrong. This will result in job steps that encounter errors, but do not halt project execution, and do not cause projects to be marked as failures. So what we do is catch the error, immediately throw it back out so that the error makes it to the console for logging, and then throw a non zero return code to ensure the project gets marked as failed.

Hopefully though, if we’ve done our job right, the console output from this first job run will show us a list of all available modules, including Pester. If you don’t see Pester in the list try adding a line to the script to output the $PSModulePath variable and ensure the Pester module is in one of those directories.

The First Tests

First we’ll save a script to the workspace folder just like the one above. I am going to call it Get-CustomLocalGroupMembership .ps1 with the contents below:

So let’s note a couple things. When I look at that script in PowerShell ISE, that $env:workspace variable isn’t going to mean anything. To test that script effectively in PowerShell ISE you may need to assign an $env:workspace variable in your session manually before testing to ensure it will execute as you expect without having to do make modifications you might forget to remove and break your project.

Next, the list of local users. I actually created them for the purposes of this demo, so feel free to do so yourself to follow along.

Now let’s look at the project’s only build step:

That was easy. Invoke-Pester will automatically do a recursive search in the current working directory ($env:workspace) for any files with <name>.tests.ps1 as the name format and execute them. This project step is very clear and easy to read.

If you noticed that after I just got done saying that we should almost always put our code in try/catch blocks, I didn’t do it here, you aren’t wrong. The trick is that Invoke-Pester is going to take care of this for me with the -enableExit parameter. If any errors occur during execution Invoke-Pester is kind enough to bubble up the error for me and return a non zero exit code. Even if there are no unexpected exceptions; if I simply have failed tests, it will return all of the errors and return an exit code equal to the number of failed tests.

So let’s run the project and see what we get. Our output should look like below:

This is really fantastic. We can see not only that tests failed, but exactly what user we expected to present and wasn’t found. Take some time to click around in the job and look at all the nice results you get, and realize it only gets better as tests start to pass later.

Now let’s see if we can get the output file and read it to figure out how we can make some use of it.

Well, that’s ugly but it will work to help us figure out what to do next. We can use this kind of testing to figure out exactly how to query the xml file to get the data we need, and what we see there is very close. I don’t know NUnitXML format well enought to tell you what query will get you the data you need. I just know XML well enough to keep querying until I have what I need. Later you’ll see the query I came up with to make the project to fix the permissions work. So let’s move on and set up the fix.

Before you attempt to implement the configuration below, create a new Free Style Project called “Fix Permissions Job Step” if you want to follow along, and then add the Post-Build actions shown below to the Permissions Testing project.

Next, in the Fix Permissions Job project we will tell it to copy the xml result file from the permissions test project. You will also see the build step that invokes a pester test. The code that follows will be the content of the AddUsersToAdmins.tests.ps1 file that the build step is invoking along with the code for a helper function it needs.

Make sure that the test users are NOT a part of the admins group and then run the test job. What you should see is a tests job that runs and fails three tests, then executes the fix job which will add the users and mark those tests as passed.

Once you do that, go ahead and kick off the permissions test project again and you will see the tests not only pass this time, but Jenkins knows the tests failed last time and doesn’t mark them as just passed, but as fixed.

Lets say that those user permissions being missing is a serious problem. In the past you might have set an alert on them being missing so someone could fix it. Now, there is only a need to send out an alert if the attempt at FIXING it fails, which Jenkins will know about as soon as it attempts to fix it for you.

Conclusion

This kind of operational validation can be extended to testing things like ensuring a web site is up. Don’t just test to ensure that the w3wp service is running and that the SQL service is running. You can actually run Invoke-WebRequest and test to ensure that you get a return code of 200 and that the elements you expect to find are present in the web page, using Pester, and if they aren’t, you can run further tests in follow up projects to make automated attempts at solving some of the common issues you know might cause an outage.

You won’t get a midnight alert because Jenkins fixed it for you, but you can see in your build statuses the next morning that something went wrong and take a look at what it was based on the tests that failed.

Thanks for reading and of course if you have any questions, hit me up on Twitter!

The Problem

I was recently asked to come up with a method to ensure files are encrypted at rest after they are transferred from client servers onto my employers servers. We have constraints on the way we do things that can make it difficult to install third party software to get things like this done, so I often find myself in the position of having to come up with my own code to do this kind of thing. That being said, cryptographer is hard, even for really smart people, so I have no intention of actually rolling my own crypto.

Thankfully, with Powershell, it’s turn the .NET crypto classes into a usable tool for my purposes. Over the next few blog posts I’ll be using this MSDN page as the basis for a custom encryption module that can be deployed to servers without having to install any third party software at all.

How Certificate Based Cryptography is Going to Work

Since you’re reading a blog mostly about Powershell scripting, I’m going to assume you are working in a Windows environment. The most natural way then to encrypt and decrypt files will be using certificates managed in the Windows Certificate store. The thing is, if you know how encrypting an SSL web session works, Public/Private key cryptography isn’t actually very good for encrypting large amounts of data like text files or media files.

If you read the linked MSDN page closely you get something like the following workflow.

Create a certificate with a public and private key pair

Export the public key and copy to the server the source sensitive files

For each file that we encrypt we will do the following

Generate a new AES symetric encryption key and IV

Use the key and IV to encrypt the data file.

Use the public key to encrypt the key and the IV and prepend them to the encrypted data file.

When the files arrive on the destination server we can use the private key decrypt the key and IV for each data file and then use the decrypted keys decrypt the larger data file.

Building the Module

The first two functions we need are one to create a certificate and one to check and see if the one we are creating already exists. These are easy since we can just use self signed certificates. Self Signed certs are fine for this use since we aren’t asking browsers or other systems to trust them. They are purely for our own use.

These are pretty thin wrappers over existing commandlets, but they serve the purpose at hand. For instance, in this case I know that if I’m generating a self signed cert, I’m always going to want to export the public key for use on a remote server. So why write the commands to do the export myself each time I generate a cert? Wrap it in a function and viola, the cert is conveniently exported each and every time.

Also, notice in the function New-ClientCert that each cert we create gets immediately moved to the Address Book cert store. This is because we aren’t going to be asking the system to trust these certs in any way. I don’t need a browser to accept connections encrypted with these certs. I just need a place to keep named certificates.

This function is where we see the code from the linked MSDN article translated into Powershell. Going over how it functions in too much detail could take a while, but it’s basically outlined by the work flow detailed above.

That’s it for this post as I’ve run out of time, but in later posts we are going to fill out the support functions we need to make a usable module, and of course, we’ll work on getting the encrypted data back again into plain text or usable data.

In my last post I demo’d building an event based GIU app in Powershell Studio. You probably noticed though that some of the code to handle long running tasks in a background job was less than ideal. To recap it was handled as follows:

Create a timer object

Assign a code block to the tick event that knows how to poll for the status of all of your background jobs

Create and start the jobs running

Start the timer ticking

The code block checks for job results and hopefully cleans up after itself when the jobs are done.

It certainly works, but it can hardly be called clean. In fairness, without C#’s background workers, Powershell is at a cleanliness disadvantage, but I think there’s still a better way. What if the process looked more like:

Register an event handler that just waits for your job to finish without sitting there and cycling over and over again.

Create and start your job

Inside the code of the job, fire an event when the job is done that calls your registered event handler

The event handler consumes the results of the long running task and cleans up the job when it’s done.

Considerations in this code

Session Variables: Creating a session variable takes a little time, but it saves you time later. If you are making a GUI find an unobtrusive place in your app’s execution like the FormShown event to create session objects or even a loading spinner before the user is allowed to start if you need to. Create sessions on the local server and to any remote servers you want to execute jobs on. That way they will be ready to go when you need to pass them to Invoke-Command.

Job Names: There might be a better way to do this, but to allow the code to clean up after itself without looping through all background jobs, it’s important that the event handling code have a way to know which job it should delete. In typical use you would probably build a string inside a loop for each job name you need, and pass it into the job.

Job Name Param: The script block you pass into the job needs to take at least one parameter so it can receive the name you’ve assigned the job. This is important because the jobs result set needs to include this name so the callback code block knows which job it’s receiving results from.

Register-EngineEvent in Remote Code Block: It’s not great that you have to repeat the event name so many times, but it’s import that you register the engine event in the main session, and also in the background job code block. In the background code block though you use the -Forward parameter to ensure the event you raise later gets forwarded up to the parent session.

-EventArguments: This should be the result of your long running operation.

-Sender: The sender will be your job name so the callback codeblock knows which job to clean up.

Making use of values: If you have a GUI app it’ really easy to make use of the jobs results. You can find the UI element you want to modify and assign the value where it’s needed. If you are not in a GUI app you can either create the variable ahead of time and do the assignment in the callback code block, or assign a new variable in the code block, but make sure the scope the variable so that it still exists after the code block completes.

Memory Usage: I looked for a decent way to do this using runspaces because I think the memory usage is probably lower, but I didn’t find a decent way to make it happen. So keep in mind that this works well, but keep an eye on RAM Usage in testing. If you start a loop over a large number of object I can imaging memory consumption getting out of control pretty quickly. But of course, using the looping method the same consideration applies.

Why Not Runspaces?: Run spaces have a lot of advantages besides lower memory usage. One of them is persistence. The runspace doesn’t need to be cleaned up when it’s done with a task. You can also give it lots function definitions and just ask it to execute them as needed. The problem I ran into was that it was a lot harder to get the runspace to communicate results and data back up to the parent runspace than it was to communicate downward.

Anyway, I hope you like it and please let me know if you think there are ways to improve it! You can reach me on Twitter @RandomNoun7

I was on Twitter the other day and saw that @juneb_get_help tweeted asking people to talk about things they had built with Powershell Studio. I responded, but I thought it really deserved a blog post to talk about the app and the kinds of things you can do with it.

The Business Problem

The company I work for installs software inside their clients’ networks. We provide the client with a detailed spec of what we would like the servers they give us to look like, but it is very common for servers to be just a little bit wrong. Since we use Puppet for configuration management, it’s important to verify that servers are in the correct state before we accept them.

The app I wrote is dropped onto a single server in the environment, and when provided a list of servers to test, invokes a series of remote jobs that instruct each box to test itself for correct state and report back the results.

Why Powershell Studio

As we will see shortly, Powershell studio is a good solution for this because it allows you take advantage of the convenience of Powershell for administering servers, while also taking advantage of event based programming and easily wrapping scripts in a GUI that suits the needs of slightly less technical users.

The App

In the screenshot above you can see the design view for the app. The app consists of a main form, a tab control with tabs for the stages of testing, with sub controls for data. In the tab you see here I have a couple text boxes for Active Directory account names. Before the machines are tested I use the app to ensure that the Active Directory accounts we asked for have been created.

Notice on the right side the name says textBoxSQLAccount. That is going to be the name of a variable created by the Studio to reference that text box.

In this screenshot we are adding an event handler to the lower text box. When you click ok you get a code block for the object’s event handler.

If you’re familiar with Powershell syntax you will recognize a variable with a code block. In the background, when Powershell Studio builds the app, it ensures that code is executed when that event is fired just as the name implies. To keep things neat, the code is factored out into a function and I simply pass the textbox, since both boxes need to run the same check when the Leave event fires.

One of the cool things about Powershell Studio is how the really great auto complete incentivises you to write good code. The function Validate-Textbox was defined with a parameter of type System.Windows.Forms.Textbox. Powershell Studio knows it, so when auto complete comes up, it only shows me the variables of the correct type.

With these event handlers in place, every time my cursor leaves the text box the Validate-TextBox function is called with the current text box as the parameter, which then passes the text value to Verify-ADObject. If an object is returned we know it exists and the textbox turns green, and if not we get a red box. Since this is a demo app this is enough, but in reality we would want some checks in place to ensure such things as that a value actually exists in the text box in case someone just clicked on the text box by accident and then left.

Some of you at this point may be wagging your fingers at me, and believe me I know. I shouldn’t be making network calls on the UI thread. I don’t disagree, but I’m also lazy, and this check happens very quickly so I’m not worried about it for these text boxes. In the next tab though I’m going to do it right.

This tab is going to install the roles we need on the server. In reality Puppet can take care of this for us, but this is a decent way to demo the next concept I want to cover. Combining Powershell jobs and event driven programming we can spin off long running processes into a background thread that keeps our UI from blocking while the servers go about their business.

The IIS tab consists of little more than a DataviewGrid control, a progress bar and a button. After we add servers to the list in the dataview control we handle the button’s Click event to start installing the Web Server roles that we need.

Earlier in the form code that you don’t see here I created a timer object, but didn’t start it. In the next code section I’ll get the list of servers from the DataGridview control, spin off the jobs to the background, and then add a code block to the timer’s tick event before starting the timer. The effect is that the invocation of Begin-Install completes and the UI thread is unblocked. In the background however, the timer is still running and with each tick it calls a code block that checks on the status of the jobs we spun off in the background. The Get-JobStatus function that gets called knows how to find the Datagridview control and update the appropriate rows as the servers report their status back to the background job. Here’s the code.

In the real version the process of just finding the list of servers and spinning off the jobs can take a noticeable amount of time, so really the entire operation should be in the background, but for a demo, just spinning off the remote portion makes it easier to follow what’s happening.

Troubleshooting

One of the advantages to using Powershell studio forms apps like this is that it allows me to export the entire app not as an executable finished product, but just as a big long script. The reason this can be nice is if there are bugs with the program, I don’t have to install the entire Powershell Studio on a client machine to debug. I export the script, set my break points in Powershell ISE, and I can debug on the clients machines using only the tools freely available on any Windows Server. With that in mind, the entire demo app is pasted below as a script. It’s very very rough, just thrown together over the course of a few hours to make this blog post happen, so please don’t there are bugs, and there’s no error handling, etc, I know. but if you want to let me know of any better ways to do this stuff, or just have general thoughts, don’t hesitate to let me know. You can find me on twitter @RandomNoun7

Have you ever had to transfer files over FTP to and from Windows and Linux systems and had to deal with an administrator that just could not comprehend why the files he or she is giving you aren’t coming out right?

Maybe they are transferring the files to you and they keep showing up with no line breaks. The problem if course is pretty simple on the surface. Windows uses Carriege Return (Char(13))/Line Feed(Char(10)) together to represent line breaks, while many other operating systems only use CR. Of course if you’re not very lucky you are dealing with an OS or an import program that expects something even weirder like RS(Char(30)).

So yeah, the problem is simple to understand, and usually simple to fix. Use ASCII Transfer mode in FTP instead of Binary and most of the time the problem goes away. But what do you do when the admin on the other side doesn’t believe you that the file is fine from your perspective? Or maybe they insist that they DID FIX IT!!! How do you convince someone that their encoding is wrong, or that yours is fine? Maybe it’s not worth the struggle. How do you fix a file so that they can consume it no matter what?

With UTF32 it gets a little more complicated. We start with the byte order mark and then have a bunch of unused bytes between each letter. Our Char 10 and 13 are still there though. We’ve taken up more space but this is still a fairly plain windows file.

Lastly we see what the bytes look like in a file that looks fine to the linux admin but looks like just a blob of text to us. This simple file is easy to fix manually, but if you’re trying to set up automated data imports on a Windows system, this can be a real pain.

Fixing the File

So now that we’ve seen how we can inspect the file, what can we do if the admin on the other end just doesn’t know how to fix this. And by the way, this doesn’t always mean they are incompetent. I’ve been told by a very smart admin that getting this right transferring in and out of AIX is just hard.

That last command really shows us how we can make the other guys life easier for very little effort on our part. If you aren’t super familiar with Powershell it’s worth looking at exactly how it works.

Get-Content reads a file’s content, but it will break up each line into a discreet string object, stripping its line endings in the process. The syntax forces the entire file to be processed at once and the newly created array of string objects is handed off to the -join operator. We join by char 10 in this case to give us Linux line endings. We pass that resulting string off to Set-Content choosing ASCII as our encoding (encoding can be whatever the recipient wants), ensuring that we use -NoNewline so we don’t get a Windows line ending appended at the very end of the file. Now you can do a binary file transfer and the Linux system is happy.

Need to terminate lines with a “~”? yeah I’ve seen it. Just use -join [char]126. Any crazy line terminator they want, you can provide.

This also gives us insight into how to fix Linux line endings that they can’t figure out how to fix for us.

Get-Content c:\BrokenFile.txt | Set-Content c:\FixedFile.txt

In this case we take advantage of the fact that while many older Windows programs adhere slavishly to Windows Cr\Lf line endings, Powershell really does attempt to be smarter, so it has been designed such that many commandlets like Get-Content understand Linux line endings by default. Again it strips the line endings as it breaks the files lines into an array of strings. As those string objects are passed on to Set-Content though, it adds them to file one at a time, but this time it uses the standard Windows line endings, and just like that, a file that just a second ago looked like one long line of gibberish is fixed.

One last thing, just to save you a minute of frustration, notice that I did not write to the same file as I read the data from. When Powershell starts reading a file and breaking the lines into string objects, the first of those strings will reach Set-Content before Get-Content has actually finished reading the file. If you want to convert a file in place you will have to stage the data somewhere else first, or Set-Content will just encounter a file that is still open for reading and throw an error that it can’t write to the file because it’s still locked.