Monthly Archives: June 2016

I’ve been diving back in to – well, dipping my toe in the chilly waters of – PowerShell for some scripting here at my Data Processing job.

Several years ago, I learned the hard way (i.e., after writing a couple hundred lines of Ruby script) that although much of our processing automation was written without unit tests, that does NOT apply to any automation that *I* want to write. Not if I want to put it into production, that is.

I resisted unit testing and TDD for some time (Why? Well, that’s a story for another time), but I finally got testing religion last year with some Python scripting.

I could continue with the Python, but I think PowerShell is a better fit for our environment here.

Most modern programming languages have a choice of testing frameworks to choose from, but for PowerShell there’s only one that I know of – Pester.

Matt’s blog is subtitled “Tales from an Automation Engineer”, so his perspective on testing is a little different from the usual software testing guru. In particular, he points out that when it comes to infrastructure (and Data Processing, IMO), the things that are mocked / “stubbed out” in most software development environments are the things that we want to test:

But infrastructure code is different

Ok. So far I don’t think anything in this post varies with infrastructure code. As far as I am concerned, these are pretty universal rules to testing. However, infrastructure code IS different…

If I mock the infrastructure, what’s left?

So when writing more traditional style software projects (whatever the hell that is but I don’t know what else to call it), we often try to mock or stub out external “ifrastructureish” systems. File systems, databases, network sockets – we have clever ways of faking these out and that’s a good thing. It allows us to focus on the code that actually needs testing.

However …if I mock away all of these layers, I may fall into the trap where I am not really testing my logic.

More integration tests

One way in which my testing habits have changed when dealing with infrastructure code is I am more willing to sacrifice unit tests for integration style tests…If I mock everything out I may just end up testing that I am calling the correct API endpoints with the expected parameters. This can be useful to some extent but can quickly start to smell like the tests just repeat the implementation.

Typically I like the testing pyramid approach of lots and lots of unit tests under a relatively thin layer of integration tests. I’ll fight to keep that structure but find that often the integration layer needs to be a bit thicker in the infrastructure domain. This may mean that coverage slips a bit at the unit level but some unit tests just don’t provide as much value and I’m gonna get more bang for my buck in integration tests.

Matt’s opinion accords with my intuition about my Data Processing environment. In the DP realm, the part of the script that can be tested without accessing the production environment (or at least a working model of the production environment) can be trivial. This is probably the main reason our existing production automation doesn’t have full testing coverage. (Well, that, and the fact that as far as I know there’s no testing framework for the automation software we use).

So I think my approach will be something like Matt’s – unit test where it’s useful and non-trivial, and more integration tests (a “thicker layer” as Matt says) to get full (or at least adequate) coverage.

I’m trying to develop some scripts to handle data on some of the web servers we push data to at work.

I’m using PowerShell, because I can be fairly sure it will be available on the local hosts that we connect to the web servers from.

There is a “PSFTP” module that wraps .NET calls in PowerShell commands, but it’s taken a few searches to find the best way to use it in our environment here.

The Technet article shows examples of using the PowerShell commands in the module, but the way the example sets up the initial FTP connection opens a dialog box (outside of PowerShell) for the user to enter the FTP password. This is ok for the example, but wouldn’t work for automated FTP applications.
Happily, I found a WindowsITPro article that shows how to use the PowerShell ConvertTo-SecureString
cmdlet to create the FTP credential using a stored password in plaintext (or from a configuration file), rather than having to be logged in to the server to enter the password and allow the script to continue:

Using the ConvertTo-SecureString cmdlet, I’ve been to create a connection and list the contents of the remote (FTP server) directory.

One thing I’ve had to look at with the Technet and WindowsITPro examples is the “-UsePassive” parameter on the Set-FTPConnection command. I’ve never had to be concerned with the difference between Active and Passive FTP modes before, but it appears that Passive mode has problems with the way our firewall is set up. I found a StackOverflow article about the difference between Active and Passive FTP, but I will probably need to keep consulting the article until I’ve really got a handle on it.