Trying to prove that Skynet should be running on PowerShell!

Category Archives: API

Can you automate anything with Azure Automation?
While there are some limitations on what it can and cannot do, I thought I could have a bit of fun using some of the fairly new features in Azure Automation to show that even though the main purpose of this service is to automate management tasks in the cloud (and in your local datacenter using hybrid workers), since it’s built on PowerShell, there really isn’t that much you cannot do.

I’ve been trying to come up with a scenario that is a bit of fun, and at the same time shows how you can use features like hybrid workers and webhooks to overcome almost any obstacles you have when automating something that spans over different services and locations.

This is what I came up with:Let’s say I’m on my way home from work, it’s autumn and the rain is pouring down. I feel tired, cold, and just want to come home and grab a nice cup of coffee. To detect if something is running out of resources (in this case me), and to trigger something that can fix it (in this case, coffee) is a pretty common scenario in IT.

And to simulate that a process like this might span over multiple services where some are in the cloud (in this case iCloud, twitter and a weather service), while others are in your local datacenter (or in this case the coffee brewer in my kitchen), we are going to run parts of the code in Azure Automation and parts of it on a hybrid worker, everything orchestrated with Azure Automation.

The steps involved, at a high level, will be:

Fetch the location of my phone through iCloud

Use that information to fetch the weather data at that location, and check if it’s going to rain

If rain was detected, start with sending out a tweet asking if I would like some coffee, if the reply is positive, brew some coffee.

If no reply to the tweet is detected, it will send out an e-mail with a link to a page where the coffee brewer can be started with a button (that calls a runbook through a webhook)

So let’s get started!

So, where am I? And how’s the weather?
I don’t work at the same place everyday, so I don’t want to hard code the location where the weather is checked. Wouldn’t it be great if I could just fetch this information dynamically somehow? Well, with Azure Automation and some PowerShell, I can.

Since I carry around a smartphone with a GPS all day long, I thought that would be a good source for location details, and since you can fetch your location information through iCloud when you have an iPhone, this was the method I chose to do it.

Disclaimer: This code is for educational purposes ONLY, I do not take any responsibility if you use this outside of the ToS for the different services utilized here.

So I started with creating a PowerShell-function that could fetch my phones location through iCloud, if you want to take a look at it, it’s available here.

We then need to fetch some weatherdata at that location, luckily, I’ve already built a function like that before, blog post available here.

So, I have the tools to fetch my current location and the weather at that location. But how do to use this in Azure Automation?

Importing custom modules in Azure Automation
This is actually really simple! You can import almost any PowerShell module into Azure Automation, as long as you zip it up in folder with the same name as your module file. So I took my two functions above and put them into a WebUtilities.psm1-file. I then put that file into a WebUtilities-folder, and finally zipped it all up as “WebUtilities.zip”. If you want to learn more about how to create integration modules for Azure Automation, including creating an optional file containing information about a Azure Automation connection-variable, more information about that is available here.

We then need to import this into Azure Automation. The screenshots that follows are from the “classic portal”, but you can do this in the preview portal as well:

First find the automation account you want to use, go to assets, and then click “Import Module” at the bottom:

Browse to your zip-file and click open to select it and press “Complete” down in the right corner:

Azure Automation will then begin to import the module and extract the activities it contains, you can follow the process at the bottom of the page:

These functions are now available in our PowerShell Workflows and PowerShell runbooks. Neat huh?

(The custom modules you import will not, at the time I’m writing this, be pushed to your hybrid runbook workers automatically. The Azure Automation team is working on that though, so it will happen eventually. In the meantime, you need to do this yourself.)

Writing the code…
It is now time to use the functions and actually write the code needed to tie everything together. There are many cool new features regarding Azure Automation but one of my favorites are the PowerShell ISE AddOn the Azure Automation team is working on, if you work with Azure Automation I can’t recommend you to check out the GitHub repository for it enough, and ever since I did the build straight from the source it has been working pretty well considering it’s still a very early release.

This is how my setup looks (ISESteroids, another great product, is also used here):

In addition to enabling you to use all of the features of the PowerShell ISE (and ISESteroids if you use that), this AddOn enables you to for example; fetch your runbooks straight from Azure, upload changes, run the code locally with emulated activities, test the code in Azure, and manage your assets so they are available when you test the code locally.

The productivity boost you get from this in comparison to the text authoring and testing experience in the portal, at least in my experience, is huge. So go ahead and try it out!

So, back to the code itself. As stated above, the steps involved here will be:

Fetch the location of my phone through iCloud

Use that information to fetch the weather data at that location, and check if it’s going to rain

If rain was detected, start with sending out a tweet asking if I would like some coffee, if the reply is positive, brew some coffee.

If no reply to the tweet is detected, it will send out an e-mail with a link to a page where the coffee brewer can be started with a button (that calls a runbook through a webhook)

And since the PowerShell community is so awesome, this is a pretty common scenario aswell, you build a few functions of your own, and you find some from others. Just zip it up and import it in the same way as the above functions. To use the MyTwitter-module, you also need to add API keys, just follow Adam’s instructions and you’ll be fine!

If you have read some posts at this blog before, you probably know that I enjoy creating home automation scripts quite a lot, and I’ve named this little project Jarvis after the famous AI, the ‘JarvisGroup’ specified above (Start-AzureAutomationRunbook cmdlet) is the hybrid worker group that runs some of these scripts. If you want to learn more about hybrid runbook workers and how to deploy them, check out this link.

Currently, you can’t use webhooks to trigger runbooks on a hybrid worker, as a workaround, I have another runbook that uses the Start-AzureAutomationRunbook cmdlet to trigger it on the hybrid worker instead, the code of that looks like this:

The module containing the Connect-TelldusLive and Set-TDDevice cmdlets are installed on the target hybrid worker since that’s where it will execute (and as stated above, the module won’t be pushed out to hybrid workers automatically from Azure Automation even if you have imported them there, but that will be fixed in the future).

So, we’re all set now…

But, does it all work?
Well, you’d obviously have to come by for coffee some time to see this for yourself, but yes, it actually does! 🙂

The code for that form with the token masked (be aware that posting a form like this on a public website without authentication is a MAJOR security risk depending on the runbook type, it’s only for demo purposes in this case):

Summary
I hope this post have helped you to see how flexible Azure Automation actually is. PowerShell is truly versatile and a great “glue-language” to tie different services together. Even though using Azure for turning on a coffee brewer might be a bit overkill, if it’s possible to integrate a weather service, an iPhone, e-mail, twitter and a coffee brewer using it, it can probably manage your IT environment aswell, don’t you think? 🙂

Building an advanced function that can consume information on the web is pretty powerfull and I use it for all kinds of things.

In this post I will try to guide you through the process on how to build one for more or less any service, but the example will be the Swedish postal service.

I usually start with a web browser that has some developer features, for example Google Chrome. Go to the website and press Ctrl+Shift+i, select the “network tab” and enter whatever information you need to send to the service, in this case the ID of the package I want to trace.

In this example it should look like this: (I have chosen to use the English version of the website):

Press the submit button and look at the beginning of the network trace. You usually find a GET or POST request there, in this case it is a GET-request.
In this example it looks like this:

You can right click that row and select “Copy link address”, which in this case is “http://www.posten.se/en/Pages/Track-and-trace.aspx?search=MyPackageID”.

Now open whatever PowerShell script environment you prefer, for example the PowerShell ISE. Start with sending the same request from PowerShell, that can be done by using Invoke-WebRequest (if you are using PowerShell v3 or higher). Start with putting a variable where “MyPackageId” is.

The “UseBasicParsing” switch is not mandatory here, but if you don’t need the html returned to be parsed into different objects it is a bit quicker.

We now need to parse the html-code stored in the “Content”-property to get what we want. This can be a bit time consuming, but with a little help from Chrome it gets easier.

Press the magnifier button and hover the mouse over parts of the site or parts of the HTML-code (if you select the “Elements-tab”) and you will soon find what part of the HTML code you need.

In this example the table-tag. Screenshot:

Now we need to do some string manipulation to get the parts we need properly formatted. In this case we want to split the HTML to get the parts between the start of the table and the end of it. What we have left is the rows with all the package events, find something that splits them up in to nice pieces, in this case the “tr class=” tag. The first of the rows that gets returned are some table information (containing a unique ID that might change) and the table columns, so we want to skip those. A oneliner that does all of this looks like:

We can now loop through these items, parse them and build an object out of them. Each one of these items has three columns; a date, a location and a comment/tracking event. The columns are enclosed in the “TD”-tags so we can split them up at those.

When you have all the values we need we create the object and send it to the pipeline. Could look something like this:

If you haven’t heard of it, Infoblox is making appliances and solutions for enterprise networks. One of them is the Infoblox Trinzic DDI which manages DNS, DHCP and IPAM.

Infoblox does not seem to have any plans for delivering a PowerShell module for managing their product, which is a bit weird since it would probably make it a bit easier for network admins trying to move windows engineers away from Active Directory Integrated DNS, and instead let the Infoblox appliance host all or some of their zones (which seems to be a common enough debate among enterprises).
But in their defence they have released a REST-based API for this appliance which enables administrators to basically build whatever tools they want to manage anything from DNS records to IPAM.

I’ve therefore started to build a module for managing DNS-records hosted by the Infoblox, but since my experience with this product is fairly limited, I thought I would post what I’ve done so far to get opinions from people with more experience with this product.

I’ve only done a few cmdlets (advanced functions) for managing a few common DNS record tasks. So far these are:

How? Well, you need to find someone who sells groceries through a website, and can deliver them to your house.

The rest is just webrequests!

This is not only (but maybee mostly… 😉 ) made as a part of my weird quest for doing strange things with PowerShell, but also a small part of my home automation project. I’ll give you some usage examples:

Let’s say the weather-cmdlet posted earlier gives you reports about rain, wind and cold weather for the coming week, how about adding some popcorn to your basket and other things you want for a nice movie-night at home?

I just started on a new part of my home automation project, controlling my receiver (Onkyo TX-NR616).

I thought that this might be a good opportunity to do a “walk-through” on how to create a script for something that clearly was not meant to be controlled with PowerShell, in this case a surround receiver.

The basic methods and principles used here should be applicable to more or less anything you want to automate and I hope someone might find it useful!

Small disclaimer:
I take no responsibility in how you use the information in this post, make sure you know what you are doing! Even though any damage to a receiver or other equipment should be unlikely, I wont give you any guarantees…

So… The steps involved are basically:

Figure out how the item (website, device, software…) is controlled. Does it have a documented API? Is it using REST? Website? Mobile app?

Figure out how to utilize this with PowerShell. If it is a REST-based API, you are lucky, websites or webservices are also highly automatable (always check ToS before doing it though…), if it has a mobile app or similar, it most definitely is using some sort of API which may or may not be easy to figure out.

I usually build a simple script to start doing some testing. Once I figured out how the API works, I turn that into an advanced function.

Enjoy doing yet another thing with the blue console of wonder!

I will now try to describe how I used this basic approach to control my receiver!

So, how is it controlled?
In this case, I knew there was a mobile app available, and therefore, there must be some kind of API. After some research I found out that they are using something called “Integra Serial Communication Protocol”, or eISCP as it is called when used over an ethernet connection.

But what if we didn’t have any documentation? Or if we just want to use a few commands (like power on/off), do we really need to learn the complete documentation?

That depends, sometimes you can just “copy -> paste” the commands from a trace. That will be described under the next part of this post; “Using a network trace”.

Using a network traceDocumentation obviously helps a lot, but if we pretend there was none available, should you just give up? Of course not! 🙂

One approach if that is the case is to basically just sniff the network traffic. You can do this with whatever tool you prefer, two free examples are Wireshark and Microsoft Network Monitor.

If you can find a windows application for the device, that helps since you can just sniff your local network traffic. Another alternative if you want to stick to a windows computer but all you can find to control your devices is a mobile app, you could try to run routing and remote access (or internet connection sharing) to just make the traffic go through your PC, which then acts as a router. After that, your network analyzer/sniffer should pick up the traffic, which hopefully will not be encrypted (there are ways of solving that too, but we’ll stick to the simpler scenarios here…).

When you got this ready, just start your sniffer and your application used to control your device, and record whatever command you want to be able to automate.

This is a screenshot of a Wireshark trace recording a “power on” command sent to the Onkyo receiver:

Right click the first package sent to receiver and choose “follow TCP stream”. That makes it a bit easier to see what is happening, it looks like this:

The colouring makes it easier to see what gets sent to the receiver and what gets sent back, the first command (light red colouring above) shows what the app sent to turn on the receiver, the dots just represent “non printable characters”, so the command looks somewhat like “ISCP !1PWR01” which does seem like a “power on” command!

Now select “C Arrays” instead of Hex Dump, select the package you want and copy the text, like this:

You now have the whole command needed to turn the device on! Since what you will be sending is byte arrays and not HEX, you need to convert the HEX to Bytes instead. To do this, open a PowerShell prompt, paste the text you copied, add a pipe and write “% { [BYTE]$_ }”, press enter two times and they are converted! Should look like this:

Those bytes are what you need to send using the Send-method of System.Net.Sockets.Socket. More on that follows under the next part of this post!

Sending TCP streams with PowerShellWe now know what we should send, but how?
There are many examples around on how to do this, I will post one that at least I thought was fairly straight forward.

The code looks like this: (Credit to this thread for getting me started!)

And the receiver turns on! This code lacks quite a lot, like a method of reading the reply, closing the socket, error handling etc., but it will hopefully give you an idea on how to tackle something like this! If you want something slightly more polished, check the last part of this post…

But I want to do a lot of different things, not just turn it on/off!
I will fix a module for controlling Onkyo receivers as soon as I get more time, in the meantime I have made a “quick and dirty” advanced function that allows you to add the command to a parameter. To give you an idea:

If you look in the documentation I linked to above, you will probably figure out the commands for more or less anything. I have tested this on my receiver to for example use Spotify, change volume, change inputs and so on…

The module will probably include something like this function to enable a wider range of commands, but will also have cmdlets prepared for powering on, selecting different inputs (including subcategories under “Network”).

This function should just be considered an example on how you can do something like this with PowerShell, and was made since most of you might not be using Onkyo-receivers but still would like some sample code.

As a part of my home automation project I wanted to be able to check the weather, and since I’m trying to do as many parts of the automation as possible with PowerShell, I needed to write a cmdlet for this. But where to get the data?

I was very happy to find out that the Swedish Meteorological and Hydrological Institute (SMHI) actually has a published API, open and free to use!

The API is very simple, after you send a request with longitude and latitude it sends back a json-object with weather data for that location (tagged with the location of the closest weather station used for the report) for up to 10 days.

Note: As a visitor pointed out in the comments, SMHI does not have weather data for all countries, I’ve tested a couple of locations in Europe which seems to work fine though. You will get a “error 400 bad request” error if the location is out of scope.

The cmdlet does some small changes to the returned object (expands a few properties and changes some shortnames to something that is easier to understand, converting some values to percentages and so on…) and returns each forecast as a separate object. The forecasts can cover anything from 1 hour to a day or so, so make sure you check the start/end dates.