Our 6 module course on Building the Test Automation Framework starts with Amazon Web Services (AWS). We’re going to use AWS, and more specifically, the Elastic Compute Cloud (EC2) to build our test environment and automation system. It all starts with configuring and running up the virtual machines we need to run everything on.

If you haven’t read our introduction to Building the Test Automation Framework you can read more here:

This first module is all about understanding Amazon Web Services (AWS) and Elastic Compute Cloud (EC2).

We’re going to cover six key areas:

First, creating an AWS account (you can skip this part if you already have one)

Secondly, configuring two security groups.

Thirdly, creating a security key pair.

Fourthly, we’ll run up two Virtual Machines (one Windows master and one Unix slave)

Fith, we’ll install Putty for our Secure Shell (so we can connect from the windows machine to the unix machine)

Lastly, Monitoring Usage

The last few parts we’ll look at checking your AWS usage and making sure you run your virtual servers down. We want to avoid you getting charged for using these Amazon virtual machines (more on this in a moment).

Why Use Amazon Web Services?

As a tester why would you want to use Amazon Web Services (AWS)? Well it’s fast to get setup, it’s scalable and it’s cost effective. Let’s address each of those points in turn.

So you need a new environment to test in. If don’t already have the hardware you’ll need to purchase the hardware, install the operating system, configure the network, etc, etc. If you have an AWS account all you need to do is pick a machine image and fire it up. No purchase, no ordering, no hardware configuration and no operating system install. It’s all there in minutes.

The AWS Compute side of things isn’t called Elastic Compute Cloud (EC2) for nothing. If you don’t need it you run it down. If you need more of it you scale it up. What you need is there, when you need it – on demand. No acquiring physical machines that are obsolete within a few years.

The whole concept is you use, and pay for, what you need when you need it. The traditional model is that you purchase a PC and pay the full cost up front. If you only use your PC for half the time (e.g. you don’t use it throughout the night) you don’t pay half price for it. You still pay full price. And then there’s all the hidden costs, like network infrastructure, power, racking, cooling, etc. This all adds up, but you never see it. Okay, you can rack up big costs fast with AWS if you’re not careful. Manage it well though and you’ll save money.

Yes this is a new way of working (well not even that new anymore), but it’s a smart way for testers to work.

Prerequisites

You don’t need a lot to cover this or any of the other module in this series. However, you will need the following:

1. A windows desktop or laptop that runs RDP
You can check if you have RDP on your machine with:

press the Windows button + r

in the Run window type ‘mstsc’

this should bring up the Remote Desktop Connection window

2. A credit card
No, this course doesn’t cost you anything. However, to sign up for an AWS account, even if you stick to using the AWS free tier, you’ll need to provide credit card details. Sorry that’s not our requirement – it’s Amazon’s requirement.

We have designed this course so that you use the Amazon Free Tier. So even though you have to provide your credit card details you should not be charged. HOWEVER, a word of WARNING! It is YOUR responsibility to monitor your usage and keep track of your AWS resource usage. If you go above the free tier allocation Amazon WILL BILL YOU. If this happens it will be YOUR bill to pay. We can’t be held responsible for any charges you incur! Sorry.

What You’ll Learn

In this module, covered by the 11 parts listed below, we’ll go through the practical aspects of using AWS. By the end of the module you’ll understand the following concepts:

What will your setup look like when we’re finished?

Over the course of the 6 modules you’ll get to this….

For this module you’ll run up the Windows master machine (that we’ll run Jenkins on later) and one slave Unix machine (that we’ll use to build and install the application under test on). Don’t worry if you’re not familiar with Unix or you’re not familiar with Windows. We’ll cover all the steps you need. By the end of this module we’ll have created the following:

You’ll start out with a master/control machine, running Windows Server 2008 (yes I know that’s old but it’s less resource hungry and delivers everything we need). This machine will be running Jenkins which will control all the other machines. You’ll access this machine using RDP from your desktop or laptop.

Then you’ll add a Unix (Ubuntu) machine where we’ll build, install and test our Application Under Test (AUT). Normally you’ll have build and test on different machines but for this module one machine will do. The application we’ll be working with is Rocket Chat (see https://Rocket.chat). More on why we’ve chosen Rocket Chat back in the introduction post here.

Over time, and as you progress through these modules, we’ll be adding various other nodes/hosts to this network. They’ll include Selenium, JMeter and SoapUI nodes.

So we’re looking for a distributed network of nodes that will run within their own Amazon Virtual Private Cloud (VPC). We’ll make sure the machines have public DNS records so that we can access these machines from outside of the VPC. Within that VPC you’ll configure the security groups so that these machines only allow access for the services we need. Finally, we’ll connect both machines (Windows and Unix) by installing Putty (a Secure Shell client) on the Windows machine. Putty will allow us to SSH from the Windows machine to the Unix machine. If you’ve not come across Putty and SSH we’ll cover it all in just a moment.

For this module we’re only going as far as setting up the AWS virtual machines. We’ll be looking at installing Jenkins and setting up the AUT in the next module.

Let’s get started!

Part 1: Creating an AWS Account

We’re going to build our test system using Amazon Web Services. This allows us to spin up virtual machines in the cloud on demand. Once you’ve signed up for an AWS account you’ll have access to all of these services through the AWS dashboard:

You’ll only need the EC2 service (along with the assoicated EBS storage, security and machines images) for now. So how do we create our AWS account?

If you already have an AWS account you can skip the following and jump to AWS Fundamentals.

Be warned though……. if you have an existing AWS account you may be out of your 12 month Free tier period. Or you may have nearly exghasted all of your free tier resources. So Amazon may charge you for running up the virtual servers covered here. It’s YOUR responsibility to monitor and, if necessary, pay for any usage above the free tier. If you don’t know how to monitor this we show at the bottom of this module.

And that’s it. At this point you should have an AWS account, you should be logged in and see the management console that lists all of the available services.

Part 2: AWS Fundamentals

These fundamentals are all you’ll need to complete all of our modules in this series. It’s all you’ll need to get to productive with your testing on AWS. Trust me, despite all the different services and options, it’s easy to get started with AWS. You just need to grasp the following:

Services: Amazon provide a range of different services. A service can be thought of as the type of work a particular cloud resource provides. These services are grouped into categories like Compute, Storage, Database, Networking and others.

As an example, in the ‘Compute’ category, we have the EC2 (Elastic Compute Cloud) service along with other services like VPC (Virtual Private Cloud). In the ‘Storage’ category we have the EBS (Elastic Block Store) service along with other storage services like S3 (Simple Storage Service).

All of the services available are listed in the first page you see when you login to AWS:

For the purpose of these modules we’ll be focusing on EC2, EBS and VPC. Each of these are described in more detail below.

Zones and Regions: are used to reduce latency between the end users and the services that are provided. Amazon provisions these services in data centers located in different regions. For example services are provided from locations like US East (N. Virginia), EU (Ireland), Asia Pacific (Singapore), and many others. Note that not all services are provided in each region.

When you start services you’ll want to make sure that you have the correct regieon selected so that the service is started in your regeion. Also, and this is important, when you create VPCs, Security Groups and Key Pairs these are all linked to a region. Everything you need should be defined and created in the same regeion. So it’s best to select the region up front and stick with it.

Management console: the AWS management console gives you the ability to start, configure and manage the services you need. For example you can spin up an EC2 instance and from the management console start an RDP (remote desktop) session on that EC2 instance.

The key components of the management console are highlighted in this image:

EC2: is an abbreviation for Elastic Compute Cloud. Essentially EC2 is the service that delivers resizeable computing capacity. When you select EC2 in the management console you are given the capability to start, configure and manage virtual machines in Amazons cloud. These virtual machines are known as ‘Instances’.

EC2 Instances: from the management console you can run up virtual machine instances running all sorts of different operating systems, with different hardware platforms and with a range of different default software installed. Each virtual machine you run up is referred to as an ‘instance’. For example you can run up a virtual machine instance that has Windows 2008 Server, with 2 CPUs, 4 GB memory on a 64 bit architecture.

EC2 Instance Types: Each ‘Instance’ you run up will be of a specific type. A ‘Type’ defines the the CPU, memory, storage and networking capability of the ‘Instance’. Types typically range from ‘T2.nano’ (low capacity on all fronts) to different families of ‘Large’. In the example above we have an instance Type of ‘t2.micro’ which has 1 vCPU and 1GiB of memeory.

Images and AMIs: each time you start a virtual machine you don’t want to have to install and configure the operating system and additional software from scratch. To save you building the instance from scratch Amazon give you the capability to start the machine with a predefined image already installed. These images are known as AMIs (Amazon Machine Images). AMIs define the operating system, architecture (32 bit or 64 bit), launch permissions and storage for the root device.

Elastic Block Store and Volumes: a virtual machine is no good without storage. Amazon provide a huge number of options here.

The most common data storage service is known as EBS (Elastic Block Store). EBS is a service that provides storage volumes that can be attached to running ‘Instances’, the data stored is persistent (e.g. data is retained across reboots, shutdowns, etc) and each EBS volume can persist independently from the life of the ‘Instance’ if need be.

Also available, and commonly used, is Instance Store storage. This storage service is automatically provisioned with some EC2 instances and AMIs. The defining feature is that the storage is physically attached to the host machine. You should be aware though that Instance Storage is NOT persistent. Data is not retained across shutdowns (stopping or terminating a machine).

For us, we’ll be using EBS. EBS because the data stored is persistent. Also, EBS is automatically provisioned on some AMI’s. The AMI’s we’ve selected for this course all have EBS storage (e.g. so you’ll have storage that is automatically setup and that will persist when you stop or terminate the machine).

You can see the storage type you have associated with your Instance here:

Network and Security: when you start creating instances from your Amazon management console you might notice that Amazon creates a Virtual Private Cloud (VPC) for you automatically. Every Instance you create is placed in your VPC automatically too. You will find you click on the console home icon then select VPC you’ll be taken to the VPC dashboard. If you didn’t notice then this is where you’ll see details for the VPC that your instances are placed in..

Pretty much all of the setting here you don’t need to worry about. AWS takes care of all of this for you. However, there’s one setting ‘DNS Resolution’ that you will need to update and set to ‘yes’. We’ll talk about how and why later but this makes all the Intances you create visible with public hostnames and Ip addresses outside of your VPC. More on this later.

Once Instances are run up you’ll see them associated with your VPC here…

This VPC is essentially the same as the networks you’d find for you physical PCs and laptops on in your office. Just be aware that by default your instances are NOT given public IP addresses and host names. Everything is defined by default with local VPC IP addresses and host names. Again we’ll look at configuring the VPC to provide public IP addresses and host names in a moment.

Security Groups: Built in to the whole AWS framework are some pretty clever security capabilities. Key to running our instances is the concept of Security Groups. Each security group contains a set of fire wall rules. Each time you start an instance you specify the security group you want to associate to the instance and the instance inherits these firewall rules.

So for the example above when we run our AUT (Rocket Chat) on one of our instances the AUT needs the rules defined to allow access to the AUT services that will be running.

What we’ll do for your setup is define a couple of security groups that contain rules for each Instance that you want to setup. Then each time we start and instance that will be running either Rocket Chat or some other applications we’ll associate the required security group with the instance.

The concept here is that a public and (related) private key are created. Amazon encrypt the password with the public key. Amazon provide you with the encrypted password that no one can decrypt. That is unless you have the associated private key. Assuming you do have the private key can then decrypt the encrypted password.

So to log in to any Windows or Unix Instance you create you will need to create a public and private key pair first. Only once you have the key pair can Amazon encrypt the login details, and you decrypt the login details.

It’s slightly different for Windows and Unix Intsances here. For Windows instances it’s the login password that is encrypted and used to login. For Unix instances it’s the key pair that’s needed in conjunction with SSH (Secure Shell) to login (there are no passwords for Unix machines).

More about all of this in a moment. For now though just be aware that you can create your public and private keys in the EC2 dashboard in the management console here…

We’ll go into much more detail about this in a moment. Just remember that once you have the private key you MUST NOT lose it. You can ONLY download the private key you need ONCE!!!! Don’t lose it!!!!

So what’s next then?

Well we understand the fundamentals so lets start running up our instances and building our test system. First we’ll need to configure some security groups and setup a security key pair. Once this is complete we’ll be ready to run up our Windows 2008 instance and our Ubuntu Unix instance.

Part 3: Configuring Security Groups

First make sure you select the right region (select this from the management console top right). Security Groups are created FOR each region and can’t be used across regions.

Once you’ve selected the region we need to create 2 security groups. The first will be used for our Windows master machine. The second security group for our Unix machine that is running Rocket Chat (the application under test).

You can configure these two security groups by following these steps….

What this will give you is a machine that lets you access Jenkins which is web based serving up HTTP and/or HTTPS traffic to your local browser. It will also all SSH (Putty) access so that you can create a terminal on the Unix instance you’ll be setting up. Finally RDP access is provided so that you can create an RDP session from your local PC or laptop.

Note that we’re selecting a ‘Source’ of ‘My IP’ so that only YOUR MACHINE will have an inbound connection to the instances we run up that use this security group. No other machines will have access (note that if you’re on a laptop and you move locations you may end up being blocked yourself). You can leave the source as 0.0.0.0/0 for each rule but it’s less secure.

What this will give you is access to the Rocket Chat AUT that serves up HTTP/HTTPS traffic to your local server. The Rocket Chat application also needs to provide and consume data from the the Mongo Database that is accessed on port 27017. It will allow SSH for terminal access from the Windows master machine. That’s all we need for now.

Just as a note you’ll see under security groups that by default you already have one security group created. This group will have a source id that refers to the same security group id (e.g. sg-xxxxx). It’s a circular reference if you like but just means that any instances in this security group can access any other instances in the same security group.

Once you’ve completed this you should have something like this…

Part 4: Creating a Security Key Pair

Again, make sure you have the right region selected (select this from the management console top right). Key pairs are create FOR each region and can’t be used across regions.

Now you need to create your security key pair. The private key you obtain from this process you’ll need to keep safe as we’ll need it later in the process. It’ll be needed to decrypt the windows password and to login to the Unix machine using SSH/Putty.

To create your key pair follow these steps:

1. on the EC2 Dashboard page click on “Key Pairs” menu item
2. on the key pairs page click the ‘Create Key Pair’ button
3. give the Key Pair a name (e.g. FirstKeyPair)
4. at this point you should be prompted to download the .pem file
5. click okay and save the .pem file somewhere safe

DON’T LOSE THIS FILE!

You can open this .pem file with a text editor (e.g. notepad) if you like. You’ll see it’s just an RSA Private Key. Dont’ worry about this for now though. You just need to know that we’ll need this later when we want to get access to our instances.

Now we’re ready to start the fun stuff!

Part 5: Running up the Windows Instance (VM)

There’s four things we need to create our Windows Instance:

AMI – Microsoft Windows Server 2008 R2 Base – ami-c5a7bea4

Instance Type – t2.micro

Security Group – Windows-Master (we created this earlier)

Private Key – .pem file (from our key pair created earlier)

Next then, on the EC2 console/home page click “EC2 Dashboard”. You should see a resource list like this….

From here you click the ‘Launch Instance’ button. You should be able to follow the steps using the ‘four’ items of info listed above to create your first instance. Just accept all the defaults as you go through the steps.

At this point you’ll be asked to select a Key Pair. We’ll use the Key Pair we created earlier. The public key from this pair will be used to encrypt the windows password for the instance that is about to be created. If you don’t have the .pem file with the private key in you’ll never be able to decrypt the password and you’ll never gain access to your instance. You’ve to the private key safe right?!

Next then, go back to the EC2 console/home page click “EC2 Dashboard”. From here you click the ‘Launch Instance’ button again. You should be able to follow the steps using the ‘four’ items of info listed above to create your second instance. Just accept all the defaults as you go through the steps.

At this point you’ll be asked to select a Key Pair. We’ll use the Key Pair we created earlier. The public key from this pair will be used to encrypt the windows password for the instance that is about to be created. If you don’t have the .pem file with the private key in you’ll never be able to decrypt the password and you’ll never gain access to your instance. You’ve to the private key safe right?!

* This instance does not come with EBS storage. It comes with SSD storage. That is we have storage that is not Persistant. When we ‘stop’ or ‘terminate’ this instance we’ll lose all our data. That’s fine for what we want to do but just be aware that nothing is retained on this instance.

If you’d like a more detailed account of how to launch a Linux Instance I’d recommend this:

If you go to the EC2 Dashboard now you should see something like this…

And if you click on ‘2 Running Instances’ you should see a list of your instances like this….

Now we’re ready to login to the windows master machine, install putty and then connect to the unix machine. At which point it’ll be job done. So not much left now.

Part 7: Connecting to the Windows Master Machine

Pretty straight forward to connect to the Windows machine. Just follow these steps:

Right click and select ‘Get Windows Password’

Select the path to the ‘Private Key’ you saved earlier

Click the ‘Decrypt Password’ button

Write down the Windows Admin password and the Public DNS host name

IMPORTANT: If you don’t get a public DNS host name you’ll have to complete these steps and change this setting:

Click ‘Services’ (top menu bar)

Select ‘VPC’ (in the drop down menu)

Select ‘Your VPCs’ (in the side menu)

Right click on your VPC entry

Select ‘Edit DNS hostnames’ (in the context sensitive menu)

Select the ‘Yes’ radio button and then ‘Save’

Then return to ‘Services -> EC2’ and your list of running hosts

At this point you should have the IP address, Hostname of your Windows server, your user name (Administrator) and your password. You can right click on your ‘Windows Instance’ entry in the list and select ‘Connect’

Open the ‘Remote Desktop File’. When you open the file with the default application ‘Remote Desktop Connection’…

… you should then be able to connect using the credentials you have. With the RDP session established and access to the Windows server desktop we’re in a position to start installing Putty and connecting to our Unix machine.

Please note that you’ll need to connect using this method each time you need access. Don’t bother saving the RDP file as this will have a specific Public DNS record / Host name. When you restart this machine the Public DNS / Host name will change. This won’t be a problem if you always connect following the steps above because Amazon takes care of updating the RDP file with the correct IP address and host name each time.

Part 8: Installing Putty and SSH on the Windows Machine

Not sure if you’ve come across Putty yet but this is a great little (actually not that little if you consider how much has gone into this suite of tools) application for connecting via a secure shell from a windows machine to a unix machine. If you set it up correctly you can open a Shell session (command prompt) on the Unix machine at a click of a button without even entering a password. Here’s how:

Open Internet Explorer within the RDP session on your Windows server

Either search for Putty or enter this URL

http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

(MAKE SURE YOU GO TO THE ‘chiark.greened.org.uk’ SITE)

Download the Putty Installer: putty-0.66-installer.exe

Run the installer selecting all the defaults

At this stage you should be able to see all the Putty tools in the Start menu. The tools we’ll be needing are…

PuTTYgen: allows us to convert our .pem key (see below)

Pagent: authentication agent that runs on the windows machine

PuTTY: Secure Shell client that connects to our unix machine

Part 9: Connecting to the Unix Client Machine

Three very straight forward steps to getting connected to our Unix machine. Very important we set it up correctly though. First, because it makes our life easier (quickly creating a terminal session on the unix machine) and secondly because later our Jenkins setup will connect automatically from this Windows machine to the Unix machine using SSH. SSH is a key component in our process of automating everything.

Step 1. Convert our .pem key
So the .pem private key Amazon gave us for connecting to our Windows and Unix machines isn’t supported by default by Putty. But it’s easy to convert it to the right format. Just….

On your desktop/laptop copy your .pem file (e.g. FirstKeyPair.pem)

On the windows server paste the .pem file to the desktop

On the windows server start ‘PuTTYGen’

In ‘PuttyGen’ select SSH-2 DSA

Then click ‘Load’

Load the .pem file (making sure you select ‘All Files (*.*)’

Enter a passphrase and click the ‘Save private key’ button

DON’T FORGET THE PASSPHRASE YOU USE!

Save the new .ppk file to the desktop (call it FirstKeyPair.pkk if you like)

This should leave you with the following icon on the desktop

Step 2. Run Putty Agent
Now we have our .pkk file that Putty can use we setup Putty Agent. Putty Agent runs in the background and holds our private key (.pkk file). When you make a connection to the Unix machine Putty take the key from the agent and uses it to establish a secure connection. Don’t forget that when AWS created our Unix instance it used the public key as part of the process for setting up the connection credentials. So when we try to connect using our private key it should work in conjunction with the other half of the key pair, the public part of the key pair. So lets setup Putty Agent:

Double click the putty icon on the desktop

Enter your Passphrase and click okay

Check you have the putty agent running

So you should see Putty Agent running in the task tray. If you double click on the icon you should see that your private key is loaded.

Now all we need to do is open the connection up to the unix machine.

Step 3. Start Putty and connect with SSH
Now we’ll start Putty and connect to our Unix machine. We’ll configure Putty correctly so that it’s easy to connect each time we need a terminal open on our Unix machine. Complete thesesteps:

From the Start menu start Putty

From your AWS Management console find and copy your Unix Private IP

Paste the IP Address into the Putty window

Make sure SSH is selected

Enter a saved session name (e.g. Unix-client)

Click ‘Save’

Click ‘Connection -> Data’ in the side menu

Enter ‘ubuntu’ for the Auto-login username

Click ‘Session’ in the side menu

Click the ‘Save’ button again

Click ‘Open’

At this point you should be prompted with a security alert. First time round it is valid to select ‘Yes’. And from here you should go straight into a SSH shell session on your Unix client machine with NO login required.

At this point you can just type ‘Exit’ and return in the shell window.

When ever you need a connection to this machine you can just carry out the followign action….

This will take you straight into the SSH shell prompt on your unix machine (no password required).

To be honest we don’t have much requirement to actually use the shell prompt for what we need. However, Jenkins does require SSH setup and configured to use it as a build and install node. If this is working we should be okay for the next stage.

The next stage then is installing Jenkins on the Windows machine and setting the Unix machine up as a Jenkins node. Before we jump to that though, a couple of small points…

Part 10: The Difference Between AWS Terminate and AWS Stop

You’ll notice in the management console that you have 2 options for bringing your servers down (both for Windows and Unix).

Stop: when you stop an instance the instance is shutdown. If it has EBS storage (like our Windows server does) the data on this storage is maintained. If you have SSD storage (like our Unix server does) the data on this storage is NOT maintained. When an instance is shutdown you can restart the instance when you need it again. Things like intance ID, EBS storage, private DNS and private IPs are maintained and restored. Things like Public DNS and Public IPs may change (they will on our setup).

Note that when an instance is in a ‘Stopped’ state you are not charged for is use. However, any EBS storage that is maintained you will be charged for. If you don’t want to be charged then you will need to ‘Delete’ the volume OR Terminate the instance.

Terminate: If you terminate the instance everything is deleted. Terminate an instance if you no longer need it as you can NOT restart it or connect to it again. So only use this if you don’t need any of the data anymore. You can also use this option if you want to make absolutly sure you are not charged anymore. Any EBS storage you have when you terminate an instance is deleted too (so you won’t be charged for EBS storage after you’ve terminated either).

NOTE that during this course we…

– DO NOT want to terminate the Windows instance. We need to retain the data on this machine as it’ll be our master machine running Jenkins.

– We don’t really want to terminate our Unix instance either. However, on this machine no data will stored that we need to keep. So if you do terminate it that’s okay as we can just run up another instance with the same AMI.

– We recommend that you ‘STOP’ instances when you are not using them. This will mean that you are not charged for the instances. However, YOU WILL continue to be charged (if you go outside of your free tier allocation) for EBS storage that is retained.

For this reason we strongly advise that you monitor your AWS spend.

Part 11: How to Check Your AWS Spend

Within the AWS management console you’ll find a ‘Billing and Cost Management’ option:

In here you’ll see what your current balance is (should remain at $0.00 if you stay within your free tier constraints). The most important section though is th e’Top Free Tier Services by Usage’ section.

Monitor the stats here to see how close you are to going over your free tier allocation. If you’ve started with a clean AWS account for this course you shouldn’t go over your free tier allocation.

IT IS YOUR RESPONSIBILITY TO MONITOR THIS. CHARGES INCURED ARE YOUR CHARGES TO PAY.

If you want to be absolutely sure that you don’t go over the free tier allocation then I would recommend reading this….

Conculsion

So we’ve started with a new AWS account. Then we went through the sign up process to create a new account. We’ve learnt about the AWS fundamentals covering Virtual Private Clouds, Elastic Cloud Computing and storage. From here we created our first Windows and Unix instances from Amazon Machine Images. To finish off we set up our servers so that we have RDP and SSH access. In short we started out with nothing and now we have our own cloud environment with virtual machines and storage.

In the next module we’ll be installing Jenkins and looking at the process of automating the build/install of our application under test. Lastly, if you’d like notifications about this course and the Pdf course notes please feel free to sign up below.