Tag: EC2

AWS EC2 instances can be used to create a HTTP proxy server, so when a client browser using the proxy browses the internet, the AWS EC2 instance’s public IP address effectively becomes their IP address. This may be useful for anonymity, for example if you’re browsing the Internet from home but want to mask your IP address.

Furthermore, you can even have the IP address of your AWS EC2 instance change, by releasing and attaching a new AWS Elastic IP to it, thus “rotating” the public IP of the HTTP proxy. This way you can achieve even more anonymity by using an ever changing IP address.

This is a guide on how to use an AWS EC2 instance (particularly Linux) to create a rotating HTTP proxy. We’ll achieve this using the AWS Java SDK.

To get started start, install tinyproxy on your EC2 instance. SSH into it, and run the following command:

sudo yum -y install tinyproxy –enablerepo=epel

Then edit /etc/tinyproxy/tinyproxy.conf. Note the port, which should be 8888 by default. Make sure the following options are set:

You may also want to add the same command (without the sudo) to /etc/rc.local so tinyproxy is started whenever the EC2 instance is restarted. There’s a proper way to indicate in Linux what services to start on system startup, but I’m forgetting how, and being too lazy to look it up right now :). Adding this command to /etc/rc.local will certainly do the trick.

Now set your web browser (or at the OS level) to use an HTTP proxy by pointing the settings to the public IP address of the EC2 instance. If you don’t know the IP already, you can get it using the AWS EC2 web console. Or by typing the following command on the EC2 server shell:

wget http://ipinfo.io/ip -qO –

You can now go to Google and type in “What is my IP address”. Google will show you, and you’ll notice that it’s not your real IP, but the public IP of the EC2 instance you’re using as a proxy.

Before we move on, let’s set up some security group settings for the EC2 instance to prevent access. This is necessary so not everyone on the Internet can use your proxy server. The best way to go about this is to use the AWS EC2 web console. Navigate to the security group of the EC2 instance, and note the “Group Name” of the security group (we’ll use that later). Add a custom inbound TCP rule to allow traffic from your IP address to port 8888 (or whatever you configured the proxy to run on).

Next what you need to do is to attach new network interfaces to your EC2 instance (one or multiple). This is so that you can have additional network interfaces that you can map an elastic IP address to, as you don’t want to mess with the main network interface so you can have at least one static IP so you can connect to your EC2 instance for whatever reason. The other network interfaces will rotate their public IPs by attaching and releasing to Elastic IPs (AWS seems to have an endless pool of Elastic IPs, you get a new random one every time you release an Elastic IP and reallocate a new one… this works in our favor so we get new IPs every time).

To attach an Elastic Network Interface to your EC2 instance, check out this documentation: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html. Also note that depending on the type of EC2 instance, you only get to allocate a certain number of network interfaces (for t2.micro, I believe the limit is 1 default and 2 additional (so 3 total)). Lastly, take note of the Elastic Network Interface IDs and their corresponding private IP addresses, once you create them. We’ll use them in our java code.

Now, below is a Java code segment that can be used to assign and rotate Elastic IPs to your EC2 instance, which then become the IPs used as proxy. Note at the top of the code there are a number of configuration parameters (static class level variables) that you’ll need to fill out. And of course you’ll need to have the AWS Java SDK in your classpath.

The method associateAll() will associate the Elastic Network Interfaces provided with new Elastic IPs. And the method releaseAll() will detach the Elastic IPs from the Elastic Network Interfaces and release them to the wild (and thus a subsequent associateAll() will then return new IPs). associateAll() will return an ArrayList of Strings corresponding to the new Elastic IPs attached to the EC2 instance. And these IPs can then be used as the HTTP proxy (tinyproxy will automatically bind itself to the proxy port (8888) on the new public IP addresses, so you can connect to them from your client/browser).

Also note that associateAll() will authorize the public IP of the machine running this code by adding it to the EC2 security group to allow connection to TCP port 8888 (or whatever you configured your HTTP proxy port to be) going into the EC2 instance.

An important note on cost! If you allocate and release Elastic IPs too many times, AWS starts charging you (I think the first couple hundred(?) are free, but after that they start charging and it can add up!). And there is also a cost for leaving an Elastic IP address allocated.

Most of the development/experimentation I was doing with scikit-learn’s machine learning algorithms was on my local development machine. But eventually I needed to do some heavy duty model training / cross validation, which would take weeks on my local machine. So I decided to make use of one of the cheaper compute optimized EC2 instances that AWS offers.

Unfortunately I had some trouble getting scikit-learn to install on a stock Amazon’s EC2 Linux, but I figured it out eventually. I’m sure others will run into this, so I thought I’d write about it.

Note: you can of course get an EC2 community image or an image from the EC2 marketplace that already has Anaconda or scikit-learn and tools installed. This guide is for installing it on a stock Amazon EC2 Linux instance, in case you already have an instance setup you want to use.

In order to get scikit-learn to work, you’ll need to have pandas, scipy and numpy installed too. Fortunately Amazon EC2 Linux comes with python 2.7 already installed, so you don’t need to worry about that.

Start by ssh’ing into your box. Drop into rootshell with the following command (if you’re going to be typing “sudo” before every single command, might as well be root by default anyway, right?)

sudo su

First you need to install some development tools, since you will literally be compiling some libraries in a bit. Run the following commands:

yum groupinstall ‘Development Tools’
yum install python-devel

Next you’ll install the ATLAS and LAPACK libraries, which are needed by numpy and scipy:

yum install atlas-sse3-devel lapack-devel

Now you’re ready to install first all the necessary python libraries and finally scikit-learn:

Well, this applies to any Linux instance that you may be remotely connected to, depending on how sshd is configured on the remote server. And depending on how your localhost (developer machine) ssh config is done. But essentially in some instances the sshd host you’re connecting to times you out pretty quickly, so you have to reconnect often.

This was bothering me for a while. I usually am off and on all day on Linux shell on EC2 instances. And it seemed every time I come back to it, I’d be timed out, causing me to have to reconnect. Not a huge deal, just a nuisance.

To remedy this, without changing the settings on the remote server’s sshd config, you can add the following line to your localhost ssh config. Edit ~/.ssh/config file and add the following line:

ServerAliveInterval 50

And it’s as simple as that! It seems that AWS EC2s are set up to time you out at 60 seconds. So a 50 second keep-alive interval prevents you from getting timed out so aggressively.

This is a short post to explain how to run AWS CLI commands from a crontab.

First you’ll need to install and set up the AWS CLI. More information here: http://docs.aws.amazon.com/cli/

Once you’ve set up AWS CLI, you’ll notice that there is a “.aws” folder created in the HOME folder for the user you’re logged in as. If it’s root, it would be “/root/.aws”.

The problem with running AWS CLI commands from crontab is that crontab sets HOME to “/”, so the “aws” command will not find “~/.aws”.

In order to get around this, you simply need to set HOME=”/root/” (or whatever the HOME is for the user AWS CLI was set up under). This can be done in the shell script that is being called by crontab, or if the aws command is directly in crontab, the crontab command could be something like the following:

HOME=”/root” && aws cli

And that’s it!

Like this:

AWS CLI (command line interface) is very useful when you want to automate certain tasks. This post is about dumping a whole S3 bucket from the command line. This could be for any purpose, such as creating a backup.

First of all, if you don’t already have it installed, you’ll need to download and install the AWS CLI. More information here: http://docs.aws.amazon.com/cli/latest/userguide/installing.html

To configure AWS CLI, type the command:

aws configure

It will ask for credentials: the Access Key ID, and the Access Secret Key. More information on how to set up a key is here: http://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html

And that’s it! You now have the power of manipulating your AWS environment from your command line.

In order to dump a bucket, you’ll need to first make sure that the account belonging to the AWS Key you generated has read access to the bucket. More on setting up permissions in S3 here: http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html

To dump the whole contents of an S3 bucket, you can use the following command:

aws s3 cp –quiet –recursive s3:///

This will copy the entire contents of the bucket to your local directory. As easy as that!