The paradigm of big data software development has cemented itself as a firm niche in recent years. The Hadoop ecosystem has been a strong underpinning of big data processing because it has so many robust tools and out of the box services that address the most

The paradigm of big data software development has cemented itself as a firm niche in recent years. The Hadoop ecosystem has been a strong underpinning of big data processing because it has so many robust tools and out of the box services that address the most fundamental problems of big data data processing. AWS has a dedicated service for running MapReduce jobs called Elastic Map Reduce and in this post I'll show you how to make a basic cluster of Ec2 instances and submit a MapReduce job to it. This job will perform a word count the words in a series of job requirements.

Getting Started with EMR

To get started with EMR you might consider making an IAM group to manage big data type permissions. In any case you'll need the AmazonElasticMapReduceFullAccess an AmazonElasticMapReduceforEC2Role polices to gain access to the EMR console. We'll also use the MapR Distribution for Hadoop. The example given here is more focused on setting up a cluster and running a demo job than the mechanics for writing custom MapReduce jobs. A MapReduce cluster is essentially a fleet of Ec2 instances and so it's important that you have an Ec2 Key Pair ready to use for your cluster. Your Ec2 Key Pairs are displayable for the Ec2 dashboard.

Starting the Cluster

While you certainly can use the AWS console to start the cluster I'm going to start mine form a local terminal. It's important that if you choose to do this that you local terminal is configured with AWS keys to allow make suitable api class. I'm using the command line here for simplicity because it shows all the options I'm using to start the cluster. Using the command

will magically start your cluster of 3 instances and will return the clusters id on the command line. If you return to the EMR console you'll see this id as the cluster starts.

There are a few important details to notice with the command that we used. First notice that the KeyName is the name of one of your Ec2 key pairs. It's important to not have ".pem" on the end of the key or the cluster will fail to start. Also of note is that we are using MapR to make Hadoop easier to deal with and ultimately more dependable because it uses a no-NameNode architecture.

Running a Job

Once your cluster is up and running the master node will have a security group allowing ssh access. You can find a connection string for that node in the summary of your cluster.

Once you are on the master node we can run a MapReduce job on the cluster.

Processing Job Ads with MapReduce

To give an example of running a MapReduce job on our cluster we are going to create a text file with several job descriptions. We will then run a word count map reduce job to find the most relevant skills for all of the proposed jobs. To that end I created a text file in /mapr/MapR_EMR.amazonaws.com/in where I have added the following job descriptions

Word Count

If you are unfamiliar with "word count" it's the "hello world" of MapReduce. However, in this case it will find us the most commonly used words in job requirements and hence tell us the most important skills to look for in prospective resumes. Running the command hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar wordcount /mapr/MapR_EMR.amazonaws.com/in/ /mapr/MapR_EMR.amazonaws.com/out/

will run a "word count" MapReduce job against our job descriptions. Here is an example of successful output

What Happened?

If we look at the directory in/mapr/MapR_EMR.amazonaws.com/out we will see the output of the reducers. We can see the results for each node in the cluster. Here is a sample of the output from one node

From this we can get a broad sense of what experience is most desired in the current job market. For example, we see here that "enterprise" experience is highly desired.

Amazing!

This has been super cool, but it only scratches the surface of what is possible with Big Data processing in AWS. You can make your own custom MapReduce jobs, package them as *.jar files and submit them to the cluster. The AWS infastructure is powerful and robust. Let's see what you can do with it.

]]>Backing Up Your Mobile Data

It's been about 15 years since the first camera phones came out, but now everyone has one and they take lots of pictures. Do you have a back up strategy for all of those pictures and videos? In this post I'll show you how to

It's been about 15 years since the first camera phones came out, but now everyone has one and they take lots of pictures. Do you have a back up strategy for all of those pictures and videos? In this post I'll show you how to use Aws together with Dropbox to automatically backup all of your mobile data with very little effort and money.

The Plan ...

Before getting started here you'll need an Aws account and Dropbox account. I'm going to show you how to install Dropbox on a Linux ec2 instance and activate the Dropbox Deaemon. If you then put the Dropbox app on your phone the ec2 instance will have access to all your photos. That much alone already accomplishes automated, encrypted backup. The only issues is you'll end up having to pay for Dropbox after you dump enough files in there. Dropbox generously has a free 2 gig account so you can run a cron job to move your files from Dropbox to S3. It's true you'll have to pay for S3 usage, but it's cheap. You can make it cheaper by using life cycle polices on your bucket and promoting your media to Amazon Glacier. This strategy gives you cheap persistent storage and frees you from ever worrying about losing your phone or every hooking it up to a computer ever again.

Dropbox on Linux

Dropbox is a very stable product and installing it on Linux is pretty simple. At this point we're assuming you have a Linux ec2 instance up and running and have made a dedicated Dropbox bucket to store your phones media in. To follow this guide be sure to give you instance the S3 full access role. For this experiment I used an Ubuntu instance. To install Dropbox all you need to do is run the command. cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf -

Now you can use the command ~/.dropbox-dist/dropboxd to start the Dropbox daemon. This will return an url for you to follow in your browser. Once you hit the endpoint and login to your Dropbox account you will have associated the ec2 instance to your account. When done correctly you'll see

Now you can use the Dropbox provided Python script to start the Dropbox Client. You can download it here. Once you have copied over the Python script you can use the command python dropbox.py start to start synchronization. Once Dropbox is fully activated and ready to sync your files it will look like this

It's worth noting at this point that you can check your Dropbox run status with the aforementioned Python Script. For example python dropbox.py status will tell if you the daemon is running or has stopped. The command python dropbox.py help will give even more options. If you ever snapshot the EBS volume for this instance it's important that you'll need to restart the daemon manually. Once Dropbox is successfully installed it will add the directory structure Dropbox/Camera\ Uploads/. This is the directory where Dropbox will sync your mobile files. I want to move all the data to a bucket for easy access and storage. To accomplish that I wrote a small script using the Aws command line interface to move the files over to S3. I'm calling my script s3sync and it looks like this.

Be sure to change the permissions of the script to make it executable by using something of the form chmod 755 s3sync Now it's easy to call that script daily from the crontab. To edit your cron jobs use the command crontab -e and add the line @hourly cd /home/ubuntu/ && ./s3sync. Now when you take a picture on your phone it goes into Dropbox via the app and your ec2 instance will fire your S3 migration script hourly to keep you within the free limits of Dropbox.

Do Your Really Need S3 ?

The short answer to that question is no. However, S3 does give you the most flexibility and transparency as to where your files actually go. In reality, you could just move the Dropbox files to a directory on the local instance and then snapshot the root EBS volume. You can even automate that snapshot via CloudWatch. The snapshot is stored in the same storage class as S3 so you get the same availability and durability as S3.

Time to go PhotoCrazy

There are other backup strategies that differ from what I have given here, but there is tremendous peace of mind having a strategy in place. I know that people will say they use Amazon Prime to store their photos or icloud does it automatically. I only see this and them holding your data hostage for further payment.

]]>It's All Fun and Games Until the Bill Comes

Cloud computing is arguably the wave of the future. Major companies like Netflix, Kellog's and even McDonalds are now using AWS to fuel their internet business objectives. This means that the average developer or system administrator is going to need to

]]>https://blogs.sequoiainc.com/track-your-aws-bill-and-learn-how-to-optimize-for-cost/c015524b-4994-48d3-9351-1d002053f731Tue, 26 Dec 2017 12:23:51 GMTIt's All Fun and Games Until the Bill Comes

Cloud computing is arguably the wave of the future. Major companies like Netflix, Kellog's and even McDonalds are now using AWS to fuel their internet business objectives. This means that the average developer or system administrator is going to need to be ahead of the curve on these technologies to stay relevant in their current job or to have a chance in the job market. However, for many people they are afraid to get their feet wet with AWS because of the inevitable bill that shows up. It's well known that AWS will give you one free year to play around but that is hardly enough time to explore deep security concerns, private networking and and general best practices in the cloud. Typically after the free year most people walk away from AWS because they are just plain scared of the bill. Understanding billing is an important part of the use of cloud architecture. That's why in this post I'll show you how you keep a careful eye on your AWS bill with notifications and even how to use different instance types for massive savings over time.

Failure to Plan is Planning to Fail

There are a few ways to stay on top of your AWS bill but one of the most obvious ways is through billing notifications. I like to get billing notifications through text message because they are harder to ignore than email. The two services you need to accomplish this are CloudWatch and SNS. You can set an alarm in CloudWatch based on your bill and use it to trigger a notifications that are in turn sent to an SNS subscriber. I currently have my alarms set for $3,$10 ,$20 and $100. Here is what it looks like

From this screen shot you can see that my bill is over $3 but under $10. I can easily make another alarm for $250 by clicking create an alarm. They tricky detail here is that it makes you pick a metric before you can create the alarm. Set your metric as EstimatedCharges and be sure to select your denomination. You should be at this point

In the next step you will define the actual alarm and configure your desired action. I'm going to send myself a text message via SNS. You can take it a step further and configure autoscaling to terminate instances if you want to. My $250 alarm is now set

Saving Money on Instance Type

A central part of the AWS ecosystem are the Ec2 instances. If you are looking to save money on your bill it's worth your while to understand the configurations available with Ec2. The basic facet to understand here are the instances types. People tend to overlook these because they took some training that was all focused on getting an instance up and running and maybe serving some general purpose but they never really thought about the bill. Ec2 defaults to launching instances as on demand. This is the most flexible but it's also the most expensive. To save money on your bill you shouldn't neglect reserved and spot instances. As of October 2017 AWS has gone to billing in 1 second increments for Ec2,EBS and EMR. The key point here is the on demand rate is always higher than the reserved rate depending how you structure the term commitment. The term commitment is essentially the down payment.

Massive Savings on a Reserved Instance (An Example)

I'm currently running t2.micro instance as a reserved instance. Check it out

You can see here I paid $115 upfront for a three year term on a t2.micro instance but my but my usage and recurring charges are 0. Meanwhile if we check official on demand rates here we'll see that same t2.micro instance cost $.0116 per hour. If we apply that rate over the three year duration we quickly see that the on demand instance will cost 0.0116 * 24 * 365 * 3=$304.85. This translates to a savings of about 63%.

Saving More with Spot Instances

In our last price comparison we saw a major discount using a reserved instance. It's often said that all magic has it's price and the caveat to this magical discount was a full payment up front. The rate for a spot for instance will be even cheaper. In fact at the time of writing it is $0.0037 per hour. What is the price for this magic? The answer is instability. Spot instances allow you to bid on a rate. When an instance satisfying that rate becomes available you get the instance for that rate you bid. This can give you up to a 90% savings. However, if the rate fluctuates above your bid your instance will be terminated after a two minute notice. Most people won't tolerate instability but the typical use case of spot instances is big data analytics. This allows you to run all you analytics at the cheapest possible price.

Money Doesn't Buy Happiness ... but I had to Find Out For Myself

AWS billing is a complicated subject but what we have seen here is that it is easy to track and that a little of strategy goes a long way to cut the bill down. To protect yourself from a huge unexpected AWS bill you should consider your needs and priorities and then optimize accordingly. It takes some effort, but it's for your financial benefit.

]]>Combine

This past October, the Sequoia Team released Combine 3.5 and debuted Harvest which improved our C2S emulation capabilities and established a platform for low-to-high software development. Since then we have been working to not only further mature our C2S migration capabilities but also expand support to other operational

This past October, the Sequoia Team released Combine 3.5 and debuted Harvest which improved our C2S emulation capabilities and established a platform for low-to-high software development. Since then we have been working to not only further mature our C2S migration capabilities but also expand support to other operational air-gapped environments.

That work has culminated in Combine Version 4.0 which provides support for the newly released SC2S region! Combine Version 4.0 will provide SC2S simulation for workload migration in parallel to the existing C2S simulation capabilities.

Combine Version 4.0 will be available starting in January 1st, 2018!

Roadmap

As we open this new year Sequoia is excited about the opportunities for Combine, and helping our customers on their C2S, and now SC2S, migration journeys!

Our development roadmap for Combine remains focused on improved Migration Dashboard (sorting, filtering, and smart analytics that offer solutions for common problems), self service options for TAP role and TAP user creation, and other great improvements!

For other consulting needs or to enlist Sequoia as a partner in migrating your software to C2S or SC2S please contact us www.sequoiainc.com.

]]>Introduction

As a quick glance through past posts will demonstrate, I love the Raspberry Pi platform! I have been using the Pi since it was released, and have owned at least a dozen (dozens maybe?) of them over the years. I was therefore delighted to see that Amazon added Rasbian

As a quick glance through past posts will demonstrate, I love the Raspberry Pi platform! I have been using the Pi since it was released, and have owned at least a dozen (dozens maybe?) of them over the years. I was therefore delighted to see that Amazon added Rasbian support to their new AWS EC2 Systems Manager platform!

AWS EC2 Systems Manager (Systems Manager for short) is an agent based platform for configuring, controlling, and governing on premise servers from within the EC2 console. By installing a Systems Manager agent on your server, you can execute commands remotely, ensure servers remain in specific state, and enforce configuration management requirements.

The agent connects to Systems Manager through an "activation" which is represents a collection of servers that are managed as a single unit.

NOTE: An interesting advantage of Systems Manager is that it is a "pull" not a "push" model. The installed agent polls Systems Manager for commands, and therefore servers can be safely managed from behind a firewall or from within a private subnet since. Systems Manager never directly connects to the managed servers.

Connecting your Raspberry Pi to Systems Manager requires a few steps:

Establish the necessary IAM permissions

Create an activation

Install agent and register with activation

Once the agent is up and running you can then perform Systems Manager operations such as:

Create a command

Run a command

This is just scratching the surface of Systems Manager, but will give you a good foundation from which to learn more!

Establish IAM Permissions

In order to give each "activation" the necessary access to AWS resources (for logging, interacting with systems manager and ec2, reading data from s3, and so forth) you need to create an IAM Role.

NOTE: When you create an activation from the AWS Management Console, there is an option to create the role automatically. I find however, that you learn a more if you create IAM resources manually. If you prefer to let AWS do this behind the scenes, skip tot the next section.

You will need to first create an IAM Policy to attach to your role. Below is the default policy that covers all the core permissions required by Systems Manager:

You now have the IAM permissions necessary for your activation to execute successfully!

Create Activation

You are now ready to create an activation! Once you have an activation you will be able to register servers to it. Those servers will operate with the permissions and properties of the activation.

To create an activation you access the EC2 section of the AWS Management Console. On the left hand navigation bar there is a section for Systems Manager Services and Systems Manager Shared Resources.

Under Systems Manager Shared Resources you will see a link for Activations.

Click on Create Activation.

You will see a simple form. The only required fields are the Activation Description and the IAM Role Name.

If you elected to create your IAM permissions above, then select Select an existing custom... and then select your role name from the dropdown that appears. Otherwise leave the default value selected and AWS will create the IAM Role for you behind the scenes.

NOTE: If you do not have the Trust Policy set to include ssm.amazonaws.com you can still select the policy but it will fail internally.

When you create your activation you will be given an activation code and activation id. These act like a secret key and access key for your servers to connect to Systems Manager and assume the IAM Role associated with their activation.

Save and backup these values!

Excellent! Now that your activation is created it is ready to have servers registered to it!

Install Agent

There are Systems Manager agent builds for a wide variety of platforms. For the various platform's installation instructions consult the Amazon documentation.

For reference, I have curated the install instructions for Raspberry Pi here:

NOTE: Replace the <CODE>, and <ID> place holders with your activation's code and id, and replace the <REGION> place holder with the region in which your activation resides (for example, us-east-1).

If that is successful, you should see your Raspberry Pi appear as an instance on the Managed Instances tab under Systems Manager Shared Resources!

Now that your instance is being managed by Systems Manager you have access to a whole world of configuration management and enforcement.

We will conclude this blog with a simple example to demonstrate (and validate) two way communication with the Raspberry Pi from within Systems Manager.

Create Command

The simplest thing you want to do on a remote server is execute a command. Systems Manager has a mechanism for creating and managing commands, and executing them remotely on managed instances.

Before you can run a command, you need to create a definition that defines it and provides context for how / where it should execute. Systems Manager manages these definitions of operations as Documents.

NOTE: There are several types of documents for different applications.

Once your command is created, you are ready to execute it on your Raspberry Pi!

Run Command

To run the command select the Run Command tab under the Systems Manager Services section. Click the Run a Command button.

You are presented with a form that lets you select a command, select which instances to execute it on, and specify some parameters relevant if you are executing on many servers simultaneously.

Select the command you created above, and your Raspberry Pi instance id, then click the Run button.

You will now see an entry on the Run Command dashboard representing your running command.

You can view the command result by clicking on the command entry, clicking the Output tab on the detail view, and clicking the View Output link. This will show you the command line output from running the command!

If all goes well your command will successfully execute!

Conclusion

Systems Manager is a powerful platform for configuration management and remote access to servers. I hope this gentle introduction to it was useful and provides a good foundation for you to use Systems Manager further!

Feel free to email me at smouring@sequoiainc.com with any questions, comments, or feedback!

]]>Overview

I would be willing to bet even the most mundane computer users has a graveyard of old computers laying around. There is a variety of reasons this can happen from the hinges busted apart on your laptop to you just plain filled your computer up with photographs of your

I would be willing to bet even the most mundane computer users has a graveyard of old computers laying around. There is a variety of reasons this can happen from the hinges busted apart on your laptop to you just plain filled your computer up with photographs of your first born. For this post I grabbed a fifteen year old paperweight laptop from my basement and replaced the keyboard and freshly installed Ubuntu 17.10 to make a full working and usable machine. Essentially I took this thing

and turned it into this

It's All About the Pentiums!

This was really just a fun experiment because I had never really opened up a laptop and messed with it internally until writing this blog post. The laptop needed a new keyboard and it was all too easy to order the correct one from Amazon for less that $10. Incidentally, you can buy individual laptop keys. I've used this site in the past. The keys are made up of the key, the hinge and the cup. I also wanted to get a nice set of screw drivers for computers and other house hold electronics. I ended up buying this set

from Amazon and it more than suited my purposes. With the new keyboard and fancy screw driver I was ready to breathe new life into my laptop.

Replacing the Keyboard

Replacing the keyboard turned out to be the easy part. Getting access to the screws that hold the keyboard down was a different story. All brands of laptops are different. The answer for me was getting the screws off the hinges and removing the plate above the keyboard that covers the screws that hold down the keyboard. Here is how it looked once I go that piece off.

In this picture you can see the keyboard is flipped forward and held in by one ribbon cable. Also notice that upper plate that has the power button is held onto the motherboard by a tiny ribbon cable. It's important at this point to not drop screws or ribbon cable fasteners into the bowls of the laptop. If this happens you'll have to take many more screws out and search for the lost city of Atlantis and the tiny pieces you dropped. Once you gently remove the ribbon cable holding down the keyboard you are ready to fasten on the new keyboard. Here is how it looks once you are ready to lay down the new keyboard

Enough Hardware, Let's Talk Software

Once the new keyboard was in it was time to format and install the operating system. This machine previously had Windows XP which was out of support so I used Ubuntu Linux because it's drop dead simple to install and seeing as I was already $30 into this I was running out of patience. I found a tool called Rufus to make a bootable Usb drive and install Ubuntu. Rufus was super easy to use and here is a screen shot so you can get a feel for what it's like

After making my bootable Usb drive and changing the boot order on my laptop it was a breeze to install Ubuntu and get up and running.

Wireless Networking

The internal native wifi adapter and Ubuntu were not getting along so I found an easy to use Usb wifi adapter that that worked via plug-n-play on Ubuntu. I ordered it from Amazon here. Here is how it ended up looking

It's About the Journey not the Destination

You might be asking "what you are you going to do with your new laptop ?". Naturally, I wrote this blog post on it, but I didn't do all of this because I desperately needed a laptop. I did it for the experience and because my Netflix queue was looking lean. However, if you have old computer hardware laying around you should consider saving any relevant data from it and either updating it, or disposing of it. If you are feeling ambitious you can strip your old machines for parts and sell them on Ebay. Everyone should contemplate this situation because all computer consumers will eventually have to make these decisions.

]]>Introduction

Bitcoin, Ethereum, Monero... oh my! Cryptocurrencies are much in the news recently and have seemed to catch the public imagination! Advocates praise their advantages and potential to disrupt, while detractors are cautioning against their peril and warning of a financial bubble... which is not surprising since cryptocurrencies combine an

Bitcoin, Ethereum, Monero... oh my! Cryptocurrencies are much in the news recently and have seemed to catch the public imagination! Advocates praise their advantages and potential to disrupt, while detractors are cautioning against their peril and warning of a financial bubble... which is not surprising since cryptocurrencies combine an intriguing technical foundation with the profit seeking thrill that is innate to human nature.

In the past I have made several forays into CryptoCurrency mining. More recently I have built multiple Ethereum mining rigs (another blog post for another time) and been exposed first hand to the trade off between up front investment and long term return.

One aspect of cryptocurrency mining that really caught my attention was browser mining... Although I was originally exposed to it on less thanreputableterms, I see browser mining as a novel alternative mechanism for monetizing your website.

I hate ads. I know they power the internet. I know they enable websites to continue providing quality content. I know they are Google's primary source of revenue. Still. I hate being sold to. I hate the intrusion.

Browser mining offers an intriguing alternative... Instead of asking for your time and attention, browser mining borrows some CPU cycles and runs up a couple extra pennies on your electricity bill.

By allowing your browser to mine cryptocurrency you are paying for the content you are consuming, just in a much less invasion manner than having to be subjected to an ad.

Whether or not browser mining is ultimately a replacement for advertising is unlikely, but it is an intriguing alternative and something I would love to see as a voluntary alternative on websites.

As soon as I learned about the concept, I was eager to try it out. I found it to be surprisingly simple!

CoinHive

Although featured in the nefarious hacks referenced above, CoinHive is a legitimate purveyor of browser mining middle man services.

CoinHive is essentially a mining pool... It divides up mining work into small parts, and sends them to its agents, who complete the work and send back their results. Each agent gets a share of the overall profit from mining. Once enough shares has been accumulated, CoinHive distributes the Agent's profit to a configurable wallet address.

NOTE: CoinHive mines Monero. The reason for this is that unlike other major cryptocurrencies, Monero is both GPU and ASIC resistant. This means that mining with a GPU is not overwhelmingly better than mining with a CPU, and mining cannot be performed with custom hardware (ASICs). This levels the playing field and allows folks leveraging just CPUs to mine competitively.

At the time of this writing, Monero is worth about $100 per unit. A single CPU mining 24/7 is capable of mining about 1 or 2 Monero per year.

Most visitors to a website are not going to be mining Monero 24/7 however, but this is the beauty of the model. There is no difference between 1 CPU mining for 10,000 seconds, or 1,000 CPUs mining for 10 seconds. The more traffic you drive to your site the more your mining profits will increase!

To get started, you need to create a CoinHive account. Follow the instructions on the site.

Monero Wallet

CoinHive is a middle man. It requires a wallet address to send your Agent's profits to. A wallet is just a public/private key pair used to receive and send Monero. You can create a Monero Wallet offline (or online) using a desktop client, or online using one of the Monero wallet websites.

I recommend using the Monero desktop client. It gives you full control of you wallet files, the ability to create backups, and puts the responsibility of security in your hands.

On the other hand, the Monero desktop client requires that you download your own entire copy of the Monero blockchain, and is only accessible from a single computer. If you would rather have more convenience, consider a cloud wallet such as MyMonero.

Once you have created a wallet, enter the public address into your account on CoinHive.

You are now ready to start mining!

Deployment

To test this browser miner, I added it to my personal website / scratch pad: stephenmouring.name.

NOTE: CoinHive allows you to create multiple "sites" with are unique key pairs that identify your mining agent. I created a single site for my purposes, but if you had a large deployment over multiple websites using multiple key pairs would allow you to track revenue from each individual website.

Very simple. The first <script> block remotely loads CoinHive's mining library. This is where all the magic and complexity of the mining happens, but they have created a simple API to wrap that and a simple interface to integrate with.

var miner = new CoinHive.Anonymous('<YOUR SITE KEY>');
miner.start();

This is really all there is to it! As soon as this code executes, the miner will start mining.

Every three seconds, I update the div on the page with the current hash rate. This gives me a benchmark for performance and visual indication of activity.

Conclusion

That is all there is! Remarkable simple. An intriguing use of technology. An interesting alternative to advertising? Tell me what you think!

Questions? Comments? Email me at smouring@sequoiainc.com!

]]>Combine

This past June, Sequoia was pleased to announce the release of Combine 3.0. Since that release, we have helped many more customers in their transition to C2S. This engagement has helped us refine our migration experience and made us aware of new customer needs.

This past June, Sequoia was pleased to announce the release of Combine 3.0. Since that release, we have helped many more customers in their transition to C2S. This engagement has helped us refine our migration experience and made us aware of new customer needs.

We have leveraged that experience into new features and improvements for Combine 3.5 and also the creation of a new product Sequoia Harvest!

We have the pleasure of announcing Combine 3.5 and Harvest 1.0 for immediate release, and would like to cover the new features and many improvements we have made!

Combine 3.5 Features

We have added several new features and improved several existing features in Combine 3.5.

New Interface

We worked with a front end engineering team to rewrite the Combine interface. This gave us an opportunity to build a cleaner, more extensible UI using modern web standards.

The above image is the updated landing page for account/role selection.

The above image is the updated Migration Dashboard.

This new interface provides a more clear user experience and paves the way for future functionality driven by customer feedback!

C2S Service / Service Features

In Combine 3.5 we have significantly improved the fidelity of our C2S Service and Service Features simulation. This includes further adjustments to EC2 to lock down EC2 service features not available in C2S, as well as new restrictions for ELB, RDS, CloudWatch, CloudTrail, Config, and Kinesis.

The Enterprise role for the AWS Config service was ported to Combine from C2S enabling Combine customers to test AWS Config workloads.

Updates to Enterprise roles were made including removing some restrictions for Marketplace, Trusted Advisor, KMS, EMR, Kinesis, and Redshift.

These changes to our service and service feature modeling improve the fidelity of the Combine simulation and increase the speed at which migration issues can be detected, while reducing the number of false positives.

Internal Improvements

Several internal changes where made to PKI infrastructure to better model the PKI infrastructure used by the customer agencies. This should simplify testing particularly for CAP integration scenarios.

Several internal changes were also made to server infrastructure to increase the stability of the TAP server. This will also prevent private IP address reassignments which invalidate PKI certificate aliases.

Introducing Harvest

Last but not least, Sequoia is pleased to announce the release of Sequoia Harvest! Built on the same C2S emulating technology as Combine, Harvest is tailored for use in custom software development and DevOps workloads.

Harvest is proud to offer pre-installed software from our partner network (which includes Telos, NVidia, and evident.io) in the same C2S simulating environment that powers Combine.

This allows our customers to do custom low-to-high software development against a C2S simulation with the added benefits of consolidated billing for partner software.

Harvest is the solution for unclassified, low-to-high development workloads.

Roadmap

Moving into next year we are excited about the growth opportunities for Combine and Harvest, and helping our customers on their C2S migration journey!

Our next major goal is building a similarly high quality simulation environment for the upcoming SC2S region to serve customers who want to migrate to that space.

Our development roadmap for Combine and Harvest includes an improved Migration Dashboard (sorting, filtering, and smart analytics that offer solutions for common problems), self service options for TAP role and TAP user creation, and other great improvements!

For other consulting needs or to enlist Sequoia as a partner in migrating your software to C2S please contact us www.sequoiainc.com.

]]>Keeping Up with the Times

Some tools are just fundamental to the development of any type of software such as version control and IDE's. In this post I'll give you an overview of the AWS tools that can be used for development. I'll show you how to use AWS CodeCommit

]]>https://blogs.sequoiainc.com/aws-as-a-software-development-platform/5b0bdf36-ffea-4abe-8362-3cac32aec9a7Mon, 23 Oct 2017 12:19:56 GMTKeeping Up with the Times

Some tools are just fundamental to the development of any type of software such as version control and IDE's. In this post I'll give you an overview of the AWS tools that can be used for development. I'll show you how to use AWS CodeCommit for private git repos and even a full cycle of local commit to open internet deployment with a single click. Interested ? Check it out.

Github ... ain't nobody got time for that

Most developers are accustomed to Github as a remote repository for their Git repos. Aws has their own private and encrypted upstream repository called CodeCommit. You can push and pull via ssh just like Github but CodeCommit gives you access to Aws services as well as controlling who can access your code.

Git Access with CodeCommit

The setup for CodeCommit starts with IAM but is very similar to uploading ssh keys into Github. Let's start by making a new IAM user and assigning some permissions to push and pull code. First we create a new user

Now set his permissions.

Be sure that you take note of the new users access key and secret key so that you will be able to access Aws services programatically from the api. From the security credentials tab for the user you will be able to upload an ssh public key. While you are here be sure to generate and download https git credentials for CodeCommit. We'll use these later for git commits from Eclipse Oxygen. Here is what the whole thing looks like

Once you have uploaded your ssh key an ssh key id will be generated by Amazon. Now click on show ssh key and copy this to the keyboard. In the same directory where you generated your ssh key that you uploaded, create a file called aws-key and paste in what you have what you copied. You'll also need to create a config file and populate it with

If you have done it correctly the repo will clone just like it would from Github.

CodeStar and Eclipse

Many developers like to use Git from the command line. You can certainly do that with the setup I have just given you. However, your code will likely be written in an IDE and if you want to control Git from that IDE you can have that freedom also. Now that we have established local git connectivity let's make a full blown deployed app. Head over to CodeStar in the console and let's use an Aws template for a Java webapp deployed on ElasticBeanStalk. .

Aws will now generate code and deploy via the ElasticBeanStalk service. If you head over to ElasticBeanStalk you'll see AWS has deployed your application to a url. Here is the default index.html.

IDE Integration with CodeCommit and ElasticBeanStalk

At this point Aws has auto generated application code that we can checkout via CodeCommit in Eclipse. For this tutorial I downloaded the latest version of Eclipse called Oxygen. From the Help menu in Eclipse you can access the Marketplace and download the Aws Toolkit. This gives smooth CodeStar Integration.

. When this completes you can import a CodeStar Project from the Aws menu bar here

Make a commit and Push to AWS

I'm just going to change the text on the default index.html to demonstrate a push and build in ElasticBeanStalk. The Aws Toolkit will setup the git integration for you, so if you select your project in the project explorer shift+ctrl+# will pull up your commit menu where you can commit and push straight to Aws.

After your commit ElasticBeanStalk will automatically detect it. The code will be built and deployed to the application server. We can see that our commit worked successfully below

Wait There's More!

There is plenty more that you can do with the infrastructure that we have laid out here. The CodeStar Service even facilitates Jira integration and makes it unbelievably simple to forklift your whole development team into the Aws environment. There are even more services like CodeBuild for building and testing. Also CodeDeploy is a free service for continuous integration. CodeDeploy even has autoscaling integration that gives a self healing quality and and high availability. A big question is where does this leave traditional tools like Jenkins and Github? Only time will tell, but it's hard to argue with this level of sophistication and integration.

]]>AWS is well known for touting their tennants of highly available, self healing and fault tolerance. In this post we'll look at using resources that justify those claims and show some of the magic that you can realize using the AWS architecture. We will be creating several Ec2 instances as]]>https://blogs.sequoiainc.com/highly-available-self-healing-and-fault-tolerant-applications-on-aws/6448b796-12f6-4e60-b579-f2a747f6aea0Sun, 24 Sep 2017 15:40:38 GMTAWS is well known for touting their tennants of highly available, self healing and fault tolerance. In this post we'll look at using resources that justify those claims and show some of the magic that you can realize using the AWS architecture. We will be creating several Ec2 instances as part of a launch configuration and use an elastic load balancer to manage the traffic to them. We'll then use an autoscaling group and see that when killing instances new ones will take their place. When an instance is terminated the load balancer will stop sending traffic to that instance while the autoscaling group fire a new instance to replace the terminated one.

Making a Load Balancer

It's important to realize that load balancers and autoscaling groups are all part of the Ec2 service. This means that the Ec2 menu will be the starting point for all services that we are going to leverage in this tutorial. In the lower left side bar of the Ec2 menu click on load balancer and create a new load balancer. The load balancer creation is pretty straight forward but be sure to pick a VPC and you'll need to add subnets. Yours should look something like this.

You'll need to configure the health check. I've set my ping protocol to TCP for this example. You likely don't need to change anything else here unless you are doing something exotic.

The Launch Configuration

The next step is to have a launch configuration. This is very similar to launching a single instance from the Ec2 menu but here you will provision several instances to put in your load balancer. Be sure in the configuration details tab to choose "Assign a public IP to every instance". Here is what it looks like:

We'll also configure some user data too so we can clearly see the load balancer hitting an individual instance. Use the script below to install an Apache Webserver and serve a basic web page.

The AutoScaling Group

You can use the autoscaling creating wizard here. The key point here is that we want to set metrics that will cause AWS to fire new instances. This is the justification that AWS is self healing and fault tolerant.

Cattle not Pets

If one instance had a problem, the health check would see this, terminate that instance, and replace it with a new healthy instance. This is the self healing process. Often the idiom of "cattle not pets" is used to describe this overall process. Cattle are meant for killing and consumption whereas pets are something that are helped and cared for. It's a much lower level of effort to automatically kill the server and start another one than it is to diagnose and restart.

Setting the Autoscale alarms

Pick the vpc and subnets and you are now ready to add scaling alarms.

Beware of the max size.

Create an alarm to add an instance.

Don't forget to take action once the your alarm is set

Go check your Ec2 dashboard and watch your instances launch.

After your instances have time to register with the load balancer you can follow the DNS address of the load balancer found here.

When you append "/test.html" to the above uri you'll see the web page we created in the user data on one of the instances registered with the load balancer. We can see it working below.

After terminating an instance you'll see the autoscaling magic as a new instance is kicked off and put in the load balancer. The load balancer keeps a record of this activity and you can see below where I killed an instance and auto scaling started another one to replace it.

Where Does it Leave Us?

The key point here is that we have extended the simple use of an Ec2 instance by using the load balancer. The load balancer is arguably the AWS resource that gives you high availability. In this example we have used it to host a web server and serve a simple web page. Self healing was a concept demonstrated by the combination of launch configuration and autoscaling. The launch configuration allowed us to provision instances to put in the load balancer while auto scaling allowed us to set alarms and take specific action to increase or decrease the number of instances based on specific metrics. The ability of the load balancer to have health checks and stop sending traffic to unhealthy instances is what makes AWS fault tolerant.

Take it Further

The extensibility of what we have discussed here is nearly endless in the context of AWS cloud computing. It would be easy to take this further and use Route 53 to purchase a domain name and create a record set pointing to the elastic load balancer. Then the user would see your application or website and under the hood you would have the security and high availability of AWS. I hope you can take some of these concepts and architect your own solutions on AWS.

]]>Going Deeper

In this blog post we'll go beyond standard security and create our own VPC with a private subnet. We'll then create a bastion host in a public subnet that we'll leverage to instantiate a NAT gateway to enable our instance in a private subnet to receive updates from

In this blog post we'll go beyond standard security and create our own VPC with a private subnet. We'll then create a bastion host in a public subnet that we'll leverage to instantiate a NAT gateway to enable our instance in a private subnet to receive updates from the internet even though it has no route to an internet gateway.

Layered Security

There are many layers of security available in AWS. Most people are content with the security group acting as a firewall at the instance level. This is probably mostly because the Ec2 creation wizard prompts them to select a security group or create a new one. This alone, can act a very effective firewall. Restricting ssh traffic to originate from your own external IP address is a very effective filter. However, AWS allows you to firewall an entire subnet and create your own virtual network where your security paranoia will be given free reign to run amok.

AWS Security Infrastructure

Upon deeper inspection of the AWS infrasturue you'll probably figure out that the difference between a public and a private subnet is the route to the internet gateway. If we create our own VPC and attach a internet gateway to it then any subnet that has a route table associated with it and a route to the internet gateway becomes a public subnet. You can get a deeper level of security if you launch your instances into a private subnet and then only shell into them from a public instance that acts as the bastion host. Let's walk through it so we can understand the risk/reward ratio at each step of the configuration.

A Custom VPC

Your AWS account comes with a default VPC that will likely suit most of your needs. While I don't recommend that you delete it, you certainly can if you are the adventurous type. You can make a VPC from scratch and I recommend that you don't use the creation wizard. Building it from scratch will force you to encounter each security concern. Once your VPC is created you can also create a new internet gateway and associate it to your new VPC. I'm using 10.0.0.0/24 for the CIDR range of my VPC. This will impact my choice of CIRD blocks for my subnets later.

Creating Subnets in your VPC

I'm making two subnets. One is public with a route to the internet gateway you created above, and one is private. Here is how it looks so you are familiar with the creation process.

For your bastion host create a route table, associate your public subnet and define a route to your internet gateway. Notice the tab for subnet association as well as the the tab for route enumeration.

While you are busy creating route tables make another and associate your private subnet you created above. You won't need any other routes. This will keep the riffraff of the internet out of your business.

Launch your Bastion Host and Private Instance

We are ready to launch the bastion host. Follow the basic creation wizard but make sure you use your new VPC, your public subnet and enable a public IP so you can actually shell into it. You can similarly launch your private instance but make sure you use your private subnet and disable the public IP. It will be created with a private ip that your bastion host will be able to use.

Risk Reward Ratio

We now have the benefit of increased security because our instance is in a private subnet but it can't receive updates because it has no route to the internet to receive them. Let's see how it works. Shell into your bastion host and copy over your cert with the copy command scp -i temp.pem temp.pem ec2-user@54.209.50.20:/home/ec2-user From the bastion host you can now login to your private instance. This is possible because of the default local route that Amazon provides. Once you are in your private instance you'll notice that sudo yum update fails because your private instance has not route to the internet because it is in a private subnet. We can get around this issue by using the NAT Gateway service and hence get strong security with no consequences.

Getting Updates Through a NAT Gateway

Create a NAT gateway from the VPC menu.
Be sure to use the public subnet and you can create a new elastic IP. Here is what it looks like.

Update your private route table to have a route to the NAT like this.

With this new route in place return to your private instance and again run sudo yum update and you'll see it gets the updates successfully. Check it out.

If you have been using AWS for a for a few years you'll notice that the NAT gateway is a new service. You used to have to create a separate instance to act as your NAT.

Wrapping Up

I hope this post has showed you some of the security possibilities that are available in AWS. Play with them yourself and discover the ideal setup for your needs. Security is the biggest argument against cloud computing so you should invest in a deep understanding on this issue.

]]>Creating a Linux Virtual Machine From the Command Line

In this post we'll create an Azure account, install the Azure command line tool and use it to create a Linux Virtual Machine within your newly minted Azure account.

In this post we'll create an Azure account, install the Azure command line tool and use it to create a Linux Virtual Machine within your newly minted Azure account.

Getting setup on Azure

The first order of business is to get get a new Azure account. Head of to azure.microsoft.com to get started.

Before you sit down to do this you'll need a microsoft.com account, a credit card and your cell phone for verification purposes. You'll find this experience relatively painless and in no time you'll be inside the Azure portal. They have have some simple walk-throughs for you once you are inside. To make your reading of this article more entertaining I'll actually spin up an Ec2 instance in AWS, install the Azure Command line Interface on it and then proceed to create a Linux machine from the command line. We'll finish by shelling into Azure from AWS.

Install Azures Command Line Tool

When I did this experiment I ran right off an Ubuntu Server Ec2 instance. However, there is really no necessity in doing this. You can certainly get this up and running straight from your local command line. To get the Azure command line tool installed run the following commands

With the last command being executed the Azure command line tool will be installed and you can verify the version by running azure --version as show below.

Logging into Azure from the Command Line

Microsoft makes this easier than you would'd think. Simply running the command azure login and url with security token will be returned. To successfully authenticate all you need to do is follow the url and enter the security token in the browser. Here is what it looks like.

Following the url in the browser completes the authentication step.

Creating a Linux Box from the Command Line

At this point you are ready to create a virtual instance. This is just a matter of knowing the right commands and giving a correct configuration. Start the configuration by running azure config mode arm. This will allow you to configure Azure resources in the command line context. You also need to create a new resource group. Keep note of what you call your group because you'll need it again when you go to create the actual instance. azure group create blogTest eastus will create the "blogTest" group in the eastus region. At this point you need to run some commands to allocate the foundational resources to create your instance. To this end run azure provider register Microsoft.DataFactory and then azure account list. Here you will see your fee account listed with it's id. To complete this resource allocation run azure account set <SubscriptionId> Where is the given id returned from the previous command. You're now read to start the image creation process with the command azure vm quick-create. You'll then be prompted for several things, but keep in mind your previous group name and the fact that you allocated resources in eastus. You'll be prompted to create user-name and password. Be sure to keep note of them. The only odd ball thing you'll be asked for is the ImageURN. You can create an Ubuntu image with the input canonicalubuntuserver:14.04.2-LTS:standard Since you are working from the command line Microsoft does insist on a minimal password length of 8. Here is a screen capture of this whole process so you can try it for yourself.

This process will take a few minutes but when it completes a summary will be echoed to the terminal. This summary will in particular give the fully qualified domain name of your new instance that will allow you to shell into it. Below is completion the creation process.

ssh and Kick The Tires!

At this point you have made an Ubuntu instance on Azure from the command line and are ready to shell into it. From the summary screen above we have sufficient credentials to login. Indeed, running ssh blogauthor@blogd-eastu-axziavndl262-pip.eastus.cloudapp.azure.com and using the password you chose earlier we will be fully logged into our new instance. Below is a screen capture of the full login process.

Back to the Azure Portal

Return to your actual Azure Portal in your browser and see all the resources you just created from the command line have updated into the portal. Here's how mine looks now.

We didn't need the command line to accomplish this but it let us explore and combine facets of Azure in a non obvious way.

AWS or Azure?

So which cloud platform is better? That is a deep question. Personally, most of my cloud computing experience has been with Amazon but I do have to say Microsoft has made a very smooth and seamless user experience. I think a complete answer to that question would be both platforms have their strengths and weakness but the correct tool for the job depends on the specific task at hand. If you learn one platform well and ignore the alternatives then you might miss the optimization that another platform could give you. When the only tool you have is a hammer it's funny how everything start to look like a nail.

]]>Streaming Twitter Data to Kinesis Firehose
In this post we'll dive into data mining social media with Kinesis Firehose. Specifically, we'll obtain Twitter API keys and write a Python script to find tweets related to congressional activity by filtering with the word "congress".

Making a Twitter Application
The first thing we'll need is a Twitter account. After you login make sure you add your phone number to your account via the settings console here https://twitter.com/settings/. Be sure to use your real phone number because Twitter will text you to verify ownership. You can text back to stop all future text messages. If giving out your number to Twitter makes you nervous I can assure I have gotten no spam since attempting this experiment. The phone number turns out to be a necessary step because you can not register an app to your account without one. Now make your app at https://apps.twitter.com/. Click the Create New App button and fill out the corresponding form. Only the name and description fields are required. Any dummy url can be used. Once this is done you'll have the required API keys to to start sending Twitter data to Kinesis Firehose.

Digging into Kinesis
Within AWS, Kinesis is a broad topic. We are going to specifically use Kinesis Firehose and stream tweets straight into an S3 bucket. Before you can start be sure to give yourself the AmazonKinesisFirehoseFullAccess role. Notice that this is different than the KinesisFullAccess role. Then from the KinesisFirehose console choose to create a new delivery stream and be sure to name it "twitter-stream". This is important if you are going to run the code I supply you here. Be sure to choose to stream your data in S3 rather than Redshift. While there are configurations in here for transforms, logging, and encryption you can leave them all disabled and create a new bucket for your destination. The only optimization I would recommend are the buffer settings for S3. I have mine set like this.

Towards the end of setup process you'll need to choose an IAM role to run the stream. The path of least resistance here is to let Amazon create a new role for you. This will quickly wrap up the Kinesis setup for you. To finish up, head over to S3 and see your new empty Kinesis bucket.

Python Streaming Code
We will be using Python for our application code and you can clone the repository with the command https://github.com/jdav999/twitter_stream.git. You can do this from an Ec2 instance or your local machine. However, you'll need to have pip installed and then pip install tweepy, pip install ConfigParser, and pip install boto3. Next you'll need to add your Twitter API keys to the config file. I've left a skeleton for all the api keys here

Once your are fully configured and authorized you are ready to kick off the stream. However, before you do it's worth browsing the code and noticing how it authenticates and then filters for the phrase "congress". This will give us congressionally relevant tweets. Check it out here

Turning the Hose On
If you have everything set correctly then python twitter_firehose.py will start the stream and you'll see raw tweets fly by like this. More importantly, you'll see your S3 bucket start to populate like this

You can even open one of the objects from the buckets and inspect the raw tweet

I've highlighted some of the tweets actual payload to demonstrate how our code actually filtered for "congress". At this point quite a lot of data science could be thrown at this data and we can hunt for clues of future congressional plans. Relevant data in here includes mentions of congress, the disseminators name, the names of those who retweet congressional related information, and their followers. Incidentally there are some really cool big data tools in AWS to help sift through this kind of data, but that will have to wait for another time.

The last thing that is interesting to look at here is the stream in the Kinesis Console. In particular you can look at the monitoring graphs of your stream and see how it's performing. Here is how mine looks right now

Conclusions
I hope I introduced you to some new technology and showed you how to integrate with the Twitter Platform. Get your hands dirty and have some fun.

]]>Introduction

Since getting an Amazon Echo for father's day, Alexa has established herself in our life with remarkable alacrity.

She has finally enabled my wife to fully enjoy our music collection (since my excessive organization of our data unintentionally made it hard for her to search for what she wants)

Since getting an Amazon Echo for father's day, Alexa has established herself in our life with remarkable alacrity.

She has finally enabled my wife to fully enjoy our music collection (since my excessive organization of our data unintentionally made it hard for her to search for what she wants).

My kids immediately accepted Alexa as a fixture of the household, asking her to tell them jokes, tell them stories, and even if she could "go to the store and buy them a puppy".

I have now arranged for Alexa to control the lighting in the master bedroom. Most nights as we try to put our newborn to sleep there is a regular progression of "Alexa dim the bedroom lights to 50%", "Alexa dim the bedroom lights to 30%", "Alexa dim the bedroom lights to 10%", "Alexa turn off the bedroom lights"...

The next step in my Alexa journey was, naturally, to write my own Alexa Skill. So this weekend, I set aside some time to do that... And I was pleasantly surprised to find that it was both remarkably easy and really fun!

There are essentially two components to an Alexa Skill: the definition and the implementation. To illustrate this, I will walk through one of the skills I wrote this weekend.

I called this skill "Work Days" and it responds to several phrases (such as "how many work days are left in the year?"), by counting the number of work days remaining in the year, and reporting this number back to the user. (An important utility for a developer who is on yearly contracts!)

So let's get started!

Skill Definition

To create an Alexa Skill you need to set up an account on the Amazon Developer site.

NOTE: It is much easier to test your Alexa Skill if you use the account associated with your devices as your developer account. Otherwise you need to share devices between your developer account and your regular account.

In the Amazon Developer site you will create an Alexa Skill entry (which is much like an "app store" listing). In this entry you will provide metadata about your skill, define the methods in your implementation ("intents") and specify what words or phrases should trigger those methods ("utterances").

To get started, log into the Developer site, select the Alexa tab, and click Get Started > under the Alexa Skills Kit icon.

This will then start a wizard which walks you through the process of completing the entry.

The first step of the wizard is the Skill Information form. For a basic skill there are really only two things of interest here, the Name and the Invocation Name.

The Name is what will be displayed in the Alexa Skill store. The Invocation Name is the key word that enables your skill when a user address Alexa.

In our example, "How Many Work Days Left?" is our Name, and the phrase work days is our Invocation Name. Therefore our skill can be accessed by addressing Alexa like this: "Alexa, ask work days ..."

The next step of the wizard is the Interaction Model form. Here is where you define your "intents" (which map to methods in your implementation) and "utterances" (which trigger your intents when spoken to Alexa).

Intents are defined via JSON (see documentation and schema here). Intents are like hooks or triggers that map an action in the skill to a block of implementing code.

I have one primary intent: WorkDaysIntent that will handle the majority of the skill's work. I also implemented the built in AMAZON.HelpIntent to hook into the Alexa's help interface. We will see these intent names again when we provide the implementation for the skill in the next section.

Utterances

Utterances are the bread and butter of the user interaction. For each intent you can provide multiple sentences that are used to trigger the intent.

The utterances defined for our skill are as follows:

WorkDaysIntent say how long does the work year lasts
WorkDaysIntent say how many works days are left
WorkDaysIntent say how many works days are left in the year
WorkDaysIntent say how many works days are left this year
WorkDaysIntent say how many works days are there
WorkDaysIntent say how many works days are there left
WorkDaysIntent say how many works days remain
WorkDaysIntent say how much longer

NOTE: Utterance can include "slots" which are essentially variables that can be included in the utterance to turn it into a template. For example if I wanted an utterance to include a number or a date, I could define a slot and reference it in the utterance. In a future blog post, I will demonstrate an Alexa Skill with slots.

Once a user speaks a sentence that matches one of your utterances, it will cause the corresponding intent to be invoked. The more utterances you provide for an intent, the more flexible and forgiving your skill will be to the user if they misspeak.

We will take a quick pause from the Alexa Skill entry wizard, save our progress and shift to talk about implementation. When we have our implementation done, we will return to the wizard and finish our skill definition.

Save your work thus far, and make a note of the ID value found on the top banner underneath you skill's name.

Skill Implementation

Skills are implemented via either an AWS Lambda function or through a custom web application. Since Lambda is considerably simpler, less expensive, more scalable, and generally more fun I opted for that route.

I am using nodejs for this example and will be using the nodejs Alexa SDK.

Log into your AWS account, and go to the Lambda page and click the Create a Lambda function button. (You may have to click Get Started if this is your first Lambda function).

You will be prompted to select a blueprint. Since we are using nodejs, I recommend using the alexa-skill-kit-sdk-factskill template. This template automatically loads the nodejs Alexa SDK for you.

NOTE: If you choose Blank Function you will need to download the nodejs Alexa SDK and bundle it with your code as a zip file. I had trouble getting this to work from Windows due to issues with zip file paths. Probably user error, but proceed with caution.

Next you will need to set the trigger for your Lambda function to be Alexa Skills Kit.

Lastly, you will need to set a Name and Description for your function, select the latest nodejs Runtime, and paste in your implementation code.

A couple things to note. First, notice that I provided my skill's ID in the APP_ID variable. This prevents other skills from invoking my Lambda function.

Second, notice that each intent has a matching function mapped to it. These functions are invoked whenever a matching utterance for that intent is spoken. The Alexa SDK provides methods (such as emit() to handle responses).

Once your Lambda function is created, save a copy of the Lambda function's ARN.

We will now return to the Amazon Developer site to finish filling out the skill entry.

The next step in the wizard is Configuration. Here you have a choice to attach your skill to a Lambda function or a custom web application. Use the ARN of your Lambda function that you saved in the prior step.

Testing

The fourth step in the skill entry wizard is for Testing. Amazon provides a number of ways to test your Alexa Skill. I would recommend that you do two things.

First, make sure that the skill is enabled for your account. This will allow your devices to use the skill without having to publish it to the marketplace.

Secondly, you will want to use this page to generate a test event for your Lambda function.

You can type a sample utterance into the Text test section and it will generate a Request to your Lambda function and the Response it receives.

You can use the request in the Lambda console (by opening up your Lambda function and clicking the Test button) to troubleshoot issues with your function. If you have any bugs, sending the test event from the Lambda function page will give you an error message whereas the Test page of the skill wizard will not.

Once you are able to successfully test your skill via the Text test panel, you can try it on one of your own devices!

Assuming your devices are attached to the same account you are using in the developer console, they should be automatically updated with the skill as you make changes to it. Just turn to your nearest Alexa and say:

"Alexa, ask work days, how many works days are left in the year?"

Congratulations! You have written your first Alexa Skill!

Conclusion

Best of luck to you as you teach your Alexa new skills!

Questions? Comments? Email me at: smouring@sequoiainc.com!

]]>Introduction

A few years ago my company switched our corporate blog platform from a WordPress instance to Ghost.

Ghost is an outstanding blogging tool. It is simple. Elegant. Easy to use. It meets my personal Occam's Razor of usability: all the features I need but no more.

A few years ago my company switched our corporate blog platform from a WordPress instance to Ghost.

Ghost is an outstanding blogging tool. It is simple. Elegant. Easy to use. It meets my personal Occam's Razor of usability: all the features I need but no more.

The only problem I had was finding out how to upload images. Ghost had the capability to link to images that were already hosted, but for the last two years I could not figure out how to upload images directly into Ghost.

Instead of reading the manual, Googling a solution, or asking for help, I did what any self respecting programmer would do... I wrote my own image server running globally over HTTPS on a custom domain name! And now, with this blog, you can too!

(Full Disclosure: After building my image server, I finally found the interface on Ghost to upload images... User error or bad interface design? You decide. But the experience of creating an elegant solution to a personal problem is both rewarding and educational, and you can use an image server (or a file server which is what this really is) for many other things anyway, so here we go!)

Your Own Image Server

AWS S3 is a solid starting place for an image server. In addition to its reliability and relatively low cost, S3 has the capability for making a bucket publicly accessible, which means file stored in that bucket can be served out directly over HTTP/HTTPS. With the roll out of the new console interface for AWS, this process has become considerably more streamlined.

Log into your AWS Management Console. Go to S3 and create a bucket.

On the Set permissions step of the Create Bucket Wizard you will have a chance to set the Manage public permissions option to Grant public read access to this bucket.

This will enable HTTP/HTTPS access to this bucket.

Once this is done you will want to create a directory structure inside your bucket to manage your content. The exact organization is up to you. I created an images folder with a blog subfolder since in the future I might use this "image server" for other types of images or even for other types of file altogether.

As an exercise, go ahead and upload an example image file.

Be aware that even though the bucket is public, each file you upload will also need to be made public as well.

Once you have uploaded a file, you can see the URL to access that file in the file's Properties page in the S3 Console.

Congratulations! You now have a simple and reliable image server built on S3!

Globally

But why stop there?

What if you want all your images to be cached globally at dozens of data centers around the world to minimize their load times??

This lofty goal is easily achievable by putting an AWS CloudFront distribution in front of your S3 Bucket!

To do this, go to the AWS Management Console and go to CloudFront. Click Create Distribution and opt for a Web distribution type.

For Origin Domain Name you will want to select your S3 bucket (pre-populated in the dropdown list). You can also specify a S3 folder prefix under Origin Path. This lets you create different CloudFront distributions for different S3 folders, which means you can cache different kinds of content with different rules. You can also choose to disable the direct S3 URLs via the Restrict Bucket Access option (although this is not necessary in our case).

CloudFront is a complex beast, with many options to control security, caching, and other distribution characteristics. For our purposes it is safe to accept the default options for now and we will revisit some of them later.

Once your distribution has been created and is fully deployed you can now use its DNS name to access your content like this:

https://dxnjrp250socj.cloudfront.net/blog/example_image.jpg

Congratulations! You now have a global image server!

Securely

But wait! There is more!

What if you want to serve your content over your own domain name via HTTPS?

Note: If you do not have a domain name, you can easily register one through AWS Route 53. In my example below, I am leveraging my stephenmouring.name domain that I already had registered.

To make this work you need to achieve two steps. First you need to have your domain name redirect traffic to your CloudFront distribution. then, you need to create a signed certificate for your domain name and associate it with your CloudFront distribution.

The first step requires you to add a record to your domain name's record set. In my case I am using Route 53 to manage my domain name DNS. Since I use stephenmouring.name for many different purposes, I chose to use a subdomain: images.stephenmouring.name

I added an A record to my record set that pointed the subdomain images as an Alias (make sure to check Yes in the Alias toggle box!) to my CloudFront distribution as an Alias Target.

So now, all images.stephenmouring.name traffic will be directed to the CloudFront distribution. The second step is to prepare the CloudFront distribution to accept that traffic!

We do this by requesting a custom security certificate from the AWS Certificate Manager service. I recommend that you request a single certificate that can be used for all your subdomains (using a *. prefix as shown below). This will save you the hassle of requesting a certificate for each subdomain in the event that you want to use a certificate in AWS in the future.

So here is a major gotcha. Amazon requires that you verify that you own a domain before you can register a certificate for it. It does this verification through an email. It will use the email registered with the domain in WHOIS. However, if you opted for the privacy option when registering your domain it will use a set of five common admin emails (admin@<hostname>, info@<hostname>, etc.)

If there is no way to respond on one of those emails you will be unable to register your certificate! This represents a considerable catch 22 since we are only using the domain for CloudFront and it does not have any infrastructure behind it.

My workaround is not for the faint of heart (but was really fun nonetheless!) I signed up for a free account via Zoho set it up as the provider for my custom domain. I then created an admin account (which meant I had an admin@stephenmouring.name email) that was capable of accepting the verification from AWS Certificate Manager.

This took a lot of work but was a good learning experience and is a good workaround to avoid having your personal details publish in WHOIS for your domain name!

Once you have your certificate in hand, you can return to AWS CloudFront, and edit your distribution. You will want to change two things: the CNAME used and the security certificate.

Once your distribution finishes deploying your changes you can then access images over your custom domain name like this:

https://images.stephenmouring.name/blog/example_image.jpg

Congratulations! You now have a secure image server running on your own domain name!

Conclusion

Is this an overkill solution for hosting images to link from a blog site? Yes. Is this a rewarding learning experience that teaches you about S3, CloudFront, Route 53, DNS and other good things? Yes, absolutely!

I hope you found this useful! Email me at smouring@sequoiainc.com with any comments, questions!