Random thoughts, musings, and tech stuff

04 Aug 2015, 08:33

Hello to whomever is reading this, you’ve landed on my new blog. This will be my primary collection of junk on the web. I’ve never been super happy with having my content live outside of my control, and this effort intends to fix that issue. That basically means that I will be posting here a lot, and share this content out with to other outlets.

This is a work-in-progress. I’ve already imported three separate blog’s worth of content here, and I’m continuing to add the things that I’d like to keep. So, if you find things broken, or a bit off, hopefully I’ll get to fixing it.

On a technical note, I’ve been wanting to try out a static site generator for a while, and landed on Hugo. It’s written in GoLang, and was the only tool I tried that worked as advertised out of the box. It also happens to be quite speedy.

Dockerfile

testfile

As you’re writing your Dockerfile, it may be helpful to run the commands against a real
image as you go, so that you can more easily predict what’s happening. Here is how to do
that:

docker pull debian:latest
docker run -t -i debian:latest /bin/bash

If you want to generate a docker image based off of what you did in the interactive session
vs a Dockerfile, you can take note of the image id in the console after root. E.g.
root@28934273 # specifies the user is root and the image id is 28934273. Then, you can
run whatever commands you’d like, exit the session, and run the following:

You can use the image you just created as a base image in another of your Dockerfiles, so that
you can interactively set up your image initially, and then in the second step, add any CMD
statements to actually run your software.

Code

02 Jun 2015, 06:21

Using Cardboard with unsupported VR viewers

The Project Tango booth at Google I/O 2015

On Saturday, I was hanging out with some friends at the SFVR Project Tango Hackathon & VR Expo. The Tango team had handed out Tango devices and Durovis Dive 7 VR masks to develop with. I was feeling pretty dead from Google I/O, and the surrounding parties, but Etienne Caron and Dario Laverde were busy doing their best to build something on the Tango using the Cardboard SDK. They both fired up demos, and immediately found an issue. The screen was not configured correctly for the device, or the viewer.

Giant Cardboard hanging at Google I/O 2015

It turns out that Cardboard expects a configuration that is typically provided by the manufacturer of the viewer, but the Dive doesn’t come with one. Etienne noticed that there was nothing documented in the Cardboard API that related to screen or viewer configs. He and I started to poke at the debugger, trying to figure out if we could find the place that those values get set. We traced it back to a file on the sdcard, but when we looked at it, we realized that it was in serialized protobuf format. My initial thought was to copy the files that read the protobuf and decode the file, but we realized that there was an easier way, the Cardboard View Profile Generator.

Etienne and I generated config files, and Dario helped us test. Dario was also working on some other Cardboard related issues. Here’s what we did:

19 May 2014, 15:18

Witness, a simple Android and Java Event Emitter

Source. I found this in an image search for ‘witness’. Had to use it. =)

I’ve been working on a project for Google Glass and Android that requires asynchronous messages to be passed around between threads. In the past, I’ve used Square’s Otto for this, but I wasn’t thrilled with the performance. Additionally, Otto makes some assumptions about which thread to run on, and how many threads to run with, that I wasn’t crazy about. Before continuing, I should point out that Otto is configurable, and some of these basic assumptions can be dealt with in the configs. Regardless, I felt like writing my own.

Code

Witness is a really simple event emitter for Java. The idea here is to build a really simple structure that can be used to implement observers easily. It’s sort of like JavaScript’s Event Emitter, and the code is short enough to read in its entirety.

I’ll start with Reporter because it’s so simple. Essentially, this is just an interface that you implement if you want your class to be able to receive events.

The Witness class maps class types to Reporter instances. This means that for a given data type, Witness will fan out the event to all registered Reporters. It uses an ExecutorService with a pool size of 10 threads to accomplish this as quickly as possible off of the main thread:

Usage

To receive events for a specific datatype, the receiving class needs to implement the Reporter
interface, then register for whatever data types the following way:

Update

It has the effect that events can be delivered on arbitrary threads. The event handler must be threadsafe, must synchronize access to shared or stateful resources, and be careful not to create deadlocks. Otoh if you use a single event queue, you avoid this kind of complexity in the first place. I’d opt for the latter, threadsafe programming is something you want to avoid if you can.﻿

I should note that my usage is all on Android, where I’m explicitly specifying the thread that the events will run on using Handlers. I haven’t used this in a non-Android environment, and I’m not entirely sure how to implement the equivalent behavior for regular Java.

20 Mar 2014, 18:11

AWS S3 utils for node.js

I just published my first package on npm! It’s a helper for S3 that I wrote. It does three things, it lists your buckets, gets a URL pair with key, and deletes media upon request.

The URL pair is probably the most important, because this allows you to have clients that put things on S3 without those clients having any credentials. They can simply make a request to your sever for a URL pair, and then use those URLs to put the thing in your bucket, as well as a public GET URL, so that anyone can go get it out.

I had previously written about using the AWS SDK for node.js here. It includes some information about making sure that you have the correct permissions set up on your S3 bucket, as well as how to PUT a file on S3 using the signed URL.

19 Feb 2014, 20:57

Crashlytics is awesome!

I recently started playing with Crashlytics for an app that I’m working on. I needed better crash reporting than what Google Play was giving me. I had used HockeyApp for work, and I really like that service. My initial thought was to go with HA, but as I started looking around, I noticed that Crashlytics offers a free enterprise level service. No down-side to trying it!

I gave it a shot, they do a nice job with Intellij and Gradle integration for their Android SDK, so setting up my project was quite easy. I tested it out, and again, it was very simple and worked well. The reporting that I got back was quite thorough, more than what anybody else that I’m aware of gives you. It reports not just the stack trace, but the state of the system at the time of your crash. If you’ve ever run into an Android bug, this sort of thing can really come in handy.

But, then I ran into an issue. I had some thing that was acting funny, so I pinged the Crashlytics support. I was pretty sure that it was an Android bug, but hadn’t had time to really nail down what the exact problem was. After a short back and forth, I let them know that I’d try to dig in a little more when I had time, but that I was busy and it might not be until next week. The following day, I received a long, detailed response, that included Android source code, to explain exactly the condition that I was seeing. I was floored. They had two engineers working on this, figuring out exactly what the problem was, and what to do with it. I don’t think that I could imagine a better customer service experience!

As a note, I have no affiliation with Crashlytics outside of trying out their product for a few days. Their CS rep did not ask me to write this. I was so impressed that I wanted other people to know about it.

01 Feb 2014, 19:45

Demos using AWS with Node.JS and an AngularJS frontend

I recently decided to build some reusable code for a bunch of projects that I’ve got queued up. I wanted some backend components that leveraged a few of the highly scalable Amazon AWS services. This ended up taking me a month to finish, which is way longer than the week that I had intended to spend on it. (It was a month of nights and weekends, not as much free time as I’d hoped for in January.) Oh, and before I forget, here’s the GitHub repo.

This project’s goal is to build small demo utilities that should be a
reasonable approximation of what we might see in an application that uses the aws-sdk node.js module. AngularJS will serve as a front-end, with no direct access to the AWS libraries, and will use the node server to handle all requests.

The same schema will be used for both Dynamo and RDS, almost. RDS uses an mkey field in the media table, to keep track of the key. Dynamo uses a string id, which should be the key of the media object in S3.

Using the above schema, we set up a couple Dynamo tables. These can be treated in a similar way to how you would treat any NoSQL database, except that Dynamo’s API is a bit onerous. I’m not sure why they insisted on not using standard JSON, but a converter can be easily written to go back and forth between Dynamo’s JSON format, and the normal JSON that you’ll want to work with. Take a look at how the converter works. Also, check out some other dynamo code here.

There are just a couple of things going on in the DynamoDB demo. We have a method for getting all the users, adding or updating a user (if the user has the same id), and deleting a user. The getAll method does a scan on the Dynamo table, but only returns 100 results. It’s a good idea to limit your results, and then load more as the user requests.

The addUpdateUser method takes in a user object, generates an id based off of the hash of the email, then does a putItem to Dynamo, which will either create a new entry, or update a current one. Finally, deleteUser runs the Dynamo API method deleteItem.

The following are a few methods that you’ll find in the node.js code. Essentially, the basics are there, and we spit the results out over a socket.io socket. The socket will be used throughout most of the examples.

This one’s pretty simple, RDS gives you an olde fashioned SQL database server. It’s so common that I had to add the ‘e’ to the end of old, to make sure you understand just how common this is. Pick your favorite database server, fire it up, then use whichever node module works best for you. There’s a bit of setup and configuration, which I’ll dive into in the blog post. Here’s the code.

I’m not sure that there’s even much to talk about with this one. This example uses the mysql npm module, and is really, really straightforward. We need to start off by connecting to our DB, but that’s about it. The only thing you’ll need to figure out is the deployment of RDS, and making sure that you’re able to connect to it, but that’s a very standard topic, that I’m not going to cover here since there’s nothing specific to node.js or AngularJS.

The following are a few methods that you’ll find in the node.js code. Essentially, the basics are there, and we spit the results out over a socket.io socket. The socket will be used throughout most of the examples.

This one was a little tricky, but basically, we’re just generating a unique random key and using that to keep track of the object. We then generate both GET and PUT URLs on the node.js server, so that the client does not have access to our AWS auth tokens. The client only gets passed the URLs it needs. Check out the code!

The s3_utils.js file is very simple. listBuckets is a method to
verify that you’re up and running, and lists out your current s3 buckets. Next up, generateUrlPair is simple, but important. Essentially, what we want is a way for the client to push things up to S3, without having our credentials. To accomplish this, we can generate signed URLs on the server, and pass those back to the client, for the client to use. This was a bit tricky to do, because there are a lot of important details, like making
certain that the client uses the same exact content type when it attempts to PUT the object. We’re also making it world readable, so instead of creating a signed GET URL, we’re just calculating the publicly accessible GET URL and returning that. The key for the object is random, so we don’t need to know anything about the object we’re uploading ahead of time.
(However, this demo assumes that only images will be uploaded, for simplicity.) Finally, deleteMedia is simple, we just use the S3 API to delete the object.

There are actually two versions of the S3 demo, the DynamoDB version, and the S3 version. For Dynamo, we use the Dynamo media.js file. Similarly, for the RDS version, we use the RDS media.js.

Looking first at the Dynamo version, getAll is not very useful, since we don’t really want to see everyone’s media, I don’t think this even gets called. The methods here are very similar to those in user.js, we leverage the scan, putItem, and deleteItem APIs.

The same is true of the RDS version with respect to our previous RDS example. We’re just making standard SQL calls, just like we did before.

You’ll need to modify the CORS settings on your S3 bucket for this to work. Try the following configuration:

The following are a few methods that you’ll find in the node.js code. Essentially, the basics are there, and we spit the results out over a socket.io socket. The socket will be used throughout most of the examples.

SES uses another DynamoDB table to track emails that have been sent. We want to ensure that users have the ability to unsubscribe, and we don’t want people sending them multiple messages. Here’s the schema for the Dynamo table:

Emails: {
email: "steve@example.com",
count: 1
}

That’s it! We’re just going to check if the email is in that table, and what the count is before doing anything, then update the record after the email has been sent. Take a look at how it works.

Sending email with SES is fairly simple, however getting it to production requires jumping through a couple extra hoops. Basically, you’ll need to use SNS to keep track of bounces and complaints.

What we’re doing here is for a given user, grab all their media, package it up in some auto-generated HTML, then use the sendEmail API call to actually send the message. We are also keeping track of the number of times we send each user an email. Since this is just a stupid demo that I’m hoping can live on auto-pilot for a bit, I set a very low limit on the number of emails that may be sent to a particular address. Emails also
come with a helpful 'ubsubscribe’ link.

The following are a few methods that you’ll find in the node.js code. Essentially, the basics are there, and we spit the results out over a socket.io socket. The socket will be used throughout most of the examples.

We’re also listening for SNS messages to tell us if there’s an email that’s bounced or has a complaint. In the case that we get something, we immediately add an entry to the Emails table with a count of 1000. We will never attempt to send to that email address again.

I have my SES configured to tell SNS to send REST requests to my service, so that I can simply parse out the HTML, and grab the data that I need that way. Some of this is done in app.js, and the rest is handled in bounces.js. In bounces, we first need to verify with SNS that we’re receiving the requests and handling them properly. That’s what confirmSubscription is all about. Then, in handleBounce we deal with any complaints and bounces by unsubscribing the email.

The AngularJS code for this is pretty straightforward. Essentially, we just have a service for our socket.io connection, and to keep track of data coming in from Dynamo and RDS. There are controllers for each of the different views that we have, and they also coordinate with the services. We are also leveraging Angular’s built-in events system, to inform various pieces about when things get updated.

There’s nothing special about the AngularJS code here, we use socket.io to shuffle data to and from the server, then dump it to the UI with the normal bindings. I do use Angular events which I will discuss in a separate post.

Here’s the AWS doc on setting up deployment with git integration straight from your project. It’s super simple. What’s not so straightforward, however, is that you need to make sure that the ports are set up correctly. If you can just run your node server on port 80, that’s the easiest thing, but I don’t think that the instance that you get from Amazon will allow you to do that. So, you’ll need to configure your LoadBalancer to forward port 80 to whatever port you’re running on, then open that port in the EC2 Security Group that the Beanstalk environment is running in.

Once again, do use the git command-line deployment tools, as it allows you to deploy in one line after a git commit, using git aws.push.

A couple of other notes about the deployment. First, you’re going to need to make sure that the node.js version is set correctly, AWS Elastic Beanstalk currently supports up to v0.10.21, but runs an earlier version by default. You will also need to add several environment variables from the console. I use the following parameters:

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

AWS_RDS_HOST

AWS_RDS_MYSQL_USERNAME

AWS_RDS_MYSQL_PASSWORD

Doing this allowed me to not ever commit sensitive data. To get there, log into your AWS console, then go to Elastic Beanstalk and select your environment. Navigate to ‘Configuration’, then to ‘Software Configuration’. From here you can set the node.js version, and add environment variables. You’ll need to add the custom names above along with the values. If you’re deploying to your own box, you’ll need to at least export the above environment variables:

29 Jan 2014, 18:51

Hacking Crappy Customer Support

The Situation

We had an issue at work the other week. Basically, we were running into some pretty serious problems with one of the SAAS services that we use (I will leave which one to the reader’s imagination). This is a mission critical service for a couple of our offerings, and it’s not particularly cheap, at $200/month for the pro service (which is what we have). This is a service that is structured in such a way that for anybody that uses it, it’s likely to be a mission critical service. This is all well and good, except that they don’t seem to bother answering support emails. We’ve had emails to them go totally unanswered before. However, until last week, we hadn’t run into an issue important enough that we really, really needed a response.

We found ourselves in a situation where we had taken a dependency on a third-party service, ran into an issue, and were getting no help from the provider. We had guessed at a work-around that turned out to be the right answer for an immediate fix, but we still needed a proper fix for this, else we would need to make larger changes to our apps to better work-around the problem. The vendor was not answering the urgent emails, and provided no phone number for the company at all.

The Hack

I had an idea. The provider has an Enterprise tier, we could contact the sales team, and say that we’re looking to possibly upgrade to Enterprise, but that we had some questions that needed to get answered first. We structured our questions in a way that first asked what we needed to know, and then asked if the Enterprise tier might solve the issue, or if they were working on a fix. This tactic worked. A couple of us separately sent emails to the Enterprise sales team (they, too, do not have a phone number listed) and received responses fairly quickly. We got our questions answered after a couple rounds of emails.

Concessions

It’s true, they didn’t promise any support, even at the pro level; we had mistakenly assumed that we’d at least be able to get questions answered via email. We should have probably done a bit more research before choosing a provider. The provider may not have been set up to handle that much in terms of support. However, they are not a small company, and could at least offer paid support as needed.

Conclusions

As far as whether or not we stick with them, we’ll see. I’m not too thrilled about paying a company $200/mo and getting the finger from them whenever something of theirs is broken. But, there are other constraints, and you can’t always get everything you want. Sometimes there simply isn’t the time to go back and fix everything that you’d like to.

python sweetness: How to lose $172,222 a second for 45 minutes

This is probably the most painful bug report I’ve ever
read,
describing in glorious technicolor the steps leading to Knight
Capital’s \$465m trading loss due to a software bug that struck late
last year, effectively bankrupting the company.