So today I faced an interesting problem. How do I capture email chains and post them on a website? The solution was to create an email that will always be CC’ed onto the email. This email will then forward the conversation to our Rails app. We can easily do this off Mailgun and Heroku.

So the first thing you need to do is to create the route for Mailgun to pass along the email to. So add a new route to your route table and create a controller for it:

Make sure you have no authentication blocking this route and if you do, you probably need some skip_before_filter to allow Mailgun to access this route. After this, you will need to tell Mailgun this route along with an email you created with them. So you can log into Heroku, go to Mailgun, create an email, and go to the routing page. Then just fill out your information. It should look something like this:

Now you have a working controller to capture your email and an email address that will forward your emails. Next up is to create your email:

Now when this email goes out, it will be sent to your destination as well as this capture email you’ve made. So it will create a post automatically for your first initial email. Then whenever someone reply-all to the thread, it will be captured along the way and stored in your database to display on your Rails app.

While this is nice, if you have multiple email threads going on at once with different users, it will become very hard to keep track of which replies belong to which user on your site. A simple solution is to create a special token for the initial post and append it to the footer of your email. This footer can say something like “Make sure to reply-all to store this thread {THREAD#abc123}”. This way in your controller, you can create a regex method (which I’ll leave to you) that will parse through the body and find that token.

Now this creates more problems since you probably want the first email to show immediately on your site and you want the token saved as well. This is because emails usually are running in the background on a delayed job, it would be a bad experience for the user to not see their message show up immediately. So we have to create a post immediately when they click send. But the problem is now when your Mailgun email gets that email, it will create a duplicate post. Luckily there is a way around that:

So Mailgun allows you to past a JSON along in your email. This is great because its a simple way to pass along information that this is the first email in this thread. Then the reply-all will not pass along this JSON header so if we do see it, we will know for sure to ignore the email and not create a post.

Now when we send our email, that data will be passed along in our header which we can parse easily in our controller:

class MailgunController
def add_to_thread
# all the mail information will be in params
unless params["X-Mailgun-Variables"] and params["X-Mailgun-Variables"]["first_email"]
Post.create(:message => params["body-plain"])
end
render :json => {:success => true }
end
end

We know can capture email threads with our Rails app. Testing this though is really hard since Mailgun cannot access your localhost. So unless you want to keep pushing to production and testing it (which is extremely time consuming), there are ways to open up your localhost to the internet. There are many services out there that will do it for you. I would recommend ngrok (https://ngrok.com/) which is free for most basic features. After you run this tunneling program, it will return to you a new public URL which will point back to your machine. Go back to Mailgun routing page and add in that URL and point it to your controller. Now you can easily test your emails on your local machine and hopefully come up with some neat email features.

Usually for most of my programs these days on Rails, I would deploy my code through Heroku (https://www.heroku.com/) which makes things extremely easy and free for very small projects. It has a bunch of add-ons which add lots of functionality to your website. Though I’m not gonna talk about that today because there are plenty of other guides out there on how to set up and Heroku is pretty popular already to host your apps. The problem I was running into was a way to upload my current side projects. One is an HTML5 game while the other are Android apps.

I wasn’t going to create a whole app just for my HTML5 game because the game only consist of the HTML, Javascript and CSS. There was no need to create a whole project. So I was looking around for a quick easy way to upload my files for my friends to test out without going through the trouble of registering a domain and hosting a website. After some quick research, I found out that I could host basic HTML, Javascript, and CSS files on Dropbox (https://www.dropbox.com/help/16/en).

It should create a Public folder for you. After that, you just paste in all your code and click on your index.html. There should be a button at the top that says “Copy public link”. And then you can share that with your friends!

For my Android apps, there was no easy way to share it. I could always just release the APK but that would be annoying to constantly upload somewhere and then put into your Android device. So I decided to just upload it to Google play. It was a lot easier than I thought it would be.

You have to pay a one time $25 which I guess is there to make sure bad quality apps don’t get through. After paying that, you just have to upload your APK. I had my project in Eclipse and it had a nice feature to export your project as an APK file. You just right click it and export it as a signed Application Package.

Then you will have to create a key to sign it with and there’s a bunch of forms to fill out. After that though, you will have your APK file. You then go back to the Google Play to upload this app. After that, you will need to go through a checklist to pick which country and region could use your app, the pricing, and you’ll need at least two images to display in the app store. When everything is good to go, it will give you an option to publish it and it will be up in the app store within a few hours.

It shows you cool stats as well and it seemed my app was mostly downloaded by people in Asia (I guess that makes sense since lion dance is a lot more popular over there). But it’s interesting how people find and download apps so quickly without me doing anything. I thought I would have to force people to download it but this worked out better. I’ll do a post soon about Android development and using AndEngine.

Image uploading has always been a problem for me at my current job. This is because we use paperclip to process our image into many different sizes (thumbnails, small squares, large image, etc). Processing these images take some time and if there’s enough users trying it at the same time, the request will timeout. This was a big issue for us because as our company scales, the timeouts were happening more frequently. So this was a critical issue that needed to be addressed immediately. So our solution was to use Amazon S3 and delayed jobs (https://github.com/collectiveidea/delayed_job).

The idea was to upload the image up to Amazon S3, which will then return an image URL. So this uploading part does not touch our backend at all so even large images should be fine. It will not eat up any web threads. Once we get the URL from the cloud storage, we will send that to our backend. This will create an image with this URL and will use delayed jobs to process that image. Delayed jobs will basically put it in a queue and use workers to process the image in the background. This way, the user can get quick feedback and it will not hold up a web thread. The drawback to this is that the image might take long time to process if there are hundreds of jobs in the queue. The user will have a poor experience if they wonder why their image isn’t showing. But there are ways around that by using the temporary image URL from Amazon S3.

So first things first. We need to create the image model. It only really needs a two fields for now. Just the file attachment and the unprocessed image URL from Amazon S3.

The IMAGE_STYLES are just all of the different sizes of the images we need to cut. Then we have a process_image method that will open up the URL. By putting it into the image field, Paperclip will process that image for us.

The next step is to create a signed URLs controller. This controller will be used to create the Amazon S3 upload policy document and signature. This will be called when you upload an image. It will generate a json with these info and pass this along with the image to Amazon S3. It’s basically our credentials to make sure we have the rights to upload an image to our Amazon S3.

Next is a controller to actually upload our image URL we get from Amazon S3. It will create the image model and then use delayed jobs to put the processing into a queue. It’s pretty straight forward the the json_for_image just either checks if the image is there or not. If the image is processed, it will grab the processed image and if not, it will use the unprocessed_image_url. This is just for preview. If you don’t need that, you can just pass a success.

Now that our backend is complete, we need to work on the image uploader form. We need a good image uploader and there is a really nice one online (https://github.com/blueimp/jQuery-File-Upload). It has tons of neat features such as progress bar, chunked uploads, client-side image resizing, and the list goes on. But I’m using it mostly for the drag and drop support as well as the multiple file upload at once. So here is the HTML part in HAML:

You will need to have the Amazon S3 protocal, bucket, domain, and access_key_id key ready. I also have created a drop zone which you can point to any div where you can drag and drop images to. After you set this up, you will need to initialize the plugin in the Javascript (this will be in coffeescript):

As you can see in the add block, it will hit our signed_urls controller to fetch that json we set up earlier. If it succeeds, it sets the key, policy, and signature for Amazon S3 and then submit the data. When that’s done, Amazon will pass back a bunch of data but we only want the location of that image. So we grab that and created a Backbone model with that URL to submit. Though I only printed out “Success” in this example, you can use the image json to show the preview.

And that concludes how to upload mass amount of pictures without locking down your site. I will have posts later on about how we had weird critical bugs with this and how we managed cropping. Thanks!

I’m not going to cover every little detail of my game since that would be too much work and writing but I will write about interesting things that I stumble upon. Today I would just be explaining some of my code for the core gameplay. The first was adjusting the movements from last time:

This was a great way to get basic movements in since it would just sit on top of the mouse. So regardless of how fast the mouse was moving, or taken off screen and put back on screen at another spot, the ship would either move really fast or teleport. The game wouldn’t be about dodging bullets well in advance anymore but rather just clicking on empty spots on the screen to teleport your ship there. So I made a minor adjustment.

_my_ship_target_x is a global variable I set to null. So when the mouse is on the canvas, the location is stored. And when the mouse leaves the canvas, that variable is set to null. So then I created a method called myShipMove and I called it in my update method which is being called by the Ticker 60 times a second.
This way, the ship will try to get closer to this target position every cycle.

So I created another variable called _my_ship_speed to adjust the speed. One big thing I added was checking if the distance between the ship and the target was less than the ship speed. Without that, it was impossible to get to that exact location of the mouse because the change in location wasn’t a small enough interval to hit that exact location. So it would cause weird jitteryness which was solved when I told it not to get to that target position if the difference was too small.

The next part of my core game play was the holding and releasing of bullets. Again, my game idea was that our spaceship would not be able to shoot any bullets but rather to hold and release enemy’s bullet. But as you hold the bullets, their directions will change to retarget your ship again but their speed will drastically drop. So if you hold the bullets on for too long, you will still get hit by the bullets. So first step was to create the shield to indicate the hit area this pull.

Everything is straight forward but my mouse up and mouse down. I created two booleans: _flux_activate and _flux_capture. And I set them both to false when I instantiated them. Flux capture is used to get that exact cycle the bullet entered the shield and the moment the stage is clicked. Any bullets in the shield at that time will be pulled in. Then the idea is to set that boolean to false right after that cycle. This will allow any other bullets to enter the shield and not be pulled in even while you’re holding down the bullets. While flux activate is to know if the mouse is still being held down. So on mouse up, those variables are reset back to false and a pushBullets method is called which I’ll explain in a bit.

So I updated my enemyBulletTravel method to check if the bullets are being pulled in, and then retarget the bullets and move them to my own bullet array. This way, I can keep track of my own bullets since they will be moving with the ship and they have their own attributes, such as not being able to hurt the enemies yet. So I also need to update myShipMove to make my pulled bullets move with me. And the last part of the puzzle is to move the pulled bullets after retargeting it.

Lots of things going on here but it’s just the same thing for bullets moving. Nothing really interesting to note. So now we go back to the pushBullets method I was referring to earlier. This method will grab all the bullets in the pulled and invert the bullets direction and increase the speed. Then I have another method to move it and check for collisions with the enemy to delete:

And that’s the basic gameplay for Flux. The next step is the art because I want to make sure the dimensions work well and that the game looks polished. Then it’s off to level design, more enemies, powerups, menu screen, and just cleaning up the game.

One of my new years resolution was to create a mobile game this year and I will be using this blog to document my journey. It will be a great learning experience for me and it will allow me to mess with all these new tools. So I have a game idea that has been constantly evolving. I think the process of game design is a very interesting topic and I would like to share the thoughts that lead up to my game design.

So mobile games usually have very simple buttons. Usually just a click on the screen. This is because there’s not much space on a tiny screen for anything else and multiple buttons would just be hard on the user to actually see the screen with their fingers blocking everything. So I thought back on old classic simple games which I played before. And I remembered Jezzball. Jezzball is an extremely old Windows game. The point of the game is to separate the balls by building walls without the balls touching the walls while it’s constructing. The game only required two buttons: one to switch directions and one to build the wall. So in order to limit that to one button click, I decide to make the walls circular. So by holding on to a certain area on the screen, a wall would build around the bouncing balls! One problem with that though is that it’s hard to see the ball you’re trying to trace with your fat fingers blocking it. Would be great for a PC with a mouse but a terrible experience for a mobile user. Also capturing balls all day doesn’t seem that captivating so let’s throw in something that would make it more interesting. Adding in higher stakes.

I decide to throw in a space ship and make this one of those 2D shooter airplane game. That way, you can protect your ship from all these bouncing balls. Though that would be hard to explain why there are bouncing balls in the first place. So it was time to add enemy ships who shoot circular bullets where I have to capture. But that idea of your finger covering up the ball still haunted me. I knew it was an interesting concept but the player would be frustrated by this problem. And that’s not something I want my players to feel when they play my game. But I LOVE the idea of not having your own bullets and using what’s around you to win.

The name Flux came from a Heroes of Newerth character. His ultimate power isn’t about doing damage, but about dragging or pushing away the enemies. That idea is so interesting because it leads to insane plays depending on the environment and the cards he is dealt. So with that in mind, I thought about creating a ship where you can either pull in or push out bullets. Combining that with only one button makes this challenging but we’ll just have to wait and see how enjoyable it will be.

Since I don’t have an any iOS, I decide to create it for the Android. But after reading up and setting up my environment, I remembered how painful and ugly Java was. And since I love Javascript, I decide to make an HTML5 game instead.

Setup was a pain. Got VMPlayer to boot up Ubuntu. Installed and setup all my necessary files and programs for this project (node, git, etc). I created a Github repository (https://github.com/jdnguyen/flux). And my IDE of choice has always been something from JetBrains because I’ve been using RubyMine before and it has been amazing. So I got their Javascript IDE, WebStorm. I also installed Coffeescript and hooked it up to WebStorm. Coffeescript just makes it so much easier to code and read. I setup a watcher on the project so it could translate any Coffeescript files immediately to Javascript after each save. I can explain this long tedious task in another post if anyone is interested.

Next up was to actually start an empty project. Quick Googling showed me HTML5 Boilerplate (http://html5boilerplate.com/). It basically starts an empty HTML5 project for you. Which is amazing because it takes into account all browsers, setup all your necessary files, and puts them in an extremely neat format. Made my life so much easier.

Next step, finding a good game engine. I found this great HTML5 Engine called EaselJS (http://createjs.com/#!/EaselJS) which seemed to have what I need: Basic shape drawing, mouse interaction, hit detection, and really easy to update and draw frames. I downloaded the minified Javascript file and loaded that into my project.

Time to start actual coding. First step is to draw my canvas where the game will take place. This should go inside your index.html file in the body block (I took out the brackets because the blog was being weird about showing html):

canvas id="canvas" width="500" height="500"

You should probably add a black background-color to the stylesheet for this canvas to make the background black (since we’re in space). Everything now will be in Coffeescript because it’s beautiful. I put it all into main.cofeee which would translate to main.js. I initialized the canvas and ship:

This creates a canvas, create my ship, puts it on the bottom of the screen, and it will move along the x-coordinate with my mouse. stage.update() draws the screen so I have to update the stage each time the mouse moves around the campus. Basic controls are done. Next step is drawing enemies and making them shoot bullets.

I created an array that would store all the enemies. Then I created a method to spawn enemies. These ships will spawn on the far top right of the screen and move left. The enemy would also hold a shootDelay which will be how many cycles before it shoots another bullet and shootCounter to keep track of the current cycle:

Now that we have a method to spawn enemies, we need to call it. So I created two more global fields. One to keep track of how many cycles til it spawns another enemy and what cycle we’re currently on. Now we need this to run constantly regardless of human’s interaction. So there needs to be a game loop that keeps track of enemy’s movement and spawning. Good thing for EaselJS, this is relatively easy to set up. So I end up adding this this:

So the Ticker is set to 60 frames per second. That means it will run my update method 60 times a second. I set my spawn counter to 240. That means an enemy will spawn every 6 seconds. But now the enemies are not moving yet. They’re just spawning in the top right corner. We need to get them to move left until they’re off the screen and then unhook them because we don’t want to keep track of things off the screen. There’s no point to it and it will just slow down the game. Also I like to point out that I removed the stage.update() from the mouse movement. The game is already redrawing 60 times a second right now which is more than enough. Redrawing extra when the mouse is moving would only cause unnecessary code execution.

I created another method called enemyShipMovement and called it inside my update method:

It goes through my array of enemy ships and move it to the left by 2 pixels each cycle. Since the coordinates of the ship is in the top left corner, I have to make sure that the right corner of the ship is off the screen before deleting the ship. So I add the ship’s width to the x coordinate to make sure that it’s less than 0 before deleting. Deletion comes in two parts. I have to first remove the actual shape from the stage and then remove the ship’s hash info from my array. You might have noticed I traversed through the array from the back as opposed to starting i = 0. I did that at first but it caused me a bug. When I delete ships, it would throw off the i counter and skip through ships moving. Now we have ships moving to the left and deleting themselves once they his the left side of the screen.

Time to create the bullets. I created two different methods. One to spawn bullets and one to move the bullets. Spawning the bullets is a bit harder though. It’s spawning on the enemy with the intention of hitting the player. Once the bullet leaves the enemy, it will travel at a certain speed and direction. So I have to keep these info on the bullet. When something moves, it moves with a velocity in the x and y direction. The bullet should always carry the seem speed regardless of the angle it is shooting so we’re gonna need some basic math. The Pythagorean theorem to be more exact.

As you can see, I pass the bullet the starting location of the ship. I spawn it in the center to the enemy ship with it’s target being the center of my ship. The _bullet_speed is a global variable with the speed you want. It just multiples with our directional velocity. With some basic math I’m sure you guys can figure out, I get the x and y velocity and store it into a hash with the shape itself. I then pump that into an array to loop through. I create the second method: the bullet flying.

It looks the same as the ship moving but it can move along the x and y coordinate and that the bullet gets deleted once it goes to the bottom of the screen. I add this method to my update method so it will move the bullets 60 times a second. Now how do I shoot these bullets? I need to tell the ship when to add create these bullets for it to fly. I add on to the enemy ship movement method:

If my ship isn’t destroyed, they it will add to the shoot counter. Once the shoot counter fills up, it will create a bullet from it’s current location and then reset the counter again. If you run the program, you will see plenty of bullets flying at your space ship with enemies flying over. You can tweek with the global variables to get them to shoot more bullets or fly faster.

Last thing I want to do today is to get basic hit detection with the bullets hitting your ship. And with EaselJS, this is really easy. So I add on to the enemy bullet travel method:

If the bullet is not off the screen, it will check the bullets location with my current ships location. EaselJs has a method called hitTest which will return true of the shapes are colliding in any way. If it is true, I destroy the bullet and print out a message. And there we have it for day 1. Flying ships, bullets and hit detection.

None of them gave me the results I wanted. I tested each query by printing out the size of every items queried:

Page.where(:items => {"$size" => 0}).collect{|a| a.items.count}

They all had zeros in the array that was returned by the collect. Clearly this was not working so I took a step back and read up on Mongo. I pulled up their operators list (http://docs.mongodb.org/manual/reference/operator/query/) and tried a few different combinations. First one was trying greater than. Maybe $not wasn’t just working for some odd reason.

Page.where(:items => {"$size" => {"$gt" => 0}})

Same result and I’m back to the drawing board. The next one that caught my eye was exists. So I tried it like this:

Page.exists(:items => true)

This returned me a much smaller array and as I inspected the results, I realized most of these pages have at least one item. Success! But not really. I realized it caught nil’s but not empty arrays. Empty arrays were not caught be the exist but that gave me hope. I could easily just query up where all items are nil’s or [].

Page.where(:items.nin => [[], nil])

Gave me all pages with at least one item. Perfect. But that got me thinking: there has to be a way to query by size. What if for some odd reason I wanted to query up all pages with at least two items? This method would not have worked. I decide to venture more into this problem for the sake of learning.

After more researching and asking around, I realized that you can use Javascript for Mongo querying. Mind. Blown. This changes everything. So I first tried this query using Javascript:

Page.where("this.items.length > 0")

This threw back this weird error: “10071 error on invocation of $where function:\nJS Error: TypeError: this.items has no properties nofile_a:0″. It seems like it was calling length of something that did not exist. I had to NULL check first. So then I tried:

Page.where("this.items && this.items.length > 0")

Perfect! Gave me exactly what I wanted. And with this query, I can look up pages with at least 5 items easily. I read up more on this and it turns out that this was only possible as of version 3.1.0 of Mongoid. So make sure your Mongoid version is up to date and begin messing around.

Sunspot is a Solr-powered search for Ruby. I have never used it before working at my current job but it makes searching and pagination so easy. Solr is this awesome search platform that will index models almost immediately after being updated. It has plenty of other cool features but I will be focusing on the indexing portion of it.

We have had a weird problem for the past few months but we ignored it for as long as we could because it was a tough problem to solve. So we have these sale pages:

class Sale
#fields aren't important for this
end

These sale pages can hold from one to hundreds of items. These sales can range from season themes to fashion style of a certain starlet. So the important thing to take out of this is that an item can be part of many different sales or none at all. Also the ordering of the items on the sale page matters greatly to make them look appealing.

So we have our item model which can be part of many different sales and these items can have different positions on each of these sale. So we created a sale usage model which will be embedded in the item but contain all it’s position for every sale.

Now if I want to grab all items part of the sale, I can do a quick Mongoid query:

Item.where("sale_usages.sale_id" => @sale.id)

The problem with this is that I can’t sort it by the position on the sale usages. There’s no quick and easy way to do this with Mongo. This is because there can be many different sale usages in items and there’s no way to know which of these embedded objects is the one I want to sort by. So we avoided this problem for months by passing the items to backbone and sort it through backbone collection. Sounds like a great solution but there is one problem with this. If there are too many items on the sale, the page timeouts because there are simply too many items to load. This problem is usually very easily avoidable by adding pagination to the page. But wait! We can’t paginate because we’re doing all the sorting on the frontend. So there’s no way to tell what’s on the first page unless we send all the items to the user, which will not work if there’s 200+ items.

Our solution? Sunspot. Sunspot has an interesting concept of creating dynamic fields (https://github.com/sunspot/sunspot/wiki/Dynamic-fields) on the indexed item. So the idea was to create these special fields on the item which will be indexed for fast querying and ordering on Sunspot. We concluded that the best way was to use the sale id as the field name and the position as the value.

I proceeded to read every documentation and thread I can find on dynamic fields but there weren’t much to look at. So I copied and pasted whatever I found online to test. Trial and error is always the best way to learn.

So from what I gathered, this is how you create a dynamic integer field. So I added this to my item searchable block:

searchable do
dynamic_integer :sale_usages do
sale_usages.inject({}) do |hash, sale_usage|
hash.merge(sale_usage.sale_id => sale_usage.position)
end
end
end

That will create a field that will look like this:

{"123456789" => "0", "987654321" => "23"}

Looks exactly like what I wanted so I re-indexed all my items and added the search query to my item controller where the pagination take place:

Since the item position is default at 0 and all the positions have to be greater than that, I queried for the sale.id as the field name with a value greater than -1. That did fetch all the items in the sale so I knew the indexing of the dynamic field worked. But the ordering did not work. It threw back a “wrong constant name 123456789Sort” error. It seemed like it was appending the word “Sort” to the end of sale id and it could not find that constant name. Quick Googling came up with nothing so it was back to trial and error.

Tried all sort of things but they all failed. One attempt was to index the dynamic field as sale.id + “Sort” to get around this error but that didn’t help. Another attempt was to convert the dynamic field name from a string to a symbol but that also didn’t work. But as we looked closer at that last attempt, we realized it might be the quotations in the symbol. It looked like this by the way after calling to_sym on the sale id:

{:"123456789" => "0", :"987654321" => "23"}

The colon represents that it is indeed a symbol but it treated the quotations as part of the field name too. I opened up my console and fired up a bunch of different cases to see what made the quotations show up. And here were my results: