Pages

2010-10-28

In my previous post I showed how to take images uploaded to the Blobstore and resize them before storing them into the Datastore. This was used on PicSoup reliably for 6 months or so. I really wanted to find a solution that could use faster image serving services like Picasa but ran into terms of service issues.

Shortly after writing that blog entry App Engine SDK 1.3.6 was released along with the new fast image serving facilities. There are some good tutorials which explain this in detail so I won't repeat that here.

What I needed for Picsoup was to use the new image serving but to be able to manipulate the images in the Blobstore. This is so I can allow the user to upload images larger than 1MB but then resize them to 800x600 to store them. Typically a JPEG at this size is less than 100KB whereas the originals tend to be around 3MB. I also wanted to provide a rotation feature so the user can correct the image orientation after uploading.

The key to this problem is how to get an image out of the Blobstore, manipulate it, and put it back. The Blobstore API has no methods for writing directly to it, you can only write by uploading data through an HTTP POST and thus creating a new Blob. So the problem breaks down into three steps:

My domain object, a competition entry (ce), has a BlobKey string property. I use the ImageService to make an image from the blob and rotate it to create a new image.

Step 2: Upload the new image to the Blobstore

This is the step that I imagine the App Engine team will get around to adding to the API at some point. It would be nice to have a function to complement makeImageFromBlob(BlobKey) called makeBlobFromImage(Image). In the mean time I have written my own multipart/form-data post routine:

id - a domain object key id used to update the datastore reference to the new blob

cmd - a command string used to determine how to handle the uploaded data

imageBytes - a byte array of the new image that is to be uploaded

it then creates a multipart/form-data payload to send via the URLFetchService. The Deadline has been set to 10 seconds - the current maximum - but as you can see there is still a try..catch block around urlFetch.fetch(req) to catch timeouts. More about this later.

Step 3: Update Datastore references to the new Blob and remove the old Blob

The Blobstore calls back to "/blobimage" as defined earlier in sendToBlobStore when it has finished storing the new blob. So a doPost method is required to handle the incoming callback. When we have finished processing the callback we have to send a redirect and therefore we have to have a servlet request handler ready to respond as well. A possible quirk I've noticed here is that the browser follows the redirect via a GET whereas the URLFetchService follows it with another POST request, therefore the handler has to be available for both.

Here you can see that my handlers for the redirects just send the word SUCCESS. My GWT code reads this and then makes further RPCs to update the front-end. The section to explain here is under the "Handle upload callbacks" comment. What I'm doing here is simply taking the first entry from the UploadedBlobs map and using the key to determine how to process the callback. The key is the "cmd" parameter we passed in earlier to the sendToBlobStore method. I have removed a few handlers from this example for brevity but you can see here how I can have different processing for an initial upload from a browser versus an internal upload following a rotate transformation.

The rotate operation we ran in Step 1 passed in the cmd "save" meaning the saveHandler is called:

In saveHandler there is a little bit of Objectify code to update the datastore object to reference the new blob. The old blob is then deleted. Note my little hack in getServingUrl to iron out a difference between the Development and Production environments.

Timeouts

I arrived at the design above following a number of experiments. The main problem that shapes the solution this way is the URLFetchService timeout. The maximum deadline at the moment is 10 seconds which seems like plenty of time but an IOException for a timeout is regularly thrown. For some reason (any explanation gratefully received) when there is only 1 instance of the app running in production the deadline is always reached. As soon as there are 2 or more instances running this stops happening. Unfortunately the exception thrown is just an IOException and not something more specific like URLFetchDeadlineExceededException which would be much nicer. On the development server this timeout is never reached.

To get around this timeout issue you just have to make sure that any critical code goes into the Blobstore callback handler. For example, I save the change to the domain object in saveHandler and not in my original call in Step 1. In my GWT front-end I have routines to check that the transformation is complete and show a spinner while waiting.

2010-07-31

My App Engine project: PicSoup, is a weekly photo voting competition with a twist (read the Help tab at the site for instructions). The app is all about images. Users upload (or email in) their competition entry which is then resized and displayed on the site for everyone to vote for.

The Blobstore is used for the initial upload to get around the 1MB limit. Sadly this is not available for emails so the user is limited to 1MB for email entry. The large image is then resized and stored as a blob in the Datastore like this:

Note that I'm using the Cache-Control header with a max-age of two weeks. The image with the given id never changes so in theory this could be set to forever. This caching is very important because otherwise the app gets hit every time for the image. Users of PicSoup frequently visit the site to check for new entries .

The downside to this is that sometimes the Datastore can be very slow. I've watched images appear like they used to on an old 56k modem! Google were having some problems with Datastore performance and it is way better now but it's not as fast as accessing a static file on a dedicated server.

The performance and the Datastore usage quota put me off keeping a higher resolution image but the site really needed it. So I started developing a way to store the big images in Picasa. The documentation is really good and I soon had this working. Now when the user uploaded their image the small image would still be stored in the Datastore as above but then a task would be enqueued on the Task Queue to transform the image from the Blobstore again and then upload it to my Picasa account:

Before I started the UI work to display the image from Picasa I checked the Terms of Service and realised this solution may be contrary to item 5.9:

5.9 In order to use the Picasa Web Albums API with your service, all End Users on your service must have previously created their own individual Picasa Web Albums accounts. You must explicitly notify End Users that they are accessing their Picasa Web Albums accounts through your service. In other words, you may not create one or more Picasa Web Albums accounts for the purpose of storing images on behalf of users without those users creating their own individual Picasa Web Albums accounts.

Now, strictly, I'm not sure I'm storing the images on behalf of the users - they've kind of donated them to me and my app. I searched around for some clarification and found that there are plenty of people trying to do this sort of thing and the answer is always no. Have a look at this search in the forum.

So, I've removed Picasa from my app and I'm now using the Datastore to hold an 800x600 image as well. (If you go to PicSoup today [31-July-2010] only the most recent entries have the high-res view available, just click the small image). Now that the Datastore performance has improved this is not so bad.

I've looked at Flickr and Photobucket as well and they also seem to have a clause like this in their terms.

Does anyone know of a service where this is allowed?

UPDATE See my new post which explains how to use the new Blobstore based fast image serving service.

2010-05-16

I recently started adding email notifications to PicSoup (my GAEJ application). I followed the examples in the documentation and wrote a really simple function to allow me to send a simple text email to one recipient:

This notifies a particular user and uses their settings to get the appropriate "personal name". I also wanted to use this call to send notifications to myself. Being the administrator I could do this using the Admins Emailed quota. To do this I thought I could use the special "admins" recipient with my sendEmail function like this:

Sadly I discovered that this doesn't work. It silently fails to send the email to anyone! It turns out that this is because I have included "PicSoup Administrators" as the "personal name" in the InternetAddress object. In order to make this work I changed my sendEmail method to ignore the "personal name" for emails to admins:

2010-04-12

When I'm using a mobile or an iPod Touch the web sites that detect my device and show me a mobile friendly page are more likely to be added to my favourites. For example Digg has a mobile version of their site which you are automatically taken to.

You can detect a mobile device by looking at the HTTP User Agent and Accept headers. Here are these two headers for an Android phone as captured in my application log:

Detecting all the many and varied mobile devices out there would be a real pain without the help of MobileESP. This project supports multiple languages but I'm using Java on Google App Engine so all I needed to do was place the single java source file in my project.

I then wrote a servlet to handle requests to the home URL which redirects to the normal or mobile version of the site like so:

Unfortunately there is a minor issue with this. It seems that some gadget containers ignore the urlparam attribute. When the gadget is hosted in the iGoogle home page it works fine but in Blogger and some other places the id parameter was not coming through in the querystring. Instead you get something like like

http://yourappid.appspot.com/gadget.html?up_profileId=123

Basically the UserPref name prefixed with up_. This meant a minor change to the GWT code:

2010-02-27

I've just found how easy it is to create Google Gadgets and host them on App Engine. All you need to do is create a simple page in your App Engine application to be the content of the gadget and then host a URL content type gadget specification that points to that page.

In my case I'm using a very simple GWT application. Make sure that it looks good at 300 pixels wide. The height can vary but don't make it too tall because people are less likely to install it if it takes up too much space.

The URL content type approach allows you to make a normal web page using any tools you like and you can debug it like any normal page.

To put it all together simply save your gadget spec into the war directory of your GAE application and deploy it so it's available on a URL like this: http://yourappid.appspot.com/gadgetspec.xml

Now you can test it in the iGoogle Sandbox which (after you've signed up) allows you to test unpublished gadgets. Add this developer gadget to make life easy.