Monday, December 24, 2007

I just got this idea, and i don't know if it works at all. I'm just playing with the idea and thought i might just as well write it down here immediately.

Amazon (S3 and EC2)

Amazons launched a couple of services, one of them is EC2 (Elastic Computing Cloud) and one of them is S3 (Simple Storage Service). S3 is a distributed storage solution where you can store an unlimited amount of data online. This storage is actually distributed over lots and lots of (virtual) servers around the globe.Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites.When using S3 you only have to pay about $0.18 per GB storage a month and $0.16 per GB data transfer a month.EC2 stands for 'Elastic computing cloud' and makes it possible to launch an image of a server online. This virtual server will be launched somewhere in the computing cloud, which again is distributed over lots of servers.Just as Amazon Simple Storage Service (Amazon S3) enables storage in the cloud, Amazon EC2 enables "compute" in the cloud. Amazon EC2's simple web service interface allows you to obtain and configure capacity with minimal friction. It provides you with complete control of your computing resources and lets you run on Amazon's proven computing environment. Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change. Amazon EC2 changes the economics of computing by allowing you to pay only for capacity that you actually use.You pay about $0.10 per instance hour. I can imaging it saves a lot of money, since you only launch the amount of servers you need at the moment and you are able to scale it down or up when you like to.

Terracotta

Open Terracotta is Open Source JVM-level clustering software for Java. It delivers clustering as a runtime infrastructure service, which simplifies the task of clustering a Java application immensely, by effectively clustering the JVM underneath the application, instead of clustering the application itself.

Open Terracotta's JVM-level clustering can turn single-node, multi-threaded applications into distributed, multi-node applications, often with no code changes. Terracotta plugs into the Java Memory Model in order to maintain semantics of Java (Java Language Specification), such as pass-by-reference, thread coordination, and garbage collection across the cluster. Open Terracotta's JVM-level clustering is enabled through declarative configuration (XML), and provides fine-grained, field-level replication, which means that objects do not need to implement Java serialization.

With a simple load balancer it doesn't matter to which server the request is send, because all servers are on the grid and share the same memory and thus all user sessions are available on all servers. When adding a new server to the grid it takes automatically the state of the application and can receive requests immediately. This makes it very easy to scale up when needed.

Another idea is to use Amazons S3 Service (distributed online storage) for file storage, because this also is unlimited scalable. All data transfer between the EC2 cloud and S3 is free!

Winning combination?

The idea is to create a standard java backend application (the usual spring/hibernate/etc application). Then we take that application and launch it in a terracotta grid. That makes it scalable over multiple servers sharing the same resources through terracotta. Take it a step further by creating an image and launch it in Amazons EC2 cloud.

Possible challenges (problems do not exists, right?)

All images in the EC2 cloud are stateless, when the server images goes offline or restarts all state is gone. Every image starts from a blank image, so you can't write to disk. But that shouldn't be necessary if we write all files to S3. Another thing is databases, if we have like 5 images in the cloud all running terracotta and thus share the same memory/state. Then we don't have to do persist anything to the database, and just keep it in memory. Terracotta could drastically help to reduce the database load. But we can't keep everything in memory, you don't want to keep historic data that is practically not used anymore in memory. You only want to keep objects in memory that you are likely to query a lot. Where do we store the data that we don't want in memory? I heard about the Amazon Simple Database (SDB), which is going to get launched pretty soon (at the moment of writing this). I didn't look into it yet, but sounds promising.

I'm going to work this out and try this later, i you have any comments or suggestions let me know!

Thursday, November 01, 2007

Kevin Hoyt has written some nice examples in his blog (http://blog.kevinhoyt.org/) about using EXIF info in AIR applications.

I've been writing an application that is using EXIF info to get the jpg thumbnail from it without having to read the entire file.. Now this is working as a charm BUT..

There are multiple version of the EXIF standard and there are different manufacturers using it different. I have tested my application with the code from kevin hoyt, it worked on my Sony digital camera, but didn't work on another Sony camera and also failed on a Canon camera. Where the failing sony didnt succeed to retrieved jpeg bytes at all. I could retrieve bytes from the canon picture but these are not valid jpeg..

I read through some of the EXIF documentations of the official standard but it can't figure it out how to create a parser that always succeeds.

That plus the fact that most users make their landscape pictures portrait using the windows picture viewer, which just dumps all EXIF information from the JPG. Even when using photoshop to do this is deletes your EXIF.

This all makes it pretty useless for any application to depend on, until someone comes up with a superparser compatible with all manufacturers.

So back to reading the entire file just to show a quick thumbnail (which makes it impossible to create like a thumbnailbrowser).

Tuesday, September 04, 2007

I'm working on an AIR application that has to load up a lot of pictures from a users system. I have to be able to load about 120 photos (of 3MB each), open them, resize it and save it in memory and cache a resized photo to disk. So i created a manager which has an internal queueing system which loads the photos one by one and doing the operations before continueing on to the next photo. The most processor time goes into loading the photos from disk into memory, which is a asynchronous job, so the GUI has plenty of time to update and keep track with a simple loading bar.

Now the weird stuff, the calls to load the next photo uses 'Application.application.callLater(loadPhoto)', on a Windows machine this is no problem at all, and it loads 120photos easily (i've tested it with up to 500). Now if you change that sencentence to 'setTimeOut(loadPhoto, 500)', then it suddenly stops working. It loads up to 70photos and then sais 'Invalid bitmapdata'. This is probebly because the garbage collector is too slow and can't free the memory intime, resulting in memory allocation errors. When i'm running the same thing on a MacBook Pro (which is faster then my window laptop, and has twice the RAM), then suddenly both scenarios don't work and i can only load up to 17 photos. RAM Climbing very fast up to 500megs and crashes...

Now thats sucks big time!

Then i came across the web and found a garbage collection hack, i didn't feel like that was going to be the solution but gave it a try anyway and the miracle happend and everything runs like a charm, no problems at all on windows or the mac.

Tuesday, July 31, 2007

Flex has some options for image caching on its own, but it leaves the image caching for the browser to handle. This creates differences between browsers (i have written applications that caches really well in IE but not so well in FF). This browser cache isn't very reliable, plus it takes a really small amount of time to retrieve (because it is http based and it needs a http call to retrieve) what causes the image to flicker. What i really want is some sort of application wide caching mechanism for images.

Another thing is that it works (offcourse) also for AIR applications, so it should work both with online and offline files. The thing that makes these files unique is the location of the file, what could be either 'http://...' or 'file://...'

The solution is really simple. Create a class that can store images in a map (like a java hash-map). This class should be a singleton and have a couple of methods like:- lookup(location:String)- cacheImage(location:String, image:Bitmap)

And everywhere in your application when you use an Image object just set the source to the bitmap from the repository (instead of a url). How to load the bitmap from an online or offline file is a whole different story (and much more complicated). I will discuss that in my next blog entry.

Sunday, July 08, 2007

StringTemplate is a template engine written by Terence Parr, the author of the popular ANTLR parser generator. It is based upon the idea of Enforcing a strict Model-View separation in Template Engines. Its features allows you to restructure you templates to be more maintenance friendly: so therefore I chose the title "Refactoring your Templates".

What is a template?

Templates are textual files with "holes" in it that that can be filled with data. Besides these holes templates also need facilities for: looping through items, conditional formatting, template fragments and string formatting.

How does JSP handle this?

JSP uses scriptlets inside the template: small fragments of code. These code fragments give full access to the Java language. The danger of using java code for control flow is that templates are not fully separated from underlying data computations. For instance: programmers may choose to quickly embed a JDBC query inside a template. They also lack facilities for reuse: object-oriented features like inheritance and components are not available.

How does StringTemplate do better?

Templates contain no code fragments.

Templates can be named and may accept attributes.

Templates can be grouped in a single file.

Template groups can use inheritance to reduce template duplication.

Templates do not embed code fragments, that allows you to use the same template on different platforms. Naming templates allows you to call templates inside other templates and those calls can receive arguments. Grouping templates into a single file is convenient when you break up templates into smaller parts. Inheritance of templates allows you to override named templates in a group. This feature is nice if you need similar but slightly different results. For example when your program generates SQL and you want to support different SQL database dialects.

How do I use StringTemplate?

This article is a quick introduction to StringTemplate, for more information please see the documentation. The StringTemplate essentials in six bullets:

Holes inside the template are marked using dollars, for instance: $username$.

Every substitution argument must be calculated beforehand and assigned, using the setAttribute("username", "Foo Bar").

Examples and documentation are in the excellent StringTemplate documentation. More advanced features of StringTemplate are:

Template group files

Template inheritance

Group files use the *.stg extension and embed a number of named templates. When you have lots of small templates, this might be more convenient.

Templates are ordered in named groups. It is possible to use an existing template group as a basis and override/extend some templates.

Conclusion

StringTempate helps ANTLR to add a new language backend without adjusting a single line of code. This is a showcase of the power of StringTemplate and how it helps to define complex templates in a clean way.

StringTemplate available for different platforms: Java, C# and Python. It is an interesting project if you need clean templates, certainly worth a look!

Friday, July 06, 2007

This blog entry is quick read summary about code complexity. It gives a definition of code complexity and several signs how to notice code complexity. It also provides a brief description how to measure code complexity and suggests a way to keep it low.Most of the concepts are taken from Structure 101 help section.

This blog entry is based upon my code complexity presentation for www.greetz.com.

Andy Hunt extends the analogy concisely:"Just like real debt, it doesn't take much to get to the point where you can never pay it back, where you have so many problems you can never go back and address them."

What can we do? Measure Complexity!

Although we cannot specify the amount of effort needed. We can calculate a complexity index.

To calculate cyclomatic complexity build a graph that describes the control flow of the program. The nodes of the graph correspond to the commands of a program. A directed edge connects two nodes if the second command might be executed immediately after the first command.

The number of linearly independent paths through the first method is 3, and the second is 2.

RefactoringDefenition of Refactoring:improving the design of existing code.

Always Refactor when adding new features to you code.

Quote Chris Chedgey:"You use personality, charisma, leadership and/or donuts to convince your team that henceforth, they will not add any more complexity debt to the code base. Now watch what happens..."

Wednesday, July 04, 2007

I'm creating my first AIR application, or better i'm porting my existing webapplication to an AIR project. The application is all about online imaging and the one thing it is missing is local file access; a perfect setup for an AIR application.

First i wanted to just make my Flex project AIR compatible.. but weird enough that is not possible. I added a Apollo buildcommand and apollo nature to my project file, but flexbuilder can't handle both a flex nature and a apollo nature, so he just picks one and youre project isn't flex compatible anymore (you get an error when lauching through the Run... buttons).

So i went for the next approuch, too seperate projects with the same codebase. There are several options to achieve this.1. Create libraries and modules for common code2. Create a symlink to the source dir

The first one looks the best option but needs a lot of refactoring at the moment in the project, thats what we are going to do when there is more time (this is a proof of concept). The latter is the easyest solution. I Created an AIR project and created a symlink by clicking: New -> Folder -> Advanced -> 'Link folder to the file system' and linked it to the source dir.

Now i still continue to work on my Flex Project and once in a while i can create an AIR Installer file by simply clicking the button in the other project.

Next i created a 'Seamless Installer'. This piece of code simply checks wether the user has the AIR runtime and let the user install your application including the runtime or just the application.

You create this 'badge' by copying the badge.swf from 'sdks/moxie/samples/badge/badge.swf'and create a html file like:

Tuesday, June 12, 2007

Almost every project nowdays needs an automatic build server for integration tests, while this is common practice in java oriented companies it is fairly new to Flex developers.First of all you need some kind of versioning system for your code (you really really really can't go without a versioning system for your code!).. I am using subversion (http://subversion.tigris.org/) for my projects.At one point conflict resolution is not enough when you have multiple developers working on a piece of software with a versioning tool, you have to know day by day if the commits from you and youre co-workers are not conflicting with one another on a functional level (i'm not talking about compile time issues here). So you need to compile the code on the server and run some tests to automatically check if everything still works fine.Herefore i use the cruisecontrol (http://cruisecontrol.sourceforge.net/) build server together with Ant build scripts and FlexUnit tests.There are tons of tutorials and howto's about how you install cruisecontrol, subversion and flexUnit.

There is a project in adobe labs called Flex Ant Tasks, wich enables you to compile your flex/actionscript code from ant. It is pretty straightforward if you are familiar with Ant, the only thing i had some problems with is the wrapper creation. The 'standard' way for this is to use the predefined html-wrapper task, that works fine but takes a template from the FLEX-SDK install dir. I have custom html-wrapper in my project and i don't want to copy them on every development machine to the FLEX-SDK. I can't find a way to configure the ant script so that it points to some other directoy in my project. So i had to write that part myself..

The following code is a complete ant script and build.properties file which you can use as a template/reference for you own build scripts. Just place the build.xml and build.properties in the root of your project and alter the variables the way you like. The script is pretty ok documented so i won't go in detail on this. Don't forget that you have to install ANT to run the builds ;-P

I also included asDocs in my ant scripts, which generates javaDoc style documentation for your actionScript project. I don't have it fully running myself yet because the asDoc compiler is much stricter then the flex builder sdk compiler, so i get a lot of errors on unused imports and stuff and i just don't have the time to fix it (for now).

Tuesday, May 15, 2007

I get a lot of questions asking how I deal with configuring environment specific settings. I'll explain my solution here.

The following solution depends on the Spring framework.

1) have a separate package per environment (e.g. production, test, local)2) put a properties file in this directory (e.g. webapp.properties)3) configure the spring context4) set a system environment variable to choose your environment

Take a close look at the propertyPlaceHolderConfigurer; there is a default configured, and one with an EL expression. If ${env.dir} is not specified, it will not find the properties file, and ignore this because of the ignoreResourceNotFound setting.

The properties in the junit.properties file, are overwritten by any settings in the next properties file.

The only thing left to do is specify the environment setting 'env.dir'. This can be done in several ways; in the servlet container configuration, or on the system environment, or as a parameter to the jvm (-Denv.dir=test).

Wednesday, April 18, 2007

Appearantly the flex compiler includes a context in the swf. When the swf is downloaded and tries a RemoteObject or other request to the server, it uses the precompiled context of your webapp. So if you are developing at "http://localhost/appcodename", then after a couple of weeks hard work you create a war file like "financeApp.war" and you deploy it to like "http://domain.com/financeApp/". Then suddenly your remote requests give a weird "Send Failed" error... I found out that the url it is requesting is wrong (after hours of debugging) and ended up recompiling the SWF files with the correct context (i set the context from the flex builder -> project properties -> Flex Server. So i changed 'appcodename' to 'financeApp'

And this message pops up randomly when accessing a flex generated html page with does nothing more then embedding the swf file.

I'm not sure what causes this error, only some flexplugins have this problem and only occasionnally. It has something to do with the history management of your browser. The solution to this problem is to remove the IFrame where flex does its history management (history.htm). The error message dissapears when removing the IFrame. I am not sure why or how, but i recommend not to compile your apps with history management if you don't use it.

Friday, March 16, 2007

Great, i saw there was a new release of Flex builder on the Adobe site the other day.. I am currently using Flex Builder 2, and there is a new release called 2.0.1. So i thought, ok thats a small bugfix or something, nothing major (hence the 0.1 version). So i downloaded the update from the adobe site and i ended up with downloading a 144MB update. Hmz that can't be right for a small bugfix/code change right?

Next i installed the update, all fine, but when i started Eclipse on the project i'm currently working on i ended up with this error and a bunch of others: Definition mx.binding.RepeaterComponentWatcher could not be found.

I searched on google and found a bunch of people with the same problems, making up all kinds of tricks and workarounds to get it working. The problem is the .swc files in the update are different and _NOT COMPATIBLE_ with the .swc files from the 2.0 release. No way i can make it work with the serverside code running on 2.0 with FDS and the other programmers who use 2.0 because it is incompatible.

So i am wondering why the hell adobe releases an update with API changes and incompatible, non-backwards compatible code. On top of all they all it a x.0.1 release, what indicates it is a minor release. They problebly should call it a 3.0 release but that would be bad for marketing, so they just release this crap, then don't release it at all! Or at least inform people it doenst work with anything but 2.0.1.

Anyways my message to Adobe: this sucks, thnx for wasting my time.

I uninstalled everything and reinstalled Flex Builder 2 and everything is working fine again.written by Marcel Panse

Wednesday, January 17, 2007

One of the problem with FlexBuilder, which is integrated in the eclipse IDE, is that you will need a Flex Project in your workspace to make use of all the cool FlexBuilder tools.You can't just integrate flex in your existing java-app, because the FlexBuilder won't recognize the project as a Flex project.I've looked into the differences of these projects and it really isn't that hard to create a flex project with java support.

Just create a Flex project and edit the .project file it generates with it. Add another build command like: