April 2008

Apr 30, 2008

I first wrote about the H2 Database last August. I've been using it since and so far I'm very happy with it. The article I wrote is still getting hits from people searching for information on H2 (here's a fun fact, if you search Google for H2, you'll get the H2 Database before the H2 Hummer) . I'm now working on improving the performance of my application. Like everything else in H2, it is very simple to trace the database performance. This article demonstrates several techniques.

Note: the tools shown here are relevant to H2 starting from version 1.0.70. Since H2 is still work in progress, this may change in the future. I'll do my best to keep this article up to date. I also included some useful links to the original H2 Tutorial.

Generating Trace Files

H2 can output trace files which include the performance information. These trace files contain all the commands executed and their execution details. In the examples hereunder, I will use a database called mydatabase.

First, you need to specify the trace level. You do this by setting a property called TRACE_LEVEL_FILE. The best way is to simply specify this upon connection to the database. This is done by adding ";TRACE_LEVEL_FILE=n", where n is the trace level, at the end of your database connection string. The trace levels are: 0=off, 1=error, 2=info, 3=debug. Use trace level 2 or 3 in order to get the performance information. For example:

jdbc:h2:file:~/mydatabase;TRACE_LEVEL_FILE=3

From that point, H2 will start writing trace logs to the directory where the database file is located. The trace file is called mydatabase.trace.db. Once the file reaches the maximum size, it is rotated to mydatabase.trace.db.old (the old one is erased). You can set the size of the log file by setting TRACE_MAX_FILE_SIZE. My connection URL, looks like this:

jdbc:h2:file:~/mydatabase;TRACE_LEVEL_FILE=3;TRACE_MAX_FILE_SIZE=20

H2 is very tolerant regarding trace files. You can rename the file and it will create a new one instead. You can set the trace level at any given time, not only upon connection. You can use the H2 console for that, just connect to your database instance and execute the command:

The first line shows the date and time. The second line shows the following information:

l:nn - The length of the SQL. Will not appear if there are no parameters (see below).

#:nn - The number of rows returned or affected. Will not appear if no rows were found or affected.

t:nn - The execution time in milliseconds. Will not appear if the time was less than 1 ms.

The SQL command itself with all the parameters, in case you're using a prepared statement. You can simply copy the SQL with the parameters and execute it through the H2 Console, which is very handy for debugging.

The trace file is your first stop when checking your performance. You can check every single command and see how it performs.

SQL Command Statistics

H2 includes several tool, among them a utility that generates statistics from a trace file. The tool is called ConvertTraceFile. The tool takes a trace file as input and has two outputs: a Java class (not relevant) and an SQL script which contains all the SQL commands and the execution statistics.

You should specify full paths for h2.jar and your trace file if necessary. Alternatively, make sure h2.jar is in your CLASSPATH. The SQL script will be in a file called mydatabase.sql. You may also specify a different file name (use -? to see all the available options).

The SQL script file holds all the SQL commands which were executed when the trace file was active. At the end of the file, you will find the command statistics section which has the following form:

--

self

accu

time

count

result

sql

--

89%

89%

3309

12

234

SELECT * FROM EMPLOYEES WHERE EMP_ID = ?;

--

5%

94%

192

1015

1015

UPDATE EMPLOYEES SET NAME=?, DEPT=?, MANAGER=? WHERE EMP_ID=?;

--

2%

97%

87

421

421

UPDATE DEPARTMENTS SET NAME=? WHERE ID=?;

--

0%

97%

26

8123

8123

CALL AUTOCOMMIT();

--

0%

98%

16

18

736

SELECT * FROM DEPARTMENTS WHERE ID = ?;

--

0%

98%

14

424

740

SELECT COUNT(*) FROM DEPARTMENTS;

The columns are:

Self - The percentage of the given command's time from the total time.

Accu - The percentage of the accumulated time from the total time. The accumulated time is the time of the commands listed so far.

Time - The total time of the command executions in milliseconds.

Count - the number of times the command was executed.

Result - the total number of fetched or affected rows.

SQL - the command itself

Needless to say, the statistics you get can easily be imported to OpenOffice Calc or Micsoroft Excel for further manipulation (you can use a fixed column import).

Conclusion

I like simplicity and this is one of the main reasons I enjoy using H2. The above tools enable you to get statistics and detailed information on SQL queries and commands executed in H2.

Going back to Windows is annoying as expected. The worst annoyance comes from the difference in the location of the Mac "Command" and the Windows "Ctrl", especially when it comes to copy/paste: in OS X it's Command-X and in Windows it's Ctrl-X, but the Command is placed where the Alt key is in Windows. I also miss Quicksilver and Exposé. I know, there are alternatives, but I just couldn't be bothered to install any.

On the upside, my environment is surprisingly the same. Here's some examples:

Eclipse - exactly the same. Right down to the plug-ins. The keyboard shortcuts are a bit different which takes some time to get used to.

H2 database - completely cross platform. I didn't even bother to install it, just downloaded the generic package and unzipped it. Works beautifully and exactly the same (maybe faster?).

Firefox - yes, I have all my add-ons running. It's feels at home.

Open Office - I had to work on some documents. On the Mac I use NeoOffice. My very complex spreadsheet didn't even notice the platform change.

I use Cygwin which gives a Unix prompt, much like the one I get in OS X. Cool.

After some minor tweaking the application I'm developing was up and running. It was working beautifully. I was working fluently and almost forgot I switched the platform. I made some progress this week, but...

All my tasks are managed in OmniFocus - I couldn't find anything that can open my backup files on PC.

All my brainstorming and designs are in OmniOutliner format - again, I couldn't read it. Not even close.

I had some notes in xPad. I managed to partially recover the ones I needed.

Some after thoughts after this "fun" week:

The Mozy backup service allows downloading backups through a web clients. However, Mac backups are downloaded as DMG files. It took me time to open these on Windows. I finally used IsoBuster, which isn't free. I'm just wondering what's wrong with plain-old ZIP files.

I have backups of my sources and documents. I missed out on some important stuff - like Eclipse workspace settings (including launch configurations and what not), my test projects, etc. Luckily, I'm going to get my MacBook Pro early next week with the hard-disk intact.

It took me hours to install the Eclipse & Firefox with all the plug-ins and settings. This gave me a great idea - write a plug-in which synchronizes the settings and plug-ins. You could then download a distribution of Eclipse/Firefox with this plug-in and the plug-in will install your other plug-ins and do all the configuration for you. Sounds like a million dollar idea to me.

Finally, this is the price of going solo and not having an IT department behind you. I knew that when I "signed up". This helped reduce the anger and pain involved. I knew this is going to happen some day and I just said to myself: "well, today is the day".

Apr 21, 2008

Last week, my MacBook Pro died on me, as I wrote in my previous post. I started working on my PC, which is a very old and slow machine (a Pentium 4). After a couple of hours I realized that it won't work without upgrading my machine's memory. However, I couldn't upgrade the memory since my machine would not accept 1 GB memory modules. They told me it has something to do with the BIOS, which needs an update. So, I got the BIOS updating software, updated my BIOS and, as you might expect, ended up bricking my PC. Later that day I got a word from the lab that my MacBook Pro has toasted its' logic board and they're sending in a new one (which will take time).

Two machines. Two totaled logic boards. Two days. I'm off the grid.

I have no problem being disconnected from the net. I usually do that when I take a vacation: no internet, no e-mail, minimal phone calls. However, sitting at home with no computer was a strange feeling. Luckily, I could browse my Gmail from my 3G cell phone, although this wasn't very satisfying. Now, I got a spare machine from my brother and I'll do my best not to trash it. I should be getting my PC back today with a revamped processor.

Apr 17, 2008

In an unfortunate turn of events, my MacBook Pro has decided to take a vacation. It's spring time, so there must be something in the air. Maybe it's the upcoming Passover, which is known to be "the festival of freedom". Anyway, it just died on me. The Apple service in Israel has room for improvement: they promise to fix it in two weeks. Two weeks. What am I going to do in the mean while?

I wasn't really shocked when it happened. My personal experience tells me that laptops, no matter how good they are, will need service every year or so. It's probably because all the miniature and delicate components cramped in this overheating space, which constantly gets rattled and banged. My previous laptop was an IBM ThinkPad T41. It was a good and sturdy machine, but I did visit the IT department every now and then. The advantage of working in a big company is that I got a replacement laptop (with my hard drive) whenever such a mishap occurred, so I could keep on working. Now, I'm decommissioned.

I'm making the best of this time to finally test my product on Windows. It took me a few hours to restore my environment. I backed up my source repository, but not much else, which means I have to restore my workspace. I had to download all the relevant products (Eclipse, Spring, H2, ...) and install them. I also had to work around issues of platform compatibility, mainly resource locations. However, once all this was done, I was up and running.

I appreciate the platform compatibility, but It's not perfect. The Eclipse UI is based on SWT. SWT is much better looking than Swing since it is wrapping native UI controls. Which means it may behave a bit different under each platform. It is cross-platform, but it is far from being bug free and requires the developer to test and debug the product on all platforms. My product is running, but not without glitches. Well, I have the upcoming two weeks to get rid of those. And maybe I'll get a little freedom myself.

Apr 14, 2008

We've seen some hype lately around the next Eclipse release dubbed E4.
The theme of E4 would be to take Eclipse online. This is not the first
attempt (I wrote about it 6 months ago), however, now it turned from a
sporadic effort to a full blown platform migration.

The Eclipse Duality

Before looking deeper into this trend, there's a much needed
distinction to be made. Eclipse, as a brand, stands for two different
entities: the Eclipse IDE (Integrated Development Environment) and the
Eclipse
RCP (Rich Client Platform). It's easy to get confused, but
it's
not the same thing. It's almost like the difference between Microsoft
VisualStudio and the .Net platform.

The Eclipse IDE is a great tool for software development. You can use
it to develop in Java, C++, Ruby and many other languages. On the other
hand, the Eclipse Platform is a framework for developing Java based
cross-platform desktop application. The Eclipse IDE uses the Eclipse
Platform as its' base. The Eclipse IDE is the best tool for developing
applications for the Eclipse platform, however, one may use any IDE
which supports Java development for that purpose.

In this post, I will focus on the Eclipse IDE, looking forward to E4.

Who
Needs an Online IDE?

The trend of taking desktop applications online and providing them as a
service (SaaS)
is not new. It was only a matter of time until it gets
to the IDEs. It's taking longer for the IDE market to be affected,
probably because it is a much smaller market than, let's say, word
processors. The online word processors market is already saturated with
products like Google
Docs and Zoho
Writer.

Most online IDEs are very specific (unlike generic IDEs like Eclipse),
geared towards a given service or platform. The IDE is not the service
being provided, but rather a tool for the user to consume other
services. For example, the service can be an online database and the
online IDE would allow developing solutions on top of this service:
database definition, screen design, coding and even debugging. The
deployment is seamless, using the provided service.

Examples for such services are Coghead,
Zoho Creator,
Bungee Builder,
Microsoft PopFly
and Yahoo Pipes.
These are all proprietary services.
Some use their own languages and all of these services are restricting
the deployment to their servers. This is usually their underlying
business model: providing the service for executing the complete
application.

There are a few services which are more generic by nature, based on
standard
languages. For example, Heroku
is using Ruby on Rails (RoR) and can be used develop and deploy a full
RoR application. A
unique feature in Heroku is the ability to export your code. Since the
code is generic RoR, you can then deploy it on any server that supports
RoR. Heroku is based on Amazon
EC2 platform. I won't be surprised to
see Python-oriented services based on Google's AppEngine
soon. Another
interesting example in this area is AppJet
which uses JavaScript.

This online IDE roundup cannot be complete without mentioning CodeIDE
and ECCO.
Both provide completely generic online IDEs. They seem like
perfect tools for people learning how to program, allowing the user to
instantly start
coding and see the results. Although these are not complete IDEs, they
provide a glimpse of what a generic online IDE might look like.

All the services above have mainly two thing in common: they do not
require any installation and moving from coding to execution
is just a
few clicks away. Usually, there are considerable overheads
in IDE
installation, environment configuration and server deployment and
execution. This can all be avoided, saving a great deal of time (and
agony, speaking from personal experience).

Nevertheless, none of these services is even close to the rich feature
set you
get from a true desktop IDE like Eclipse or VisualStudio. This gap is
not unique to IDEs, though. People switching from Microsoft Word (or
Apple Pages) to Google Docs will feel the difference. Sure, Google Docs
has all the basic features. However, it is still very far from a
full-fledged desktop word processor.

The move to server-agnostic applications is still in its infancy but
will soon have a major effect on enterprise computing. The legacy
applications won't go away, even if the exciting stuff is being done on
Internet-based apps, they said. But it won't stay that way. Today, 70
percent to 80 percent of corporate applications require Windows to run,
but the Gartner analysts expect a tipping point in 2011, when the
majority of these applications will be OS-agnostic, such as Web
applications.

The Eclipse IDE is just joining the crowd. We're going to see an
increasingly large number of
online applications replacing their desktop counterparts. IDEs that
won't offer an "online option" will be falling behind.

The
Future

As I mentioned, there are some immediate benefits from having an online
IDE, especially: no installation and instant deployment. IMHO, this is
just the tip of the iceberg. Here are just a few thoughts of what we
might expect:

Online IDEs open new capabilities of sharing and
collaboration. Consider doing pair-programming
with your colleague,
which is sitting in another continent.

This can be even more useful when it comes to outsourcing
and your coworker is an occasional developer. For example, I need the
services of an expert DBA. I can find one online in elance and work
together on
my project immediately.

On-demand services open new possibilities: instead of
buying a profiler for $500, maybe I'll just pay for the time which I'm
using it.

Mash-ups: I can use Google
Page Creator to design my project's web
pages and Yahoo Pipes to define a web service. All as part of the same
project. Today, Eclipse is already a mash-up of OSGi services.
Tomorrow, it may be a mash-up of web services or REST.

All the online services I mentioned in this post had to
developed their own IDEs. I'm sure future vendors will be more than
happy to use an existing online IDE and develop plug-ins on top of it.
As the number of online services provided grow, the need becomes more
imminent.

I'm just guessing here, but it does open up new directions and
opportunities.