The knowledge gap

I’ve been a professional programmer now for over a decade, and during this time I’ve met a LOT of excellent programmers. Most of them are partially autodidactic and learned the business side of programming in school. But non of these guru’s/experts use their skills to teach the next generations. Sure, they attend programming conferences, give talks, join user groups etc. But all the other attendees are already good programmers! Companies usually send their brightest programmers to these conferences, not the beginning employees. This means there is already a gap in knowledge and it is increasing. The smart are getting smarter, and nobody is really helping the beginners…

Teachers salary

During my time in college I wasn’t very happy with the programming teachers. They had some experience with the business side, they teached us about UML, gantt charts, and (then still hot) waterfall project planning. When it came to real programming, problem solving, algorithms, best practices… they didn’t really teach us much.
Most of the time we had more programming experience and skill than the teachers.

Although I haven’t been in college for a couple of years, except two guest appearances at the University of Amsterdam, I don’t think much has changed. The whole problem is the status and rewards of being a teacher. The average salary of a college teacher (here in the Netherlands) is about half, maybe two thirds of the average professional programmers salary. No wonder why most average or better programmers don’t become teachers! Only the programmers with least experience and skill, or teachers which later on learned programming end up becoming programming teachers.

Mentoring/apprenticeship

In the ancient Greece and Roman times teachers were important professions. Especially the most experienced craftsmen, they did not only master their craft, but earned most money by passing their skills as master/mentor to their protégés. This doesn’t happen anymore, the big names in programming now don’t have apprentices to pass on their knowledge. In the best case they write books, which will take years before the school teachers have read them and adopt the ideas.

Devoxx4Kids?

I’ve already mentioned Devoxx4Kids in previous blogposts. Although this is a great initiative (which I fully support, we’re holding our own version in the Netherlands!). It helps getting kids excited in IT/programming, but it won’t help getting the level of education to a higher level.

Highest status

Being a teacher/mentor should have the highest social status in our modern world, it should be the highest paid profession and all the guru’s/experts should want to become teachers. But it isn’t. It is poorly paid and absolutely not the sought after job. How can we change this?

At work we’ve had a lot of discussion about epics lately. We seem to have forgotten what they are and how we learned to deal with epics. It became a ‘theme’ or just a big pile of stories which sounded like they belong together. But epics are much much more.

An epic is a goal, and just one goal only. All the stories in the epic should contribute to that epic! Also the epics goal should be a business goal, never a technical goal. The epic should describe a SMART described measurable goal without hinting towards a (technical) solution.

Epic manifesto

In our project we tend to forget what epics are for and why they are so important after some time.
To remind everybody we decided to write down a manifesto for the epic:

An epic has just one goal, not more and not less.

An epic is always measurable, thou should always ask “why” until the business value is clear.

An epic has a fixed goal, new insights will lead to new epics.

An epic should have a solution conceived with and verified by the stakeholders, users, product owner and the team.

An epic contains only stories that help to achieve the goal.

An epic is ‘done’ when the goal is achieved.

An epic is only estimated in relative complexity, never in units of time.

An epic is always in the correct order on the backlog (by the product owner).

An epic is realized by the team with the minimal viable solution.

Does your epic break one of the rules above? It is an epic failure!

This is going on several walls in our building just to remind us how important correct epics are. If the initial goal is wrong, all the stories are wrong as well. The epic is the base of all the solutions you’ll be creating and must be founded properly.

Any thoughts on this subject, don’t hesitate to reply. Agile projects get better by sharing best practices and trying to improve!

A couple of years ago, when our family was planning a trip to South Africa, I bought a nice DSLR. We had friends with excellent cameras, huge lenses and wild life photography experience. So instead of trying to compete with them I decided to buy a camera suited for filming. It was the rise of filming-DSLRs, and even television shows were adapting DSLRs for filming. I settled on a Canon 550D also known as Rebel T2i.

Just before I left for South Africa I decided to take the camera to the Devoxx 2011 conference. I didn’t even have a tripod at that time! After shooting some quick footage I decided to edit everything to make a nice movie and share it with the organiser. He liked it so much he placed it on the main website for a while. This is what I made:

Devoxx 2012

The following year Stephan asked me to be the official videographer for Devoxx 2012! So I went back and made another video, it seems like I’m turning from regular visitor/developer to videographer. I took some more time to analyse the conference to include the main sponsors, show every aspect of the conference and to get across the Devoxx atmosphere (including the James Bond theme, we all watched SkyFall during the conference!):

Devoxx4Kids 2012

The best time I had as videographer was during Devoxx4Kids. It is a separate event first hosted in 2012. The kids were having a great time and that is clearly visible in the video. Kids aren’t shy at all and much easier to approach then adults. That is why I did some quick interviews and I love the result, I might do that in the future for other conferences as well:

Devoxx UK 2013

The latest conference I filmed was Devoxx UK. For the first time the Devoxx conference has crossed the big blue sea to go to the United Kingdom. I’ve tried to make the resulting video look a bit more professional although I’m still just a Java developer with a hobby:

The future…

The last three years I’ve filmed at three weddings, four conferences and edited together our vacations to South Africa, Norway and some trips into the Alps (wintersport). As an amateur videographer (with a growing list of references) there still is a lot to learn, but I’m really enjoying the process! Any tips/hints/tricks are absolutely welcome, and I’m open for invitations!

Fitnesse

Fitnesse is a very popular test framework, used in a lot of projects. Using a wiki you describe the tests as plain text, which get translated into ‘fixtures’, code that is called that, in turn, can call your application. An example of a fixture we use:

This is a very simplified example on how you could use Fitnesse to test your application. But the idea is very powerful, there are implementations for Java (initially) and now .NET, Groovy etc. Also, most of the time the system you’re testing doesn’t have a Java API but instead you’d like to test the user interface. With most Java applications being web applications there are numerous plugins to connect Fitnesse with for example Selenium to ease web application testing. The fixtures fill webpages and analyze the resulting HTML response.

At my current client (Port of Rotterdam) we’re building a very rich, partially offline, Swing application. This application has a backend and numerous clients running the Swing application. We’ve used Fitnesse to test the backend code from the start, and the last year we’ve put in some effort to also use Fitnesse for the Swing application. We’ve called this “Nonkey”, being an automated version of our Monkey integration testing we’d been doing up to that moment.

UISpec4J

The first ingredient we need to test Swing GUI’s in Fitnesse is UISpec4J. This is an excellent Swing test framework that allows you to manipulate GUI elements from code. The framework is made with unit-tests in mind, but can also be used in integration testing. In our SetUp page in Fitnesse we call the following code:

privateWindowmainWindow;privateJFramemainFrame;//Called from the Suite Set Up page:
publicvoidlaunchNonkey(){mainWindow=WindowInterceptor.run(newTrigger(){@Overridepublicvoidrun()throwsException{newOurApplication().launch();// Get the main JFrame from Spring, used directly in some cases instead of UISpec4J
mainFrame=BeanFactory.getBean("mainFrame",MainFrame.class);}});}

The WindowInterceptor captures the Swing application and allows for manipulation in a later stage. For example:

This fixture causes the Swing application to fill in the username, password and press the login button. It is easy to read and can be launched from Fitnesse with a single line:

|login with username|roy |and password |test123 |

Nice! We are done, we can test Swing GUI’s in Fitnesse now! Well no… not really, unfortunately there are a couple more problem we ran into, regaring the Event Dispatch Thread (EDT).

Event Dispatch Thread (EDT)

The EDT, sometimes called Swing Thread, is a special thread. Everything you do in Swing should always happen on this thread. Setting components to visible, pressing buttons etc, all need to be done in the EDT. If you don’t do this correctly the application gets into a lot of race conditions and it can even cause the application to hang.

When we combined Fitnesse with UISpec4J and started making a lot of tests we noticed that sometimes mysteriously a test would fail without changes. It turned out to be a problem with the EDT. UISpec4J doesn’t worry too much about the EDT, and when we click a button or setText (like the examples above) it does this in the current thread. In Fitnesse this thread is the thread that called the fixture, obviously not the EDT.

We brainstormed a lot about how we could fix this, maybe change UISpec4Js code? This turned out to be a hellish task. How about setting SwingUtilities.invokeAndWait() in every fixture? That could work but is also a massive undertaking. We decided to take a look at how Fitnesse calls the fixtures. This is always done from a ‘StatementExecutor’. So we decided to build a custom StatementExecutor which causes the fixtures to all run in the EDT.

To do this we need a custom SlimService. First I tried to extend the normal SlimService, but this wouldn’t work for some odd reason (still unexplained, should try this again). We decided to copy the SlimService code and make our own:

Make sure that the class can be found (for example, package it in a JAR and place it on Fitnesses classpath). And you’re done! Using a ThreadCheckingRepaintManager (workings described here) we checked to see if the tests are now repainting/manipulating Swing from the correct thread, and it works like a charm.

Good luck testing with Fitnesse in the near future, if you have any problems/improvements/ideas please let me know (in the comments below).

In the next couple of years dead will become an important aspect of the social networks. A time will come that most profiles on Facebook will belong to people who’ll never be able to post another update. Currently due to inactivity most dead profiles will be deleted or become inactive, but I think dead will take a much more important role on the internet.

My parents and even grandparents have a Facebook account now, but in twenty or thirty years, some people I’m connected to won’t be alive anymore. It would be great to be able to browse the lives of people who are no longer around us. I’d love to read about my deceased grandmother for example, just to see how she lived her life. So instead of deleting inactive accounts, death should become more accepted and a part of our social web as well.

A couple of weeks ago I went to Devoxx4Kids. Devoxx is a very well known Java conference (the best in the world actually!). This time they organized an event especially for kids ranging from 10 to 14 years old. The goal is twofold, the main goal is to just give them a good time, the second goal is to introduce them to programming and technology in a fun way.

To show the world this is a great idea, Stephan invited me to film the event and capture the happy faces of the kids enjoying themselfs. And now after Devoxx 2012, I’m allowed to show it to everybody:

Stacktraces in Java are wrong. This is a conclusion that I’ve drawn after years of working with novice programmers.

What is the first thing novice programmers do when they encounter a stacktrace? They start reading it, and this is wrong! The first thing you should do is scroll down. That is because the most important piece of the stacktrace is most likely at the bottom.

Caused by…

Stacktraces in Java are usually wrapped numerous times, because all the frameworks and the application server wants to add their own little bit of information around the original exception. The problem is that Java uses the “Caused by” way of printing the stacktrace. With the “Caused by” you’ll get a lot of information about what happened (afterwards) first, until you get to the actual reason.

For example:

com.portofrotterdam.hamis.service.offline.WebServiceException: Error during webservice call.
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doWebserviceCall(BaseSynchronizingWebservice.java:111)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doCallWithCallback(BaseSynchronizingWebservice.java:97)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doActualCall(BaseSynchronizingWebservice.java:77)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.access$000(BaseSynchronizingWebservice.java:44)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice$1.run(BaseSynchronizingWebservice.java:63)
at java.lang.Thread.run(Unknown Source)
Caused by: com.portofrotterdam.hamis.service.offline.WebServiceException: Problem while getting data from webservice and translation:
at com.portofrotterdam.hamis.service.webservice.WebServiceProxy.callWebService(WebServiceProxy.java:60)
at com.portofrotterdam.hamis.service.webservice.WebServiceProxy.callWebService(WebServiceProxy.java:42)
at com.portofrotterdam.hamis.service.webservice.GetAllTemplateInspectionCheckListsSynchronizingWebservice.callWebservice(GetAllTemplateInspectionCheckListsSynchronizingWebservice.java:26)
at com.portofrotterdam.hamis.service.webservice.GetAllTemplateInspectionCheckListsSynchronizingWebservice.callWebservice(GetAllTemplateInspectionCheckListsSynchronizingWebservice.java:8)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doWebserviceCall(BaseSynchronizingWebservice.java:107)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doCallWithCallback(BaseSynchronizingWebservice.java:97)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doActualCall(BaseSynchronizingWebservice.java:77)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.access$000(BaseSynchronizingWebservice.java:44)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice$1.run(BaseSynchronizingWebservice.java:63)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException
at com.sun.deploy.security.CPCallbackHandler.isAuthenticated(Unknown Source)
at com.sun.deploy.security.CPCallbackHandler.access$1300(Unknown Source)
at com.sun.deploy.security.CPCallbackHandler$ChildElement.checkResource(Unknown Source)
at com.sun.deploy.security.DeployURLClassPath$JarLoader.checkResource(Unknown Source)
at com.sun.deploy.security.DeployURLClassPath$JarLoader.getResource(Unknown Source)
at com.sun.deploy.security.DeployURLClassPath.getResource(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at com.sun.jnlp.JNLPClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:351)
at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:345)
at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:337)
at com.portofrotterdam.hamis.service.webservice.WebServiceProxy.callWebService(WebServiceProxy.java:48)
at com.portofrotterdam.hamis.service.webservice.WebServiceProxy.callWebService(WebServiceProxy.java:42)
at com.portofrotterdam.hamis.service.webservice.GetAllTemplateInspectionCheckListsSynchronizingWebservice.callWebservice(GetAllTemplateInspectionCheckListsSynchronizingWebservice.java:26)
at com.portofrotterdam.hamis.service.webservice.GetAllTemplateInspectionCheckListsSynchronizingWebservice.callWebservice(GetAllTemplateInspectionCheckListsSynchronizingWebservice.java:8)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doWebserviceCall(BaseSynchronizingWebservice.java:107)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doCallWithCallback(BaseSynchronizingWebservice.java:97)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doActualCall(BaseSynchronizingWebservice.java:77)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.access$000(BaseSynchronizingWebservice.java:44)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice$1.run(BaseSynchronizingWebservice.java:63)
at java.lang.Thread.run(Unknown Source)

The first two traces aren’t at all important, it tells us something went wrong with a webservice call. But the bottom trace gives us the real reason, a NullPointerException!

Caused!

I’d rather see stacktraces being inverted, so instead of saying:

**Webservice failed

which is caused by -> Webservice failed during translation

which is caused by -> A NullPointerException!**

It should say:

**A NullPointerException

which caused -> Webservice failed during translation

which caused -> Webservice failed…**

So using the example above it would become:

java.lang.NullPointerException
at com.sun.deploy.security.CPCallbackHandler.isAuthenticated(Unknown Source)
at com.sun.deploy.security.CPCallbackHandler.access$1300(Unknown Source)
at com.sun.deploy.security.CPCallbackHandler$ChildElement.checkResource(Unknown Source)
at com.sun.deploy.security.DeployURLClassPath$JarLoader.checkResource(Unknown Source)
at com.sun.deploy.security.DeployURLClassPath$JarLoader.getResource(Unknown Source)
at com.sun.deploy.security.DeployURLClassPath.getResource(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at com.sun.jnlp.JNLPClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:351)
at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:345)
at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:337)
at com.portofrotterdam.hamis.service.webservice.WebServiceProxy.callWebService(WebServiceProxy.java:48)
at com.portofrotterdam.hamis.service.webservice.WebServiceProxy.callWebService(WebServiceProxy.java:42)
at com.portofrotterdam.hamis.service.webservice.GetAllTemplateInspectionCheckListsSynchronizingWebservice.callWebservice(GetAllTemplateInspectionCheckListsSynchronizingWebservice.java:26)
at com.portofrotterdam.hamis.service.webservice.GetAllTemplateInspectionCheckListsSynchronizingWebservice.callWebservice(GetAllTemplateInspectionCheckListsSynchronizingWebservice.java:8)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doWebserviceCall(BaseSynchronizingWebservice.java:107)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doCallWithCallback(BaseSynchronizingWebservice.java:97)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doActualCall(BaseSynchronizingWebservice.java:77)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.access$000(BaseSynchronizingWebservice.java:44)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice$1.run(BaseSynchronizingWebservice.java:63)
at java.lang.Thread.run(Unknown Source)
Caused: com.portofrotterdam.hamis.service.offline.WebServiceException: Problem while getting data from webservice and translation:
at com.portofrotterdam.hamis.service.webservice.WebServiceProxy.callWebService(WebServiceProxy.java:60)
at com.portofrotterdam.hamis.service.webservice.WebServiceProxy.callWebService(WebServiceProxy.java:42)
at com.portofrotterdam.hamis.service.webservice.GetAllTemplateInspectionCheckListsSynchronizingWebservice.callWebservice(GetAllTemplateInspectionCheckListsSynchronizingWebservice.java:26)
at com.portofrotterdam.hamis.service.webservice.GetAllTemplateInspectionCheckListsSynchronizingWebservice.callWebservice(GetAllTemplateInspectionCheckListsSynchronizingWebservice.java:8)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doWebserviceCall(BaseSynchronizingWebservice.java:107)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doCallWithCallback(BaseSynchronizingWebservice.java:97)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doActualCall(BaseSynchronizingWebservice.java:77)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.access$000(BaseSynchronizingWebservice.java:44)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice$1.run(BaseSynchronizingWebservice.java:63)
at java.lang.Thread.run(Unknown Source)
Caused: com.portofrotterdam.hamis.service.offline.WebServiceException: Error during webservice call.
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doWebserviceCall(BaseSynchronizingWebservice.java:111)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doCallWithCallback(BaseSynchronizingWebservice.java:97)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.doActualCall(BaseSynchronizingWebservice.java:77)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice.access$000(BaseSynchronizingWebservice.java:44)
at com.portofrotterdam.hamis.service.webservice.BaseSynchronizingWebservice$1.run(BaseSynchronizingWebservice.java:63)
at java.lang.Thread.run(Unknown Source)

This is much easier to understand (IMHO). The first exception you see is now also the first thing that happened. Much clearer for the novice programmer.

Some more tips when you encounter a stacktrace:

Look for the first line that starts with YOUR package, this is where your control over the code starts, where you might be able to fix it

If there are a lot of other packages mentioned before YOUR package, Google it! You are probably not alone

giving.github.com

Giving.github is a project that tries to bring people together. There are a lot of programmers looking for fun projects in the world, and so are scientists/charities and other people with problems!

Be sure to check it out, maybe you can help somebody as well.

Mara-Naboisho Lion Project

One of the problems posted to giving.github has something to do with Lion collars. The Mara-Naboisho Lion Project has a couple of collars and some files, but no idea what the data inside the files mean. Their initial description was:

The Mara-Naboisho Lion Project has asked for our help decoding and plotting the positions of lions fitted with GPS collars.

These are presumably SMS PDUs, although simple online decoders don’t seem to make much sense of them. There’s the ASCII text “IRIDIUM” in them, presumably being the SMSC or sender or something string, as IRIDIUM is a satellite telephone system and presumably the bearer when the collars send up their positions.

What format is it, and how do we decode it?

Not SMS PDU

This is where I started analyzing, and I quickly ruled out that the hex-code is pure PDU (which can easily be translated). It seems to be some dialect/own protocol.

The first major breakthrough was realizing that there is one big header that both files have in common, and the real message starts with 0A. And the two messages after that have a repeating pattern:

This matches the date in the title of the first sms: Collar07854_100806 210058.SMS
It even matches the timestamp minus 2 hrs in the title!
-——————————————————————————–

More information

At this point the progress stopped a bit, we couldn’t get the other bytes translated. There seemed to be certain integer values: 0039 d0ed for example. This probably isn’t floating point (which we thought it would be, latitude/longitude/altitude) because floating point numbers have the exponent in the first 11 bits. When a number starts with 00’s it doesn’t have the exponent part filled.

So we asked the project for some more information, for example about the recorded data (where/when) and the manufacturer of the collars.
They gave us this KML file, and this company name: Vectronic-Aerospace.

Lat/lon, I’ve got you!

The biggest part we wanted to decipher was the lan/lon location (that is what the researchers want to have). I came to this breakthrough after reading a PDF file from Vectronic-Aerospace’s website. It has some screenshots from their original software (why aren’t they using that free software?!). In these screenshots I noticed integer coordinates very close to the integers I’ve managed to get from the hex-data! It was called ECEF data, this is X,Y,Z data! Until now my guess was that the SMS files had lat/lon coded in binary, but it has XYZ information.

After a quick google I managed to find some code to translate from ECEF coordinates to lat/lon and it gave me the following location:
Lat: 52.43067054630929
Lon: 13.525755535406619
Height: 88.99500464554876

When plotted in Google Maps it gave me a location in Germany, at first I thought it was another dead end, but then I looked here. It turns out it was the EXACT location of Vectronic Aerospace, the creators of the collar! Problem solved.

What’s next?

There are still some mysteries, for example… what are the last bytes? Looking at the screenshots in the PDF I suspect it would be main battery voltage, backup voltage and temperature. The project also probably wants to integrate the collar files with the website. And there are of course a lot more giving.github.com problems to solve!

With the sudden surge of visitors from Reddit (Hello!) I decided to take look at Google Analytics. And two things caught my eye:

First of all, Google Analytics ‘Realtime’ is awesome, you can see minute by minute how many people are on your website and what they are looking at, where they come from (internetwise and geographically).

Second, my website gets the right type of visitors, I’m so proud at you guys and girls…

Last week I’ve created a ray marcher 3d engine which renders the Mandelbulb. And I’ve translated it into pure Javascript a couple of days later. After the translation I decided I should optimize the code a little for speed, so I made some speed improvements in the Javascript code. The main optimization was using an array for the vector3d instead of a class/function.

Rendering the Mandelbulb on a 400x400 canvas now took just 1850ms in Javascript (Chrome, V8). Which is very fast! Even faster than my Java implementation (running on Java 1.6.0.33 -server, which was faster than Java 7). But the Java code didn’t have some of the speed optimizations. So I re-translated the Javascript code back to Java. It produced the following numbers (lower is better performance):

What has happened here? The output is the same, why is Java so much slower than Javascript? I would have suspected the opposite…

I fired up the profiler to see what was causing the Java code to be so slow, and it turned out the method it spend most time in was Math.pow(). Other slow methods were Math.acos(), cos(), sin() etc. It turns out that the Math library isn’t very fast, but there is an alternative, FastMath. Apache Commons has implemented a faster Math library for commons-math. Lets see what changing Math.* to FastMath.* does to the performance:

This is already much better. But still the method causing most delay is FastMath.pow(). Why is Javascript so much faster? The method is made so you can calculate the power of two doubles, not only integer values. But I’m only doing Integer powers (7 and 8 to be precise). So I decided to implement my own method:

The speed with this new method is much better and seems comparable with Javascript. Maybe this is an optimization the V8 engine does by default? Who knows.

The slowest method in the program now is FastMath.acos. From highschool I know that acos(x) can also be calculated as atan(sqrt(1-x*x)/x). So I created a own version of acos. When benchmarked, the different methods: Math.acos(), FastMath.acos() and FastMath.atan(FastMath.sqrt(1-x*x)/x), the result is again surprising:

The custom acos() function is a bit faster than FastMath.acos() and a lot faster than Math.acos(). Using this function in the Mandelbulb renderer gives us the following metric:

So it turns out that with a bit of tweaking we can get the Java version faster than Javascript, but I would have never imagined Java would be slower in the first place. The Chrome V8 guys really did an amazing job improving the speed of their Javascript VM. Mozilla isn’t far behind, they are getting +/- 2200 ms in the benchmark. Which is also faster than Java.Math and FastMath! It seems that V8’s math implementation has some optimizations that Java could really use. The tricks used above don’t make any difference with the Javascript version.

Edit 1: Is Javascript faster than Java?

Well surprisingly in this case it is. With the code a 100% the same, using arrays as vector and Math.* the code actually runs faster in my browser!

Edit 2: People have been asking me: What could have been done to make it faster in Java? And, why is it slow?

Well the answer is twofold:

1) The Math libraries are made for ‘double’ in Java. Having a power() method work with doubles is much harder than working with just integer numbers. The only way to optimize this would be to overload the methods with int-variants. This would allow much greater speeds and optimizations. I think Java should add Math.pow(float, int), Math.pow(int, int) etc.

2) All the Math libraries have to work in all situations, with negative numbers, small numbers, large numbers, zero, null etc. They tend to have a lot of checks to cope with all those scenario’s. But most of the time you’ll know more about the numbers you put in… For example, my fastPower method will only work with positive integers larger than zero. Maybe you know that the power will always have even numbers…? This all means that the implementation can be improved. The problem is, this can’t be easily achieved in a generic (math) library.

This afternoon Will (the friend I mentioned in this previous post) showed up again on Google Talk.
He set me a new challenge: “How about translating your 3d engine/mandelbulb code into pure Javascript?”

This seemed impossible, but I quickly realized the code I had was very easy to convert. It is my first piece of Javascript larger than 10 lines I think, but it is working!

Any Javascript developers who would like to take the code and improve on it (trust me, there is more then enough room for improvement) go ahead! I’m releasing it under creative commons, do with it what you like, just be sure to mention me!

Last friday a friend of mine was talking about ray marching, the Mandelbulb and programming his own 3D fractal engine. He also kind of challenged me to do the same… So I picked up the challenge and set to work on my own 3D (CPU only) ray marching fractal engine (in Java). It was a very steep learning curve for a programmer with limited math knowledge, but I’m pretty pleased with the first results!

My first 3d engine (tm)

On sunday I had the first things ready, lighting (Blinn-Phong), soft shadows, but still I had no perspective build on (all rays travelled in the same direction) and I had no way to change the view point/camera position:

Next step was to render something other than spheres and cubes, and get the camera position under control, that breakthrough came monday evening:

My first mandelbulb! The perspective is still flat and it misses detail, we’ve also rendered through the camera/near field.
So I made some more improvements and managed to render a nice wallpaper for myself:

Yesterday I’ve been playing around with optimizing the code a little bit (less memory usage, already twice as fast as it was, but still slow). And I’ve added some glow and the ability to add ‘distance fog’. Also I’ve recorded my first movie, just to show some moving fractals:

(yes, it has a glitch with the pink mandelbulb, can’t be bothered to fix…)

This afternoon I was reading about rotoscoping. Rotoscoping is a technique where an artist takes a filmed movie and animates a drawn (cartoon) character frame by frame. The end result is highly realistic. The technique was first used in 1915’s with an ink cartoon called Out of the Inkwell and has been used in many cartoons since.

But what about video games?

It turns out that Prince of Persia was the first computer game to use rotoscoping. That game brings back a lot of good memories to me, and probably everybody my age.

After a quick search I found this little gem of a video:

The person in the video is David Mechner. He is the younger brother of Jordan Mechner, the creator of Prince of Persia.
Jordan used the clips shown above to rotoscope the movements of the main character in the game.

I had never realised why I loved Prince of Persia so much in my youth. But now I think it is probably because of rotoscopy! The animations are so realistic and life like, it has brought a whole new level of realism into video games.

Here is another video of Jordan, showing even more rotoscoping in Prince of Persia:

And the icing on the cake, a video about rotoscoping in Mortal Kombat 1:

A couple of days ago I’ve received my Arduino Mega in the mail. Together with a breadboard, some plug-wires and Arduino-starters kit (with some resistors, capacitors, and a couple of sensors).

Arduino?

The Arduino is a handy little board with micro controller. It allows you to connect your computer using USB to the micro controller and easily program and upload the program to the board. I’m a very experienced programmer with almost no experience in electronics.

I know what a resistor is and what it does, but I have no idea how/when to use which resistor. But I’d love to learn more, maybe even build some toy robots in the end. A final goal would be to have a working humanoid with 20+ servo’s, but that goal might be a bit out of reach…

The examples

The first thing I’ve done after unboxing the Arduino was plugging it in my laptop and browsing to the arduino.cc website. There you can download the Arduino IDE.

In the ‘Learning’ section (http://arduino.cc/en/Tutorial/HomePage) you’ll find everything you need to get started. There are diagrams on how to wire everything into the breadboard, and there are also code examples.

Some stuff I made using the ‘Learning’ page:

Blinking LED (the hardware Hello World, yay!)

Slowly fading blinking LED

Knight rider, KITT LED bar

Tiny keyboard with 5 buttons

Light reactive theremin

And some projects that aren’t on the example page, but could be made using the things I’ve learned:

Extended keyboard with 5 buttons and LEDs!

Light reactive LED bar (showing the amount of light in LEDs)

Knock-sensor, LED shows when it ‘feels/hears’ a knock

All the above was made in a single evening! And there is much much more that you could make with the Arduino. I haven’t even started using servos and/or DC motors. The only problem left is lack of imagination!

/*
keyboard
Plays a pitch that changes based on a changing analog input
circuit:
* 3 force-sensing resistors from +5V to analog in 0 through 5
* 3 10K resistors from analog in 0 through 5 to ground
* 8-ohm speaker on digital pin 8
created 21 Jan 2010
modified 9 Apr 2012
by Tom Igoe
This example code is in the public domain.
http://arduino.cc/en/Tutorial/Tone3
*/#include"pitches.h"constintthreshold=10;// minimum reading of the sensors that generates a note
// notes to play, corresponding to the 3 sensors:
intnotes[]={NOTE_A4,NOTE_B4,NOTE_C3};voidsetup(){}voidloop(){for(intthisSensor=0;thisSensor<3;thisSensor++){// get a sensor reading:
intsensorReading=analogRead(thisSensor);// if the sensor is pressed hard enough:
if(sensorReading>threshold){// play the note corresponding to this sensor:
tone(8,notes[thisSensor],20);}}}

The language has borrowed a lot of syntax from C, but there are some tricky issues (in the eyes of a programmer). For example: