My name is Anders

and I create stuff.

I am a creative statistics nerd that loves graphs and photos. During the days I am the team leader for the Swedish team of web developers at Procera Networks, mainly focused on creating awesome and intuitive visualizations of our data. But when the sun sets and I take to my private projects anything can happen, mostly bad things. But on rare occurances there are great success and those might get posted here on the blog.

About the blog

This is my new blog, less photos and more stuff but please remember that it's still work in progress, sortof, so if you find stuff that doesn't work or doesn't look right please drop me a line and I'll fix it asap.

Basicly the blog is divided into three main categories, Projects, Experiments and Random. Under the first category I'll post stuff that has a bit more work behind it, sometimes even a finnish. Under Experiments there will be exactly that, experiments, tests and failures, things that never made it into the Projects category. Lastly there's Random, where anything might happen, so step with care. In a while I'll add a Gallery-category, for all the photos but I haven't really figured out how I want it yet.

Projects

"A planned endeavor, usually with a specific goal and accomplished in several steps or stages."

Here I post completed and successfull projects, things that took some time and effort. Things that make me proud just slightly embarrassed. Some of these projects will be collaborations with other people but most of them will be solo projects.

Dashoids

The Telia Dashboard for Dreamhack started to feel a bit dated so Hannes and I started thinking of doing a new dashboard, but in a different way. We brainstormed all the possible metrics at DH that we could visualize, everything from traffic in the network to how many cans of soda had been sold and the distribution between sexes regarding shower-count.

Since we din't have a lot of time we had to cut down on what data we were to collect, we ended up with seven (expanded to eight during the last event).

Total traffic

The accumulated traffic amount in bytes (or rather terrabytes) from the event start.

Bandwidth usage

The current throughput through the Procera PREs, updated every second to get live graphs. It peaked at around 20Gbit/s usage of the availible 40Gbit/s

Connected devices count

The number of unique IP-adresses seen during the event, using the Procera PIC statistics.

Top games

The number of connections per game, classified by the Procera PRE. We use the Python API to connect to the PRE every second and get the number of connections.

Gameplay time

The accumulated time for all games, this ended up near 4000 days. This is a feature of the Procera PIC.

DHTV viewers

This uses the Twitch-api and a list of official streams to periodically connect and retrieve the viewer count of each stream and summarize them.

WIFI-clients

Thanks to STKJ and Phivil we got SNMP access to the Cisco WIFI gear and could get accuarate client count every second.

Dreamhack Game players

After a request from the Dreamhack Game guys we created a special rule in the Procera PRE to count the number of visitors to DHGames website and publish this.

For the visuals we decided early that we wanted something flashy using WebGL, it did't take long until we settled on a GL background and cards in the foreground that flipped horizontally when switching between the content of the cards. For the background we started with falling cubes with a nice DOF falloff. But it didn't really work, so we tried out other thing and finally settled on the wavy background that is in the final version.

During the event we came up with some ideas for future improvements. One problem is that some of the numbers are interpretations of reality, like Connected devices, so for those items we'll investigate other (better?) metrics to use to represent that data. Also the most requested feature was more linegraphs, so we'll look into how we can incorporate that into the current design.

DH Dashboard

I did get the opportunity to play with data from the PacketLogic machines that was deployed at DreamHack summer 2011. Basically we parsed a log containing a subset of all connections, extracted the outside IP (independent from if it was source or destination) and ran that through MaxMinds GeoIP API to get lon/lat. The data was mangled a bit and then fed to a website with an WebGL globe that showed the amount of connections as bars on the globe. You can see the globe "in action" in this presentation made by Karl 'IcE^' Anderssson, the teamleader for the Network-team at Dreamhack.

The original webgl-globe with bars representing data amount

The globe was shown to Telia who provided the bandwidth for the event and they were interested in a more advanced version for the next event. I knew that there would be PacketLogics at the next event also so i didn't have to change much code in the backend and concentrated on figure out what would be interesting for Telia and the participants at the event and also would look cool.
Since this was done on my spare time it took a couple of weeks in Photoshop before i had something that I was satisfied, then the next step, creating all the shiny meters in HTML5.

Quite a few hours of adventures in the world of JavaScript i hade a working frontend and realized that i needed a couple more scripts in the back to feed it with data. Not having my own PacketLogic to code against was a bit of a problem but the documentation is quite good and I got example snippets from a friend that works at Procera. Those scripts worked fine with some small tweaks at the start of DHW11.
The kind Virus84 was kind enough to lend me three blades in the blade server he had at DreamHack. The servers had enough power to handle both the geolocation decoding of about 80k IP/sec and the massive flood of visitors, after the three day event there had been more than 120,000 visitors. One problem was disk space, I had underestimated the amount of connections and overestimated how much disk there would be, but a quick update and small bash script made sure that no old cached data remained.

The final version, the realtime-graph is flat because the screenshot taken after the end of the event

The dashboard contains six different panels. In the first row we have: a WebGL globe with a heat map counting connections, a VU-meter counting unique hosts on the network and a display of the total amunt of data passing out and in from the network, and to the right a speedometer displaying in- and out-speed in real time.
At the left in the bottom row there's a graph showing traffic over time for the whole event, the middle panel show the same for the last 5 minutes and was updated in real time, and to the right there was a graph showing the percentage of top 8 games played.

For DreamHack Summer 2012 some small changes of the graphic was done in the frontend but the big changes was in the backend. Since i didn't know what kind of hardware the dashboard could run on I spent some time optimizing the slowest parts which was the geolocation and image rendering. For the first attempt the geo-data was stored in a MySQL database and every 15 minutes the amount of connections during that period was summed up and extracted for the heatmap-like globe. Unfortunately the database didn't really handle the data gracefully and the SELECT took longer and longer the more data that was put in, so every other hour all rows was deleted and the DB optimized.

For DHS12 I decided to use flat files instead, the script that did the geolocation work was optimized and changed to hold the data in memory until it dumped it out into a text file every 15 minutes. The text file was then used by the script generating the heat map. The changes and optimizations had quite a large impact on performance, during DHS12 I had a couple of dualcore Opteron machines and one of these machines managed to quadruple the performance of the quad Xeons I used last event.

Telia, who also sponsor a internet café in Stockholm wanted a "Telia branded" version of the dashboard to run at the café so me and IcE started work on it. Since the café don't have any of the cool Procera equipment that's available at DreamHack IcE started working on a completely new backend and i worked on teliafying the frontend.

Illusions

Just before DreamHack Summer 2011 Hannes "Hanes" Rahm and I sat down and talked about that it would be cool to actually submit something for the Wild-competition at DH which only was two weeks away. After a couple of hours brainstorm and research on Internet we had an idea, and Illusions is the result.

This was the first Wild/Animation entry released at DreamHack by me and Hanes. Released under the group Zink-Interactive who by placing first both in 64-bit demo and combined demo previously aren't newcomers at DreamHack. After two days work we had a lot of footage to cut together and sync with the soundtrack made by Hanes. At the end we finished at second place, with Raddox on the first place with remix of Skurt, not too bad for a first try.

Unfortunately wasn't I able to be at DreamHack when the video was shown or at the prize ceremony, but thanks to the wonderful production I could sit in a hotel room in Malmö and watch it live.

Telia DH Dashboard mk II

The new 4-way split design.

After having used the old dashboard a couple of events Telia contacted me if I could create a new one, and that I could. This time I got my friend from previous adventures on this blog, Mr Hannes. His master knowlege about design and UX combined with my backend expertese led us to a novel setup.

Insted of only using one screen we wanted to give the dashboard more impact and show more stuff larger. After coming up with a couple of new ideas we finally went with splitting the dashboard into several screens. Since the there was four racks for the servers, and the width of a rack was almost exacly the same as the hight of the TV we used, we rotated them 90 degrees. The dashboards we created needed to be usable in horizontal mode also, since most (all?) of the people who viewed them via the web had horizontal screens.

With all the extra space we decided that we could afford to use one of the screens for branding only. So the left-most screen contained no data, only a clear Telia branding with an image and slogan they use elsewhere. On the other three screens we show the number of devices on the network, the current throughput from the three main halls and the accumalitive data volume as a stack of CDs compared to different well known buildings from around the world.

Steer your browser towards https://dhdashboard.se/ during an event and you'll be able to see the dashboard in person.

Experiments

"A test under controlled conditions, demonstrate a known truth, examine a hypothesis, or determine the efficacy of something previously untried."

When I experiment and test stuff it might end up here, things like failed projects or tests for a project than never will become reality.

Snails

After the success with Illusions we started thinking of a new project, we wanted something more like a short film with a script and characters. It took several lunches before we came to the conclusion that we needed small animals that were easy to control, and didn't run away as soon as we started to film. The only animal that fitted that and that we could get hold of was snails.

Hanes photographing The Real World while we're bringing some of it back in buckets

So, off to my girlfriends parents house and to the raspberry bushes. Under the leaves there were hundreds of snails, in all sizes. We collected some of them in a can. Then we drove down to a meadow close by to get some material for the set, we figured we wanted two different settings. We built the set at Hanes place, and started to film a documentary in Attenborough style. The story was something like that the snails was a bit like elephants, and when they were old and weak the traveled to secret grounds to die.

Hanes building the jungle set

The two sets was first a jungle-set with a lot of moist vegetation and a desert set with sand and dry stuff. The snails started in the jungle set and started their travel towards the hole iron wheel in the desert. By minimal modifications of the set and different angles between shots we managed to get quite a few shots out of the small sets

Hanes filming the desert set, with the secret iron ring at the top right

One negative side of using real material for the sets were all the tiny bugs that lived among the dirt and grass, some ruined the shots by running by and others wandered of on the floor. At the end of a days shooting we had all the shots we needed and carried it all outside in buckets and cleaned up.

A couple of days later I got a message on IRC from hanes, the disk where all the footage was stored had crashed, in a bad way. After finding some deep level data recovery software and running it he was able to save about half of the stuff, but the disappointment of losing soo much footage dampened our lust for working on the project and it was never finished.

Random

This is random posts, anything that I want to share with the world and which don't fit into the other categories. It can be anyting from pre-experiment tests to one-off ideas that I probebly won't do more work on.

Illusions - BTS

There're was quite a lot questions about how the effects in Illusions was created. Some guessed that we created a new flow-dynamics algorithm others had even stranger ideas. So here's a quick post about the process that finally led us to the effects in the video.
First, it's all real fluids that anyone has in the fridge, into ordinary cold water.

Hanes looking at some of the footage while shooting in my kitchen.

It all started with that we (that's me and Hannes) saw some videos of ink in water (quite far from the result we finally got) on Youtube and thought that we might be able to do something cool with that. We created a small "studio" setup in my kitchen, the main parts was a 200W light in a shoebox, a cake-plate covered in aluminium foil with a 5cm hole in the center and a plastic container from IKEA (hey, we're swedes!). Later we had to add a fan to cool down everything since the lamp gets quite warm.

The "contraption" that made it work, with food colouring (failure) in water.

With the setup ready and the water tap close by we started to add stuff while filming it. We started out with different food-colour liquids (bad), soy (bad), oil (bad), mixtures of the previous (worse!), it was mostly a failure. Then we reversed colours of the setup, instead of dark on white we removed the white background and replaced it with a black cloth. Now we started with milk at it was perfect, behaved quite close to what we hoped and as time progressed we got more and more amazing shots. To get a little variety we tried different temperatures, cold in cold, hot in cold etc. and it changed how fast and how much the milk dissolved in the water. To dispence the milk we ended up using syringes. We had three different sizes and for some of the clips we used no more then 1-2ml of milk was used, and despite that we used up quite a lot milk.

After we switched to dairy products, black background instead.

With all kind of hot and cold milk tested we ran across the street and bought milk with different fat contents and also cream. The more fat it contain the more 'thready' the result was, also it took longer for cream to dissolve. So after an hour or two of failures we continued pouring lactose-products into water for several hours, before we had enough clips to be satisfied.

Hannes wrote some music and we sat at his place to cut it into something presentable. It took far longer than we expected before the result was presentable, and sometime after midnight we decided to cut it from the planned 3 minutes down to 2:30 minutes. All and all we're very happy with the result, and it got quite a lot of attention around the world. It might even get us into a music video, more about that in the future.

We had to empty and wash the container after shot

You can find more and other information about this at the official Zink-Interactive blog.