Chris' tech bloghttp://blog.papauschek.com
Daily challenges of a web startup.Mon, 08 Aug 2016 14:05:22 +0000en-UShourly1https://wordpress.org/?v=4.4.11Why and How we are using Scala at journihttp://blog.papauschek.com/2014/07/why-and-how-we-are-using-scala-at-journi/
http://blog.papauschek.com/2014/07/why-and-how-we-are-using-scala-at-journi/#commentsSat, 12 Jul 2014 23:09:35 +0000http://blog.papauschek.com/?p=253Continue reading →]]>Last year we started building our web/mobile startup journi (called miavia back then), and I considered 3 different backend languages before finally settling on Scala:

PHP. I didn’t want to continue working with a language that had grown into a monstrosity.

C# .NET. Solid, stable language with functional programming features, but I wanted an open source solution instead, because I found Windows Servers less affordable and more difficult to automate than Linux Servers.

Java. The JVM is as solid and fast as the .NET CLR, but Java was lagging 5-10 years behind C# in language features.

Enter Scala.

I had worked with Scala in 2010 and found it interesting, but at the time it seemed unusable for production. This had changed when I came across Play Framework 2 in 2013 and found that it was exactly what I needed.

I am not a language advocate, so don’t be surprised that I’m not trying to convince anyone to use Scala. But after 18 months of using Scala we found that:

Handling collections and options is very natural and readable, as control flow does not depend on exceptions and generally needs fewer tests.

Generally I found it easier to understand and write correct code that uses immutable variables and a functional programming style.

As the project grows bigger, we can more easily move forward with changes and refactor things. If the project compiles after a lot of renames, deletions, and so on, we have a lot of confidence that the code is still correct.

Scala’s Futures API makes it easy to interface with third party API endpoints while dealing with network timings and exceptions.

We use a lot of existing Java libraries. Very little work is needed to integrate with third-party services, because of the existing Java ecosystem.

Where performance is needed, you can easily get it. Sometimes, the most readable code does not perform as needed. There are a lot of monitoring tools for Scala out there. They can profile your app in production and give you detailed performance characteristics of your methods.

This outweighs the disadvantages:

In the beginning, newcomers from Java won’t immediately grasp how to use Options and other monads. Learning functional style of programming takes time, especially knowing when to use which paradigm.

You need a fast computer in order to have decent compile times. This is only a problem when checking out new code that has a lot of changes. While developing, the incremental compiler is quite fast.

Sometimes, because there are so many stable Java frameworks and new Scala frameworks, you spend more time evaluating which library to pick for basic things like database access. I’m pretty sure this will improve as the community matures.

I get asked quite often why we chose Scala in the backend, so this blog post serves as a reflection on this topic. Feel free to comment if you have more questions

]]>http://blog.papauschek.com/2014/07/why-and-how-we-are-using-scala-at-journi/feed/2Scala Akka Workshop – Evolving a Robot in an Actor Clusterhttp://blog.papauschek.com/2014/02/scala-akka-workshop-evolving-robot-actor-cluster/
http://blog.papauschek.com/2014/02/scala-akka-workshop-evolving-robot-actor-cluster/#commentsTue, 25 Feb 2014 14:16:27 +0000http://blog.papauschek.com/?p=202Continue reading →]]>It is quite extraordinary what is possible nowadays with a group of interested developers and powerful tools such as Scala and Akka.

Three months ago we had our first Akka Workshop here at the Vienna Scala User Group (VSUG), in which 25 developers participated in a real-time networked programming challenge. They had to compete in the non-zero-sum game of Prisoners Dilemma, which was powered by a central host, and network communication implemented with Akka.

Building an Akka Cluster

In the second workshop I wanted to raise the stakes a bit: what if we could make use of Akka’s clustering features? What if we could develop a cluster and a distributed algorithm on top of that cluster, during a 3 hour workshop?

Kicking off the Akka Workshop at Sektor5, Vienna

Rafael, co-organizer of the VSUG, found a programming challenge that would be perfect for the task: a robot that has to navigate a grid-like environment and collect cans to get points. He got the idea from the book Complexity: A Guided Tour by Melanie Mitchell.

The environment of the robot: a 10×10 field surrounded by walls. The robot has to pick up as many cans as possible.

The challenge was that the robot could only see a few surrounding cells, and it’s difficult to determine the best decision to make in each situation. The robot has no memory and has to act quickly – it has only 200 turns to pick up all the cans.

Tricky: the robot can only see its immediate surroundings (walls and cans). This makes a total of 128 different situations. In each of these, it can do one of 6 things: Go Up, Down, Left, Right, PickUp a can or Walk Randomly.

The main reason this task was perfect because the author of this particular robot challenge used evolutionary algorithms to find the best programming for the robot. And evolutionary algorithms can easily be distributed in a cluster.

First Steps during the Workshop

We prepared the workshop in a similar manner to the last one – there was a public GitHub repository for everyone to get started quickly. This time, a lot of the logic necessary for evaluating a robot for an evolutionary algorithm was already implemented. But everything else had to be done during the workshop: the Akka cluster itself, and the evolutionary algorithm on top of it.

In the first hour we tried to get the cluster running. Akka’s clustering implementation is pretty straightforward – after exchanging a few IP addresses, we could already see a few machines pop up on our cluster visualization.

Developers who connected their laptops to the cluster

But there was something odd: a few machines dropped out of the cluster after a while, and stopped receiving messages from other machines. After a little bit of investigation, we found out that not everyone could ping other machines in the wireless network we used. It was pretty weird, considering that no-one had problems reaching my laptop, or connecting to the internet. It seemed to be some kind of network split.

Luckily, Yves, owner of Sektor5 where the event was hosted, explained to me that some Linux network drivers have issues with wireless routers broadcasting in multiple frequencies, and ultimately we could solve the problem by forcing those clients to use 2.4 Ghz.

Starting the Robot (R)evolution

After addressing some of the networking problems, the basic Akka cluster was running and we could focus on implementing the distributed evolutionary algorithm.

The basic idea was this: developers could create solutions to the robot problem, and broadcast them to the cluster. Robots were simply represented by a list of 128 decisions (one for each situation it could encounter). As shown before, in each of those situations the robot could either go Up, Down, Left or Right, PickUp (a can) or move randomly.

Wolfgang explaining the robot problem and evolutionary algorithms

So it was simple to create initial solutions to the problem. Also, code was provided to evaluate those solutions and calculate the number of points each robot would get. The robot would be evaluated on 20 different fields of size 10×10, and got points for each can it collected (or minus points for running into walls).

Now the challenge was to combine solutions to create better ones, in the spirit of evolutionary programming. Wolfgang was careful to introduce everyone to the basics of genetic operations, so that everyone would know how to implement the basic algorithm.

The initial, randomly created robots performed poorly and started with negative points as low as -100k for running against walls, and trying to pick up cans that weren’t there. Quickly, however, the first positive solutions appeared on the screen and reached up to 90k points. This was already a 90% solution to the problem. The best solution, collecting all cans (which is probably not possible), would be 100k points.

Once a robot solution was created in the cluster, they were broadcasted to everyone else so others could base their solutions on them. We also tracked the evolutionary origin of robots, so we could see how often solutions jumped from cluster node to cluster node, and how many improvements happened on each machine.

On this particular robot, we already saw several developers contributing to its evolution. We watched for a few minutes to see if it would evolve to achieve even more points – or “higher Fitness”, as it is called in evolutionary programming lingo. But it seemed to be stuck, and not improving at all.

Mutant Robots to the Rescue

So it seemed we were stuck in a local optimum, something very common with evolutionary algorithms. If there is not enough randomness in the system, solutions quickly converge to a solution that cannot draw on other “approaches”, because it was evolved from a very limited set of solutions.

We thought that increasing the randomness, or “mutation rate” in our system, could lead to a better solution. Also, we decided to throw away our current population, reset the cluster, and start from new.

After everyone implemented mutations, small random changes to the robots decisions, we watched the cluster start with new solutions, slowly improving and getting over the 50k points mark. From then it quickly reached again 90k, but this time it did not stop there.

Final Solution

The robot improved more and more, and finally stopped improving at 97980 points – that’s almost 98% of the cans collected!

Best robot with fitness 97980 and its generations history. 10 people contributed to this robots evolution.

Impressive – I didn’t think that we would actually get a solution that was so good, in such a short time! In just three hours our 26 participants implemented a cluster with Akka, and an evolutionary algorithm on top of it.

In Philip’s Actor implementation we can see that he regularly broadcasted the best robot to the cluster, for everyone to improve on it. Also, he added other robots received from the cluster to his population, and he always kept the best 100. By constantly sending himself CrossoverRandomRobots messages, he made sure that his CPU was fully utilized and evolving.

We can see that he crossed-over the code of two random parent robots, cutting them at a random position and recombining them. The mutation code (not shown here) was also very interesting. His strategy was to go through every single position in the robot code, and see if a random replacement of that number resulted in a better robot. It seems it wasn’t necessary to mutate more than one number at the same time. Check out the full code for more details.

Behind the Scenes

One of the big challenges was to provide a simple API for the developers, that can be quickly understood in the timeframe of a workshop. Judging from what people told us after the workshop, we succeeded in reducing accidental complexity. So nobody had any troubles understanding how the evolutionary algorithm can be implemented.

Thanks everyone for participating in this workshop, it was a lot of fun. I also want to thank all the organizers of the VSUG who made this workshop possible, especially Wolfgang and Rafael.

]]>http://blog.papauschek.com/2014/02/scala-akka-workshop-evolving-robot-actor-cluster/feed/6World map coding challenge (Android)http://blog.papauschek.com/2014/02/world-map-coding-challenge-android/
http://blog.papauschek.com/2014/02/world-map-coding-challenge-android/#respondSat, 22 Feb 2014 18:51:18 +0000http://blog.papauschek.com/?p=283Continue reading →]]>The goal of this coding challenge is to build a world map that can be scrolled and zoomed into by the user. Here’s a video that shows how the end result might look like:

Requirements

Make sure the user can scroll the map by panning vertically or horizontally (see video)

Make sure the user can zoom in by pinching the screen (see video)

The solution should run on Android 6.0 (e.g. on the Nexus 5X Emulator).

Bonus requirements:

Extra kudos if you make it perform well (e.g. no stuttering during scroll).

Bonus points for smooth scrolling with inertia.

How to draw the vector data

The JSON file is in the GeoJSON format, which contains data about countries and their geometry. The file for this coding challenge contains arrays of polygons that should be used for drawing. Coordinates of the vector points range roughly from -20.000.000 to +20.000.000, so you may have to rescale the values to fit on screen.

(Each “feature” represents a country, and the “coordinates” property in the above example shows the list of x and y coordinates of the country’s polygons to be drawn to create the world map. More information about the GeoJSON format and its specification here: http://geojson.org/ )

For this coding challenge, only Polygons and MultiPolygons (which is a collection of Polygons for larger countries) are used.

Please don’t use any third-party frameworks or libraries for this that are not available by default in the Android API 22+.

Further information

It’s best for us if you send your solution as a .zip file with an Android Studio project or Gradle Project that we can build right away. You may also put it on your GitHub account and send us a link.

Good luck, and be sure to send us even a partial solution rather than nothing at all! We’ll send back to you a complete solution for the problem in any case.

]]>http://blog.papauschek.com/2014/02/akka/feed/0World map coding challenge (iOS)http://blog.papauschek.com/2014/01/world-map-coding-challenge/
http://blog.papauschek.com/2014/01/world-map-coding-challenge/#respondFri, 10 Jan 2014 09:00:03 +0000http://blog.papauschek.com/?p=234Continue reading →]]>The goal of this coding challenge is to build a world map that can be scrolled and zoomed into by the user. Here’s a video that shows how the end result might look like:

Requirements

Make sure the user can scroll the map by panning vertically or horizontally (see video)

Make sure the user can zoom in by pinching the screen (see video)

The solution should run on iPhone 6 on iOS 9.

Extra kudos if you make it perform well outside the iOS simulator (e.g. on slower iOS devices).

How to draw the vector data

The JSON file is in the GeoJSON format, it contains data about countries and their geometry. The file for this coding challenge contains arrays of Polygons and MultiPolygons that should be used for drawing. Coordinates of the vector points range roughly from -20.000.000 to +20.000.000, so you may have to rescale the values to fit on screen.

Please don’t use any third-party frameworks or libraries for this that are not available by default on iOS 9.

(Each “feature” represents a country, and the “coordinates” property in the above example shows the list of x and y coordinates of the country’s polygons to be drawn to create the world map. More information about the GeoJSON format and its specification here: http://geojson.org/ )

For this coding challenge, only Polygons and MultiPolygons (which is a collection of Polygons for larger countries) are used.

Further information

It’s best for us if you send your solution as a .zip file with an Xcode project or workspace that we can build right away. You may also put it on your GitHub account and send us a link.

Good luck, and be sure to send us even a partial solution rather than nothing at all! We’ll send back to you a complete solution for the problem in any case.

]]>http://blog.papauschek.com/2014/01/world-map-coding-challenge/feed/0Play Framework Evolutions with Slick 2.0 Code Generatorhttp://blog.papauschek.com/2013/12/play-framework-evolutions-slick-2-0-code-generator/
http://blog.papauschek.com/2013/12/play-framework-evolutions-slick-2-0-code-generator/#commentsSun, 29 Dec 2013 18:40:03 +0000http://blog.papauschek.com/?p=167Continue reading →]]>The new Slick 2.0 database library comes with a simple code generator that generates Scala code for all your tables so you can quickly start coding against your database. It takes into consideration not only the tables and columns, but also primary keys and foreign key relationships.

Integration with Play Framework Evolutions – No More Boilerplate

So I thought it might be interesting to integrate this code generator with the Play Framework 2.2. Play has an Evolutions Plugin that takes care of database migrations. A basic evolutions file looks like this:

Of course, if we later remove the “first_name” column from the user table (in a later migration for example), we want the compiler to tell us that the column is no longer there. This is a huge productivity boost when dealing with many tables, and avoids writing a lot of boilerplate code.

How can this be done?

My basic idea was to write small Play module that takes care of the code generation. Dependency on the Play Framework is needed because we need the Evolutions Plugin to know how our current database looks like.

You can find my working prototype of this here on GitHub. If you just want to give it a try in your own project, take the play module from my code and make the necessary changes to your SBT build file.

If you’re interested in how it works, read on…

Introducing the DbGen Play Module

The sample prototype project uses a Play sub-module called “dbgen” for code generation. The “dbgen” module consists of just one class (PlaySlickCodeGenerator.scala) and takes care of applying evolutions from the main project into an in-memory database (configured here: /dbgen/conf/application.conf). The main method starts a fake Play application, invokes the Evolutions plugin, and then calls the Slick code generator.

Of course, now we want to have a seamless integration into our build process while developing. SBT takes care of this, we just have to provide a new build task.

Automatic Code Generation when Evolutions change

The new build task should only run if our Evolutions (and therefore our database) have changed. Lets take a look at our Build.scala. With the main module and the DbGen submodule, it looks like this:

This defines both modules, and that the main project depends on the dbgen code generator. Also, we define a new code generator task slickCodeGen that can be invoked from the play console (if we want to start it manually), as well as an entry for automatic source code generation (sourceGenerators in Compile).

The actual SBT task is also defined in the Build.scala, but not listed here for brevity. It basically checks whether evolutions have changed, and starts the PlaySlickCodeGenerator to generate the code if necessary. The output is placed into the “target\scala-2.10\src_managed\main” directory alongside other generated code by Play (routes, templates, etc.). It will be automatically picked up by IntelliJ IDEA and other Scala IDEs, for auto-completion support and other goodies.

The code generator can also be started manually from the play console using the “gen-tables” task:

What you can do with this

You can now continue using Play evolutions, and your Slick tables will be automatically generated. No more Slick boilerplate.

When you make changes to your database, hit refresh and you’ll see the new tables and columns appear in auto-complete. Even have all the metadata like foreign key relationships.

Get compiler hints whenever database changes break your code.

Known Limitations:

The SQL in the Evolutions files has to be compatible with both your production database and the H2 in-memory database. H2 can be configured to be compatible with most databases. This example uses MySQL, hence it activates the MySQL compatibility mode of H2.

Request for comments

There’s got to be a better way to integrate the Evolutions Plugin with SBT. If you have an idea to make the solution cleaner or avoid having to use a fake Play Application, please let me know in the comments. If you found this useful, please let me know as well

]]>http://blog.papauschek.com/2013/12/play-framework-evolutions-slick-2-0-code-generator/feed/63 Things Learned from using Akka in a Coding Workshophttp://blog.papauschek.com/2013/11/3-things-learned-from-using-akka-in-a-coding-workshop/
http://blog.papauschek.com/2013/11/3-things-learned-from-using-akka-in-a-coding-workshop/#commentsWed, 27 Nov 2013 11:47:35 +0000http://blog.papauschek.com/?p=127Continue reading →]]>Here are 3 learnings from last weeks Akka coding session, in which all participants connected their Actors in order to participate in a Prisoner’s Dilemma competition.

1. Prepare Your Networking Infrastructure for Akka

Akka needs TCP connections in order to work. In some venues you will run into issues with Router Firewalls that block certain ports, and other surprises. We were lucky that our host didn’t block any ports, but we had a backup Wireless Router just in case.

When using Akka with its default settings, some machines will use their local loopback address (e.g.”127.0.0.1″) for their akka host. In this case, their Actors will not be able to receive any messages. The solution is to get their LAN ip address manually (e.g. via ifconfig) and put it into the application.conf:

akka.remote.netty.tcp.hostname = "192.168.0.81"

2. Expect Latency

Hosting system timing out and receiving late responses out of context (see debug messages on the left side)

Dealing with 25 machines worked remarkably well. During the workshop, each participant had at least one Actor connected to our “hosting Actor”. The average time the hosting Actor waited for a response from a participants Actor was about 100 milliseconds (keep in mind we used a wireless network).

However, as soon as the number of messages on the network climbed to about 100 messages per second, the hosting Actor started to ignore some of the other actors because it waited over 1 second for a response. The hosting Actor used Akka’s ask pattern to wait for the players responses in context:

As the messages took too long to arrive, some responses were collected by the host at a time when the ask pattern already ran into the 1 second timeout. Suddenly, the hosting actor started receiving those messages out of context, because the virtual actor created by the ask pattern was already gone.

3. Let It Crash

Part of the fun of a workshop like this is that people will start to mess with the system (yes, I’m looking at you, Wolfgang! )

That’s actually a good thing, because a lot of things can be learned. In our case, there was an “exploit” that allowed people to connect as many actors as they wanted to the central host, and so at one point our high score was flooded with players. The system kept running, but the tournament started to slow down because of the high number of players.

We did, however, run our hosting system in Untrusted Mode. So at least, this makes it difficult for everyone to start and stop Actors on the system remotely

]]>http://blog.papauschek.com/2013/11/3-things-learned-from-using-akka-in-a-coding-workshop/feed/4Scala coding competition: Akka and the Prisoner’s Dilemmahttp://blog.papauschek.com/2013/11/scala-coding-competition-akka-and-the-prisoners-dilemma/
http://blog.papauschek.com/2013/11/scala-coding-competition-akka-and-the-prisoners-dilemma/#commentsFri, 22 Nov 2013 15:10:07 +0000http://blog.papauschek.com/?p=86Continue reading →]]>More than 20 people with no prior Akka experience, 10 minutes preparation talk, one idea: getting everyone to write Actors participating in a realtime competition to beat others in the game of Prisoner’s Dilemma.

Sounds challenging?
Yes, it is.

Did we manage to do it?Yes, we did!

Before I explain what we did during the workshop, big thanks to the Scala Vienna User Group organizers and participants (special thanks to Rafael and Manuel for helping out), and to Stockwerk for hosting us! Also, Heiko Seeberger, Director of Education at Typesafe, came all the way from Munich – thanks for joining the workshop!

Actor what?

The workshop was split into 3 steps: First, I quickly introduced the concept of Actors and provided some code examples. And then the first challenge: develop your first actors in preparation for the game of Prisoner’s Dilemma. Once everyone was able to code basic Actor communication with Akka, we could move on to the final part: get everyone connected.

Real time tournament of Prisoner’s Dilemma

Using actor communication, all participants connected their Prisoner actors to the Tournament hosting actor, participating in a real-time round robin tournament. The tournament system continuously picked actors from the pool to play a game of Prisoner’s Dilemma against each other. Each actor represents a Prisoner in the game, and has to decide whether or not to cooperate with the other actor. Making the right decisions gives you points in the game, and we displayed the results live on a highscore.

Prisoner’s Dilemma explained

The game of Prisoner’s dilemma is actually pretty simple. Both prisoners have to make a decision. Each of them knows the name of the other prisoner, and has to decide: do you cooperate or defect? If both cooperate, they get some points. If both defect, they don’t gain anything. But if one cooperates, and the other one defects, the cooperator looses points and the defector gets it all, exploiting the other prisoner.

An Evil Twin strategy

If you are a prisoner in the Prisoner’s Dilemma, you better remember who the bad guys are, otherwise you’re not going to make a lot of points in the game.

After running the tournament for a while, one participant using an “Evil Twin” strategy scored most of the points. Peters strategy was straightforward: Be nice for a while, but then switch strategy and try to exploit everyone for a while, before turning back to being nice again. It’s quite challenging to implement a strategy like this in just a few minutes time. So, congratulations Peter! You can see his Evil Twin code here on GitHub.

Connected Actors for the win – what’s next?

The gamification aspect of the workshop was a lot of fun. Everyone could develop different strategies and watch how they play out against the other ones in real time.

Now one question remains: what’s next? If you have some ideas, some games perhaps, or other things that could be implemented with live-feedback during a workshop, please let me know! Comments and suggestions are always welcome.

Even though Play Framework 2 is a young framework, we found it to be quite stable for production use. Still, here are some points you may want to consider when working on a serious web project using Play 2.2.

This is a series of blog posts touching different aspects of web development and deployment. Here’s the list of topics:

3.) Logging (coming soon)

By default, there will be only one log file created by Play, this may not be what you want on your production server. How-to coming soon.

4.) Developing with Play: CSS/less compilation (coming soon)

If you have lots of stylesheets, or when using Twitter Bootstrap, the built in less compiler can become annoyingly slow during development. Thankfully there is an easy workaround using the faster lessc compiler. How-to coming soon.

5.) Developing with Play: Javascript reverse routes (coming soon)

Play can automatically generate javascript reverse routes which can be used in your javascript front-end. In production, you will probably want to minify and compile your javascript to one big file for performance reasons. How-to coming soon.

]]>http://blog.papauschek.com/2013/10/using-play-framework-2-in-production/feed/3Play Framework 2 in production: HTTPS using nginxhttp://blog.papauschek.com/2013/10/play-framework-https-nginx/
http://blog.papauschek.com/2013/10/play-framework-https-nginx/#commentsFri, 04 Oct 2013 15:00:18 +0000http://blog.papauschek.com/?p=30Continue reading →]]>If you provide some sort of user login on your page, you’ll probably want to use HTTPS/SSL. Play Framework 2 supports secure connections using the recommended setup with a reverse proxy such as nginx.

This means your application server will run your play framework application on a private port (the default is 9000), as well as an nginx web server which listens to the public ports (HTTP/80 and HTTPS/443), and redirects those requests back to your play app.

Nginx Configuration for HTTPS

Installing nginx on most linux distributions is usually a one-liner (please refer to the nginx documentation). Here is the configuration we use for setting up nginx as the reverse proxy. In this example we assume that your domain name is example.com and that your compiled Play application files (using play dist) are located in /var/example/target.

UPDATE: The some paths may have changed for Play Framework 2.3+ and later. See comments.

Checking for SSL in your play app

Note that in this example configuration, ALL traffic will use HTTPS. If you want to use both HTTP and HTTPS, you’ll have to remove the redirects from http to https. You can check in your play app which protocol the request uses: