Knowledge of Knights and Nerds

Last week I got the privilege to do the closing key note at the ITMasters2015 in Amsterdam. I like to share the written version of this speech with everyone. I jumping right in and skipping all the „Good evening“ talk 🙂

I am traveling a lot these days and I have been at some Meetups recently to speak about MicroService and their cultural impact in the IT. Also as you might have recognized from my accent, I am German and with that I could probably consider myself a bit stiff when it comes to social etiquette. I prefer to shack hands, firm but not to hard, but please no hugging.

Let me briefly tell you a story: When I was preparing to held a talk in Ireland the host asked me how I would like to be introduced: “just some short words, then I will join you on the stage we shack hands and I am taking over with introducing myself”. He didn’t consider this to be adequate and told me we should just keep it simple. All right then, let’s keep it simple.

So the host stepped on the stage, said some warm words about the event and finally introduced me. My queue to walk on stage! Suddenly I feel how my arm moving. The host, a bit surprised, also lifts his arm. So we were approaching each other, having both are arms ready for a firm but manly handshake, but shortly before we reach the point of no return I remember, this wasn’t what we agreed on and quickly let my arm slip down, as did the host, resulting in a very weird passing by.

So, what is this story about? I would consider myself smart enough to not be a shame for our species still I did this wrong. Because Change is hard!

But change is a constant in this profession and you need to find your filters early to not be overwhelmed by the massive amount of information that are being available for consumption. You probably can visualize yourself sitting at home frustrated about how many new read worthy links your Twitter Feed offers you – and you just have been away for a short bathroom break. So buckle up, you are about to enjoy a great ride, this is just going more and more insane.

When I started as CTO for Lieferando in Germany, this title wasn’t actually earned. This was a honey pot to get me on board for 400€ a month and a mobile phone. In the end I was a 24 year old student, so this was a shitload of money and I got an IPhone 1 so I got this going for me. But I had to understand within one week that the code I was writing for University was worth nothing in business such as ours. Here speed matters so I coded the hell out of my keyboard and created a prototype within 1 month just to figure out 12 month later that I should have at least apply some things I have learned during my studies to save time in the long run, but here we are now. So ultimately I needed to do what is the fear of every project manager, the refactoring and with it all the symptoms: projects got delayed and investors got annoyed. I realized how much of my code actually represents the health of the company and that this is all it had in terms of assets. Code isn’t just a tool, it is the centre of the company and it needs constant change to improve, as does your company. There is no such thing as a perfect product, period!

So first live lesson learned: Change is the new normal

So I remember myself sitting in those chairs – if I was lucky to get a chair, because Berlin University are badly crowded – So I was romanticizing about all the problems I am going to solve in my career. All those fancy languages I am going to make use of and all those technologies I can experiment with. Fairly this is the amazing part about the profession of ours – we usually have more solution than problems to solve while to be honest most of the problems we are creating ourselves.

Speaking of us….

How blind of me to realize very late: the main problem in IT, this is you and I – and by this I mean people in general. The boundaries towards laying your hands on code have been struck down with the start of the Internet. It feels like training courses and tutorials making up the majority of the Internet – if you would ignore porn. We already have issues with over opinionated IT people such as Linus Torvalds who repetitively freaks out in the Unix kernel mailing lists, swearing and figuratively spitting towards the recipient with some rather rude words.

And I quote from one of his public comments in a pull request: “The above code is sh*t, and it generates shit code. It looks bad, and there’s no reason for it. So I really see no reason for this kind of complete idiotic crap.“.

Let’s not be this way, this kind of hatred becomes common in the IT business and is isolating groups. I haven’t yet heard someone publicly say: “Yes I do like Java, but I also like PHP”. Honestly, most of the opinions are just reflected from something someone just read on a blog and adapted because in IT you apparently need to have a strong opinion about everything! The opinions can mostly be separated in two definitions

Religion: developers defend the ideas they have without needing much evidence to back it up.

Elitism: some people are being elitist and rejecting technology because of reasons that have nothing to do with technology.

We as the engineers of a young faculty need yet to learn a lot more than just engineering skills and need to recognize the responsibility that has been put into our hands with the talents we sharpen during university. We as coders shape the world – or to be more hip: we disrupt business, but also social conventions and culture. Nothing will be spared, nothing is sacred – resistants’ is futile.

The applications that we write become part of our daily life and if we are lucky even of the life of others. Code is everywhere even where we do not recognize this – fricking coffee machines are running Linux these days, this is the world we live in: a big network of things that communicate with each other. Buzzword alarm: Internet of Things!

We need to recognize our code as a language such as our spoken word and as such it has as much influence towards others as if we were speaking to them directly! As Chomsky figured some time ago: the only difference of our natural language compared to programming language is the presents or absences of context – or to say for programming language there is only one interpretation while the spoken and the written word underlies tens of interpretations.

So first recap: we have natural language that has multiple interpretations and we have programming language providing us with multiple solutions to far less problems … great. Seems to be an easy assignment to find an agreement of what the best solution is that a team should go for, right? Right!

I experienced this by heart when I became the CTO from takeaway.com 8 month ago. To give you some context: Lieferando, a German based food delivery start-up, the company I have been the CTO for 5 years before, has been acquired by takeaway.com in early 2014. Therefore the business wasn’t new for me, but this didn’t matter. Two application stacks, two teams but more importantly two cultures have been forced together. Sadly not the only contradiction I had to face. While in Berlin I was not questioned anymore the team Enschede was and is very doubtful about every step that I am doing. Both not ideal situations, because without being questioned you cannot reflect on the work you are providing and are exposed to suffer from imposter syndrome – you miss the constant change because of missing criticism. But with constant doubt it is also hard to move together with a team.

So I needed to learn in a foreign language in a foreign country to explain my ideas of a high quality product in such a manner that the developers could translate this to code without me watching over them all the time. I never ever improved more in such a short time.

So second life lesson: Code is your language; with code you express your abstraction of the reality and document your thoughts. But your thoughts must be put into words first.

Furthermore code creates networks, architecture emerges, and infrastructure is being setup. Organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations – not my words, but those of Melvin Conway. A man, who recognized something so simple yet so intriguing. The structure of the team we are working in and the way this team is exchanging information constraints the outcome of the system we aim to build. Once again it hit me that my social reality is reflected in the system my code produces. If you think about that ‘team’ by definition can also be an open source community. Obviously they are building more decoupled modularized components, because this is the way they need to communicate with all those remote, probably even unknown persons while product focused teams are rather building monolithic applications since they have all their communication happen in one room.

So third live lesson learned: Architectural decisions are mostly not related to educational level of the teams but to the communication patterns happening within the team. Those are that define the business and the architecture.

To sum this up! Code influence society, but code is also shaped by society. This is a bilateral exchange. You can never force a team to change their architecture if you don’t take the persons into account involved in this project and if you cannot explain your ideas to them. They will have to change their communication structure to make this work as well. So swarm out and work with different teams, don’t stick to the same peers for coding all the time, you will stop learning at some point in time. Get insights from other companies in other countries to fully understand the way your code changes if you are experiencing different languages and cultures. Incorporate the business you are working in into your application and you will be a great success. Or at best contribute to open source projects, there you get the full charge!

In this sense I like to quote Linus Torvalds once more

In open source, we feel strongly that to really do something well, you have to get a lot of people involved.

With the strive for changing software architectures towards MicroServices events step even more into the focus of attention. Some developers have already been familiar with Event Driven Programming but not so with Event Driven Architectures like distributed systems of MicroServices in particular. Though logically both should be kind of the same approach, I experienced developers implemented things differently.

If we look on an Event Model in any monolith, we do things like notifying other parts of our system about things that just happened.

Java

1

2

3

4

publicvoidlogin(){

Publisher publisher=newPublisher();

publisher.emit(newLoginEvent());

}

While in a MicroService environment, where events become the essence of the entire system, they are degraded to commands instead of actual events (once again the Oxford Dictionary defines an event as: A thing that happens or takes place, especially one of importance) . I also have seen those kind of things in systems where only part of the application have been decoupled and communication is done via any kind of Bus or Queue system.

Java

1

2

3

4

5

publicvoidsendEmailToCustomer(){

Queue queue=newAmqpQueue();

queue.setQueue("email_queue");

queue.send("maybe some json with the task to send out an email");

}

So what do we have here? We have a glitch in the matrix … while before we did tell the system what just happened, we now inform someone what has to be done. So we actually switched from Event Driven to Command Pattern. I assume this lies in the nature, we do not trust that others react upon our notes so we rather tell people to do something. Command Pattern is a bossy pattern and is as ineffective as micro management for distributed systems.

There is so much wrong with this…

First of all MicroServices ain’t MicroServices if they are not loosely decoupled. In this sense Command Pattern makes no sense at all, since one service needs to know about what the other needs. That is not loosely decoupled.

Events should be about what just happened not about what is to be done. Events represent the present not the future.

Every time someone needs to inform other developers to change their events if they have changed or extended their implementation

So let’s have a look at this picture. In the above example we have three programs (P1, P2, P3), which are sending requests to the fourth program (P4) to react upon and fulfil a task. Every time P4 changes it’s implementation we need to notify the developers of P1, P2, P3 to change their request if the requirement for the event has changed, e.g. additional data needs to be provided.

In the next example we still have three programs, but they all emitting different events (e1, e2, e3) and inform the system about what just happened on their side. P4 has attached to all those events, since those are events he wants to react upon. Now if P4 is changing its implementation it only has an impact on P4 because he has control.

Sure P1, P2 and P3 sometimes change their events as well, but I do belief that the Open/Closed principal doesn’t apply only to classes but also to the definition of events (no matter if json, protobuf or anything else). You can extend it any time, but you should never change what is already present. This way P4 will always work with the events from P1, P2 and P3 even if they have extended their events dramatically. P4 just doesn’t care, since the event, which was used during the implementation phase, is still a subset of the current event therefore still valid for P4.

It is like a rule, it always happens, you can’t hide from it, it will find you…

No matter how smart you are designing your service or the UI of your application. The user will end up with weird ways of using it, I guarantee you! Apparently it is the way the human brain is defined, otherwise I cannot make sense out of it, since it doesn’t happen only to software. Even in the wild people do see intend in things where others don’t.

Maybe here is the reasoning for IT designing weird error messages, since they just don’t get how the customer might react. It is like the US law forcing companies to come up with messed up rules on their products to avoid complete misuse. Ever heard about the „cat microwave lawsuit“? No, than you belong to the 26% of people that don’t know about it, but it’s fine, it is an urban myth, while still a good example for weird „please don’t use it that way“ sentences in some instructions.

So what it is with instructions in the first place. A common rule in UI design is: „if you need to explain it you done it wrong“. But this rule has been most of the time sarcastically interpreted as: „you need to dumb it down for the user“. This assumption – since as we know assumption are the root of all evil – leads directly into the user experience hell.

First of all the user isn’t dumb, he or she just tends to misinterpret things that are provided by the developers in the first place. So dumbing things down should actually resolves to removing things that confuse or distract. The devil lies in the detail therefore simplicity isn’t something that can be simply achieved. Jeff Atwood describes it very nicely in his blog post about „the god login„, a login box and how much effort his team has put into that to make it easy to use and understandable. At the end it is just a login box, those nasty things that a developer needs to implement multiple times in his career. But it causes agony to the users if done wrong!

Second, the misusage of software might actually give a hint about the absence of certain functionalities. Happened to my team, when we introduced the possibility to assign restaurants to a franchise. It didn’t take long and franchises like „normal“ and „no contract“ or „testing phase“ have been introduced. While someone else needed this feature to group chains together to create one invoice others saw their chance of finally grouping restaurants to query them more easily for statistical purposes.

Third, never assume, test! Assumption is the root of all evil and you should never expect that a customer has no other choice but using the software this way. Test the shit out of your platform and you will be wining about that customer tend to like the pink buttons more than the green ones, even though the latter ones have been suggested by a UX guru. Don’t worry about it, behavior isn’t rational most of the time and so you are incapable by definition to release something which you have scientifically proven to be understandable and working fluently. No chance! Do A/B testing and react upon the results.

Fourth, don’t belief your customers – Ok … ehm what? In the last paragraph I stated clearly to go out and react upon customer behavior. And now I am urging you to not trust them in the first place? Sure … no really this makes perfect sense, because you should never be biased or blinded because of numbers. Those numbers might lie and give you a wrong feedback, so always revalidate and take a look on the long term impact once you have changed your buttons to shiny pink because of the success of the last A/B test on mothers day.

Last but not least, don’t be frustrated about that situation, think about it from another perspective: Users are using your software, that is something good. Just accept and deal with the weird user behavior as valued user feedback, even though you need to interpret it. Listen to the noise of the crowd!

Disclaimer: I am the IT Lead of Takeaway responsible for lieferando, but those thoughts and fails are my own!

To sum it up – we screwed up and we were so confident to succeed.

But let’s start from the beginning. We planed a big marketing campaign at lieferando that created a big buzz in all the social media platforms and we were aware of a big impact on our servers. Since our system is designed this way to handle quite a big load with far less servers than our competitors we were confident we will handle the load and looked forward to prove everybody wrong about their premature statements about us getting run over.

But we weren’t ignorant, we planned load testing, prepared backup plans, tested fallbacks and simulated server outages. We thought we were ready, but apparently we were not. So what did go wrong?