Deathowl's Nest

Moderately humble toughts from the land of Devops and tools.

After almost 2,5 years of never ending fun and hard work, at the end of February I’ll leave one of the greatest tech companies of central Europe, Ustream.

Why?

I haven’t been fired, and I haven’t gone insane neither, I just had a great opportunity, to work at a startup from almost the very beginning. And my good old ego wouldn’t allow me to let this opportunity slip away. I think at my age(I’m 23, doh, so old[no, not really), you have to take some risk. I enjoyed every moment of this 2,5 years at Ustream, and thanks for them again, for being as awesome as they are. I Love you all guys, and I always will.

So, what now?

So, I’m starting to work at Endticket from March, as the Infrastructure guy of the company, and start to establish new and polish existing processes and workflows related to build, deployment, monitoring, alerting, uptime and so on, you know the usual stuff, in the name of Devops.

And what will happen to the open source stuff built at Ustream?

I won’t abandon any of my open source projects. I’ll remain to be the maintainer of Openduty, Specialist, and Errbit Proxy. Plus, expect some Slack related hotness to arrive until the end of February, because I want to see that stuff open sourced, as long as I have the chance to do so.

Misc

I already have some pretty crazy and useful stuff in mind to build there, and the good news is: Everything not tightly coupled to the business logic will be open sourced, because of the previously mentioned Ego related craziness of mine. And open source is good for the company as well.

So I just came back from the sunny Barcelona yesterday, and I have to admit, Barcelona is az amazingly beautyful city. But more importantly Velocity is an amazing conference. This post will be a short recap from my point of view grouping my key takeaways from that conference.

Open source tools:

So my favourite topic, I’ve heard great talks about tools we already know and love, like Etsy’s very own nagios-herald, and some new ones i didn’t really hear about yet, but was totally amazed by, let this be just a short list, and check them out yourself:

OpenSpeedMonitor : An open source tool built atop of Webpagetest to mesure performance

I’ve seen some pretty awesome commercial tools, which were new for me and blew my mind as well.
Let’s progress with a list which one is not odered, because i can’t decide order based on awesomeness.

Ruxit What Ruxit does is the following: It has an agent on your box, and monitors your system, this is really common for a monitoring system, right ? :) But the mindblowing little added value comes here: They call it contextual alerting. So, it creates correlations between incidents, so when you get hit by an alert you can see what was the root cause, what was the problem that originated from that roout cause, and the other little failing checks that originate from the same root as well. Saves lots of time for on call guys. Awesomesauce, right?

Cedexis Cedexis provides free R.U.M solution, and a paid Load balancing service. What Cedexis does, is they collect the network data in they free RUM tier, so they have a global map of the network traffic. And based on that, believe it or not, they provide the best routing between your service and the enduser. DNS load balancing based on global metrics, well this is my reaction:

Technologies:

We’ve heard lots of great talks around various technological topics, covering containers, anomaly detection, an particularly good talk by Google’s very own Ilya Grigorik about optimizing TLS performance. Just WOW.

Theoretical stuff:

As I mentioned before, we had great talks in the topic of anomaly detection, and oh boy, fiddling with time series comes with glorious amount of deep deep math, the talk by Arun Kejariwal was heavy on math, but I enjoyed every moment of it. The other anomaly detection talk was by the guy, who is highly recognizable by his ever changing facial hair, Theo Schlossnagle, the summary of his talk can be found here. We also heard pretty good amount of talks around microservices, deployment, and uptime monitoring.

Linux Containers

Linux containers are a really hot topic nowadays, and it’s totally understandable why. It’s the ops equivalent of the microservice oriented code. And as so, it’s meant to serve as a backbone of a microservice oriented infrastructure.
There was a presentation, which one ripped apart the real meaning of linux containers, how they really work, and what docker, or LXC actually abstract away from the enduser. Slides
The other great tutorial in this topic was about Core OS and Kubernetes.Material at Github

Summary

To summarize, I learned a lot, networked a lot, met lots of nice people, and enjoyed the show maximally. I’m really thinking of going back next year as well.

This post is just my personal opinion, I don’t want to tell You , to finish or don’t finish university, it’s up to You to decide. Thanks.

So, it’s official now, I’m an university dropout. And actually I don’t really mind it.

Why?

You don’t have to have a CS degree of any level to be a great engineer.

Mathematics was lectured on a high standard, it was cool like this, but sadly, you can’t say the same about programming at all. We only had programming courses, because we had to have them (you know, CS faculty.).

lots of the material (approx. around 90% in Hungary, on ELTE) is basically useless to someone, who has seen some code before university. I’ve picked up that 10% of the material, that I’ve found useful, so I’ve wasted no time

Instead of the 90% useless material I can learn 90% useful stuff. For example Go, I’m into learning Go right now.

Since i work at my current place, I took 2 passive semesters, because I wanted to focus on the things i do there. I like to work on problems, and learn useful stuff a lot more, than to sit at a course and listen to a class about “How to write a class in Java” 4 hours long (it’s a true story actually)

I don’t want to state, that you don’t need deep theoretical knowledge to be a great software engineer, but you can learn the needed material on your own, maybe it will be a bit harder, but it can be done. Have you ever heard of coursera? I’m pretty sure you all did.

Experience is worth more than the diploma

Basically I had to make a choice, go to night course after work (giving up work was not even on my mind, I love it way too much), and give up on every spare time, pet project, social activity. Well you see my point,I think.
So that was it.

As an Infrastructure Engineer lately I’ve the lovely task to develop new, or select existing Open Source tools which fit our needs at Ustream, and even extend them to fit our needs better. Such tools have a massive potential to improve developer productivity in many ways, and I’m a developer, I want to be productive, and I want to help other devs to be more productive too. This is why I love it so much. I remember how astonished I was, when I used for example Vagrant for the first time. My reaction was an immediate WOW. It was a really cool experience, to be honest, not to speak of more “visually intense” tools like Graphite. After that , I went to Monitorama, and I still refer to it, as the most inspiring conference I ever visited. I fell in love with tools there irreversibly.
Now more than 1,5 years have passed since then, and now I can proudly tell, I’ve got my (at least partially) own tools.

The first ever tool i had the chance to work on was Errbit, which is an error logging and aggregation service, like airbrake.io . I really enjoyed working on it, and working with it, because it solved a real problem, logging errors can be hard.

Then came GitLab, which is a super awesome self hosted git repository management thingy, something like Github enterprise , just for free.

And then the totally own stuff : Openduty which is the result of the first ever Ustream Hackathon (see previous post).

After that I even had the chance to contribute to some awesome piece of code written written by one of my best friends working at Prezi called Changelog

And I’m still working on great stuff inside and outside the office. So yes, tools are awesome, and make life more awesome too, if they are used well. Automation makes life better, so:

There was a the first ever Hackathon held at Ustream a good month ago, this is just a short summary about my experiences, and how fun it was.

Long story short, i led a team named Call of Duty and we reached the 4. place out of 12 teams, so it was a pretty decent achievment, i’ve got to say.

The story of the Hackathon, you have an idea, you collect your team, and you get one day to make a working protoype based on your idea. We wrote an alerting and incident escalation application, much like Pagerduty, and it is free, so the name DutyFree was absolutely trivial.

Soon you’ll find the source code on Ustream’s github account, we just got to clean it up a little.
Here is a good photo taken at 4 a.m of the team.

So after 4 years i decided to move from Byethost(i also deleted my EC2 instance at the same time btw.) to Digital Ocean. I also decided to give up wordpress and give Octopress a try. It was easy to set up, and finally i have a nice static super fast blog set up. I was getting tired of having a box on EC2 to do stuff, and have another one to host my blog, now it’s all in one place and feels much better. And thanks for DigitalOcean for their awesome pricing, which is more than affordable.YAY.

Errbit and it’s hosted counterpart Airbrake are great tools to capture and track your Application’s Exceptions. The only problem is, the interface they provide is a restful API in which one you can post errors in a nice XML format. The only problem with this is http, or if I go deeper tcp. If your app has a really bad time , and gets flooded with exceptions (for example in case of cascading errors eg.: a database outage) your application could get into serious locks if you try to send all the exceptions directly. Yes of course you could use a queue to store exceptions, and process them asynchronously, or you could just do the whole thing via udp.

That’s why i started to hack on my pet project, really creatively named “Err-proxy” , which one is really an error proxy listening on UDP for error messages and forwarding them on the regular HTTP way to the Airbrake or Errbit server. It’s written in node.js (Yes, I’m coding in node, no thank you ,I feel all fine, it’s just the right tool for the problem), and inspired by statsd a lot. It’ll be in working state in a few days, and i’ll share the github repo here(It’s a private repository at the moment). Here it is: https://github.com/ustream/Errbit-proxy

So the talks, and the people i met on the first day were cool, but the workshops on the second day. Man, i mean really, it was the icing on the cake.

Collectd, Riemann, Dashing, Graphite scaling, and last but not least a little about Descartes. But my favourite one was the talk of Abe Stanway about Kale. My mind just blew up with the ideas how many things become possible and less sucky with an automated anomaly detection system. But i liked the Dashing talk also as much as this one, so i can’t decide. I’ve got a love about pretty dashboards and automation, get over it.