Thoughts from Technology Artisans

Crafting the Code

As we all know there are a lot of things going on all the time in our field. I am now sharing some thoughts that were first brought to public in ClojuTRE conference in September. If you weren’t there, don’t worry. Youtube can get you up to date about the speeches. In ClojuTRE we published that Siili is starting a new kind of business solution around Clojure. That is what we did. Now .core has taken its first steps as independent business unit within Siili.

The concept started to form at the beginning of May and after some chewing, we have stated guidelines to our formed group. “Simple made easy.” That’s it. When you work with software you might guess that this isn’t easy at all. The idea behind this is related to modern day craftsmanship and our aim is to make the best possible solutions for complex problems.

So as you probably guessed from the first paragraph .core is about Clojure. Why we chose Clojure? We think that the simple made easy concept is best reached using Clojure. All Turing complete languages are capable of solving computational problems. I’m not denying that. But what we do is best reached using Clojure.

There are many interesting things coming at .core in 2019 and we are hoping to share these things as they come. Keep an eye out for this blog and of course Siili official website. To get more information about this thing I strongly recommend you to drop by this years first Clojure meetup hosted at Siili.

In Tampere we have a saying “aika paljon on virrannut vettä Tammerkoskessa”. I guess that could be roughly translated “a lot has happened since”. And I am pleased about the direction things are heading.

I visited the ClojureD conference in Berlin on 24.2.2018 during some cold winter days. I went with my colleagues to gather some insights and listen to interesting talks on different subjects around Clojure. As a result I decided to challenge myself (and tests I write), inspired by Torsten Mangners good presentation titled:

Writing tests that suck less – “What” vs “How”.

Disclaimer: This post takes no credit whatsoever on the subject or claims to be a best way of doing things. This text relies on Torsten Mangners presentation but there is also some personal interpretation.

The Gauntlet

As Torsten reasoned that writing pure unit tests in Clojure (or any other language) is usually a small task. You invoke your unit with inputs and assert the results. Tests ideally document our software (as an added side-effect of verification).

Let’s see a simple, self-documenting unit tests.

;; the implementation (the unit, the HOW)
(defn add-numbers
[numbers]
(apply + numbers))
;; the test that should describe to a (technical) reader WHAT we are testing
(deftest add-two-numbers-together
(testing "If two numbers can be added together"
(is (= 3 (add-numbers '(1 2))))))

So it’s pretty straightforward, we don’t need to know the HOW or any other implementation details and because we are testing a pure unit with no dependencies we don’t need to prepare anything obfuscating and arcane.

Writing tests should not be an effort for a developer – but it starts to feel so when we run out of pure functions and go over to integration testing. It shouldn’t be too hard to write the actual tests – we just end up with a bunch of namespace_test.clj(s) files that can end up being many times longer than the actual code including a lot of HOW:s. Arcane and magical ceremonies, where we roll out mocks, stub and whatnot – finally ending in one or a few strange assertions.

If you jump into a project and find out that the documentation basically is the source code, you would be glad to find out that the tests would clearly tell us WHAT they test and prove. We don’t want to investigate and reverse-engineer the HOW part. We could just read what the software internals in it’s current state is documented and supposed to do.

At some point in time, as a software developer, you would need to know the HOW but that’s more of the part of writing the implementation (and the tests).

Throwing The Gauntlet

Thus I challenge myself to separate the bloat of HOW out my tests that repeatedly ended up looking like this:

This test mocks two successful HTTP calls, invokes the handler (entry point of the serverless function) and performs some assertions.

For those who are curious about what’s going on in here – this is a test for a serverless REST API handler in a Slack Bot backend. It’s written in ClojureScript and runs on Javascript function runtime on Azure. See Siili Solutions / Hedge on GitHub – a serverless framework to deploy ClojureScript functions on Azure and AWS.

This test checks how our serverless API responds to clients if it succeeds when contacting the Slack API.

So is this test complex? Maybe not too complex. But when you test more complicated logic it probably will be. Also there’s lot of potential arcane ceremony that doesn’t add to test readability like async done, (done), go, <! that is purely about the HOW in my tests. And those things just repeat over and over, multiple times per test namespace.

And my mocks are just a different combination of successes and failures, so they were just wrapped into their own functions.

Conclusion

You could still reason that this could even be more simplified. Macros could help me to get rid of even more noise. But in the end of the day, Torstens presentation at ClojureD made me more motivated about trying to write tests that suck less. Or die trying.

In the end of February five Siilis took off from Helsinki to visit Berlin. ClojureD conference to be exact. Two tracks, plenty of talks and focus in Clojure. This is an overview from the talks that we visited and because Joy Clark had awesome notes on talks too I asked permission to share her notes here as well. Big thumbs up for the visual content!

Teaching Clojure

For me the first talk of the day was about teaching. We have ran our junior program with Clojure for two years as we speak so new views and how others find teaching in this domain was interesting. Mike Sperber shared his point on systematic thinking and how it affects the way we build software.

Maria: a beginner-friendly coding environment for Clojure

Next talk was about Maria by Dave Liepmann. A simple really beginner friendly place to learn fundamentals in Clojure. In his talk there was quite a few demos on how fast one can get on with coding.

Fast results are important when you start something new and Maria offers fast response to your coding and shows results immediately. Of course when you start doing bigger and more advanced stuff your pace of getting things done and ready will be different.

Writing test that suck less

Last talk before lunch was about testing by Torsten Mangner. This talk dived a little deeper to the fundamentals of testing. Talk began wit a confession and gave us the baseline.

“Writing unit tests in Clojure is easy, since testing pure functions is trivial. But the more high-level our tests become, the more they have to deal with the state and side-effects of your application. It becomes harder and harder to write proper tests. And worst of all: they are harder and harder to understand and to maintain, greatly diminishing the value of those tests as a documentation of your software.”

A Dynamic, Statically Typed Contradiction

After lunch we had the pleasure of diving into mathematics. As Andrew Mcveigh talked about lambda calculus and its application in a “Hindley-Milner“ based type system and how it can be mapped onto Clojure. This solution can also be used to “type-chec” a subset of Clojure code.

Unfortunately my attention was to fully hang on with the maths and our ways with Joy had departed so, no good pictures from this talk. Sorry!

Onyx, virtual machines, jokes and afterparty

Next up for me was Vijay talking about Onyx and its uses for data crunching. An overall look for the uses. Masterless systems, stream and batch processing, flow control and principles of Clojure.

After coffee we got the pleasure of seeing a walkthrough how to implement Clojure on a new virtual machine. Heavy stuff and a great idea.

Last official talk before the lightning talks (I had to skip those because I was hungry and needed a Döner 😉 ) was called defjoke. Live coding a fully compliant spec for humour.

All talks were taped and are available soon in http://clojured.de/ check it out. Other great photos and one-liners can be found in https://twitter.com/clojuredconf

Thanks from Siili to great organisers and speakers. Hoping to see you soon in other adventures!

I remember last year binge-watching the Serverlessconf recordings online thinking “If there is one conference I want to go this is the one!”. The talks were so utterly good that I felt bad for not having witnessed it live. I was looking for the coming Serverlessconfs but they seemed to be coming nowhere near me. Then suddenly they announced the event in Paris and I went crazy. Let´s do this!

On day #1 there were multiple workshops on various technology stacks. AWS, Microsoft Azure, Google Cloud, OpenFaaS and IBM Cloud Functions.

I chose a workshop that was organized by A Cloud Guru where we built a Youtube like video service website with AWS API Gateway, Lambda, Elastic transcoder, Google Firebase and Auth0.

The present and future of Serverless observability | Yan Cui

The first presentation of the day was about the present and future of Serverless observability. Yan Cui was pleasant to listen as he had a very fluent presentation style and insightful observations. He also pointed out an interesting whitepaper Serverless Application Lens for the AWS Well-architected framework.

The document covers common serverless applications scenarios. I encourage you to read it to understand AWS best practices and strategies to use when designing architectures for serverless applications.

The next presentation was given by Subbu Allamaraju from Expedia Inc. Expedia is American travel company that owns some of the most popular travel sites like Hotels.com and Trivago. He approached the concept from an interesting perspective highlighting some points on benefits of the serverless paradigm and in contrast common counter arguments like today’s limitations and constraints, personal perspectives of cloud, portability, lock-in, cost etc. He highlighted how past habits, culture and legacy can really slow down adoption.

The presentation provided ideas on how to tackle these blockers on the journey towards a platform of the future, where functions and events fit, and how to prepare ones colleagues for that future.

Bret McGowen & Monika Nawrot showed demos around the Google Cloud offering. Monika explained how Google Serverless offering has grown over the years and had a Google Cloud Functions demo with thumbnail creation function, storage & logging. She highlighted also that there is a possibility to use a Function emulator for debugging functions.

Bret McGowen vent quickly through some of the best Backend as a Service (BaaS) offerings out there and then focused on the Google Firebase. He explained that the Realtime database was the first Firebase feature but there are many more built-in features like Cloud storage, Functions, Hosting, etc.

He also made a bold statement which people seemed to find quite amusing. “More serverless than serverless!” =) . It sounded like he was selling artificial sweetener by claiming “Sweeter than sugar” and I found it triggering my bullshit detector. Luckily the demos totally convinced everybody as he showed the Firebase realtime update capability. Apparently Firebase is also handling situations when a client connection drops. The data syncs instantly when a client gets back online. All the Firebase features can be implemented directly in the client-side code using an SDK. The sugar on the top was BigQuery that could query strings on pretty much the entire universe in just mere seconds.

Kassandra Perch talked about visualizing Serverless architectures and what does a healthy serverless app look like. Her demo included inspecting for example error rates and counting cold starts. She showed how to include IopipeLib, trace and profiler in a lambda function & how to use those. Interestingly the IOPipe wraps the whole lambda handler. The recommended tool IOPipe has a slogan “See inside you Lambda functions” You can find it here: https://www.iopipe.com/

Frédéric Lavigne introduced IBM´s Serverless offering and clarified the difference between IBM Cloud Functions & Apache OpenWhisk. OpenWhisk is open source and used in the IBM managed service. The IBM Cloud team contributes to the open source project. Runtimes include: Node 6, Node 8, PHP 7.1, Python 3 and Swift. Additionally, OpenWhisk can run any custom code put in a Docker container.

In the demo he defined a smiley face as SVG inside the Function code. This was enabled by the RAW HTTP handling setting. Unfortunately, there was some demo effect with API Gateway as he tried to show how to use Web actions and API Gateway. IBM Cloud Functions have some built-in event sources such as Cloudant, Message Hub and Github. Currently they are developing a composer for cloud function flow control.

The presentation included some interesting customer cases such as the DOV-E Hypersonic-based Mobile Device Communication. They had Coca Cola TV advertise trigger a hypersonic message on the viewers phone. Viewer could not hear the communication, but they received a message on the phone asking whether they would like to drink some Coca Cola right there and then.

Event specifications, state of the serverless landscape, and other news from the CNCF Serverless Working Group | Daniel Krook

Daniel Krook had a very informative talk about Event specifications, state of the serverless landscape, and other news from the CNCF Serverless Working Group. CNCF stands for Cloud Native Computing Foundation which drives the adoption of the new Serverless computing paradigm.

They have multiple projects under their wings such as Kubernetes, Prometheus, Fluentd to name a few. However, currently there are no serverless projects. They consist of four active CNCF Working groups (Continuous Integration, Networking, Storage, Serverless) and have now a whitepaper available for review. It is hosted in Github for collaboration purposes and they are accepting pull requests to keep up with the fast phase of development in Serverless. Please find it here: http://www.github.com/cncf/wg-serverless

Accelerating DevOps with Serverless | Michael H. Oshita

Michael H. Oshita talked about Accelerating DevOps with Serverless. His excellent demo showcased how to provide automated branch testing infrastructure for Devs without burdening Ops. Technically the setup uses Github hooks that trigger Circle CI, provisions using Terraform templates and Lambda functions are orchestrated with Step Functions. The infrastructure is created on pull request and destroyed when branch is merged and deleted. See the ECS Deity -project here: http://www.github.com/ijin/ecs-deity

End-To-End Serverless | Randall Hunt

The most entertaining talk of the day was a live coding session with technical evangelist Randal Hunt. He´s opening line was surprisingly in French and the goofball show made everyone crack up. He also asked audience for suggestions on what to build there on the spot in front of the audience. Not something most presenters would want to try I imagine. The randomness didn´t seem to bother viewers and he did manage to finish the live demo just in time which made the organizers happy. The energetic personality and laid back attitude with relevant content was a great combination and I hope to see more of this in the future. If there is one recording you should watch, it would be this one in my opinion.

With Great Scalability Comes Great Responsibility | Dana Engebretson

Dana Engebretson had a pleasant talk telling an interesting story about her journey to Serverless architecture for analytical purposes. Humorous croissant baking tips and to-the-point analogies made it inspiring and entertaining.

Overall the talks were versatile and excellent. There were also some less technical talks but nothing really outside the Serverless focus which was good as it didn´t seem like the organizers had struggled finding relevant and good presentations for the event. I recommend getting involved in your local cloud computing communities. Serverless development is super-efficient and enjoyable so if you haven´t tried it I suggest you do so without further due.

The author Jouni Leino has 15 years of experience in IT. He lives in Stuttgart Germany working as a Field Application Engineer in the Automotive industry. As Serverless Competence Lead he´s mission is to guide customers and colleagues towards a more cost effective and productive future.Follow in TwitterLinkedIn

At the start of my Siili career as an apprentice, our group was introduced to a mysterious tool: Hedge. Our apprentice project was to build an internal tool using Hedge as a cloud platform abstraction library and a deployment tool. After our apprentice program had come to an end, I was offered an opportunity to continue the development of Hedge in an internal project. Because we had already used Hedge I had some experience as a framework user. The task I was offered was to implement basic Amazon Web Services(AWS) support into Hedge.

What is this Hedge you are talking about?

Hedge offers automated serverless function code generated from common ring compatible handler methods, libraries for various levels of abstractions, and a set of common commands to build handlers, create artifacts, and to deploy created artifacts. Hedge is an open source software and is available in our github.

“Hedge is a platform agnostic ClojureScript framework for deploying ring compatible handlers to various environments with focus on serverless deployments.”

Back to my journey with Hedge development

The scope of platform agnostic in Hedge was uncertain for me when I started the development. First, I implemented a simple feature parity for AWS code creation and deployment with Serverless Framework. Later the definition of the platform agnostic, roadmap, and context of Hedge became more clear.

When I started working with Hedge I had little Clojure experience and almost no hands-on experience with AWS. For AWS deployment I had checked how other tools do deployment and had chats with my peers. These information sources gave good advice on how to handle deployment.

Why Clojure(Script)? ClojureScript in backend?!

Clojure(Script) is popular and it is in high demand in Finland and at Siili Solutions. Using ClojureScript gives access to ClojureScript and JavaScript libraries. The functional paradigm has steep learning curve, but after learning the basics it is easy to realize that immutable data fits well with serverless handlers. Multithread-safety adds more security if one of the supported clouds is re-using processes with multiple threads.

One of the Clojure build tools, Boot, is also extensively used with Hedge. Boot tasks are great for creating and chaining commands. It is fast and easy to develop a set of tasks which for example build input files, create all artifacts, and finally deploy artifacts to cloud. Then those tasks can be combined into one large task which does all with one command or uses small tasks to store artifacts into disk and later deploys artifacts to cloud.

Why platform agnostic framework?

Following code snippets reveals the first problems with current serverless platforms:

Above code snippets are simple Hello World serverless function handlers but the problem is already visible.

Handler function signatures(line 3)

Logging(line 4)

Output(line 5) handling(line 12)

Exit condition signaling(line 12)

all are different.

The main goal of Hedge is platform agnosticism: once the code and configuration has been written for one cloud platform, it is re-deployable to another cloud provider by changing deployment command and a few configuration directives. This improves code re-usability and limits the risk of being locked with specific cloud provider.

To make the implementation easier, Hedge supports only a small common subset of provided cloud features. Limiting number of supported features might be a risk and might lower acceptance rate amongst developers. It is unknown if developers will use a framework which allows using only a small fraction of cloud features. The small number of features definitely narrows down the creativity of developers and might also limit usage of Hedge for some projects.

Our implementation

Hedge adds abstraction layers for function handler code, infrastructure and deployment. Following code snippets are examples of abstraction layers of current development version.

Above snippet is example how to run Hedge. Help command lists all available command and simple aws/deploy or azure/deploy commands can be used to deploy project to selected cloud.

Deployment is abstracted with boot tasks. Identical set of command can be used for build, artifact creation and deployment.

Hedge HTTP function handlers resembles Ring handers which is well-known abstraction for HTTP in Clojure. Hedge reads user-supplied configuarion files and serververless function hander then creates cloud-specific wrapper code between a user supplied handler function and cloud’s native handler entry points. The wrapped code handles differences between cloud providers and in the future, there will be libraries for the rest of code abstraction.

Hedge is currently under heavy development and some of the listed features are still on roadmap.

Lessons learned

During this project I had to learn how create Cloudformation templates and stacks from template. Cloudformation templates for lambda functions with API Gateway endpoints first seemed overwhelming, but AWS SAM simplified template creation. AWS SAM is still a young technology and I did not find any big projects using it.

I had to find one important piece of information from StackOverflow. Cloudformations templates using SAM must be deployed using change sets. Luckily it was documented there, since otherwise I probably would have spent a lot of time debugging a feature which was not supposed to work.

As the documentation of SAM is still vague and mostly in GitHub only, I personally can recommend using SAM for serverless Cloudformation templates.

Both homoiconic nature and functional side make clojure very distinctive language. They are both inherent in making clojure what it is and i would not take one before the other.

As we have blog posts covering the reasons for FP here i’m going to take FP as a paradigm more or less granted in this post.

Why do i see clojure as a great language in the context of functional programming?

Learning and unlearning

First of all it’s hard to learn a programming paradigm in a language that does not embrace it. A wrong tool drives you to make wrong choices. It leaves you without actually learning the paradigm. It is by far the best thing to do to learn a paradigm using tools meant for it. Eric Normand wrote a good blog post about this problem.

To understand the benefits of FP, you have to use languages meant for it. Languages like Haskell (and by extension Elm and Purescript), Clojure, F#, Scala, Other lisps, Erlang, or OCaml.

I take it that you’re not interested in why not Haskell family, F# or OCaml, but i’ll anyhow go through that part. Most of these languages have very limited use in actual business right now. Currently Haskell and OCaml don’t have any place in programming for business. Which is a pity and no fault of the languages . Elm is currently used in front end and as such is still lacking in backend. Purescript seems to be the most versatile and available of these. It works well both in front and on node.js making it a viable language used in professional fullstack dev.

Scala then is a very solid language running on an widely accepted platform: the JVM. It is a very good all around language, there is no doubt about that. It also has a very good yet complex type system. System that can guide a good FP programmer to discover things about their domain. There are yet few things making it lacking in comparison to Clojure.

Main problem in learning is unlearning. The familiar look of Scala drives people to not learn it. It’s easy to hack something with it but actually using Java with different syntax. Learning first Clojure leads to faster understanding of Scala itself. Adding to this rudimentaries of Haskell helps even more. Learn You a Haskell for Great Good! is a good resource for this.

The unique way of thinking about problems and the language that describes them. This way of thinking is very valid and a good tool to learn for any programmer. Biggest reason for this is the homoiconic nature of the language. The great ability it brings in metaprogramming.

Scala is currently only viable for backend use. Unlike scala Clojure has many supported platforms. It supports JVM, JS including node and browser, and CLR. All which acceptable platforms for business use. This makes it possible to share the same code in front end and backend. With it you can make a consistent whole. Creating front end components that are in sync with their microservice counterparts. In scala this is not possible as Scala.js is lacking in actual production grade stability.

So yes scala is an interesting language but not the best to learn FP. Especially so for Java developers.

Motivation as a driver for effectiveness

It’s actually quite hard to have valid measurements for coder efficiency. Yet one thing we do know. Study after study has found that developer motivation is elemental to productivity.

With unenthustiastic devs its best to inspire them. One of the main benefits comes from the fact that clojure is easy to use and makes coding actually fun again. This is a quote you do hear quite a lot from clojure devs.

As this is about inspiring people, you should go with what is good and nice and feels good and brings joy to you. As then you’ll be more inspiring.

Final words

It also goes down to the provable fact that Imperative Paradigm is more complex than simpler FP. Inside FP clojure is a very very very easy way to learn it. Much easier one to reap the rewards than most others.

If you have a bunch of coders struggling with imperative stuff. And you want to get the benefits of FP as fast as possible. Clojure is the fastest one to learn.

Choosing a functional paradigm language has been a hot topic for a while. Functional vs. Object Oriented Paradigm is a constant discussion topic within the developers.

Before thinking the benefits of FP, the working environment must be suitable for modern work

The key of success at the management point of view is not to micromanage. Let the team choose their working methods and tools. The project is always a compromise between time, features and given resources. It’s all about how the project management sees that triangle.

Every tool and way to work has their own place. Telling the developers HOW to do work instead of WHAT to do is always a problem. It kills creativeness and get the authority outside the dev team. Too often the author is someone who doesn’t know about the actual developer work.

Treat the professional developers with this Sledge Hammer principle: “Trust me, I know what I’m doing”. Developers should know exactly how the program works, because code is the most detailed specification of the solution. It’s all about communication between the people – and between the human and the computer.

When the environment is suitable for working, then we can discuss about the benefits of FP

So why should I choose FP instead of OOP/imperative? What benefits it would bring the table? Here’s some thinking I’ve done in the past years based on my experiences:

Functional programming makes especially list handling a way easier. There are effective functional languages like Clojure or Scala. Another option is to use functional libraries like Ramda or Lodash. I like flexible code, so I prefer using scalable languages, like ES6, Scala, Java8 or C#/LINQ. With OOP and FP combined I have more options available. It’s also a safe choice when there ain’t so much pure functional programmers in the team (yet). Scaling the Object Oriented code with functional flavour is understandable compromise with functional averages like me. Actually I’m on my way of using only functional languages.

Here’s some benefits I found on functional paradigm which could save some money

It’s stateless and then also immutable, so no side effects

Object Oriented Paradigm is all about sharing the lifecycle between the objects. OOP makes problem solving often too complex, and complexity costs. Mutable state makes side effects possible, so code is harder to test with a full coverage. Developing with FP forces to code small functions. Simplified syntax (especially with pure FP languages) gives more time to think the actual business logic. Focusing on one thing at the time is clever.

When the state is immutable, it’s way easier to scale up and still control the whole

When programming high capacity systems, it’s crucial to design the software architecture to be scalable for parallel processing. If the program isn’t handling any state inside the runtime, it’s way easier to scale by using micro services or FaaS container platforms like AWS or Azure. Then the runtime isn’t blocked to serve only one client for a long time – there’s more time to handle many short tasks instead of a one big blocker.

So no more if-else-if-else or switch-case spagetti, or recursive for loop hell. Only function chains like map, filter or reduce one-liners.

It might be hard to transform the brain in the right mode after coding years with OOP. Functional code readability might cause many WTFs in the beginning, but – trust me – It’s worth of it. Like Twelve-factor manifest says “Execute the app as one or more stateless processes“. Start by one and try to write more if it feels right and suitable for the context.

Learning functional programming makes you a better OOP developer. You don’t have to be an expert, but it’s beneficial to learn something new. It’s hard at the beginning, but with the supportive team it’s possible to achieve.

Functional languages are simple by design when you let it be so.

Here’s a recap of my points:

Let the developers to choose the right tools and processes for building the high quality solution

As a team member, encourage the colleagues at least try to code with FP

Functional code is simple and easy to maintain. when it works, it works (no side-effects like in OOP). Broken code is easier to recognise and refactoring is less risky operation

Data must persist outside the code (like in Redux architecture). It shortens or eliminates debugging marathons. Less risk for wasting the time hunting the bugs from the production environment