Macs, Modularity and More

QCon London 2014 Day 2

Day 2 at QCon started off with a talk by Tim Lister (co-author of
Peopleware). One of the great things about QCon is that you get to meet
interesting peope from all walks of life, and I had the pleasure of
chatting with Tim at drinks the night before.

Forty Years of Teams

In 1987, the book “Peopleware: Productive Projects and Teams” was published,
written by Tim Lister and Tom DeMarco. The third edition was published in
June 2013, and it’s a book which has become a standard for understanding
team dynamics.

Tim’s main observation was that most problems with projects are not a result
of any single or sudden change, but of problems with people. He used the
phrase “Dead Fish” to identify projects which were clearly identifiable as
failures for years before the ultimate fail point, but despite being common
knowledge the project is kept alive. Economic interests, power plays and
a need to meet bugetary quota are some examples of where clearly failing
projects are kept alive to achieve side-effects other than the project’s
success.

He mentioned the benefits of working in a great team – “it’s great to know
your colleagues have your back” was the phrase he used. Working for and
with people smarter than you can result in successful delivery and learning;
on the other hand, “if you don’t have the trust of the team or you aren’t
excited to go to work then you should use your feet.”

Finally he inspired many by talking about software and being one of the
biggest changes in the lsat 50 years, and how conferences such as QCon are
about moving an idea from one thick skull to another. He also suggested that
writing is a great way to distill ideas and opinions (which reminds me, I
need to blog more …)

Maneuverable Architecture

Michael Nygard talked about
maneuverable architecture, and started by talking about maneuverability
based on the investigation of
John Boyd
who looked at aircraft performance and how quickly they could respond in
dogfights by converting kinetic energy to potential energy and vice-versa.

In addition, he proposed an Observe, Orient, Decide, Act loop
(also known as OODA)
which is the process of deciding what to do; specifically, by speeding this
up or reducing the time taken to perform the actions can result in being
able to react faster.

The same thing can happen in a business context, by being able to reduce
the design/implement/test/release cycle. By being more agile (or maneuverable)
it is possible to reduce time to market and thus costs.

One of his observations was that immutable data is easier to share and scale
than mutable data. By using identifiers consisting of hashes of the content,
the state of a system can be externalised into a list of identifiers. This is
similar to how Git works in terms of referencing content. His example was of
a shopping cart, with an identifier based on its content, and then a state
reference from a user to point to the current version of a basket (referenced
by its ID). By namespacing IDs it is possible to avoid content collisions
based on different types.

This allows a system of micro services (aka μservices) to be built and
a standardised API to be able to interact with them. By layering the services
together it is possible to implement security or filtering to the requests.

The argument is that by decomposing services and making the feature an
emergent property of the system instead of a hard-coded one (for example;
a process which periodically calls a URL or a service that sends a mail,
instead of a single process which walks through and mails based on content
of the server).

APIs and Platform as a Service

Paul Fremantle talked about APIs and
platform as a service (PaaS) as being the next stage in the evolution of
virtualisation. Instead of VMs and OSes being virtualised (Infrastructure
as a Service) the next layer up (Platform) is being virtualised. An API is
like a virtualised service itself; by providing a connection point between
two services. He also linked to
A company without an API is like a computer without internet.

Paul also observed that having an API and a way of monetising that API is
necessary for being able to ensure growth. He used Twitter as an example
of a service whose API got massive growth but without an initial monetising
plan, which then had to be retrofitted. But even with a monetisation plan in
place, having a free sign-up for developers (or at least access to a developer
or test environment) is a way of ensuring that developers can get started
with as little friction as possible.

To build an API-first system, start with developing an API and ensure that
all clients go through that API. That way, the system’s contents is exposed
via the same API that everyone uses, which means that there aren’t first-class
versus second-class applications. In addition, as new features are added to
the API then all clients can take advantage of it.

Migrating to Micro Services

Adrian Cockroft talked about his
experience of migrating Netflix to micro services (whilst wearing Google
Glass). His key enabler is that speed wins in the marketplace; by skating to
where the marketplace will be you can out-innovate the competition.

To do this, reducing friction from product development is the key. Automating
everything and trusting developers allows them to innovate without needing
meetings, approvals, sign-offs, cross-checks, or anything else which gets in
the way of the desired result – providing functionality to serve the business.

Importantly, by focussing on what you are good at (and what’s core to your
business) you can let go of things that you aren’t; including using open-source
tools instead of trying to fork/wrap your own APIs, and making the non-secret
sauce parts of your environment as open-source applications. Doing so increases
your reputation of the company – your company’s repositories on GitHub is
its on-line resume;. By licensing permissively (Apache, MIT, BSD etc.)
you trade-off control for ubiquity.

By doing this, Netflix were able to take full advantage of Amazon’s cloud
infrastructure, and provisioned 192TB of SSD storage based on a hallway
conversation in 20 minutes, and within the day had performed load testing
across two data centres with replicated content totalling 18TB of data
replicated in less than 5 hours between regions, and then load testing with
less than half a second delay in replication between the east and west coast.

Typical reactions to the prior Netflix talks:

You guys are crazy! Can’t believe it (2009)

What Netflix is doing won’t work (2010)

It only works for Unicorns like Netflix (2011)

We’d like to do that but can’t (2012)

We’re on our way using Netflix OSS code (2013)

By reducing all friction in the decision cycle, developers can innovate
faster and get changes into production.

In Netflix’s case, they achieved this by automating their deployment and
testing processes, so a full stack could be provisioned on the fly.
By building microservices, and having each developer team responsible for
their own services, it is possible to build a complex ecosystem from
modular parts. These services can be versioned so that all calls are using
a fixed version; this allows a production version of the service to be used
whilst test versions are used by a sample of users or for testing live in
production.

The recommendation is to provide solid client APIs for connecting to services
as well as the service itself; this reduces friction in the users who need
to connect to the service to interact with the runtime. Adrian suggested that
MongoDB was successful over Cassandra because the former had plenty of good
drivers in many languages, although with investment the Cassandra service
may be better in different circumstances. Using non-blocking calls with
Reactive Java and using Netty for non-blocking IO gave an increase in
performance without being overly concerned about threading issues.

Lambdas and Streams

Simon Ritter talked about Lambdas
and Streams in Java 8, due out at the start of EclipseCon 2014 in a couple of
weeks.

For those that aren’t familiar, the Lambda expressions will be able to provide
a similar change as inner classes, allowing in-line functions to be assigned
to a Single Abstract Method (SAM) type. These will actually be functional
types rather than instances of classes, so they don’t have this.

1

Runnablestuff=()-&gt;System.out.println("Hello Lambda");

He also covered the new streaming API which allows a sequence of collection
elements to be processed in a stream, much like an iterator allows elements
to be processed. However, by moving this into a stream it’s possible to perform
stream combinations such as map, filter and flatMap to construct
different streams on the fly. Importantly, these streams are lazy and so
aren’t evaluated until the end of the process (the terminal condition) such as
sum(), collect() etc.

Java 8 is shaping up to be the most significant release since generics were
added in Java 5. Although not as functional or flexible as languages like
Scala, it will be a great help to developers who continue to use Java.

Back to the Future

Emma Langman explained why my train home to Milton Keynes was delayed two
weeks ago. Time for a drink.