Featured in Architecture & Design

Monal Daxini presents a blueprint for streaming data architectures and a review of desirable features of a streaming engine. He also talks about streaming application patterns and anti-patterns, and use cases and concrete examples using Apache Flink.

Featured in AI, ML & Data Engineering

Joy Gao talks about how database streaming is essential to WePay's infrastructure and the many functions that database streaming serves. She provides information on how the database streaming infrastructure was created & managed so that others can leverage their work to develop their own database streaming solutions. She goes over challenges faced with streaming peer-to-peer distributed databases.

Go Fourth and Grow

Yesterday marked the fourth anniversary of the open-source project. (Go1 was released in March 2012; the fourth anniversary is calculated from the initial project creation). Go was created by Robert Griesemer, Rob Pike and Ken Thompson at Google, joined by Ian Taylor and Russ Cox before it went open-source. From the FAQ:

Go was born out of frustration with existing languages and environments for systems programming. Programming had become too difficult and the choice of languages was partly to blame. One had to choose either efficient compilation, efficient execution, or ease of programming; all three were not available in the same mainstream language. Programmers who could were choosing ease over safety and efficiency by moving to dynamically typed languages such as Python and JavaScript rather than C++ or, to a lesser extent, Java.

Go is an attempt to combine the ease of programming of an interpreted, dynamically typed language with the efficiency and safety of a statically typed, compiled language. It also aims to be modern, with support for networked and multicore computing. Finally, it is intended to be fast: it should take at most a few seconds to build a large executable on a single computer. To meet these goals required addressing a number of linguistic issues: an expressive but lightweight type system; concurrency and garbage collection; rigid dependency specification; and so on. These cannot be addressed well by libraries or tools; a new language was called for.

Go's most recent release was 1.1, released in May 2013. Go 1.2 is expected to launch later this month, and a compatibility document sets forward the compatibility of future programs with Go 1, suggesting a forward-looking approach to compatibility, something which other compiled languages lack.

Since its initial release, a number of companies have started using Go, including Docker, a tool for packaging applications in lightweight containers; Packer, a tool for automating the creation of machine images; Bitly's NSQ, which provides a distributed messaging platform; Canonical's JuJu, which provides an infrastructure automation system, and many others. Even the Free Software Foundation are talking about replacing Java with Go as a default language. The Docker team have given a presentation on why they chose to use Go.

The main benefit for devops people is that Go programs are always statically linked into a single executable which can be deployed onto the target platform. As a result, the system does not need to have any additional binaries pre-installed or has any concern with running different versions side-by-side without conflict. This compares favourably to packaging systems like Linux and Java which have their libraries spread out over multiple locations and require either a specific setup in place or a resolver to build a path ahead of time. (The downside is that common libraries are repeated in the static binaries of all programs, but disk space is cheap. Memory pressure on servers, which would otherwise be able to load a single image of the library is perhaps a more realistic concern for systems with many processes running, but the devops tools tend to not run for any length of time.)

The main benefit for high availability and multi threaded programs is the use of Go's goroutines, which permit multiple threads of execution to serve incoming requests in a way similar to Erlang or Occam. All three are based on Hoare's Communicating Sequential Processes or CSP. Instead of having to deal with memory and threading access, CSPs each have an incoming queue of messages/tasks, and can in turn send those messages/tasks to other CSPs. There are blocking and asynchronous operations and each (blocking) CSP can have a return value typically sent as a return message to the originating CSP.

Cloud Foundry has recently moved from Ruby to Go as its command line tool, to avoid any particular pre-requisites to use the command. In fact, it is now a lightweight tool for communicating with back-end services, and instead of the client performing complex validation routines it now communicates with a back-end service to do the heavy lifting. This means the tool now no longer needs to be updated as frequently, and is smaller as well.

With Go 1.2 expected in the upcoming weeks, there is one change to the language in that three-index slices are now possible. Where before a a[1:10] expression would slice out the elements from the array a from 1 to 10, now a[1:2:10] will slice out every other value. A minor clarification on where nil is used will cause a panic if dereferenced. Other changes are listed in the release notes.

Go's critics point out that the language uses an older return-code style of programming instead of the more widely adopted exception based ones, and that unlike Erlang (which as a functional language has no mutable state), goroutines can pass maps around which are unsynchronized by default and can therefore lead to memory corruption. What is clear is that Go applies well to particular niche use-cases, and whatever its comparisons to languages like Java and Erlang, Go can be seen as a lightweight and perhaps safer alternative to raw C.

Community comments

Three-arg slices are different in Go

Your message is awaiting moderation. Thank you for participating in the discussion.

Three-argument slices are a little different in Go vs. other languages. In Go, slices have a capacity, basically the number of items that memory's been allocated for. (append() allocates a new, larger memory region when the capacity would be exceeded.)

Three-argument slices in Go 1.2 will return a slice header with the capacity field artificially reduced, so that code that receives the slice can't write outside a specified region of the underlying array. That's a mouthful, but bottom line, it's useful for custom memory-management code and for improving code safety in general. The release notes say more (I'd link, but putting URLs in comments seem to get them filtered).

1.2 also includes: performance improvements in the standard library and for networking on Windows and BSD; changes to goroutine scheduling; and tools for saving and loading objects in memory as text, JSON, or binary data.