http://dominik.honnef.co/Dominik Honnef2016-10-23T22:08:51ZDominik Honnefhttp://dominik.honnef.cotag:dominik.honnef.co,2016-10-24:/posts/2016/10/harvesting_email_addresses_from_public_slacks/Harvesting email addresses from public Slacks2016-10-23T22:08:51Z2016-10-23T22:08:51Z
<p>Over the last year, Slack, the communication tool for teams, has been
becoming popular in the open source community, replacing alternatives
such as IRC and Gitter. This was – rightly so – surprising to the
people behind Slack. After all, Slack is designed for companies. This
is evident in the lack of features that open communities would want,
and the presence of features and design decisions that only really
work with a group of trusted people.</p>
<p>Most of these design decisions might only appear odd or slightly
annoying. One decision, however, makes spammers really happy: Since
Slack assumes that people in a team know each other, it sees no reason
to hide email addresses. This makes it trivial for spammers to join a
public Slack, run a simple line of JavaScript in their browser, and
get hundreds to thousands of active email addresses:</p>
<pre><code>&gt; TS.model.members.forEach(function(item){console.log(item.profile.email)});
b*************@gmail.com
1*****@gmail.com
j******@gmail.com
d*************@gmail.com
e*******@gmail.com
4*****@gmail.com
...
</code></pre>
<p>I’ve censored the email addresses in this blog post, but the real
invocation returns full addresses. I don’t feel particularly bad about
showing the code here because it is trivial to come up with and in
active use by spammers already.</p>
<p>Luckily, Slack has added a feature to hide email addresses. It took
them several months and countless requests, though that shouldn’t be
too surprising as – again – their focus lies on paying businesses, not
free communities. For a while now, admins of Slack teams have had the
option to hide members’ email addresses. The instructions for doing so
can be found in their
<a href="https://get.slack.help/hc/en-us/articles/228020667-Display-members-email-addresses">help center</a>.
Unfortunately there are still plenty of large public Slacks that
haven’t enabled this setting, and so the spamming continues…</p>
tag:dominik.honnef.co,2016-10-22:/posts/2016/10/go-and-strip/Stripping Go ELF binaries is safe2016-10-22T01:16:28Z2016-10-22T01:16:28Z
<p>A common misconception among Go developers is that stripping Go
binaries – that is using the <code>strip(1)</code> Unix utility – is unsupported
and leads to broken programs. This misconception is perpetuated on
mailing lists and other forums without providing actual proof for why
it is supposedly unsafe.</p>
<p>This article will try to explain how this misconception came to be and
why it is not a real concern anymore.</p>
<h3 id="history">History</h3>
<p>The misconception hasn’t always been that. At various points in time,
stripping Go binaries could in fact break them. These cases were
always considered bugs and were fixed eventually.</p>
<h4 id="invalid-elf-binaries">Invalid ELF binaries</h4>
<p>The majority of issues were caused by slightly incorrect ELF binaries.</p>
<p><a href="https://golang.org/issue/261">Issue 261</a> – created in 2009 and fixed
in 2010 by <a href="https://golang.org/cl/994044"><span class="cl">CL 994044</span></a> – and
<a href="https://golang.org/issue/1242">issue 1242</a> – created in 2010 and
fixed in 2011 by CLs <a href="https://golang.org/cl/4639077">4639077</a> and
<a href="https://golang.org/cl/4808043">4808043</a> – are the two prominent
issues that existed with stripped binaries. In both cases, the ELF
binaries didn’t have the required layout or headers to make strip work
correctly.</p>
<h4 id="ld--s">ld -s</h4>
<p>Go’s linker has itself a flag to omit writing the symbol table and
debug information. This flag had bugs causing crashes as well. These
bugs were more recent, occurring in 2013 and 2015. However, the
flag and <code>strip</code> are not related and <code>strip</code> continued functioning
correctly during that time. Nevertheless, these new bugs reinforced
the idea that <code>strip</code> was still broken. Furthermore, these bugs have
been fixed as well. They were tracked by the issues
<a href="https://golang.org/issue/6245">6245</a> and
<a href="https://golang.org/issue/10254">10254</a>, fixed by CLs
<a href="https://golang.org/cl/13751045">13751045</a>,
<a href="https://golang.org/cl/10835">10835</a> and
<a href="https://golang.org/cl/11695">11695</a>.</p>
<h3 id="current-situation">Current situation</h3>
<p><a href="https://groups.google.com/forum/#!topic/golang-dev/ABppMOjYP6w">In early 2016, the question of stripping Go binaries was brought up on the golang-dev mailing list</a>.
Here, Keith Randall and Matthew Dempsky confirm that stripping ELF
binaries should already be safe, and if it wasn’t that it’d be
considered a bug. Russ Cox further confirms that it is working and
“has worked for the past five years or so”.</p>
<p>In an experiment at <a href="https://golang.org/cl/20584"><span class="cl">CL 20584</span></a>, Michael Hudson-Doile proves that
stripping Go binaries does not cause any tests to fail, with the
exception of an odd crash on Darwin, which does not affect ELF
binaries.</p>
<h3 id="conclusion">Conclusion</h3>
<p>To conclude: stripping Go binaries, at least on systems that use ELF
binaries, is safe and has been safe since 2011. Similarly, using the
Go linker’s <code>-s</code> flag has been safe for many years, various
short-lived bugs aside.</p>
<p>This article did not explicitly investigate the stripping of PE and
Mach-O binaries. However, there are no known issues with stripping PE
binaries, and stripping Mach-O binaries only caused an issue in the
experiment in <a href="https://golang.org/cl/20584"><span class="cl">CL 20584</span></a>. No issues in real-world use cases are known.</p>
<h3 id="addendum-another-misconception-about-stripping-go-binaries">Addendum: Another misconception about stripping Go binaries</h3>
<p>In addition to the misconception that stripping Go binaries breaks
them, there have been several related misconceptions. The biggest one
is that <code>strip</code> would make panic stacktraces useless by stripping
function names, file names and line numbers. This is not the case.
While <code>strip</code> will remove various useful debug information, it will
not affect how panics are rendered. This also implies that <code>strip</code> is
not sufficient to guarantee that all identifying information are
removed from an executable.</p>
tag:dominik.honnef.co,2016-10-07:/posts/2016/10/kube-cert-manager/Managing Let's Encrypt certificates in Kubernetes with kube-cert-manager2016-10-07T07:05:46Z2016-10-07T07:05:46Z<p><a href="http://kubernetes.io/">Kubernetes</a> provides a way of scheduling and
managing resources such as containers, storage and load balancers in a
highly automatic and easy fashion. It’s a prime example of
<a href="https://en.wikipedia.org/wiki/Infrastructure_as_Code">Infrastructure as Code</a>.</p>
<p><a href="https://letsencrypt.org/">Let’s Encrypt</a>, by its use of the
<a href="https://letsencrypt.github.io/acme-spec/">ACME protocol</a>, provides a
simple and automatic way of obtaining TLS certificates that are
trusted by all major browser vendors, completely for free.</p>
<p>In a lot of scenarios it can make sense to combine Kubernetes and
Let’s Encrypt and to not have to deal with creating or renewing
certificates by hand ever again. This is where
<a href="https://github.com/PalmStoneGames/kube-cert-manager">kube-cert-manager</a>
comes into play: it offers an ACME client that is integrated with
Kubernetes; requesting, renewing and deleting certificates as needed.</p>
<h3 id="features">Features</h3>
<p>kube-cert-manager provides numerous features that make managing Let’s
Encrypt certificates a breeze.</p>
<p>The most obvious feature is, of course, requesting certificates. It can
be instructed to do so in two different ways. The first, slightly more
manual way, is by creating a Certificate resource. This resource will
encapsulate all the necessary information, such as the domain name.
kube-cert-manager, when seeing such a resource, will go and talk to
Let’s Encrypt, request a certificate, and upon success create a new
Kubernetes Secret to store the certificate in. At this point, you can
use the secret in any way you usually would usually, most likely by
mounting it in a container to use the certificates in a web server or
similar.</p>
<p>The more automatic way hooks into Kubernetes Ingress objects. Ingress
objects describe external load balancers and URL routing rules. Part
of their job is terminating TLS connections. For this, the user can
specify a list of hosts and Secret names that should contain the
certificates. If you apply the right annotations to the Ingress
resource, kube-cert-manager will pick them up and automatically
populate the secrets with certificates.</p>
<p>Requesting certificates from Let’s Encrypt requires passing a
challenge-response protocol to prove that one is in control of the
domain. kube-cert-manager, via the use of
<a href="https://github.com/xenolf/lego">lego</a>, provides both HTTP and DNS
based challenges. For the DNS challenge, it supports
<a href="https://github.com/PalmStoneGames/kube-cert-manager/blob/master/docs/providers.md#dns-providers">over a dozen of DNS providers</a>,
such as Google Cloud DNS or Digital Ocean.</p>
<p>Of course requesting certificates is only half the job: They also need
to be renewed regularly: kube-cert-manager will do this automatically
for you. If your load balancer or services support detecting file
changes to reload certificates, you won’t need to do anything. For Go
programs, there is
<a href="https://github.com/PalmStoneGames/kube-cert-http">kube-cert-http</a>.
For other software you will have to write your own adapters and reload
logic, but that should be relatively simple.</p>
<p>kube-cert-manager will also delete old, unused certificates, but that
goes without saying.</p>
<h3 id="installation">Installation</h3>
<p>The project comes with
<a href="https://github.com/PalmStoneGames/kube-cert-manager/blob/master/README.md">detailed documentation</a>
on installation and deployment. I will refrain from copy &amp; pasting it
here.</p>
<h3 id="examples">Examples</h3>
<p>The following examples are supposed to give you a feel for what it is
like working with kube-cert-manager. Check the official documentation
for a full reference of the resource formats.</p>
<h4 id="certificate-objects">Certificate objects</h4>
<p>The first example demonstrates the use of Certificate objects. These
are standalone resources that specify a certificate and cause
kube-cert-manager to request a certificate and store it in a secret.
The secret will exist for as long as the Certificate object exists.</p>
<pre><code>apiVersion: "stable.k8s.psg.io/v1"
kind: "Certificate"
metadata:
name: "psg-dot-io"
spec:
domain: "psg.io"
email: "admin@psg.io"
provider: "googlecloud"
</code></pre>
<p>A secret that is created this way can be used like any other
secret, for example by mounting it into a container, like in the
following example:</p>
<pre><code>spec:
containers:
- name: my-app
image: ...
args:
- "-tls-cert=/etc/tls/psg.io/tls.crt"
- "-tls-key=/etc/tls/psg.io/tls.key"
volumeMounts:
- name: psg-io
mountPath: /etc/tls/psg.io
volumes:
- name: psg-io
secret:
secretName: psg.io
</code></pre>
<h4 id="ingress-objects">Ingress objects</h4>
<p>The next example demonstrates the use of Ingress resources.
kube-cert-manager will automatically extract the domain names and
secret names from Ingress resources and populate the secrets with
certificates.</p>
<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
annotations:
stable.k8s.psg.io/kcm.enabled: "true"
stable.k8s.psg.io/kcm.provider: "googlecloud"
stable.k8s.psg.io/kcm.email: "admin@psg.io"
spec:
tls:
- hosts:
- psg.io
secretName: hello-secret
rules:
- host: "psg.io"
http:
paths:
- path: /hello-world
backend:
serviceName: helloworld
servicePort: 80
</code></pre>
<p>Given this example, the load balancer created by an Ingress controller
will terminate TLS connections using a Let’s Encrypt certificate that
is stored in the secret named <code>hello-secret</code>.</p>
<h3 id="summary">Summary</h3>
<p>Kubernetes automates infrastructure, Let’s Encrypt automates
certificates, and kube-cert-manager is the beautiful marriage of the
two. It can be added to an existing Kubernetes setup without much
effort and is definitely worth checking out.</p>
tag:dominik.honnef.co,2016-04-16:/posts/2016/04/merging-git-repositories/Merging multiple git repositories2016-04-16T17:24:43Z2016-04-16T17:24:43Z<p>Today I had to merge a bunch of existing git repositories into a new
<a href="http://danluu.com/monorepo/">monorepo</a>. I had the following
requirements:</p>
<ul>
<li>The entire history of each repository had to stay intact. This
includes the requirement that <code>git log -- somefile</code> works without
needing any workarounds.</li>
<li>Each repository $repo should be merged into /$repo/ in the monorepo.</li>
<li>Being able to merge new changes wasn’t an issue, because the
original repositories were going to be deleted.</li>
</ul>
<p>The first two requirements meant that we could use neither subtree
merges – as those break file history – nor ordinary merges – as those
would put the contents of $repo in the root, not a subdirectory.</p>
<p>The solution is to use <code>git filter-branch</code> to rewrite the paths in the
repositories, and only then merge the repositories into the monorepo.
The final script looked like this:</p>
<pre><code>new_repo=/tmp/scratch/new_repo
base=git@bitbucket.org:customer
projects=(proj1 proj1)
mkdir -p "$new_repo"
cd "$new_repo"
git init
git commit --allow-empty -m "Initial commit"
for prj in "${projects[@]}"; do
git remote add "$prj" "$base/$prj"
git fetch "$prj"
git filter-branch -f --index-filter \
'git ls-files -s | sed "s%\t\"*%&amp;'"$prj"'/%" |
GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &amp;&amp;
mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"' "$prj/master"
git merge -m "Merge $prj" "$prj/master"
git remote rm "$prj"
done
</code></pre>
<p>Part of that script is based on an
<a href="http://stackoverflow.com/a/28914695/309058">answer on Stackoverflow</a>.</p>
<p>It should be easy enough to adapt this script for different needs
(merging repositories from multiple sources, into different
subdirectories and so on.)</p>
<p><strong>Update:</strong> Beginning with Git 1.9, <code>git merge</code> refuses to merge
unrelated histories unless the <code>--allow-unrelated-histories</code> flag is
used.</p>
tag:dominik.honnef.co,2015-06-22:/posts/2015/06/go-musl/Statically compiled Go programs, always, even with cgo, using musl2015-06-22T06:37:09Z2015-06-22T06:37:09Z<p>
<a href="#flavor-be-gone">(Click here to skip the flavor text and go straight to the commands.)</a>
</p>
<p>Most people appreciate the fact that Go programs compile statically,
as it makes deploying Go programs a lot easier. Unfortunately, they
aren’t always statically compiled.</p>
<p>Using cgo, either in a 3rd party package or in the standard library,
usually requires linking to a libc. Both <code>net</code> and <code>os/user</code> are two
packages known for using cgo and requiring a libc. On most Linux
systems, that leaves us with programs dynamically linking against the
glibc. Usually, that’s not a problem in itself, if we deploy our
programs to machines that have a glibc of a matching version. If they
don’t have a matching glibc, however, things tend to go south.</p>
<p>The first solution that may come to mind is to just link the glibc
statically. However, that rarely works, as various warnings will tell
you. The glibc just doesn’t like that. Also, it might create certain
legal problems. Some people, the FSF in particular, interpret static
linking as copying, and thus any binaries linked statically with the
glibc that we distribute would have to abide by the GPL.</p>
<p>The second solution, which most people go for, is to avoid cgo
altogether. They use the <code>netgo</code> tag to build a pure-Go version of
the <code>net</code> package, and they set CGO_ENABLED=0. Of course this has the
drawback of not being able to use cgo. That means you can’t even use
<code>os/user</code> to get the name of the current user.</p>
<p>Luckily, there is a third solution: Don’t use the glibc. The glibc
isn’t the only libc implementation on Linux. There are others, and
musl is one of them. It’s a lightweight libc (it’s a lot smaller than
the glibc) which supports all the relevant features, was written with
static linking and correctness in mind, and is licensed under the MIT.
It seems to be the perfect fit for our requirements.</p>
<p>
<a name="flavor-be-gone" id="flavor-be-gone"></a>
</p>
<h3 id="installing-musl">Installing musl</h3>
<p>Installing musl is quite straightforward and follows the usual 3
steps: configure, make, make install:</p>
<pre><code>$ wget http://www.musl-libc.org/releases/musl-1.1.10.tar.gz
$ tar -xvf musl-1.1.10.tar.gz
$ cd musl-1.1.10
$ ./configure
$ make
$ sudo make install
</code></pre>
<p>This will install musl in <code>/usr/local/musl</code> – it will not conflict
with your glibc. Alternatively, you can pass <code>--prefix</code> to
<code>./configure</code> to install it elsewhere; for example in your home
directory.</p>
<p>musl comes with a wrapper script called <code>musl-gcc</code>, which invokes gcc
with all the options required for using musl. We can test it on a pure
C program:</p>
<pre><code>$ cat &gt; hello.c &lt;&lt;EOF
#include &lt;stdio.h&gt;
int main()
{
printf("hello, world!\n");
return 0;
}
EOF
$ /usr/local/musl/bin/musl-gcc -static hello.c
$ file a.out
a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped
$ ./a.out
hello, world!
</code></pre>
<p>We end up with a 12 KB large, statically compiled program. No glibc in
sight.</p>
<p>Using it with Go is a bit more complex, but still simple enough:</p>
<pre><code>$ cat &gt; hello.go &lt;&lt;EOF
package main
// #include &lt;stdio.h&gt;
// void helloworld() { printf("hello, world\n"); }
import "C"
import (
"log"
"net"
)
func main() {
log.Println(net.LookupHost("google.com"))
C.helloworld()
}
EOF
$ CC=/usr/local/musl/bin/musl-gcc go build --ldflags '-linkmode external -extldflags "-static"' hello.go
$ file hello
hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped
$ ./hello
2015/06/22 06:01:40 [216.58.211.14 2a00:1450:4016:804::200e] &lt;nil&gt;
hello, world
</code></pre>
<p>We have a fully static binary that is using musl’s DNS
resolver<a href="#footnote-1">¹</a>, as well as a custom C function. The output
binary is 2.8 MB large. How large would it be if we linked dynamically
instead? Still 2.8 MB. The musl libc is quite small and its maximum
overhead will be about 450 KB – that is, if you use all of it, which
you are unlikely to.</p>
<p>But back to the actual command and what we’re doing here:</p>
<ol>
<li>We set the CC environment variable. This tells Go which compiler
and linker to use, in our case the musl-gcc wrapper</li>
<li>We use <code>--ldflags</code> to pass two options to Go’s linker:
<ol>
<li><code>-linkmode external</code> instructs Go’s linker to always use an
external linker, that is gcc (more about this later)</li>
<li><code>-extldflags "-static"</code> instructs Go’s linker to pass the
<code>-static</code> flag to the external linker</li>
</ol>
</li>
</ol>
<p>These 3 settings are all that is required to use musl with Go.</p>
<p>Do note that our basic musl setup only works for cgo that uses the C
standard library. It will not work for external dependencies. For
example, if we wanted to use <code>github.com/hanwen/usb</code> – bindings for
libusb – we’d also have to build libusb with musl first. An
alternative approach here would be to have a VM that uses one of the
musl-based Linux distributions, such as
<a href="https://www.alpinelinux.org/">Alpine Linux</a>. That way, we don’t have
to deal with compiling those libraries.</p>
<h3 id="drawbacks">Drawbacks</h3>
<p>Unfortunately, there are some minor drawbacks to using musl. The first
one is compatibility. Earlier we said that musl was written with
correctness in mind. The problem is that some software out there isn’t
correct, and won’t build with musl. On the one hand, the glibc is a
lot more forgiving and will accept certain code that it shouldn’t. On
the other hand, the glibc comes with its own custom extensions. If
code depends on these, it will not work with musl.</p>
<p>Luckily, this sounds worse than it is. Thanks to several Linux
distributions using musl as their default libc, a lot of these issues
are being fixed. Also, they most often affect large applications, not
libraries like we might use with cgo.</p>
<p>Another drawback is that while musl supports C++, it doesn’t so as
easily as with C. We can’t just use the musl-gcc wrapper, but instead
would need to build a libstdc++ with musl. It’s doable, but outside
the scope of this article. Unfortunately, the lack of easy C++ support
also means that we can’t use Go’s race detector with musl, as it is
written in C++.</p>
<p>Yet another drawback is that the <code>os/user</code> package currently doesn’t
work with musl. It’s a bug that is easily fixable and I hope that it
will be fixed in Go 1.6. In the meantime, you can apply a patch I’ve
included at the end of this article.</p>
<p>The final drawback is compilation speed. Remember that <code>-linkmode</code>
flag? When you’re not using cgo in any 3rd party packages, but only
due to the standard library (<code>net</code>, <code>os/user</code> and so on), Go doesn’t
normally use an external linker. Instead, it does all the linking
itself. Forcing it to use the external linker means that we now have
to invoke gcc, which will be a bit slower. It’s a minor slowdown,
though. For a simple hello world program, the difference between
internal and external linking is roughly 0.09 ms (0.24 ms vs 0.33 ms). It
won’t kill us.</p>
<p>Enjoy your static binaries!</p>
<p><a name="footnote-1" id="footnote-1"></a>
¹: Do note that musl doesn’t support NSS, so using musl’s DNS resolver
is not much different from using netgo: You won’t be able to use
things like mDNS.</p>
<h3 id="the-osuser-patch">The os/user patch</h3>
<pre><code>--- c/src/os/user/lookup_unix.go
+++ w/src/os/user/lookup_unix.go
@@ -9,7 +9,6 @@ package user
import (
"fmt"
- "runtime"
"strconv"
"strings"
"syscall"
@@ -55,17 +54,15 @@ func lookupUnix(uid int, username string, lookupByName bool) (*User, error) {
var pwd C.struct_passwd
var result *C.struct_passwd
- var bufSize C.long
- if runtime.GOOS == "dragonfly" || runtime.GOOS == "freebsd" {
- // DragonFly and FreeBSD do not have _SC_GETPW_R_SIZE_MAX
- // and just return -1. So just use the same
- // size that Linux returns.
+ bufSize := C.sysconf(C._SC_GETPW_R_SIZE_MAX)
+ if bufSize == -1 {
+ // DragonFly and FreeBSD do not have _SC_GETPW_R_SIZE_MAX.
+ // Additionally, not all Linux systems have it, either. For
+ // example, the musl libc returns -1.
bufSize = 1024
- } else {
- bufSize = C.sysconf(C._SC_GETPW_R_SIZE_MAX)
- if bufSize &lt;= 0 || bufSize &gt; 1&lt;&lt;20 {
- return nil, fmt.Errorf("user: unreasonable _SC_GETPW_R_SIZE_MAX of %d", bufSize)
- }
+ }
+ if bufSize &lt;= 0 || bufSize &gt; 1&lt;&lt;20 {
+ return nil, fmt.Errorf("user: unreasonable _SC_GETPW_R_SIZE_MAX of %d", bufSize)
}
buf := C.malloc(C.size_t(bufSize))
defer C.free(buf)
</code></pre>
tag:dominik.honnef.co,2014-12-01:/posts/2014/12/go-tools/An incomplete list of Go tools2014-12-01T00:55:57Z2014-12-01T00:55:57Z
<p>Go is an amazing language, but not just because of its language
features, but also because of its ecosystem and how easy it is to
write tools that operate on Go code.</p>
<p>While languages and frameworks like Java and .NET have popular IDEs
(e.g. Eclipse and Visual Studio) that can do formatting, refactoring,
code generation etc, the Go ecosystem focuses on small standalone
tools and reusable parsers. While Eclipse’s features are locked into
that IDE, a huge and clunky piece of software, Go has tools like gofmt
that can be used from the command line or any editor that allows
calling external programs. Write a tool once, and everyone gets to use
it. It doesn’t matter if you use acme, vim, Emacs or Sublime Text, you
can benefit from a vast number of tools.</p>
<p>This article presents an incomplete list of official and 3rd party
tools for working with Go – it’s inherently incomplete, as people keep
writing new tools at a steady pace. Furthermore, not everyone may like
every tool on this list. Some tools serve very specific needs, some
are very opinionated and some are controversial.</p>
<p>The following index is sorted alphabetically for ease of use, but the
article itself is sorted by theme for a nicer reading experience.</p>
<ul>
<li><a href="#benchcmp">benchcmp</a></li>
<li><a href="#benchviz">benchviz</a></li>
<li><a href="#errcheck">errcheck</a></li>
<li><a href="#generics">Generics</a></li>
<li><a href="#gogenerate">go generate</a></li>
<li><a href="#gocover">go tool cover</a></li>
<li><a href="#govet">go vet</a></li>
<li><a href="#gocode">gocode</a></li>
<li><a href="#godef">godef</a></li>
<li><a href="#godepgraph">godepgraph</a></li>
<li><a href="#gofmt">gofmt</a></li>
<li><a href="#goimports">goimports</a></li>
<li><a href="#golint">golint</a></li>
<li><a href="#gorename">gorename</a></li>
<li><a href="#goreturns">goreturns</a></li>
<li><a href="#impl">impl</a></li>
<li><a href="#oracle">oracle</a></li>
<li><a href="#prettybench">prettybench</a></li>
<li><a href="#stringer">stringer</a></li>
</ul>
<h3 id="gofmt">gofmt</h3>
<p>Hopefully you know gofmt by now, but if you don’t: Run <code>gofmt</code> on
your Go code, and it formats your code according to the only
acceptable way of formatting it. That’s it.</p>
<h3 id="govet">go vet</h3>
<p>Go comes with its own linter, <code>go vet</code>, for analysing Go code and
finding common mistakes. So far, it can check your code for the
following mistakes:</p>
<ul>
<li>Useless assignments</li>
<li>Common mistakes when using sync/atomic</li>
<li>Invalid <code>+build</code> tags</li>
<li>Using composite literals without keyed fields</li>
<li>Passing locks by value</li>
<li>Comparing functions with nil</li>
<li>Using wrong printf format specifiers</li>
<li>Closing over loop variables the wrong way</li>
<li>Struct tags that do not follow the canonical format</li>
<li>Unreachable code</li>
<li>Misuse of <code>unsafe.Pointer</code></li>
<li>Mistakes involving boolean operators</li>
</ul>
<p>The following (constructed) piece of code tries to trigger as many
errors as possible, to show <code>go vet</code>’s behaviour:</p>
<pre><code>package main
import (
"crypto/tls"
"fmt"
"sync"
"sync/atomic"
"unsafe"
)
// +build this,is,too,late
type T struct {
A int `not a canonical tag`
}
func foo() []byte { return []byte{} }
func bar(mu sync.Mutex) {} // Locks have to be passed as pointers
func main() {
fmt.Sprintf("%s", 1) // Wrong verb
fmt.Sprint("%s", 1) // Wrong method
var a int
a = a // Useless assignment
_ = tls.Certificate{nil, nil, nil, nil} // These fields should be keyed
if foo != nil { // We meant to check foo() != nil
}
x := uintptr(0)
_ = unsafe.Pointer(x) // Misuse of unsafe.Pointer
for _, x := range []int{1, 2, 3} {
go func() {
fmt.Println(x) // Closing over x wrong, will probably produce "3 3 3"
}()
}
var b int32
b = atomic.AddInt32(&amp;b, 1) // We shouldn't assign to b
var c bool
if c == false &amp;&amp; c == true { // Always false
}
if c || c { // Redundant
}
return
fmt.Println("unreachable") // Unreachable code
}
</code></pre>
<p>And <code>go vet</code>’s opinion of our code:</p>
<pre><code>/tmp/vet.go:11: +build comment must appear before package clause and be followed by a blank line
/tmp/vet.go:14: struct field tag `not a canonical tag` not compatible with reflect.StructTag.Get
/tmp/vet.go:18: bar passes Lock by value: sync.Mutex
/tmp/vet.go:52: unreachable code
/tmp/vet.go:21: arg 1 for printf verb %s of wrong type: int
/tmp/vet.go:22: possible formatting directive in Sprint call
/tmp/vet.go:25: self-assignment of a to a
/tmp/vet.go:27: crypto/tls.Certificate composite literal uses unkeyed fields
/tmp/vet.go:29: comparison of function foo != nil is always true
/tmp/vet.go:33: possible misuse of unsafe.Pointer
/tmp/vet.go:37: range variable x captured by func literal
/tmp/vet.go:42: direct assignment to atomic value
/tmp/vet.go:45: suspect and: c == false &amp;&amp; c == true
/tmp/vet.go:48: redundant or: c || c
</code></pre>
<h3 id="golint">golint</h3>
<div class="tool-install">Install with <code>go get github.com/golang/lint/golint</code></div>
<p>While <code>go vet</code> checks your code for actual programming errors, golint
checks your code for style violations (so it’s not really a linter –
naming things is hard.)</p>
<p>Gofmt already takes care of whitespace-related style questions, but
it cannot cover things such as variable names. Golint follows the
style that is used internally by Google, which is generally accepted
by the open source Go community as well – in essence it checks the
various style questions discussed <a href="https://code.google.com/p/go-wiki/wiki/CodeReviewComments">in the
wiki</a>.</p>
<p>Golint does not emit errors or warnings, but “suggestions”: These
suggestions can be wrong at times, and code that golint complains
about isn’t necessarily wrong – it might just be hitting a false
positive. Nevertheless, it’s more often right than wrong, and you
should definitely run golint on your code from time to time and fix
those suggestions that it is right about.</p>
<p>The amount of style violations golint checks for is a bit longer
than the list of modules <code>go vet</code> has, so what follows is only a
demonstration of golint’s checks, it’s not exhaustive. For the
following piece of code</p>
<pre><code>package lint
import (
"errors"
. "fmt"
)
var SomeError = errors.New("Capitalised error message")
type unexported int
type Exported int
func (this unexported) Foo() {}
func (oneName Exported) Foo() {
}
func (anotherName Exported) Bar() {
Println("Hi")
}
</code></pre>
<p>golint has the following suggestions:</p>
<pre><code>/tmp/lint.go:5:2: should not use dot imports
/tmp/lint.go:8:5: exported var SomeError should have comment or be unexported
/tmp/lint.go:11:6: exported type Exported should have comment or be unexported
/tmp/lint.go:15:1: exported method Exported.Foo should have comment or be unexported
/tmp/lint.go:18:1: exported method Exported.Bar should have comment or be unexported
/tmp/lint.go:8:5: error var SomeError should have name of the form ErrFoo
/tmp/lint.go:13:1: receiver name should be a reflection of its identity; don't use generic names such as "me", "this", or "self"
/tmp/lint.go:18:1: receiver name anotherName should be consistent with previous receiver name oneName for Exported
</code></pre>
<h3 id="errcheck">errcheck</h3>
<div class="tool-install">Install with <code>go get github.com/kisielk/errcheck</code></div>
<p>Ignoring errors in Go is more difficult than ignoring exceptions in
some other languages, but it’s not impossible. Some people even say
that it’s way too easy. Call a function like <code>(*os.File).Close</code>
without looking at its return value, and that’s an error gone by
unnoticed.</p>
<p>The errcheck tool has been written to help you find exactly those
function calls. Running errcheck on a package will report all function
calls that have unchecked errors.</p>
<p>The following code has three unchecked errors, one for each method
call:</p>
<pre><code>package main
import "os"
func main() {
f, _ := os.Open("foo")
f.Write([]byte("Hello, world."))
f.Close()
}
</code></pre>
<p>Running errcheck with the default options on it will produce the
following output:</p>
<pre><code>foo/foo.go:7:9 f.Write([]byte("Hello, world."))
foo/foo.go:8:9 f.Close()
</code></pre>
<p>Do note that it is not printing the call to <code>os.Open</code> – that’s because
assignment to <code>_</code> is not interpreted as “unchecked” by default. That’s
because assigning errors to <code>_</code> can be a valid pattern, for when one
truly doesn’t care about an error, or because it’s known that an error
cannot occur (that’s true for some implementations of the io.Writer
interface, for example). If you do worry about someone assigning to
<code>_</code> out of laziness, you can run errcheck with the <code>-blank</code> flag:</p>
<pre><code>foo/foo.go:6:5 f, _ := os.Open("foo")
foo/foo.go:7:9 f.Write([]byte("Hello, world."))
foo/foo.go:8:9 f.Close()
</code></pre>
<p>Furthermore, errcheck comes with options for ignoring certain packages
or functions, to avoid false positives. Together with <code>go vet</code>, it
should be part of your quality assurance scripts.</p>
<h3 id="gocover">go tool cover</h3>
<p>Go 1.2 added support for generating code coverage reports, and while
there’s an excellent blog post titled <a href="http://blog.golang.org/cover">“The cover
story”</a>, it might not be receiving
enough attention.</p>
<p>Code coverage reports provide a quick and easy way of finding untested
code. Go’s support for these reports consists of two components:
Support in <code>go test</code> for generating coverage profiles and <code>go tool
cover</code> to generate reports from it.</p>
<p>Set the <code>-coverprofile</code> option while running <code>go test</code> and it will
generate a coverage profile. You can then pass the profile to <code>go tool
cover</code> and generate different kinds of reports. Currently, there are
two forms of reports: A per-function listing of code coverage, and an
HTML report that renders your code in a color-coded fashion, showing
covered lines of code in green, uncovered lines in red, and irrelevant
lines (such as comments, type definitions, etc) in grey.</p>
<pre><code>fmt/format.go: init 100.0%
fmt/format.go: clearflags 100.0%
fmt/format.go: init 100.0%
fmt/format.go: computePadding 84.6%
fmt/format.go: writePadding 100.0%
fmt/format.go: pad 100.0%
fmt/format.go: padString 100.0%
fmt/format.go: fmt_boolean 100.0%
fmt/format.go: integer 100.0%
fmt/format.go: truncate 100.0%
fmt/format.go: fmt_s 100.0%
fmt/format.go: fmt_sbx 100.0%
</code></pre>
<figure>
<a href="/img/go_tools/cover.png"><img alt="go tool cover in the browser" src="/img/go_tools/cover.png" width="736" /></a>
<figcaption>go tool cover in the browser</figcaption>
</figure>
<p>Because the generation of a profile and actually using it are
separated, one can also use other tools than <code>go tool cover</code> to
interpret profiles. <a href="https://gist.github.com/guelfey/6253836">Vim</a> and
Emacs have support for Go’s coverage profiles, and so does
<a href="https://github.com/smartystreets/goconvey">GoConvey</a>.</p>
<figure>
<img alt="go tool cover support in Emacs" src="/img/go_tools/cover.emacs.png" />
<figcaption>go tool cover support in Emacs</figcaption>
</figure>
<h3 id="benchcmp">benchcmp</h3>
<div class="tool-install">Install with <code>go get golang.org/x/tools/cmd/benchcmp</code></div>
<p>Benchmarks are a crucial tool when optimizing code. Anyone who has
asked “Is A or B faster?” will have (or should have!) heard an answer
along the lines of “Write a benchmark!”. Go’s <code>testing</code> package has
built-in benchmarking facilities that produce useful statistics about
speed and allocations.</p>
<p>Comparing numbers by hand, however, can be tedious, and usually we’re
interested in relative changes: Is my new version faster, and by how
much? The benchcmp tool can answer this question. It takes two text
files as input, the output of <code>go test -bench</code> for the old and new
version of your code, and outputs a delta, measured in per cent, for
each benchmark.</p>
<p>The following output is the result of running the benchmarks for the
<code>image/png</code> package in Go 1.3 and Go 1.4 and running the output
through <code>benchcmp</code>:</p>
<pre><code>benchmark old ns/op new ns/op delta
BenchmarkPaeth 5.22 5.30 +1.53%
BenchmarkDecodeGray 1037540 1018369 -1.85%
BenchmarkDecodeNRGBAGradient 4391776 3838947 -12.59%
BenchmarkDecodeNRGBAOpaque 3148985 3145987 -0.10%
BenchmarkDecodePaletted 747152 803097 +7.49%
BenchmarkDecodeRGB 2861484 2751671 -3.84%
BenchmarkEncodeGray 3912599 3940407 +0.71%
BenchmarkEncodeNRGBOpaque 12066176 12072256 +0.05%
BenchmarkEncodeNRGBA 13863837 13898732 +0.25%
BenchmarkEncodePaletted 3908491 3939133 +0.78%
BenchmarkEncodeRGBOpaque 12021250 12040434 +0.16%
BenchmarkEncodeRGBA 33623263 48841059 +45.26%
benchmark old MB/s new MB/s speedup
BenchmarkDecodeGray 63.16 64.35 1.02x
BenchmarkDecodeNRGBAGradient 59.69 68.29 1.14x
BenchmarkDecodeNRGBAOpaque 83.25 83.33 1.00x
BenchmarkDecodePaletted 87.71 81.60 0.93x
BenchmarkDecodeRGB 91.61 95.27 1.04x
BenchmarkEncodeGray 78.52 77.96 0.99x
BenchmarkEncodeNRGBOpaque 101.84 101.79 1.00x
BenchmarkEncodeNRGBA 88.63 88.41 1.00x
BenchmarkEncodePaletted 78.60 77.99 0.99x
BenchmarkEncodeRGBOpaque 102.22 102.06 1.00x
BenchmarkEncodeRGBA 36.55 25.16 0.69x
</code></pre>
<p>We can easily see that most functions didn’t have significant changes
in speed, but <code>BenchmarkEncodeRGBA</code> sticks out, as its performance has
nearly halved in Go 1.4.</p>
<p>If your benchmarks don’t produce stable output, maybe because your
system isn’t entirely idle, or because your CPU is applying thermal
throttling of some sort, you can repeat a single benchmark multiple
times and tell benchcmp to pick the best result, by passing the
<code>-best</code> option. You can also omit any results that are the same
between the two versions, and sort results by magnitude of change.</p>
<p>benchcmp used to be written in Perl and was part of the Go
distribution, but it has since been rewritten in Go and must be
installed separately.</p>
<h3 id="prettybench">prettybench</h3>
<div class="tool-install">Install with <code>go get github.com/cespare/prettybench</code></div>
<p>While we’re on the topic of benchmarks, let’s mention prettybench, a
small utility that makes the output of <code>go test -bench</code> a bit nicer,
by correctly aligning all columns and printing column headers. Simply
pipe <code>go test</code>’s output through prettybench. The downside of using
prettybench, however, is that it has to buffer output, so you won’t
see any results until all benchmarks have finished running.</p>
<p>Before:</p>
<pre><code>PASS
BenchmarkCSSEscaper 1000000 2843 ns/op
BenchmarkCSSEscaperNoSpecials 5000000 671 ns/op
BenchmarkDecodeCSS 1000000 1183 ns/op
BenchmarkDecodeCSSNoSpecials 50000000 32 ns/op
BenchmarkCSSValueFilter 5000000 501 ns/op
BenchmarkCSSValueFilterOk 5000000 707 ns/op
BenchmarkEscapedExecute 500000 6191 ns/op
BenchmarkHTMLNospaceEscaper 1000000 2523 ns/op
BenchmarkHTMLNospaceEscaperNoSpecials 5000000 596 ns/op
BenchmarkStripTags 1000000 2351 ns/op
BenchmarkStripTagsNoSpecials 10000000 260 ns/op
BenchmarkJSValEscaperWithNum 1000000 1123 ns/op
BenchmarkJSValEscaperWithStr 500000 4882 ns/op
BenchmarkJSValEscaperWithStrNoSpecials 1000000 1461 ns/op
BenchmarkJSValEscaperWithObj 500000 5052 ns/op
BenchmarkJSValEscaperWithObjNoSpecials 1000000 1897 ns/op
BenchmarkJSStrEscaperNoSpecials 5000000 608 ns/op
BenchmarkJSStrEscaper 1000000 2633 ns/op
BenchmarkJSRegexpEscaperNoSpecials 5000000 661 ns/op
BenchmarkJSRegexpEscaper 1000000 2510 ns/op
BenchmarkURLEscaper 500000 4424 ns/op
BenchmarkURLEscaperNoSpecials 5000000 422 ns/op
BenchmarkURLNormalizer 500000 3068 ns/op
BenchmarkURLNormalizerNoSpecials 5000000 431 ns/op
ok html/template 62.874s
</code></pre>
<p>After:</p>
<pre><code>PASS
benchmark iter time/iter
--------- ---- ---------
BenchmarkCSSEscaper 1000000 2843 ns/op
BenchmarkCSSEscaperNoSpecials 5000000 671 ns/op
BenchmarkDecodeCSS 1000000 1183 ns/op
BenchmarkDecodeCSSNoSpecials 50000000 32 ns/op
BenchmarkCSSValueFilter 5000000 501 ns/op
BenchmarkCSSValueFilterOk 5000000 707 ns/op
BenchmarkEscapedExecute 500000 6191 ns/op
BenchmarkHTMLNospaceEscaper 1000000 2523 ns/op
BenchmarkHTMLNospaceEscaperNoSpecials 5000000 596 ns/op
BenchmarkStripTags 1000000 2351 ns/op
BenchmarkStripTagsNoSpecials 10000000 260 ns/op
BenchmarkJSValEscaperWithNum 1000000 1123 ns/op
BenchmarkJSValEscaperWithStr 500000 4882 ns/op
BenchmarkJSValEscaperWithStrNoSpecials 1000000 1461 ns/op
BenchmarkJSValEscaperWithObj 500000 5052 ns/op
BenchmarkJSValEscaperWithObjNoSpecials 1000000 1897 ns/op
BenchmarkJSStrEscaperNoSpecials 5000000 608 ns/op
BenchmarkJSStrEscaper 1000000 2633 ns/op
BenchmarkJSRegexpEscaperNoSpecials 5000000 661 ns/op
BenchmarkJSRegexpEscaper 1000000 2510 ns/op
BenchmarkURLEscaper 500000 4424 ns/op
BenchmarkURLEscaperNoSpecials 5000000 422 ns/op
BenchmarkURLNormalizer 500000 3068 ns/op
BenchmarkURLNormalizerNoSpecials 5000000 431 ns/op
ok html/template 62.874s
</code></pre>
<h3 id="benchviz">benchviz</h3>
<div class="tool-install">Install with <code>go get github.com/ajstarks/svgo/benchviz</code></div>
<p>Another way of visualising benchmark results is via benchviz, a tool
that generates graphical output. It depends on benchcmp and uses bar
charts to show positive and negative change. Simply pipe the output of
<code>benchcmp</code> to <code>benchviz</code> and it will produce SVG output on stdout.
Running <code>benchviz -h</code> will reveal several settings that can control
the way the image will look like.</p>
<p>By default, it will look similar to the following image.</p>
<figure>
<img alt="benchviz output for the benchmarks from earlier" src="/img/go_tools/benchviz.svg" />
<figcaption>benchviz output for the benchmarks from earlier</figcaption>
</figure>
<h3 id="gogenerate">go generate</h3>
<p>A new subcommand that was added in Go 1.4 is <code>go generate</code>, which is
supposed to replace various uses of makefiles and shell scripts in Go
packages.</p>
<p>All <code>go generate</code> does is look for directives (in the form of special
comments) in your Go code that direct it to execute programs. These
programs usually generate new Go files, hence the name.</p>
<p>Example uses of <code>go generate</code> include running yacc to turn grammar
files into Go code, generating Go code from protobuff definitions or
embedding binary files in Go programs.</p>
<p>A core principle of <code>go generate</code> is that it has to be run manually
and is supposed to be run by the developer of the package. It will not
run on <code>go build</code>, and by extension not on <code>go get</code>, either. A
developer is supposed to run <code>go generate</code> and then check the
generated files into version control. That way, when a user downloads
the package, he already has all the Go files he needs to compile it
and doesn’t need to run arbitrary tools (which would also be a major
security risk.)</p>
<p>What <code>go generate</code> explicitly is not is a general build system à la
make – it does not offer any form of dependency analysis and instead
simply runs commands it encounters in the order it finds them.</p>
<p>For a detailed description on how to use <code>go generate</code>, run <code>go
generate -h</code>.</p>
<h3 id="stringer">stringer</h3>
<div class="tool-install">Install with <code>go get golang.org/x/tools/cmd/stringer</code></div>
<p>Constants in Go are a compile-time concept. Once compilation finishes
and you run your program, code isn’t aware of constants or their names
anymore. But what if you have a set of typed constants and want to
implement the <code>Stringer</code> interface for them? In the past, the easiest
solutions were using a switch, or having a slice or map of names.
Writing it felt a bit silly (why do I need to write each word as a
constant name <em>and</em> as a string?) and you had to remember to update it
after adding a new constant.</p>
<p>With the new <a href="#gogenerate"><code>go generate</code> command in Go 1.4</a>, and the
<code>stringer</code> tool, however, we can easily automate this.</p>
<p>Given something like</p>
<pre><code> package date
type Day int
const (
Monday Day = iota
Tuesday
Wednesday
Thursday
Friday
Saturday
Sunday
)
</code></pre>
<p>running <code>stringer -type Day</code> on it will create a new file (<code>day_string.go</code>) that looks like</p>
<pre><code>// generated by stringer -type Day; DO NOT EDIT
package date
import "fmt"
const _Day_name = "MondayTuesdayWednesdayThursdayFridaySaturdaySunday"
var _Day_index = [...]uint8{6, 13, 22, 30, 36, 44, 50}
func (i Day) String() string {
if i &lt; 0 || i &gt;= Day(len(_Day_index)) {
return fmt.Sprintf("Day(%d)", i)
}
hi := _Day_index[i]
lo := uint8(0)
if i &gt; 0 {
lo = _Day_index[i-1]
}
return _Day_name[lo:hi]
}
</code></pre>
<p>Not only is this process fully automatic, it also generates a more
efficient implementation than a human would write. The specific
implementation that is used depends on the number and density of your
constants.</p>
<p>Add</p>
<pre><code>//go:generate stringer -type Day
</code></pre>
<p>to your <code>.go</code> file (the one containing the constants), and every time
you run <code>go generate</code>, the <code>Stringer</code> implementation will be
regenerated for you.</p>
<h3 id="impl">impl</h3>
<div class="tool-install">Install with <code>go get github.com/josharian/impl</code></div>
<p>Another tool that’s inspired by an IDE feature is impl, which, given
an interface, will create method stubs for implementing that
interface.</p>
<p>Now, interfaces in Go are usually small, often only consisting of one
or two methods, so impl isn’t as badly needed as stub generators in
e.g. Java, but it can still provide a small speedup if integrated with
your editor. It’s probably mostly useful when writing tests and
wanting to stub out multiple interfaces. Here, impl allows you to
generate stub implementations for all interfaces with only a few
keystrokes.</p>
<h3 id="generics">Generics</h3>
<p>It is impossible to have a tool like <code>go generate</code> and not have people
write tools to imitate generics. It is equally impossible to write a
list of Go tools without people mentioning their favourite tool for
imitating generics. As there are as many tools for this problem as
there are different interpretations of what generics really are,
there’s little point in listing all of them individually. If you’re
interested in imitating generics/code generation in Go, you might want
to look at one of the following tools:</p>
<ul>
<li><a href="https://github.com/clipperhouse/gen">gen</a></li>
<li><a href="https://github.com/cheekybits/genny">genny</a></li>
<li><a href="https://github.com/ncw/gotemplate">gotemplates</a></li>
</ul>
<p>All of these tools put their own spin on the code generation problem,
so it’s up to you to find out which one you like the most.</p>
<h3 id="goimports">goimports</h3>
<div class="tool-install">Install with <code>go get golang.org/x/tools/cmd/goimports</code></div>
<p>Does <code>./foo.go:3: imported and not used: "log"</code> make you twitch? Do
you wish Go would just allow you to import unused packages? Or are you
even tired of having to manually import all those packages you use?
Then goimports might be for you. It’s a drop-in replacement for
gofmt that, additionally to formatting your code, also manages your
imports for you.</p>
<p>It removes unused imports and adds missing imports, as long as doing
so is not ambiguous. If multiple packages with the same name, such as
<code>text/template</code> and <code>html/template</code> can be used to make the code
compile, then goimports errs on the side of safety and imports
neither. But that’s a rare occurrence in everyday code. More often
than not, package names are sufficiently unique, especially because
goimports takes into account which identifiers you use.</p>
<p>Before:</p>
<pre><code>package main
func main() {
f, _ := os.Open("foo")
log.Println("opened file")
io.Copy(os.Stdout, f)
}
</code></pre>
<p>After:</p>
<pre><code>package main
import (
"io"
"log"
"os"
)
func main() {
f, _ := os.Open("foo")
log.Println("opened file")
io.Copy(os.Stdout, f)
}
</code></pre>
<p>Since it’s a drop-in replacement for gofmt, all editors that permit
changing the gofmt executable can use goimports.</p>
<h3 id="goreturns">goreturns</h3>
<div class="tool-install">Install with <code>go get sourcegraph.com/sqs/goreturns</code></div>
<p>Inspired by the way goimports magically fixes incomplete code by
adding import statements, goreturns fills in incomplete return
statements with zero values.</p>
<p>Given the following snippet</p>
<pre><code>func F() (*MyType, int, error) { return errors.New("foo") }
</code></pre>
<p>running <code>goreturns</code> on it will transform it into</p>
<pre><code>func F() (*MyType, int, error) { return nil, 0, errors.New("foo") }
</code></pre>
<p>The primary idea here is that returning errors usually also requires
returning one or more zero values, and if there are several error
conditions on which to return, it can become annoying to manually
write those zero values over and over again.</p>
<p>Goreturns acts as a drop-in replacement for goimports, by calling
goimports after it’s done doing its own thing.</p>
<p>The following 30 seconds long screencast shows goreturns in action.</p>
<iframe width="736" height="414" src="//www.youtube.com/embed/hyEMO9vtKZ8" frameborder="0" allowfullscreen=""></iframe>
<h3 id="godef">godef</h3>
<div class="tool-install">Install with <code>go get code.google.com/p/rog-go/exp/cmd/godef</code></div>
<p>One of those features that a lot of IDE users love, and why people
coming from languages like Java often ask for a “Go IDE”, is “go to
definition” – being able to jump to the definition of constants,
variables and functions.</p>
<p>Luckily, godef has us covered. Its most interesting mode of operation
allows printing the file/line number in which a given identifier is
defined. Given that information, it’s trivial to implement a “go to”
feature in any modern editor. Support for acme is included with godef
itself, vim and Emacs have support in
<a href="https://github.com/dgryski/vim-godef">vim-godef</a> and go-mode
respectively.</p>
<p>Another mode that godef supports is printing information about an
identifier. Given a variable of some struct type, godef can list all
fields of that struct. Given a constant, it can show you the
constant’s value.</p>
<p>Godef works across all packages in your GOPATH as well as your GOROOT
(i.e. the standard library), so any identifier you have, you can check
out its code. This is a crucial feature while reading or even
debugging someone else’s code. Even traversing deep into the standard
library is no problem. And to make things even better, godef is so
fast that the moment you decide to jump somewhere, you’re already
there.</p>
<p>The following short clip shows godef in action:</p>
<video controls="">
<source src="/video/go_emacs/godef_mp4.mp4" type="video/mp4"></source>
<source src="/video/go_emacs/godef_ogg.ogg" type="video/ogg"></source>
Your browser does not support the video tag.
</video>
<h3 id="gocode">gocode</h3>
<div class="tool-install">Install with <code>go get github.com/nsf/gocode</code></div>
<p>Another popular feature of IDEs is auto completion. And as with most
popular IDE features, the Go community came up with a reusable tool
for it: gocode. Gocode provides context-sensitive auto completion and
uses a client/server architecture, where the server is a caching
daemon, and the clients are a command line utility as well as all
popular editors.</p>
<p>It uses Go object files (that is, <code>go install</code>‘d packages) to complete
identifiers of imported packages, and parses the source of the current
package to provide completion for said package. Due to the efficient
layout of Go object files, the fast parser and gocode’s caching, auto
completion completes almost instantly (compared to auto completion in
a popular Java IDE, which may remind you that Java is also a brand of
coffee and you’d really like one now.)</p>
<p>Do note that gocode itself just provides information about identifiers
– how your tools or editors make use of it is up to you – Emacs, for
example, has several traditional completion backends, but also <a href="https://github.com/syohex/emacs-go-eldoc">eldoc
integration</a>, to show the
signature of the function call at point in a status bar.</p>
<h3 id="oracle">oracle</h3>
<div class="tool-install">Install with <code>go get golang.org/x/tools/cmd/oracle</code></div>
<p>The Go oracle is a source analysis tool capable of answering a
multitude of questions about Go programs, such as:</p>
<ul>
<li>What is the type of this expression? What are its methods?</li>
<li>What’s the value of this constant expression?</li>
<li>Where is the definition of this identifier?</li>
<li>What are the exported members of this imported package?</li>
<li>What are the free variables of the selected block of code?</li>
<li>What interfaces does this type satisfy?</li>
<li>Which concrete types implement this interface?</li>
<li>What are the possible concrete types of this interface value?</li>
<li>What are the possible callees of this dynamic call?</li>
<li>What are the possible callers of this function?</li>
<li>What objects might this pointer point to?</li>
<li>Where are the corresponding sends/receives of this channel receive/send?</li>
<li>Which statements could update this field/local/global/map/array/etc?</li>
<li>Which functions might be called indirectly from this one?</li>
</ul>
<p>If one were to address the technical aspects of the Oracle, one could
mention that it uses pointer analysis on an SSA form of Go programs to
approximate an answer to these questions, and if you want to know more
about the design of the Oracle, <a href="https://docs.google.com/document/d/1WmMHBUjQiuy15JfEnT8YBROQmEv-7K6bV-Y_K53oi5Y/edit">there is an in-depth design
document</a>
– but for the sake of keeping it simple, and because I can’t possibly
represent the design in an accurate way, we’ll refer to what the
Oracle is doing as “magic”.</p>
<p>We do, however, need to peek behind the curtain a bit and explain what
is necessary for the magic to work. In order to answer most questions,
the Oracle needs to trace all possible program flows. For this, it
needs both a <em>scope</em> and <em>entry points</em>. The scope defines the set of
files or packages it considers during its analysis. The entry points
can either be <code>main</code> packages or tests – the Oracle will trace the
program flow from these entry points, and the union of all flows is
what it will use to answer questions.</p>
<p>This is why the Oracle is able to answer questions such as <em>Who is
calling this function?</em> – because it saw who was calling it. If no
entry point leads to the call of said function, it has no callers. If
no entry point leads to a channel operation on a given channel, no
operations will be listed, and so on.</p>
<p>To demonstrate some of the coolest features of the Oracle, we’ll be
asking the following questions:</p>
<ul>
<li>What channel operations are happening for a given channel? - What
concrete methods will a dynamic method call invoke? - Who is calling
our function?</li>
</ul>
<h4 id="channel-peers">Channel peers</h4>
<p>For our first question, we’ll look at the following piece of code:</p>
<pre><code>package main
import "fmt"
func odd(ch chan int) {
i := 1
for {
ch &lt;- i
i += 2
}
}
func even(ch chan int) {
i := 0
for {
ch &lt;- i
i += 2
}
}
func main() {
ch := make(chan int)
go odd(ch)
go even(ch)
for i := 0; i &lt; 10; i++ {
fmt.Println(&lt;-ch)
}
}
</code></pre>
<p>Specifically, we’ll look at the channel receive operation in
<code>fmt.Println(&lt;-ch)</code>, and we will inquire who is sending us values on
that channel. Executing the corresponding query will result in the
following output:</p>
<pre><code>$ oracle -pos="/tmp/demo2.go:#263" peers /tmp/demo2.go
/tmp/demo2.go:27:15: This channel of type chan int may be:
/tmp/demo2.go:22:12: allocated here
/tmp/demo2.go:8:6: sent to, here
/tmp/demo2.go:16:6: sent to, here
/tmp/demo2.go:27:15: received from, here
</code></pre>
<p>It tells us everything there is to know about <code>ch</code>: Where it was
created, where it is being sent to, and where it is being received
from. The interesting thing here is that the Oracle is able to tell
that we’re passing <code>ch</code> to two functions, <code>odd</code> and <code>even</code>, and that
these functions are sending to <code>ch</code>. If we remove one of the
goroutines, the corresponding send operation will disappear from the
output, too.</p>
<p>In big, highly concurrent programs, it might not always be obvious
where we’re receiving a value from or where it is being sent to, so
being able to ask this kind of question can be very valuable. A common
example would be a <code>quit</code> channel that can be closed in many different
places, and received from in multiple places as well.</p>
<h4 id="callees">Callees</h4>
<p>For our second question (<em>What method are we calling?</em>) we’ll be using
the following small program:</p>
<pre><code>package main
import (
"fmt"
"math/rand"
"time"
)
type A struct{}
type B struct{}
func (A) String() string { return "A" }
func (B) String() string { return "B" }
func Print(s fmt.Stringer) {
fmt.Println("Hello " + s.String())
}
func main() {
rand.Seed(time.Now().UnixNano())
if rand.Intn(2) == 1 {
Print(A{})
} else {
Print(B{})
}
}
</code></pre>
<p>We’ll look at the <code>Print</code> function and ask what concrete methods we
might be calling when calling <code>s.String()</code>:</p>
<pre><code>$ oracle -pos="/tmp/demo.go:#227" callees /tmp/demo.go
/tmp/demo.go:16:33: this dynamic method call dispatches to:
/tmp/demo.go:12:10: (main.A).String
/tmp/demo.go:13:10: (main.B).String
</code></pre>
<p>The Oracle knows which types we might end up passing to <code>Print</code>. If we
had more calls to <code>Print</code> with even more types, these would be listed
as well.</p>
<h4 id="callers">Callers</h4>
<p>For our final question (<em>Who is calling this method?</em>), we’ll be
reusing the program from the previous question, looking at <code>A.String</code>
instead and asking who might be calling this method:</p>
<pre><code>$ oracle -pos="/tmp/demo.go:#116" callers /tmp/demo.go
/tmp/demo.go:12:10: (main.A).String is called from these 1 sites:
/tmp/demo.go:16:33: dynamic method call from main.Print
</code></pre>
<p>The same information that allows the Oracle to tell which methods
we’re calling also allows it to tell where a method ends up being
called from, even if these calls are dynamic.</p>
<h4 id="editor-integration">Editor integration</h4>
<p>The Oracle isn’t really meant to be used from the command line.
Instead, it’s supposed to be used as the universal solution to Go
source analysis in editors. The Oracle can generate output in plain
text, JSON or XML, which should cater to most editor’s needs. It comes
with Emacs integration, Vim has support for it via
<a href="https://github.com/fatih/vim-go">vim-go</a>. And if your editor of
choice isn’t offering integration with the Oracle yet, there’s also
<a href="https://github.com/fzipp/pythia">Pythia</a>, a browser-based solution.</p>
<p>For more information about the Oracle’s features and how to use it,
check out the <a href="https://docs.google.com/document/d/1SLk36YRjjMgKqe490mSRzOPYEDe0Y_WQNRv-EiFYUyw/view">user
guide</a>.</p>
<h3 id="gorename">gorename</h3>
<div class="tool-install">Install with <code>go get golang.org/x/tools/cmd/gorename</code></div>
<p>Renaming identifiers can range in difficulty from <em>very easy</em> to <em>oh
god why…</em> – unless you’re using gorename, a relatively new tool for
type-aware renaming of identifiers – that is constants, variables,
types, functions and methods.</p>
<p>It is type-aware and workspace-aware. Workspace-aware means that while
you can absolutely use gorename to rename local variables, which will
have no effect on the code outside of a function or even a package,
you can also use it to rename exported identifiers. In that case, gorename
will check all packages in your GOPATH for uses of that identifier,
and rename those uses as well.</p>
<p>Being type-aware, and generally smart, means that gorename will refuse
to make changes that stop the code from compiling correctly. If
changing an identifier would cause a naming conflict or a shadowing
error, gorename will abort. In the same way, if renaming a method
would stop a type from implementing an interface that it has to
implement, gorename will abort. Renaming a method in an interface,
however, will cause gorename to update all known implementations and
uses of the interface.</p>
<p>It has to be noted that gorename is “as strong as a compiler”, which
means that if a compiler is able to complain about a type error,
gorename is able to deal with it, too (either by renaming even more
identifiers, or by aborting). However, there are more dynamic ways of
using interfaces which can only be checked at runtime, and gorename
isn’t able to detect these cases (as it’s an undecidable problem).
While it is very rare that renaming an identifier will cause code to
break due to this, it is still required to validate the changes that
gorename makes.</p>
<p>Support for go-mode is included with gorename, support for vim can be
found in <a href="https://github.com/fatih/vim-go">vim-go</a>.</p>
<h3 id="godepgraph">godepgraph</h3>
<div class="tool-install">Install with <code>go get github.com/kisielk/godepgraph</code></div>
<p>Another handy utility is godepgraph, a tool for generating
dependency graphs (in Graphviz format) for Go packages.</p>
<figure>
<a href="/img/go_tools/godepgraph.dot.png"><img alt="Dependency graph of golint, rendered with dot" src="/img/go_tools/godepgraph.dot.png" width="736" /></a>
<figcaption>Dependency graph of golint, rendered with dot</figcaption>
</figure>
<p>Packages in the standard library are colored green, while packages
from your GOPATH are colored blue. If a package imports the pseudo
package <code>C</code>, it will be colored orange, but <code>C</code> itself will not show
up in the graph, as it isn’t a real package. Also note how godepgraph
doesn’t delve into the dependencies of the standard library, it only
shows the packages that are imported directly, and even those can be
hidden with a command line flag.</p>
<figure>
<a href="/img/go_tools/godepgraph.circo.png"><img alt="Dependency graph of golint, rendered with circo" src="/img/go_tools/godepgraph.circo.png" width="736" /></a>
<figcaption>Dependency graph of golint, rendered with `circo`</figcaption>
</figure>
<p>Because godepgraph just outputs a directed graph in Graphviz format,
you can tweak the final image in many ways: You can use different
Graphviz filters, such as dot or circo, and you can adjust all kinds
of options affecting the layout. But using <code>dot</code> and default settings
usually works well enough – for small packages, anyway.</p>
tag:dominik.honnef.co,2014-01-10:/go-tip/2014-01-10/What's happening in Go tip (2014-01-10)2014-01-10T11:54:50Z2014-01-10T11:54:50Z
<p>It’s 2014, we’ve survived Christmas and New Year’s Eve and the Go team
has begun work on the next release. And thus the <em>What’s happening in
Go tip</em> series has started again.</p>
<p>For this iteration of the series, there will be some small
adjustments. The main change will be the lack of a strict weekly
cycle. There might be multiple articles a week, or none. This is
partly due to personal reasons and partly because it allows a more
flexible approach to reporting changes. In the same vein, individual
articles might be shorter than before, for the sake of addressing new
developments faster.</p>
<p>The second change is the addition of coverage of proposals and
discussions that haven’t seen any code changes yet. This is due to the
fact that Go 1.3 will see some big changes (primarily the planned
rewrite of the compilers in Go) that should be covered early on.</p>
<p>In this issue of the series we will, however, concentrate on the new
<code>sync.Pool</code> type since it’s the first major addition to the standard
library in Go 1.3.</p>
<h3 id="whats-happening">What’s happening</h3>
<ul>
<li><a href="#syncpool">Addition of the sync.Pool type</a></li>
<li><a href="#dev">Small change to the development process</a></li>
</ul>
<h4 id="syncpool">Addition of the <code>sync.Pool</code> type</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/41860043"><span class="cl">CL 41860043</span></a>, <a href="https://golang.org/cl/43990043"><span class="cl">CL 43990043</span></a>, <a href="https://golang.org/cl/37720047"><span class="cl">CL 37720047</span></a>, <a href="https://golang.org/cl/44080043"><span class="cl">CL 44080043</span></a>, <a href="https://golang.org/cl/44150043"><span class="cl">CL 44150043</span></a>, <a href="https://golang.org/cl/44060044"><span class="cl">CL 44060044</span></a> <a href="https://golang.org/cl/44050044"><span class="cl">CL 44050044</span></a>, <a href="https://golang.org/cl/44680043"><span class="cl">CL 44680043</span></a>, <a href="https://golang.org/cl/46010043"><span class="cl">CL 46010043</span></a></p>
<p>Projects like the JVM invest a lot of time and effort in improving
their garbage collectors to battle the high amount of garbage they
need to deal with. Go, on the other hand, generally takes the approach
of avoiding garbage in the first place, requiring a less sophisticated
garbage collector by giving control back to the programmer.</p>
<p>To this end, several parts of the standard library have pools of
reusable objects. The <code>regexp</code> package maintains state machines for
concurrent use of a regexp, the <code>fmt</code> package has printers and other
packages have their own pools as well, or would like to have them.</p>
<p>This approach, however, has two issues. The obvious one is code
duplication: all packages have to implement their own pools, even
though the implementation often is identical. A subtler issue is the
lack of draining the pools. These simple implementations never release
memory, which defies the point of a garbage collected language and can
lead to unnecessarily high memory usage.</p>
<p>Because of this, Brad Fitzpatrick
<a href="https://code.google.com/p/go/issues/detail?id=4720">suggested</a> adding
a public <code>Cache</code> type to the <code>sync</code> package. This suggestion triggered
a lot of discussion. Should Go provide such a type in the standard
library or should it instead be private? Should it actually evict
items, and if so, when? Should it even be named <code>Cache</code>, or rather
<code>Pool</code>?</p>
<p>Let me explain the difference between a cache and a pool, and why it
is important for this discussion. The type that Brad Fitzpatrick
requested is actually a pool: A set of interchangeable values where it
doesn’t matter which concrete value you get out, because they’re all
identical. You wouldn’t even notice when, instead of getting a value
from the pool, you get a newly created one. Caches, on the other hand,
map keys to concrete values. A prominent example are disk caches,
where files from a slow storage medium are cached in the system’s main
memory for faster access. If the cache contains values for the keys
(in this example file names) <code>A</code> and <code>B</code>, and you request the value
associated with <code>A</code>, you definitely do not want to get <code>B</code> out. The
fact that values in a cache aren’t identical also adds a lot more
complexity to their eviction policies, namely the question of which
values to evict when. The
<a href="http://en.wikipedia.org/wiki/Cache_algorithms">Wikipedia article on cache algorithms</a>
lists 13 different algorithms, from the well known LRU cache to more
complex ones like the
<a href="http://en.wikipedia.org/wiki/LIRS_caching_algorithm">LIRS caching algorithm</a>.</p>
<p>With that out of the way, the only real question concerning our pool
is when to drain it. And pretty much every possibility has been
suggested: somewhen before GC, somewhen after GC, clock-based or by
employing weak pointers. And all suggestions had their own drawbacks.</p>
<p>After a lot of back and forth, Russ Cox eventually
<a href="https://groups.google.com/forum/#!searchin/golang-dev/gc-aware/golang-dev/kJ_R6vYVYHU/LjoGriFTYxMJ">suggested an API and draining policy</a>
that was very simple: Drain the pools during garbage collection. This
proposal reminds us of the fact that the purpose of the <code>Pool</code> type is
to reuse memory between garbage collections. It should not circumvent
garbage collection, it should only make it more efficient.</p>
<p>Brad implemented said proposal in <a href="https://golang.org/cl/41860043"><span class="cl">CL 41860043</span></a> and, after some more
back and forth, the CL was committed to the Go repository. Of course
this CL wasn’t the end of the story. First, all existing pools had to
be replaced with <a href="http://tip.golang.org/pkg/sync/#Pool"><code>sync.Pool</code></a>.
This happened in the CLs <a href="https://golang.org/cl/43990043"><span class="cl">CL 43990043</span></a>, <a href="https://golang.org/cl/37720047"><span class="cl">CL 37720047</span></a>, <a href="https://golang.org/cl/44080043"><span class="cl">CL 44080043</span></a>, <a href="https://golang.org/cl/44150043"><span class="cl">CL 44150043</span></a>, <a href="https://golang.org/cl/44060044"><span class="cl">CL 44060044</span></a> and <strong>not</strong> in <a href="https://golang.org/cl/44050044"><span class="cl">CL 44050044</span></a>. <a href="https://golang.org/cl/44050044"><span class="cl">CL 44050044</span></a> was
special in that it tried to use <code>sync.Pool</code> in the <code>encoding/gob</code>
package, which has local free lists. And <em>local</em> is the keyword. The
pools in <code>encoding/gob</code> belong to instances of a <code>Decoder</code> and are
both short-lived and already handled appropriately by the GC. A free
list would exist as long as the decoder did, and then be thrown away.
Russ Cox
<a href="https://codereview.appspot.com/44050044/#msg10">commented on the CL</a>,
making clear what the purpose of <code>sync.Pool</code> is, and what it isn’t. To
this end, Rob Pike submitted and committed <a href="https://golang.org/cl/44680043"><span class="cl">CL 44680043</span></a>, which expands
the type’s documentation to state its purpose more clearly.</p>
<blockquote>
<p>Pool’s intended use is for free lists maintained in global
variables, typically accessed by multiple goroutines simultaneously.
Using a Pool instead of a custom free list allows the runtime to
reclaim entries from the pool when it makes sense to do so. An
appropriate use of sync.Pool is to create a pool of temporary
buffers shared between independent clients of a global resource. On
the other hand, if a free list is maintained as part of an object
used only by a single client and freed when the client completes,
implementing that free list as a Pool is not appropriate.</p>
</blockquote>
<p>From the comment (and prior discussions) it’s also apparent that the
addition of <code>sync.Pool</code> is tentative and might be reverted before the
release of Go 1.3, if it doesn’t live up to its promises. This is also
tracked by
<a href="https://code.google.com/p/go/issues/detail?id=6984">Issue 6984.</a></p>
<p>While this marks the end of this exposition on <code>sync.Pool</code>, it’s not
the end of the pool’s journey yet. There’s still <a href="https://golang.org/cl/46010043"><span class="cl">CL 46010043</span></a>, which
improves the very simple initial implementation to become better
suited for concurrent use, but it hasn’t been reviewed yet at this
point.</p>
<h4 id="dev">Small change to the development process</h4>
<p>Beginning with the Go 1.3 cycle, some small adjustments have been made
to the development process. These changes only affect people who are
either directly involved in the development process or who, like me,
follow changes closely.</p>
<p>A new mailing list,
<a href="https://groups.google.com/forum/#!forum/golang-codereviews">golang-codereviews</a>,
has been created and made the default Cc target of new CLs, instead of
the old
<a href="https://groups.google.com/forum/#!forum/golang-dev">golang-dev</a>. The
idea behind this is to reduce the noise on golang-dev, to allow actual
conversations.</p>
<p>At the same time, <a href="https://go-dev.appspot.com/">a new dashboard</a> has
been created, allowing committers to easily track open issues and CLs.
Anyone who is interested in how the Go team works is encouraged to
read <a href="http://golang.org/s/go-dev-howto">the usage instructions</a> for
the new dashboard.</p>
tag:dominik.honnef.co,2013-12-11:/posts/2013/12/go-stdlib-cgo/Cross-compiling Go and hidden uses of cgo2013-12-11T00:12:36Z2013-12-11T00:12:36Z
<p>One of Go’s strengths is its built-in support for cross-compilation.
Setting up and using cross-compilers in Go is completely painless and
mainly involves setting some environment variables.</p>
<p>There is, however, one drawback to cross-compiling Go programs: It’s
not possible to use cgo. Cgo is used for writing bindings to C libraries
so that they can be used directly in Go.</p>
<p>Most of the time when people consider this drawback, they’re only
thinking about their own packages. If they don’t explicitly use cgo,
surely cross-compiling should be fine? Yes and no. Unfortunately,
there are several packages in the Go standard library that make use of
or even rely on cgo.</p>
<h3 id="the-net-package">The <code>net</code> package</h3>
<p>The <code>net</code> package is the prime example for its use of cgo to use the
system resolver, used for making domain name lookups. At the same
time, it’s a component that works really well without cgo in most
cases, because it also implements its own resolver in pure Go, which
will work fine for most people, unless they need the features that are
only available with the system resolver, such as LDAP.</p>
<h3 id="the-osuser-package">The <code>os/user</code> package</h3>
<p>A better hidden use of cgo can be found in the <code>os/user</code> package.
<code>os/user</code> needs cgo to determine the current user or to look up any
other user by ID. When cgo isn’t available, functions like
<code>user.Current()</code> and <code>user.Lookup()</code> will always return an error,
saying that these functions aren’t implemented.</p>
<p>(Plan 9 is an exception to this rule. <code>user.Current()</code> works fine
without cgo, while other functions in this package never work, even
when <code>cgo</code> is enabled.)</p>
<h3 id="the-cryptox509-package">The <code>crypto/x509</code> package</h3>
<p>The last example is <code>crypto/x509</code>, which parses X.509-encoded keys and
certificates, i.e. the kind of certificates that SSL/TLS use. These
certificates form a chain of trust, going all the way up to the root
certificates. In order to verify a certificate, all certificates in
that chain need to be verified, which means that Go needs access to
them. Unfortunately, in OS X, the root certificates are stored in
Apple’s Keychain software, and interfacing with that requires cgo. The
saddening result of this is that a cross-compiled Go binary targeting
OS X will not be able to use SSL/TLS correctly. Instead, it might
return the following error:</p>
<pre><code>x509: failed to load system roots and no roots provided
</code></pre>
<h3 id="conclusion">Conclusion</h3>
<p>All three examples have in common that even though they might break
your application when cross-compiled, the compilation process itself
will succeed. Only at execution time will you realize that something
is wrong. And if you’re bad at handling your errors (which, for
obvious reasons, you shouldn’t be), you might even run into nil
pointer dereferences.</p>
<p>This article isn’t supposed to mark cross-compilation as something
unusable, but be aware of its limitations and, if possible, compile
directly on the target platform.</p>
tag:dominik.honnef.co,2013-09-20:/go-tip/2013-09-20/What's happening in Go tip (2013-09-20)2013-09-20T11:54:50Z2013-09-20T11:54:50Z
<h3 id="go12">Go 1.2 Release Candidate 1 released</h3>
<p>This week’s article will be very short. Why? Because the first release
candidate of Go 1.2 has been released! And because that release
includes a list of all changes – many of which I have covered before
as well as all the ones I haven’t yet – there is little point to
reiterating them all again here.</p>
<p>Go check out the
<a href="https://groups.google.com/forum/#!topic/golang-nuts/iNX9jpufYqY">announcement</a>
as well as the <a href="http://tip.golang.org/doc/go1.2">release notes</a>.</p>
<p>The following sections will be of special interest to you, since they
cover changes I haven’t talked about yet.</p>
<ul>
<li><a href="http://tip.golang.org/doc/go1.2#use_of_nil">Use of nil</a>,</li>
<li><a href="http://tip.golang.org/doc/go1.2#cover">test coverage</a>,</li>
<li><a href="http://tip.golang.org/doc/go1.2#archive_tar_zip"><code>archive/tar</code> and <code>archive/zip</code> changes</a>,</li>
<li><a href="http://tip.golang.org/doc/go1.2#fmt_indexed_arguments">indexed arguments in <code>fmt</code></a></li>
<li><a href="http://tip.golang.org/doc/go1.2#gc_changes">Compiler and linker changes</a></li>
<li><a href="http://tip.golang.org/doc/go1.2#minor_library_changes">many minor changes</a></li>
</ul>
<p>With the first release candidate released and Go in a feature freeze,
there will probably not be any interesting changes to cover until Go
1.2. That’s why I will probably not be releasing any articles for the
weeks to come, unless there are interesting changes beyond normal bug
fixes.</p>
<p>Thanks for sticking with me for the past few weeks and see you once Go
tip becomes exciting again!</p>
tag:dominik.honnef.co,2013-09-13:/go-tip/2013-09-13/What's happening in Go tip (2013-09-13)2013-09-13T18:02:50Z2013-09-13T18:02:50Z
<p>This week we’ll concentrate on changes to <code>go vet</code> and <code>gofmt</code>, two
important parts of Go’s tooling, as well as some changes to cgo, some
of which are especially important to users of OS X.</p>
<h3 id="whats-happening">What’s happening</h3>
<ul>
<li><a href="#govet">Improved go vet</a></li>
<li><a href="#gofmt">Improved gofmt</a></li>
<li><a href="#cgo">Changes to cgo</a></li>
</ul>
<h4 id="govet">Improved go vet</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/12936046"><span class="cl">CL 12936046</span></a>, <a href="https://golang.org/cl/12811043"><span class="cl">CL 12811043</span></a>, <a href="https://golang.org/cl/10895043"><span class="cl">CL 10895043</span></a>, <a href="https://golang.org/cl/13102043"><span class="cl">CL 13102043</span></a>, <a href="https://golang.org/cl/10409047"><span class="cl">CL 10409047</span></a></p>
<p>A number of changes make <code>go vet</code> even more useful for checking the
correctness of your code:</p>
<p><code>go vet</code> <a href="https://golang.org/cl//12936046">will now warn</a> about recursive <code>Stringer</code> implementations. A
common mistake is writing code like the following:</p>
<pre class="go"><code>func (t T) String() string {
return fmt.Sprintf("%v", t)
}
</code></pre>
<p>This will lead to infinite recursion because the <code>%v</code> verb will also
use the <code>String()</code> method of <code>T</code>. <code>go vet</code> now correctly warns about
this.</p>
<p>Another <code>printf</code>-related improvement is
<a href="https://golang.org/cl//12811043">the detection of redundant calls to <code>String()</code> and <code>Error()</code></a>.
Given a type <code>T</code> that implements the <code>Stringer</code> interface and a value
<code>t</code> of type <code>T</code>, the following call to <code>String()</code> is redundant:</p>
<pre class="go"><code>fmt.Fprintf("%s", t.String())
</code></pre>
<p><code>go vet</code> will now suggest that this method call is redundant, because
<code>fmt</code> will already call it for us. The same applies to <code>Error()</code>.</p>
<p>One last <code>printf</code>-related improvement is <a href="https://golang.org/cl//10895043">improved checking of types</a>
for various verbs.</p>
<ol>
<li>
<p><code>%b</code> will no longer allow complex numbers, while <code>%e</code>,
<code>%f</code> and <code>%g</code> do.</p>
</li>
<li>
<p><code>%p</code> will only accept types that actually represent pointers (maps,
channels, slices, <code>unsafe.Pointer</code> and functions/methods.)</p>
</li>
<li>
<p><code>%s</code> will allow <code>[]byte</code> in addition to <code>string</code>.</p>
</li>
<li>
<p>Matching is recursive, which means that <code>map[K]V</code> and <code>[]T</code> will be
allowed if the verb matches <code>K</code>, <code>V</code> and <code>T</code>. That is, <code>go vet</code> will
for example not complain about</p>
<pre class="go"><code> fmt.Printf("%d", []int{1, 2, 3})
</code></pre>
</li>
</ol>
<p><a href="https://golang.org/cl//13102043">The next change</a> has nothing to do with <code>printf</code>, but function
comparison. In Go, functions and methods can be compared against nil
(with <code>==</code> and <code>!=</code>). This check, however, only makes sense when using
variables; named functions and methods will never be nil. More
importantly, Go 1.1 added method values, which can lead to code like
the following, which will compile but not do what you intended:</p>
<pre class="go"><code>if t.SomeMethod != nil {
// ...
}
</code></pre>
<p>We wanted to compare the return value of <code>SomeMethod()</code> with nil, but
we forgot the parentheses. This way, the check will always evaluate to
<code>true</code>. <code>go vet</code> will now warn about this kind of pointless
comparison.</p>
<p>The last <code>go vet</code> change <a href="https://golang.org/cl//10409047">is about shadowing</a>. One particularly nasty
class of programmer mistakes in Go is unintentional shadowing of
variables, especially in combination with the <code>:=</code> operator.</p>
<p>A new, optional, module that will check for probably unintentional
shadowing has been added. It is still considered an experimental
feature and too noisy, which is why it needs to be enabled explicitly,
by using <code>go tool vet -shadow *.go</code>.</p>
<p>An overview of all kinds of shadowing it does and does not warn about
can be found in the
<a href="https://code.google.com/p/go/source/browse/cmd/vet/testdata/shadow.go?repo=tools">included test data</a>.
Many of these errors are of stylistic nature to avoid programmer
confusion and do not actually affect the program, but that isn’t
unusual, <code>go vet</code> is often about good style as well as correctness.</p>
<h4 id="gofmt">Improved gofmt</h4>
<p>Relevant Cls: <a href="https://golang.org/cl/12837044"><span class="cl">CL 12837044</span></a></p>
<p><a href="https://golang.org/cl/12837044"><span class="cl">CL 12837044</span></a> improves <code>gofmt</code>’s handling of import statements. For one,
it improves sorting of imports to also consider import name and
comment, in addition to the import path. The major improvement,
however, is the automatic removal of duplicate imports.</p>
<p>Given the following file</p>
<pre class="go"><code>package main
import (
"fmt"
"fmt"
)
func main() { fmt.Println("test") }
</code></pre>
<p><code>gofmt</code> will remove the duplicate <code>fmt</code> import and only leave one of
them. This is especially interesting when using editors that allow
adding imports via a menu or keypress (for example go-mode in Emacs
with <code>C-c C-a</code>) because now you can blindly add an import when you
think you need it, without having to make sure that the import didn’t
exist already¹.</p>
<p><code>gofmt</code> will make sure to always delete the right entries. Imports must
have identical or no import names to be checked for duplicates, and
only imports with no attached comments will be removed.</p>
<p>Given</p>
<pre class="go"><code>import (
"fmt" // bar
"fmt" // foo
a "foo"
b "foo"
b "foo"
)
</code></pre>
<p><code>gofmt</code> will only delete the duplicate <code>b "foo"</code> import and leave the
rest alone.</p>
<p>¹: The Go compiler refuses to compile a program with duplicate
imports.</p>
<h4 id="cgo">Changes to cgo</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/12350044"><span class="cl">CL 12350044</span></a>, <a href="https://golang.org/cl/13413047"><span class="cl">CL 13413047</span></a></p>
<p>An obvious trend in OS X is the preference of clang over gcc, to the
point that Xcode 5 comes with a <code>gcc</code> that symlinks to clang. Because
of that, cgo will use clang instead of gcc on OS X 10.8 and later.</p>
<p>The move to clang also requires Go to provide its own wrapper for
<code>C.malloc</code> because <code>malloc</code> on OS X 10.9 is defined as taking a
<code>ulong</code> instead of a <code>size_t</code> – a change that doesn’t matter to C in
this case, but does matter to Go because of a stricter type system. A
side-effect of this wrapper is that it implements a feature asked for
in <a href="http://code.google.com/p/go/issues/detail?id=3403">Issue 3403</a>:
Crashing if <code>C.malloc</code> (and by extension <code>C.CString)</code> cannot allocate
memory instead of returning nil.</p>
<p>And in case it wasn’t clear enough: If you’re using OS X 10.9 / Xcode
5, you do need Go tip or cgo will not work.</p>
tag:dominik.honnef.co,2013-09-07:/go-tip/2013-09-07/What's happening in Go tip (2013-09-07)2013-09-06T23:20:10Z2013-09-06T23:20:10Z
<p>Welcome back to yet another update on Go tip. We’ll be looking at
another batch of smaller changes from all over the place; there’s
something for everyone.</p>
<p>And if you’re waiting for some juicier changes, let me just say that
they’re on my radar, but I am reserving them for the finale. Well,
really for the big hole that the feature freeze will cause.</p>
<p>Also, I’m sorry for releasing this article a day late. I hope I didn’t
ruin anyone’s Friday at work :-( – I will try to release the next
article on time again.</p>
<h3 id="whats-happening">What’s happening</h3>
<p>This week we will be looking at:</p>
<ul>
<li><a href="#compress">Reusable Writer in compress/flate</a></li>
<li><a href="#template">Improvements to text/template</a></li>
<li><a href="#sql">More control over database/sql’s connection pool</a></li>
<li><a href="#netgo">The netgo build tag</a></li>
<li><a href="#tooling">Small tooling changes</a></li>
</ul>
<h4 id="compress">Reusable Writer in compress/flate</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/12953048"><span class="cl">CL 12953048</span></a>, <a href="https://golang.org/cl/13171046"><span class="cl">CL 13171046</span></a>, <a href="https://golang.org/cl/13435043"><span class="cl">CL 13435043</span></a>, <a href="https://golang.org/cl/12894043"><span class="cl">CL 12894043</span></a></p>
<p>Continuing the trend of less garbage, the Writers in <code>compress/flate</code>,
<code>compress/zlib</code> and <code>compress/gzip</code> have a new <code>Reset()</code> method, which
allows resetting a Writer’s internal state without needing to allocate
new memory. According to
<a href="http://code.google.com/p/go/issues/detail?id=6138">issue 6138</a>, this
saves 1.4 MB of allocations per Writer you would have to create.</p>
<p><a href="https://golang.org/cl//12894043">In a somewhat related change</a>, a test in <code>archive/zip</code>
has been brought down from 34 seconds to 0.23 seconds, by avoiding an
allocation in <code>archive/zip</code> itself and by not using <code>compress/flate</code>
in the test at all. While this hardly affects people who use
<code>archive/zip</code>, it’s a welcome speedup to anyone who has to run Go’s
entire test suite.</p>
<h4 id="template">Improvements to text/template</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/13091045"><span class="cl">CL 13091045</span></a>, <a href="https://golang.org/cl/13509046"><span class="cl">CL 13509046</span></a>, <a href="https://golang.org/cl/13327043"><span class="cl">CL 13327043</span></a></p>
<p>Two small additions to <code>text/template</code> (and, by extension,
<code>html/template</code>) might make writing templates just a bit easier.</p>
<p>The <a href="https://golang.org/cl//13327043">first addition</a> is <code>{{else if}}</code>, which can be
used instead of <code>{{else}}{{if}}</code>. Naturally this also means one less
level of nesting, so that this:</p>
<pre><code>{{if A}}
a
{{else}}
{{if B}}
b
{{end}}
{{end}}
</code></pre>
<p>will turn into this:</p>
<pre><code>{{if A}}
a
{{else if B}}
b
{{end}}
</code></pre>
<p>A welcome simplification!</p>
<p>The <a href="https://golang.org/cl//13091045">second addition</a> are built-in comparison functions
for basic types, that is booleans, (unsigned) integers, floats,
complex numbers and strings. The <code>eq</code> (equality) comparison is special
in that it accepts a variable number of arguments (at least two),
comparing the first argument to all the other arguments in an OR
relation. So <code>{{eq arg1 arg2 arg3}}</code> would be equivalent to
<code>arg1==arg2 || arg1==arg3</code>. It is important to note that <code>eq</code> is an
ordinary function, so there’s no short circuit and all arguments will
be evaluated.</p>
<p>Furthermore, these functions are a bit more lax than the Go type
system, allowing comparison between different sizes of the same type
(e.g. <code>int32</code> and <code>int16</code>), as well as different types with the same
underlying type (e.g. <code>type Foo int</code> and <code>int</code>)</p>
<p>Here’s a table of the new functions as well as their equivalent Go
syntax:</p>
<table>
<thead>
<tr>
<th>Operator</th>
<th>Function</th>
</tr>
</thead>
<tbody>
<tr>
<td>==</td>
<td>eq</td>
</tr>
<tr>
<td>&gt;=</td>
<td>ge</td>
</tr>
<tr>
<td>&gt;</td>
<td>gt</td>
</tr>
<tr>
<td>&lt;=</td>
<td>le</td>
</tr>
<tr>
<td>&lt;</td>
<td>lt</td>
</tr>
<tr>
<td>!=</td>
<td>ne</td>
</tr>
</tbody>
</table>
<h4 id="sql">More control over database/sql’s connection pool</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/10726044"><span class="cl">CL 10726044</span></a></p>
<p>A long awaited improvement (at least by some people) is more control
over the connection pool managed by <code>database/sql</code>. Thanks to <a href="https://golang.org/cl/10726044"><span class="cl">CL 10726044</span></a> it is now possible to limited the total number of
simultaneous open connections, by using
<a href="http://tip.golang.org/pkg/database/sql/#DB.SetMaxOpenConns"><code>(*DB).SetMaxOpenConns(int)</code></a>.</p>
<p>It is an important addition to the already present
<a href="http://tip.golang.org/pkg/database/sql/#DB.SetMaxIdleConns"><code>SetMaxIdleConns</code></a>.
Limiting the number of idle connections wouldn’t prevent an unbounded
number of connections to be created if they were needed, it would only
close most of them once done. This could still lead to more
connections than wanted, hitting database or resource limits.</p>
<p>By setting the maximum number of open connections, however, we ensure
that we will never exceed that number of connections. Also, the
maximum number of open connections can be bigger than the maximum
number of idle connections, allowing bursts without keeping all
connections around. This should remove the need for custom pooling,
limits and semaphores in a lot of database code.</p>
<h4 id="netgo">The netgo build tag</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/7100050"><span class="cl">CL 7100050</span></a></p>
<p>Go binaries, if not using cgo, are usually statically linked. When
importing the <code>net</code> package (directly or indirectly), however, one
ends up with a dynamically linked binary, linking against libc. That’s
because the <code>net</code> package, by default, uses cgo to use the system’s
resolver, to support for example LDAP. The <code>net</code> package does come
with a pure-Go resolver. Until now, it was only used when cgo wasn’t
enabled, which means compiling the standard library (or at least
<code>net</code>) as well as your own package with <code>CGO_ENABLED=0</code>. But that has
the downside of disabling cgo for your entire project, which might not
be desirable.</p>
<p>A new build tag, <code>netgo</code>, alleviates this problem by providing a way
to build the standard library and your project with a pure-Go <code>net</code>
package while still enabling cgo for other uses. In order to reinstall
the standard library with <code>netgo</code>, you run</p>
<pre><code>go install -a -tags netgo std
</code></pre>
<p>You also need to specify <code>-tags netgo</code> when building your own package,
otherwise cgo will overrule and rebuild <code>net</code> every time you build
your package. This does allow installing a <code>netgo</code> standard library as
the default with cgo as a working alternative, but it also has the
added cost of recompiling the <code>net</code> package every time you do not use
the <code>netgo</code> tag.</p>
<h4 id="tooling">Small tooling changes</h4>
<p>There are two small changes to tooling that are worth mentioning.</p>
<h5 id="deletion-of-go-doc">Deletion of go doc</h5>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/12974043"><span class="cl">CL 12974043</span></a></p>
<p>The first is the deletion of <code>go doc</code>. Now, mind you, <code>go doc</code> is not
the same as <code>godoc</code>. <code>go doc</code> accepts different (more limited)
arguments than <code>godoc</code> and apparently the consensus was to remove it
and focus on improving <code>godoc</code> instead.
<a href="http://code.google.com/p/go/issues/detail?id=4849">Issue 4849</a> kind
of leads up to the deletion and <a href="https://golang.org/cl/12974043"><span class="cl">CL 12974043</span></a> is ultimatively deleting
it.</p>
<p>This change goes to show that Go’s compatibility promises only cover
the language and standard library, not tools.</p>
<h5 id="go-get-for-git-repositories">go get for git repositories</h5>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/12923043"><span class="cl">CL 12923043</span></a>, some more I unfortunately do not have the
numbers of.</p>
<p>If you ever used <code>go get</code> on a git repository and later went to look
at the actual clone, you might have noticed that HEAD was in a
detached state, and that every <code>go get -u</code> would put it back in
detached state, with HEAD pointing at a specific commit on a remote
branch.</p>
<p>And even though that isn’t technically a problem (you could still just
check out the master branch), it would lead to confusion and
occasional loss of data when one wasn’t aware of the fact.</p>
<p>In Go 1.2, however, <code>go get</code> will always create a “proper” (as in
expected) clone, with an active master branch, and <code>go get -u</code> will
use <code>git pull</code> and all its effects, such as aborting in case of
conflicts or uncommitted changes. Old clones from before Go 1.2 will
<a href="https://golang.org/cl//12923043">automatically be updated</a> to the new format by <code>go get
-u</code>.</p>
tag:dominik.honnef.co,2013-08-30:/go-tip/2013-08-30/What's happening in Go tip (2013-08-30)2013-08-30T18:28:10Z2013-08-30T18:28:10Z
<p>Welcome back to another week of Go tip. This time we’ll mostly be
focusing on smaller changes and API improvements. Enjoy!</p>
<h3 id="whats-happening">What’s happening</h3>
<p>In this week’s article we will be looking at:</p>
<ul>
<li><a href="#better-citizen">Go becoming a better citizen</a></li>
<li><a href="#gif">Support for encoding GIF images</a></li>
<li><a href="#zip">User-extensible compression methods for archive/zip</a></li>
<li><a href="#hash">Easier hashing with MD5 and SHA</a></li>
<li><a href="#test-dependencies"><code>go get</code> &amp; test dependencies</a></li>
<li><a href="#subrepos">Subrepositories</a></li>
</ul>
<h4 id="better-citizen">Go is becoming a better citizen</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/12541052"><span class="cl">CL 12541052</span></a>, <a href="https://golang.org/cl/12650045"><span class="cl">CL 12650045</span></a>, <a href="https://golang.org/cl/13037043"><span class="cl">CL 13037043</span></a>, <a href="https://golang.org/cl/13038043"><span class="cl">CL 13038043</span></a>, <a href="https://golang.org/cl/13348045"><span class="cl">CL 13348045</span></a></p>
<p>Many of the design decisions behind Go would occasionally lead to
scenarios where a misbehaving Go program could destabilize the whole
operating system. For example, Go’s split stacks could grow
indefinitely, until swap death occurs. Another example are goroutines
and how they’re being distributed onto system threads. While languages
with more explicit threading make it (somewhat) obvious when and how
many threads are going to be created, goroutines hide that fact. One
particularly dangerous fact is that if a syscall or a call to cgo
blocks, it will receive its own thread automatically. This means that
thousands of concurrent blocking calls of that kind will lead to
thousands of threads. One realistic scenario of this happening would
be thousands of concurrent DNS lookups.</p>
<p>These issues have been known and acknowledged for a while, but Go 1.2
will finally address them, by introducing various limitations, some of
which are configurable, while others are not.</p>
<p>To tackle infinite recursion,
<a href="https://golang.org/cl//12541052">a fixed limit on the size of stacks has been added</a>.
For 64-bit systems, that limit is 1 GB and for 32-bit systems it is
250 MB. Do note that the specific limits might still be adjusted if
they turn out to be “implausibly large”.</p>
<p>A somewhat related change is <a href="https://golang.org/cl/12650045"><span class="cl">CL 12650045</span></a>, which moves all variables
that are bigger than 10 MB to the heap. Admittedly, that change was
originally made to
<a href="http://code.google.com/p/go/issues/detail?id=6077">avoid an issue in the compiler</a>,
but it will also make sure that big allocations won’t hit the stack
size limit.</p>
<p>On the topic of limits, there is now
<a href="https://golang.org/cl//13037043">a limit on the number of threads that may exist</a>.
Exceeding that limit will result in a crash (as opposed to trying to
stay within the limit, which could lead to deadlocks). The limit is
set to 10,000 threads by default, but is configurable through
<code>runtime/debug.SetMaxThreads()</code>. This limit addresses the issue of
syscalls or cgo calls unexpectedly leading to too many threads.</p>
<p>A more specific case of limiting threads is that of DNS lookups in the
<code>net</code> package, which has now
<a href="https://golang.org/cl//13038043">been limited to a fixed 500 concurrent lookups</a>. This
change has been made in addition to the
<a href="https://golang.org/cl//10079043">coalescence of inflight DNS lookups</a> that I mentioned
last week. Not only will these two changes ensure that you won’t
bombard your system with new threads, it also works around the issue
that some resolvers cannot handle more than 1024 concurrent lookups.</p>
<p>This set of changes reduces a lot of the “risks” and hidden growth
behaviors that Go has, by enforcing upper limits. It is, however,
still important that you, as the programmer, are aware of these
limits. While forcefully crashing your program makes sure that the
system stays stable, it’s not exactly a good thing to happen in
production systems. You will still have to avoid infinite recursion
for obvious reasons, and you should still keep track of how many
potential threads you are going to create. While the <code>net</code> package
feels responsible for maintaining a hard limit, other packages do not
and it usually is the programmer’s responsibility.</p>
<h4 id="gif">Support for encoding GIF images</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/10977043"><span class="cl">CL 10977043</span></a>, <a href="https://golang.org/cl/10896043"><span class="cl">CL 10896043</span></a>, <a href="https://golang.org/cl/10890045"><span class="cl">CL 10890045</span></a></p>
<p>While it has been possible to decode GIF images for a long time,
encoding GIF images wasn’t possible until now.</p>
<p>Go 1.2 will add a GIF encoder, and because encoding GIF images
requires quantization, it also adds facilities for implementing
quantizers. For this, two new interfaces have been added to the
<a href="http://tip.golang.org/pkg/image/draw/"><code>image/draw</code></a> package.</p>
<p>The first interface is
<a href="http://tip.golang.org/pkg/image/draw/#Drawer"><code>Drawer</code></a>, which
encapsulates the idea of drawing a section of a source image onto a
destination image. A concrete drawer could for example implement
<a href="https://en.wikipedia.org/wiki/Floyd%E2%80%93Steinberg_dithering">Floyd-Steinberg error diffusion</a>.
That one is actually included with Go and used by default by the GIF
encoder.</p>
<p>The second interface is the
<a href="http://tip.golang.org/pkg/image/draw/#Quantizer"><code>Quantizer</code></a>
interface that includes the single method <code>Quantize</code>, whose
responsibility will be to take an image and return a color palette of
wanted size. This palette will be used by the GIF encoder to produce a
paletted images. Go does not come with an implementation of Quantizer,
though, and the GIF encoder will use the
<a href="http://plan9.bell-labs.com/magic/man2html/6/color">Plan 9 palette</a> by
default. Alternatively there’s also the
<a href="https://en.wikipedia.org/wiki/Web_colors#Web-safe_colors">“Web-safe palette”</a>,
aka the Netscape Color Cube. Both palettes can be found in the
<a href="http://tip.golang.org/pkg/image/color/palette/"><code>image/color/palette</code></a>
package.</p>
<p>And because everybody likes animated GIFs, the new encoder also
includes a function for producing those:
<a href="http://tip.golang.org/pkg/image/gif/#EncodeAll"><code>EncodeAll</code></a>.</p>
<h4 id="zip">User-extensible compression methods for archive/zip</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/12421043"><span class="cl">CL 12421043</span></a></p>
<p><a href="https://golang.org/cl/12421043"><span class="cl">CL 12421043</span></a> improves <code>archive/zip</code> to allow adding new compression
methods without having to modify <code>archive/zip</code> directly. While
previously a hard-coded switch for known and implemented methods was
used (at the time only
<a href="https://en.wikipedia.org/wiki/DEFLATE">DEFLATE</a> and storing without
compression), it now allows for registering new methods (compressors
and decompressors separately).</p>
<p>In total the ZIP specification allows 11 different compression
methods, out of which Go only supports 2. With this change, adding new
methods will be possible without having to modify <code>archive/zip</code> directly. It
also allows swapping out implementations for better ones¹.</p>
<p>¹: At least as long as <code>archive/zip</code> doesn’t include them, because it
is not possible to register a compression method that has already been
registered.</p>
<h4 id="hash">Easier hashing with MD5 and SHA</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/10624044"><span class="cl">CL 10624044</span></a>, <a href="https://golang.org/cl/10571043"><span class="cl">CL 10571043</span></a>, <a href="https://golang.org/cl/10629043"><span class="cl">CL 10629043</span></a>, <a href="https://golang.org/cl/10630043"><span class="cl">CL 10630043</span></a></p>
<p>Like a lot of things in Go that transform streams of bytes, the
packages for cryptographic hashes (MD5, SHA1 etc) are implemented as
<code>io.Writer</code>’s. You create a hash, you write data to it, you then ask
for the result. In addition to that, the <code>hash.Hash</code> interface has
been designed with performance in mind, which is why the <code>Sum()</code>
method takes a slice as input, to allow buffer reuse. And even though
passing <code>nil</code> is possible, it’s still a somewhat involved process and
often leads to confusion for newbies who are used to easier solutions
a la “take these bytes, give me the result”.</p>
<p><a href="http://play.golang.org/p/bunKo0g3u1">The following piece of code</a>
demonstrates the current process:</p>
<pre class="go"><code>data := []byte(`hello, world`)
h := sha1.New()
h.Write(data)
hash := h.Sum(nil)
fmt.Printf("H(data) = %x\n", hash)
// Output: H(data) = b7e23ec29af22b0b4e41da31e868d57226121c84
</code></pre>
<p>And because the Go team agrees that there should be an easier way to
calculate hashes, Go 1.2 will add simple package-level functions that
take the bytes to hash as input and return the hash. The previous
example can be reduced <a href="http://play.golang.org/p/H3x5t8JEq7">to this</a>:</p>
<pre class="go"><code>data := []byte(`hello, world`)
hash := sha1.Sum(data)
fmt.Printf("H(data) = %x\n", hash)
// Output: H(data) = b7e23ec29af22b0b4e41da31e868d57226121c84
</code></pre>
<p>
<del>Please remember that even though the new API is easier to use, you
should only use it when performance is not a concern and you are only
calculating one/a few hashes. Otherwise, using the old interface, with
reused buffers, will prove beneficial.</del>
</p>
<p><code>Sum()</code> has been implemented with “no allocations” in mind, which is
why it returns an array instead of a slice. Given the fact that the
only difference between the new package-level <code>Sum()</code> and the old API
is the lack of appending to a slice¹, the new function should be just
as fast, if not marginally and almost unmeasurably faster.</p>
<p>¹: Even though <code>Sum()</code> will have to create a new digest for every
invocation, so will the old API internally, since it operates on a
copy of the digest.</p>
<h4 id="test-dependencies"><code>go get</code> &amp; test dependencies</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/12566046"><span class="cl">CL 12566046</span></a></p>
<p><a href="https://golang.org/cl/12566046"><span class="cl">CL 12566046</span></a> adds the <code>-t</code> flag to <code>go get</code>, which will make it
download test dependencies, something
<a href="http://code.google.com/p/go/issues/detail?id=5126">that wasn’t possible before</a>.
Do note that this will not download test dependencies recursively, but
only for the specified package.</p>
<h4 id="subrepos">Subrepositories</h4>
<p>The Go repository includes a number of subrepositories, most
prominently
<a href="http://code.google.com/p/go/source/browse/?repo=tools">go.tools</a>,
which contain some components of Go that are either not directly part
of it or that are being developed independently of Go itself, such as
godoc, which has recently been moved to said subrepository.
<a href="https://plus.google.com/u/0/+AndrewGerrand/">Andrew Gerrand</a>
correctly pointed out that I should be taking a look at these
subrepositories as well. And I will, in due time, as soon as I catch
up with the changes on the main repository. Apparently there have been
some nice changes to <code>go vet</code> that we will be checking out soon!</p>
tag:dominik.honnef.co,2013-08-23:/go-tip/2013-08-23/What's happening in Go tip (2013-08-23)2013-08-23T04:10:35Z2013-08-23T04:10:35Z
<p>Last week I published <a href="/go-tip/2013-08-15/">the first article</a> in an
ongoing series on changes in Go tip. It was met with a lot of positive
feedback, so here’s the second article. Thank you for your support and
I hope you enjoy this article just as much as the previous one.</p>
<h3 id="whats-happening">What’s happening</h3>
<p>This time we’ll be looking at:</p>
<ul>
<li><a href="#slicing">A new slicing syntax</a></li>
<li><a href="#performance">Performance improvements</a></li>
<li><a href="#p256">Fast, constant-time P-256 elliptic curves</a></li>
<li><a href="#godoc">Where has godoc gone?</a></li>
</ul>
<h4 id="slicing">A new slicing syntax</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/10743046"><span class="cl">CL 10743046</span></a>, <a href="https://golang.org/cl/12931044"><span class="cl">CL 12931044</span></a></p>
<p>The first change we will be looking at is probably the most
controversial addition to Go 1.2: A new slicing syntax that allows
setting the capacity of a slice. But before going into why it’s
controversial let’s see what it’s about first.</p>
<p>Everyone should be familiar with the slicing operation <code>mySlice[i:j]</code>,
creating a new slice, ranging from <code>i</code> to <code>j</code>. <code>mySlice</code> and the new
slice will share the same backing array and manipulating one slice
will affect the other. That’s all pretty well known and expected
behaviour.</p>
<p>What some people might not realize is that the two slices will share
the capacity, too. A slice has a length and a capacity; the number of
elements it can see right now and the number of elements in can
potentially access. The capacity is mostly relevant when using
<code>append()</code>¹: When appending to a slice, it first checks if there’s
enough room left in the underlying array (the capacity). If there is,
it creates a new slice with bigger length, pointing to the same
backing array, the same memory.</p>
<p>¹: One can also manually reslice past the length, if the capacity
allows it. This is basically what <code>append()</code> is doing.</p>
<p>Consider the
<a href="http://play.golang.org/p/rvV8gLTPvZ">following piece of code</a>:</p>
<pre class="go"><code>orig := make([]int, 10, 20)
subSlice := orig[5:10]
fmt.Printf("len(orig) = %d, cap(orig) = %d, len(subSlice) = %d, cap(subSlice) = %d\n",
len(orig), cap(orig), len(subSlice), cap(subSlice))
// Output: len(orig) = 10, cap(orig) = 20, len(subSlice) = 5, cap(subSlice) = 15
orig = append(orig, 1)
fmt.Println(orig)
// Output: [0 0 0 0 0 0 0 0 0 0 1]
subSlice = append(subSlice, 2)
fmt.Println(orig)
// Output: [0 0 0 0 0 0 0 0 0 0 2]
</code></pre>
<p>It demonstrates that <code>subSlice</code>, albeit being sliced to a specific
window of <code>orig</code>, still has access to the same kind of memory – minus
the first 5 elements, since slices cannot grow backwards. It also
demonstrates that using <code>append()</code> leads to “weird” results, where
we’d overwrite elements in the original slice.</p>
<p>Now, why is this an issue in real code? Imagine you’re writing your
own memory allocator based on <code>[]byte</code> – something that’s common when
dealing with a lot of tiny allocations that shouldn’t slow down the
GC. When the user asks for <code>N</code> bytes of memory, you return a slice
that is a slice somewhere from that <code>[]byte</code>, with length <code>N</code>. Now the
user does something stupid: He appends to it. As we’ve seen before,
this append will “leak” into memory beyond the length of the slice,
and beyond what the memory allocator intended. You just overwrote
someone else’s memory!</p>
<p>A safe alternative that still allows implementing a custom memory
allocator but preventing arbitrary memory corruption would be to limit
the capacity of the slice we return, so that an append wouldn’t modify
memory it shouldn’t. And that safe alternative is the new slicing
syntax: <code>mySlice[i:j:k]</code> – The first two elements behave as usual,
slicing from <code>i</code> to <code>j</code>. The third element affects the capacity, where
the capacity will be calculated as <code>k - i</code>.</p>
<p><code>k - i</code> as the capacity simply means that <code>k - 1</code> is the absolute
index into the underlying array that will be reachable by the new
slice, same as <code>j - 1</code> being the index for the length. By making <code>k</code>
equal to <code>j</code> you create a slice that cannot look past its length. And
now we can write an allocator that returns <code>N</code> bytes of memory and
doesn’t allow you to look any further.</p>
<p>In the introduction I wrote that this addition is controversial. Maybe
you can already guess why: If you’ve kept thinking to yourself “why
would I need this”, you guessed why. This feature is used very rarely.
Only some very specific use-cases require it, and looking at the
standard library won’t return many uses of it, either. That doesn’t
mean the feature shouldn’t exist at all, since it does solve a valid
problem, but it has been argued that the slicing syntax shouldn’t be
extended. Functions like <code>setcap()</code> have been suggested to avoid
adding new syntax to Go, but in the end the Go team went with the new
slicing syntax.
<a href="https://groups.google.com/forum/#!msg/golang-dev/cL9CCxa2jUc/K-cAjmQ9NCUJ">Here’s the full discussion</a>
if you’re interested in the back and forth. And yes, this will be in
Go 1.2.</p>
<p>There’s also
<a href="https://docs.google.com/document/d/1GKKdiGYAghXRxC2BFrSEbHBZZgAGKQ-yXK-hRKBo0Kk/pub">a design document</a>,
describing the original motivations and concerns.</p>
<h4 id="performance">Performance improvements &amp; Garbage be gone</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/11573043"><span class="cl">CL 11573043</span></a>, <a href="https://golang.org/cl/9129044"><span class="cl">CL 9129044</span></a>, <a href="https://golang.org/cl/12894043"><span class="cl">CL 12894043</span></a>, <a href="https://golang.org/cl/8179043"><span class="cl">CL 8179043</span></a>, <a href="https://golang.org/cl/9492044"><span class="cl">CL 9492044</span></a>, <a href="https://golang.org/cl/9432046"><span class="cl">CL 9432046</span></a>, <a href="https://golang.org/cl/8819049"><span class="cl">CL 8819049</span></a>, <a href="https://golang.org/cl/9459044"><span class="cl">CL 9459044</span></a>, <a href="https://golang.org/cl/12603049"><span class="cl">CL 12603049</span></a>, <a href="https://golang.org/cl/12708046"><span class="cl">CL 12708046</span></a>,
<a href="https://golang.org/cl/10079043"><span class="cl">CL 10079043</span></a>, <a href="https://golang.org/cl/8670044"><span class="cl">CL 8670044</span></a>, <a href="https://golang.org/cl/12680046"><span class="cl">CL 12680046</span></a>, <a href="https://golang.org/cl/12694048"><span class="cl">CL 12694048</span></a>, <a href="https://golang.org/cl/11874043"><span class="cl">CL 11874043</span></a>, <a href="https://golang.org/cl/12072045"><span class="cl">CL 12072045</span></a>, <a href="https://golang.org/cl/9915043"><span class="cl">CL 9915043</span></a>, <a href="https://golang.org/cl/12662043"><span class="cl">CL 12662043</span></a>, <a href="https://golang.org/cl/9462044"><span class="cl">CL 9462044</span></a></p>
<p>Go 1.2 will see a lot of performance improvements. These improvements
fall into two categories: Raw higher speed thanks to better code and
better compilers, and less work for the memory allocator and garbage
collector thanks to less garbage.</p>
<p>Brad Fitzpatrick, in his quest for high performance HTTP, has been
doing a lot of work on the <code>net/http</code> package, and related packages
you often use with it. Most of his changes fall into the “less
garbage” category. Since most changes are rather simple, I’ll only
list their CL numbers so you can check them out. They all include
benchmarks in their description: <a href="https://golang.org/cl/9129044"><span class="cl">CL 9129044</span></a> (faster JSON encoding), <a href="https://golang.org/cl/12894043"><span class="cl">CL 12894043</span></a> (faster ZIP compression), <a href="https://golang.org/cl/8179043"><span class="cl">CL 8179043</span></a>, <a href="https://golang.org/cl/9492044"><span class="cl">CL 9492044</span></a> and <a href="https://golang.org/cl/9432046"><span class="cl">CL 9432046</span></a> (several HTTP improvements)</p>
<p>There are, however, some CLs that are particularly interesting. One
set of CLs are <a href="https://golang.org/cl/8819049"><span class="cl">CL 8819049</span></a>, <a href="https://golang.org/cl/9459044"><span class="cl">CL 9459044</span></a> and <a href="https://golang.org/cl/12603049"><span class="cl">CL 12603049</span></a> – The first two
add a buffer pool and buffer reuse to <code>bufio</code> and the last one removes
said buffer pool and buffer reuse, replacing it with a simple <code>Reset</code>
method. The story behind this is that these buffers would be handed to
users, and these could, by ignoring the API, potentially corrupt data
of other users, leading to hard to debug issues that are alike to “use
after free” errors, which Go explicitly avoids by not providing a
manual way of freeing memory. Additionally, it would set a precedent
of unnecessarily complex code for the sake of some additional
performance.</p>
<p>The newly added <code>Reset</code> method manages to reach similar performance
improvements, at the cost of forcing the user to actively reuse
buffers instead of relying on the package for it. <code>net/http</code> would be
one of these users, making use of <code>Reset</code> in <a href="https://golang.org/cl/12708046"><span class="cl">CL 12708046</span></a>. Do note that
the included benchmark is relative to the <code>bufio</code> pool, not Go 1.1.</p>
<p>For the original discussion around this, see
<a href="http://code.google.com/p/go/issues/detail?id=6086">issue 6086</a> by
Russ Cox</p>
<p>One last change in Brad’s name is <a href="https://golang.org/cl/10079043"><span class="cl">CL 10079043</span></a>, which adds coalescing
of duplicate in-flight DNS lookups. Or put differently: If doing the
same DNS lookup a hundred times before one of them finishes, only one
actual request will be made to servers.</p>
<p>But not only Brad did some work, other people contributed great
improvements as well.</p>
<p>The most notable one would be <a href="https://golang.org/cl/8670044"><span class="cl">CL 8670044</span></a>, which implements an
integrated network poller for Windows, leading to 30% performance
improvements on that platform for networking operations – A luxury
that Go 1.1 already brought for Linux.</p>
<p>Another notable change would be <a href="https://golang.org/cl/11573043"><span class="cl">CL 11573043</span></a>, which considerably speeds
up <code>sync.Cond</code> – And completely gets rid of memory allocations in the
process.</p>
<p><a href="https://golang.org/cl/12680046"><span class="cl">CL 12680046</span></a> and <a href="https://golang.org/cl/12694048"><span class="cl">CL 12694048</span></a> add fast routes to <code>encoding/binary</code>,
handling slices of integer types, making these common types a lot
faster and cheaper.</p>
<p>DES encryption has become five times faster (which, sadly, is still
very slow) thanks to <a href="https://golang.org/cl/11874043"><span class="cl">CL 11874043</span></a> and <a href="https://golang.org/cl/12072045"><span class="cl">CL 12072045</span></a>.</p>
<p>bzip2 decompresses 30% faster (<a href="https://golang.org/cl/9915043"><span class="cl">CL 9915043</span></a>) and the DNS client in the
<code>net</code> package produces ~350 bytes less of garbage (<a href="https://golang.org/cl/12662043"><span class="cl">CL 12662043</span></a>).</p>
<p>And last but not least, <code>io.Copy</code>
<a href="https://golang.org/cl//9462044">now prioritizes WriterTo over ReaderFrom</a>, which can
lead to <del>tremendous</del> noticeable performance improvements (and
virtually no-ops in the case of <code>ioutil.Discard</code>) when types implement
both functions.</p>
<h4 id="p256">Fast, constant-time P-256 elliptic curves</h4>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/10551044"><span class="cl">CL 10551044</span></a></p>
<p>One of my readers pointed out this very interesting change which adds
a constant-time P-256 implementation, which at the same time happens
to be a lot faster than the previous implementation.</p>
<p>And while performance is nice, the really important bit here is
“constant-time”: Since the function takes the same amount of time for
all possible inputs, it avoids being vulnerable to
<a href="http://en.wikipedia.org/wiki/Timing_attack">timing attacks</a>, a form
of
<a href="http://en.wikipedia.org/wiki/Side_channel_attack">side channel attacks</a>
in which timing information is used to break a cipher.</p>
<h4 id="godoc">Where has godoc gone?</h4>
<p>Using Go tip always requires you to be more aware of the development
process and to expect breakage, weird behaviour and sudden changes.</p>
<p>Nevertheless, not everyone who uses Go tip has the time to check its
entire development history, especially if you’ve been told to “try Go
tip and see if your problem has been fixed”. As such, a fair number of
people have recently been surprised and confused when, after compiling
Go tip, <code>godoc</code> wouldn’t work anymore, either because there was no
<code>godoc</code> binary, or because it would print an error like this one:</p>
<pre><code>readTemplate: open /Users/rsc/g/go/lib/godoc/codewalk.html: no such file or directory
</code></pre>
<p>In either case, the fix is to run <code>go get
code.google.com/p/go.tools/cmd/godoc</code> – <code>godoc</code> has been moved to the
<code>go.tools</code> subrepository.</p>
<h3 id="the-future">The future</h3>
<p>Phew. We’re still catching up with all the development that happened
since Go 1.1, but we’re slowly getting there. Luckily current Go tip
development has slowed down, not yielding a lot of interesting
changes. And where important things change, like the <code>godoc</code> change,
we mix them into the articles.</p>
<p>Attentive readers might have noticed that the previous article
promised a look at shared linking support. Unfortunately it didn’t
make its way into this article, to make room for more important
changes and because the change to slicing syntax already took a bit to
digest.</p>
<p>What’s planned for next week? More changes!</p>
tag:dominik.honnef.co,2013-08-15:/go-tip/2013-08-15/What's happening in Go tip (2013-08-15)2013-08-15T17:52:52Z2013-08-15T17:52:52Z
<p>This is the first part of an ongoing series on changes and
developments in Go tip, i.e. the development branch of Go.</p>
<p>In this series I intend to look at noteworthy recent CLs¹ and
proposals and comment on their significance and impact. For CLs, I
will be concentrating on those that have either been already committed
or that have been rejected, if they had been of significance. For
proposals I will look at their current state and summarize the
discussions where possible.</p>
<p>¹: CL is short for Change List and is, in essence, what GitHub calls a
pull request. Go maintains a linear history of changes, and every
change needs to be submitted as a CL and peer reviewed before it can
be committed to the repository.</p>
<h3 id="the-format">The format</h3>
<p>I am intending to post weekly updates, talking about the changes that
happened during that week. Close to the release of a new version of Go
there will naturally be fewer/no interesting changes. In that case I
might either pause posting updates, or post “Nothing to see this time”
updates.</p>
<p>Since this is the first post in the series and due to unlucky timing
it will have special status. For one, there have been <em>a lot</em> of
changes between Go 1.1 and tip, and catching up on all of them will
require more than one article. “Luckily”, Go is nearing its feature
freeze (September 1), and a lot of the Go team will be on summer
vacation until the middle of September. This means there will be
little (accepted) change, allowing me to spend some more time talking
about past changes.</p>
<h3 id="disclaimer">Disclaimer</h3>
<p>There are some important things to note before diving into the
article.</p>
<p>The first is that I am not part of the core Go team and as such my
interpretations might sometimes be wrong. I am trying my best to be as
accurate as possible, but if I do mess up, let me know and I will
correct any errors.</p>
<p>Second, just because something got committed doesn’t mean that it will
actually be part of the next release of Go. It might be reverted for
various reasons or it might be part of the second next release.
Reverted changes will be covered in future articles, and I will try my
best to annotate changes that are not intended for the next release.</p>
<h3 id="whats-happening">What’s happening</h3>
<p>In the first article in the series I will be looking at:</p>
<ul>
<li>C++ support in cgo (kind of)</li>
<li>Function pointers in cgo</li>
<li>Improved marshaling through the addition of new interfaces</li>
<li>Preemptive scheduling</li>
</ul>
<h4 id="cgo">cgo</h4>
<h5 id="c-support-in-cgo-kind-of">C++ support in cgo, kind of</h5>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/8248043"><span class="cl">CL 8248043</span></a></p>
<p><a href="https://golang.org/cl/8248043"><span class="cl">CL 8248043</span></a> adds support to the cgo tool to include C++ files in
the build process. It adds the CXXFiles field to the internal
representation of packages and support for the CPPFLAGS and
CXXFLAGS variables.</p>
<p>Now, this doesn’t mean that you can directly use C++ classes and
methods in Go. After all, these are still hard to map to the
concepts of Go and especially name mangling wouldn’t make your
life easy. Instead you will still have to write C wrappers for
the C++ APIs and bind to those wrappers instead. What this change
allows, however, is to compile those wrappers during the build
process, instead of being forced to add a precompiled library to
the project.</p>
<h5 id="function-pointers-in-cgo">Function pointers in cgo</h5>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/9835047"><span class="cl">CL 9835047</span></a></p>
<p><a href="https://golang.org/cl/9835047"><span class="cl">CL 9835047</span></a> adds partial support for function pointers to cgo.
While it won’t permit calling functions from function pointers
directly, it will allow storing and passing them around, giving
them a proper type. Therefore it will be possible to receive
function pointers from C, store them in Go, and pass them back to
C to let C call them.</p>
<h4 id="improved-marshaling">Improved marshaling</h4>
<p>Go 1.2 will include some very nice improvements to encoding, by
introducing a general text marshaler and an improved marshaling
interface for <code>encoding/xml</code>.</p>
<h5 id="generic-text-and-binary-marshalers">Generic text and binary marshalers</h5>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/12541051"><span class="cl">CL 12541051</span></a>, <a href="https://golang.org/cl/12703043"><span class="cl">CL 12703043</span></a>, <a href="https://golang.org/cl/12751045"><span class="cl">CL 12751045</span></a>, <a href="https://golang.org/cl/12681044"><span class="cl">CL 12681044</span></a>, <a href="https://golang.org/cl/12705043"><span class="cl">CL 12705043</span></a>, <a href="https://golang.org/cl/12706043"><span class="cl">CL 12706043</span></a></p>
<p>The first part of the improvements is the addition of an
<a href="http://tip.golang.org/pkg/encoding/"><code>encoding</code> package</a>, defining the
<code>TextMarshaler</code>/<code>TextUnmarshaler</code> and
<code>BinaryMarshaler</code>/<code>BinaryUnmarshaler</code> interfaces.</p>
<p>These two sets of interfaces define generic (un)marshaling of values
to/from UTF-8 encoded text and binary form respectively. The idea
behind them is that packages like <code>encoding/json</code>, <code>encoding/xml</code> and
<code>encoding/gob</code> can all rely on these interfaces instead of forcing the
user to implement three different sets of interfaces (or more if new
<code>encoding/*</code> packages are added.)</p>
<p>One issue that might be familiar to some of you is that encoding a
<code>net.IP</code> to JSON would result in a base64 encoded string instead of a
human-readable representation of an IP. And encoding it to XML is
<a href="http://play.golang.org/p/S2AwOUJyzp">even more futile</a>, not to
mention that neither would be able to unmarshal again.</p>
<p>That is because previously, net.IP would’ve had to implement the JSON
<code>Marshaler</code> interface, and XML didn’t provide any way to implement
custom marshaling.</p>
<p>Now, by implementing the <code>TextMarshaler</code> and <code>TextUnmarshaler</code>
interfaces, a type can have a single machine-parseable textual
representation that all encoding packages can use. And if text isn’t
an option, there’s the <code>BinaryMarshaler</code> and <code>BinaryUnmarshaler</code>
interfaces, which right now <code>encoding/gob</code> uses and prefers over the
textual ones.</p>
<p>Of course it will still be possible to implement encoding-specific
behaviour for e.g. JSON. The generic interfaces will be tried last,
after checking that no encoding-specific interfaces have been
implemented.</p>
<p>All existing encoding packages in the standard library support the new
interfaces (<a href="https://golang.org/cl//12703043">JSON</a>, <a href="https://golang.org/cl//12751045">XML</a>,
<a href="https://golang.org/cl//12681044">gob</a>) and both <a href="https://golang.org/cl//12705043">net.IP</a> and
<a href="https://golang.org/cl//12706043">time.Time</a> implement them.</p>
<p>If you’re interested, there is also
<a href="https://docs.google.com/document/d/1u1a0ZxxBHWx5OCNNHXXR-yXl5ry5RM7DLlYy-5bjBE8/pub">a design document</a>.</p>
<h5 id="xmlmarshaler-and-xmlunmarshaler">xml.Marshaler and xml.Unmarshaler</h5>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/12603044"><span class="cl">CL 12603044</span></a>, <a href="https://golang.org/cl/12556043"><span class="cl">CL 12556043</span></a></p>
<p>The second and very important improvement is the addition of
marshaling interfaces to <code>encoding/xml</code>. While previously XML
marshaling behaviour could only be affected via struct tags, it is now
possible to implement entirely custom behaviour, both for values that
are marshaled as tags
(<a href="http://tip.golang.org/pkg/encoding/xml/#Marshaler"><code>Marshaler</code></a> and
<a href="http://tip.golang.org/pkg/encoding/xml/#Unmarshaler"><code>Unmarshaler</code></a>) and as
attributes (<a href="http://tip.golang.org/pkg/encoding/xml/#Marshaler"><code>MarshalerAttr</code></a>,
<a href="http://tip.golang.org/pkg/encoding/xml/#UnmarshalerAttr"><code>UnmarshalerAttr</code></a>)</p>
<p>Since XML is more complex than JSON by having different contexts in
which data can appear (tags, attributes, comments) and more
complicated parsing and state (escaping, namespaces, …), the design
of the XML interfaces differs from the JSON ones in one significant
aspect: While the JSON marshaling interfaces deal with byte slices,
the XML interfaces give you an <code>Encoder</code>/<code>Decoder</code> value, allowing you
to use Go’s parser to parse or emit tokens or tags.</p>
<p>For that purpose, two new functions have been added to <code>xml.Encoder</code>,
namely
<a href="http://tip.golang.org/pkg/encoding/xml/#Encoder.EncodeElement"><code>EncodeElement</code></a> and
<a href="http://tip.golang.org/pkg/encoding/xml/#Encoder.EncodeToken"><code>EncodeToken</code></a>.</p>
<p>But if your requirements don’t include complex handling of tokens you
can still use the new interfaces to encapsulate a common pattern:
Copying your data into another data structure that reflects the
desired XML structure and marshal that. Previously, this step had to
be done before the call to <code>xml.Marshal</code>, now it can be hidden away;
similar for <code>xml.Unmarshal</code>.
<a href="http://play.golang.org/p/v9Nt57Fj03">This playground snippet</a>
demonstrates some of the possibilities, but it’s hard to cover all of
them in a succinct way without real world examples. And of course you
cannot execute that code in the playground because it depends on
changes in Go tip.</p>
<p>There is a
<a href="https://docs.google.com/document/d/1G-AMeUPoyTnbZDkuCJA89DjJwTjdWFctIZ_N9NgA9RM/pub">design document</a>.
for these interfaces as well, but be warned that the final
implementation differs slightly by separating the Marshaler interface
into two interfaces.</p>
<h4 id="scheduler">Scheduler</h4>
<h5 id="basic-preemptive-scheduling">Basic preemptive scheduling</h5>
<p class="relevant_cls">Relevant CLs: <a href="https://golang.org/cl/10264044"><span class="cl">CL 10264044</span></a></p>
<p>One of the oldest issues on the Go project has been the request for
preemptive scheduling (Issue 543). Until now, Go’s scheduler has been
strictly cooperative, with scheduling happening during specific events
(channel operations and locks).</p>
<p>While cooperative scheduling isn’t familiar to many modern-day
programmers and might confuse them in some cases, that’s not the main
problem. The main problem is that the garbage collector, too, needs to
wait for all goroutines to yield before it can do its job.</p>
<p>When the garbage collector wants to run, it prevents all goroutines
from being scheduled. A goroutine that is already running, however,
will run until it yields. This can lead to long GC pauses, where all
cooperative goroutines have already been stopped but one misbehaving
goroutine is still running and won’t yield for the foreseeable future.
During that period, neither other goroutines nor the GC can run.</p>
<p>Preemptive scheduling solves this problem by, well, preempting
goroutines. No longer do they choose when to yield.</p>
<p>So, does that mean that <code>for {}</code> will no longer lock up the scheduler?
Not quite. The amount of preemption in Go 1.2 is limited to preemption
at function calls.</p>
<p>Whenever a function gets called, its preamble checks whether there is
enough stack left or if more needs to be allocated. The scheduler
abuses this check by pretending that more stack needs to be allocated
and thus forcing the “need more stack” function to run. This function
is aware of preemption, realizes that it doesn’t really need more
stack and instead preempts the goroutine.</p>
<p>This does imply that a goroutine that never uses any channel
operations, locks, or function calls will still not be preempted, but
such code should be rare, and the amount of preemption might well be
increased in future versions of Go.</p>
<h3 id="the-future">The future</h3>
<p>I hope that this article could give you a rough overview over the kind
of changes that are happening in Go tip, as well as the format of this
series. Of course there are a lot more changes that happened between
Go 1.1 and now, but as I stated in the introduction, because of the
upcoming summer slowdown, I spread out these changes over several
articles. And after all, the idea of this series is to provide
digestible summaries of what happened. A huge article with all changes
combined would contradict that idea.</p>
<p>Please let me know if/how you liked this article and the idea of the
series and whether it is something you want me to continue.</p>
<p>You can also contact me via <span class="information" title="dominikh on freenode; also in #go-nuts">IRC</span>,
<a href="mailto://dominikh@fork-bomb.org">email</a>,
<a href="https://profiles.google.com/111031610679270691456">G+</a> or
<a href="http://twitter.com/dominikhonnef">Twitter</a> and let me know about
specific changes that you want me to cover. To give you an overview
over what I have planned for the upcoming articles: I will be talking
about shared linking, performance improvements and a new slicing
syntax!</p>
tag:dominik.honnef.co,2013-08-11:/posts/2013/08/emacs-go-2/Writing Go in Emacs (cont.)2013-08-11T17:35:31Z2013-08-11T17:35:31Z
<p><a href="/posts/2013/03/writing_go_in_emacs/">Back in March</a> I demonstrated
various ways in which
<a href="https://github.com/dominikh/go-mode.el">go-mode</a> can help you write
Go more efficiently.</p>
<p>Today I want to post a quick follow-up, showing two new ways to
improve your performance. One is a very handy 3rd party package, while
the other is a direct addition to go-mode.</p>
<h3 id="go-eldocel">go-eldoc.el</h3>
<p><a href="https://github.com/syohex/emacs-go-eldoc">go-eldoc.el</a> by
<a href="http://syohex.org/">Syohei Yoshida</a> implements
<a href="http://www.emacswiki.org/emacs/ElDoc">ElDoc</a> functionality for
go-mode. For those of you who aren’t familiar with ElDoc, it’s a minor
mode that uses the echo area to show the argument list of the function
call you are currently writing or editing.</p>
<p>
<img src="/img/go_emacs/go-eldoc.png" alt="Screenshot of go-eldoc.el" />
</p>
<p>For example, if you were typing <code>fmt.Printf(</code>, it would tell you that
the signature of the function is <code>Printf(format string, a
...interface{}) (n int, err error).</code> It will also highlight the
current argument in the list.</p>
<p>Under the hood, go-eldoc.el uses
<a href="https://github.com/nsf/gocode">gocode</a> to determine function
signatures, which means it can make use of gocode’s well known
accuracy.</p>
<h3 id="code-coverage-analysis">Code coverage analysis</h3>
<p>When writing tests, one measure you continuously want to look at is
code coverage. In tip, Go’s test tool supports generating coverage
data for your code and provides a way to generate a single HTML file
out of it, which is already a very nice way to check coverage.</p>
<p>
<img src="/img/go_emacs/go-tool-coverage-html.png" alt="Screenshot of go tool coverage HTML output" />
</p>
<p>But wouldn’t it be even nicer if your editor could display that
information directly? In a buffer that behaves just like a Go buffer
with editing and navigation capabilities? That’s what go-mode now
supports! It can parse the coverage information that <code>go test</code>
generates and overlay it on your code, in a second,
<a href="http://www.gnu.org/software/emacs/manual/html_node/emacs/Indirect-Buffers.html">cloned buffer</a>.
This buffer behaves just like a normal go-mode buffer, offering the
same level of navigation and editing capabilities.</p>
<p>
<img src="/img/go_emacs/go-coverage.png" alt="Screenshot of go-coverage" />
</p>
<p>So how hard is it to generate and display coverage information? Not
hard at all. In order to generate coverage information, you need to
pass the <code>-coverprofile</code> option to go test, specifying a file to print
the information to. An example invocation could look like <code>go test
-coverprofile=c.out</code>. This can easily be done via <code>M-x compile</code> or a
terminal.</p>
<p>In order to display the information, run <code>M-x go-coverage</code> in the
buffer you want to view the information for. It will prompt you for
the coverage file, then open a cloned buffer with a gray/red/green
overlay. Gray represents untracked code (comments, types and other
code that cannot be run by itself), red represents code that wasn’t
covered by your tests and green represents code that was covered.</p>
<p>If you want to update the displayed information after a new run of <code>go
test</code>, you can just use <code>M-x go-coverage</code> in the coverage buffer and
it will update it from the same file you used last time.</p>
<p>There is one more feature that <code>go test</code>’s coverage output supports,
and that’s detailed accounting on how often a line of code ran,
instead of a boolean yes or no. To enable this, pass
<code>-covermode=count</code> (the default is <code>set</code>) to <code>go test</code>. To display the
information you still use <code>M-x go-coverage</code>.</p>
<p>
<img src="/img/go_emacs/go-coverage-gradient.png" alt="Screenshot of go-coverage with gradient" />
</p>
<p>The brighter the green is, the more often the code has been run.
Hovering over any of the lines will give you a tooltip with the
absolute number of times.</p>
<p>It is important to note that you need Go tip for this to work. Go 1.1
does not include coverage generation yet. And, as usual, it’s best to
use <a href="https://github.com/dominikh/go-mode.el">go-mode.el from GitHub</a>,
even though the latest changes are slowly finding their way upstream.</p>
tag:dominik.honnef.co,2013-03-11:/posts/2013/03/emacs-go-1/Writing Go in Emacs2013-03-11T08:27:27Z2013-03-11T08:27:27Z
<p>Using the right tools plays a big role in getting your job done
efficiently. In the case of programming Go, most of your time will be
spent in an editor. It is therefore key that your editor doesn’t get
in your way and, optimally, assists you wherever it can.</p>
<p>In this article, I will talk about go-mode for Emacs, its history and
features, as well as useful extensions to it.</p>
<p>Even though this article is targeted at Emacs users, parts of it,
mainly the feature list, might be of interest to users of other
editors as well, to see what’s possible and to adopt some of the
ideas. You will want to skip the <em>History</em> and <em>Obtaining it</em> sections
though.</p>
<h3 id="history-of-go-mode">History of go-mode</h3>
<p>Go has been including a mode for Emacs for a long time now. Until
recently, however, it was rather fragile. It regularly messed up
indentation or syntax highlighting, for quite some time it could
completely lock up your editor and it didn’t feel really integrated in
Emacs, lacking support for functions like <code>beginning-of-defun</code>. It
also didn’t include a lot of extra features.</p>
<p>All in all, it was better than having to use nano, but not by much.</p>
<p>All that changed around February this year, when I rewrote the mode
from scratch. I took special care to build it on top of a solid
foundation, making use of Emacs’s parsing facilities, as opposed to
the previous version of the mode, which implemented its own parser.
Apart from generally bug-free behavior, this also allowed building a
lot of functions on top of it.</p>
<h3 id="obtaining-it">Obtaining it</h3>
<p>There are two ways of obtaining go-mode and both are very similar. The
first way is using the one that comes with the
<a href="http://tip.golang.org/misc/emacs/">Go distribution itself</a>, the
second way is to get it from
<a href="https://github.com/dominikh/go-mode.el">GitHub</a>, where I push all new
changes before they find their way into the Go repository.</p>
<p>When using the one that comes bundled with Go, be sure to use the one
that comes with <em>tip</em>, the development version of Go. The current
release, 1.0.3, includes the old, outdated version of go-mode.</p>
<h4 id="installation">Installation</h4>
<p>If using go-mode from Go tip, put both <code>go-mode.el</code> and <code>go-mode-load.el</code>
into a directory of your choice, add it to Emacs’s load paths via</p>
<pre class="cl"><code>(add-to-list 'load-path "/place/where/you/put/it")
</code></pre>
<p>and require it via</p>
<pre class="cl"><code>(require 'go-mode-load)
</code></pre>
<p>This will install autoload entries and associate <code>*.go</code> files with
go-mode.</p>
<p>If using go-mode from the git repository, the procedure is similar,
but requires an additional step to generate the <code>go-mode-load.el</code>
file. From within Emacs, run <code>M-x update-file-autoloads</code>, point it at
the <code>go-mode.el</code> file and tell it to generate a <code>go-mode-load.el</code> file.
From that point on the procedure is the same as before.</p>
<h3 id="features">Features</h3>
<p>For Emacs, a programming mode is often about more than just
indentation and syntax highlighting. Integration with the tools of the
language is often key, easy navigation of code, even crossing package
boundaries, is appreciated and some people even rely on on-the-fly
syntax checking.</p>
<p>go-mode, either directly or by using the work of others, offers all of
this and more.</p>
<p>We’ll discover how to read documentation, format and navigate code,
how to use Go’s own pastebin, how to get syntax checking, autocompletion and snippets.</p>
<h4 id="reading-documentation">Reading documentation</h4>
<p>Using the <code>godoc</code> function, you can invoke the identically named Go
tool from within Emacs and read package documentation in a view-mode
buffer. Additional feature: You can tab complete import paths.</p>
<h4 id="formatting-code">Formatting code</h4>
<p>Maybe one of the most important tools of Go is <em>gofmt</em>, which
automatically formats your code to the one true coding style, used by
every Go developer.</p>
<p>In Emacs, there are two ways to use <em>gofmt</em>. One way is to invoke
<em>gofmt</em> manually with the identically named function <code>gofmt</code>, which
will patch the current buffer according to gofmt. The other way is to
use a before-save-hook to run <code>gofmt</code> every time you save a Go buffer.
Enabling that hook is as easy as doing</p>
<pre class="cl"><code>(add-hook 'before-save-hook 'gofmt-before-save)
</code></pre>
<p>If you have used this hook before and complained about your cursor
jumping around, don’t worry: This has been fixed in the new go-mode.
At the very most, your cursor will be placed on the beginning of the
current line, if it had to be reformatted.</p>
<h4 id="managing-imports">Managing imports</h4>
<p>On a first glance, managing imports in Go doesn’t seem like much of a
deal. You either add import statements at the top of your file or you
remove them. But when rapidly developing a new application, this can
cause a lot of jumping around. Did you just realize that you forgot to
import the fmt package? Previously, this meant going to the beginning
of the buffer, advancing to the block of import statements (if it
existed yet), going to its end and adding your new import. And when
you realized you didn’t need it after all, you had to repeat the same
procedure to remove it again.</p>
<p>The new go-mode has three functions for working with imports:
<code>go-import-add</code>, <code>go-remove-unused-imports</code> and <code>go-goto-imports</code>.</p>
<h5 id="go-import-add">go-import-add</h5>
<p><code>go-import-add</code>, bound to <code>C-c C-a</code> by default, will prompt you for an
import path (again supporting tab completion) and insert it in the
import block, creating it if necessary. If an import already existed
but was commented, it will be uncommented.</p>
<p>If prefixed with <code>C-u</code>, it will ask you for an alias, too. An annoying
procedure of moving around and mental context switching has just been
reduced to a keystroke.</p>
<h5 id="go-remove-unused-imports">go-remove-unused-imports</h5>
<p>Instead of offering a function for removing a single import, go-mode
will detect all unused imports and delete them (or comment them) once
you run <code>go-remove-unused-imports</code>. It is not bound to a key by
default, but you can bind it yourself if you want to. Personally I
have bound it to <code>C-c C-r</code>:</p>
<pre class="cl"><code>(add-hook 'go-mode-hook (lambda ()
(local-set-key (kbd "C-c C-r") 'go-remove-unused-imports)))
</code></pre>
<h5 id="go-goto-imports">go-goto-imports</h5>
<p>If you decide you want to look at your imports or edit them manually,
go-goto-imports will take you to them automatically, placing your
cursor after the last import. It isn’t bound to a key, either, mainly
because I couldn’t come up with a good default that didn’t violate
Emacs guidelines. But you can bind it manually, just like before:</p>
<pre class="cl"><code>(add-hook 'go-mode-hook (lambda ()
(local-set-key (kbd "C-c i") 'go-goto-imports)))
</code></pre>
<h4 id="navigating-code">Navigating code</h4>
<p>Also important for efficient programming is quick navigation of code.
<code>go-goto-imports</code> was only the tip of the iceberg.</p>
<p>go-mode supports <code>beginning-of-defun</code> (<code>C-M-a</code>) and <code>end-of-defun</code>
(<code>C-M-e</code>), two core Emacs functions for navigating between functions.
Additionally, functions such as <code>narrow-to-defun</code> and <code>mark-defun</code>
rely on these two functions.</p>
<p>If that isn’t enough, you can also use
<a href="http://emacswiki.org/emacs/ImenuMode">Imenu</a> to jump to specific
function or type declarations.</p>
<p>Even if you long for a feature that’s usually only provided by IDEs,
or by making use of etags, namely jumping to the declaration of a
symbol under your cursor, go-mode got you covered.</p>
<p>go-mode integrates with
<a href="https://code.google.com/p/rog-go/source/browse/exp/cmd/godef/">godef</a>,
an amazing little tool written by
<a href="https://twitter.com/rogpeppe">Roger Peppe</a>. godef is able to parse
your code, and the code of other packages, and the code of the Go
standard library, and can tell you what exactly the symbol you’re
looking at means and where it has been defined.</p>
<p>go-mode makes use of this to provide the two functions
<code>godef-describe</code> and <code>godef-jump</code>. <code>godef-describe</code> will tell you what
you’re looking at, while <code>godef-jump</code> will take you to its definition.
And yes, this works across files, packages and into the standard
library, without needing any tags. And it has almost no measurable
delay.</p>
<p><code>godef-describe</code> is bound to <code>C-c C-d</code> and <code>godef-jump</code> is bound to
<code>C-c C-j</code>. If you want to, you can bind <code>godef-jump</code> to <code>M-.</code>, which
is the default key for <code>find-tag</code>, something you might already be
using for other programming languages:</p>
<pre class="cl"><code>(add-hook 'go-mode-hook (lambda ()
(local-set-key (kbd \"M-.\") 'godef-jump)))
</code></pre>
<p>There are two important notes:</p>
<ol>
<li>
<p><strong>Update (2013-03-19): The godef features in go-mode are now part
of the Go repository.</strong> As of writing this article, the godef
functions haven’t reached the Go repository yet. They are only
available in the git version. This will hopefully change in the next
couple of days.</p>
</li>
<li>
<p>You will obviously need to install godef, which is as easy as doing
<code>go get code.google.com/p/rog-go/exp/cmd/godef</code>.</p>
</li>
</ol>
<video controls="">
<source src="/video/go_emacs/godef_mp4.mp4" type="video/mp4"></source>
<source src="/video/go_emacs/godef_ogg.ogg" type="video/ogg"></source>
Your browser does not support the video tag.
</video>
<h4 id="interacting-with-the-playground">Interacting with the Playground</h4>
<p>Go has its own pastebin, called the
<a href="http://play.golang.org/">Playground</a>. When asking for help with code,
or sharing solutions to those who need help, it is often expected to
use the Playground, especially because it can execute code.</p>
<p>It makes sense that go-mode integrates with the Playground. It offers
<code>go-play-buffer</code> and <code>go-play-region</code> to send the current buffer or
region to the Playground and store a link in your kill ring.</p>
<p>It can also directly download a paste from the Playground into Emacs,
by using <code>go-download-play</code>.</p>
<h4 id="on-the-fly-syntax-checking">On-the-fly syntax checking</h4>
<p><a href="http://www.emacswiki.org/emacs/FlyMake">FlyMake</a> is Emacs’s solution
to on-the-fly syntax checking. Many programming modes offer support
for it (some more easily than others).</p>
<p>
<img src="/img/go_emacs/goflymake.png" alt="Screenshot of goflymake" />
</p>
<p>In the case of Go,
<a href="https://twitter.com/dougmaceachern">Doug MacEachern</a> wrote
<a href="https://github.com/dougm/goflymake">goflymake</a>, which consists of a
small Go binary and some elisp to integrate it with Emacs. Because Go
compiles blazingly fast, using goflymake doesn’t cause any performance
penalties. Personally I am letting FlyMake compile my Go buffers every
time I insert a newline. More conservative settings would compile
after a certain amount of idle time, or when saving the buffer.</p>
<h4 id="autocompletion">Autocompletion</h4>
<p><a href="https://github.com/nsf/gocode">gocode</a>, written by
<a href="https://github.com/nsf">nsf</a>, provides the kind of autocompletion
that you might know from IDEs. It provides context-sensitive
autocompletion and works across a variety of editors, including Emacs.
The project’s README includes detailed installation instructions.</p>
<p>Note: it suggests using auto-complete-mode and mentions company-mode
as an alternative. Personally I recommend trying company-mode first.
auto-complete-mode isn’t as lightweight and can be harder to
configure.</p>
<p>
<img src="/img/go_emacs/gocode.png" alt="Screenshot of gocode" />
</p>
<h4 id="snippets">Snippets</h4>
<p>Some people like them, some think they’re useless: Snippets. Short
words that, upon pressing TAB, turn into longer words, sentences or
code constructs. One of the more popular implementations of snippets
for Emacs is <a href="https://github.com/capitaomorte/yasnippet">YASnippet</a>,
and it is YASnippet that I wrote a
<a href="https://github.com/dominikh/yasnippet-go/tree/master/go-mode">small number of Go snippets</a>
for.</p>
<h4 id="finding-unchecked-errors-with-go-errcheck">Finding unchecked errors with go-errcheck</h4>
<p>Last but not least, there’s
<a href="https://github.com/dominikh/go-errcheck.el">go-errcheck</a>, offering
Emacs integration for <a href="https://github.com/kisielk/errcheck">errcheck</a>.
errcheck is a tool written by
<a href="http://www.kamilkisiel.net/">Kamil Kisiel</a> for finding and reporting
unchecked errors in your Go code.</p>
<p>
<img src="/img/go_emacs/go-errcheck.png" alt="Screenshot of go-errcheck" />
</p>
<p>go-errcheck offers the <code>go-errcheck</code> function, which will run errcheck
on the current package and report errors in a compilation buffer.
Additionally, it supports setting errcheck flags on a default and
per-package basis.</p>
<h3 id="conclusion">Conclusion</h3>
<p>I hope I was able to give you a good overview of the features that you
can enjoy while writing Go code in Emacs. If I forgot any cool
packages, or if you have suggestions or comments, please leave me a
message. Check the <a href="/contact/">contact page</a> to find ways of
contacting me.</p>
tag:dominik.honnef.co,2012-12-05:/posts/2012/12/go-type-identity/Go: On implicit type conversions, type identity and a little gotcha2012-12-05T22:04:26Z2012-12-05T22:04:26Z
<p>A well known design decision in the Go programming language is the
absence of implicit type conversions. An <code>int</code> is always an <code>int</code> and not
an <code>int32</code> and certainly not a <code>float</code>. And a <code>net.IP</code> is not a <code>[]byte</code>… or
is it?</p>
<p>Knowing only the “no implicit type conversions” principle, the
<a href="http://play.golang.org/p/xNBc9evi96">following code</a> should not compile:</p>
<pre class="go"><code>package main
import (
"fmt"
"net"
)
func main() {
var foo net.IP
var bar []byte = []byte{1, 2, 3, 4}
foo = bar
fmt.Println(foo)
}
</code></pre>
<p>But it does! Now, the definition of <code>net.IP</code> is <code>type IP []byte</code>, but we
also know that this should be an entirely new type and not an alias,
because Go doesn’t have type aliases, either. So why does this compile
and execute properly?</p>
<p>The secret lies in Go’s understanding of
<a href="http://golang.org/ref/spec#Type_identity">type identity</a> and
<a href="http://golang.org/ref/spec#Assignability">assignability</a>. The
following part is of special importance:</p>
<blockquote>
<p>A value x is assignable to a variable of type T (“x is assignable to T”) in any of these cases:</p>
<p>[…]</p>
<ul>
<li>x’s type V and T have identical underlying types and at least one of V or T is not a named type.</li>
</ul>
</blockquote>
<p>And in our case, <code>[]byte</code> is not a named type (unlike <code>byte</code> or
<code>net.IP</code>, which are), which means that values of type <code>net.IP</code> and
<code>[]byte</code> can be freely assigned to each other. And of course the same
applies to all other identical scenarios, <code>net.IP</code> and <code>[]byte</code> are
just examples.</p>
<p>The remaining question is: Why did they add this special case to the
language specification? Go is not a language of odd exceptions, but at
first glance, this exception does seem very odd. The only sensible
explanation I came across are function types. Consider the <a href="http://play.golang.org/p/r6ivafqGwr">following
code</a>:</p>
<pre class="go"><code>package main
import "fmt"
type BinaryOp func(int, int) int
func Do(fun BinaryOp, a, b int) int {
return fun(a, b)
}
func main() {
result := Do(func(a, b int) int { return a + b },
1, 2)
fmt.Println(result)
}
</code></pre>
<p>If <code>BinaryOp</code> (a named type) and <code>func(int, int) int</code> (an unnamed
type) weren’t identical, writing anonymous functions would be
<a href="http://play.golang.org/p/RTit2DS_TG">quite a bit uglier</a>.</p>
<p>I don’t know if the initial example, identity of named and unnamed
slice types, was intentional or an accident caused by an overly broad
specification, but I’d recommend not making use of it and sticking to
explicit conversions, even if they’re not necessary. Clarity of code
trumps saving a few keystrokes.</p>
tag:dominik.honnef.co,2012-04-16:/posts/2012/04/go-string-encodings/Dealing with encodings in Go2012-04-16T15:31:41Z2012-04-16T15:31:41Z
<p>One not well documented aspect of Go is how it handles string
encodings, or how strings are actually treated internally. This
article tries to shed some light on basic aspects of that, to get you
started.</p>
<h3 id="its-all-utf-8-right">It’s all UTF-8, right?</h3>
<p>A source of confusion is the fact that Go <strong>source code</strong> has to be
encoded in UTF-8, meaning that string literals, variable/function
names etc. most consist solely of UTF-8 code points. This does,
however, not mean that strings in Go must only contain valid UTF-8
data. For example, the following string literal is okay to use, even
though it does not describe a valid UTF-8 string: <code>"Hello,
\x90\xA2\x8A\x45"</code>.</p>
<p>In fact, the string datatype is effectively nothing more than an
immutable byte array. This also means that using the index operator
will return a specific byte, not a <a href="http://en.wikipedia.org/wiki/Code_point">rune</a>:</p>
<pre class="go"><code>fmt.Printf("0x%x", "世界"[1]) // =&gt; 0xb8
</code></pre>
<p>Furthermore, the <code>len()</code> function will also return a string’s length in
bytes, not runes:</p>
<pre class="go"><code>fmt.Println(len("Hello, 世界")) // =&gt; 13
</code></pre>
<h3 id="but-isnt-go-aware-of-utf-8">But isn’t Go aware of UTF-8?</h3>
<p>Even though strings are mere byte arrays, Go does have proper support
for UTF-8. For example the <code>range</code> operator will iterate over a string
by yielding runes, instead of bytes:</p>
<pre class="go"><code>s := "Hello, 世界"
for _, r := range s {
fmt.Printf("%c ", r) // =&gt; H e l l o , 世 界
}
</code></pre>
<p>And by importing the <code>unicode/utf8</code> package, it is also possible to
validate strings, or more importantly, get their rune count (which
effectively can be compared to the string’s length as we’d interpret
it):</p>
<pre class="go"><code>s := "Hello, 世界"
fmt.Println(utf8.RuneCountInString(s)) // =&gt; 9
fmt.Println(utf8.ValidString(s)) // =&gt; true
</code></pre>
<h3 id="and-how-do-i-deal-with-other-encodings">And how do I deal with other encodings?</h3>
<p>Even though Go has good support for UTF-8 (and minimal support for
UTF-16), it has no built-in support for any other encoding. If you
have to use other encodings (e.g. when dealing with user input), you
have to use third party packages, like for example <a href="http://code.google.com/p/go-charset/">go-charset</a>.
go-charset makes it quite easy to convert between different string
encodings.</p>
<p>If you want to further work with input data, the best idea is to first
convert it to UTF-8 and then use Go’s built-in features:</p>
<pre class="go"><code>package main
import (
"code.google.com/p/go-charset/charset"
"fmt"
"io/ioutil"
"strings"
"unicode/utf8"
)
import _ "code.google.com/p/go-charset/data" // include the conversion maps in the binary
func main() {
s := "Hello, \x90\xA2\x8A\x45" // CP932 encoded version of "Hello, 世界"
r, _ := charset.NewReader("CP932", strings.NewReader(s)) // convert from CP932 to UTF-8
s2_, _ := ioutil.ReadAll(r)
s2 := string(s2_)
fmt.Println(s2) // =&gt; Hello, 世界
fmt.Println(len(s2)) // =&gt; 13
fmt.Println(utf8.RuneCountInString(s2)) // =&gt; 9
fmt.Println(utf8.ValidString(s2)) // =&gt; true
}
</code></pre>
<h3 id="conclusion">Conclusion</h3>
<p>Even though this article is rather short, I hope that it was able to
convey the following main points:</p>
<ol>
<li>Strings in Go are mere byte arrays</li>
<li>There is built-in functionality for dealing with UTF-8</li>
<li>Any other encoding is best converted to UTF-8 first</li>
</ol>
<p>For further reading, I recommend <a href="https://groups.google.com/forum/?fromgroups#!topic/golang-nuts/1yL7IsqADSw">a discussion on the Go mailing list</a>,
which also sheds light on why strings behave the way they do (hint:
it’s mostly about performance, and freedom).</p>
tag:dominik.honnef.co,2012-03-28:/posts/2012/03/backwards-compatibility/Breaking backwards compatibility in good conscience2012-03-28T10:57:31Z2012-03-28T10:57:31Z
<p>A couple of days ago I released Cinch 2.0 – version 2.0 of the
most-used IRC library written in Ruby. The purpose of this article,
however, is not to talk about all the new features and improvements in
Cinch (check the <a href="http://bit.ly/GLoXVP">changelog</a> if you’re curious) but why it
breaks backwards compatibility and how that’s not a bad thing.</p>
<p>The Ruby community basically has two main camps: Those who think that
breaking backwards compatibility is no problem, as long as your
version numbers signalise it (c.f. <a href="http://semver.org/">Semantic Versioning</a>), and
those who think that breaking backwards compatibility is proof of bad
API design and should be avoided at all costs.</p>
<p>Loren Segal, creator of <a href="http://yardoc.org/">YARD</a>, belongs to the latter camp.
Notably, the current version of YARD (0.7.5 as of writing) is still
compatible with the earliest releases. According to him, he still uses
the same plugins and templates that he wrote for version 0.2.x,
released roughly 4 years ago.</p>
<p>He says that this is possible because he put a lot of thought into the
API, planning <em>before</em> writing code. While I agree that maintaining
backwards compatibility is a noble cause, and that planning <em>can</em>
help, I don’t think that everything can be planned perfectly. I think
that Loren accomplished a great feat by planning the API in such a
solid way that it worked out for him, but it might not for others, and
it might not be a sign of bad will.</p>
<p>I myself put quite some thought into the API of Cinch before releasing
version 1.0 – I tried to consider as many scenarios as possible,
planning ahead and avoiding future conflicts. And I failed. Cinch was
my first IRC library with focus on writing multithreaded bots, and
while I tried to get everything right, I overlooked many little
details. Details that only became apparent when users came to me,
pointing out problems they were having.</p>
<p>While I could’ve maintained <em>some</em> parts of the flawed API in version
2.0, other parts had to be removed completely because they were plain
broken and unfixable. And I didn’t want to maintain the flawed parts,
either. Why? Because they added unnecessary complexity to the code
base of Cinch, and because on the long run it won’t matter. Cinch is
now 1.5 years old and it will be around for much longer. In a couple
of years, nobody will care that the API changed. In fact it will
probably not take more than a couple of months.</p>
<p>My point is that it’s a good idea to try and avoid breaking the API,
but only as long as it makes sense and as long as it doesn’t have a
negative effect on your code, or the rest of the API, in the long run.
And if you’ve made a mistake while designing the API, it is better to
admit the mistake and fix it than to live with it for the rest of your
project’s life.</p>
<p>Everyone who has taken a look at the <a href="https://github.com/lsegal/yard">code base of YARD</a>
knows that it indeed has been designed very well, but I think this is
an exceptional case and not the norm and it shouldn’t be assumed that
everyone can achieve the same. Maybe it was even a bit of luck that it
worked out so well.</p>
<p>Of course this isn’t a free ticket for breaking APIs every second
release, either. If you have to constantly break backwards
compatibility, maybe you should put at least <strong>some</strong> thought into
design first.</p>
tag:dominik.honnef.co,2011-10-29:/posts/2011/10/mdraid-dmcrypt-write-performance/Linux RAID + dm-crypt: Poor write performance2011-10-28T23:41:47Z2011-10-28T23:41:47Z
<p>I’ve been running a RAID 6 with four active disks for quite some time
now, and performance always seemed adequate, achieving about 230MB/s
read and 90MB/s write (obviously in rather artifical tests using dd;
actual file operations might vary).</p>
<p>Now it occured to me, that encrypting my data might actually be a good
idea, so I looked into cryptsetup for adding encryption on top of my
RAID, using LUKS. Now, anyone familiar with RAIDs should know that
alignment is key to proper performance. cryptsetup offers an option
called <code>--align-payload</code> to align data at a multiple of 512 bytes.
Based on the chunk size I am using (512KB) and the number of disks,
2048 seemed to be the right value for the setting.</p>
<p>Weirdly enough, though, it didn’t matter what value I used: 2048,
1024, none at all or completely arbitrary values: Write performance
would always suffer and be around 30MB/s, with my CPU being almost
idle. I spent hours trying to figure out what I might’ve been doing
wrong, why the option wasn’t having any effect and so on, until I
found an aligning-unrelated setting called <code>stripe_cache_size</code>, which
defaults to 256 on many systems, mine including. After setting that
option to a higher value of 8192, my write speeds rocketed up to
190MB/s for both plain md0 as well as the encrypted device.</p>
<p>The option can be set by simply writing a value to
<code>/sys/block/&lt;your_md_device&gt;/md/stripe_cache_size</code>, so for example: <code>echo
8192 &gt; /sys/block/md0/md/stripe_cache_size</code>.</p>
<p>Please be aware that the setting describes the number of pages to
cache, so a higher value will consume more memory.</p>
<p>Also note that I did not rely on the speed output of dd but instead used
nmon, to avoid getting higher speeds caused by buffers. In fact, the
system will continue writing data while dd already thinks it is done.</p>
tag:dominik.honnef.co,2011-08-25:/posts/2011/08/electronics-3/Electronics: My first experiences with various shops2011-08-25T14:53:02Z2011-08-25T14:53:02Z
<p class="preface">This is part three of a series of me describing my new experiences with
electronics.</p>
<p>First of all, sorry for the late third article, but ever since I
received my Arduino I was really busy playing around with it.</p>
<p>Quite a bit has happened since my last article and I will try to catch
up as quickly as possible. One thing that happened is that I had to
order quite a bit more tools and components than I initially thought.</p>
<p>In this article I will concentrate on my experiences with the various
shops I’ve ordered from so far and how they performed.</p>
<h2 id="baustelnde">bausteln.de</h2>
<p>I ordered my Arduino starter kit from bausteln and they were quick to
send me a package. Everything was well packaged, nothing was missing
and they even included a neat little plastic box for storage.</p>
<p>There was, however, a little problem: The included breadboard is
about half the size of the one depicted in images. Now, I know that
relying on images is my fault, and the article description didn’t
include any measurements, either, so I really shouldn’t have expected
a breadboard of a specific size.</p>
<p>But my problem isn’t really with the small breadboard (even twice the
size would still be too small for any serious work) but the fact that
I’ve tried to contact the owner of the shop (it’s a single person) and
never got a reply. I’ve also tried to write a product review to warn
potential new customers, but said review has to be accepted by the
shop owner and he never did.</p>
<p>I do not know if he simply doesn’t care or just missed the messages,
but I do not feel comfortable doing business with someone who cannot
even reply to emails or monitor his shop.</p>
<h2 id="futurlec">Futurlec</h2>
<p>Packages from Hong Kong take a long time… Three weeks to be exact,
at least if you live in Germany. But obviously <a href="http://www.futurlec.com/">Futurlec</a> is
not to blame nor did I expect any faster shipping.</p>
<p>I was, however, honestly surprised about their quality. Everything was
packaged well, components were new, nothing was bent, nothing was
broken and nothing was missing.</p>
<p>As you might remember, I ordered a couple of “value packs” from
Futurlec. Basically bags full of discrete components of different
sizes/values. What I expected was one big bag per component type, full
of stuff, unsorted and with bent legs. I was surprised to see that
each value had its very own small bag, properly labelled as well.</p>
<p>Should I ever have to buy components in bulk again, I will definitely
consider Futurlec a second time.</p>
<h2 id="reichelt">Reichelt</h2>
<p><a href="http://www.reichelt.de/">Reichelt</a> is a German electronics store, selling everything
from components to tools and gadgets. Unlike other German competitors
like Conrad, Reichelt has cheap prices and doesn’t completely ignore
the hobbyist/maker market.</p>
<p>So far I made two purchases, mostly containing soldering equipment,
but also a few ICs and more importantly a big breadboard. Both
packages arrived fast (within 1-2 days).</p>
<p>Everything was packaged well, everything worked properly, I am happy.</p>
<h2 id="pollin">Pollin</h2>
<p>The 4th shop I ordered from is <a href="http://www.pollin.de/shop/index.html">Pollin</a>, another German store.
They’re kind of an electronics store, but they concentrate mostly on
buying and selling remaining stock and the like. Sometimes they sell
random LCD screens without any documentation, or even old Russian ICs.</p>
<p>I ordered 20 little plastic boxes for storing my components, 0.85€ per piece.</p>
<p>Shipping was moderately fast (3 days), everything was packaged well
etc. The boxes aren’t of best quality, but hey, I paid 85ct per box
and it has a reason they’re that cheap. Render me satisfied.</p>
tag:dominik.honnef.co,2011-07-26:/posts/2011/07/electronics-2/Electronics: Starting small2011-07-26T17:57:50Z2011-07-26T17:57:50Z
<p class="preface">This is part two of a series of me describing my new experiences with
electronics.</p>
<p>As I said in my last post, I want to start with simple hardware. The
reasons for this should be pretty obvious: neither do I have a
concrete idea for a project, nor am I certain that I’ll stay
interested for long enough. Consequently, I don’t want to spend too
much money.</p>
<p>A rough limit I have set for myself is about 60€.</p>
<h3 id="disregard-females-acquire-hardware">Disregard Females, Acquire Hardware</h3>
<p>The first step is to find a platform to work with, where the following
aspects are the most important ones to me:</p>
<ul>
<li>the developing environment has to be compatible with Linux</li>
<li>it has to be affordable</li>
<li>it has to be solder-free</li>
<li>it mustn’t be too limited</li>
</ul>
<p>Friends quickly pointed in the direction of Arduino, an open-source platform.</p>
<h4 id="what-is-arduino">What is Arduino?</h4>
<p>Arduino is both a community and a hardware platform. The idea of
Arduino is to provide an open platform for average people, especially
artists (with a wide meaning of artist).</p>
<p>Open, in this context, means that the circuit design as well as the
source code are freely available to anyone. An implication of this is
that anyone can design (and sell) his own Arduino. A nice side-effect
is that no single entity can control the price and thus Arduinos are
very affordable, starting at 20€ for the basic ones.</p>
<h4 id="what-arduino-to-get">What Arduino to get?</h4>
<p>There’s not “one Arduino” that you can buy, but different designs for
different tasks. There are Arduinos especially made for
<a href="http://www.arduino.cc/en/Main/ArduinoBoardLilyPad">sewing into clothes</a>,
<a href="http://www.arduino.cc/en/Main/ArduinoBoardNano">extremely small ones</a>
for small devices and a
<a href="http://arduino.cc/en/Main/ArduinoBoardUno">variety of palm-sized ones</a>
for prototyping (which is what we’re after).</p>
<p>The current “revisions” of the main prototyping boards are called
<a href="http://arduino.cc/en/Main/ArduinoBoardUno">“Arduino Uno”</a> and
<a href="http://arduino.cc/en/Main/ArduinoBoardMega2560">“Arduino Mega 2560”</a>,
where the only difference is that the Mega has more pins, more memory
and is more expensive.</p>
<p>The Arduino Uno can be bought for about 20-30€ (depending on where you
live) and the Arduino Mega for about 40-60€.</p>
<p>Considering that I want to start small, the Arduino Uno is the perfect
choice, especially because I still have to get more components to
actually be able to do something with it.</p>
<h4 id="starter-kits">Starter kits</h4>
<p>Luckily, a lot of shops sell starter kits consisting of an Arduino Uno
and some basic components. The most promising one I found is the
<a href="http://www.sparkfun.com/products/10173">“SparkFun Inventor’s Kit”</a>,
which comes with an Arduino, a lot of sensors and some actuators, for
65€. For more information on the exact list of parts, click the link.</p>
<p>Unfortunately, shipping to Germany is rather expensive, and even
though some German shops sell the very same kit, they sell it for more
than 65€, so that exceeds my limit, especially the one imposed by my
bank account.</p>
<p>I did, however, find
<a href="http://bausteln.de/shop/arduino/sets/einsteigerset.html">a comparable kit</a>
at a German shop called “bausteln” for only 59€. It comes with nearly
the exact same components, just lacking two of the sensors and a DC
motor. Since I already own multiple DC motors, and the missing sensors
can easily be bought separately later, I chose to buy that kit.</p>
<h4 id="buying-in-bulk">Buying in bulk</h4>
<p>I never realised how cheap resistors, transistors and diodes can be if
bought in bulk, at least if buying at shops that specialise on that
kind of selling, like <a href="http://www.futurlec.com">Futurlec</a> does.</p>
<p>Beside all the single components they sell, they also sell
<a href="http://www.futurlec.com/ValuePacks.shtml">bags full of them</a>.</p>
<p>$2.95 for 300 resistors? Bought! $2.95 for ceramic capacitors? Bought!
$5.95 for Linear ICs? You guessed it, bought! $4.95 for 100 LEDs?
Nope, I already have plenty of LEDs, thanks anyway. And so it goes on
for most of the “value packs” Futurlec sells. Those are certainly a
great way to get you started. For about $25 plus shipping (which is
also extremely affordable) you get bags full of everything you’ll need
as a starter.</p>
<h4 id="my-shopping-list">My shopping list</h4>
<p>And this is what I eventually ended up buying:</p>
<ul>
<li><a href="http://bausteln.de/shop/arduino/sets/einsteigerset.html">Arduino Einsteigerset</a> – $72</li>
<li><a href="http://www.futurlec.com/ValuePacks.shtml">Value packs</a>
<ul>
<li>300 1/4W resistors – $2.95</li>
<li>200 ceramic capacitors – $2.95</li>
<li>100 electrolytic capacitors – $3.95</li>
<li>50 Linear ICs – $5.95</li>
<li>100 diodes – $2.95</li>
<li>100 transistors – $4.95</li>
</ul>
</li>
<li><a href="http://www.futurlec.com/cgi-bin/search/qty_price.cgi?part_no=LEDBARRED">Red bargraph</a> – $0.30</li>
<li><a href="http://www.futurlec.com/cgi-bin/search/qty_price.cgi?part_no=JUMPERKIT">Jumper wires</a> – $4.90</li>
</ul>
<p>In total that’s $100.9 or 70.19€, plus shipping. That’s very close to
my limit of 60€ so I am satisfied.</p>
<p>More articles coming once I received my packages.</p>
tag:dominik.honnef.co,2011-07-25:/posts/2011/07/electronics-1/Electronics: Introducing myself to the world of electronics2011-07-25T17:29:30Z2011-07-25T17:29:30Z
<p class="preface">This is part one of a series of me describing my new experiences with
electronics.</p>
<h3 id="introduction">Introduction</h3>
<p>I’ve never really been interested in electronics (ie. circuits,
microcontrollers etc). Sure, I understand, on a basic level, how most
of the gadgets and computers that I own work, but I wouldn’t be able
to build anything myself or explain what e.g. a diode is used for.</p>
<p>The closest I ever got to working with electronics was owning a kit
for children, consisting of a couple transistors, resistors,
capacitors and LEDs. My interest didn’t last long though, especially
because I cooked one of the transistors early on, rendering me unable
to build most of the circuits from the manual.</p>
<p>Being a geek, however, I am always interested in acquiring new skills
that somehow relate to computers and electronics are just a logical
extension (or rather simplification) of computers. Furthermore I found
the following reasons to learn more about electronics:</p>
<ul>
<li>Parts of it might be considered common knowledge among my circles</li>
<li>It might be a nice skillset for my CV, especially the microcontroller aspect</li>
<li>My studies at university will require me to work with microcontrollers, anyway</li>
<li>I heard it is supposed to be a fun experience!</li>
</ul>
<p>So what am I going to do now? I’ll look for an affordable
microcontroller and components needed for basic circuits and go from
there. More on that in my next articles.</p>
tag:dominik.honnef.co,2011-07-13:/posts/2011/07/sketchup-selection-manager/SketchUp Plugin: Selection Manager2011-07-13T15:00:33Z2011-07-13T15:00:33Z
<p>Disappointed with <a href="http://www.smustard.com/script/SelectionMemory">SelectionMemory</a> and not willing to pay $5
for <a href="http://www.smustard.com/script/SelectionMemory2">SelectionMemory2</a> (not mentioning the bad interface it has), I went
to write my own plugin for saving and restoring selections.</p>
<p>The main benefits compared to other solutions are:</p>
<ul>
<li>It is free</li>
<li>It has no artificial limit with regard to how many selections can be saved</li>
<li>It is driven via the context menu</li>
<li>Selections can be saved with the model and won’t be lost when closing SketchUp.</li>
</ul>
<p>For the readme as well as the plugin itself, check the <a href="https://github.com/dominikh/sketchup-selection_manager">git repository at GitHub</a>.</p>
tag:dominik.honnef.co,2011-05-01:/posts/2011/05/rsemantic/rsemantic – Latent Semantic Analysis for Ruby2011-05-01T14:39:47Z2011-05-01T14:39:47Z
<h3 id="introduction">Introduction</h3>
<p>Some time ago, I started reading
<a href="http://astore.amazon.com/mojombo-20/detail/1594487715"><em>Where Good Ideas Come From: The Natural History of Innovation</em></a>
and one thing it mentions is an application called “DEVONthink”.
DEVONthink is an application you’d use for storing all kind of
documents (RSS feeds, emails, notes, articles) and the by far most
interesting feature is its ability to recommend similar documents,
without asking the user to provide categories/tags or summaries by
himself. Instead, DEVONthink automatically analyses all your documents
and calculates the similarity.</p>
<p>The really interesting part here is that DEVONthink is able to tell if
different words have similar meanings, e.g. “canine” and “dog”. That
means that if you’re reading a new article which talks mostly about
dogs, DEVONthink will be able to suggest an article about canines.
This isn’t done by a fixed list of words with similar meanings, but
again by analysing all documents and checking which words get used
together often.</p>
<p>Unfortunately, DEVONthink is only available for OS X and not for
Linux. Besides that, it works with a centralised database, while I
believe in the beauty of plain text files. That is why I began
researching how DEVONthink is able to successfully find similar
documents and I quickly found Latent Semantic Analysis, short
LSA.</p>
<h3 id="latent-semantic-analysis">Latent Semantic Analysis</h3>
<p>Now, I do not want to explain LSA in detail, but summarized it works
as follows: First, a term-document matrix gets constructed, storing
which terms occur how often in which documents. Such a matrix usually
is quite sparse, meaning that it is mostly populated with zeros. LSA
then decomposes the matrix into three parts (this is known as
<a href="http://en.wikipedia.org/wiki/Singular_value_decomposition">Singular Value Decomposition, SVD</a>), reduces the singular values to
the K biggest values and then recomposes the matrix. This effectively
reduces the dimensions of the vector space, putting similar words
closer together.</p>
<p>For a detailed (and mathematically accurate) description, check out
the <a href="http://en.wikipedia.org/wiki/Latent_semantic_analysis">Wikipedia article about LSA.</a></p>
<h3 id="lsa-in-ruby">LSA in Ruby</h3>
<p>Trying to avoid having to do the math by myself, I looked for a ready
library for Ruby that’d offer LSA. I found rsemantic, which by itself
looked promising, not only offering LSA but also <a href="http://en.wikipedia.org/wiki/Tf-idf">tf-idf</a> (a way to
give common words negative scores).</p>
<p>The only problem with rsemantic was, that it depends on <a href="http://rubyforge.org/projects/linalg/">linalg</a>,
another Ruby library, which provides binding to <a href="http://en.wikipedia.org/wiki/LAPACK">LAPACK</a>, a Fortran
library for linea algebra. Now the problem with linalg is that it
doesn’t compile on modern systems, at all. Apparently it requires gcc
3.x, and the maintainer has seemingly abandonned the project quite
some time ago.</p>
<p>Looking for an alternative to either rsemantic (which wasn’t actively
developed anymore, either) or linalg, I found the GNU Science Library
(GSL), for which someone wrote working Ruby bindings.</p>
<p>The next step was obvious: port rsemantic to GSL. At the same time, I
was able to improve other parts of rsemantic, cleaning up the whole
code base and drastically improving the performance, to a point where
the only time consuming process is doing the SVD itself.</p>
<h3 id="the-future-of-rsemantic">The future of rsemantic</h3>
<p>Luckily, the developer of rsemantic is still interested in the project
(and even actively using it, according to him). He therefore merged my
changes and added me as a contributor to the
<a href="https://github.com/josephwilk/rsemantic">git repository at GitHub</a>, allowing me to further work on the
library.</p>
<p>A new version of the gem will be released as soon as both of us are
happy with all the recent changes.</p>
<p>Furthermore, I am working on an improved API for rsemantic. While
currently it is pretty low-level (and not that object-oriented), I am
working on a far more high-level API. Following are two examples,
first for the new API, then for the old one.</p>
<h4 id="new-api">New API</h4>
<pre class="ruby"><code>require "semantic"
require "pp"
corpus = Semantic::Corpus.new([], :transforms =&gt; [:LSA])
# The directory /tmp/some_texts/ contains Wikipedia articles about
# various animals (beagle.txt canidae.txt cat.txt dog.txt gecko.txt
# german_shepherd.txt reptile.txt salamander.txt)
Dir.glob("/tmp/some_texts/*.txt").each do |file|
# Here we add each article to the corpus (collection of documents).
# `:name` is an attribute we're assigning to the document. There is
# no limit in the names and amount of attributes you can attach to
# each document.
corpus &lt;&lt; Semantic::Document.new(open(file), :name =&gt; File.basename(file))
end
# Build the index, Depending on the CPU, available memory and most
# importantly the size of the corpus, this can take some time.
corpus.build_index
# Compares the article about dogs to all other articles and returns a
# score for each article, representing the similarity.
# In the example output we can see, that rsemantic correctly says that
# "dog.txt" is by far the most related to
# {german_shepherd,beagle,canidae}.txt, and not to e.g. reptiles.
pp corpus.documents.find { |d| d[:name] == "dog.txt" }.related
# {#&lt;Semantic::Corpus 8 documents, @options={:transforms=&gt;[:LSA]}&gt;=&gt;
# [#&lt;Semantic::SearchResult:0x0000000283db30
# @document=#&lt;Semantic::Document @attributes={:name=&gt;"gecko.txt"}&gt;,
# @score=0.05935175420757509&gt;,
# #&lt;Semantic::SearchResult:0x0000000283dbd0
# @document=#&lt;Semantic::Document @attributes={:name=&gt;"reptile.txt"}&gt;,
# @score=0.09322815190241214&gt;,
# #&lt;Semantic::SearchResult:0x0000000283db08
# @document=#&lt;Semantic::Document @attributes={:name=&gt;"cat.txt"}&gt;,
# @score=0.14170011833185264&gt;,
# #&lt;Semantic::SearchResult:0x0000000283db80
# @document=#&lt;Semantic::Document @attributes={:name=&gt;"salamander.txt"}&gt;,
# @score=0.17269285169838045&gt;,
# #&lt;Semantic::SearchResult:0x0000000283db58
# @document=#&lt;Semantic::Document @attributes={:name=&gt;"canidae.txt"}&gt;,
# @score=0.2107975969277189&gt;,
# #&lt;Semantic::SearchResult:0x0000000283dba8
# @document=#&lt;Semantic::Document @attributes={:name=&gt;"beagle.txt"}&gt;,
# @score=0.4041096720670102&gt;,
# #&lt;Semantic::SearchResult:0x0000000283dbf8
# @document=#&lt;Semantic::Document @attributes={:name=&gt;"german_shepherd.txt"}&gt;,
# @score=0.508139138880475&gt;,
# #&lt;Semantic::SearchResult:0x0000000283dc20
# @document=#&lt;Semantic::Document @attributes={:name=&gt;"dog.txt"}&gt;,
# @score=1.0000000000000688&gt;]}
</code></pre>
<h4 id="old-api">Old API</h4>
<pre class="ruby"><code>require "semantic"
require "pp"
corpus = Semantic::Corpus.new([], :transforms =&gt; [:LSA])
documents = []
document_names = []
Dir.glob("/tmp/some_texts/*.txt").each do |file|
documents &lt;&lt; open(file).read
document_names &lt;&lt; File.basename(file)
end
# This automatically builds the index for all documents
search = Semantic::Search.new(documents, :transforms =&gt; [:LSA])
document = document_names.index("dog.txt")
h = {}
search.related(document).each_with_index do |score, index|
h[document_names[index]] = score
end
# The resulting scores are the same, but we have to manually keep
# track of file names, indices, and the scores are not sorted, either.
pp h
# {"dog.txt"=&gt;1.0000000000000688,
# "german_shepherd.txt"=&gt;0.508139138880475,
# "reptile.txt"=&gt;0.09322815190241214,
# "beagle.txt"=&gt;0.4041096720670102,
# "salamander.txt"=&gt;0.17269285169838045,
# "canidae.txt"=&gt;0.2107975969277189,
# "gecko.txt"=&gt;0.05935175420757509,
# "cat.txt"=&gt;0.14170011833185264}
</code></pre>
tag:dominik.honnef.co,2010-10-27:/posts/2010/10/tmux-vs-screen/Why You Should Try tmux Instead of screen2010-10-27T01:42:05Z2010-10-27T01:42:05Z
<h3 id="what-screen-is">What screen is</h3>
<p>If you’re reading this article, you probably already know what <em>GNU
Screen</em> is: the most well-known terminal multiplexer in the world of
Unix. If that doesn’t ring any bells, I recommend reading
<a href="http://en.wikipedia.org/wiki/GNU_Screen">the Wikipedia article about GNU Screen</a>
before continuing with this article.</p>
<h3 id="an-alternative">An Alternative</h3>
<p>There are, however, also some other terminal multiplexers besides GNU
screen, most notably <a href="http://sourceforge.net/projects/tmux/">tmux</a>. While technically screen and tmux serve the
same purpose – multiplexing terminals – their functionality and
quality differ drastically.</p>
<h3 id="the-history-of-screen-and-tmux">The History of screen And tmux</h3>
<p>Development of screen started in 1987; additions and fixes have
been contributed ever since. On the one hand, that means that screen
is a solid and proven piece of software; On the other hand, though, it
also means that the codebase isn’t the cleanest one nor easy to
understand. As a result of this, new additions are rare and
development mostly concentrates on fixing existing bugs.</p>
<p>Tmux, on the other side, saw the light of day in late 2007. And
because the developer was able to learn from screen’s mistakes while
at the same time picking up existing and field-tested features, tmux
has a solid and extensible code base – and it is actively developed by
a very responsive developer.</p>
<h3 id="actual-new-features">Actual New Features</h3>
<p>So, tmux has a better codebase and has (nearly, more on that later)
all the features screen has – but chances are high that you are not
interested in the code at all, so why should you care? Because tmux
has a lot of features that screen has always been missing.</p>
<h4 id="a-sensible-configuration-format">A Sensible Configuration Format</h4>
<p>If there was only one shortcoming of screen that you were allowed to
name, which one would it be? Right, the format of the configuration
files…</p>
<p>Let’s look at a prominent example: a hardstatus (called “status line”
in tmux) that contains the computer’s hostname, a list of all windows,
the load average and a clock:</p>
<pre class="text"><code>hardstatus alwayslastline '%{gk}[ %{G}%H %{g}][%= %{wk}%?%-Lw%?%{=b kR}(%{W}%n*%f%t%?(%u)%?%{=b kR})%{= kw}%?%+Lw%?%?%= %{y}][%{G} %l%{y}] %{y}[%{G} %m/%d %c %{y}]%{W}'
</code></pre>
<p>
<img src="/img/tmux_screen/screen_status_bar.png" alt="screen status_bar" />
</p>
<p>If I didn’t tell you what that option did, were you able to understand
it, or more importantly, able to write it yourself? No, you probably
wouldn’t. (If you, in fact, are able to: congratulations, and sincere
condolences).</p>
<p>So, how would that look like in tmux? Here’s how:</p>
<pre class="text"><code>set -g status-left ' #[fg=green,bold]#H#[default]'
set -g status-right '#[fg=yellow,bold]#(cut -d " " -f 1-4 /proc/loadavg)#[default] #[fg=cyan,bold]%Y-%m-%d %H:%M:%S#[default] '
</code></pre>
<p>
<img src="/img/tmux_screen/tmux_status_bar.png" alt="tmux status_bar" />
</p>
<p>Okay, so it’s easy to see where colors are being set (and yes, they do
differ from the screen example, just ignore that. It would look the
same for other colors, too), it’s kind of obvious what “#H” means (and
the man pages of tmux have a complete list of all available codes), we
instantly can see how the load average is displayed, and the clock
format string should be obvious.</p>
<p>But wait, where’s the list of windows? That’s a built-in feature of
tmux and thus doesn’t have to be set explicitly.</p>
<p>Of course there are a lot of more options, like the following one,
which loads the emacs keymap:</p>
<pre class="text"><code>set-window-option -g mode-keys emacs
</code></pre>
<p>Another major aspect of tmux’s config format is that it is basically a
batch script: it is a simple list of commands that will be executed
when tmux is started – they’re the same commands that you use in
keybinds, the command prompt and shell scripts (more on that in a
minute). If you want to always create two windows when a session is
created, you can put this into your configuration:</p>
<pre class="text"><code>new-window -n some_name
new-window -n another_name
</code></pre>
<h4 id="scriptable">Scriptable</h4>
<p>Another major aspect of tmux is that nearly all commands can be
invoked from the command line, allowing complex scripts controlling
tmux sessions. <code>tmux new-window</code>, for example, would create a new
window. Splitting, moving and resizing windows, setting options and a
lot more actions are possible as well. Even the status bar can
dynamically be changed.</p>
<h4 id="emacs-and-vi-keymaps">Emacs And Vi Keymaps</h4>
<p>tmux supports two different keymaps, imitating either Emacs or vi.
Those keymaps will be used in copy mode and the command prompt.</p>
<h4 id="mouse-support">Mouse Support</h4>
<p>Yeah, power users will smile at this feature, but novices might
actually like it: panes can be selected using the mouse, and in copy
mode, the mouse wheel is supported as well.</p>
<h4 id="menus">Menus</h4>
<p>Nice to have, tmux has menus for selecting windows and sessions.
<img src="/img/tmux_screen/tmux_menu.png" alt="menu in tmux" /></p>
<h4 id="multiple-buffers">Multiple buffers</h4>
<p>Much like the kill ring in Emacs, tmux can keep more than one
selection/piece of copied text in memory.</p>
<h4 id="bsd-license">BSD License</h4>
<p>Unlike GNU Screen, tmux is licensed under the simplified BSD license.
Obviously this will please everyone who dislikes the GPL – but there’s
even a real benefit from the choice in license: tmux is part of
OpenBSD since version 4.6.</p>
<h3 id="differences-to-screen">Differences to screen</h3>
<h4 id="server-model">Server Model</h4>
<p>One speciality of tmux is that when creating your first session, it
spawns a server which manages said session and all future sessions. In
fact it is also possible to change the current session on the fly,
using commands or even a fancy navigatable menu.</p>
<p>With tmux, it is not uncommon to have a set of different sessions (one
for work, one for server management, and so on) and to quickly switch
between them, effectively creating another level of grouping of
windows.</p>
<p>Sessions can also be grouped, which means that they share the same set
of windows but have independent options and allow multiple users to
work with the same windows – much like <code>screen -x</code>, just a tad more
flexible.</p>
<h4 id="window-splitting">Window Splitting</h4>
<p>This one is probably the most confusing aspect of tmux, to people
coming from screen. While splitting in screen means displaying
multiple windows at once, in tmux it means splitting a window into
multiple panes. It is, however, possible to take a pane and convert it
into a window and vice versa. It is, however, not possible to display
a window in a pane, without converting it to a pane.</p>
<p>Tmux does, however, natively support both horizontal <strong>and</strong> vertical
splits, without the need for a custom patch, as is the case with
screen.
<img src="/img/tmux_screen/tmux_splits.png" alt="screen status_bar" /></p>
<p>Additionally, tmux has presets for pane arrangements, eliminating the
need for arranging splits by hand.</p>
<h4 id="missing-features">Missing Features</h4>
<p>As mentioned earlier, there is in fact a feature missing in tmux:
built-in serial and telnet support, while screen is able to directly
connect to for example a serial null modem. Even more so, tmux will
probably never include that feature, because the developer considers
it bloat and says, that other, specialized tools should be used for
that task instead.</p>
<h3 id="final-words">Final words</h3>
<p>I hope that this article is giving you enough reasons to at least try
tmux. Of course there might be users who are just happy with screen or
even users who need its support for serial consoles, but most of you
should be able to at least give it a try. I can assure you that you
will not regret it.</p>
<p>I also recommend reading the
<a href="http://wisconsinlinux.org/tmux-manpage-from-cvs/tmux.1.txt">man pages of tmux</a>,
which are written in a very clear and readable way, and just playing
with some of the options.</p>
<h3 id="links">Links</h3>
<dl>
<dt>Wikipedia article about GNU Screen</dt>
<dd><a href="http://en.wikipedia.org/wiki/GNU_Screen">http://en.wikipedia.org/wiki/GNU_Screen</a></dd>
<dt>tmux</dt>
<dd><a href="http://sourceforge.net/projects/tmux/">http://sourceforge.net/projects/tmux/</a></dd>
<dt>man pages of tmux</dt>
<dd><a href="http://wisconsinlinux.org/tmux-manpage-from-cvs/tmux.1.txt">http://wisconsinlinux.org/tmux-manpage-from-cvs/tmux.1.txt</a></dd>
</dl>
tag:dominik.honnef.co,2010-08-04:/posts/2010/08/newton-cinch-merge/Newton and Cinch – One awesome framework2010-08-03T23:45:20Z2010-08-03T23:45:20Z
<p>A lot of time (3 months, actually) has passed since I’ve last blogged
about
<a href="/posts/2010/05/are_lightweight_irc_frameworks_too_lightweight_/">Ruby and available IRC frameworks for writing bots</a>.
Back then I planned to fork Isaac or Cinch, in order to add proper
abstraction and object-orientation to them. Well, I did fork Isaac,
realized that it was, in all honesty, quite a mess and instead wrote
my own framework, mostly from scratch. I did, however, call it
“Newton”, completly ignoring the fact that Isaac was named after Isaac
Hayes and not Isaac Newton. I still thought it was a suiting name for
what was supposed to be a fork.</p>
<p>Well, in my obviously biased opinion,
<a href="http://github.com/dominikh/newton">Newton</a> is the finest Ruby
framework for writing IRC bots. It has it all:</p>
<ul>
<li>a nice DSL</li>
<li>proper debug output</li>
<li>threads</li>
<li>support for all relevant parts of the various IRC RFCs, including support for ISUPPORT</li>
<li>support for plugins (in a
<a href="http://github.com/dominikh/newton/tree/plugins">separate branch</a>)</li>
</ul>
<p>There is, however, one thing that Newton is missing: Users. Shockingly
enough (*cough*), good code (remember? biased opinion…) and
features alone won’t attract a lot of users. Good advertisment does,
though, and since you probably have never heard of Newton before, in
all those months, there doesn’t seem to be any good advertisment to be
going on.</p>
<p>This is where <a href="http://github.com/injekt/cinch">Cinch</a> is coming into
play. Cinch has remarkably many users, even though I think that not
even its developer knows how that’s possible. Anyway, influenced by
Newton (and a funserious rivalry between Cinch and Newton), the
developer of Cinch decided to adapt “some” of my ideas, including more
abstraction, tracking changes in state (joining/leaving users etc.)
and developing a plugin system (admittedly, he did it first, Newton
stole that idea). Threading was also on his to-do list.</p>
<p>And what’s the second most obvious thing to do, if two projects share
the same ideals and both desire attributes of the other project?
Right, consider merging the two projects (the most obvious thing is
starting a huge flame war, of course).</p>
<p>And this is why there won’t be any new Newton releases. Instead, most
of the current codebase of Cinch will be replaced by Newton, while
some features of Cinch (its pattern matcher, mostly) will be
backported to Newton. Yes, this will mean that all existing Cinch bots
will not be compatible to the upcoming version, but on the other side,
new and updated bots will be able to profit from a lot of new awesome
features, which will be documented in a later post, and of course an
official README. The current work on the merge can be seen in the
<a href="http://github.com/injekt/cinch/tree/merge">merge branch</a>, which will
become the master branch in a few weeks.</p>
<p>Lastly, for real facts on why the merge is good for Cinch and Newton:</p>
<table>
<thead>
<tr>
<th>Good for Cinch, because…</th>
<th>Good for Newton, because…</th>
</tr>
</thead>
<tbody>
<tr>
<td>Awesome new features (threading, OO, plugins)</td>
<td>A new name, known by a lot of people</td>
</tr>
<tr>
<td>Two developers instead of one</td>
<td>Reworked documentation, which will be friendlier to newbies</td>
</tr>
<tr>
<td> </td>
<td>Two developers instead of one</td>
</tr>
</tbody>
</table>
<p>That’s it for this article, stay tuned…</p>
tag:dominik.honnef.co,2010-05-01:/posts/2010/05/lightweight-irc-frameworks/Are lightweight IRC frameworks too lightweight?2010-05-01T19:07:07Z2010-05-01T19:07:07Z
<p>Recently I’ve been looking for a small framework for writing IRC bots
in Ruby. Admittedly, I have been successfully running a bot based on
<a href="http://butler.rubyforge.org/manual/quickstart.html">Butler</a> for quite some time now, but I have always been
looking for a more lightweight solution, which does not come with a
predefined plugin or authentication structure.</p>
<p>Now there are probably three major players in the field of IRC bots,
namely <a href="http://github.com/RISCfuture/autumn">Autumn</a>, <a href="http://github.com/ichverstehe/isaac">Isaac</a> and just recently <a href="http://github.com/injekt/cinch">Cinch</a>.</p>
<p>I won’t further examine <a href="http://github.com/RISCfuture/autumn">Autumn</a> here, as it is to IRC bots what Rails
is to web development: Anything but lightweight.</p>
<p><a href="http://github.com/ichverstehe/isaac">Isaac</a> and <a href="http://github.com/injekt/cinch">Cinch</a> are quite similar: They both feature
a small DSL for quickly developing bots, while <a href="http://github.com/injekt/cinch">Cinch</a> puts
more emphasis on the idea of plugins (<a href="http://github.com/ichverstehe/isaac">Isaac</a> is more about
parsing messages in general) and its own pattern language to match
input.</p>
<p>But there is another thing they have in common: No proper
object-oriented interface… Instead of passing around channel and user
objects, all the developer sees are strings. This also means that
instead of doing something as intuitive as</p>
<pre class="ruby"><code>some_channel.part
# or
some_channel.kick(user, "some reason")
# or
some_channel.topic # retrieve the topic
some_channel.topic = "new topic" # set a new topic
# or
some_user.hostname
</code></pre>
<p>one will have to fall back to stuff like this (the examples are based
on <a href="http://github.com/injekt/cinch">Cinch</a> but will be similar for <a href="http://github.com/ichverstehe/isaac">Isaac</a>):</p>
<pre class="ruby"><code>bot.part(some_channel) # now this still makes sense
bot.kick(some_channel, user, "some reason") # not too nice anymore
bot.topic(some_channel) # plain wrong
bot.topic(some_channel, "new topic") # plain wrong
# there simply is no method of retrieving a user's hostname,
# except from doing a WHOIS and parsing the output yourself
</code></pre>
<p>Of course one might argue that those frameworks/DSLs are lightweight
after all and that the user could write helper methods if he really
wanted to have a rich interface, but that would also mean that I had
to use those helper methods every other line, just to deal with proper
objects.</p>
<p>It is quite questionable if “lightweight” is even allowed to mean “not
properly abstracted” or if abstraction is something that can be taken
for granted if working with Ruby.</p>
<p>Either way, as nice as the DSLs of <a href="http://github.com/ichverstehe/isaac">Isaac</a> and <a href="http://github.com/injekt/cinch">Cinch</a>
look, they are still pretty useless to me in their current state.
Instead of reinventing the wheel, however, I will probably fork
<a href="http://github.com/ichverstehe/isaac">Isaac</a> or <a href="http://github.com/injekt/cinch">Cinch</a> and create a “still lightweight”
framework based on that. In the end, I am planning to have an API
which effectively allows code like the following:</p>
<pre class="ruby"><code>on :channel, /^print my host$/ do
# in this example, we will work with an instance of User,
# instead of a plain string with just a nickname. Same goes for `channel`
channel.send "Your host is #{user.host}"
end
on :channel, /!annoy (.+)$/ do |target|
# indeed a helper for cases in which we have to deal with strings
Channel(target).each_user do |victim|
victim.send "hi, you have been annoyed!"
end
end
</code></pre>
tag:dominik.honnef.co,2009-10-11:/posts/2009/10/ruby-dynamic-method/Dynamically defining a method on a single instance in Ruby2009-10-10T22:00:00Z2009-10-10T22:00:00Z
<p>Given the following initial code…</p>
<pre class="ruby"><code>class Dog
def initialize(name)
@name = name
end
def bark
puts "woof"
end
end
charlie = Dog.new("Charlie")
lucas = Dog.new("Lucas")
</code></pre>
<p>… one might want to teach one of the dogs a new trick, while
the others shouldn’t automatically learn it. One possible
solution could look like the following:</p>
<pre class="ruby"><code>def charlie.jump
puts "wee, #@name is jumping around"
end
</code></pre>
<p>Another way would be to provide a teach_trick method, which
dynamically defines a new method:</p>
<pre class="ruby"><code>class Dog
# [...]
def teach_trick(name, &amp;block)
(class &lt;&lt; self; self; end).class_eval do
define_method name, &amp;block
end
end
end
charlie = Dog.new("Charlie")
lucas = Dog.new("Lucas")
charlie.teach_trick :dance do
puts "#@name is dancing around!"
end
charlie.dance # &gt;&gt; Charlie is dancing around!
lucas.dance # ~&gt; -:23:in `&lt;main&gt;': undefined method `dance' for #&lt;Dog:0x9a62e70 @name="Lucas"&gt; (NoMethodError)
</code></pre>
<p>What this does is it defines a new method on the eigenclass of the
instance of a dog, so when the called method, <em>dance</em> in
this case, is not found in the instance methods of the class Dog, it
searches in the eigenclass of the instance instead.</p>
<p>One advantage of this solution is that the actual implementation of
how new tricks are taught is hidden, the user does not need to know he
is defining a method anymore, he is just teaching his dog a new trick.
Another example is when you have to define a bunch of methods
depending on some input, like a hash, which is completly dynamic.</p>
<p>One disadvantage, on the other hand, is that calling methods defined
by define_method is a bit slower than those defined by def, but the
factor is so small that you probably won’t notice it in real
applications:</p>
<pre class="ruby"><code>Benchmark.bmbm(10) do |x|
x.report("eigenclass:") { 1_000_000.times {d1.trick} }
x.report("instance def:") { 1_000_000.times {d2.trick} }
end
# &gt;&gt; Rehearsal -------------------------------------------------
# &gt;&gt; eigenclass: 0.440000 0.000000 0.440000 ( 0.453996)
# &gt;&gt; instance def: 0.320000 0.000000 0.320000 ( 0.322817)
# &gt;&gt; ---------------------------------------- total: 0.760000sec
# &gt;&gt;
# &gt;&gt; user system total real
# &gt;&gt; eigenclass: 0.450000 0.000000 0.450000 ( 0.451315)
# &gt;&gt; instance def: 0.310000 0.000000 0.310000 ( 0.308632)
</code></pre>
tag:dominik.honnef.co,2009-10-05:/posts/2009/10/getting-help-on-irc/About getting help on IRC2009-10-04T22:00:00Z2009-10-04T22:00:00Z
<p>There is one thing that either we, the helping people, or those that
search for help with their code don’t get, and that’s what
kind of answers are to be expected.</p>
<p>Every too often I see people joining certain channels, in particular
#ruby and #ruby-lang, asking questions about why their code
doesn’t work or how to implement feature XY. And of course they
show others their existing code, which usually isn’t too great,
looking like a direct port from another language or simply like the
one of a newbie (not that being a newbie is bad in any way, we all
started as one).</p>
<p>Usually what follows isn’t a straight forward answer like
“write this to get your feature” or “change that on
line X to get it working” but a general discussion about the
whole piece of code, how one shouldn’t write it like that at all
and that, if it was done right, the problem wouldn’t have
occured at all.</p>
<p>The purpose of this definitely is not to offend, discriminate or
degrade anyone, but to teach the people how to code properly, how they
can save time, nerves and money, and what it takes to become a good
programmer. Programming isn’t about just hacking together a few
lines that, by chance, end up doing what you expected them to do, but
to create a well designed application, that can be maintained and
extended at need.</p>
<p>Nevertheless we can see two completly different kinds of reaction. The
first one is the person being thankful for all the great tips by the
“pros”, even though he might be confused because of all
the information at first. The other one though gets angry and tells
everyone how he “wanted to fix a problem, not rewrite the whole
damn thing from scratch”, generally feeling offended by all the
well-meant advices.</p>
<p>So, the question is, who is right? Are we supposed/allowed to comment
on the overall quality of the publicly posted code, or do we only
exist to provide copy&amp;paste solutions to people desinterested in
quality of design? In my opinion, most if not all of those that spend
their time helping others, want to make a change, want to
<strong>educate</strong>, as opposed to just providing solutions that
work in the given context, not fixing the bigger, the real problems.</p>
<p>I think that if the majority of help seeking people wanted us to stay
shut about the code in a whole and just expected us to provide help
with what exactly they asked, we would end up with not many people
helping anymore.</p>
<p>So, what can <strong>you</strong> do in case you need help with a
problem but do not want any help with your design and patterns? Well,
the easiest option clearly is not to join those channels and try to
figure it out by yourself. But I think another, not too bad, option is
to actually seek help in those channels, but by explicitly saying that
you only <strong>want</strong> (need is a bad word, because you
probably do need) help with a certain problem and that you either have
no time for or no interest in other advices. While this might put off
some people you eventually will end up with one or another helping you
with your particular problem, not wasting his, her or your time with
information other than the ones you need.</p>
<p>In my opinion it definitely is wrong to ignore those advices made by
professionals, but it is even more wrong to get angry about them.
They’re well-meant, usually correct and you should respect them
if you want to program in a proper manner. Yet, if you do not care
about any of that, at least be so fair and tell us in advance so we
can save our time.</p>
<h4 id="update-as-of-20091007">Update, as of 2009/10/07</h4>
<p>Okay, I’ve read all the comments about my article (both the ones
posted on my blog and the ones on
<a href="http://www.reddit.com/r/programming/comments/9rhf7/getting_help_on_irc_why_people_criticize_your_code/">reddit</a>)
and it looks like I made one major mistake when writing this article:
I was only thinking of the two main Ruby channels on FreeNode. While
my article, in my opinion, totally holds true for those two channels
it might as well be true that certain other channels are literally
polluted by people who actively try to bash and piss off newbies.
Alternatively, other channels seem to have a high amount of people
with no deeper knowledge, who, nevertheless, try to force their
opinion upon others, maybe not even backing it up.</p>
<p>So, what are people with problems supposed to do then? Avoid IRC
alltogether? Hope to be lucky to have picked a language with a decent
community? Only try to get help on IRC if you are able to tell apart
bollocks from qualified help? I don’t know, but maybe you got
the ultimate solution? And if so, maybe you don’t mind sharing
it with us? Comments appreciated!</p>
tag:dominik.honnef.co,2009-08-08:/posts/2009/08/apache-lighttpd-proxy/Apache behind a remote lighttpd proxy and the caveats2009-08-07T22:00:01Z2009-08-07T22:00:01Z
<h4 id="background">Background</h4>
<p>The subdomain I use belongs to a friend, who isn’t able to
set A or CNAME entries, so he uses lighttpd + mod_proxy to redirect it
to my server.</p>
<h4 id="the-problem">The Problem</h4>
<p>And that’s where the problems started. Because of the proxy,
the HTTP REMOTE_ADDR isn’t the one of the visitor anymore but
the one of the proxy, which in turns collides with log analysis, GeoIP
and alike.</p>
<h4 id="the-solution">The Solution</h4>
<p>In general, a simple solution would be to use the Apache module
“rpaf”, which uses the X-Forwarded-For header to properly
set REMOTE_ADDR.</p>
<p>In my case, that wasn’t the whole solution though. Instead of
an IPv4 address, the lighttpd installation of my friend transmitted an
IPv6 one, which then again didn’t work with GeoIP either, as it
only supports IPv4. In this case, the solution was to run a local
lighttpd, which itself uses mod_extforward to make use of
X-Forwarded-For, and then properly forward the request to Apache, this
time using IPv4.</p>
<p>Oh, also important: rpaf has to be loaded before GeoIP, otherwise
the latter uses the wrong IP for lookups.</p>
tag:dominik.honnef.co,2009-08-07:/posts/2009/08/conditionally-call-super/Only call super when possible2009-08-06T22:00:00Z2009-08-06T22:00:00Z
<p>Today we were faced with an interesting question in #ruby-lang:</p>
<blockquote>
<p>&lt;Judofyr&gt; is there a way to find out if it’s safe to call “super”?<br />
&lt;Judofyr&gt; aka. it’s defined on the superclass</p>
</blockquote>
<p>After some ugly solutions (like self.class.ancestors.any? and super rescue nil) apeiros came up with a nicer one:</p>
<blockquote>
<p>&lt;apeiros_&gt; super if defined? super</p>
</blockquote>
<p>Example:</p>
<pre class="ruby"><code>class Test
def foo
1
end
end
class Test2 &lt; Test
def foo; super if defined? super; end
def bar; super if defined? super; end
end
t = Test2.new
t.foo # =&gt; 1
t.bar # =&gt; nil
</code></pre>
tag:dominik.honnef.co,2009-08-04:/posts/2009/08/smalltalk-best-practice-patterns-review/"Smalltalk Best Practice Patterns" reviewed2009-08-03T22:00:00Z2009-08-03T22:00:00Z
<p>Today I read the first volume of Smalltalk Best Practice Patterns, or
to be precise, a draft I found on Google. While I am a Ruby programmer
and never even touched Smalltalk in my life, it is an undeniable fact
that Ruby and Smalltalk have many things in common, or to put it in
another way, that Ruby (or any real OO language really) was inspired
by Smalltalk.</p>
<p>The book is, like the title already says, about the “best
practice patterns”, which covers naming conventions, coupling,
splitting up methods and much more.</p>
<p>The book starts with an introduction into what patterns are, what they
are needed for and how those in the book were developed and describes
the intention of the book by one simple sentence:</p>
<blockquote>
<p>This book is about making code that works for you.</p>
</blockquote>
<p>It then, basically, consists of a long list of different patterns,
which are described by a title, patterns preceding and following it,
the problem stated as a question, possible problems with the pattern,
the solution and a discussion of the solution.</p>
<p>Two important things about the patterns in this book, that distinguish
them from other collections of patterns, are that they are discussed
(instead of simply saying that they are the way to go) and that they
are put into context, as most patterns can’t work if not
combined with others.</p>
<p>Having read the book, I can say, that most if not all of the patterns
covered in the book are also applicable to the Ruby world, even though
some of them had to be slightly adjusted due to differences between
Ruby and Smalltalk. But those changes only affected details, like the
names of methods, and not the idea behind them.</p>
<p>It was kind of hard for me to decide if and to whom I should recommend
this book. But I think this puts it the best way: If you are already
familiar with the language you are using and also interested in proper
design patterns, you should absolutely read this book, provided that
you are able to adapt some Smalltalk specific terminology to the ones
that suit the language you use. And even if you don’t want to
pick up the author’s patterns, the first 40 pages of the book
still help you defining your own patterns.</p>