Rob Reid/index.xml
Recent content on Rob ReidHugo -- gohugo.ioen-gbMon, 20 Aug 2018 09:38:28 +0100runtime.Goexit/runtime-goexit/
Mon, 20 Aug 2018 09:38:28 +0100/runtime-goexit/
<p>If you&rsquo;ve ever needed to kick off multiple goroutines from <code>func main</code>, you&rsquo;d have probably noticed that the main goroutine isn&rsquo;t likely to hang around long enough for the other goroutines to finish:</p>
<pre><code class="language-go">package main
import (
&quot;fmt&quot;
&quot;time&quot;
)
func main() {
go run(1, &quot;A&quot;)
go run(5, &quot;B&quot;)
}
func run(iter int, name string) {
for i := 0; i &lt; iter; i++ {
time.Sleep(time.Second)
fmt.Println(name)
}
}
</code></pre>
<p>It&rsquo;ll come as no surprise that this program outputs nothing and exits with an exit code of 0. The nature of goroutines is to be asynchronous, so while the &ldquo;A&rdquo; and &ldquo;B&rdquo; goroutines are being scheduled, the main goroutine is running to completion and hence closing our application.</p>
<p>There are many ways to run both the &ldquo;A&rdquo; and &ldquo;B&rdquo; goroutines to completion, some more involved than others. Here are a few:</p>
<h5 id="a-name-synchronous-a-run-a-goroutine-synchronsly"><a name="synchronous"></a>Run a goroutine synchronsly</h5>
<p>If you&rsquo;re confident that one of your goroutines will run for longer than the other, you could simply call one of the routines synchronously and hope for the best:</p>
<pre><code class="language-go">package main
import (
&quot;fmt&quot;
&quot;time&quot;
)
func main() {
go run(1, &quot;A&quot;)
run(5, &quot;B&quot;)
}
func run(iter int, name string) {
for i := 0; i &lt; iter; i++ {
time.Sleep(time.Second)
fmt.Println(name)
}
}
</code></pre>
<pre><code class="language-bash">$ go run main.go
B
A
B
B
B
B
&lt;EXIT 0&gt;
</code></pre>
<p>This of course falls down if the goroutine you&rsquo;re waiting on takes less time than the other, as the only thing keeping your application running is the goroutine you&rsquo;re running synchronously:</p>
<pre><code class="language-go">go run(5, &quot;A&quot;)
run(1, &quot;B&quot;)
</code></pre>
<pre><code class="language-bash">$ go run main.go
B
&lt;EXIT 0&gt;
</code></pre>
<p>&hellip;so not a workable solution unless you&rsquo;re running things like long-running web servers.</p>
<h5 id="a-name-waitgroup-a-sync-waitgroup"><a name="waitgroup"></a>sync.WaitGroup</h5>
<p>A more elegant solution would be to use <code>sync.WaitGroup</code> configured with a delta equal to the number of goroutines you&rsquo;re spawning. Your application will run to completion after all of the goroutines exit.</p>
<p>In the following example, I&rsquo;m assuming that we don&rsquo;t have access to the <code>run</code> function and so am dealing with the <code>sync.WaitGroup</code> internally to the <code>main</code> function.</p>
<pre><code class="language-go">package main
import (
&quot;fmt&quot;
&quot;sync&quot;
&quot;time&quot;
)
func main() {
var wg sync.WaitGroup
wg.Add(2)
go func() {
defer wg.Done()
run(1, &quot;A&quot;)
}()
go func() {
defer wg.Done()
run(5, &quot;B&quot;)
}()
wg.Wait()
}
func run(iter int, name string) {
for i := 0; i &lt; iter; i++ {
time.Sleep(time.Second)
fmt.Println(name)
}
}
</code></pre>
<pre><code class="language-bash">$ go run main.go
B
A
B
B
B
B
&lt;EXIT 0&gt;
</code></pre>
<p>This is a more elegant solution to the <a href="#synchronous">hit-and-hope</a> solution as it leaves nothing to chance. As with the above example, you&rsquo;ll likely want/need to keep the wait group code within your <code>main</code> function, so provided you don&rsquo;t mind polluting it with synchronisation code, you&rsquo;re all good.</p>
<p>If you need to add/remove a goroutine, don&rsquo;t forget to increment the delta, or your application won&rsquo;t behave as expected!</p>
<h5 id="a-name-channels-a-channels"><a name="channels"></a>Channels</h5>
<p>It&rsquo;s also possible to use channels to acheive this behaviour, by creating a buffered channel with the same size as the delta you initialised the <code>sync.WaitGroup</code> with.</p>
<p>In the below example, I once again assume no access to the <code>run</code> function and keep all synchronisation logic in the <code>main</code> function:</p>
<pre><code class="language-go">package main
import (
&quot;fmt&quot;
&quot;time&quot;
)
func main() {
done := make(chan struct{})
go func() {
defer func() { done &lt;- struct{}{} }()
run(1, &quot;A&quot;)
}()
go func() {
defer func() { done &lt;- struct{}{} }()
run(5, &quot;B&quot;)
}()
for i := 0; i &lt; 2; i++ {
&lt;-done
}
}
func run(iter int, name string) {
for i := 0; i &lt; iter; i++ {
time.Sleep(time.Second)
fmt.Println(name)
}
}
</code></pre>
<pre><code class="language-bash">$ go run main.go
B
A
B
B
B
B
</code></pre>
<p>The obvious added complexity and the fact that the synchronisation code needs to be updated if a goroutine needs to be added/removed detract from the elegance of this approach. Forget to increment your channel&rsquo;s reader delta and your application will exit earlier than expected and forget to decrement it and it&rsquo;ll crash with a deadlock.</p>
<h5 id="a-name-goexit-a-runtime-goexit"><a name="goexit"></a>runtime.Goexit()</h5>
<p>Another solution is to use the runtime package&rsquo;s <code>Goexit</code> function. This function executes all deferred statements and then stops the calling goroutine, leaving all other goroutines running. Like all other goroutines, <code>Goexit</code> can be called from the main goroutine to kill it and allow other goroutines to continue running.</p>
<p>Exit wise, once the <code>Goexit</code> call is in place, your application can only fail. If your application is running in an orchestrated environment like Kubernetes (or you&rsquo;re just happy to tolerate non-zero exit codes), this might be absolutely fine but it&rsquo;s something to be aware of.</p>
<p>There are two ways your application can now exit (both resulting in an exit code of 2):</p>
<ul>
<li>If all of the other goroutines run to completion, there&rsquo;ll be no more goroutines to schedule and so the runtime scheduler will panic with a deadlock informing you that <code>Goexit</code> was called and that there are no more goroutines.</li>
<li>If any of the other goroutines panic, the application will crash as if any other unrecovered panic had occurred.</li>
</ul>
<p>With all the doom and gloom out the way, let&rsquo;s take a look at the code:</p>
<pre><code class="language-go">package main
import (
&quot;fmt&quot;
&quot;runtime&quot;
&quot;time&quot;
)
func main() {
go run(1, &quot;A&quot;)
go run(5, &quot;B&quot;)
runtime.Goexit()
}
func run(iter int, name string) {
for i := 0; i &lt; iter; i++ {
time.Sleep(time.Second)
fmt.Println(name)
}
}
</code></pre>
<pre><code class="language-bash">$ go run main.go
B
A
B
B
B
B
fatal error: no goroutines (main called runtime.Goexit) - deadlock!
&lt;STACK OMITTED&gt;
&lt;EXIT 2&gt;
</code></pre>
<p>Succinct, if a little scary!</p>
<p>This solution understandably won&rsquo;t be for everyone, especially if you&rsquo;re working with inexperienced gophers (for reasons of sheer confusion, &ldquo;my application keeps failing&rdquo; and &ldquo;nice, I&rsquo;ll use this everywhere&rdquo;) but it&rsquo;s nevertheless an interesting one, if only from an academic perspective.</p>
how i test endpoints/how-i-test-endpoints/
Wed, 18 Apr 2018 09:37:45 +0100/how-i-test-endpoints/
<p>When developing a service, you&rsquo;ll want to ensure that you have decent test coverage of its endpoints. I&rsquo;ve seen endpoint coverage achieved in a number of ways, some interesting, some <em>interesting</em> and I&rsquo;d like to share my method.</p>
<p>If you&rsquo;ve arrived here, you&rsquo;re probably no stranger to validating input to endpoints and it&rsquo;s this use case that I&rsquo;ll be demonstrating here.</p>
<p>First and foremost, I&rsquo;m using table-drive tests. In the context of service testing, they allow me to make multiple, requests to a service, using similar request bodies, without changing the test code I&rsquo;m using to invoke the service, or polluting my test cases with huge, largely identical code.</p>
<p>To keep things simple, I&rsquo;ve written a very contrived example of a service, which satisfies the following contract:</p>
<ul>
<li>Takes an <code>application/json</code> body which contains a first and last name.</li>
<li>It asserts that both the first and second name have been provided and are of sensible lengths.</li>
<li>It returns a full name, which is the concatenation of first and last name.</li>
</ul>
<h5 id="code">Code</h5>
<p>I&rsquo;m using <a href="https://github.com/asaskevich/govalidator">govalidator</a> to validate the input and in order to make the various error states obvious, I&rsquo;m using custom error message. So you&rsquo;ll have to excuse the verbosity:</p>
<pre><code class="language-go">type fullNameRequest struct {
First string `json:&quot;first&quot; valid:&quot;runelength(1|100)~first name should contain 1-100 characters,required~first name required&quot;`
Last string `json:&quot;last&quot; valid:&quot;runelength(1|100)~last name should contain 1-100 characters,required~last name required&quot;`
}
</code></pre>
<p>The endpoint is simple, it binds the request body to a <code>fullNameRequest</code> struct and validates it. If the request isn&rsquo;t valid, it&rsquo;ll bail with a <code>422</code> and if it&rsquo;s valid, the full name will be returned:</p>
<pre><code class="language-go">func getFullName(w http.ResponseWriter, r *http.Request) {
var request fullNameRequest
if err := bindAndValidateJSON(r.Body, &amp;request); err != nil {
http.Error(w, err.Error(), http.StatusUnprocessableEntity)
return
}
w.WriteHeader(http.StatusOK)
w.Write([]byte(fmt.Sprintf(&quot;%s %s\n&quot;, request.FirstName, request.LastName)))
}
</code></pre>
<p>While not essential to this example, the helper methods I&rsquo;m using for binding and validating are copy/paste-friendly, so I&rsquo;ve included them in case you want them:</p>
<pre><code class="language-go">func bindAndValidateJSON(rc io.ReadCloser, target interface{}) error {
if err := bindJSON(rc, target); err != nil {
return err
}
_, err := govalidator.ValidateStruct(target)
return err
}
func bindJSON(rc io.ReadCloser, target interface{}) error {
defer rc.Close()
body, err := ioutil.ReadAll(rc)
if err != nil {
return err
}
return json.Unmarshal(body, target)
}
</code></pre>
<p>To run the server, it&rsquo;s the standard two-liner:</p>
<pre><code class="language-go">func main() {
http.HandleFunc(&quot;/&quot;, getFullName)
http.ListenAndServe(&quot;:1234&quot;, nil)
}
</code></pre>
<p>Let&rsquo;s give it a whirl, to see if we get what we&rsquo;re expecting from the various requests we could throw at it:</p>
<pre><code class="language-bash">curl localhost:1234/ -d '{&quot;first&quot;: &quot;Rob&quot;, &quot;last&quot;: &quot;Reid&quot;}'
Rob Reid
curl localhost:1234/ -d '{&quot;last&quot;: &quot;Reid&quot;}'
first name required
curl localhost:1234/ -d '{&quot;first&quot;: &quot;Rob&quot;}'
last name required
</code></pre>
<h5 id="test-code">Test Code</h5>
<p>Rather than paste one massive test method, I&rsquo;ve separate each section out into it&rsquo;s component parts. If you&rsquo;re familiar with <a href="https://blog.golang.org/subtests">table-drive tests</a>, the following will still read fairly naturally to you. Here&rsquo;s a high-level workflow of what the test is doing:</p>
<ul>
<li>For each of the test cases:
<ul>
<li>Create a valid request for the endpoint using a struct.</li>
<li>Modify it to make it invalid.</li>
<li>Marshal the stuct to something we can throw at the endpoint.</li>
<li>Perform some expectations.</li>
</ul></li>
</ul>
<p>Like in the service code, I&rsquo;ve included any helpers I&rsquo;ve used to A) keep the code clean and B) give you something to copy/paste if you think it&rsquo;d be useful:</p>
<pre><code class="language-go">// ToJSONBody turns a struct into an io.Reader request body.
func ToJSONBody(tb testing.TB, i interface{}) io.Reader {
j, err := json.Marshal(i)
if err != nil {
tb.Fatalf(&quot;error stringifying: %v&quot;, err)
}
return strings.NewReader(string(j))
}
// Equals performs a deep equal comparison against two
// values and fails if they are not the same.
func Equals(tb testing.TB, exp, act interface{}) {
tb.Helper()
if !reflect.DeepEqual(expected, actual) {
tb.Fatalf(&quot;\n\texp: %#[1]v (%[1]T)\n\tgot: %#[2]v (%[2]T)\n&quot;, exp, act)
}
}
</code></pre>
<p>At the top of the test, the test cases are defined as a struct that has a name, a function and some expectations. The interesting bit here is the function, <code>mod</code>. This function allows me to pass a pointer of a <code>fullNameRequest</code> and modify it:</p>
<pre><code class="language-go">cases := []struct {
name string
mod func(r *fullNameRequest)
expCode int
expBody string
}
</code></pre>
<p>Next, are the test definitions themselves. I&rsquo;ve included just enough of them to demonstrate what&rsquo;s happening. Notice that I&rsquo;m not creating a new <code>fullNameRequest</code> for every test case, so the usefulness of this pattern comes into its own as struct size increases.</p>
<pre><code class="language-go">{
name: &quot;missing first name&quot;,
mod: func(r *fullNameRequest) {
r.FirstName = &quot;&quot;
},
expCode: http.StatusUnprocessableEntity,
expBody: &quot;first name required\n&quot;,
},
{
name: &quot;first name too long&quot;,
mod: func(r *fullNameRequest) {
r.FirstName = strings.Repeat(&quot;a&quot;, 101)
},
expCode: http.StatusUnprocessableEntity,
expBody: &quot;first name should contain 1-100 characters\n&quot;,
},
{
name: &quot;Testy McTestface&quot;,
expCode: http.StatusOK,
expBody: &quot;Testy McTestface\n&quot;,
}
</code></pre>
<p>In the main body of the function, we&rsquo;re creating a decent request and passing it to the <code>mod</code> function if one has been provided. We&rsquo;re then passing the request through the <code>ToJSONBody</code> function to derive an <code>io.Reader</code> to chuck at the endpoint. Finally, we&rsquo;re asserting that the expected status code and response body have been returned.</p>
<pre><code class="language-go">t.Run(c.name, func(t *testing.T) {
body := fullNameRequest{
FirstName: &quot;Rob&quot;,
LastName: &quot;McSomething&quot;,
}
if c.mod != nil {
c.mod(&amp;body)
}
w := httptest.NewRecorder()
r := httptest.NewRequest(&quot;GET&quot;, &quot;/&quot;, ToJSONBody(t, body))
getFullName(w, r)
Equals(t, c.expCode, w.Code)
Equals(t, c.expBody, w.Body.String())
})
</code></pre>
test tags/test-tags/
Fri, 13 Apr 2018 09:37:12 +0100/test-tags/<p>One of the most useful go build flags I&rsquo;ve used recently is tags. Its purpose is to look at the build tags you&rsquo;ve specified and use them to drive what&rsquo;s sent to the compiler for compilation.</p>
<p>The tags flag is also available to the go test command, which means you can toggle swathes of tests on and off, without having to resort to code changes such as using flag in your TestMain function or - heaven forbid - writing logic into your tests themselves.</p>
<p>As part of a massive Go microservice rollout and to save everyone having to write boilerplate code or learn an ORM, a colleague and I wrote a tool to generate Go structs from MySQL tables and we called it <a href="https://github.com/LUSHDigital/modelgen">modelgen</a>. In addition to outputing generated structs, we also provide some helper structs to handle primative types going into and out of nullable database fields.</p>
<p>These helper structs are bundled into a file called x_helpers.go and their tests are also bundled for completeness, into a file - unsurprisingly - called x_helpers_test.go. These structs are thoroughly tested, which means when the project that references them calls go test -v, there&rsquo;s a lot of output.</p>
<p>The following example demonstrates how we suppress these tests, without resorting to code changes.</p>
<p>To keep things tidy, the following file is a stand-in for x_helpers_test.go. It contains one failing test to make it obvious when the test is executed:</p>
<pre><code>//+build helpers
package main
import &quot;testing&quot;
func TestFail(t *testing.T) {
t.Fatal(&quot;Oh noes!&quot;)
}
</code></pre>
<p>At the top of the file, you&rsquo;ll notice the inclusion of the build tag &ldquo;helpers&rdquo;. This translates to &ldquo;compile this, if the helpers tag has been specified and don&rsquo;t if it hasn&rsquo;t&rdquo;. A test run with the tag specified shows that this test is being executed as expected:</p>
<pre><code>go test --tags helpers
--- FAIL: TestFail (0.00s)
blah_test.go:8: Oh noes!
FAIL
exit status 1
FAIL sandbox 0.039s
</code></pre>
<p>When we run the tests without the &ldquo;helpers&rdquo; tag, the compiler won&rsquo;t compile this file and hence won&rsquo;t run its tests:</p>
<pre><code>go test
PASS
ok sandbox 0.039s
</code></pre>
<p>I like this method for supressing test output for a number of reasons:</p>
<ul>
<li>It&rsquo;s familiar because the use of build tags is widespread in Go.</li>
<li>The default behaviour of go test in this setup is to run the unit tests we&rsquo;re expecting to run and ignore those we&rsquo;re not.</li>
<li>We can toggle the tests to run by including the tag in our build command.</li>
<li>A build tag is less invasive than a code change.</li>
</ul>
errors and defer/errors-and-defer/
Sat, 31 Mar 2018 09:36:36 +0100/errors-and-defer/
<p>The benefits of thorough testing can&rsquo;t be understated, especially when it comes to anything that&rsquo;s in any way magical.</p>
<p>For a language that avoids magic, I&rsquo;d consider Go&rsquo;s named return values and defer pretty magical, when used together. A big assumption and a very subtle syntax oversight and this magic would have lead to a <strong>serious</strong> production issue had it not been for a test that saved the day.</p>
<p>In my latest project, I&rsquo;m performing a lot of database reads/writes, so require database atomicity. Working in a RDBMS/NewSQL environment, that naturally means reaching for transactions. Rather than rolling a new transaction manually every time I need one, I&rsquo;m using a helper function that wraps the creation and commit/rollback of all transactions. Here&rsquo;s a severely cut-down, version of my final implementation, which contains a <strong>critical</strong> bug.</p>
<p>Without cheating, or wishing to sound like a disgraced children&rsquo;s TV presenter, can you guess what it is yet?</p>
<pre><code class="language-go">func executeTx(db *sql.DB, actions func(*sql.Tx) error) error {
tx, err := db.Begin()
if err != nil {
return err
}
defer func() {
if r := recover(); r != nil {
tx.Rollback()
panic(r)
} else if err != nil {
tx.Rollback()
} else {
tx.Commit()
}
}()
return actions(tx)
}
</code></pre>
<p>Let&rsquo;s count the number of rows in <code>some_table</code> before we start to play with it:</p>
<pre><code class="language-bash">+----------+
| count(*) |
+----------+
| 100 |
+----------+
</code></pre>
<p>In the following example, we delete everything from <code>some_table</code>, then return an error, to force our transaction rollback code to kick in and prevent the deletion from being committed:</p>
<pre><code class="language-Go">err := executeTx(db, func(tx *sql.Tx) error {
_, err := tx.Exec(&quot;DELETE FROM `some_table`&quot;)
if err != nil {
return err
}
return errors.New(&quot;oh noes&quot;)
})
</code></pre>
<p>Nothing spooky going going on here. Let&rsquo;s count the number of rows in <code>some_table</code> again, to make sure that our transaction code is doing what it&rsquo;s meant to be doing:</p>
<pre><code class="language-bash">+----------+
| count(*) |
+----------+
| 0 |
+----------+
</code></pre>
<h4 id="oh-my-shit">OH MY SHIT!</h4>
<p>We&rsquo;ve just ensured that the exact opposite of what we want to happen, will happen, every time an error occurs. Now imagine this behaviour had gone unchecked and we&rsquo;d have released it into the wild.</p>
<p>Happily, this code went nowhere near a production database. Here&rsquo;s the unit test that saved the day (error handling omitted for your viewing pleasure):</p>
<pre><code class="language-go">package main
import (
&quot;database/sql&quot;
&quot;errors&quot;
&quot;log&quot;
&quot;testing&quot;
&quot;gopkg.in/DATA-DOG/go-sqlmock.v1&quot;
)
func TestDBRollback(t *testing.T) {
db, mock, _ := sqlmock.New()
mock.ExpectBegin()
mock.ExpectExec(&quot;^DELETE FROM (.+)&quot;)
mock.ExpectRollback()
err := executeTx(db, func(tx *sql.Tx) error {
tx.Exec(&quot;DELETE FROM `some_table`&quot;)
return errors.New(&quot;oh noes&quot;)
})
if err = mock.ExpectationsWereMet(); err != nil {
log.Fatal(err)
}
}
</code></pre>
<p>In this test, I&rsquo;m asserting that a database transaction is started, has a <code>DELETE</code> statement performed against it and is rolled back. All using the wonderful <a href="gopkg.in/DATA-DOG/go-sqlmock.v1">sqlmock</a> package from <a href="https://www.datadoghq.com/">DataDog</a>.</p>
<pre><code class="language-bash">go test -v
=== RUN TestDBRollback
2018/03/30 17:56:36 there is a remaining expectation which was not matched: ExpectedRollback =&gt; expecting transaction Rollback
exit status 1
FAIL pocs/db/mysql 0.033s
</code></pre>
<h5 id="the-problem-solution">The problem/solution</h5>
<p>If you follow best practice guildlines, or regularly read code online, you might be in the habit of making named return values naked or returning the result of function calls if the return values match your function&rsquo;s return values.</p>
<p>In my case, it was a bit of both that ended up shafting my function, as the original code (see <a href="#references">references</a>) didn&rsquo;t contain the bug.</p>
<p>The issue was two-fold: I had converted a named return value to naked return value and made the assumption that the deferred function would be able to close over the error created during <code>db.Begin()</code>.</p>
<p>To allow the error be closed over as expected, either of the following will work:</p>
<p><strong>Named return value</strong></p>
<pre><code class="language-go">func executeTx(db *sql.DB, actions func(*sql.Tx) error) (err error)
</code></pre>
<p><strong>Capture the error before attempting to return it</strong></p>
<pre><code class="language-go">err = actions(tx)
return err
</code></pre>
<p>If we run the test again with either of these changes in place, we&rsquo;ll see that the rollback is now happening as expected and our data is safe&reg;:</p>
<pre><code class="language-bash">go test -v
=== RUN TestErrorRollback
--- PASS: TestErrorRollback (0.00s)
PASS
ok pocs/db/mysql 0.037s
</code></pre>
<h5 id="a-name-references-a-references"><a name="references"></a> References</h5>
<p><a href="https://stackoverflow.com/a/23502629/304957">Helper function inspiration</a></p>
gotcha iota/gotcha-iota/
Wed, 01 Nov 2017 09:35:57 +0100/gotcha-iota/<p>I ran into a curious little gotcha with Go&rsquo;s <code>iota</code> construct today and I wanted to share it with you.</p>
<p>Without cheating, what would you expect the following code to output?</p>
<pre><code class="language-go">package main
import &quot;fmt&quot;
const (
one = 1 &lt;&lt; iota
two
)
func main() {
fmt.Println(one, two)
}
</code></pre>
<p>If you said <code>1 2</code>, you&rsquo;d be correct! There&rsquo;s nothing fishy going on here.</p>
<p>Now let&rsquo;s suppose we need to add another constant. As it&rsquo;s important and we want it to be visible, we&rsquo;ve added it to the top of our <code>const</code> block:</p>
<pre><code class="language-go">package main
import &quot;fmt&quot;
const (
greeting = &quot;Hello, Enums!&quot;
one = 1 &lt;&lt; iota
two
)
func main() {
fmt.Println(one, two)
}
</code></pre>
<p>What would you expect the program to output now?</p>
<p>If you said <code>1 2</code>, you&rsquo;d now be wrong! The output of this program is now <code>2 4</code>. <code>iota</code> has been incremented, so <code>one</code> is left-shifted by 2 instead of 1, throwing all subsequent enum values out.</p>
<p><strong>Without proper unit tests, this is the kind of subtle bug that could take down a production application, or worse, not take it down.</strong></p>
<p>When you understand the <a href="https://golang.org/ref/spec#Iota">ConstSpec</a>, it becomes clearer why this is the case but as <code>greeting</code> isn&rsquo;t logically part of our enumeration (or even an integer), I would have expected <code>iota</code> to behave differently in this case.</p>
<p>The only correct fix in this example is to move the new constant out of the enumeration <code>const</code> block and into a <em>new</em> <code>const</code> block. This is because each <code>const</code> block resets <code>iota</code>, essentially grouping the constants together, regardless of type and the order in which they appear.</p>
httptest/httptest/
Thu, 19 Oct 2017 09:34:18 +0100/httptest/
<p>Go&rsquo;s <code>httptest</code> package provides a really simple, elegant way to test your HTTP services. It allows you to create requests to and capture the responses from anything that implements the <code>http.Handler</code> interface:</p>
<pre><code class="language-go">type Handler interface {
ServeHTTP(ResponseWriter, *Request)
}
</code></pre>
<p>There are many ways to acheive this and each is tested slightly different. A colleague of mine recently asked for some help testing his HTTP server and I&rsquo;m hoping that this post might help others test theirs, regardless of how they&rsquo;ve implemented it.</p>
<p>I&rsquo;ll provide a super simple example of each server flavour, along with an example of how to test it using the <code>httptest</code> package.</p>
<h4 id="background">Background</h4>
<p>Before we dive in, I&rsquo;ll explain the basics of the <code>httptest</code> package, to highlight why it&rsquo;s so useful.</p>
<p>An HTTP handler in Go takes two arguments, an <code>http.ResponseWriter</code>, which is an interface that describes how to provide a response to the caller and a pointer to an <code>http.Request</code> which contains the request URI, the request body and various other properties. Therefore, in order to make a request (at runtime or in tests), you&rsquo;ll need to provide these.</p>
<p>Happily, the <code>httptest</code> package provides functions that simplify the creation of both. Namely, <code>httptest.NewRequest</code> and <code>httptest.NewRecorder</code>.</p>
<pre><code class="language-go">// Everything required by a ResponseWriter.
resp := httptest.NewRecorder()
</code></pre>
<p>The call to <code>httptest.NewRecorder</code> returns an <code>httptest.ResponseRecorder</code> which implements the <code>http.ResponseWriter</code>. This satisfies the first parameter of <code>ServeHTTP</code>.</p>
<pre><code class="language-go">// Everything required to make a GET request with no body.
req := httptest.NewRequest(http.MethodGet, &quot;/&quot;, nil)
</code></pre>
<p>The call to <code>httptest.NewRequest</code> returns a bare-bones instance of an <code>*http.Request</code> containing an HTTP verb (in this case &ldquo;GET&rdquo;), a target URL (in this case &ldquo;/&rdquo;) which becomes the <code>RequestURI</code> and a body (in this case <code>nil</code>). This satisfies the second parameter of <code>ServeHTTP</code>.</p>
<p>Remembering that everything in Go&rsquo;s world of HTTP boils down to a call to <code>ServeHTTP</code> with two parameters, you now have everything you&rsquo;ll need to test an HTTP server.</p>
<p>To keep things simple and still copy/paste friendly, I&rsquo;ve removed all package names and imports from the following examples. In the 3rd-party handler sections, I&rsquo;m using the Echo server mux, which can be installed as follows:</p>
<pre><code class="language-bash">$ go get -u github.com/labstack/echo/...
</code></pre>
<pre><code class="language-go">import (
&quot;github.com/labstack/echo&quot;
)
</code></pre>
<h4 id="flavours">Flavours</h4>
<ul>
<li><a href="#builtin">Built-in <code>http.HandleFunc</code> handlers</a></li>
<li><a href="#builtin-servemux-direct">Built-in <code>http.ServeMux</code> handlers (direct)</a></li>
<li><a href="#builtin-servemux-routed">Built-in <code>http.ServeMux</code> handlers (routing)</a></li>
<li><a href="#builtin-servemux-embedded">Built-in <code>http.ServeMux</code> handlers (embedded)</a></li>
<li><a href="#builtin-wrapped">Wrapped <code>http.HandleFunc</code> handlers</a></li>
<li><a href="#mux-direct">3rd-party mux handlers (direct)</a></li>
<li><a href="#mux-routing">3rd-party mux handlers (routing)</a></li>
<li><a href="#mux-embedded">3rd-party mux handlers (embedded)</a></li>
</ul>
<h5 id="a-name-builtin-a-built-in-handlers"><a name="builtin"></a>Built-in handlers</h5>
<p>Go&rsquo;s built-in handlers are vanilla HTTP handlers that satisfy the <code>http.Handler</code> interface when a call to <code>http.HandleFunc</code> is used in conjunction with a function (anonymous or otherwise) with the signature <code>func(w http.ResponseWriter, r *http.Request)</code>:</p>
<h5 id="server-code">Server code</h5>
<pre><code class="language-go">func main() {
http.HandleFunc(&quot;/&quot;, hello)
log.Fatal(http.ListenAndServe(&quot;:1234&quot;, nil))
}
func hello(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusTeapot)
w.Write([]byte(&quot;hello&quot;))
}
</code></pre>
<p>This is the quintessential Hello, World! HTTP server example. It exposes a single endpoint that returns the word &ldquo;hello&rdquo; and a status code of 418 (I&rsquo;m a teapot).</p>
<h5 id="test-code">Test code</h5>
<pre><code class="language-go">func TestHello(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, &quot;/&quot;, nil)
resp := httptest.NewRecorder()
hello(resp, req)
if resp.Code != http.StatusTeapot {
t.Fatalf(&quot;exp %d but got %d&quot;, http.StatusTeapot, resp.Code)
}
if resp.Body.String() != &quot;hello&quot; {
t.Fatalf(&quot;exp %s but got %s&quot;, &quot;hello&quot;, resp.Body.String())
}
}
</code></pre>
<p>In this example, we only want to test our <code>hello</code> function, as that contains our business logic. We therefore make a call directly to it and bypass the HTTP routing to &ldquo;/&rdquo; provided by the <code>*http.Server</code> (which is configured in the call to <code>http.ListenAnServe</code>).</p>
<p>By now, the only line that might seem unfamiliar to you is the direct call to <code>hello</code> with our <code>httptest.ResponseRecorder</code> and <code>*http.Request</code> parameters.</p>
<p>As the call to <code>NewRecorder</code> returns a pointer to an <code>http.ResponseRecorder</code>, it&rsquo;ll be mutated inside the call to <code>hello</code>. This is what allows us to interrogate the response after making the request.</p>
<p>The tests pass because we&rsquo;ve configured the <code>hello</code> function to return a status code of &ldquo;418&rdquo; and a body of &ldquo;hello&rdquo;; all of which is captured by the <code>httptest.ResponseRecorder</code>.</p>
<h4 id="a-name-builtin-servemux-direct-a-built-in-http-servermux-direct-handlers"><a name="builtin-servemux-direct"></a>Built-in <code>http.ServerMux</code> direct handlers</h4>
<p>Go&rsquo;s <code>http.ServeMux</code> is a multiplexer that you can configure routes and handlers against. It&rsquo;s a step between calling <code>http.HandleFunc</code>, which abstracts you away from the <code>http.Handler</code> interface and a 3rd-party multiplexer that adds additional layers of abstraction.</p>
<h5 id="server-code-1">Server code</h5>
<pre><code class="language-go">func main() {
mux := http.NewServeMux()
mux.Handle(&quot;/&quot;, &amp;handler{})
log.Fatal(http.ListenAndServe(&quot;:1234&quot;, mux))
}
type handler struct {
}
func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusTeapot)
w.Write([]byte(&quot;hello&quot;))
}
</code></pre>
<p>In this example, we&rsquo;ve created a <code>handler</code> struct which implements the <code>http.Handler</code> interface because of its <code>ServeHTTP</code> method receiver. With this configuration, we could add additional handlers of varying complexity, to handle different routes.</p>
<p>It&rsquo;s worth noting that <code>mux.Handle(&quot;/&quot;, &amp;handler{})</code> is equivalent to <code>http.Handle(&quot;/&quot;, &amp;handler{})</code>, so the test will work against either, without modification.</p>
<h5 id="test-code-1">Test code</h5>
<pre><code class="language-go">func TestHello(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, &quot;/&quot;, nil)
resp := httptest.NewRecorder()
h := &amp;handler{}
h.ServeHTTP(resp, req)
if resp.Code != http.StatusTeapot {
t.Fatalf(&quot;exp %d but got %d&quot;, http.StatusTeapot, resp.Code)
}
if resp.Body.String() != &quot;hello&quot; {
t.Fatalf(&quot;exp %s but got %s&quot;, &quot;hello&quot;, resp.Body.String())
}
}
</code></pre>
<p>This time round we&rsquo;re invoking a method receiver on <code>*handler</code>, rather than calling the top-level function. Everything else remains exactly the same.</p>
<p>Once again, the routing logic to &ldquo;/&rdquo; isn&rsquo;t being tested, we&rsquo;re just testing the function that&rsquo;ll get called <em>after</em> the requset has been routed. If you&rsquo;d like routing to form part of your unit tests, you might want to consider using <code>http.ServeMux</code> as follows.</p>
<h4 id="a-name-builtin-servemux-routed-a-built-in-http-servermux-routed-handlers"><a name="builtin-servemux-routed"></a>Built-in <code>http.ServerMux</code> routed handlers</h4>
<h5 id="server-code-2">Server code</h5>
<pre><code class="language-go">func main() {
mux := newMux()
http.ListenAndServe(&quot;:1234&quot;, mux)
}
func newMux() (m *http.ServeMux) {
m = http.NewServeMux()
m.HandleFunc(&quot;/a&quot;, func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte(&quot;hello&quot;))
})
m.HandleFunc(&quot;/b&quot;, func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte(&quot;goodbye&quot;))
})
return
}
</code></pre>
<p>In this example, we attach all of our handlers to the <code>*http.ServeMux</code> using <code>HandleFunc</code> and not <code>Handle</code>. This hides the <code>ServeHTTP</code> methods for the handlers themselves, allowing the test code to call the <code>ServeHTTP</code> method on the <code>*http.ServeMux</code> struct directly; testing our routing logic.</p>
<h5 id="test-code-2">Test code</h5>
<pre><code class="language-go">func TestHello(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, &quot;/a&quot;, nil)
resp := httptest.NewRecorder()
m := newMux()
m.ServeHTTP(resp, req)
if resp.Body.String() != &quot;hello&quot; {
t.Fatalf(&quot;exp %s but got %s&quot;, &quot;hello&quot;, resp.Body.String())
}
}
</code></pre>
<p>This time, I&rsquo;ve bound two handlers, one for route &ldquo;/a&rdquo; and another for &ldquo;/b&rdquo;. The test will pass, as the handler configured for route &ldquo;/a&rdquo; returns hello but change the request&rsquo;s target URI to &ldquo;/b&rdquo; and you&rsquo;ll receive the following error:</p>
<pre><code class="language-bash">--- FAIL: TestHello (0.00s)
main_test.go:19: exp hello but got goodbye
</code></pre>
<p>Our routing is being tested!</p>
<h4 id="a-name-builtin-servemux-embedded-a-built-in-http-servermux-embedded-handlers"><a name="builtin-servemux-embedded"></a>Built-in <code>http.ServerMux</code> embedded handlers</h4>
<p>In your production code, you&rsquo;re more likely to want to hide the <code>*http.ServeMux</code> along with other dependencies in a struct of your own. It&rsquo;s easy to adapt the <a href="#builtin-servemux-routed">Built-in <code>http.ServeMux</code> handlers (routing)</a> example to hide the <code>*http.ServeMux</code>. There are a couple of ways you could do this but the easiest way is to use embedding:</p>
<h5 id="server-code-3">Server code</h5>
<p>In this example, we declare a <code>mux</code> struct and &ldquo;embed&rdquo; the <code>*http.ServeMux</code>, which automatically exposes the <code>ServeHTTP</code> method on our <code>mux</code> struct and the tests pass without modification.</p>
<pre><code class="language-go">func main() {
mux := newMux()
http.ListenAndServe(&quot;:1234&quot;, mux)
}
type mux struct {
*http.ServeMux
}
func newMux() (m *mux) {
m = &amp;mux{
http.NewServeMux(),
}
m.HandleFunc(&quot;/a&quot;, func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte(&quot;hello&quot;))
})
m.HandleFunc(&quot;/b&quot;, func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte(&quot;goodbye&quot;))
})
return
}
</code></pre>
<h4 id="a-name-builtin-wrapped-a-wrapped-built-in-handlers"><a name="builtin-wrapped"></a>Wrapped built-in handlers</h4>
<p>The <code>http.HandlerFunc</code> type reduces a web request to a simple function call. Functions are first-class citizens in Go, meaning you can call one <code>http.HandlerFunc</code> from another. The preceding <code>http.HandlerFunc</code> is referred to as &ldquo;middleware&rdquo;.</p>
<p>Go&rsquo;s support for closures gives you the added benefit of closing over variables and keeping them within the scope of your final handler. This is useful for passing around loggers and database sessions etc.</p>
<p>In the following example, I create a logging handler and chain it onto my <code>hello</code> handler:</p>
<h5 id="server-code-4">Server code</h5>
<pre><code class="language-go">func main() {
l := log.New(os.Stdout, &quot;&quot;, 0)
http.HandleFunc(&quot;/&quot;, withLogging(hello, l))
http.ListenAndServe(&quot;:1234&quot;, nil)
}
func hello(w http.ResponseWriter, r *http.Request) {
w.Write([]byte(&quot;hello&quot;))
}
func withLogging(h http.HandlerFunc, l *log.Logger) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
l.Println(&quot;before&quot;)
defer l.Println(&quot;after&quot;)
h(w, r)
}
}
</code></pre>
<p>When the endpoint is hit, we&rsquo;ll execute our logging middleware first, then our <code>hello</code> handler.</p>
<h5 id="test-code-3">Test code</h5>
<pre><code class="language-go">func TestHello(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, &quot;/&quot;, nil)
resp := httptest.NewRecorder()
buf := &amp;bytes.Buffer{}
logger := log.New(buf, &quot;&quot;, 0)
middleware := withLogging(hello, logger)
middleware(resp, req)
if resp.Body.String() != &quot;hello&quot; {
t.Fatalf(&quot;exp %s but got %s&quot;, &quot;hello&quot;, resp.Body.String())
}
output := buf.String()
if output != &quot;before\nafter\n&quot; {
t.Fatalf(&quot;exp %s got %s&quot;, &quot;before\nafter\n&quot;, output)
}
}
</code></pre>
<p>In the test, we create a <code>*log.Logger</code> and set its <code>io.Writer</code> to be a <code>*bytes.Buffer</code>, meaning we can interrogate whatever&rsquo;s logged bye our middleware. We wrap the <code>hello</code> handler in the logging middleware and make the request via the logging middleware, just like we would with our <code>hello</code> handler directly.</p>
<p>The end-user sees &ldquo;hello&rdquo;, just as they did before and we&rsquo;re able to call the <code>String()</code> method on the logger&rsquo;s writer to assert that our logging middleware wrote the expected log lines.</p>
<h4 id="a-name-mux-direct-a-3rd-party-server-mux-direct"><a name="mux-direct"></a>3rd-party server mux (direct)</h4>
<p>Every server mux is different and will require a different approach to testing. Under the covers however, everything is still just <code>ServeHTTP</code>, so the tests will look familiar.</p>
<p>I&rsquo;m a big fan of the <a href="https://echo.labstack.com">Echo</a> framework. Its API made sense to me from the get go and it has become my go-to server mux for web services. In the case of Echo, handlers are similar to those expected by the stdlib&rsquo;s <code>http.HandleFunc</code> function except for the fact that they wrap the <code>*http.Request</code> and <code>http.ResponseWriter</code> into an <code>echo.Context</code>. This behaviour is shared by most 3rd party multiplexers.</p>
<h5 id="server-code-5">Server code</h5>
<pre><code class="language-go">func main() {
e := echo.New()
e.GET(&quot;/&quot;, hello)
e.Start(&quot;:1234&quot;)
}
func hello(c echo.Context) (err error) {
return c.String(http.StatusTeapot, &quot;hello&quot;)
}
</code></pre>
<h5 id="test-code-4">Test code</h5>
<pre><code class="language-go">func TestHello(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, &quot;/&quot;, nil)
resp := httptest.NewRecorder()
e := echo.New()
c := e.NewContext(req, resp)
if err := hello(c); err != nil {
t.Fatalf(&quot;exp not error not %v&quot;, err)
}
if resp.Code != http.StatusTeapot {
t.Fatalf(&quot;exp %d but got %d&quot;, http.StatusTeapot, resp.Code)
}
if resp.Body.String() != &quot;hello&quot; {
t.Fatalf(&quot;exp %s but got %s&quot;, &quot;hello&quot;, resp.Body.String())
}
}
</code></pre>
<p>In this example, the only deviation from standard handler testing is the creation of the wrapping context. Contexts in the Echo framework are pooled in a <code>sync.Pool</code> and created with the very same <code>NewContext</code> method that we&rsquo;re using in the test, so it&rsquo;s a great way to test Echo&rsquo;s runtime context creation beaviour too.</p>
<p>Like the majority of tests in this post, this test bypasses routing and shows how to test just the handler we care about. The following example tests Echo&rsquo;s routing behaviour.</p>
<h4 id="a-name-mux-routing-a-3rd-party-server-mux-routing"><a name="mux-routing"></a>3rd-party server mux (routing)</h4>
<p>In order to test routing with the Echo mux, we&rsquo;ll need to be able to call <code>ServeHTTP</code> on <code>*echo.Echo</code> in our test. To do this, I&rsquo;ve moved its creation into a method called <code>newMux</code>, which performs all of the route configuration and provides something for us to call <code>ServeHTTP</code> on.</p>
<h5 id="server-code-6">Server code</h5>
<pre><code class="language-go">func main() {
e := newMux()
e.Start(&quot;:1234&quot;)
}
func newMux() (e *echo.Echo) {
e = echo.New()
e.GET(&quot;/a&quot;, hello)
e.GET(&quot;/b&quot;, goodbye)
return
}
func hello(c echo.Context) (err error) {
return c.String(http.StatusTeapot, &quot;hello&quot;)
}
func goodbye(c echo.Context) (err error) {
return c.String(http.StatusTeapot, &quot;goodbye&quot;)
}
</code></pre>
<h5 id="test-code-5">Test code</h5>
<pre><code class="language-go">func TestHello(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, &quot;/a&quot;, nil)
resp := httptest.NewRecorder()
mux := newMux()
mux.ServeHTTP(resp, req)
if resp.Code != http.StatusTeapot {
t.Fatalf(&quot;exp %d but got %d&quot;, http.StatusTeapot, resp.Code)
}
if resp.Body.String() != &quot;hello&quot; {
t.Fatalf(&quot;exp %s but got %s&quot;, &quot;hello&quot;, resp.Body.String())
}
}
</code></pre>
<h4 id="a-name-mux-embedded-a-3rd-party-server-mux-embedded"><a name="mux-embedded"></a>3rd-party server mux (embedded)</h4>
<h5 id="server-code-7">Server code</h5>
<p>In your production code, you&rsquo;re more likely to want to hide the 3rd-party server mux along with other dependencies in a struct of your own.</p>
<p>Testing an embedded Echo mux is just as easy as testing a naked one, you just have to get at its <code>ServeHTTP</code> method. I achieve this by exposing it via a method receiver on my <code>server</code> struct:</p>
<pre><code class="language-go">func main() {
s := newServer()
s.start(&quot;:1234&quot;)
}
type server struct {
router *echo.Echo
}
func newServer() (s *server) {
s = &amp;server{}
s.router = echo.New()
s.router.GET(&quot;/a&quot;, s.hello)
s.router.GET(&quot;/b&quot;, s.goodbye)
return
}
func (s *server) start(addr string) (err error) {
return s.router.Start(addr)
}
func (s *server) hello(c echo.Context) (err error) {
return c.String(http.StatusTeapot, &quot;hello&quot;)
}
func (s *server) goodbye(c echo.Context) (err error) {
return c.String(http.StatusTeapot, &quot;goodbye&quot;)
}
func (s *server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
s.router.ServeHTTP(w, r)
}
</code></pre>
<p>It&rsquo;s worth noting that I could have called my <code>ServeHTTP</code> method anything, as it&rsquo;s not essential for the tests. However, it does allows us to use <code>server</code> directly in a <code>http.Handler</code> and semantically, it&rsquo;s clear to others what&rsquo;s happening so there are definitely benefits.</p>
<h6 id="test-code-6">Test code</h6>
<pre><code class="language-go">func TestHello(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, &quot;/a&quot;, nil)
resp := httptest.NewRecorder()
s := newServer()
s.ServeHTTP(resp, req)
if resp.Code != http.StatusTeapot {
t.Fatalf(&quot;exp %d but got %d&quot;, http.StatusTeapot, resp.Code)
}
if resp.Body.String() != &quot;hello&quot; {
t.Fatalf(&quot;exp %s but got %s&quot;, &quot;hello&quot;, resp.Body.String())
}
}
</code></pre>
<p>As with the <a href="#builtin-servemux-routed">Built-in <code>http.ServeMux</code> routed handlers</a> tests, I&rsquo;ve created two endpoints, one that returns &ldquo;hello&rdquo; and one that returns &ldquo;goodbye&rdquo; to highlight that we&rsquo;re testing the routing logic as well. Make a change to target &ldquo;/b&rdquo; instead of &ldquo;/a&rdquo; and you&rsquo;ll get an error.</p>
struct options/struct-options/
Fri, 22 Sep 2017 09:33:43 +0100/struct-options/<p>It&rsquo;s important to be explicit about the dependencies in your application. If your <code>Server</code> struct requires access to a database, it makes sense to force consumers to provide it with the means to connect to that database during creation. Peter Bourgon&rsquo;s brilliant <a href="https://peter.bourgon.org/go-best-practices-2016/#dependency-management">Go best practices, six years in</a> post makes a brilliant case for other mandatory explicit dependencies and I urge you to read it.</p>
<p>What about optional dependencies? When starting a new application, I&rsquo;m always faced with the decision of how best to manage optional dependencies. There are loads of possible ways but I&rsquo;ve settled on the pattern adopted by many open-source projects including the <a href="https://nats.io">NATS messaging system</a>.</p>
<p>Here&rsquo;s an example from the NATS client package, which separates mandatory dependencies (URL in this case) from optional ones:</p>
<pre><code class="language-go">func Connect(url string, options ...Option) (*Conn, error) {
</code></pre>
<p>This allows the package consumer to spin up a basic NATS client with zero knowledge of the <code>Connect</code> function other than that it takes a URL, which is pretty self-explanatory:</p>
<pre><code class="language-go">nats.Connect(&quot;nats://localhost:4222&quot;)
</code></pre>
<p>Need to provide root certificates for a TLS client? No problem, just use one of the variadic <code>Option</code> parameters:</p>
<pre><code class="language-go">nats.Connect(&quot;tls://localhost:4443&quot;, nats.RootCAs(&quot;./certs/ca.pem&quot;))
</code></pre>
<p>The <code>Option</code> function makes all of this possible and is defined as follows:</p>
<pre><code class="language-go">type Option func(*Options) error
</code></pre>
<p>It&rsquo;s simply a function that takes a pointer to the NATS client&rsquo;s <code>Options</code> struct and returns an error if anything went wrong. The beauty of this approach? The consumer never actually deals directly with the <code>Options</code> struct, they just declaratively build up the constructor with their own overrides for default properties.</p>
<p>Let&rsquo;s have a look at the <code>Connect</code> function in its entirety to see how these properties are getting applied:</p>
<pre><code class="language-go">func Connect(url string, options ...Option) (*Conn, error) {
opts := GetDefaultOptions()
opts.Servers = processUrlString(url)
for _, opt := range options {
if err := opt(&amp;opts); err != nil {
return nil, err
}
}
return opts.Connect()
}
</code></pre>
<p>The first thing you might notice is how clean the function is; there&rsquo;s nothing spooky going on:</p>
<ol>
<li>Create an <code>opts</code> variable to hold our default configuration. If the consumer hasn&rsquo;t provided any <code>Option</code> parameters, we&rsquo;ll have a perfectly sensible NATS default client.</li>
<li>NATS clients can learn about new servers via INFO messages from the server but can also connect to any number of servers at start-up. This line just breaks up a comma-separated list of hosts into individual servers.</li>
<li>For each of the options the consumer provides, apply them over the default values.</li>
<li>Use the <code>Options</code> struct to connect to the server and return the connection.</li>
</ol>
<p>The <code>GetDefaultOptions</code> function is similarly non-spooky and does exactly what you&rsquo;d expect it to do:</p>
<pre><code class="language-go">func GetDefaultOptions() Options {
return Options{
AllowReconnect: true,
MaxReconnect: DefaultMaxReconnect,
ReconnectWait: DefaultReconnectWait,
Timeout: DefaultTimeout,
PingInterval: DefaultPingInterval,
MaxPingsOut: DefaultMaxPingOut,
SubChanLen: DefaultMaxChanLen,
ReconnectBufSize: DefaultReconnectBufSize,
}
}
</code></pre>
<p>The eagle-eyed among you may notice the omission of anything TLS related from the <code>Option</code> example I gave above&hellip; I&rsquo;d guess that this is because the default options returned by this function are important to the running of the server. The server can run without TLS configuration, so its default is null (and hence omitted) but it&rsquo;d struggle to run without a sensible timeout configured etc.</p>
<p>Here&rsquo;s a very contrived and easily copy/pasted example I&rsquo;ve thrown together to allow for some tinkering with the idea (and a <a href="https://play.golang.org/p/aUOjl_PFx_">playground</a> link):</p>
<pre><code class="language-go">package main
import (
&quot;fmt&quot;
&quot;log&quot;
)
func main() {
s, err := newServer(1234, certs(&quot;certs&quot;), logs(&quot;logs&quot;))
if err != nil {
log.Fatal(err)
}
fmt.Println(s)
}
type server struct {
certs string
logs string
port int
}
type option func(s *server) error
func certs(value string) option {
return func(o *server) error {
o.certs = value
return nil
}
}
func logs(value string) option {
return func(o *server) error {
o.logs = value
return nil
}
}
func newServer(port int, options ...option) (*server, error) {
s := server{
port: port,
certs: &quot;/etc/certs&quot;,
}
for _, opt := range options {
if err := opt(&amp;s); err != nil {
return nil, err
}
}
return &amp;s, nil
}
</code></pre>
stupid channel tricks (p2) semaphores/stupid-channel-tricks-p2-semaphores/
Fri, 01 Sep 2017 09:33:00 +0100/stupid-channel-tricks-p2-semaphores/<p>Semaphores are like mutexes that allow N threads to access a shared resource instead of just 1. Go&rsquo;s buffered channels make creating semaphore-like behaviour a doddle. To create a semaphore that allows 10 threads to concurrently access a shared resource, its simply <code>make(chan struct{}, 10)</code> etc. The only thing our threads need to lock/own is an item of the buffered channel, so it makes sense to use an empty struct, as it uses zero memory.</p>
<p>To keep the interface to our semaphore clean, we&rsquo;ll start by hiding the channel in a struct with a sensible name:</p>
<pre><code class="language-go">type semaphore struct {
channel chan struct{}
}
</code></pre>
<p>At creation time, we&rsquo;ll want to initialise the semaphore with an underlying channel size, so let&rsquo;s create a <code>new</code> method next:</p>
<pre><code class="language-go">func newSemaphore(concurrency int) (s *semaphore) {
return &amp;semaphore{
channel: make(chan struct{}, concurrency),
}
}
</code></pre>
<p>Next, we&rsquo;ll want to ask the semaphore to execute something for us. The worker doesn&rsquo;t need to hold onto a lock object, so this can be abstracted away too:</p>
<pre><code class="language-go">func (s *semaphore) execute(f func()) {
// Attempt to send an empty struct to the underlying channel. This will
// block if the channel is full (i.e. there are N concurrent operations
// already underway).
s.channel &lt;- struct{}{}
// By now, we've obtained a 'lock' and can begin execution. Execute
// function f and ensure that the empty struct is removed from the
// channel when we're finished to allow other workers to execute.
go func() {
defer func() {
&lt;-s.channel
}()
f()
}()
}
</code></pre>
<p>In the following example, we perform 10 operations using a semaphore with 5 slots. You&rsquo;ll notice from the output that the first 5 operations are completed together and the next 5 operations are completed after:</p>
<pre><code class="language-go">func main() {
s := newSemaphore(5)
for i := 0; i &lt; 10; i++ {
j := i
s.execute(func() {
fmt.Println(j, time.Now())
time.Sleep(time.Second)
})
}
fmt.Scanln()
}
</code></pre>
<pre><code class="language-bash">$ go run main.go
4 2017-09-01 07:59:04.6135725 +0100 BST &lt;- first 5
0 2017-09-01 07:59:04.6135725 +0100 BST
2 2017-09-01 07:59:04.6135725 +0100 BST
3 2017-09-01 07:59:04.6135725 +0100 BST
1 2017-09-01 07:59:04.6135725 +0100 BST
5 2017-09-01 07:59:05.6147369 +0100 BST &lt;- next 5
6 2017-09-01 07:59:05.6157369 +0100 BST
7 2017-09-01 07:59:05.6157369 +0100 BST
8 2017-09-01 07:59:05.6157369 +0100 BST
9 2017-09-01 07:59:05.6167397 +0100 BST
</code></pre>
stupid channel tricks (p1) blocking/stupid-channel-tricks-p1-blocking/
Wed, 30 Aug 2017 09:29:41 +0100/stupid-channel-tricks-p1-blocking/
<p>Channels are Go&rsquo;s implementation of Tony Hoare&rsquo;s <a href="https://en.wikipedia.org/wiki/Communicating_sequential_processes">CSP</a> concurrency model. Rather than reiterate the <a href="https://tour.golang.org/concurrency/2">basics</a>, I&rsquo;ll dive straight into some silly channel tricks that I&rsquo;ve found to be useful. Note that these examples are designed to be easy-to-follow, rather than easy-to-copy-straight-into-production.</p>
<h4 id="prevent-sender-from-blocking">Prevent sender from blocking</h4>
<p>Sending to a full channel blocks by design. This prevents fast senders from saturating a channel (and your available memory) but it&rsquo;ll penalise them by forcing them to block, instead of penalising slow readers for not keeping up.</p>
<p>A good analogy for this would be a stock ticker, where only the latest stock prices can be considered relevant. We don&rsquo;t want to prevent the delivery of the latest prices just to allow a slow reader to keep up with the oldest because that&rsquo;s silly. Our bank wants to be too fast to fail.</p>
<h5 id="penalising-fast-senders">Penalising fast senders</h5>
<p>In this example, the price publisher in our stock ticker analogy blocks to allow a slow reader to chug through every price at its own pace:</p>
<pre><code class="language-go">c := make(chan int, 10)
// Fast sender
go func() {
for i := 0; i &lt; 20; i++ {
c &lt;- i
}
}()
// Slow reader
go func() {
for o := range c {
time.Sleep(time.Millisecond * 500)
fmt.Println(&quot;SUB&quot;, o)
}
}()
</code></pre>
<p>The above code outputs the following and completes in 10s:</p>
<pre><code class="language-bash">$ go run main.go
SUB 0
SUB 1
SUB 2
SUB 3
SUB 4
SUB 5
SUB 6
SUB 7
SUB 8
SUB 9
SUB 10
SUB 11
SUB 12
SUB 13
SUB 14
SUB 15
SUB 16
SUB 17
SUB 18
SUB 19
</code></pre>
<h5 id="penalising-slow-readers">Penalising slow readers</h5>
<p>The following ensures the sender never blocks in the case of a slow reader. With the use of a <code>select</code> statement, we can decide what happens if the channel we&rsquo;re sending to is full. In this case, our sender reads off and throws away an item from the channel to make space for a newer item. This means we only keep the most recent items, essentially penalising the slow reader.</p>
<p>In this example, the price publisher in our stock ticker analogy publishes all of its prices onto the channel, removing the oldest if they haven&rsquo;t been read to allow the latest to be published:</p>
<pre><code class="language-go">c := make(chan int, 10)
// Fast sender
go func() {
for i := 0; i &lt; 20; i++ {
select {
case c &lt;- i:
// If we hit the default case, the channel was full. Remove an
// item, then resend. The select statement serialises access to
// the channels in our case statements, so we're protected from
// race conditions.
default:
&lt;-c
c &lt;- i
}
}
}()
// Slow subscriber
go func() {
for o := range c {
time.Sleep(time.Millisecond * 500)
fmt.Println(&quot;SUB&quot;, o)
}
}()
</code></pre>
<p>The above code outputs the following and completes in 5s:</p>
<pre><code class="language-bash">$ go run main.go
SUB #0
SUB #10
SUB #11
SUB #12
SUB #13
SUB #14
SUB #15
SUB #16
SUB #17
SUB #18
SUB #19
</code></pre>
json time.Duration/json-time-duration/
Sat, 05 Aug 2017 09:28:53 +0100/json-time-duration/<p>From time to time, you&rsquo;ll want to serialise/deserialise a struct whose default serialisation seems counter-intuitive. Take for example, the JSON serlisation of <code>time.Duration</code>:</p>
<pre><code class="language-go">json.NewEncoder(os.Stdout).Encode(time.Hour)
// result: 3600000000000
</code></pre>
<p>I don&rsquo;t event think <em>science</em> knows what this number is.</p>
<p>Asking people to configure 1 hour as &ldquo;3600000000000&rdquo; is not only cruel, it&rsquo;s <em>asking</em> for trouble; miss a zero and you&rsquo;ve had it.</p>
<p>A much friendlier and more natural alternative is to allow for the confguration of <code>time.Duration</code> as you&rsquo;re <em>used</em> to seeing it appear (<code>1h</code>, <code>1m30s</code> etc). To acheive this, you&rsquo;ll need to perform some custom JSON marshaling and unmarshaling.</p>
<p>Here&rsquo;s a copy-paste-friendly example:</p>
<p><strong>Step 1</strong> - create a struct to house the <code>time.Duration</code> and receive the custom marshal operations.</p>
<pre><code class="language-go">type ConfigDuration struct {
time.Duration
}
func (d *ConfigDuration) UnmarshalJSON(b []byte) (err error) {
d.Duration, err = time.ParseDuration(strings.Trim(string(b), `&quot;`))
return
}
func (d ConfigDuration) MarshalJSON() (b []byte, err error) {
return []byte(fmt.Sprintf(`&quot;%s&quot;`, d.String())), nil
}
</code></pre>
<p><strong>Step 2</strong> - create a struct that consumes the new <code>ConfigDuration</code> struct:</p>
<pre><code class="language-go">type ServerConfig struct {
ReadTimeout ConfigDuration `json:&quot;readTimeout&quot;`
}
</code></pre>
<p><strong>Step 3</strong> - Unmarshal some JSON and access the <code>time.Duration</code>:</p>
<pre><code class="language-go">config := ServerConfig{}
json.Unmarshal([]byte(`{&quot;readTimeout&quot;:&quot;10s&quot;}`), &amp;config)
fmt.Println(config.ReadTimeout.Duration)
// result: 10s
</code></pre>
cv/cv/
Fri, 04 Aug 2017 09:27:58 +0100/cv/
<div align="right">
<div>v18.8.17</div>
<div><b>GitHub:</b> <a href="https://github.com/codingconcepts">codingconcepts</a></div>
<div><b>Twitter:</b> <a href="https://twitter.com/go_robreid">@go_robreid</a></div>
</div>
<h4 id="profile">Profile</h4>
<div style="text-align: justify">
I started developing software professionally in 2006 and in my time as a developer, I’ve written .NET, Go and Java applications, front-ends, APIs, application frameworks, web and Windows services and messaging solutions. I have designed and developed solutions for the police, the travel, sports betting, telecoms and retail industries and financial institutions.
<br/><br/>
</div>
<h4 id="education">Education</h4>
<div style="text-align: justify">
Kingston University (2005 - 2009) - First-Class with Honours and awarded "Best Software Engineer" by the BCS.<br/><br/>
A-Grade modules include Foundations of Algorithms, Internet Security, Software Quality, Internet Protocols and Services, Systems Analysis and Design, Business Organisation and Practice, Network Software and a double-A for my dissertation.
<br/><br/>
</div>
<h4 id="technical-skills">Technical Skills</h4>
<p><html>
<style>
td.left {
text-align: right;
font-weight: bold;
width: 150px;
}
</style>
<table style="width:100%">
<col width="130">
<col align="right">
<col align="left">
<tr>
<td class="left">Languages (primary)</td>
<td><div>Go and C#<div></td>
</tr>
<tr>
<td class="left">Languages (other)</td>
<td><div>Java, VB.NET, Ruby (and Crystal)<div></td>
</tr>
<tr>
<td class="left">Databases</td>
<td>CockroachDB (and Postgres), SQL Server, MySQL, Oracle, Sybase, MongoDB, Couchbase, Redis and InfluxDB</td>
</tr>
<tr>
<td class="left">Messaging</td>
<td>RabbitMQ, NATS and ZeroMQ</td>
</tr>
<tr>
<td class="left">Cloud</td>
<td>GCP, Algolia, Digital Ocean, AWS and Azure</td>
</tr>
<tr>
<td class="left">CI/CD</td>
<td>Docker, Kubernetes, GitLab, Jenkins, BuildMaster and ActiveBatch</td>
</tr>
<tr>
<td class="left">Practices</td>
<td> Agile, Kanban and a firm believer in the benefits of pair-programming and shift-left.</td>
</tr>
</table>
</html></p>
<h4 id="employment-history">Employment History</h4>
<h5 id="lush-november-2017-present">Lush (November 2017 - present)</h5>
<div style="text-align: justify">
At Lush, I'm a backend Charter Lead. This has me heading up a team of backend engineers, helping to identify and decide the technology roadmap, evangelising agile, leading our interdisciplinary squad standups, and mentoring other developers, testers and agile coaches.
<br/><br/>
Right from the get-go, I've been writing critical Go microservices and tools, including but not limited to:
<ul>
<li>Payment Gateway, a service to handle all UK eCommerce and in-store transactions.</li>
<li>Product Master, a service that acts as a golden source of product information.</li>
<li>Stock Management Layer, a global real-time view of stock availability.</li>
<li><a href="https://github.com/LUSHDigital/modelgen">modelgen</a>, a tool which generates boilerplate Go structs from MySQL and CockroachDB tables.</li>
<li><a href="https://github.com/zhutik/adyen-api-go">adyen-api-go</a>, wrapper for the Adyen payment system, which I've been contributing to as part of my development effort for the Payment Gateway.</li>
</ul>
An enjoyable part of my role is filling our London office with engineers and involves the on-boarding of both recruiters and candidates. As part of the on-boarding process for engineers, I've written and conduct our initial Skype interviews and technical tests and also participate in final interviews.
<br/><br/>
I also run a Code Club for staff in our Soho office. I've dusted off my Python skills and am teaching this to staff on Wednesdays afternoons. It's a great way to interact with staff I might not otherwise get the chance to interact with and gives them an appreciation of what we engineers do, without boring them!
<br /><br />
The experience I gained at that NewVoiceMedia, in a mature agile department and team, has put me at the centre of the adoption, proliferation, and honing of agile at Lush.
<br/><br/>
</div>
<h5 id="newvoicemedia-june-2016-november-2017">NewVoiceMedia (June 2016 - November 2017)</h5>
<div style="text-align: justify">
At NewVoiceMedia, I worked in the IVR (Interactive Voice Response) team. A small interdisciplinary team, who pair-programme, dev/test pair, and are held up to be the model agile team in the DevOps department.
<br/><br/>
The software created by the IVR team handles many thousands of calls at any one time, allowing customers to administer highly dynamic call flows. We were responsible for everything between and including the interfaces customers used to administer their call plans to the runtime logic routing calls through the system and various other interconnected systems.
<br/><br/>
In addition to my work in the IVR team, I propagated the use of the Go programming language across the department. I did that by way of lunch-and-learn sessions, one-on-one mentoring and providing projects kick-offs and code reviews.
<br/><br/>
My proudest achievement at NewVoiceMedia, was the spear-heading of a department-wide adoption of Go. To show the company that Go was the right tool for the job, I lead my team in the development of a mission-critical text-to-speech service. A number of mission-critical production systems have since followed suit.
<br/><br/>
</div>
<h5 id="smartodds-august-2014-june-2016">Smartodds (August 2014 - June 2016)</h5>
<div style="text-align: justify">
At Smartodds, I worked in an agile team responsible for bet placement and market data/stats retrieval for quant models. This role allowed me to function as a frontend and backend developer interchangeably and I became a go-to for the middle tier because of my work with RabbitMQ; building the company’s central message bus, which is used across the business and extensively by clients outside the business.
<br/><br/>
I was known for my love of developing proof-of-concepts and my ability to build them quickly. My versatility, sense of teamwork, readiness to help and mentor colleagues was also frequently called upon. In addition to my day-to-day duties, I could be relied upon to scout out technologies which would benefit the company. In my time at Smartodds, I oversaw the adoption of RabbitMQ, Dapper and LINQPad.
<br/><br/>
One of my favourite non-technical responsibilities at Smartodds was the developer interview process. I liaised with recruiters, vetted candidates, produced interview content and conducted phone and face-to-face interviews.
<br/><br/>
</div>
<h5 id="glencore-november-2011-august-2014">Glencore (November 2011 - August 2014)</h5>
<div style="text-align: justify">
At Glencore, I functioned as Subject Matter Expert for UX and UI development and a number of widely used internal core components, which I developed and maintained and whose use I promoted within the company. I was also a go-to for all things C#, WPF, WCF, LINQ, UX, UI, multithreading, security and build-pipeline related, helping developers of all levels write code and design interfaces.
<br/><br/>
</div>
<h5 id="barclays-capital-april-2010-november-2011">Barclays Capital (April 2010 - November 2011)</h5>
<div style="text-align: justify">
During my time at BarCap, I helped design, develop and build automated tests for the WPF-based base and precious metals trading platform. I build great relationships with the traders, sales and operations teams and held regular work groups for them.
<br/><br/>
My proudest achievement at BarCap was the design and development of a real-time WPF position monitoring interface, used heavily by traders, sales and operations.
<br/><br/>
</div>
<h5 id="navitas-solutions-june-2009-april-2010">Navitas Solutions (June 2009 - April 2010)</h5>
<div style="text-align: justify">
In my role at Navitas, I used C#, WPF, WCF, MySQL and the Boo programming language to develop bespoke solutions that interfaced with all major Global Distribution Systems (GDS) for clients in the travel industry.
<br/><br/>
My proudest achievement at Navitas was the design and development of a robust application framework, which acts as a transparent interface between Navitas and all of the GDSs. It’s used as the foundation for all of their products and is used by many thousands of travel agents today.
<br/><br/>
</div>
<h5 id="reveal-media-april-2006-june-2009">Reveal Media (April 2006 - June 2009)</h5>
<div style="text-align: justify">
My development career began at Reveal Medial during my university “industrial placement” year. During my time there, I led the design and development of a solution which is used by the majority of police constabularies around the UK. My development efforts represent a major part of an application, which handles evidential video footage from portable surveillance equipment. As part of these developments, I would frequently provide rapid proof-of-concepts, run exhibition booths and demonstrate my software to constabularies around the country.
<br/><br/>
I was asked to continue working for Reveal Media part-time during my final year at university and then full-time upon the completion of my degree.
<br/><br/>
</div>
<h5 id="other-employment-history">Other Employment History</h5>
<ul>
<li>Customer Service Leader, HMV (2004 – 2006)</li>
<li>IT Assistant, Electronic Arts Ltd, 2 weeks (work experience) 2002</li>
</ul>
<h5 id="personal-interests">Personal Interests</h5>
<div style="text-align: justify">
When I'm not developing professionally, I following the Go and Crystal development communities online, at meetups and at conferences and I make contributions to the Open Source community on GitHub and via my blog.
<br/><br/>
With any time that’s left to me, I enjoy writing and performing music. To date, I have produced 2 studio EPs and have performed around the south of England. I play bass guitar, piano, guitar and dip in and out of learning the ukulele, violin and tin whistle. I also love ballet, opera and classical music and attend performances whenever I can. In my heart however, I'm still firmly a metalhead \m/.
<br/><br/>
</div>
let's encrypt docker containers/lets-encrypt-docker-containers/
Sun, 16 Jul 2017 09:27:09 +0100/lets-encrypt-docker-containers/
<p>Let&rsquo;s Encrypt is rocking the SSL boat and the water&rsquo;s warm.</p>
<p>If you need that swanky EV banner, you&rsquo;re happy to pay $€£¥‎ for the privilege and you want to manually renew/replace expired certificates, this post is probably not for you.</p>
<h5 id="tl-dr">tl;dr</h5>
<p>Running services in containers (<a href="https://www.docker.com">Docker</a> in particular) is becoming more and more popular, as is securing stuff for free with <a href="https://letsencrypt.org">Let&rsquo;s Encrypt</a>. This post shows you how to combine the two.</p>
<h5 id="what-you-ll-need">What you&rsquo;ll need</h5>
<ul>
<li>A domain name (<a href="http://namecheap.com">namecheap</a> provide very affordable domain names)</li>
<li>A server that can run in your domain (I like to use <a href="http://digitalocean.com">Digital Ocean</a> droplets and One-Click Apps)</li>
</ul>
<h5 id="code">Code</h5>
<p>The are a number of ways to spin up an HTTPS service in Go and while I could have used the built-in <code>http.Server</code> and gained loads of configurability, I&rsquo;ll keep thing simple and use the one provided by the <a href="https://echo.labstack.com">Echo</a> server mux.</p>
<p>The code below sets up a single-endpoint API that returns &ldquo;Hello, TLS!&rdquo; to the caller. I&rsquo;ll go through each non-obvious lines in more detail:</p>
<pre><code class="language-go">package main
import (
&quot;fmt&quot;
&quot;log&quot;
&quot;net/http&quot;
&quot;github.com/labstack/echo&quot;
&quot;golang.org/x/crypto/acme/autocert&quot;
)
func main() {
r := echo.New()
r.GET(&quot;/&quot;, func(c echo.Context) (err error) {
c.String(http.StatusOK, &quot;Hello, TLS!&quot;)
return
})
r.AutoTLSManager.HostPolicy = autocert.HostWhitelist(&quot;example.com&quot;)
r.AutoTLSManager.Cache = autocert.DirCache(&quot;/certs&quot;)
r.AutoTLSManager.Prompt = autocert.AcceptTOS
if err := r.StartAutoTLS(fmt.Sprintf(&quot;:443&quot;)); err != nil {
log.Fatal(err)
}
}
</code></pre>
<p>Echo&rsquo;s <code>AutoTLSManager</code> simply wraps the <code>http.Server</code>, so if you&rsquo;re familiar with the <code>http.Server</code> TLS options, the code will feel familiar to you. If not, <a href="https://blog.cloudflare.com/exposing-go-on-the-internet/">here&rsquo;s</a> a great post about configuring an SSL <code>http.Server</code> to help you understand what&rsquo;s going on.</p>
<p>First, we let Let&rsquo;s Encrypt know what domain we&rsquo;re running in. When the first request arrives, <code>autocert</code> will talk to Let&rsquo;s Encrypt, which will issue your server with some SSL-related challenges designed to prove domain ownership.</p>
<pre><code class="language-go">router.AutoTLSManager.HostPolicy = autocert.HostWhitelist(&quot;example.com&quot;)
</code></pre>
<p>Let&rsquo;s Encrypt rate-limit requests in their production environment to 10 certificate issues per domain per week. This means you&rsquo;ll want to cache your certificates, to prevent asking for new ones every time your server starts up:</p>
<pre><code class="language-go">router.AutoTLSManager.Cache = autocert.DirCache(&quot;/certs&quot;)
</code></pre>
<p>In order to use Let&rsquo;s Encrypt, you&rsquo;ll need to accept their <a href="https://community.letsencrypt.org/tos">Terms Of Service</a>. This can be done as part of the <code>autocert</code> negotiation and can be enabled as follows:</p>
<pre><code class="language-go">router.AutoTLSManager.Prompt = autocert.AcceptTOS
</code></pre>
<h5 id="dockerfile">Dockerfile</h5>
<p>I really like small containers, they&rsquo;re quick to build, quick to push from your build machine and quick to pull onto your production machines.</p>
<p><a href="https://alpinelinux.org">Alpine</a> is a <em>really</em> small Linux distribution that adds just few megabytes to the size of your container. As the Alpine base image is so small, it doesn&rsquo;t contain the ca-certificates necessary to secure your server via Let&rsquo;s Encrypt by default, so we&rsquo;ll need to harness our Dockerfile to make sure these are in place.</p>
<p>The following Dockerfile builds on top of the <code>alpine</code> base image, installs the missing ca-certificates and mounts a volume called &ldquo;/certs&rdquo;, which we&rsquo;ll map to a directory on the host machine to allow for caching between container restarts:</p>
<pre><code class="language-Dockerfile">FROM alpine
RUN apk update &amp;&amp; apk add ca-certificates &amp;&amp; rm -rf /var/cache/apk/*
COPY . /
CMD [&quot;./hello&quot;]
EXPOSE 443
VOLUME [&quot;/certs&quot;]
</code></pre>
<h5 id="running">Running</h5>
<p>Once pulled, the Docker container can be started with the following arguments. The important one for TLS is the volume mapping <strong>-v /certs:/certs</strong>. It creates and maps a &ldquo;/certs&rdquo; directory on the host machine to a directory that will be created by <code>autocert</code> on the container, allowing certificates to be persisted between container restarts:</p>
<pre><code class="language-bash">$ docker run -d -p 443:443 -v /certs:/certs hello
</code></pre>
<h5 id="resources">Resources</h5>
<p><a href="https://github.com/octoblu/docker-alpine-ca-certificates/blob/master/Dockerfile">Alpine ca-certificates</a></p>
<p><a href="http://blog.cloud66.com/x509-error-when-using-https-inside-a-docker-container/">Ubuntu 14.04.1 ca-certificates</a></p>
expiring bolt db items/expiring-bolt-db-items/
Sun, 09 Jul 2017 09:26:22 +0100/expiring-bolt-db-items/
<p>I like a good index. I especially like the handy <a href="https://docs.mongodb.com/manual/core/index-ttl/">MongoDB TTL indexes</a> you can apply to date columns to take care of old data automatically.</p>
<p>My latest personal project manages secrets and is backed by <a href="https://github.com/boltdb/bolt">BoltDB</a>, a charming embedded key/value store, written in Go.</p>
<p>Being as simple as it is, BoltDB doesn&rsquo;t have the concept of item expiry. I spent a while Googling for elegant solutions but ended up (as we all often do), rolling my own. For this post, I&rsquo;m using the same <code>Store</code> semantics as the <a href="https://github.com/boltdb/bolt">BoltDB documentation</a> for consistency.</p>
<h5 id="tl-dr">tl;dr</h5>
<p>The trick is to create two buckets for everything you want to store, one for the items themselves and another to keep track of when they were created.</p>
<h5 id="background">Background</h5>
<p>To give a bit of context to the rest of the post, here&rsquo;s how I&rsquo;m modelling the data. The following variables are used to identify the buckets:</p>
<pre><code class="language-go">var (
bucketSecret = []byte(&quot;secret&quot;)
bucketTTL = []byte(&quot;ttl&quot;)
)
</code></pre>
<p><strong>secret bucket</strong></p>
<p>The main bucket my application depends on is the <code>secret</code> bucket. A <code>secret</code> is simply a key/value pair comprising a cryptographically secure key that point to some [soon-to-be] encrypted data. A secret value will be posted to the server and the key will be returned, giving access to the value. This concept is nothing special and certainly nothing new.</p>
<p><code>secret</code> items are represented as follows:</p>
<pre><code class="language-go">type Secret struct {
Key string `json:&quot;key&quot;`
Value string `json:&quot;value&quot;`
}
</code></pre>
<p><strong>ttl bucket</strong></p>
<p>The TTL bucket acts as a sort of audit for secrets. When a secret is posted to the database, a TTL record that contains the key and the secret&rsquo;s creation time is also written.</p>
<p>Storing the creation time in a separate key/value map allows us to scan the TTL bucket (<em>whose keys are stored in order and so can be scanned</em>) in a separate read-only transaction, minimising impact on the <code>secret</code> bucket.</p>
<h5 id="writing-data">Writing data</h5>
<p>In this snippet, we&rsquo;re storing the secret <em>and</em> the TTL records in the same transaction. The key for the <code>secret</code> bucket becomes the <em>value</em> for the <code>TTL</code> bucket. The <em>key</em> for the <code>TTL</code> bucket is a timestamp:</p>
<pre><code class="language-go">func (s *Store) Post(secret Secret) (err error) {
key := []byte(secret.Key)
return s.db.Update(func(tx *bolt.Tx) (err error) {
bSecret := tx.Bucket(bucketSecret)
if err = bSecret.Put(key, []byte(secret.Value)); err != nil {
return
}
bTTL := tx.Bucket(bucketTTL)
return bTTL.Put([]byte(time.Now().UTC().Format(time.RFC3339Nano)), key)
})
}
</code></pre>
<h5 id="care-taking">Care-taking</h5>
<p>As BoltDB doesn&rsquo;t have a process for pruning old data, I&rsquo;m simply kicking off a goroutine to manage that for me.</p>
<p>The end-to-end &ldquo;sweep&rdquo; process boils down to this:</p>
<ol>
<li><p>Find all <code>TTL</code> items whose key (a timestamp) is before a given point in time (e.g. 10 minutes old). This requires a read-only view that doesn&rsquo;t lock the <code>TTL</code> bucket. If we <em>just</em> miss a record during a sweep, we&rsquo;ll pick it up on the next one.</p></li>
<li><p>Store the <em>values</em> of each <code>TTL</code> item found in-memory. These are the <em>keys</em> of items in the <code>secret</code> bucket.</p></li>
<li><p>Perform a batch delete of all <code>secret</code> items using the in-memory keys picked up during the previous step. This requires a read-write transaction but as it&rsquo;s a batch operation, we&rsquo;re not locking the <code>secret</code> bucket for very long.</p></li>
</ol>
<pre><code class="language-go">func (s *Store) Sweep(maxAge time.Duration) (err error) {
keys, err := s.GetExpired(maxAge)
if err != nil || len(keys) == 0 {
return
}
return s.db.Update(func(tx *bolt.Tx) (err error) {
bSecret := tx.Bucket(bucketSecret)
for _, key := range keys {
if err = bSecret.Delete(key); err != nil {
return
}
}
return
})
}
</code></pre>
<pre><code class="language-go">func (s *Store) GetExpired(maxAge time.Duration) (keys [][]byte, err error) {
keys = [][]byte{}
ttlKeys := [][]byte{}
err = s.db.View(func(tx *bolt.Tx) error {
c := tx.Bucket(bucketTTL).Cursor()
max := []byte(time.Now().UTC().Add(-maxAge).Format(time.RFC3339Nano))
for k, v := c.First(); k != nil &amp;&amp; bytes.Compare(k, max) &lt;= 0; k, v = c.Next() {
keys = append(keys, v)
ttlKeys = append(ttlKeys, k)
}
return nil
})
err = s.db.Update(func(tx *bolt.Tx) error {
b := tx.Bucket(bucketTTL)
for _, key := range ttlKeys {
if err = b.Delete(key); err != nil {
return err
}
}
return nil
})
return
}
</code></pre>
<p>This solution is by no means perfect. In the <code>GetExpired</code> function, I delete keys to prevent them showing up in subsequent sweeps but this should be wrapped in a transaction so that if the <code>secret</code> deletion fails for any reason, we try to delete them again.</p>
struct tags and environment variables/struct-tags-and-environment-variables/
Mon, 19 Jun 2017 09:25:29 +0100/struct-tags-and-environment-variables/
<p>Struct tags in Go provide metadata to fields and are used heavily by the encoders in the stdlib&rsquo;s <code>encoding</code> package. Here&rsquo;s a typical use case for a struct tag:</p>
<pre><code class="language-go">import &quot;encoding/json&quot;
type ServerConfig struct {
Port int `json:&quot;port&quot;`
APIKey string `json:&quot;apiKey&quot;`
}
func main() {
bytes, _ := json.Marshal(ServerConfig{
Port: 1234,
APIKey: &quot;something secret&quot;,
})
fmt.Println(string(bytes))
}
</code></pre>
<p>The struct tags in this example give Go&rsquo;s JSON encoder an explicit name to use when marshalling/unmarshalling <code>ServerConfig</code> structs. This is simple and declarative, you&rsquo;re providing this information in-line with the struct&rsquo;s field itself, so there&rsquo;s only one source of truth. Here&rsquo;s the output of the above example:</p>
<pre><code class="language-bash">{&quot;port&quot;:1234,&quot;apiKey&quot;:&quot;something secret&quot;}
</code></pre>
<h5 id="background">Background</h5>
<p>My team recently deployed a <a href="https://12factor.net/">12-factor</a> application into Production and as per factor III of the methodology, we&rsquo;re storing our config in &ldquo;the environment&rdquo; (environment variables).</p>
<p>In vanilla Go, this means writing one of the following:</p>
<pre><code class="language-go">config := os.Getenv(&quot;ENV_KEY&quot;)
if config == &quot;&quot; {
// fail if required
}
// parse and use environment variable
</code></pre>
<p>or</p>
<pre><code class="language-go">if config, ok := ok.Lookupenv(&quot;ENV_KEY&quot;); !ok {
// fail if required
} else {
// parse and use environment variable
}
</code></pre>
<p>Of the two, my preference is the latter, because <strong>A</strong> an empty string might be valid configuration for a given struct string field, <strong>B</strong> the &ldquo;ok idiom&rdquo; is used elsewhere by the stdlib (think map access), so looks and feels natural/familiar and <strong>C</strong> the resulting code is a more syntactically obvious alternative to checking for an empty string.</p>
<p>My issue with this approach is that even when the logic is refactored out into a helper function, the responsibility of finding the value, parsing it, checking for errors and making decisions as to whether the configuration is required or not is that of the application (and the developer). Not to mention the code distance between declaring the fields on the struct and setting them from environment variables elsewhere.</p>
<p>This all feels very imperative, so I decided to write a library to take the pain out of factor III.</p>
<h5 id="enter-env-https-github-com-codingconcepts-env">Enter <a href="https://github.com/codingconcepts/env">env</a></h5>
<p>Env is a very lightweight library that allows you to express your environmental configuration declaratively. It takes care of the boring stuff, which keeps your code slick and easy to reason with.</p>
<p>Simply pass <code>env.Set</code> a pointer to the struct you wish to populate and it&rsquo;ll populate every field with an <code>env</code> tag from environment variables. For example, an <code>env</code> tag value of <code>MYAPPLICATION_PORT</code> for an integer field <code>Port</code> will find an environment variable called <code>MYAPPLICATION_PORT</code> and populate the <code>Port</code> field with the integer representation of the environment variable.</p>
<p>If your application can&rsquo;t run without a field being set, you can specify a <code>required</code> tag to ensure that an error is returned if the environment variable is not found. Another error will be returned if the environment variable found cannot be parsed to the field&rsquo;s type or if the field itself is not settable.</p>
<h5 id="field-types">Field types</h5>
<p>All of the fields you&rsquo;ll likely want to set from configuration are supported by <code>env</code> and for each type, <code>slice</code> support is also provided:</p>
<ul>
<li>string</li>
<li>bool</li>
<li>int (int8, int16, int32, int64)</li>
<li>uint (uint8, uint16, uint32, uint64)</li>
<li>time.Duration</li>
</ul>
<h4 id="slice-types">Slice types</h4>
<p>For <code>slice</code> fields, <code>env</code> will attempt to parse an environment variable as a <strong>comma-delimited</strong> collection of the data type you&rsquo;ve specified; trimming as necessary.</p>
<h5 id="example">Example</h5>
<pre><code class="language-go">package main
import (
&quot;fmt&quot;
&quot;log&quot;
&quot;time&quot;
&quot;github.com/codingconcepts/env&quot;
)
type awsConfig struct {
Secret string `env:&quot;SECRET&quot; required:&quot;true&quot;`
Region string `env:&quot;REGION&quot;`
Port int `env:&quot;PORT&quot; required:&quot;true&quot;`
Peers []string `env:&quot;PEERS&quot;`
ConnectionTimeout time.Duration `env:&quot;TIMEOUT&quot;`
}
func main() {
config := awsConfig{}
if err := env.Set(&amp;config); err != nil {
log.Fatal(err)
}
fmt.Println(config)
}
</code></pre>
<pre><code class="language-bash">$ ID=1 SECRET=shh PORT=1234 PEERS=localhost:1235,localhost:1236 TIMEOUT=5s go run main.go
</code></pre>
<p>Feel free to get in touch with issues/suggestions/pull requests!</p>
black box testing/black-box-testing/
Sat, 22 Apr 2017 09:24:42 +0100/black-box-testing/
<p>Go&rsquo;s tooling continues to delight and this one&rsquo;s a real hidden gem&hellip; I&rsquo;ve just discovered, with the help of <a href="https://www.exago.io">exago.io</a>, the <code>testing/quick</code> package in Go&rsquo;s standard library. It&rsquo;s a very simple blackbox testing package, which repeatedly executes a given block of your code with <strong>values you wouldn&rsquo;t think to try yourself</strong>.</p>
<p>Within minutes of discovering it, I&rsquo;m already starting to think differently about how I write exported functions and here&rsquo;s why.</p>
<h5 id="the-scenario">The scenario</h5>
<p>Consider the following innocuous little function. It simply returns a random number between a minimum and maximum value.</p>
<pre><code class="language-go">func Between(min int, max int) int {
return rand.Int()%(max-min) + min
}
</code></pre>
<p>The hardcore among you might even think to add a litle bit of manual testing to make sure it&rsquo;s working correctly:</p>
<pre><code class="language-go">func init() {
rand.Seed(time.Now().UnixNano())
}
func main() {
for i := 0; i &lt; 10; i++ {
fmt.Println(Between(0, 10))
}
}
</code></pre>
<pre><code class="language-bash">$ go run main.go
3
8
1
3
4
8
7
4
0
5
</code></pre>
<p>Looks great! You commit and deploy to production. Users <em>love</em> having the ability to generate a weak random number between a minimum and maximum value. Who wouldn&rsquo;t!?</p>
<p>Then I checked out my exago.io <a href="https://www.exago.io/project/github.com/codingconcepts/albert">score</a> and ran through the checklist (their gamification slant on improving code really makes sense to me):</p>
<p><code>Blackbox Tests: In addition to standard tests, does the project have blackbox tests?</code></p>
<p>After a bit of digging, I found that they were referring to Go&rsquo;s black box testing tool <code>testing/quick</code>. So I had a play&hellip; Read the following code carefully.</p>
<pre><code class="language-go">func TestBlackBoxCheckBetween(t *testing.T) {
f := func(min int, max int) bool {
result := Between(min, max)
return result &gt;= min &amp;&amp; result &lt;= max
}
if err := quick.Check(f, nil); err != nil {
t.Error(err)
}
}
</code></pre>
<p>If the basic gist of <code>TestBlackBoxCheckBetween</code> makes sense to you, your assumptions about how the function works and, more importantly, your assumptions about how <em>users will treat the function</em> are pretty much the same as mine.</p>
<p><code>testing/quick</code> detects your function&rsquo;s parameter types and repeated invokes it with generated values (a minimum and maximum in the case of <code>Between</code>). After the invocation, you perform an assertion and return <code>true</code> or <code>false</code>; <code>true</code> if the output of your function contains the expected result and <code>false</code> if it doesn&rsquo;t.</p>
<p>I gave it a run:</p>
<pre><code class="language-bash">$ go test
--- FAIL: TestBlackBoxCheckBetween (0.00s)
main_test.go:15: #1: failed on input 4106209714314777601, -2352281900722994752
FAIL
exit status 1
</code></pre>
<p>The output of this test has just made me acutely aware of how fragile my <code>Between</code> method was (not to mention every other function I&rsquo;ve <em>ever</em> written). With very little effort, <code>testing/quick</code> has just surfaced the following assumptions in my code:</p>
<ul>
<li>Users will know what values to pass in</li>
<li>I know what value users will pass in</li>
<li>Minimum will be reasonable</li>
<li>Maximum will be reasonable</li>
<li>Minimum will be less than maximum</li>
</ul>
<p>Horrified at my ignorance, I added a little check in the <code>Between</code> method and ran the test again:</p>
<pre><code class="language-go">func Between(min int, max int) int {
if max &lt; min {
min, max = max, min
}
return rand.Int()%(max-min) + min
}
</code></pre>
<p>Running the test again at this point will result in the same failure but for a different reason. Your <em>production</em> code is now better able to cope with weird input but your <em>test</em> code needs a small tweak.</p>
<p>Our assumption that result would be between the minimum and maximum values is now wrong because it&rsquo;s possible that the minimum value could be greater than the maximum value.</p>
<pre><code class="language-go">func TestBlackBoxCheckBetween(t *testing.T) {
f := func(min int, max int) bool {
result := Between(min, max)
if min &gt; max {
min, max = max, min
}
return result &gt;= min &amp;&amp; result &lt;= max
}
if err := quick.Check(f, nil); err != nil {
t.Error(err)
}
}
</code></pre>
<pre><code class="language-bash">$ go test -v
=== RUN TestBlackBoxCheckBetween
--- PASS: TestBlackBoxCheckBetween (0.00s)
</code></pre>
<p>I&rsquo;m now going to create black box tests for everything I&rsquo;ve ever written, ever.</p>