Go Modules

A discussion about how to convert multiple Go libraries in the same repository to Go modules

A discussion about how to utilize Go Modules in microservices

prerequisites

Go Modules

Go Modules is an experimental opt-in feature in Go 1.11 with the plan of finalizing feature for Go 1.13. The definition of a Go module from this proposal is “a group of packages that share a common prefix, the module path, and are versioned together as a single unit”. It is designed for resolving dependency hell problems in Go, like conflicting dependencies and diamond dependency.

An Example

As shown in the picture, the my-repo repository has two modules bar and mixi. Take the bar module as an example, it contains two packages: the bar package and the foo package. The go.mod file under the path/to/my-repo/bar directory defines the module’s path and its dependencies:

The go.mod file bundles the bar package and the foo package together as a unit. For example, the import statement in the following code will import the module path/to/my-repo/bar (which includes the foo package) rather than the package path/to/my-repo/bar/foo when Go Modules is enabled. Even though the code looks the same, the path in the import statement is recognized as the module path, not the package path, once Go Modules is used

import "path/to/my-repo/bar/foo"
func main () {
foo.DoSomething()
}

How to Enable Go Modules

In order to use Go Modules, you need to upgrade your Go to v1.11 or any later version and set the environment variable export GO111MODULE=on.

When to Use Go Modules

The major purpose of Go Modules is to let one or more packages be versioned, released and retrieved together as a single unit. Therefore, the public packages, for example, Go libraries and SDKs, are major targets of Go Modules as they need to be published properly for public use. You do not need to convert internal packages or any internal-used-only packages within a microservice repository to Go modules. These packages can directly import and use modules once Go Modules feature is enabled, even if they are not converted to modules.

Semantic Import Versioning

Semantic Import Versioning is a method proposed for adopting Semantic Versioning in Go packages and modules. The idea behind it is embedding the major version (say v2) in the package path (for packages) or the module path (for modules) with the following rules:

v1 must be omitted from the module path. This post explains the reason. You may need to follow this rule in your packages if you are thinking of converting your packages to modules one day.

The Major versions higher than v1 must be embedded in the package path or the module path so that Semantic Versioning can be applied to Go packages and modules.

The following picture demonstrates the rules above:

Releasing

With Go Modules and Semantic Import Versioning, you can release your modules by creating git tags. A tag corresponds to a version. For example, the following git command releases the bar module v2.3.3:

git tag bar/v2.3.3 && git push -q origin master bar/v2.3.3

You can read my last blog for more details about how to releases modules with Semantic Import Versioning.

All in all, Go Modules provides a way to group one or more packages as a single retrievable unit, while Semantic Import Versioning is a method for applying Semantic Versioning in Go packages and modules to make them versioned. These two things are designed for breaking a repository into multiple retrievable units (modules), so that Go can grab dependencies at the module granularity rather than the repository granularity.

Utilizing Go Modules

General Guide of Converting Go Packages to Go Modules

I wrote a dummy package called module for demonstrating how to convert one or more Go packages to a Go module.

Converting

It is very easy to convert one or more Go packages to a Go module. Take the module package as an example, here are the steps of converting it to a Go module:

Cd to the root directory of the module package: cd path/to/module

Convert the package to a module: go mod init github.com/azhuox/blogs/golang/go_modules/example/module

(Optional) you can run go mod vendor to reset the module’s vendor directory to include all the packages and modules which are required for building and testing all of the module’s packages. This is the way to provide dependencies for the older versions of Go that do not fully understand Go modules. Any version of Go >= v1.11 does not need this.

Here are the contents of the go.mod file automatically generated by Go. You can see that it defines the module’s path, glues anything under the path/to/example/module directory as a single unit and lists all of its dependencies.

1. It grabs the latest version for the packages that have been converted to modules. For example, rsc.io/quote v1.5.2.

2. It grabs the latest commit for the packages that have not been converted to modules with the format v0.0.0-{date}-{first_12_characters_of_commit_id}. For example, golang.org/x/net v0.0.0-20190328230028-74de082e2cca.

Releasing

A module can only be used as a module after it is released. A module is released by creating git tags and each tag corresponds to a version. However, there are two problems we need to solve before releasing a module.

The first problem is how to release v2 or higher Major versions. Go utilizes two methods, Major Branch and Major Subdirectory, which are provided by this proposal to solve this problem. My last blog demonstrates these two methods and compare their advantages and disadvantages. In this blog, Major Subdirectory is used for all the examples as it does not require to duplicate any code.

The second problem is we need to figure out whether to consider the conversion from Go package(s) to a Go module a breaking change or not. If so, we need to upgrade the Major version using Semantic Versioning. If not, we need to decide what versions we need to release. I prefer to just release the latest version of the package(s) listed in the CHANGELOG.md file for the following reasons:

The conversion from Go package(s) to a Go module is not a breaking change as the package(s) can still work with older versions of Go even if the package(s) are converted to a module. So it does not make sense to upgrade the Major version for this kind of change.

The conversion from Go package(s) to a Go module does not add any new feature or fix any bug. So upgrading the Minor or Patch version in this case does not make sense either.

Now let us come back to the module example and release its latest version. Here is what I did:

Appended v2 to the end of the module path (module github.com/azhuox/blogs/golang/go_modules/example/module/v2) as the latest version of the module package is v2.0.1.

Add a note under the v2.0.1 release note in the CHANGELOG.md file to indicate that the package is converted to a module in and after this version.

Consuming A Module

You can still use this package, without Go Modules enabled, by using some Go dependency management tool (e.g. dep) with the following specification. This will grab the whole repository which includes the module module for your build.

[[constraint]]
name = "github.com/azhuox/blogs"
branch = "master"

With Go Modules, what you need to do is import and use the module in your Go program and run go build. It will automatically grab the golang/go_modules/example/module/v2.0.1 module other than the whole repository for your build.

Converting Go Libraries to Go Modules

The section above already demonstrates how to convert one or more Go packages to a Go module. This section majorly talks about how to convert all the Go packages (libraries) within the same repository to Go modules.

I wrote three packages libalibb and libc under the github.com/azhuox/blogs/golang/go_modules/example/libs/ directory for the demo purpose. Among these three packages, the libb package depends on the liba package while the libc package depends on the libb and libc package.

A principle that we need to follow in this case is firstly convert the packages that have no dependency on other packages within the same repository, and then convert the packages which dependencies have been converted Go modules. This indicates that we need to convert the liba package first, then the libb package and then the libc package in this case.

The cause of this ambiguous import problem is Go grabs the whole repository github.com/azhuox/blogs v0.0.0-20190330175117-09a7dbd4a3ce to get the liba and libb package for satisfying the dependencies of the libc module. However, github.com/azhuox/blogs v0.0.0-20190330175117-09a7dbd4a3ce also includes a copy of the libc package, which confuses the Go compiler. To fix this, we need to convert the liba and libb package to Go modules and release them, so that they can be retrieved and parsed properly as two individual modules by Go.

I want to mention that the internal/pkgb package is using libc package that we just converted to a Go module above. In this case, libc is retrieved together with liba and libb from the github.com/azhuox/blogs repository when Go Modules is not enabled. But it is retrieved individually as a single unit when Go Modules is enabled.

From the project layout, you can also see that the microservice is built as a docker image with the following Dockerfile:

Converting Public Packages to Go Modules

As mentioned in the When to Use Go Modules section, only public packages need to be converted to modules. In this case, the sdks/go package is the only package that gets publicly used. Therefore, we only need to convert this package to a module and releases its latest version:

Utilizing Go Modules in the Microservice

Go Modules in this case refers to the new Go package management tool called vgo which is integrated in go tools like go get and go mod. The following steps demonstrate how to use it to manage the dependencies for the microservice:

Launch a terminal and then enable Go Modules in the terminal: export GO111MODULE=on.

Cd the root directory of the microservice.

Add a go.mod file to the root directory of the microservice: go mod init github.com/azhuox/blogs/golang/go_modules/example/micro-service.

Run or test the microservice to ensure that everything works fine: go run ./server/main.go. This will generate a file called go.sum if everything goes well.

Remove the files for the old dependency management tool, which is Gopkg.toml and Gopkg.lock in this case.

Commit the changes.

Now we successfully replace the old dependency management tool with Go Modules. However, there are two cases we need to deal with in the Continuous Integration (CI) process: with vendor or without vendor.

CI Without Vendor

Without vendors means utilizing Go Modules to dynamically grab dependencies when building docker images during the CI process. In order to do this, we need to do the following steps:

Add an environment variable ENV GO111MODULE=on in the Dockerfile to enable Go Modules.

Remove the vendor directory since we don’t need it anymore.

Commit the changes.

CI With Vendor

With vendor means we want to dump all the dependencies into the vendor directory and let the CI build the docker image based on the vendor directory. The following steps demonstrate how to do it:

Dump all the dependencies into the vendor directory: go mod vendor.

Commit the changes.

If Go Modules is enabled in the CI tool, add the -mod=vendor in the go build step in the Dockerfile: go build -mod=vendor -o /usr/bin/micro-service github.com/azhuox/blogs/golang/go_modules/example/micro-service/server && rm -rf $GOPATH/*.

Update A Dependency in the vendor Directory

Suppose we want to build docker images with the vendor directory and use the latest version of libc (say v1.5.0) in the microservice. The following steps demonstrates the update process:

Get the version: go get github.com/azhuox/blogs/golang/go_modules/example/libs/libc@v1.5.0.

Update the vendor directory: go mod vendor.

This may not work when the microservice is not using any new feature released after the current version of libc (v1.0.0 in this case). To force update it, we need to add a replace statement in the go.mod file and then run go mod vendor: