Debugging containerized Go applications

Containers are increasingly popular for deploying applications and Go applications make no exceptions from this. But how should we debug these applications that are running in the isolated environment a container offers?

For this, we turn back to the way we build these containers and include a copy of Delve, the Go debugger, and we adjust how we create/launch our application.
Let’s consider a basic web application:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

packagemain

import(

"log"

"net/http"

)

funcmain(){

log.Println("starting server...")

http.HandleFunc("/",func(whttp.ResponseWriter,r *http.Request){

w.Write([]byte(`Hello world`))

})

log.Fatal(http.ListenAndServe(":8080",nil))

}

We’ll place the code above in a package under GOPATH, for example $GOPATH/src/hello/hello.go

One important note is that your project should be in a valid Go Workspace, also known as GOPATH. You can read more about Go Workspaces in the official documentation.

We have a simple, multi-stage, Docker container which contains the building stage and the final container that results from this.

1

2

3

4

5

6

7

8

9

10

11

12

# Compile stage

FROM golang:1.10.1-alpine3.7ASbuild-env

ENV CGO_ENABLED0

ADD./go/src/hello

RUN go build-o/server hello

# Final stage

FROM alpine:3.7

EXPOSE8080

WORKDIR/

COPY--from=build-env/server/

CMD["/server"]

Place this in the root of your package, in our example under $GOPATH/src/hello/Dockerfile.

To run this, we’ll have to create a new Run Configuration, and expose our application’s port, 8080.

To test this, we’ll run the container and then launch a simple request from the IDE. To run requests from the IDE, you can create a file with the “.http” extension and then type the request as below:

1

2

3

# Run a test request

GET/

Host:localhost:8080

Let’s transform this container into one that can be debugged. Our goal is to keep as much parity as possible with the original container so that the functionality is not affected by our debugging version.

We must first compile our application with all optimizations turned off, to allow Delve to extract as much information as possible from the binary. The second part involves actually running delve into the container.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

# Compile stage

FROM golang:1.10.1-alpine3.7ASbuild-env

ENV CGO_ENABLED0

ADD./go/src/hello

# The -gcflags "all=-N -l" flag helps us get a better debug experience

Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, --security-opt="apparmor=unconfined" --cap-add=SYS_PTRACE.

GoLand will automatically map the source code location for us, so even our source code is on an operating system, and we are compiling/using the binary on a different operating system, everything will work without having to do anything special for this case.

After running this container configuration, now the last step is to connect to the debugging container and see the debugger working. Run our container and then go to “Run | Edit Configurations…” and add a new Go Remote configuration, fill in the remote host IP and port number and then run it like any other configuration.

Rerunning a request will stop the debugger as expected and you can see all the values as if you would debug a local running process.

And that’s it. Using GoLand’s debugging facilities, you can debug not only applications that run on your machine but also applications that run on remote computers or even in containers. And if you are using Kubernetes, you can also install a plugin to help you edit Kubernetes resources.

If you have not done so already, download GoLand today and send us feedback in the comments below or using our issue tracker on how we can further improve your development experience!

Please make sure you follow the steps described after the Dockerfile here:

> Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, –security-opt=”apparmor=unconfined” –cap-add=SYS_PTRACE.

The error comes most likely from the missing -security-opt / -cap-add arguments to the Docker daemon. If this is still an issue, please let us know.

Please make sure you follow the steps described after the Dockerfile here:

> Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, –security-opt=”apparmor=unconfined” –cap-add=SYS_PTRACE.

The error comes most likely from the missing -security-opt / -cap-add arguments to the Docker daemon. If this is still an issue, please let us know.

So this works when you are running your application, so am stuck at debugging tests.

So I can use the usual feature of just running test from the idea and everything works including the breakpoints. (it has been really helpful by the way)

Now the problem is at running tests that are not supported by my os for instance of of the feature am testing is “plugin” but thats not supported in osx at the moment.
How do I run this test with delve and use breakpoints in my idea

Unfortunately, debugging when using the plugins feature of Go is currently not supported by Go, and it looks like 1.11 won’t change anything for that (at least as far as I got with reading the changelog).

> How do I run this test with delve and use breakpoints in my idea

If you are using plugins, you cannot, it’s a limitation of Go, as mentioned above. If not, tests should work with the debugging feature out of the box. If they don’t please, let me know what’s going on and how to reproduce this and I’ll be happy to help you out.

I had some issues though. When I copied the options from the blog I didn’t realized at the first time that double dash and double quotes characters are replaced by another characters. So afterwards I typed options myself in docker configuration correctly and I instantly got the warning: “Can’t parse command line options: Unrecognized option: –security-opt=apparmor=unconfined”. When I clicked “Apply” the warning pop-up appeared. I closed it and clicked “OK” in “Run/Debug Configurations”, and I got the same warning window again. So there was no other way to close it but “Cancel”. When I opened it again I noticed that the configuration was saved.

When I ran Docker file I got the following message in deploy log:
“Failed to deploy ‘hello-server Dockerfile: Dockerfile’: org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: –security-opt=apparmor=unconfined”

I removed this option from the configuration and IDE was able to build an image and run it. Actually I was also able to debug the application without security-opt

Please make sure you are using the latest version of the IDE as well as the latest version of the Docker plugin. If you are, can you please share the Docker plugin version with me? As far as I’m aware, this should not work without the “security-opt” option, so I’m not sure why it’s not accepted by the Docker plugin or why it worked without it.

I used GoLand 2017.3.4 (build 173.4674.50) and Docker plugin 173.3727.15. I updated GoLand to version 2018.1.5 and there is no warning message about –security-opt
I supposed that the warning might have been connected to the error from docker build command with –security-opt: “The daemon on this platform does not support setting security options on build” (I’m working on Ubuntu 16.04 with Docker version 18.03.1-ce, build 9ee9f40).
I did another attempt with the newest GoLand and no security-opt. It worked differently. Without saying much detail I was able to debug only once, so with –security-opt it works much better.

I spent a lot of time on this topic today. The reason is that actually it didn’t work for me with the newest GoLand 2018.1.5 and the newest docker plugin 181.5087.20. I tried various combinations with and without –security-opt and it behaved differently each attempt. Even the same configuration didn’t work for me the same each time. In some cases I couldn’t stop docker container and reset seemed the only option.

I decided to try it out again on another computer that never had go, docker, and GoLand. System is different (Linux Mint 19) and docker version is a bit higher (Docker version 18.06.0-ce, build 0ffa825). I had the same problems and unpredictable results. The latest status from this computer is that it doesn’t matter if the security-opt is set or not. Debugging worked, although in some previous attempt I experienced the same issues including impossibility to stop container.

Unfortunately I was not able to achieve the same on the first computer. Configuration that works for me on the second one doesn’t work on the first one. I think it’s worth mentioning that before each attempt I deleted all the docker containers and images in order to have the same starting point.

There is a difference in behaviour on both computers, In the second one I have to send a request to the application to start debugging. On the first one it starts debugging instantly when I run “Remote Debug”. And I don’t know why. So I can debug once but the second request is processed without stopping at the breakpoint.

If you have any suggestions what could be wrong on the first computer I would appreciate any tip.

My apologies for the detail in replying to you.
Without having an error message, or the IDE logs it’s very hard to understand what the problem might be.
Please open an issue on the tracker, https://youtrack.jetbrains.com/issues/Go, and we’ll do our best to help you out.

Normally, adding --security-opt=”apparmor=unconfined” --cap-add=SYS_PTRACE should be enough to get it going. I’m using Windows as host OS as well, and I never hard to add those options to containers. Is this some pre-release Docker version? As for user root, that’s the default user in most base container images.

Is it possible to set --security-opt="apparmor=unconfined" --cap-add=SYS_PTRACE when using Docker Swarm? We’ve got a set of interdependent services that make heavy use of the Swarm-only Secrets feature.

It doesn’t look like it’s possible at the moment, if I’m reading this issue https://github.com/moby/moby/issues/25209 correctly. I’ll try and see if there’s any other way to solve this. Unfortunately without those options, due to the restrictions in Docker/the way containers are set up, debugging them won’t work.

I am use docker to deploy my app.
But my develop host is Windows, I need some operation but Linux have different file system structure to Windows. So my codes cannot directly run on Windows.
leave alone my first commit, I have solve it by Goland’s before build(Restart delve Container again)
But I cannot get app output. delve put output to stdout, I cannot get logs which my app produce.
Can we get app’s logs from container and put it into Goland Console?
I try mount log file to my workspace, and then use goland configuration LogsLog files to be shown in console, But console not show anything.
Sorry for my bad English

I think I understand now. Yes, if you need to depend on CGO, then you need to remove that line, ENV CGO_ENABLED 0 . And next, if you need gcc , then you can change this line RUN apk add --no-cache git to RUN apk add --no-cache git build-base and move it before the go build line.

According to the delve authors, you can ignore the message Could not create config directory: user: Current not implemented on linux/amd64. .

Do you also have a message right after it saying API server listening at: [::]:40000 (replace 40000 with your port of choice for the debugger)?

I tried the first Dockerfile and I could replicate the message but when I created and launched the Go Remote debug configuration, everything worked as expected. If this still is an issue for you, can you please let us know? Thank you.

It looks like everything works ok. Have you tried to also run the Go Remote configuration? This is needed as Delve will block the execution until it receives the start command from a Delve client (in this case the IDE).

My Goland IDE successfully connects to the delve server in the container, but It send my local source file path to delve when setting breakpoints, and then, of course, the delve server reports that it can’t find the location because the binary is compiled inside container with a different path.

I use Go module, the project is not under GOPATH, and the version of Goland IDE I used is 2018.3.2

Currently the workaround I used is to make a directory in the container which has the same path as my project directory in local and compile the binary under that directory.

My question is that is there any other way to do custom path remapping?

Another question is that is there any option can let me customize the behavior of IDE’s debug button which shown on the left side of my golang test function, so that I can pass the arguments to my container to run delve with target test function?

> Currently the workaround I used is to make a directory in the container which has the same path as my project directory in local and compile the binary under that directory.

You don’t need the full path to make the automatic mapping work, you just need the directory of the project to be named the same both on the host machine and in the container. There is currently a single known issue with this feature, you can read more about it and the possible workarounds for it on our issue tracker: https://youtrack.jetbrains.com/issue/GO-6736 Please see if this covers your case, and if not, please let us know how to replicate it.

> My question is that is there any other way to do custom path remapping?

No, currently we don’t support any custom path remapping as the auto-mapping feature generally works (with the exception of the above issue). What would be the use-case for such a feature?

> Another question is that is there any option can let me customize the behavior of IDE’s debug button which shown on the left side of my golang test function, so that I can pass the arguments to my container to run delve with target test function?

This will be addressed when we address https://youtrack.jetbrains.com/issue/GO-3322 which is dedicated to such use-cases. Please feel free to add more comments there as to where/how you’d use the Docker support in your workflow.