There's no way to shut down idle HTTP/1.1 or HTTP2 connections. There is a an open issue at https://golang.org/issue/9478, but no immediate plans for it. Note that once you close your listener, your server will not accept new connections; you're reusing the same connection.

If you want to forcefully close all connections, you would need to do the bookkeeping yourself using the http.ServerConnState to track the TCP connections that are active.

You could also have your handlers set Connection: close on each response, but that would make your http server much less efficient if clients ever need to make more than 1 request.

Email codedump link for Golang Keeps File Serve Socket Open After Close