What did you do?

Attempted to build some code that had dependencies on one or more packages that come from our internal GitHub repo as well as from the public github.com using the experimental vgo tool. As this refers to an internal repo (and that is the basis of the problem), there is no way for me to give you code that reproduces the problem that you can actually try. Suffice to say, though, that even the simplest import causes the issue.

What did you expect to see?

I was hoping that GitHub Enterprise would work as well as normal GitHub.

This comment has been minimized.

This comment has been minimized.

Yes, I just realized that as well. This is still workable when using the normal go build, though, because we can (and do) manually checkout our internal repos into our $GOPATH/src tree. That doesn't seem to be an option with vgo, unless I've missed something.

This comment has been minimized.

I don't know much about GitHub Enterprise. Does it have similar APIs to public GitHub? It is possible this could work better with vgo than old go get. If the tag said something like githubenterprise to tell us it was that kind of server, we could use those APIs.

This comment has been minimized.

FWIW, i have been collecting a group of company GHE customers to try to influence GitHub to actually fix this problem. (it exists in GitHub itself, as well, but it's masked by having static rules that cover it)

This comment has been minimized.

edited

Hi @sdboyer
I work for a GHE customer, and raised the issue a couple of months ago with them.
This is what they told me:

"[Thaddeus Cortez] and his team will work with Engineering to establish contact with Google's Go team to help them provide a solution on their end.
We've discussed various ways of fixing this on GitHub's side, but none of the potential solutions are ideal. The meta tag approach comes with the problem of information leakage which we consider a serious security risk. It's a trade-off we are not willing to make. And, a toggleable option in the web UI would require us to turn our static error pages into dynamic ones which would come with several different issues, most notably performance and UX implications as well as added complexity in terms of code maintenance.

We really think, the Go team needs to integrate support for other sources like GitHub Enterprise (or one of our competitor's products), and not just GitHub.com."

This comment has been minimized.

@amnonbc & @sdboyer,
I work for a GHE customer as well, and this was their response to us:

The "go-import" meta tag is used for installing Go applications from a URL, and is indeed supported on GitHub Enterprise, there is a caveat however:

To avoid leaking information about repositories when the appliance is in private mode, GitHub Enterprise doesn't return the meta tag unless you're authenticated. The ideal solution is for Go to be modified to allow authentication options, but for now, you can work around this by adding .git to your repository paths. For example:import "[ghe-hostname]/bevns/HelloWorld.git"

Or for go get:go get -v [ghe-hostname]/bevns/HelloWorld.git

Alternatively, another workaround is to manually clone the repository into your $GOPATH and then run go-install. For example:

Some other customers using Go have implemented a workaround where Go import statements refer to an internal server whose sole purpose is to direct go get to the proper repository URLs. The proxy server would respond to both HTTP and HTTPS and can return a meta tag that will point to your GitHub Enterprise instance.

This should then return a meta tag that looks something like this:<meta name="go-import" content="my.ghe.domain git https://my.ghe.domain/user/project.git">

That should then result in the correct repository being cloned.

They also added this:

Please also keep in mind that this proxy server would need to be a standalone server and should not be configured on your GitHub Enterprise server.

The best solution will come when the Go team accepts and merges #17898. That will detect the headers sent from both the GitHub Enterprise appliance and GitHub.com, and use that for verification that a Git repository is being requested.

I would also like to mention that some alternative Go package managers such as https://glide.sh/ work with the Go toolchain and support private Git servers and repositories.

This comment has been minimized.

@james-r-smith yes we use the former hack as a workaround at the moment.
The problem is that it forces the users of your library to modify their import paths in a way that is not obvious (i.e. appending .git). This adds unnecessary friction, especially to new users.

This comment has been minimized.

This comment has been minimized.

With the GitHub Enterprise 2.13 release, on-premises GitHub Enterprise instances will now send the <meta> tags expected by go get for private instances and all repositories.

For instances that are in private mode, you will still need to manage authentication for the git pull phase of go get, but there are easy workarounds such as putting an authentication token in your .netrc or the insteadOf technique to force SSH as mentioned above.

x/vgo, from what I can tell currently only supports GitHub.com, but that's to be expected considering it is still in proposal status.

A goal of introducing modules was to move away from invoking
version control tools directly, but it has become clear that we're not
ready to do that today. GitHub in particular imposes draconian limits
on HTTPS API access that they don't impose on ordinary Git access.
And we can avoid for now breaking company setups using private
Git servers.
Because GitHub Enterprise now serves ?go-get=1 queries in a way
that is compatible with old go get, dropping back to version control
tools makes that compatible with vgo too.
The next CL hooks this code into the rest of vgo.
For golang/go#24915.
For golang/go#23955.
For golang/go#24076.
Change-Id: I76bea30081047ab68286a5d095a0d55872c5a1a3
Reviewed-on: https://go-review.googlesource.com/107656
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>

This comment has been minimized.

edited

I see that this is marked as closed, but I've been unable to find an example of the solution.

I have private repos that will require API keys in deployment environments and I'm trying to figure out where I tell go.mod "when you see this import, construct this url with this token instead of whatever you would normally do".

So in the case of

go get awesome.io/go/sauce

The end result should be

git clone https://token:xxxxx@git.awesome.com/awesome/go-sauce.git

Can anyone give me an update on the best practice way to accomplish this? Something that doesn't require modifying an existing server would be preferred.

This comment has been minimized.

edited

@coolaj86 - could you use SSH keys instead of tokens (that's how I believe the authentication to GitHub Enterprise server normally works)? If so, you can add to your (and everyone's in your org 😭 )~/.gitconfig file:

[url "ssh://git@git.awesome.com"]
insteadOf = https://git.awesome.com

if (with a bit of luck such as GHE >= 2.13 - https://enterprise.github.com/releases/2.13.0/notes) your Git server replies to requests from the go tool to https://git.awesome.com/awesome/go-sauce?go-get=1 with a meta tag <meta name="go-import" content="git.awesome.com/awesome/go-sauce git https://git.awesome.com/awesome/go-sauce.git", then the go tool will end up cloning or updating from ssh://git@git.awesome.com/awesome/go-sauce.git.

Now suppose you don't have yet such a wonderful server support (GHE < 2.13 etc.) - you can still make the go tol fetch your repos and use them in modules. You would need to add a replace section to go.mod where for each "real" package (module) listed in the require section that you want to replace:

once again, notice the .git suffix in the replacement module which be not necessary with GHE >= 2.13 but I believe in all cases speeds up the checkouts/updates by avoiding an extra roundtrip with the ?go-git=1 request.

if awesome.io server returns the meta tag (<meta name="go-import" content="git.awesome.com/awesome/go-sauce git https://git.awesome.com/awesome/go-sauce.git"), you should be able to run go get awesome.io/go/sauce and get https://git.awesome.com/awesome/go-sauce.git checked out