Comments

edited

This is a tester to see if there is any support for this idea. If there is then I will write up a more detailed proposal.

Currently the go get command expects to interact with a source control system to check out source code. There is no way to package up a source archive (e.g. tar or zip) and use go get to retrieve this.

This proposal would modify the go tool to look for a source archive according to a to-be-defined scheme when the user types go get package. The go tool would download and unpack the source archive in the correct location. Alternatively the go tool could be modified to keep the archive file and support reading source from the archive directly during a build.

One possible strawman scheme would be to extend the .vcs package path syntax to allow the specification of an archive format, e.g. example.com/code/package.zip. This would be the direct URL of the source archive of the package.

There are a number of advantages to using source archives:

simpler publishing of packages: just place the archive file on a web server

removes need for installing support utilities such as git or mercurial

convenient versioning of packages by adopting a semver naming convention for the archive

This comment has been minimized.

This comment has been minimized.

As @bradfitz noted, this is suggesting similar functionality to #16545. That proposal got stuck on the behaviour of go get -u, especially in the situation where the downloaded source had been edited.

I think this complexity can be avoided by stating that source archives are immutable. go get -u example.com/code/package-1.0.zip would have no effect if the directory $GOPATH/src/example.com/code/package-1.0 already exists.

To upgrade to a new version of a package, a new package path needs to be used, e.g. go get -u example.com/code/package-1.1.zip would download and extract source files to $GOPATH/src/example.com/code/package-1.1

This comment has been minimized.

What if package-1.0.zip changed? Or more realistically, if a user put up mycode.zip and tried to use go get -u on it? Would -u be an error if it already existed on disk? Would we save the server's Content-Length & ETag? Where?

This comment has been minimized.

edited

Changing package-1.0.zip breaks the contract that source archives are immutable. The user would need to intervene and remove the old version manually. This is the same situation that occurs when getting from a vcs where the user has modified the local source and not committed it or the remote source has rebased the user's current commit out of existence. go get should not try and fix these problems.

-u would be a no-op if the directory existed. The source is assumed to be immutable so it cannot be updated if it already exists. This is similar (I think) to the behaviour of go get in a detached head situation.

I think saving the etag/sha256sum is out of scope of this change. I would expect the go get process to check the sha256sum of the file during download if a convention is defined for the naming of a file containing the sum, e.g. example.com/code/package-1.0.zip.sha256sum

This comment has been minimized.

Changing package-1.0.zip breaks the contract that source archives are immutable.

I don't think that's a sufficient answer. Users are going to violate it anyway, either on purpose or by accident, and we need an answer to why two users have different results when one ran "go get" at time 1 and the other ran "go get" at time 2, then both ran "go get -u" and things were still different. We deal with enough of those support complaints already. This would make it worse. It's easy enough to say "just go rm -rf $GOPATH/src/foo", but in this case "foo" might be a far away dependency of the thing the user actually cared about.

In other words, we already support the "-u" flag. I don't think it's acceptable to just say that sometimes "-u" doesn't work for certain packages, seemingly randomly to the user.

This comment has been minimized.

edited

As you pointed out on #16545 we don't want to implement a vcs system. If users want to modify their code in place and reuse the same package path then they should use the pre-existing go get support for common vcs systems. For source archives, versioning is achieved by naming the package paths and assuming the archive is immutable. -u would not download a replacement for a source archive but it would traverse the dependency tree and recursively update imports as normal. -u -v would indicate that the package source is assumed to be up to date.

If there were a convention for locating a remote sha256sum then potentially we could add a way of verifying that the local source has the same hash as the remote but this is only informative.

I don't think it's acceptable to just say that sometimes "-u" doesn't work for certain packages, seemingly randomly to the user.

I don't think it would appear to be at random. The user is typing in a package path that ends with .zip or .tar. This is a signal that something is different and using -v would explain that difference.

Immutability of source archives could be seen as a goal in of itself. The current mechanism for version pinning has confusing behaviour for end users. For example github.com/aktau/github-release uses semver to tag releases (and coincidentally is a tool for doing that). A user can pin a release by checking out a tag, e.g. cd $GOPATH/src/github.com/aktau/github-release; git checkout v0.5

Subsequent usages of go get -u result in a potentially confusing error:

User 1

User 2

Question: do User 1 and User 2 have the same code on disk? Will the github.com/foo/bar package be the same?

With your proposal, you no longer know. Because foo/bar might transitively depend on what you're assuming is an "immutable" package, but only because you're wishfully declaring that. A user could actually put my-foo-bar-dep.zip on an HTTP server and update it, but then Users 1 and 2 above could have different code on disk, even after a go get -u.