Comments

Binary-only packages are increasingly hard to support safely. There is no guarantee that the compilation of the binary-only package used the same versions of the dependencies that the final link does (and it would probably be too onerous to insist on that). As a result, the binary-only package may have been compiled using escape analysis results or inline function bodies for dependencies that are no longer accurate. The result can be silent memory corruption.

My memory is that we added binary-only packages originally so that a professor could hand out binary solution sets so that a student who didn't finish lab 2 could still move on to lab 3. I don't know why else anyone uses them anymore, but they're probably going to break more and more as the compiler gets more sophisticated.

Should we just remove them? If not, why not? What do people need them for?

This comment has been minimized.

edited

I'm in favor of the proposal, nonetheless I guess the important property of binary packages is they make some closed source package vendors feel "safe" about their IP.

That or plugins need to not panic (when loading) and complain about different versions of the same package and also offer unloading support (i.e: find a way to sever the link with the runtime and free evreything).

Vendoring binary packages is useful if you're selling an SDK or toolkit.

This comment has been minimized.

In our case we use it as part of our offering to Enterprise customers. Being able to provide non-source distributions and as a way to hide some of our I.P clears us with the IT departments of such companies.

Our biggest use is being able to provide a static library (the BOP package) to third parties without providing our source, we provide an SDK that should work without any connection to our Servers, hence we package the SDK as binary-only-package that includes some proprietary I.P. that is licensed.

While that approach is not the most secure way to protect our source, it is the right balance right now, we just don't include more sensitive functionality in what we distribute.

Another important characteristic is that being able to have a higher level of friction of preventing modification to what we distribute; i.e. even if we provide non-IP that has to function in a specific way, not having BOP's would make easier for third parties to modify the SDK.

I'm not entirely in favor of dropping support for binary-only-packages because I believe that many other languages support a way of distributing packages that don't have the original source in them:

This comment has been minimized.

In general I'm in favour of giving incentive to provide source. But there are legitimate binary-only cases related to not only IP but security.

What relationship would removing support for BOPs this have to plugins and related build modes (shared/c-shared/c-archive)? Could one still provide a binary in any of these forms and be able to use that? If so, then a simple solution for at least some BOP use cases would be to use a wrapper package that loads a plugin, or uses -linkshared or a c-archive via cgo and the plugin or c-archive would be outside of go get/modules and distributed separately as binary.

I have no plans to distribute such things, but would like to be able offer the capacity to others in some form. If something like the above workarounds worked, then that would provide a way to drop BOP
support and say "use workaround ..." for those who want it.

At the same time I understand there are still some issues with these other binary distribution mechanisms and modules.

This comment has been minimized.

Thanks for asking about plugins. There are no plans to remove plugin support, and suggesting the use of plugins as binary blobs instead of binary-only packages makes sense to me. Plugins have many of the same concerns, but they already have a proper check for mismatched API (avoiding the silent memory corruption problem) and it's always better to have one mechanism than two.

This comment has been minimized.

I agree that binary-only-packages are problematic and should be removed.
I tried to use plugins but you can't unload them.
I decided to live with that and restart the process in case the plugin had to be reloaded but then:
You can't load a plugin if you vendor your dependencies and one of this dependency is shared with the host.
Right now I'm using go tools (compile and link) and reuse the .a but that's kinda ugly

This comment has been minimized.

@joshlf It's hard for me to imagine any real case in which plugins are noticeably less efficient than binary-only packages. It's possible to construct an artificial case, but I doubt that would match any real implementation.

This comment has been minimized.

Thanks for asking about plugins. There are no plans to remove plugin support, and suggesting the use of plugins as binary blobs instead of binary-only packages makes sense to me. Plugins have many of the same concerns, but they already have a proper check for mismatched API (avoiding the silent memory corruption problem) and it's always better to have one mechanism than two.

Great to hear. It seems to me it wouldn't be too much to adapt the source stubs from a BOP to wrap a plugin and an init() to load it from somewhere, and well worth it to both BOP users and Go maintenance.

This comment has been minimized.

Will leave this open for another week but it sounds like we can retire binary-only packages in favor of using plugins. We could advertise in the Go 1.12 release notes that it will be the last release to support binary-only packages and they will be removed in Go 1.13.

andybons
changed the title from
proposal: cmd/go: drop support for binary-only packages
to
cmd/go: drop support for binary-only packagesOct 31, 2018

This comment has been minimized.

As a corporate programmer, the closed-source use case is very important. A language that is so RMS idealism friendly that it doesn't even allow one to use closed-source dependencies is a very hard sell in many corporate environments.

This comment has been minimized.

Consider the case when a corporation is using a licensed closed-source software package as a dependency on an internal project. Right now, if the vendor of a closed-source package wanted to make their licensed package useful to go programmers, their only option is to author it as a plugin. As stated elsewhere, this has drawbacks: complexity of .so linking, lack of non Linux support, etc.

This comment has been minimized.

I'd argue that using a binary-only package was never really an option. The binary would require an exact version of Go, as well as an exact version of all the packages it depends on. It's certainly possible, but I wouldn't say it's ever been a good or easy way to distribute software. The intended purpose, the professor scenario, is a much more restricted use case.

complexity of .so linking

Please expand on this. If you believe there's a bug or something to improve in the plugin package or the go tool, you should open a separate issue.

lack of non Linux support

What particular platform is noticeably missing plugin support? Assuming you mean Windows, that's being tracked in #19282. I believe effort would be better invested in issues like that one, instead of on maintaining binary-only package support.

This comment has been minimized.

edited

It is possible to use vendoring with plugins (without set package import path)? The use of multiple plugins is hard because they need to use exactly the same version of common dependencies. Would it be possible to use the latest minor version for each dependency otherwise?
Edit: the last point seems to be impossible to do because we cannot unload a plugin.

This comment has been minimized.

FWIW, I have a "convenience" use case for binary-only packages: cross compiled cgo packages that are inconvenient to compile on the host because everybody would have to install the relevant C cross compiler (for the target).

Concrete case: I am starting to write a go wrapper for the librobotcontrol C library for the beaglebone blue.

That C library makes no sense on the host (be it windows, mac or linux).
AFAIK to cross compile the wrapper I'd need to copy the relevant header files and libraries on the host, and also provide cgo a C cross compiler.

I was strongly considering building the wrapper natively on the beaglebone (very easy to do), and just redistribute the compiled object file so that users do not need to set up the cgo cross compilation environment.

In this case "users" would be a group of 15 years old highschool students I am mentoring for a school-related robotics project.

Anyway I can probably do this cgo wrapper as a plugin, with a bit of boilerplate code to export a type safe interface to the wrapped functions.

Do you still think to deprecate binary-only packages?
If yes I'll go the plugin way, since that would be the only supported option in the future.

This comment has been minimized.

@massimiliano-mantione Another approach you can use is .syso files. Compile all the C code into a single object file per support platform, and name it something like c_GOOS_GOARCH.syso. Then the Go code can be the same as always, and can use cgo to call into the C code provided as a .syso file.