[opensuse-go] Go Modules
Hi all, We are going to be facing quite a few issues in the near future because of Go modules. Go 1.11 added this concept of modules, which is supposed to eventually replace vendor/ and all of the other awful systems Go developers have had to deal with for the many years. Unfortunately, they are far from ideal at the moment. The current design of Go modules effectively requires that you download all of your dependencies from the internet during "go build". "go mod vendor" will generate a backwards-compatible vendor/ for you, *but* in order to stop "go build" from downloading stuff from the internet (which is disallowed in OBS obviously) you need to pass "-mod=vendor". The kicker is that only Go 1.11 supports "-mod=vendor", while "go build" supports Go modules as far back as Go 1.9. So when building currently we have to delete go.mod and go.sum. I proposed about a year ago that we should have some OBS service to deal with the vendoring problem by automating the pulling and setting up of vendor directories. Personally I think it's about time I dust off that idea and actually get to work on it -- since we now are going to need it for Go since (as I've discovered) some projects don't commit a vendor/ anymore. -- Aleksa Sarai Senior Software Engineer (Containers) SUSE Linux GmbH https://www.cyphar.com/
Aleksa Sarai writes:
We are going to be facing quite a few issues in the near future because of Go modules. Go 1.11 added this concept of modules, which is supposed to eventually replace vendor/ and all of the other awful systems Go developers have had to deal with for the many years.
Indeed, I'm very glad to have modules in Go. The transition period will be a challenge, as upstream Go projects migrate at their own pace from using 'dep' et al. and/or complex targets in their Makefile to gather dependency sources. My hope is that in the not too distant future, most Go projects will be buildable offline, without GOPATH setup, following a 'go mod vendor' call from an OBS source service, as you mention below.
In order to stop "go build" from downloading stuff from the internet (which is disallowed in OBS obviously) you need to pass "-mod=vendor".
The kicker is that only Go 1.11 supports "-mod=vendor", while "go build" supports Go modules as far back as Go 1.9.
Did you mean Go modules or 'finds dependencies in vendor/' (Go 1.6)?
So when building currently we have to delete go.mod and go.sum.
Can you expand a bit on the cases where go.{mod,sum} need to be deleted? With go1.11 I've been using env var GO111MODULE={auto|on|off}, does that cover those cases?
I proposed about a year ago that we should have some OBS service to deal with the vendoring problem by automating the pulling and setting up of vendor directories. Personally I think it's about time I dust off that idea and actually get to work on it -- since we now are going to need it for Go since (as I've discovered) some projects don't commit a vendor/ anymore.
I've been thinking about this as well and intend to contribute. What implementation are you envisioning for an OBS source service: calling the 'go mod vendor' CLI from one of bash, python, or a Go executable? Jeff -- Jeff Kowalczyk Software Engineer, Go Developer Experience SUSE Linux -- To unsubscribe, e-mail: opensuse-go+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-go+owner@opensuse.org
On 2018-11-08, Jeff Kowalczyk
Aleksa Sarai writes:
We are going to be facing quite a few issues in the near future because of Go modules. Go 1.11 added this concept of modules, which is supposed to eventually replace vendor/ and all of the other awful systems Go developers have had to deal with for the many years.
Indeed, I'm very glad to have modules in Go. The transition period will be a challenge, as upstream Go projects migrate at their own pace from using 'dep' et al. and/or complex targets in their Makefile to gather dependency sources. My hope is that in the not too distant future, most Go projects will be buildable offline, without GOPATH setup, following a 'go mod vendor' call from an OBS source service, as you mention below.
In order to stop "go build" from downloading stuff from the internet (which is disallowed in OBS obviously) you need to pass "-mod=vendor".
The kicker is that only Go 1.11 supports "-mod=vendor", while "go build" supports Go modules as far back as Go 1.9.
Did you mean Go modules or 'finds dependencies in vendor/' (Go 1.6)?
I meant Go modules, but see below (they backported some module features but I actually just misremembered what the problem was).
So when building currently we have to delete go.mod and go.sum.
Can you expand a bit on the cases where go.{mod,sum} need to be deleted? With go1.11 I've been using env var GO111MODULE={auto|on|off}, does that cover those cases?
I just realised that I misrepresented what the issue was. The problem is that I need to build restic for Leap which has an older Go compiler (1.9 is the latest I think -- with 1.10 coming out once they get the new Docker from SLES). The issue was that I needed to find a way to build with both Go 1.11 and Go 1.9 (where there was already a pre-populated vendor/ tree) and Go 1.11 would not use the vendor/ tree by default. -mod=vendor fixes this problem but obviously that doesn't work on Go 1.9. So I think you're right that GO111MODULE=off should work since older Go versions will ignore it. I will test this (the go.mod thing was mostly a hack to get the build to work so I could get my backups working again).
I proposed about a year ago that we should have some OBS service to deal with the vendoring problem by automating the pulling and setting up of vendor directories. Personally I think it's about time I dust off that idea and actually get to work on it -- since we now are going to need it for Go since (as I've discovered) some projects don't commit a vendor/ anymore.
I've been thinking about this as well and intend to contribute. What implementation are you envisioning for an OBS source service: calling the 'go mod vendor' CLI from one of bash, python, or a Go executable?
I would aim for it to be more generic so that we can fix problems with languages like Node. Rust was one other pitch I had at the time -- but I've heard that the rust2rpm story is effectively finished and so there's nothing to fix there (I don't really like the *2rpm model, but maybe I just like whinging for no good reason :P). It would have to extract the source archive, figure out what vendoring system it's using and then split out the vendor tree (either by generating a new one in the case of "go mod vendor" or other vendoring tools or by taking the existing one). Then it should hopefully generate some .spec metadata so that legal and maintenance know what on earth is inside "vendor.tar.xz". I am quite worried how nested vendors with modules will work. This in particular will be quite a nightmare because I don't think that "go mod vendor" will recursively handle vendor/s that use go modules (but I'd have to check). -- Aleksa Sarai Senior Software Engineer (Containers) SUSE Linux GmbH https://www.cyphar.com/
On Fri, Nov 9, 2018 at 5:19 AM Aleksa Sarai
On 2018-11-08, Jeff Kowalczyk
wrote: Aleksa Sarai writes:
We are going to be facing quite a few issues in the near future because of Go modules. Go 1.11 added this concept of modules, which is supposed to eventually replace vendor/ and all of the other awful systems Go developers have had to deal with for the many years.
Indeed, I'm very glad to have modules in Go. The transition period will be a challenge, as upstream Go projects migrate at their own pace from using 'dep' et al. and/or complex targets in their Makefile to gather dependency sources. My hope is that in the not too distant future, most Go projects will be buildable offline, without GOPATH setup, following a 'go mod vendor' call from an OBS source service, as you mention below.
In order to stop "go build" from downloading stuff from the internet (which is disallowed in OBS obviously) you need to pass "-mod=vendor".
The kicker is that only Go 1.11 supports "-mod=vendor", while "go build" supports Go modules as far back as Go 1.9.
Did you mean Go modules or 'finds dependencies in vendor/' (Go 1.6)?
I meant Go modules, but see below (they backported some module features but I actually just misremembered what the problem was).
So when building currently we have to delete go.mod and go.sum.
Can you expand a bit on the cases where go.{mod,sum} need to be deleted? With go1.11 I've been using env var GO111MODULE={auto|on|off}, does that cover those cases?
I just realised that I misrepresented what the issue was.
The problem is that I need to build restic for Leap which has an older Go compiler (1.9 is the latest I think -- with 1.10 coming out once they get the new Docker from SLES).
The issue was that I needed to find a way to build with both Go 1.11 and Go 1.9 (where there was already a pre-populated vendor/ tree) and Go 1.11 would not use the vendor/ tree by default. -mod=vendor fixes this problem but obviously that doesn't work on Go 1.9.
So I think you're right that GO111MODULE=off should work since older Go versions will ignore it. I will test this (the go.mod thing was mostly a hack to get the build to work so I could get my backups working again).
I proposed about a year ago that we should have some OBS service to deal with the vendoring problem by automating the pulling and setting up of vendor directories. Personally I think it's about time I dust off that idea and actually get to work on it -- since we now are going to need it for Go since (as I've discovered) some projects don't commit a vendor/ anymore.
I've been thinking about this as well and intend to contribute. What implementation are you envisioning for an OBS source service: calling the 'go mod vendor' CLI from one of bash, python, or a Go executable?
I would aim for it to be more generic so that we can fix problems with languages like Node. Rust was one other pitch I had at the time -- but I've heard that the rust2rpm story is effectively finished and so there's nothing to fix there (I don't really like the *2rpm model, but maybe I just like whinging for no good reason :P).
For Rust, as part of the work in the Fedora Rust SIG, we decided to primarily support an unbundled/unvendored model because Rust has a sane mechanism for handling dependencies. And part of the point of Fedora is to make it so that stuff is using the latest software whenever possible. The problem with allowing vendoring at all is that you have to allow vendoring at multiple levels (which complicates package review, rebuilds and updates, etc.), and while Fedora permits bundled dependencies (provided they are documented using bundled() Provides), unbundled dependencies are preferred. We designed rust2rpm to be a "generate and build" kind of tool, so it tries to do everything correctly up front for you. The newest version (which I haven't even imported into openSUSE yet, but it's on my docket) supports parsing Cargo.toml and doing filtering for only Linux-specific dependencies, which simplifies things even more and eliminates the primary reason for patching crates. I've also been writing an OBS source service for it and working on enhancements to rust2rpm to support that model.
It would have to extract the source archive, figure out what vendoring system it's using and then split out the vendor tree (either by generating a new one in the case of "go mod vendor" or other vendoring tools or by taking the existing one). Then it should hopefully generate some .spec metadata so that legal and maintenance know what on earth is inside "vendor.tar.xz".
I am quite worried how nested vendors with modules will work. This in particular will be quite a nightmare because I don't think that "go mod vendor" will recursively handle vendor/s that use go modules (but I'd have to check).
So as part of the work we're doing in the Fedora Go SIG (which I invited all of you folks to participate in some time ago[1]), we're already starting to look at this problem. A number of us are looking at Go modules as an opportunity to start unbundling and sharing dependency code, since modules need to be versioned. We've also got code for handling vendor trees and inspecting them, which is used in gofed and our rpm provides generators. We support both models today. [1]: https://lists.opensuse.org/opensuse-go/2018-09/msg00000.html P.S.: As a general comment, it's a bad idea to commit vendored code into your repository, and you should feel bad about doing so. When you do that, you are essentially throwing your hands in the air about that code and committing to supporting it independently. Moreover, every time someone does that, the code has to be inspected to figure out the licensing of the project as a whole, which makes packaging Go software more annoying. It also dis-incentivizes people from fixing issues in upstream dependency code, since it's "easier" to just fix it in your copy, effectively forking it. -- 真実はいつも一つ!/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-go+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-go+owner@opensuse.org
On 2018-11-09, Neal Gompa
On Fri, Nov 9, 2018 at 5:19 AM Aleksa Sarai
wrote: On 2018-11-08, Jeff Kowalczyk
wrote: Aleksa Sarai writes:
We are going to be facing quite a few issues in the near future because of Go modules. Go 1.11 added this concept of modules, which is supposed to eventually replace vendor/ and all of the other awful systems Go developers have had to deal with for the many years.
Indeed, I'm very glad to have modules in Go. The transition period will be a challenge, as upstream Go projects migrate at their own pace from using 'dep' et al. and/or complex targets in their Makefile to gather dependency sources. My hope is that in the not too distant future, most Go projects will be buildable offline, without GOPATH setup, following a 'go mod vendor' call from an OBS source service, as you mention below.
In order to stop "go build" from downloading stuff from the internet (which is disallowed in OBS obviously) you need to pass "-mod=vendor".
The kicker is that only Go 1.11 supports "-mod=vendor", while "go build" supports Go modules as far back as Go 1.9.
Did you mean Go modules or 'finds dependencies in vendor/' (Go 1.6)?
I meant Go modules, but see below (they backported some module features but I actually just misremembered what the problem was).
So when building currently we have to delete go.mod and go.sum.
Can you expand a bit on the cases where go.{mod,sum} need to be deleted? With go1.11 I've been using env var GO111MODULE={auto|on|off}, does that cover those cases?
I just realised that I misrepresented what the issue was.
The problem is that I need to build restic for Leap which has an older Go compiler (1.9 is the latest I think -- with 1.10 coming out once they get the new Docker from SLES).
The issue was that I needed to find a way to build with both Go 1.11 and Go 1.9 (where there was already a pre-populated vendor/ tree) and Go 1.11 would not use the vendor/ tree by default. -mod=vendor fixes this problem but obviously that doesn't work on Go 1.9.
So I think you're right that GO111MODULE=off should work since older Go versions will ignore it. I will test this (the go.mod thing was mostly a hack to get the build to work so I could get my backups working again).
I proposed about a year ago that we should have some OBS service to deal with the vendoring problem by automating the pulling and setting up of vendor directories. Personally I think it's about time I dust off that idea and actually get to work on it -- since we now are going to need it for Go since (as I've discovered) some projects don't commit a vendor/ anymore.
I've been thinking about this as well and intend to contribute. What implementation are you envisioning for an OBS source service: calling the 'go mod vendor' CLI from one of bash, python, or a Go executable?
I would aim for it to be more generic so that we can fix problems with languages like Node. Rust was one other pitch I had at the time -- but I've heard that the rust2rpm story is effectively finished and so there's nothing to fix there (I don't really like the *2rpm model, but maybe I just like whinging for no good reason :P).
For Rust, as part of the work in the Fedora Rust SIG, we decided to primarily support an unbundled/unvendored model because Rust has a sane mechanism for handling dependencies. And part of the point of Fedora is to make it so that stuff is using the latest software whenever possible. The problem with allowing vendoring at all is that you have to allow vendoring at multiple levels (which complicates package review, rebuilds and updates, etc.), and while Fedora permits bundled dependencies (provided they are documented using bundled() Provides), unbundled dependencies are preferred.
We designed rust2rpm to be a "generate and build" kind of tool, so it tries to do everything correctly up front for you. The newest version (which I haven't even imported into openSUSE yet, but it's on my docket) supports parsing Cargo.toml and doing filtering for only Linux-specific dependencies, which simplifies things even more and eliminates the primary reason for patching crates. I've also been writing an OBS source service for it and working on enhancements to rust2rpm to support that model.
That is very cool. My main issue with x2rpm-style solutions is that they make day-to-day maintenance quite hard. You need to submit loads of packages when updating, and this immediately causes conflicts in versions between different projects (though thankfully with Rust you don't have the every-commit-is-a-new-version problem that you have with Go). This then requires you to package a separate version of every dependency (and if the same version is shared then you now have to maintain it in tandem with someone else to make sure it doesn't get removed when they update their projects). Now, if this is all very easy to do then it's obviously much less of a pain. The idealist within me says that we should really be looking at merging the namespace of distribution packages and language packages rather than replicating the language packages as distribution ones, but I imagine that will take much more work. My experiences with seeing how Ruby gems were packaged (which also used an x2rpm system) doesn't inspire confidence that this model is the most viable in the long term. But I could be completely wrong. In the context of an OBS service, my proposal is mainly that the vendor.tar.xz should be generated at OBS build time and it should provide all the metadata necessary to figure out what is inside the tarball. While this does duplicate libraries between packages, I would be happy to bet that less than 10% of all projects that share libraries use the same git commit -- so the duplication is going to happen either way (it's just more maintenance annoyance to make this duplication happen with RPMs).
It would have to extract the source archive, figure out what vendoring system it's using and then split out the vendor tree (either by generating a new one in the case of "go mod vendor" or other vendoring tools or by taking the existing one). Then it should hopefully generate some .spec metadata so that legal and maintenance know what on earth is inside "vendor.tar.xz".
I am quite worried how nested vendors with modules will work. This in particular will be quite a nightmare because I don't think that "go mod vendor" will recursively handle vendor/s that use go modules (but I'd have to check).
So as part of the work we're doing in the Fedora Go SIG (which I invited all of you folks to participate in some time ago[1]), we're already starting to look at this problem. A number of us are looking at Go modules as an opportunity to start unbundling and sharing dependency code, since modules need to be versioned. We've also got code for handling vendor trees and inspecting them, which is used in gofed and our rpm provides generators. We support both models today.
[1]: https://lists.opensuse.org/opensuse-go/2018-09/msg00000.html
I was trying to package umoci for Fedora a week or two ago, and I couldn't follow the instructions on the Wiki page. gofed appeared to not like creating packages for me? Did I misunderstand what how I'm supposed to use it? (I did this in a Fedora VM to make sure that the tools were supposed to work find. The generated .spec had different macros and a different structure to the documentation.) (Obviously I am hardly in a place to talk about bad documentation -- the golang packaging Wiki is hardly an accurate description of what you should do on openSUSE. But I was very confused what the "right way" is.)
P.S.: As a general comment, it's a bad idea to commit vendored code into your repository, and you should feel bad about doing so. When you do that, you are essentially throwing your hands in the air about that code and committing to supporting it independently. Moreover, every time someone does that, the code has to be inspected to figure out the licensing of the project as a whole, which makes packaging Go software more annoying. It also dis-incentivizes people from fixing issues in upstream dependency code, since it's "easier" to just fix it in your copy, effectively forking it.
I agree in principle, but unfortunately some languages have made it hard to have a choice. Please remember that Go didn't even have a way of pinning dependency versions -- and vendoring was a community hack (eventually becoming a bit of GOPATH voodoo) that solved the problem temporarily until a better solution came out of the Go team. And we ended up with vendor/. There was literally no other option before Go 1.11 and there still is very few options in the medium term until everyone uses Go modules. -- Aleksa Sarai Senior Software Engineer (Containers) SUSE Linux GmbH https://www.cyphar.com/
participants (3)
-
Aleksa Sarai
-
Jeff Kowalczyk
-
Neal Gompa