On Fri, Nov 9, 2018 at 5:19 AM Aleksa Sarai <asarai@suse.de> wrote:

On 2018-11-08, Jeff Kowalczyk <jkowalczyk@suse.com> wrote:

...

Aleksa Sarai writes:

...

We are going to be facing quite a few issues in the near future because of Go modules. Go 1.11 added this concept of modules, which is supposed to eventually replace vendor/ and all of the other awful systems Go developers have had to deal with for the many years.

Indeed, I'm very glad to have modules in Go. The transition period will be a challenge, as upstream Go projects migrate at their own pace from using 'dep' et al. and/or complex targets in their Makefile to gather dependency sources. My hope is that in the not too distant future, most Go projects will be buildable offline, without GOPATH setup, following a 'go mod vendor' call from an OBS source service, as you mention below.

...

In order to stop "go build" from downloading stuff from the internet (which is disallowed in OBS obviously) you need to pass "-mod=vendor".

The kicker is that only Go 1.11 supports "-mod=vendor", while "go build" supports Go modules as far back as Go 1.9.

Did you mean Go modules or 'finds dependencies in vendor/' (Go 1.6)?

I meant Go modules, but see below (they backported some module features but I actually just misremembered what the problem was).

...

...

So when building currently we have to delete go.mod and go.sum.

Can you expand a bit on the cases where go.{mod,sum} need to be deleted? With go1.11 I've been using env var GO111MODULE={auto|on|off}, does that cover those cases?

I just realised that I misrepresented what the issue was.

The problem is that I need to build restic for Leap which has an older Go compiler (1.9 is the latest I think -- with 1.10 coming out once they get the new Docker from SLES).

The issue was that I needed to find a way to build with both Go 1.11 and Go 1.9 (where there was already a pre-populated vendor/ tree) and Go 1.11 would not use the vendor/ tree by default. -mod=vendor fixes this problem but obviously that doesn't work on Go 1.9.

So I think you're right that GO111MODULE=off should work since older Go versions will ignore it. I will test this (the go.mod thing was mostly a hack to get the build to work so I could get my backups working again).

...

...

I proposed about a year ago that we should have some OBS service to deal with the vendoring problem by automating the pulling and setting up of vendor directories. Personally I think it's about time I dust off that idea and actually get to work on it -- since we now are going to need it for Go since (as I've discovered) some projects don't commit a vendor/ anymore.

I've been thinking about this as well and intend to contribute. What implementation are you envisioning for an OBS source service: calling the 'go mod vendor' CLI from one of bash, python, or a Go executable?

I would aim for it to be more generic so that we can fix problems with languages like Node. Rust was one other pitch I had at the time -- but I've heard that the rust2rpm story is effectively finished and so there's nothing to fix there (I don't really like the *2rpm model, but maybe I just like whinging for no good reason :P).

For Rust, as part of the work in the Fedora Rust SIG, we decided to primarily support an unbundled/unvendored model because Rust has a sane mechanism for handling dependencies. And part of the point of Fedora is to make it so that stuff is using the latest software whenever possible. The problem with allowing vendoring at all is that you have to allow vendoring at multiple levels (which complicates package review, rebuilds and updates, etc.), and while Fedora permits bundled dependencies (provided they are documented using bundled() Provides), unbundled dependencies are preferred. We designed rust2rpm to be a "generate and build" kind of tool, so it tries to do everything correctly up front for you. The newest version (which I haven't even imported into openSUSE yet, but it's on my docket) supports parsing Cargo.toml and doing filtering for only Linux-specific dependencies, which simplifies things even more and eliminates the primary reason for patching crates. I've also been writing an OBS source service for it and working on enhancements to rust2rpm to support that model.

It would have to extract the source archive, figure out what vendoring system it's using and then split out the vendor tree (either by generating a new one in the case of "go mod vendor" or other vendoring tools or by taking the existing one). Then it should hopefully generate some .spec metadata so that legal and maintenance know what on earth is inside "vendor.tar.xz".

I am quite worried how nested vendors with modules will work. This in particular will be quite a nightmare because I don't think that "go mod vendor" will recursively handle vendor/s that use go modules (but I'd have to check).

So as part of the work we're doing in the Fedora Go SIG (which I invited all of you folks to participate in some time ago[1]), we're already starting to look at this problem. A number of us are looking at Go modules as an opportunity to start unbundling and sharing dependency code, since modules need to be versioned. We've also got code for handling vendor trees and inspecting them, which is used in gofed and our rpm provides generators. We support both models today. [1]: https://lists.opensuse.org/opensuse-go/2018-09/msg00000.html P.S.: As a general comment, it's a bad idea to commit vendored code into your repository, and you should feel bad about doing so. When you do that, you are essentially throwing your hands in the air about that code and committing to supporting it independently. Moreover, every time someone does that, the code has to be inspected to figure out the licensing of the project as a whole, which makes packaging Go software more annoying. It also dis-incentivizes people from fixing issues in upstream dependency code, since it's "easier" to just fix it in your copy, effectively forking it. -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org