On Wed, Aug 12, 2020 at 11:13, Neal Gompa <ngompa13@gmail.com> wrote:

On Wed, Aug 12, 2020 at 10:39 AM Lubos Kocman <lubos.kocman@suse.com> wrote:

...

No support for products

WTF are these?

They are kinda pointless nowadays, aren't they. We could have a product-like provide that does pretty much the same thing. Not to mention appstream operating-system metadata provides pretty much all that product does (and it's an existing standard!), we would just have to ship it

...

No support for patterns

These are just metapackages, but it wouldn't be hard to add a subcommand similar to the "group" one to manage these.

Not to mention, they already work with dnf as they are, since they are just rpm packages with placeholder content that depend, recommend and suggest some other packages.

...

No support for services

Wut?

It's one of those things where I really don't understand why it would ever be useful https://doc.opensuse.org/projects/libzypp/HEAD/zypp-services.html It also could really be a dnf plugin

...

No support for product licenses

Wut? Is this the SUSE metadata extension for packages to have EULAs? Or something else?

Always makes me wonder why stuff like this doesn't land in RPM first

...

I don't see how YaST can work with libdnf

This makes no sense to me. What do you mean by that?

I don't see how YaST couldn't work with libdnf. On the language side, considering most of YaST stuff dealing with libzypp is written in C++, that's doable. There are some parts in ruby that deal with zypper invoked in the middle of code, we could have libdnf bindings for ruby exposed so those parts can use an actual api. Maybe it's just lazyness, that I can understand. The holdup on getting dnf used here seems to be political more than technical, I don't think I wanna know how this discussion looks internally ;) LCP [Stasiek] https://lcp.world -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Wed, Aug 12, 2020 at 6:54 PM Aaron Puchert <aaronpuchert@alice-dsl.net> wrote:

Am 12.08.20 um 17:13 schrieb Neal Gompa:

...

It sounds like there's no way you can graduate any architecture. From my perspective, a supported architecture needs to exist in openSUSE:Factory. It needs to be part of the staging workflow, and needs to release at the exact same time.

How are you concluding that this is impossible? I'm missing some intermediate steps in your argument. Admittedly I have no idea if the resources are currently there. (If they aren't, maybe a generous donor can be found. Maybe aarch64 will not reach the single-core performance of x86_64 anytime soon, but maybe that can be solved with more cores.)

Just to be clear, this is not a rhetorical question, I'd just like you to get into the details.

...

I would give up on this multi-tier idea of supportability. You already have everything you're capable of: usable and supported (x86 arches) and unsupported and randomly broken (everything else).

Well, these are two tiers, right? The entire idea is that instead of saying x86_64 (& i586) is supported and the rest isn't, we say an architecture is supported if it satisfies some requirements.

The thing is, based on the conversations earlier, it seems like there is no way for openSUSE to support treating ARM, POWER, or RISC-V at the same level as x86. It's been indicated that there's not enough hardware resources to plug into the staging workflow, and the alternative architectures break too much to be included in openSUSE:Factory, and nobody wants to make it mandatory for builds to pass on these architectures before releasing snapshots.

From my perspective, that means any attempt to upgrade the support of alternative architectures is doomed to fail.

...

...

No offering of solutions if there are dependency problems (dnf just exits)

[...]

I'd be interested in understanding what makes that feature compelling, given that every time I've had to use it, it doesn't work

That's not my experience. I've had a few cases where conflicts were getting too big and I had to abort, but in most cases one of the solutions was what I wanted and worked fine.

Why is it compelling? Sometimes there are genuine conflicts, and you don't want to remove packages manually before installing a new one, and this allows you to do installation and removal in one transaction. Quite possible that the manual removal wasn't enough, and you've got more conflicts, and then you decide that you can't go further and the removal was pointless. If you let the solver figure out the entire thing at once, you avoid that painful do-while loop with rollback.

This is still possible with DNF, it just doesn't do it interactively. It will just fail and print out all the solver errors at once. You can use all those errors to adjust the transaction accordingly. You can also tell DNF to attempt to remove problematic packages all at once and attempt to resolve the transaction accordingly with "--allowerasing" The only thing we don't have in DNF right now is an equivalent of "zypper install foo -bar -baz" and "zypper remove +foo bar baz" (which do the same thing). There is "dnf swap bar foo", though. And there's the dnf shell to do more complex transactions...

...

[...] or leads to a broken system...

That would be strange, because requirements are still checked.

The point of this is to let you *ignore* dependencies and explicitly break them. It was actually an early design goal that DNF does *not* include this functionality because people often used it to subtly break their systems with less-than-obvious ill effects. Maybe it's worth revisiting this, but I'm not sure it is...

I've seen people run into strange issues when they added repos from different distributions, such as Tumbleweed repos on a Leap system, and then often dependencies don't fit. Otherwise I've not seen conflict resolution break anything, if anything the conflicts got out of hand and I had to abort.

I have seen this happen quite often while maintaining OBS servers. :( -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, Aug 13, 2020 at 12:26 AM Bernhard M. Wiedemann <bernhardout@lsmod.de> wrote:

On 13/08/2020 01.25, Neal Gompa wrote:

...

The thing is, based on the conversations earlier, it seems like there is no way for openSUSE to support treating ARM, POWER, or RISC-V at the same level as x86. It's been indicated that there's not enough hardware resources to plug into the staging workflow, and the alternative architectures break too much to be included in openSUSE:Factory, and nobody wants to make it mandatory for builds to pass on these architectures before releasing snapshots.

From my perspective, that means any attempt to upgrade the support of alternative architectures is doomed to fail.

I see the discussion the other way round: The docs say, of course we cannot promote RISC-V to primary, until someone solves the listed issues. E.g. add build hardware to OBS, have people fix build+test issues...

Either people put in as much effort as they did with aarch64 - which might therefore become as good as x86_64 soon

or people dont and those other archs remain secondary afterthoughts.

I also include AArch64 in that bucket of architectures that nobody wants to put enough effort into. Even with all the work being put into it, nobody wants to add it to openSUSE:Factory and the staging workflow because they don't want to buy the required hardware to do it. -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, Aug 13, 2020 at 2:01 PM Aaron Puchert <aaronpuchert@alice-dsl.net> wrote:

Am 13.08.20 um 01:25 schrieb Neal Gompa:

...

The thing is, based on the conversations earlier, it seems like there is no way for openSUSE to support treating ARM, POWER, or RISC-V at the same level as x86. It's been indicated that there's not enough hardware resources to plug into the staging workflow, and the alternative architectures break too much to be included in openSUSE:Factory, and nobody wants to make it mandatory for builds to pass on these architectures before releasing snapshots.

You're right that there is a long way to go from ports to where x86 is right now. But maybe there are some intermediate steps.

Technically I see no reason to couple the releases of snapshots for different architectures, you could reasonable decide to ship a snapshot for x86 only and skip it for aarch64.

Alternatively, you could stage updates separately in bigger batches, hoping that x86 staging already catches most issues and you'll only see ARM-specific issues in the second staging. Then aarch64 would lag behind, but maybe not too much.

So we have a few choices:

1) Include aarch64 in Factory staging, which needs more hardware (maybe) and could end up blocking x86 updates unreasonably. 2) Accept that Factory:ARM can sometimes be broken and we can't release snapshots for it. Can we get it back to releasable state quickly enough until the next issue appears? If not, then you're right. 3) Let Factory and Factory:ARM be separate repos and have a staging workflow Factory -> Factory:ARM. Then both repos could diverge, but Factory:ARM should be releasable most of the time.

With 1) we would be on par with x86, with 2) and 3) slightly behind, so there would be three tiers then. Still, it might be a good first step.

My experience has indicated that unless you make alternate architectures everyone's problem, it's just *not* going to get any better. For example, Fedora has x86_64, AArch64, and armv7hl as primary architectures, with ppc64le and s390x as secondary architectures. But for packagers, they are all effectively primary architectures because the build system will not let a build through without it successfully building for all architectures that it is supposed to be built for. openSUSE's problem is that they don't want to take that step to make their alternate architectures actually regularly useful by *forcing* every architecture into openSUSE:Factory. There has been no proposal from Dirk or anyone that even reasonably considers eliminating the "ports" tree. This is even sillier when you realize that SUSE Linux Enterprise doesn't bother with this malarkey and actually *does* force all architectures into the same development project because they know in order to make it supportable, they have to have a consistent tree for all architectures. Otherwise the "common codebase" is a lie.

...

This is still possible with DNF, it just doesn't do it interactively. It will just fail and print out all the solver errors at once. You can use all those errors to adjust the transaction accordingly.

You can also tell DNF to attempt to remove problematic packages all at once and attempt to resolve the transaction accordingly with "--allowerasing"

The only thing we don't have in DNF right now is an equivalent of "zypper install foo -bar -baz" and "zypper remove +foo bar baz" (which do the same thing). There is "dnf swap bar foo", though.

And there's the dnf shell to do more complex transactions...

I see. Well, some people like it more interactive and some want to automate. I don't think either is necessarily wrong, and if you have a lot of time you might just allow both.

...

...

...

[...] or leads to a broken system...

That would be strange, because requirements are still checked.

The point of this is to let you *ignore* dependencies and explicitly break them. It was actually an early design goal that DNF does *not* include this functionality because people often used it to subtly break their systems with less-than-obvious ill effects.

Maybe it's worth revisiting this, but I'm not sure it is...

Now I get what you meant. I admit that I've done this sometimes because requirements for packages were too strong. (Eventually I started filing SRs afterwards to make it a Recommends: or something like that.)

It's a somewhat questionable feature, but sometimes it's good to have the "I know what I'm doing" mode. And you can still unbreak your system with "zypper verify". But maybe it needs to be more difficult so that users don't accidentally end up there.

My belief is that this feature has actually made it easier to ignore fixing dependency issues in openSUSE projects. I think not having it would be a good way to force fixing these kinds of problems, since you have no other recourse but to play nice. -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Fri, Aug 14, 2020 at 8:39 AM Aaron Puchert <aaronpuchert@alice-dsl.net> wrote:

Am 14.08.20 um 13:23 schrieb Neal Gompa:

...

My experience has indicated that unless you make alternate architectures everyone's problem, it's just *not* going to get any better.

That's part of Dirk's proposal though, at least on paper. To quote: "The package maintainers commit to handle primary architectures with care and test-build them in the development projects before submitting changes. The package maintainers commit to handle regressions quickly and in general accept bugreports for that platform and work on a resolution."

Sorry, no. Not going to work. There is no reasonable way to enforce that. That being said, because *I* care, I actually *did* enable all architectures on the system:packagemanager:dnf[1] devel project a long time ago. But most devel projects will not do it. And the whole point of the staging workflow is so they don't need to. [1]: https://build.opensuse.org/project/show/system:packagemanager:dnf

...

For example, Fedora has x86_64, AArch64, and armv7hl as primary architectures, with ppc64le and s390x as secondary architectures. But for packagers, they are all effectively primary architectures because the build system will not let a build through without it successfully building for all architectures that it is supposed to be built for.

Most devel projects have alternative arches, so I think you're going to at least have an eye on failures there. But it's hard to prevent breakages if you just look at individual packages, because dependency changes can also break a build.

I'd love to see the stats here, because I'm pretty sure the overwhelming majority of devel projects do not.

Even with Staging you can end up breaking packages, because the Stagings don't consider the entire tree of >10,000 packages. So you'll have to rely on maintainers reacting on notifications that their package doesn't build anymore.

This is what the staging workflow is for, and if the notifications aren't working, we need better communication.

...

openSUSE's problem is that they don't want to take that step to make their alternate architectures actually regularly useful by *forcing* every architecture into openSUSE:Factory. There has been no proposal from Dirk or anyone that even reasonably considers eliminating the "ports" tree.

Part of the reason might be that this is a community project and most people can only reasonable debug issues on x86. Once I had to debug a miscompilation that was only observable on s390x and ppc64(be), and it was a pretty painful experience. (Just as a side note: I ended up finding the bug and porting back the necessary fix, so it never ended up in Factory:{PowerPC,zSystems}, whereas to my knowledge Fedora didn't do anything about it. Not sure if it ended up in any release though.)

...

This is even sillier when you realize that SUSE Linux Enterprise doesn't bother with this malarkey and actually *does* force all architectures into the same development project because they know in order to make it supportable, they have to have a consistent tree for all architectures. Otherwise the "common codebase" is a lie. SUSE employees probably have easy access to machines of all supported architectures, which makes it a bit different. Also SLE has considerably lower churn than Tumbleweed, so you're not going to need the same amount of build resources. I agree that this is overall a better model, but it's a bit harder for a rolling release community project.

Then ultimately this comes back to enabling the community to be able to fix these things. In Fedora, packagers and developers have access to test machines of all less-available architectures that they can shell into, build, test, and refine for those architectures[2]. Admittedly, because of the data center move, those are all down right now, but the general point is that they are available so that people are able to fix problems for those architectures. Solving that problem for openSUSE people should be equally easy: provide resources that the community can use for this purpose. [2]: https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintainer... -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am 14.08.20 um 14:50 schrieb Neal Gompa:

On Fri, Aug 14, 2020 at 8:39 AM Aaron Puchert <aaronpuchert@alice-dsl.net> wrote:

...

Most devel projects have alternative arches, so I think you're going to at least have an eye on failures there. But it's hard to prevent breakages if you just look at individual packages, because dependency changes can also break a build. I'd love to see the stats here, because I'm pretty sure the overwhelming majority of devel projects do not.

...

Even with Staging you can end up breaking packages, because the Stagings don't consider the entire tree of >10,000 packages. So you'll have to rely on maintainers reacting on notifications that their package doesn't build anymore. This is what the staging workflow is for, and if the notifications aren't working, we need better communication. There are two kinds of stagings: the “letter” stagings for packages in

Granted, what I wrote was based on my limited experience. Looking at a few devel projects reveals some that only build on x86 (e.g. KDE, X11:Wayland), but many build at least for aarch64 and ppc64le as well (e.g. Base:System, X11:XOrg, devel:tools). My experience is probably biased because in devel:tools:compiler we have 10 different architectures (i586, x86_64, ppc, ppc64, ppc64le, aarch64, armv6l, armv7l, s390x, riscv64), although not all are equally supported of course. But the policy is pretty clear: if we decide to promote aarch64 it needs to be active in all development projects, and that shouldn't be hard to enforce. the base system build all 2,000–3,000 packages of the base system from scratch for every submit. They also do some openQA testing, although to my understanding it's less than we do for snapshots. So we'll find issues in the base system, but not in the remaining ~10,000 packages. The “number” (adi) stagings for the remaining packages only build the package being submitted and don't do any openQA testing, so you won't find out if this breaks other packages. (There is not much to see there that you don't see in the devel project—basically only the install checker to my knowledge.)

Then ultimately this comes back to enabling the community to be able to fix these things. In Fedora, packagers and developers have access to test machines of all less-available architectures that they can shell into, build, test, and refine for those architectures[2].

Admittedly, because of the data center move, those are all down right now, but the general point is that they are available so that people are able to fix problems for those architectures.

Solving that problem for openSUSE people should be equally easy: provide resources that the community can use for this purpose.

[2]: https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintainer...

That's actually a nice idea. It's not often that I run into non-x86 issues that I can't debug from the logs, but sometimes I do, and then it would be good to have access to a native machine to debug that. Best regards, Aaron -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Fri, Aug 14, 2020 at 4:02 AM Stefan Seyfried <stefan.seyfried@googlemail.com> wrote:

Am 13.08.20 um 01:25 schrieb Neal Gompa:

...

On Wed, Aug 12, 2020 at 6:54 PM Aaron Puchert <aaronpuchert@alice-dsl.net> wrote:

...

Am 12.08.20 um 17:13 schrieb Neal Gompa:

...

...

No offering of solutions if there are dependency problems (dnf just exits)

[...]

I'd be interested in understanding what makes that feature compelling, given that every time I've had to use it, it doesn't work

That's not my experience. I've had a few cases where conflicts were getting too big and I had to abort, but in most cases one of the solutions was what I wanted and worked fine.

Why is it compelling? Sometimes there are genuine conflicts, and you don't want to remove packages manually before installing a new one, and this allows you to do installation and removal in one transaction. Quite possible that the manual removal wasn't enough, and you've got more conflicts, and then you decide that you can't go further and the removal was pointless. If you let the solver figure out the entire thing at once, you avoid that painful do-while loop with rollback.

This is still possible with DNF, it just doesn't do it interactively.

Then it's not a remotely comparable solution, sorry.

...

It will just fail and print out all the solver errors at once. You can use all those errors to adjust the transaction accordingly.

Yes, but this I can also do with plain rpm if i really want to hurt myself ;-)

...

You can also tell DNF to attempt to remove problematic packages all at once and attempt to resolve the transaction accordingly with "--allowerasing"

That's unfortunately never the solution that would seem useful or sensible to me (at least I cannot imagine it being useful).

...

...

...

[...] or leads to a broken system...

That would be strange, because requirements are still checked.

The point of this is to let you *ignore* dependencies and explicitly break them.

Well yes. "ignore" is probably an option that should be protected by an extra "i accept the risk" (uw-imap-style) confirmation prompt. It's for experts.

...

It was actually an early design goal that DNF does *not* include this functionality because people often used it to subtly break their systems with less-than-obvious ill effects.

I absolutely demand the right to shoot myself in the foot! ;-)

Yeah, uhh, no... :)

...

Maybe it's worth revisiting this, but I'm not sure it is...

...

I've seen people run into strange issues when they added repos from different distributions, such as Tumbleweed repos on a Leap system, and then often dependencies don't fit. Otherwise I've not seen conflict resolution break anything, if anything the conflicts got out of hand and I had to abort.

I have seen this happen quite often while maintaining OBS servers. :(

I do not remember this happening to me.

OBS 2.5 -> 2.6, OBS 2.6 -> 2.7, and OBS 2.8 -> 2.9 upgrades with zypper required this and it was *really* easy to accidentally break things. The end result was that I decided to switch to just destroying the whole system and rekicking it each time, because that's more reliable and nobody cares about fixing dependency breakages and making upgrades sane. Funnily enough, part of the motivation for me to start working on porting OBS to Fedora was because the maintenance process for OBS on openSUSE is miserable. (The other part was that I'm often trapped in situations where new Fedora or Ubuntu releases don't work because openSUSE provided dependencies can't support them).

Actually the zypper proposed resolutions on dependency problems being 99% useful versus the yast2 (pre-libsolv/zypper!!) proposed resolutions on dependency problems being 99% useless was *the* single selling point of zypper (actually of the satsolver) 15(?) years back.

Probably dnf does provide the same useful resolution options (as everyone uses the satsolver nowadays), but being unable to apply them interactively feels like a HUGE step backwards to me.

So here's the problem. When this feature exists, there's a lot less motivation to actually do fixes to the platform itself. You fix it for yourself, cool. And now you don't care anymore. One of the side-effects of not offering this feature in DNF is that people are a lot more motivated to make sure they don't break each other. One of the reasons that Fedora distribution upgrades are so painless is because it's part of the testing and release process to ensure this sort of thing *works*, and it's really easy to tell what's broken and how to fix it.

From my perspective, this is a net *win* in general.

Now I'm happy I did not join the dnf train yet, it would have been a total waste of time for me:

strolchi:~ # ls -1 /etc/zypp/repos.d|wc -l 18

=> you get conflicts you have to resolve once in a while with that many repos.

ngompa@yu-ohki ~> ls /etc/yum.repos.d/ | wc -l 32 Assuming those are a 1:1 map to repos (they aren't!) I can beat you on the repo count and still have a perfectly functioning system without that feature. -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org