https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c1 --- Comment #1 from Fabian Vogt --- Created attachment 847399 --> https://bugzilla.suse.com/attachment.cgi?id=847399&action=edit YaST logs (from the assets page) I attached the YaST logs from an openQA run. It's 100% reproducible so far, but sometimes it results in an "internal error" dialog instead of YaST just vanishing. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c3 --- Comment #3 from Fabian Vogt --- (In reply to Stefan Hundhammer from comment #2)

As you wrote, this happens when trying to upgrade a KDE Live 42.3 system which is not anything that we are supporting; we discussed this numerous times before.

No, this is about upgrading a "normal" 42.3 using the live media. Upgrading a live system doesn't make sense.

Does this also happen with any of the supported systems? Like e.g. Leap 15.2? Do we have an openQA test case with the equivalent failure when trying to upgrade from that supported scenario to Leap 15.3?

As I wrote in the report, there hasn't been an openQA test of that yet. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c5 Ancor Gonzalez Sosa changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ancor@suse.com Assignee|yast2-maintainers@suse.de |yast-community@opensuse.org Flags|needinfo?(fvogt@suse.com) | --- Comment #5 from Ancor Gonzalez Sosa --- As explained in many other bugs before. Installing and/or upgrading from a liveCD is a not supported scenario. At least, the YaST Team cannot make any compromise about it unless https://jira.suse.com/browse/PM-1565 becomes a reality (which would imply dropping other features). I could of course add this bug to the queue of tasks of the YaST Team... but it will likely just lie there for months or years. So assigning this to yast-community with the hope of someone taking care. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c7 Fabian Vogt changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bzeller@suse.com, | |ma@suse.com Flags| |needinfo?(ma@suse.com), | |needinfo?(bzeller@suse.com) --- Comment #7 from Fabian Vogt --- That this fails in TW now too made the list of changes much shorter, so I could do some tests. I found that this is caused by 99903f03c551288e6bb522522b547f1fc8e4ee32 in yast-packager, "Move the libzypp cache (bsc#1182928)". It does the following shortly before starting the installation: # copy the credential files (libzypp loads them from target) # and the repository cache to the target system Pkg.SourceCacheCopyTo(Installation.destdir) # symlink the cache from inst-sys to save same space (RAM!) cache_path = Pkg.ZConfig()["repo_cache_path"] || "/var/cache/zypp" log.info("Zypp cache size: #{`du -h -s #{cache_path.shellescape}`}") ::FileUtils.rm_rf(cache_path) File.symlink(File.join(Installation.destdir, cache_path), cache_path) After commenting out the "Pkg.SourceCacheCopyTo(Installation.destdir)" call, the crash is gone. What this does is effectively: cp -R /var/cache/zypp /mnt/var/cache/ rm -rf /var/cache/zypp ln -sf /mnt/var/cache/zypp /var/cache/zypp @zypp: Can you explain this? Is it a bug in YaST, libzypp or even libsolv? My theory is that it's very risky to copy cache files which are potentially in use by the current process. Buffers might not be flushed to disk yet, so this copies potentially incomplete files over and when they are reopened, the content is different. The crash looks very much like corruption of the repo data. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c9 --- Comment #9 from Fabian Vogt --- (In reply to Benjamin Zeller from comment #8)

A coredump file might help here though.

https://w3.suse.de/~fvogt/boo1183711core.xz This time it didn't crash in libsolv, but the memory corruption caused a "stack level too deep" error in ruby and then it aborted in malloc_printerr.

But depending on the state of libzypp, libsolv when the files are moved this might indeed be a problem. If we have open fd's to the cache it would be problematic, first the copied files might be garbage and second we'd still continue to use the old files ( since the kernel will keep deletes files around as long as a process has them opened ) or a mix of both.

Do you know in what stage that happens?

Not sure if libsolv keeps the cache files open, adding Michael Schr�der as well.

Before copy and link: lr-x------ 1 root root 64 M�r 31 10:05 27 -> /mnt/var/cache/zypp/solv/@System/solv lr-x------ 1 root root 64 M�r 31 10:00 28 -> /var/cache/zypp/solv/openSUSE-Leap-15.3-1_0/solv After copy and link: lr-x------ 1 root root 64 M�r 31 10:05 27 -> /mnt/var/cache/zypp/solv/@System/solv lr-x------ 1 root root 64 M�r 31 10:00 28 -> /var/cache/zypp/solv/openSUSE-Leap-15.3-1_0/solv (deleted) So the 15.3 solv file got unlinked and @System/solv got overwritten! Excluding @System from the copy prevents the crash. It might work to remove /mnt/var/cache/zypp/ before the copy, so that the open cache isn't modified. Or cp could be used with --remove-destination. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c11 Benjamin Zeller changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?, | |needinfo?(ma@suse.com), | |SHIP_STOPPER? --- Comment #11 from Benjamin Zeller --- (In reply to Fabian Vogt from comment #10)

...

https://w3.suse.de/~fvogt/boo1183711core.xz This time it didn't crash in libsolv, but the memory corruption caused a "stack level too deep" error in ruby and then it aborted in malloc_printerr. Yeah then it won't be too helpful :/.

...

Before copy and link:

lr-x------ 1 root root 64 M�r 31 10:05 27 -> /mnt/var/cache/zypp/solv/@System/solv lr-x------ 1 root root 64 M�r 31 10:00 28 -> /var/cache/zypp/solv/openSUSE-Leap-15.3-1_0/solv

After copy and link:

lr-x------ 1 root root 64 M�r 31 10:05 27 -> /mnt/var/cache/zypp/solv/@System/solv lr-x------ 1 root root 64 M�r 31 10:00 28 -> /var/cache/zypp/solv/openSUSE-Leap-15.3-1_0/solv (deleted)

So the 15.3 solv file got unlinked and @System/solv got overwritten! So that means libzypp already had a file in the destination cache and its overwritten? Maybe we should rethink the order in which this is happening here.

Excluding @System from the copy prevents the crash. It might work to remove /mnt/var/cache/zypp/ before the copy, so that the open cache isn't modified.

Yep, that seems to work. I opened a PR: https://github.com/yast/yast-packager/pull/561 Reassigning back to YaST.

...

Or cp could be used with --remove-destination.

I chose the explicit removal as this avoids mixing local and destination caches completely. If it's an issue that caches of other repos are also cleared, then the cp switch could be used instead.

Still a bit concerned about that, in a running transaction ripping the file away from under libzypp/libsolv sounds like a bad idea. And with running transaction I mean somewhere between loading the target and doing a commit. I would still like to get a ok from mlandres and mls about this. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c13 --- Comment #13 from Michael Schr�der --- Speaking for libsolv: you must not change the content of a opened solv file, i.e. you can only replace it with identical data. I'm not sure if that's happening here or not. It seems that libzypp/solv has already opened /mnt/var/cache/zypp/solv/@System, and then a different /var/cache/zypp/solv/@System is copyied to it. Which sounds like there is some fundamental problem somewhere else. (Btw, it's kind of weird that PkgFunctions::CopyToDir has a "recursive" option when it calls 'cp' with the '-a' option. Using "-a" implies recursive...) -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c15 Ladislav Slez�k changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P2 - High --- Comment #15 from Ladislav Slez�k --- Setting to P2 as this is a serious issue which might completely block the upgrade process. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c17 Lukas Ocilka changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lslezak@suse.com, | |sweiberg@suse.com Flags| |needinfo?(lslezak@suse.com) | |, | |needinfo?(sweiberg@suse.com | |) --- Comment #17 from Lukas Ocilka --- (In reply to Ladislav Slez�k from comment #15)

Setting to P2 as this is a serious issue which might completely block the upgrade process.

In fact, it might IMO be a P1 then. The code is the same for SLE 15 SP3. Needinfo for Stefan. But maybe Ladislav could describe more in which (all) cases this happens...? Thx. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 Guillaume GARDET changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |guillaume.gardet@arm.com -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c20 Michael Andres changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ma@suse.com) | --- Comment #20 from Michael Andres --- (In reply to Fabian Vogt from comment #16)

...

I'm not sure which solution to pick, which one is OK for libzypp. If we do not know than the safe option is to revert the cache patches completely.

Don't touch the cache of loaded repos. Unload all repos from the pool, close the target - then moving the repo cache data should be possible. Regarding the symlinks: zypp cache is directory based. If caches are updated we exchange directories. Symlinking individual files may not have the effect you want. BTW - did you consider to unload all repos/target, move the cache to /mnt/..., call ZConfig::setRepoCachePath("/mnt/...") and load the stuff again? And keep in mind that we build and re-build our caches as we need it. The fact that your workflow today does not experience a cache action does not mean it will nnever happen. Load a new libzypp via a DUD and all your symlinking may be obsolete. You can not control when and why a cache rebuild will be necessary. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c22 --- Comment #22 from Ladislav Slez�k --- I'll submit the fix (revert) to the SP3 code stream, personally I consider it as a P1 bug. Stefan (Weiberg), is that OK? -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c24 Fabian Vogt changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?, | |needinfo?(mls@suse.com), | |needinfo?(sweiberg@suse.com | |) | --- Comment #24 from Fabian Vogt --- (In reply to Ladislav Slez�k from comment #23)

The revert has been implemented in yast2-packager-4.3.21 (https://github.com/yast/yast-packager/pull/562)

Submitted in https://build.suse.de/request/show/238952 to SP3/Leap 15.3.

What's the sr# for TW? I suppose the needinfos can be cleared now. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c26 --- Comment #26 from Fabian Vogt --- (In reply to Ladislav Slez�k from comment #25)

SR for Factory: https://build.opensuse.org/request/show/883636

That one does not include the revert, it doesn't appear to be fixed in master AFAICT. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c28 Ladislav Slez�k changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |IN_PROGRESS --- Comment #28 from Ladislav Slez�k --- Ah, it was fixed in SP3/15.3, I forgot to merge to "master".... -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1183711 https://bugzilla.suse.com/show_bug.cgi?id=1183711#c29 Ladislav Slez�k changed: What |Removed |Added ---------------------------------------------------------------------------- Status|IN_PROGRESS |RESOLVED Resolution|--- |FIXED --- Comment #29 from Ladislav Slez�k --- Merged in https://github.com/yast/yast-packager/pull/565 Submitted in https://build.opensuse.org/request/show/887502 -- You are receiving this mail because: You are on the CC list for the bug.