[Bug 1186781] New: filesystem: usrmerge: cp: cannot create hard link: Invalid cross-device link
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 Bug ID: 1186781 Summary: filesystem: usrmerge: cp: cannot create hard link: Invalid cross-device link Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: x86-64 OS: openSUSE Tumbleweed Status: NEW Severity: Critical Priority: P5 - None Component: Basesystem Assignee: screening-team-bugs@suse.de Reporter: jonaski@opensuse.org QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- Last night I did a zypper dup update on my main machine, it completely broke the system. I had /, /usr, /opt and /var on separate ext4 partitions, and I think this might be the reason. It looks like the update was trying to create hard links from files in /usr to /bin, which won't work since those are different partitions. After this, the system was completely unusable, and I could not access any coreutils such as ls and cp, not even the shell. I ended up booting up a parted magic USB stick, chroot'ed the system, there I tried to do the same update, and got the following error, which I remember as identical to the one I got before: Checking for file conflicts: ................................................................................................................................................................................[done] ( 1/6461) Installing: filesystem-15.5-40.2.x86_64 ........................................................................................................................................................[error] Installation of filesystem-15.5-40.2.x86_64 failed: Error: Subprocess failed. Error: RPM failed: Make a copy of `/bin'. cp: cannot create hard link '/usr/bin.usrmerge/ksh93' to '/bin/ksh93': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/rpcinfo' to '/bin/rpcinfo': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/ps' to '/bin/ps': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/fsync' to '/bin/fsync': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/usleep' to '/bin/usleep': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/login' to '/bin/login': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/pgrep' to '/bin/pgrep': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/pkill' to '/bin/pkill': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/[' to '/bin/[': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/b2sum' to '/bin/b2sum': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/base32' to '/bin/base32': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/base64' to '/bin/base64': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/basenc' to '/bin/basenc': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/chcon' to '/bin/chcon': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/chroot' to '/bin/chroot': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/cksum' to '/bin/cksum': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/comm' to '/bin/comm': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/csplit' to '/bin/csplit': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/cut' to '/bin/cut': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/dir' to '/bin/dir': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/dircolors' to '/bin/dircolors': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/dirname' to '/bin/dirname': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/du' to '/bin/du': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/env' to '/bin/env': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/expand' to '/bin/expand': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/expr' to '/bin/expr': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/factor' to '/bin/factor': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/fmt' to '/bin/fmt': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/fold' to '/bin/fold': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/groups' to '/bin/groups': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/head' to '/bin/head': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/hostid' to '/bin/hostid': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/id' to '/bin/id': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/install' to '/bin/install': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/join' to '/bin/join': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/link' to '/bin/link': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/logname' to '/bin/logname': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/mkfifo' to '/bin/mkfifo': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/nice' to '/bin/nice': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/nl' to '/bin/nl': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/nohup' to '/bin/nohup': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/nproc' to '/bin/nproc': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/numfmt' to '/bin/numfmt': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/od' to '/bin/od': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/paste' to '/bin/paste': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/pathchk' to '/bin/pathchk': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/pinky' to '/bin/pinky': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/pr' to '/bin/pr': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/printenv' to '/bin/printenv': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/printf' to '/bin/printf': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/ptx' to '/bin/ptx': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/realpath' to '/bin/realpath': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/runcon' to '/bin/runcon': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/seq' to '/bin/seq': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/sha1sum' to '/bin/sha1sum': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/sha224sum' to '/bin/sha224sum': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/sha256sum' to '/bin/sha256sum': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/sha384sum' to '/bin/sha384sum': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/sha512sum' to '/bin/sha512sum': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/shred' to '/bin/shred': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/shuf' to '/bin/shuf': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/split' to '/bin/split': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/stdbuf' to '/bin/stdbuf': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/sum' to '/bin/sum': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/tac' to '/bin/tac': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/tail' to '/bin/tail': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/tee' to '/bin/tee': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/test' to '/bin/test': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/timeout' to '/bin/timeout': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/tr' to '/bin/tr': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/truncate' to '/bin/truncate': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/tsort' to '/bin/tsort': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/tty' to '/bin/tty': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/unexpand' to '/bin/unexpand': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/uniq' to '/bin/uniq': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/unlink' to '/bin/unlink': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/uptime' to '/bin/uptime': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/users' to '/bin/users': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/vdir' to '/bin/vdir': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/wc' to '/bin/wc': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/who' to '/bin/who': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/whoami' to '/bin/whoami': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/yes' to '/bin/yes': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/arch' to '/bin/arch': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/basename' to '/bin/basename': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/cat' to '/bin/cat': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/chgrp' to '/bin/chgrp': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/chmod' to '/bin/chmod': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/chown' to '/bin/chown': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/cp' to '/bin/cp': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/date' to '/bin/date': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/dd' to '/bin/dd': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/df' to '/bin/df': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/echo' to '/bin/echo': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/false' to '/bin/false': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/ln' to '/bin/ln': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/ls' to '/bin/ls': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/md5sum' to '/bin/md5sum': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/mkdir' to '/bin/mkdir': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/mknod' to '/bin/mknod': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/mktemp' to '/bin/mktemp': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/mv' to '/bin/mv': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/pwd' to '/bin/pwd': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/readlink' to '/bin/readlink': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/rm' to '/bin/rm': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/rmdir' to '/bin/rmdir': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/sleep' to '/bin/sleep': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/sort' to '/bin/sort': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/stat' to '/bin/stat': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/stty' to '/bin/stty': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/sync' to '/bin/sync': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/touch' to '/bin/touch': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/true' to '/bin/true': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/uname' to '/bin/uname': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/bash' to '/bin/bash': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/bashbug' to '/bin/bashbug': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/rbash' to '/bin/rbash': Invalid cross-device link cp: cannot create hard link '/usr/bin.usrmerge/sh' to '/bin/sh': Invalid cross-device link Something failed, cleaning up error: lua script failed: [string "%pretrans(filesystem-15.5-40.2.x86_64)"]:37: exit error: filesystem-15.5-40.2.x86_64: install skipped error: filesystem-15.5-40.1.x86_64: erase skipped Abort, retry, ignore? [a/r/i] (a): From what I remember I managed to complete the update there by downloading and copying coreutils and extracting it to /bin, I also created some symbolic links from /usr/bin/sh to /bin/sh, etc. However I managed to delete all the symbolic links in /sbin, although I could boot the system I was missing stuff in /sbin, and I decided to just do a reinstall instead. I have a couple of other machines with the same partition setup, and I need to know if I will run into the same issue there, and what I can do to fix it without repartitioning the disk. And if this will also be an issue when upgrading from LP 15.2 to 15.3. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 Jonas Kvinge <jonaski@opensuse.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonaski@opensuse.org -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c1 Usr Merge <ziwema2000-usrmerge@yahoo.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ziwema2000-usrmerge@yahoo.c | |om --- Comment #1 from Usr Merge <ziwema2000-usrmerge@yahoo.com> --- Same problem for me, /usr is on a separate partition. Installation of filesystem-15.5-40.2.x86_64 failed and broke the system. No executable from any location can be started, not even a file from /bin that has been copied into the user's home directory. The error message is simply "No such file or directory". However, the alleged missing files in /bin, /usr/bin, or any other directory are still accessible from a running file manager on the affected system, but cannot be started from there either (error message from Krusader "execvp: No such file or director"). Trying to start an executable script like a shell or python program results in "bad interpreter: No such file or directory", the script file is found and read, but not the executable specified to run it. I have also observed that according to the YaST log messages the packet compat-usrmerge-tools should have been installed but it is shown not installed in the YaST software management. I can not access any system log files as there is no way or me to get root access (/usr/bin/su: No such file or directory). The most important question for me: will my system still be functional after a reboot (I'm writing this report from a still running internet browser on an affected system, and I am afraid of restarting my system), or even better, is there a way to make my still running system functioning again without a restart? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c2 --- Comment #2 from Jonas Kvinge <jonaski@opensuse.org> --- It sounds like you are missing one of the core libraries in /lib or /lib64. I think the kernel will boot, but you will not be able to do anything because of missing coreutils and shell. But I don't think you will be able to fix anything without rebooting either. I think the system can be recovered using chroot: https://www.suse.com/support/kb/doc/?id=000018770 If you boot up a tool like parted magic, or a live opensuse DVD, you can simply copy, or create symbolic links the missing stuff over from /usr{bin,sbin} to /bin or /sbin. Worth to mention I could not chroot the system before I had created a symbolic link from /usr/bin/sh to /bin/sh, and I think I might have copied or created symbolic links for some core libraries too. It was really late, and I was very tired, but I think that's what I did. I was able to complete the zypper dup in the chroot by doing this. The reason it didn't work for me was because I did a mistake and removed all symbolic links in /sbin, because I was impatient and tried to copy all the binaries intended for /usr/sbin over in /sbin from the coreutils rpm, but I didn't take into consideration that there are more files from other packages required in /sbin too. I was impatient because it was late and I just wanted the system to boot right then, but I realized that it was probably a good idea to reinstall later anyway using a different partition setup. I decided to not spend much more time on it, so I just deleted the /boot, /, /usr, /opt and /var partitions, only keeping the first EFI 200MB partition and /home, then created a new ext 4 root partition (/) to replace the old, the whole point of my previous partition setup was to keep the minimal root filesystem separate from the rest of the system, but there is no point anymore, since you still need /usr now anyway. So my partition setup is now just: 1. EFI FAT 200MB, 2. 60GB ext4 system partition and 3. ext4 /home. I really dislike the idea of usrmerge, I like the old approach where there is a minimal root filesystem that can boot separately. I think it would be a good idea and fully possible if usrmerge was optional and only done on new installations, and there it should be optional too. I don't plan on using BTRFS either. It would be nice if the usrmerge could be fixed to work on systems where root and /usr are on separate partitions so that I don't have to reinstall the other computers too. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c3 --- Comment #3 from Jonas Kvinge <jonaski@opensuse.org> --- You need to copy binaries from /usr/bin to ~/bin (if they exist), or from the coreutils RPM if they don't exist there. Not /bin, the files in /bin are symbolic links. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c4 --- Comment #4 from Usr Merge <ziwema2000-usrmerge@yahoo.com> --- (In reply to Jonas Kvinge from comment #3)
You need to copy binaries from /usr/bin to ~/bin (if they exist), or from the coreutils RPM if they don't exist there. Not /bin, the files in /bin are symbolic links.
All binaries (including coreutils) and the links in /bin are present and readable in my still running file manager on the affected system. Furthermore, no binary on any partition (including the separate /home) can be executed. The error message is always "No such file or directory", but that seems to be a lie. While waiting patiently for reply by any openSUSE developer, I did some research and found that the package filesystem[1] executes /usr/libexec/convertfs, which can be found in the package compat-usrmerge[2]. This script checks if /usr is a mount point and if not, defines a flag CP_HARDLINK="-l" for cp, which creates hard links instead of copying files. However, when making a copy of /bin, the script executes 'cp -ax -l', not 'cp -ax $CP_HARDLINK' as I would expect. This could explain why the installation fails, but not why I can't execute binaries from any location of my system anymore. [1] https://build.opensuse.org/package/view_file/openSUSE:Factory/filesystem/fil... [2] https://build.opensuse.org/package/view_file/openSUSE:Factory/compat-usrmerg... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c5 --- Comment #5 from Jonas Kvinge <jonaski@opensuse.org> --- It's because of a missing core library. Do: ldd /bin/ls Check that the shown so's exist (except for linux-vdso.so.1). -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c6 --- Comment #6 from Jonas Kvinge <jonaski@opensuse.org> --- You probably have to do ldd on a different machine to get the list of the files. I think maybe the one I was missing was /lib64/ld-linux-x86-64.so.2 If it exists in /usr/lib64 and not /lib64, create a symbolic link. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c7 --- Comment #7 from Usr Merge <ziwema2000-usrmerge@yahoo.com> --- (In reply to Jonas Kvinge from comment #6)
I think maybe the one I was missing was /lib64/ld-linux-x86-64.so.2 If it exists in /usr/lib64 and not /lib64, create a symbolic link.
I can confirm that /lib64 contains two dead links ld-lsb-x86-64.so.{2,3} -> ld-linux-x86-64.so.2 both pointing to the same missing target, while /usr/lib64 contains the working links ld-lsb-x86-64.so.3 -> ld-linux-x86-64.so.2 -> ld-2.33.so If this is the only reason for the broken system, I'm tempted to restart my system into rescue mode and fix the links, but I would really like to see some explanation from the developers before I try this. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c8 --- Comment #8 from Usr Merge <ziwema2000-usrmerge@yahoo.com> --- Is bug 1186730[1] related? It also reports a problem with ld-linux-x86-64.so.2 after installing filesystem-15.5-40.2.x86_64. [1] https://bugzilla.opensuse.org/show_bug.cgi?id=1186730 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c9 --- Comment #9 from Usr Merge <ziwema2000-usrmerge@yahoo.com> --- (In reply to Usr Merge from comment #7)
I can confirm that /lib64 contains two dead links ld-lsb-x86-64.so.{2,3} -> ld-linux-x86-64.so.2 both pointing to the same missing target, while /usr/lib64 contains the working links ld-lsb-x86-64.so.3 -> ld-linux-x86-64.so.2 -> ld-2.33.so
I found that the only reference to ld-linux-x86-64 in /etc/ld.so.cache is /usr/lib64/ld-linux-x86-64.so.2, so do the broken links in /lib64 even matter? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c10 --- Comment #10 from Usr Merge <ziwema2000-usrmerge@yahoo.com> --- (In reply to Usr Merge from comment #9)
I found that the only reference to ld-linux-x86-64 in /etc/ld.so.cache is /usr/lib64/ld-linux-x86-64.so.2, so do the broken links in /lib64 even matter?
They do seem to matter. I can use /usr/lib64/ld-linux-x86-64.so.2 to run executables directly: $ /usr/lib64/ld-linux-x86-64.so.2 /usr/bin/ls Here is how I get a list of libraries used by ls: $ /usr/lib64/ld-linux-x86-64.so.2 /usr/bin/bash ~/bin/ldd /bin/ls linux-vdso.so.1 (0x00007ffd3c56d000) libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f54e22db000) libcap.so.2 => /usr/lib64/libcap.so.2 (0x00007f54e22d0000) libc.so.6 => /usr/lib64/libc.so.6 (0x00007f54e2101000) libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 (0x00007f54e204b000) libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f54e2044000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f54e2360000) where ~/bin/ldd is the shell script /usr/bin/ldd with /usr/lib64/ld-linux-x86-64.so.2 appended to the variable RTLDLIST. Note the difference to the example in the man page of ldd, which has /lib64/ld-linux-x86-64.so.2 without a "=>". This trick allows me to start user programs, but unfortunately, I can still get no root access for reasons I don't understand: $ /usr/lib64/ld-linux-x86-64.so.2 /usr/bin/su Password: su: Authentication service cannot retrieve authentication info $ /usr/lib64/ld-linux-x86-64.so.2 /usr/bin/sudo sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 Usr Merge <ziwema2000-usrmerge@yahoo.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|screening-team-bugs@suse.de |lnussel@suse.com -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c11 --- Comment #11 from Usr Merge <ziwema2000-usrmerge@yahoo.com> --- (In reply to Usr Merge from comment #7)
I can confirm that /lib64 contains two dead links ld-lsb-x86-64.so.{2,3} -> ld-linux-x86-64.so.2 both pointing to the same missing target, while /usr/lib64 contains the working links ld-lsb-x86-64.so.3 -> ld-linux-x86-64.so.2 -> ld-2.33.so
Here is a hint about the situation before the catastrophe: $ /usr/lib64/ld-2.33.so /usr/bin/locate lib64/ld- /lib64/ld-2.33.so /lib64/ld-linux-x86-64.so.2 /lib64/ld-lsb-x86-64.so.2 /lib64/ld-lsb-x86-64.so.3 $ /usr/lib64/ld-2.33.so /usr/bin/ls -l /var/lib/mlocate/mlocate.db -rw-r--r-- 1 nobody nobody 40964796 Jun 2 00:04 /var/lib/mlocate/mlocate.db -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c12 --- Comment #12 from Usr Merge <ziwema2000-usrmerge@yahoo.com> --- According to the log messages by Yast, there were seven more packages installed after the (ignored) failed installation of filesystem-15.5-40.2.x86_64 and before the update was stopped by the broken system unable to execute any binaries. The package glibc-2.33-6.13 was the next to last, which is probably relevant because it provides the dynamic linker. This looks like the system was actually broken by the installation of glibc on an un-UsrMerged system. Most important question for me: do I have to expect other issues than the broken symbolic links in /lib64 to the dynamic linker? I still haven't rebooted the system and I'm concerned because I can't get root access by starting su or sudo as argument of /usr/lib64/ld-2.33.so. I'm afraid the answer to my question is positive because the location of other glibc libraries on my system right now is also different from before the installation according to the 'locate' utility. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c13 --- Comment #13 from Usr Merge <ziwema2000-usrmerge@yahoo.com> --- All that was necessary to get back what appears to be a completely normal functioning system is a symbolic link in /lib64 to the dynamic linker in /usr/lib64. Under a rescue system, relative to the mounted root of the broken system (no chroot needed): $ cd lib64 $ ln -s ../usr/lib64/ld-2.33.so ld-linux-x86-64.so.2 Note that /lib64/ld-linux-x86-64.so.2 is the missing target of the two links /lib64/ld-lsb-x86-64.so.{2,3}, which showed up after the failed update. If I had been running a root terminal when the accident happened, the following might have fixed it without even booting into a rescue system: $ cd /lib64 $ /usr/lib64/ld-2.33.so ln -s ../usr/lib64/ld-2.33.so ld-linux-x86-64.so.2 I tried executing 'su' and 'sudo' as arguments of /usr/lib64/ld-2.33.so on the now functioning system and they still fail with the same errors as mentioned in comment #10, so it doesn't seem to be meant to work this way, although there is a report that says otherwise: https://programmersought.com/article/20126230768/. Here is a summary of what I have learned about this issue: The installation of filesystem-15.5-40.2.x86_64 fails when making a copy of /bin, possibly because of 'cp -ax -l' instead of 'cp -ax $CP_HARDLINK' in /usr/libexec/convertfs from the package compat-usrmerge-tools. This means the system has not been usrmerged. After the failed installation of filesystem has been ignored, the subsequently installed package glibc-2.33-6.13 breaks the system because of a missing symbolic link to the dynamic linker, which has been moved from /lib64 to /usr/lib64. Trying to run executable binaries now fails with the error message "No such file or directory". It is possible to run them as arguments of /usr/lib64/ld-2.33.so, though. Any commentary and advise by openSUSE maintainers are welcome. I am interested if I will be able to update my system without the risk of breaking it again, in particular if the package filesystem will correctly usrmerge on the next attempt. (In reply to Usr Merge from comment #1)
I have also observed that according to the YaST log messages the packet compat-usrmerge-tools should have been installed but it is shown not installed in the YaST software management.
After the fix, YaST now correctly shows the installed packages and their file lists. (In reply to Usr Merge from comment #10)
Here is how I get a list of libraries used by ls:
$ /usr/lib64/ld-linux-x86-64.so.2 /usr/bin/bash ~/bin/ldd /bin/ls
This could have be done simpler with: $ /usr/lib64/ld-linux-x86-64.so.2 --list /bin/ls -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c14 --- Comment #14 from Ludwig Nussel <lnussel@suse.com> --- (In reply to Usr Merge from comment #4)
While waiting patiently for reply by any openSUSE developer, I did some research and found that the package filesystem[1] executes /usr/libexec/convertfs, which can be found in the package compat-usrmerge[2]. This script checks if /usr is a mount point and if not, defines a flag CP_HARDLINK="-l" for cp, which creates hard links instead of copying files. However, when making a copy of /bin, the script executes 'cp -ax -l', not 'cp -ax $CP_HARDLINK' as I would expect.
Yes, that is the bug. Not sure how it slipped though as I orginally tested the split /usr setup. Sorry for that :-( -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c15 Ludwig Nussel <lnussel@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |IN_PROGRESS --- Comment #15 from Ludwig Nussel <lnussel@suse.com> --- (In reply to Usr Merge from comment #13)
After the failed installation of filesystem has been ignored, the subsequently installed package glibc-2.33-6.13 breaks the system because of a missing symbolic link to the dynamic linker, which has been moved from
*argh* don't do that! Proceeding after the filesystem package fail will totally screw up your system as you found out :-( Not sure what's the best way to recover from there. One way would be to use the rescue system to copy all files glibc has in /usr/lib64 to /lib64. Verify zypper and rpm work. Remove the offending "-l" parameter of cp in /usr/libexec/convertfs Then reboot and retry the zypper dup. In case zypper reinstalls compat-usrmerge-tools you have to fix /usr/libexec/convertfs again. Do not ignore errors from the filesystem package again :-) Anyway, fix to Factory on the way. I've also added a warning to not ignore errors. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 Ludwig Nussel <lnussel@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P1 - Urgent -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c16 --- Comment #16 from Ludwig Nussel <lnussel@suse.com> --- As soon as the fix landed in Tumbleweed it would also be possible to use the installation DVD to start an upgrade from there to fix a broken system. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c17 --- Comment #17 from OBSbugzilla Bot <bwiedemann+obsbugzillabot@suse.com> --- This is an autogenerated message for OBS integration: This bug (1186781) was mentioned in https://build.opensuse.org/request/show/898009 Factory / compat-usrmerge -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c18 --- Comment #18 from Usr Merge <ziwema2000-usrmerge@yahoo.com> --- (In reply to Ludwig Nussel from comment #15)
*argh* don't do that! Proceeding after the filesystem package fail will totally screw up your system as you found out :-(
Luckily, "totally screw up" is an overstatement in my case, as a single symbolic link made it working again. Now, maybe that's not practicable, but shouldn't packages like glibc that depend on an usrmerged system refuse to install on a non-usrmerged system?
Do not ignore errors from the filesystem package again :-)
I will add this to the list of exceptions to my golden rule for computer users: always ignore errors until you can't. (In reply to Ludwig Nussel from comment #16)
As soon as the fix landed in Tumbleweed it would also be possible to use the installation DVD to start an upgrade from there to fix a broken system.
I'm planning to do it from the online repository on my makeshift fixed system, unless someone knows a reason why this would be insane. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 Ludwig Nussel <lnussel@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1029961 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c20 --- Comment #20 from Ludwig Nussel <lnussel@suse.com> --- you are right, the params were the wrong way around. :facepalm: -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c21 --- Comment #21 from OBSbugzilla Bot <bwiedemann+obsbugzillabot@suse.com> --- This is an autogenerated message for OBS integration: This bug (1186781) was mentioned in https://build.opensuse.org/request/show/900335 Factory / compat-usrmerge -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c22 --- Comment #22 from Markus Elfring <Markus.Elfring@web.de> --- (In reply to Usr Merge from comment #13)
I am interested if I will be able to update my system without the risk of breaking it again, in particular if the package filesystem will correctly usrmerge on the next attempt.
I became curious on corresponding software evolution. See also: * YaST2 crashes when analyzing fstab files of systems to upgrade with no entries https://bugzilla.opensuse.org/show_bug.cgi?id=1186895 * https://en.opensuse.org/openSUSE:Usr_merge -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c23 Markus Elfring <Markus.Elfring@web.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |Markus.Elfring@web.de --- Comment #23 from Markus Elfring <Markus.Elfring@web.de> --- (In reply to Usr Merge from comment #19)
My /usr partition runs out of space while copying files to /usr/{bin,sbin,lib,lib64}.usrmerge, ���
I became curious how the storage requirements will be clarified further (also according to remaining preferences for separate ���usr��� partitions). -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c24 --- Comment #24 from Usr Merge <ziwema2000-usrmerge@yahoo.com> --- (In reply to Markus Elfring from comment #23)
(In reply to Usr Merge from comment #19)
My /usr partition runs out of space while copying files to /usr/{bin,sbin,lib,lib64}.usrmerge, ���
I became curious how the storage requirements will be clarified further (also according to remaining preferences for separate ���usr��� partitions).
If you want to know how much additional space on the /usr partition is needed, it should be enough to do # du -sch /bin /sbin /lib /lib64 before the UsrMerge. It shows a total of 471M for my system, but the real number should by greater because some of my files have already been moved from / to /usr by the installation of the package 'glibc' after the failed UsrMerge attempt by the package 'filesystem'. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c25 --- Comment #25 from Markus Elfring <Markus.Elfring@web.de> --- (In reply to Usr Merge from comment #24)
���, but the real number should by greater because some of my files have already been moved from / to /usr by the installation of the package 'glibc' after the failed UsrMerge attempt by the package 'filesystem'.
How will the chances evolve that any corresponding package variants can keep the involved software dependencies consistent? Would you like to achieve any documentation improvements accordingly? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c26 Ludwig Nussel <lnussel@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|IN_PROGRESS |RESOLVED Resolution|--- |FIXED --- Comment #26 from Ludwig Nussel <lnussel@suse.com> --- The fix for that now landed in Factory, will be in next Snapshot. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1186781 http://bugzilla.opensuse.org/show_bug.cgi?id=1186781#c27 --- Comment #27 from Usr Merge <ziwema2000-usrmerge@yahoo.com> --- For the record: with compat-usrmerge-tools-84.87-3.1.x86_64 from snapshot 20210620, installation of the package filesystem went through without problems on my system with a separate /usr partition, despite the problems and temporary fixes due to a previously failed installation described in earlier comments (some libraries were not found which could be fixed by symbolic links in /lib64 to libraries in /usr/lib64). -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com