Tumbleweed dup, zypper ps shows deleted files in use even after reboot
Hi all, I have a Tumbleweed machine which is kept up to date via zypper ref; zypper dup on a regular basis. About 4 or so updates I ago, I noticed that zypper ps shows multiple services using deleted files and needing to be restarted. The above has persisted despite many reboots and about 4 more updates. Using lsof, I do not see the files and services using deleted files that zypper ps keeps reporting. It looks like zypper ps always reports the same set of services using the same set of deleted files after each reboot. Anyone have any ideas why zypper ps would be doing this? Thank you in advance for your help, --Moby
So, given a PID reported by 'zypper ps', 'lsof -p PID' does not report any files? Is the process even shown by 'ps -p PID' ? -- Robert Webb On Wednesday, July 20, 2022, 03:53:50 PM PDT, Moby <moby@mobsternet.com> wrote: Hi all, I have a Tumbleweed machine which is kept up to date via zypper ref; zypper dup on a regular basis. About 4 or so updates I ago, I noticed that zypper ps shows multiple services using deleted files and needing to be restarted. The above has persisted despite many reboots and about 4 more updates. Using lsof, I do not see the files and services using deleted files that zypper ps keeps reporting. It looks like zypper ps always reports the same set of services using the same set of deleted files after each reboot.
Thanks Robert. lsof -p PID shows a number of files, none of which show as deleted. All are valid files that exist, and have not been replaced as I ran the test right after a machine reboot (multiple reboots). For example. zypper ps reports PID 30721, sshd, using file /usr/lib64/security/pam_nologin.so. It does not indicated any files as having been deleted etc.and that file exists. Regards, --Moby On 7/20/2022 20:53, Robert Webb wrote:
So, given a PID reported by 'zypper ps', 'lsof -p PID' does not report any files?
Is the process even shown by 'ps -p PID' ?
On 21.07.2022 01:30, Moby wrote:
Hi all, I have a Tumbleweed machine which is kept up to date via zypper ref; zypper dup on a regular basis. About 4 or so updates I ago, I noticed that zypper ps shows multiple services using deleted files and needing to be restarted.
The above has persisted despite many reboots and about 4 more updates. Using lsof, I do not see the files and services using deleted files that zypper ps keeps reporting. It looks like zypper ps always reports the same set of services using the same set of deleted files after each reboot.
Anyone have any ideas why zypper ps would be doing this?
Not without actual zypper output.
Thanks Andrei Below is the output of zypper ps: zypper ps | more The following running processes use deleted files: PID | PPID | UID | User | Command | Service | Files ------+-------+-----+------------------+-------------------+-------------------+---------------------------------------- ------------------------------------------ 1 | 0 | 0 | root | systemd | | /usr/lib64/libpam.so.0.85.1 (path dev=0 ,35) | | | | | | /usr/lib64/libkmod.so.2.4.0 (path dev=0 ,35) . . 20382| 1264 | 0 | root | sshd | | /usr/lib64/security/pam_nologin.so (path dev=0,35) lsof -p 20382 | grep -i delete produces no output. The above is after a fresh reboot. Regards, --Moby On 7/21/2022 01:17, Andrei Borzenkov wrote:
On 21.07.2022 01:30, Moby wrote:
Hi all, I have a Tumbleweed machine which is kept up to date via zypper ref; zypper dup on a regular basis. About 4 or so updates I ago, I noticed that zypper ps shows multiple services using deleted files and needing to be restarted.
The above has persisted despite many reboots and about 4 more updates. Using lsof, I do not see the files and services using deleted files that zypper ps keeps reporting. It looks like zypper ps always reports the same set of services using the same set of deleted files after each reboot.
Anyone have any ideas why zypper ps would be doing this? Not without actual zypper output.
Moby, the zypper man page says that this: 'zypper ps --debugFile <filename>' provides useful information. -- Robert Webb
Thanks Robert. The PIDs reported by zypper ps change after each reboot. I rebooted the machine again and ran zypper ps --debugFile zypper-dbg1 zypper ps output on the screen shows PID of 1270 for sshd using /usr/lib64/libaudit.so.1.0.0 (path dev=0,35). That file is not deleted. grep'ing zypper-dbg1 for 1270, and then looking for the line containing libaudit.so.1.0.0 shows: fmem^@tREG^@n/usr/lib64/libaudit.so.1.0.0 (path dev=0,35)^@ I see nothing but lines like the above in zypper-dbg1 for each PID and the files it is using, not sure if adds anything to what zypper ps outputs to the screen. Regards, --Moby On 7/21/2022 15:21, Robert Webb wrote:
Moby, the zypper man page says that this: 'zypper ps --debugFile <filename>' provides useful information.
On 22.07.2022 00:52, Moby wrote:
Thanks Robert.
The PIDs reported by zypper ps change after each reboot. I rebooted the machine again and ran zypper ps --debugFile zypper-dbg1
zypper ps output on the screen shows PID of 1270 for sshd using /usr/lib64/libaudit.so.1.0.0 (path dev=0,35). That file is not deleted.
grep'ing zypper-dbg1 for 1270, and then looking for the line containing libaudit.so.1.0.0 shows:
fmem^@tREG^@n/usr/lib64/libaudit.so.1.0.0 (path dev=0,35)^@
See the LSOF FAQ (https://github.com/lsof-org/lsof/blob/bcc125b7c2a160da019dd8089b999bc6d34866...): 10.4.1 What do ``path dev=xxx'' and ``path inode=yyy'' mean in the NAME column of Linux ``mem'' file types? ... When device and inode mismatches occur, lsof suppresses the reporting of link count and size. ... and the lack of link count is interpreted by zypper as indication that file was deleted. Show full output of lsof -p 1270 cat /proc/1270/maps stat /usr/lib64/libaudit.so.1.0.0 cat /proc/self/mountinfo cat /proc/1270/mountinfo
Thanks Andrei. Here is the output you requested. I will have to google around to see how to detect device and inode mismatches that you mention before I can make heads or tails of it: lsof -p 1270 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sshd 1270 root cwd DIR 0,35 202 256 / sshd 1270 root rtd DIR 0,35 202 256 / sshd 1270 root txt REG 0,35 954120 39515208 /usr/sbin/sshd sshd 1270 root mem REG 0,32 39515208 /usr/sbin/sshd (path dev=0,35) sshd 1270 root mem REG 0,32 10018537 /var/lib/nscd/passwd (path dev=0,35) sshd 1270 root mem REG 0,32 39054749 /usr/lib64/libgpg-error.so.0.33.0 (path dev=0,35) sshd 1270 root mem REG 0,32 39471728 /usr/lib64/libresolv.so.2 (path dev=0,35) sshd 1270 root mem REG 0,32 39244436 /usr/lib64/libkeyutils.so.1.10 (path dev=0,35) sshd 1270 root mem REG 0,32 39219676 /usr/lib64/libkrb5support.so.0.1 (path dev=0,35) sshd 1270 root mem REG 0,32 39219670 /usr/lib64/libk5crypto.so.3.1 (path dev=0,35) sshd 1270 root mem REG 0,32 39455014 /usr/lib64/libpcre2-8.so.0.11.0 (path dev=0,35) sshd 1270 root mem REG 0,32 39054551 /usr/lib64/liblz4.so.1.9.3 (path dev=0,35) sshd 1270 root mem REG 0,32 39053974 /usr/lib64/libzstd.so.1.5.2 (path dev=0,35) sshd 1270 root mem REG 0,32 39054546 /usr/lib64/liblzma.so.5.2.5 (path dev=0,35) sshd 1270 root mem REG 0,32 39055729 /usr/lib64/libgcrypt.so.20.3.4 (path dev=0,35) sshd 1270 root mem REG 0,32 39055044 /usr/lib64/libcap.so.2.64 (path dev=0,35) sshd 1270 root mem REG 0,32 39054900 /usr/lib64/libeconf.so.0.4.6 (path dev=0,35) sshd 1270 root mem REG 0,32 39471716 /usr/lib64/libc.so.6 (path dev=0,35) sshd 1270 root mem REG 0,32 38770934 /usr/lib64/libcom_err.so.2.1 (path dev=0,35) sshd 1270 root mem REG 0,32 39219675 /usr/lib64/libkrb5.so.3.3 (path dev=0,35) sshd 1270 root mem REG 0,32 39219668 /usr/lib64/libgssapi_krb5.so.2.2 (path dev=0,35) sshd 1270 root mem REG 0,32 39471740 /usr/lib64/libselinux.so.1 (path dev=0,35) sshd 1270 root mem REG 0,32 39054986 /usr/lib64/libcrypt.so.1.1.0 (path dev=0,35) sshd 1270 root mem REG 0,32 39053979 /usr/lib64/libz.so.1.2.11 (path dev=0,35) sshd 1270 root mem REG 0,32 39362770 /usr/lib64/libcrypto.so.1.1 (path dev=0,35) sshd 1270 root mem REG 0,32 39445620 /usr/lib64/libsystemd.so.0.34.0 (path dev=0,35) sshd 1270 root mem REG 0,32 39363195 /usr/lib64/libpam.so.0.85.1 (path dev=0,35) sshd 1270 root mem REG 0,32 39055104 /usr/lib64/libaudit.so.1.0.0 (path dev=0,35) sshd 1270 root mem REG 0,32 39471713 /usr/lib64/ld-linux-x86-64.so.2 (path dev=0,35) sshd 1270 root 0r CHR 1,3 0t0 4 /dev/null sshd 1270 root 1u unix 0xffff96ca8756f2c0 0t0 19270 type=STREAM (CONNECTED) sshd 1270 root 2u unix 0xffff96ca8756f2c0 0t0 19270 type=STREAM (CONNECTED) sshd 1270 root 3u IPv4 19278 0t0 TCP *:ssh (LISTEN) sshd 1270 root 4u IPv6 19280 0t0 TCP *:ssh (LISTEN) cat /proc/1270/maps 55f69ac1b000-55f69ac27000 r--p 00000000 00:20 39515208 /usr/sbin/sshd 55f69ac27000-55f69acb6000 r-xp 0000c000 00:20 39515208 /usr/sbin/sshd 55f69acb6000-55f69ad00000 r--p 0009b000 00:20 39515208 /usr/sbin/sshd 55f69ad00000-55f69ad04000 r--p 000e4000 00:20 39515208 /usr/sbin/sshd 55f69ad04000-55f69ad05000 rw-p 000e8000 00:20 39515208 /usr/sbin/sshd 55f69ad05000-55f69ad08000 rw-p 00000000 00:00 0 55f69ae33000-55f69ae74000 rw-p 00000000 00:00 0 [heap] 7f671e210000-7f671e245000 r--s 00000000 00:20 10018537 /var/lib/nscd/passwd 7f671e245000-7f671e24a000 rw-p 00000000 00:00 0 7f671e24a000-7f671e24f000 r--p 00000000 00:20 39054749 /usr/lib64/libgpg-error.so.0.33.0 7f671e24f000-7f671e264000 r-xp 00005000 00:20 39054749 /usr/lib64/libgpg-error.so.0.33.0 7f671e264000-7f671e26e000 r--p 0001a000 00:20 39054749 /usr/lib64/libgpg-error.so.0.33.0 7f671e26e000-7f671e26f000 r--p 00023000 00:20 39054749 /usr/lib64/libgpg-error.so.0.33.0 7f671e26f000-7f671e270000 rw-p 00024000 00:20 39054749 /usr/lib64/libgpg-error.so.0.33.0 7f671e270000-7f671e274000 r--p 00000000 00:20 39471728 /usr/lib64/libresolv.so.2 7f671e274000-7f671e27d000 r-xp 00004000 00:20 39471728 /usr/lib64/libresolv.so.2 7f671e27d000-7f671e27f000 r--p 0000d000 00:20 39471728 /usr/lib64/libresolv.so.2 7f671e27f000-7f671e280000 ---p 0000f000 00:20 39471728 /usr/lib64/libresolv.so.2 7f671e280000-7f671e281000 r--p 0000f000 00:20 39471728 /usr/lib64/libresolv.so.2 7f671e281000-7f671e282000 rw-p 00010000 00:20 39471728 /usr/lib64/libresolv.so.2 7f671e282000-7f671e286000 rw-p 00000000 00:00 0 7f671e286000-7f671e288000 r--p 00000000 00:20 39244436 /usr/lib64/libkeyutils.so.1.10 7f671e288000-7f671e28a000 r-xp 00002000 00:20 39244436 /usr/lib64/libkeyutils.so.1.10 7f671e28a000-7f671e28b000 r--p 00004000 00:20 39244436 /usr/lib64/libkeyutils.so.1.10 7f671e28b000-7f671e28c000 r--p 00004000 00:20 39244436 /usr/lib64/libkeyutils.so.1.10 7f671e28c000-7f671e28d000 rw-p 00005000 00:20 39244436 /usr/lib64/libkeyutils.so.1.10 7f671e28d000-7f671e291000 r--p 00000000 00:20 39219676 /usr/lib64/libkrb5support.so.0.1 7f671e291000-7f671e298000 r-xp 00004000 00:20 39219676 /usr/lib64/libkrb5support.so.0.1 7f671e298000-7f671e29b000 r--p 0000b000 00:20 39219676 /usr/lib64/libkrb5support.so.0.1 7f671e29b000-7f671e29c000 r--p 0000d000 00:20 39219676 /usr/lib64/libkrb5support.so.0.1 7f671e29c000-7f671e29d000 rw-p 0000e000 00:20 39219676 /usr/lib64/libkrb5support.so.0.1 7f671e29d000-7f671e2a2000 r--p 00000000 00:20 39219670 /usr/lib64/libk5crypto.so.3.1 7f671e2a2000-7f671e2ad000 r-xp 00005000 00:20 39219670 /usr/lib64/libk5crypto.so.3.1 7f671e2ad000-7f671e2b1000 r--p 00010000 00:20 39219670 /usr/lib64/libk5crypto.so.3.1 7f671e2b1000-7f671e2b3000 r--p 00013000 00:20 39219670 /usr/lib64/libk5crypto.so.3.1 7f671e2b3000-7f671e2b4000 rw-p 00015000 00:20 39219670 /usr/lib64/libk5crypto.so.3.1 7f671e2b4000-7f671e2b7000 r--p 00000000 00:20 39455014 /usr/lib64/libpcre2-8.so.0.11.0 7f671e2b7000-7f671e338000 r-xp 00003000 00:20 39455014 /usr/lib64/libpcre2-8.so.0.11.0 7f671e338000-7f671e364000 r--p 00084000 00:20 39455014 /usr/lib64/libpcre2-8.so.0.11.0 7f671e364000-7f671e365000 ---p 000b0000 00:20 39455014 /usr/lib64/libpcre2-8.so.0.11.0 7f671e365000-7f671e366000 r--p 000b0000 00:20 39455014 /usr/lib64/libpcre2-8.so.0.11.0 7f671e366000-7f671e367000 rw-p 000b1000 00:20 39455014 /usr/lib64/libpcre2-8.so.0.11.0 7f671e367000-7f671e36a000 r--p 00000000 00:20 39054551 /usr/lib64/liblz4.so.1.9.3 7f671e36a000-7f671e385000 r-xp 00003000 00:20 39054551 /usr/lib64/liblz4.so.1.9.3 7f671e385000-7f671e388000 r--p 0001e000 00:20 39054551 /usr/lib64/liblz4.so.1.9.3 7f671e388000-7f671e389000 r--p 00020000 00:20 39054551 /usr/lib64/liblz4.so.1.9.3 7f671e389000-7f671e38a000 rw-p 00021000 00:20 39054551 /usr/lib64/liblz4.so.1.9.3 7f671e38a000-7f671e390000 r--p 00000000 00:20 39053974 /usr/lib64/libzstd.so.1.5.2 7f671e390000-7f671e42b000 r-xp 00006000 00:20 39053974 /usr/lib64/libzstd.so.1.5.2 7f671e42b000-7f671e43a000 r--p 000a1000 00:20 39053974 /usr/lib64/libzstd.so.1.5.2 7f671e43a000-7f671e43b000 ---p 000b0000 00:20 39053974 /usr/lib64/libzstd.so.1.5.2 7f671e43b000-7f671e43c000 r--p 000b0000 00:20 39053974 /usr/lib64/libzstd.so.1.5.2 7f671e43c000-7f671e43d000 rw-p 000b1000 00:20 39053974 /usr/lib64/libzstd.so.1.5.2 7f671e43d000-7f671e43f000 rw-p 00000000 00:00 0 7f671e43f000-7f671e443000 r--p 00000000 00:20 39054546 /usr/lib64/liblzma.so.5.2.5 7f671e443000-7f671e464000 r-xp 00004000 00:20 39054546 /usr/lib64/liblzma.so.5.2.5 7f671e464000-7f671e46f000 r--p 00025000 00:20 39054546 /usr/lib64/liblzma.so.5.2.5 7f671e46f000-7f671e470000 ---p 00030000 00:20 39054546 /usr/lib64/liblzma.so.5.2.5 7f671e470000-7f671e471000 r--p 00030000 00:20 39054546 /usr/lib64/liblzma.so.5.2.5 7f671e471000-7f671e472000 rw-p 00031000 00:20 39054546 /usr/lib64/liblzma.so.5.2.5 7f671e472000-7f671e481000 r--p 00000000 00:20 39055729 /usr/lib64/libgcrypt.so.20.3.4 7f671e481000-7f671e568000 r-xp 0000f000 00:20 39055729 /usr/lib64/libgcrypt.so.20.3.4 7f671e568000-7f671e5a7000 r--p 000f6000 00:20 39055729 /usr/lib64/libgcrypt.so.20.3.4 7f671e5a7000-7f671e5aa000 r--p 00134000 00:20 39055729 /usr/lib64/libgcrypt.so.20.3.4 7f671e5aa000-7f671e5b0000 rw-p 00137000 00:20 39055729 /usr/lib64/libgcrypt.so.20.3.4 7f671e5b0000-7f671e5b1000 rw-p 00000000 00:00 0 7f671e5b1000-7f671e5b4000 r--p 00000000 00:20 39055044 /usr/lib64/libcap.so.2.64 7f671e5b4000-7f671e5b9000 r-xp 00003000 00:20 39055044 /usr/lib64/libcap.so.2.64 7f671e5b9000-7f671e5bb000 r--p 00008000 00:20 39055044 /usr/lib64/libcap.so.2.64 7f671e5bb000-7f671e5bc000 ---p 0000a000 00:20 39055044 /usr/lib64/libcap.so.2.64 7f671e5bc000-7f671e5bd000 r--p 0000a000 00:20 39055044 /usr/lib64/libcap.so.2.64 7f671e5bd000-7f671e5be000 rw-p 0000b000 00:20 39055044 /usr/lib64/libcap.so.2.64 7f671e5be000-7f671e5c0000 r--p 00000000 00:20 39054900 /usr/lib64/libeconf.so.0.4.6 7f671e5c0000-7f671e5c5000 r-xp 00002000 00:20 39054900 /usr/lib64/libeconf.so.0.4.6 7f671e5c5000-7f671e5c7000 r--p 00007000 00:20 39054900 /usr/lib64/libeconf.so.0.4.6 7f671e5c7000-7f671e5c8000 r--p 00008000 00:20 39054900 /usr/lib64/libeconf.so.0.4.6 7f671e5c8000-7f671e5c9000 rw-p 00009000 00:20 39054900 /usr/lib64/libeconf.so.0.4.6 7f671e5c9000-7f671e5ca000 rw-p 00000000 00:00 0 7f671e5ca000-7f671e609000 r--p 00000000 00:20 39471716 /usr/lib64/libc.so.6 7f671e609000-7f671e78e000 r-xp 0003f000 00:20 39471716 /usr/lib64/libc.so.6 7f671e78e000-7f671e7e2000 r--p 001c4000 00:20 39471716 /usr/lib64/libc.so.6 7f671e7e2000-7f671e7e3000 ---p 00218000 00:20 39471716 /usr/lib64/libc.so.6 7f671e7e3000-7f671e7e6000 r--p 00218000 00:20 39471716 /usr/lib64/libc.so.6 7f671e7e6000-7f671e7ef000 rw-p 0021b000 00:20 39471716 /usr/lib64/libc.so.6 7f671e7ef000-7f671e7fc000 rw-p 00000000 00:00 0 7f671e7fc000-7f671e7fe000 r--p 00000000 00:20 38770934 /usr/lib64/libcom_err.so.2.1 7f671e7fe000-7f671e7ff000 r-xp 00002000 00:20 38770934 /usr/lib64/libcom_err.so.2.1 7f671e7ff000-7f671e800000 r--p 00003000 00:20 38770934 /usr/lib64/libcom_err.so.2.1 7f671e800000-7f671e801000 r--p 00003000 00:20 38770934 /usr/lib64/libcom_err.so.2.1 7f671e801000-7f671e802000 rw-p 00004000 00:20 38770934 /usr/lib64/libcom_err.so.2.1 7f671e802000-7f671e804000 rw-p 00000000 00:00 0 7f671e804000-7f671e828000 r--p 00000000 00:20 39219675 /usr/lib64/libkrb5.so.3.3 7f671e828000-7f671e886000 r-xp 00024000 00:20 39219675 /usr/lib64/libkrb5.so.3.3 7f671e886000-7f671e8c3000 r--p 00082000 00:20 39219675 /usr/lib64/libkrb5.so.3.3 7f671e8c3000-7f671e8c4000 ---p 000bf000 00:20 39219675 /usr/lib64/libkrb5.so.3.3 7f671e8c4000-7f671e8d2000 r--p 000bf000 00:20 39219675 /usr/lib64/libkrb5.so.3.3 7f671e8d2000-7f671e8d4000 rw-p 000cd000 00:20 39219675 /usr/lib64/libkrb5.so.3.3 7f671e8d4000-7f671e8e1000 r--p 00000000 00:20 39219668 /usr/lib64/libgssapi_krb5.so.2.2 7f671e8e1000-7f671e918000 r-xp 0000d000 00:20 39219668 /usr/lib64/libgssapi_krb5.so.2.2 7f671e918000-7f671e924000 r--p 00044000 00:20 39219668 /usr/lib64/libgssapi_krb5.so.2.2 7f671e924000-7f671e925000 ---p 00050000 00:20 39219668 /usr/lib64/libgssapi_krb5.so.2.2 7f671e925000-7f671e927000 r--p 00050000 00:20 39219668 /usr/lib64/libgssapi_krb5.so.2.2 7f671e927000-7f671e928000 rw-p 00052000 00:20 39219668 /usr/lib64/libgssapi_krb5.so.2.2 7f671e928000-7f671e930000 r--p 00000000 00:20 39471740 /usr/lib64/libselinux.so.1 7f671e930000-7f671e94b000 r-xp 00008000 00:20 39471740 /usr/lib64/libselinux.so.1 7f671e94b000-7f671e953000 r--p 00023000 00:20 39471740 /usr/lib64/libselinux.so.1 7f671e953000-7f671e954000 ---p 0002b000 00:20 39471740 /usr/lib64/libselinux.so.1 7f671e954000-7f671e955000 r--p 0002b000 00:20 39471740 /usr/lib64/libselinux.so.1 7f671e955000-7f671e956000 rw-p 0002c000 00:20 39471740 /usr/lib64/libselinux.so.1 7f671e956000-7f671e958000 rw-p 00000000 00:00 0 7f671e958000-7f671e95a000 r--p 00000000 00:20 39054986 /usr/lib64/libcrypt.so.1.1.0 7f671e95a000-7f671e970000 r-xp 00002000 00:20 39054986 /usr/lib64/libcrypt.so.1.1.0 7f671e970000-7f671e98a000 r--p 00018000 00:20 39054986 /usr/lib64/libcrypt.so.1.1.0 7f671e98a000-7f671e98b000 r--p 00031000 00:20 39054986 /usr/lib64/libcrypt.so.1.1.0 7f671e98b000-7f671e98c000 rw-p 00032000 00:20 39054986 /usr/lib64/libcrypt.so.1.1.0 7f671e98c000-7f671e994000 rw-p 00000000 00:00 0 7f671e994000-7f671e997000 r--p 00000000 00:20 39053979 /usr/lib64/libz.so.1.2.11 7f671e997000-7f671e9a9000 r-xp 00003000 00:20 39053979 /usr/lib64/libz.so.1.2.11 7f671e9a9000-7f671e9af000 r--p 00015000 00:20 39053979 /usr/lib64/libz.so.1.2.11 7f671e9af000-7f671e9b0000 ---p 0001b000 00:20 39053979 /usr/lib64/libz.so.1.2.11 7f671e9b0000-7f671e9b1000 r--p 0001b000 00:20 39053979 /usr/lib64/libz.so.1.2.11 7f671e9b1000-7f671e9b2000 rw-p 0001c000 00:20 39053979 /usr/lib64/libz.so.1.2.11 7f671e9b2000-7f671ea58000 r--p 00000000 00:20 39362770 /usr/lib64/libcrypto.so.1.1 7f671ea58000-7f671ec17000 r-xp 000a6000 00:20 39362770 /usr/lib64/libcrypto.so.1.1 7f671ec17000-7f671eca9000 r--p 00265000 00:20 39362770 /usr/lib64/libcrypto.so.1.1 7f671eca9000-7f671ecaa000 ---p 002f7000 00:20 39362770 /usr/lib64/libcrypto.so.1.1 7f671ecaa000-7f671ecd6000 r--p 002f7000 00:20 39362770 /usr/lib64/libcrypto.so.1.1 7f671ecd6000-7f671ece6000 rw-p 00323000 00:20 39362770 /usr/lib64/libcrypto.so.1.1 7f671ece6000-7f671ecec000 rw-p 00000000 00:00 0 7f671ecec000-7f671ed03000 r--p 00000000 00:20 39445620 /usr/lib64/libsystemd.so.0.34.0 7f671ed03000-7f671ed83000 r-xp 00017000 00:20 39445620 /usr/lib64/libsystemd.so.0.34.0 7f671ed83000-7f671edb2000 r--p 00097000 00:20 39445620 /usr/lib64/libsystemd.so.0.34.0 7f671edb2000-7f671edbe000 r--p 000c5000 00:20 39445620 /usr/lib64/libsystemd.so.0.34.0 7f671edbe000-7f671edbf000 rw-p 000d1000 00:20 39445620 /usr/lib64/libsystemd.so.0.34.0 7f671edbf000-7f671edc0000 rw-p 00000000 00:00 0 7f671edc0000-7f671edc3000 r--p 00000000 00:20 39363195 /usr/lib64/libpam.so.0.85.1 7f671edc3000-7f671edcc000 r-xp 00003000 00:20 39363195 /usr/lib64/libpam.so.0.85.1 7f671edcc000-7f671edd0000 r--p 0000c000 00:20 39363195 /usr/lib64/libpam.so.0.85.1 7f671edd0000-7f671edd1000 r--p 0000f000 00:20 39363195 /usr/lib64/libpam.so.0.85.1 7f671edd1000-7f671edd2000 rw-p 00010000 00:20 39363195 /usr/lib64/libpam.so.0.85.1 7f671edd2000-7f671edd6000 r--p 00000000 00:20 39055104 /usr/lib64/libaudit.so.1.0.0 7f671edd6000-7f671eddd000 r-xp 00004000 00:20 39055104 /usr/lib64/libaudit.so.1.0.0 7f671eddd000-7f671eded000 r--p 0000b000 00:20 39055104 /usr/lib64/libaudit.so.1.0.0 7f671eded000-7f671edee000 ---p 0001b000 00:20 39055104 /usr/lib64/libaudit.so.1.0.0 7f671edee000-7f671edef000 r--p 0001b000 00:20 39055104 /usr/lib64/libaudit.so.1.0.0 7f671edef000-7f671edf0000 rw-p 0001c000 00:20 39055104 /usr/lib64/libaudit.so.1.0.0 7f671edf0000-7f671edfc000 rw-p 00000000 00:00 0 7f671ee23000-7f671ee25000 rw-p 00000000 00:00 0 7f671ee25000-7f671ee29000 r--p 00000000 00:20 39471713 /usr/lib64/ld-linux-x86-64.so.2 7f671ee29000-7f671ee51000 r-xp 00004000 00:20 39471713 /usr/lib64/ld-linux-x86-64.so.2 7f671ee51000-7f671ee5c000 r--p 0002c000 00:20 39471713 /usr/lib64/ld-linux-x86-64.so.2 7f671ee5d000-7f671ee5f000 r--p 00037000 00:20 39471713 /usr/lib64/ld-linux-x86-64.so.2 7f671ee5f000-7f671ee61000 rw-p 00039000 00:20 39471713 /usr/lib64/ld-linux-x86-64.so.2 7fff5be29000-7fff5be4a000 rw-p 00000000 00:00 0 [stack] 7fff5bfd8000-7fff5bfdc000 r--p 00000000 00:00 0 [vvar] 7fff5bfdc000-7fff5bfde000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall] stat /usr/lib64/libaudit.so.1.0.0 File: /usr/lib64/libaudit.so.1.0.0 Size: 116800 Blocks: 232 IO Block: 4096 regular file Device: 0,35 Inode: 39055104 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2022-07-21 16:08:53.425994779 -0500 Modify: 2022-06-14 17:47:30.000000000 -0500 Change: 2022-06-16 20:33:03.839130266 -0500 Birth: 2022-06-16 20:33:03.835130188 -0500 cat /proc/self/mountinfo 23 92 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:11 - proc proc rw 24 92 0:22 / /sys rw,nosuid,nodev,noexec,relatime shared:2 - sysfs sysfs rw 25 92 0:5 / /dev rw,nosuid shared:7 - devtmpfs devtmpfs rw,size=4096k,nr_inodes=1048576,mode=755,inode64 26 24 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:3 - securityfs securityfs rw 27 25 0:23 / /dev/shm rw,nosuid,nodev shared:8 - tmpfs tmpfs rw,inode64 28 25 0:24 / /dev/pts rw,nosuid,noexec,relatime shared:9 - devpts devpts rw,gid=5,mode=620,ptmxmode=000 29 92 0:25 / /run rw,nosuid,nodev shared:10 - tmpfs tmpfs rw,size=1107468k,nr_inodes=819200,mode=755,inode64 30 24 0:26 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:4 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot 31 24 0:27 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:5 - pstore pstore rw 32 24 0:28 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:6 - bpf bpf rw,mode=700 92 1 0:32 /.snapshots/21/snapshot / rw,relatime shared:1 - btrfs /dev/sda2 rw,space_cache,subvolid=2243,subvol=/.snapshots/21/snapshot 34 23 0:30 / /proc/sys/fs/binfmt_misc rw,relatime shared:13 - autofs systemd-1 rw,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13948 35 25 0:20 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:14 - mqueue mqueue rw 36 24 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:15 - debugfs debugfs rw 37 25 0:36 / /dev/hugepages rw,relatime shared:16 - hugetlbfs hugetlbfs rw,pagesize=2M 38 24 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:17 - tracefs tracefs rw 33 24 0:29 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:12 - configfs configfs rw 39 24 0:37 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:18 - fusectl fusectl rw 82 92 0:32 /.snapshots /.snapshots rw,relatime shared:36 - btrfs /dev/sda2 rw,space_cache,subvolid=276,subvol=/.snapshots 102 92 0:32 /boot/grub2/i386-pc /boot/grub2/i386-pc rw,relatime shared:42 - btrfs /dev/sda2 rw,space_cache,subvolid=257,subvol=/boot/grub2/i386-pc 81 92 0:32 /boot/grub2/x86_64-efi /boot/grub2/x86_64-efi rw,relatime shared:44 - btrfs /dev/sda2 rw,space_cache,subvolid=258,subvol=/boot/grub2/x86_64-efi 85 92 0:32 /home /home rw,relatime shared:46 - btrfs /dev/sda2 rw,space_cache,subvolid=259,subvol=/home 86 92 0:32 /opt /opt rw,relatime shared:48 - btrfs /dev/sda2 rw,space_cache,subvolid=260,subvol=/opt 87 92 0:32 /tmp /tmp rw,relatime shared:50 - btrfs /dev/sda2 rw,space_cache,subvolid=262,subvol=/tmp 89 92 0:32 /var/lib/mailman /var/lib/mailman rw,relatime shared:52 - btrfs /dev/sda2 rw,space_cache,subvolid=265,subvol=/var/lib/mailman 95 92 0:32 /var/lib/named /var/lib/named rw,relatime shared:54 - btrfs /dev/sda2 rw,space_cache,subvolid=266,subvol=/var/lib/named 96 92 0:32 /var/lib/pgsql /var/lib/pgsql rw,relatime shared:56 - btrfs /dev/sda2 rw,space_cache,subvolid=267,subvol=/var/lib/pgsql 98 92 0:32 /var/lib/machines /var/lib/machines rw,relatime shared:58 - btrfs /dev/sda2 rw,space_cache,subvolid=1182,subvol=/var/lib/machines 94 92 0:32 /var/crash /var/crash rw,relatime shared:60 - btrfs /dev/sda2 rw,space_cache,subvolid=264,subvol=/var/crash 93 92 0:32 /var/opt /var/opt rw,relatime shared:62 - btrfs /dev/sda2 rw,space_cache,subvolid=269,subvol=/var/opt 100 92 0:32 /var/tmp /var/tmp rw,relatime shared:64 - btrfs /dev/sda2 rw,space_cache,subvolid=271,subvol=/var/tmp 99 92 0:32 /srv /srv rw,relatime shared:66 - btrfs /dev/sda2 rw,space_cache,subvolid=261,subvol=/srv 88 92 0:32 /usr/local /usr/local rw,relatime shared:68 - btrfs /dev/sda2 rw,space_cache,subvolid=263,subvol=/usr/local 91 92 0:32 /var/spool /var/spool rw,relatime shared:70 - btrfs /dev/sda2 rw,space_cache,subvolid=270,subvol=/var/spool 101 92 0:32 /var/log /var/log rw,relatime shared:72 - btrfs /dev/sda2 rw,space_cache,subvolid=268,subvol=/var/log 997 29 0:63 / /run/user/0 rw,nosuid,nodev,relatime shared:530 - tmpfs tmpfs rw,size=553732k,nr_inodes=138433,mode=700,inode64 cat /proc/1270/mountinfo 23 92 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:11 - proc proc rw 24 92 0:22 / /sys rw,nosuid,nodev,noexec,relatime shared:2 - sysfs sysfs rw 25 92 0:5 / /dev rw,nosuid shared:7 - devtmpfs devtmpfs rw,size=4096k,nr_inodes=1048576,mode=755,inode64 26 24 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:3 - securityfs securityfs rw 27 25 0:23 / /dev/shm rw,nosuid,nodev shared:8 - tmpfs tmpfs rw,inode64 28 25 0:24 / /dev/pts rw,nosuid,noexec,relatime shared:9 - devpts devpts rw,gid=5,mode=620,ptmxmode=000 29 92 0:25 / /run rw,nosuid,nodev shared:10 - tmpfs tmpfs rw,size=1107468k,nr_inodes=819200,mode=755,inode64 30 24 0:26 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:4 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot 31 24 0:27 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:5 - pstore pstore rw 32 24 0:28 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:6 - bpf bpf rw,mode=700 92 1 0:32 /.snapshots/21/snapshot / rw,relatime shared:1 - btrfs /dev/sda2 rw,space_cache,subvolid=2243,subvol=/.snapshots/21/snapshot 34 23 0:30 / /proc/sys/fs/binfmt_misc rw,relatime shared:13 - autofs systemd-1 rw,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13948 35 25 0:20 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:14 - mqueue mqueue rw 36 24 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:15 - debugfs debugfs rw 37 25 0:36 / /dev/hugepages rw,relatime shared:16 - hugetlbfs hugetlbfs rw,pagesize=2M 38 24 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:17 - tracefs tracefs rw 33 24 0:29 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:12 - configfs configfs rw 39 24 0:37 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:18 - fusectl fusectl rw 82 92 0:32 /.snapshots /.snapshots rw,relatime shared:36 - btrfs /dev/sda2 rw,space_cache,subvolid=276,subvol=/.snapshots 102 92 0:32 /boot/grub2/i386-pc /boot/grub2/i386-pc rw,relatime shared:42 - btrfs /dev/sda2 rw,space_cache,subvolid=257,subvol=/boot/grub2/i386-pc 81 92 0:32 /boot/grub2/x86_64-efi /boot/grub2/x86_64-efi rw,relatime shared:44 - btrfs /dev/sda2 rw,space_cache,subvolid=258,subvol=/boot/grub2/x86_64-efi 85 92 0:32 /home /home rw,relatime shared:46 - btrfs /dev/sda2 rw,space_cache,subvolid=259,subvol=/home 86 92 0:32 /opt /opt rw,relatime shared:48 - btrfs /dev/sda2 rw,space_cache,subvolid=260,subvol=/opt 87 92 0:32 /tmp /tmp rw,relatime shared:50 - btrfs /dev/sda2 rw,space_cache,subvolid=262,subvol=/tmp 89 92 0:32 /var/lib/mailman /var/lib/mailman rw,relatime shared:52 - btrfs /dev/sda2 rw,space_cache,subvolid=265,subvol=/var/lib/mailman 95 92 0:32 /var/lib/named /var/lib/named rw,relatime shared:54 - btrfs /dev/sda2 rw,space_cache,subvolid=266,subvol=/var/lib/named 96 92 0:32 /var/lib/pgsql /var/lib/pgsql rw,relatime shared:56 - btrfs /dev/sda2 rw,space_cache,subvolid=267,subvol=/var/lib/pgsql 98 92 0:32 /var/lib/machines /var/lib/machines rw,relatime shared:58 - btrfs /dev/sda2 rw,space_cache,subvolid=1182,subvol=/var/lib/machines 94 92 0:32 /var/crash /var/crash rw,relatime shared:60 - btrfs /dev/sda2 rw,space_cache,subvolid=264,subvol=/var/crash 93 92 0:32 /var/opt /var/opt rw,relatime shared:62 - btrfs /dev/sda2 rw,space_cache,subvolid=269,subvol=/var/opt 100 92 0:32 /var/tmp /var/tmp rw,relatime shared:64 - btrfs /dev/sda2 rw,space_cache,subvolid=271,subvol=/var/tmp 99 92 0:32 /srv /srv rw,relatime shared:66 - btrfs /dev/sda2 rw,space_cache,subvolid=261,subvol=/srv 88 92 0:32 /usr/local /usr/local rw,relatime shared:68 - btrfs /dev/sda2 rw,space_cache,subvolid=263,subvol=/usr/local 91 92 0:32 /var/spool /var/spool rw,relatime shared:70 - btrfs /dev/sda2 rw,space_cache,subvolid=270,subvol=/var/spool 101 92 0:32 /var/log /var/log rw,relatime shared:72 - btrfs /dev/sda2 rw,space_cache,subvolid=268,subvol=/var/log 997 29 0:63 / /run/user/0 rw,nosuid,nodev,relatime shared:530 - tmpfs tmpfs rw,size=553732k,nr_inodes=138433,mode=700,inode64 REgards, --Moby On 7/22/2022 00:51, Andrei Borzenkov wrote:
On 22.07.2022 00:52, Moby wrote:
Thanks Robert.
The PIDs reported by zypper ps change after each reboot. I rebooted the machine again and ran zypper ps --debugFile zypper-dbg1
zypper ps output on the screen shows PID of 1270 for sshd using /usr/lib64/libaudit.so.1.0.0 (path dev=0,35). That file is not deleted.
grep'ing zypper-dbg1 for 1270, and then looking for the line containing libaudit.so.1.0.0 shows:
fmem^@tREG^@n/usr/lib64/libaudit.so.1.0.0 (path dev=0,35)^@
See the LSOF FAQ (https://github.com/lsof-org/lsof/blob/bcc125b7c2a160da019dd8089b999bc6d34866...):
10.4.1 What do ``path dev=xxx'' and ``path inode=yyy'' mean in the NAME column of Linux ``mem'' file types? ... When device and inode mismatches occur, lsof suppresses the reporting of link count and size. ...
and the lack of link count is interpreted by zypper as indication that file was deleted.
Show full output of
lsof -p 1270 cat /proc/1270/maps stat /usr/lib64/libaudit.so.1.0.0 cat /proc/self/mountinfo cat /proc/1270/mountinfo
From what I gathered from the link provided by Andrei (https://github.com/lsof-org/lsof/blob/bcc125b7c2a160da019dd8089b999bc6d34866...), zypper ps may have issues if the device or inode or both of a file do not match. I chased one of the files (/usr/lib64/libaudit.so.1.0.0) and find that the device and inode match between /proc/1270/maps and stat. Regards, --Moby On 7/22/2022 10:57, Moby wrote:
Thanks Andrei. Here is the output you requested. I will have to google around to see how to detect device and inode mismatches that you mention before I can make heads or tails of it:
lsof -p 1270 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sshd 1270 root cwd DIR 0,35 202 256 / sshd 1270 root rtd DIR 0,35 202 256 / sshd 1270 root txt REG 0,35 954120 39515208 /usr/sbin/sshd sshd 1270 root mem REG 0,32 39515208 /usr/sbin/sshd (path dev=0,35) sshd 1270 root mem REG 0,32 10018537 /var/lib/nscd/passwd (path dev=0,35) sshd 1270 root mem REG 0,32 39054749 /usr/lib64/libgpg-error.so.0.33.0 (path dev=0,35) sshd 1270 root mem REG 0,32 39471728 /usr/lib64/libresolv.so.2 (path dev=0,35) sshd 1270 root mem REG 0,32 39244436 /usr/lib64/libkeyutils.so.1.10 (path dev=0,35) sshd 1270 root mem REG 0,32 39219676 /usr/lib64/libkrb5support.so.0.1 (path dev=0,35) sshd 1270 root mem REG 0,32 39219670 /usr/lib64/libk5crypto.so.3.1 (path dev=0,35) sshd 1270 root mem REG 0,32 39455014 /usr/lib64/libpcre2-8.so.0.11.0 (path dev=0,35) sshd 1270 root mem REG 0,32 39054551 /usr/lib64/liblz4.so.1.9.3 (path dev=0,35) sshd 1270 root mem REG 0,32 39053974 /usr/lib64/libzstd.so.1.5.2 (path dev=0,35) sshd 1270 root mem REG 0,32 39054546 /usr/lib64/liblzma.so.5.2.5 (path dev=0,35) sshd 1270 root mem REG 0,32 39055729 /usr/lib64/libgcrypt.so.20.3.4 (path dev=0,35) sshd 1270 root mem REG 0,32 39055044 /usr/lib64/libcap.so.2.64 (path dev=0,35) sshd 1270 root mem REG 0,32 39054900 /usr/lib64/libeconf.so.0.4.6 (path dev=0,35) sshd 1270 root mem REG 0,32 39471716 /usr/lib64/libc.so.6 (path dev=0,35) sshd 1270 root mem REG 0,32 38770934 /usr/lib64/libcom_err.so.2.1 (path dev=0,35) sshd 1270 root mem REG 0,32 39219675 /usr/lib64/libkrb5.so.3.3 (path dev=0,35) sshd 1270 root mem REG 0,32 39219668 /usr/lib64/libgssapi_krb5.so.2.2 (path dev=0,35) sshd 1270 root mem REG 0,32 39471740 /usr/lib64/libselinux.so.1 (path dev=0,35) sshd 1270 root mem REG 0,32 39054986 /usr/lib64/libcrypt.so.1.1.0 (path dev=0,35) sshd 1270 root mem REG 0,32 39053979 /usr/lib64/libz.so.1.2.11 (path dev=0,35) sshd 1270 root mem REG 0,32 39362770 /usr/lib64/libcrypto.so.1.1 (path dev=0,35) sshd 1270 root mem REG 0,32 39445620 /usr/lib64/libsystemd.so.0.34.0 (path dev=0,35) sshd 1270 root mem REG 0,32 39363195 /usr/lib64/libpam.so.0.85.1 (path dev=0,35) sshd 1270 root mem REG 0,32 39055104 /usr/lib64/libaudit.so.1.0.0 (path dev=0,35) sshd 1270 root mem REG 0,32 39471713 /usr/lib64/ld-linux-x86-64.so.2 (path dev=0,35) sshd 1270 root 0r CHR 1,3 0t0 4 /dev/null sshd 1270 root 1u unix 0xffff96ca8756f2c0 0t0 19270 type=STREAM (CONNECTED) sshd 1270 root 2u unix 0xffff96ca8756f2c0 0t0 19270 type=STREAM (CONNECTED) sshd 1270 root 3u IPv4 19278 0t0 TCP *:ssh (LISTEN) sshd 1270 root 4u IPv6 19280 0t0 TCP *:ssh (LISTEN)
cat /proc/1270/maps 55f69ac1b000-55f69ac27000 r--p 00000000 00:20 39515208 /usr/sbin/sshd 55f69ac27000-55f69acb6000 r-xp 0000c000 00:20 39515208 /usr/sbin/sshd 55f69acb6000-55f69ad00000 r--p 0009b000 00:20 39515208 /usr/sbin/sshd 55f69ad00000-55f69ad04000 r--p 000e4000 00:20 39515208 /usr/sbin/sshd 55f69ad04000-55f69ad05000 rw-p 000e8000 00:20 39515208 /usr/sbin/sshd 55f69ad05000-55f69ad08000 rw-p 00000000 00:00 0 55f69ae33000-55f69ae74000 rw-p 00000000 00:00 0 [heap] 7f671e210000-7f671e245000 r--s 00000000 00:20 10018537 /var/lib/nscd/passwd 7f671e245000-7f671e24a000 rw-p 00000000 00:00 0 7f671e24a000-7f671e24f000 r--p 00000000 00:20 39054749 /usr/lib64/libgpg-error.so.0.33.0 7f671e24f000-7f671e264000 r-xp 00005000 00:20 39054749 /usr/lib64/libgpg-error.so.0.33.0 7f671e264000-7f671e26e000 r--p 0001a000 00:20 39054749 /usr/lib64/libgpg-error.so.0.33.0 7f671e26e000-7f671e26f000 r--p 00023000 00:20 39054749 /usr/lib64/libgpg-error.so.0.33.0 7f671e26f000-7f671e270000 rw-p 00024000 00:20 39054749 /usr/lib64/libgpg-error.so.0.33.0 7f671e270000-7f671e274000 r--p 00000000 00:20 39471728 /usr/lib64/libresolv.so.2 7f671e274000-7f671e27d000 r-xp 00004000 00:20 39471728 /usr/lib64/libresolv.so.2 7f671e27d000-7f671e27f000 r--p 0000d000 00:20 39471728 /usr/lib64/libresolv.so.2 7f671e27f000-7f671e280000 ---p 0000f000 00:20 39471728 /usr/lib64/libresolv.so.2 7f671e280000-7f671e281000 r--p 0000f000 00:20 39471728 /usr/lib64/libresolv.so.2 7f671e281000-7f671e282000 rw-p 00010000 00:20 39471728 /usr/lib64/libresolv.so.2 7f671e282000-7f671e286000 rw-p 00000000 00:00 0 7f671e286000-7f671e288000 r--p 00000000 00:20 39244436 /usr/lib64/libkeyutils.so.1.10 7f671e288000-7f671e28a000 r-xp 00002000 00:20 39244436 /usr/lib64/libkeyutils.so.1.10 7f671e28a000-7f671e28b000 r--p 00004000 00:20 39244436 /usr/lib64/libkeyutils.so.1.10 7f671e28b000-7f671e28c000 r--p 00004000 00:20 39244436 /usr/lib64/libkeyutils.so.1.10 7f671e28c000-7f671e28d000 rw-p 00005000 00:20 39244436 /usr/lib64/libkeyutils.so.1.10 7f671e28d000-7f671e291000 r--p 00000000 00:20 39219676 /usr/lib64/libkrb5support.so.0.1 7f671e291000-7f671e298000 r-xp 00004000 00:20 39219676 /usr/lib64/libkrb5support.so.0.1 7f671e298000-7f671e29b000 r--p 0000b000 00:20 39219676 /usr/lib64/libkrb5support.so.0.1 7f671e29b000-7f671e29c000 r--p 0000d000 00:20 39219676 /usr/lib64/libkrb5support.so.0.1 7f671e29c000-7f671e29d000 rw-p 0000e000 00:20 39219676 /usr/lib64/libkrb5support.so.0.1 7f671e29d000-7f671e2a2000 r--p 00000000 00:20 39219670 /usr/lib64/libk5crypto.so.3.1 7f671e2a2000-7f671e2ad000 r-xp 00005000 00:20 39219670 /usr/lib64/libk5crypto.so.3.1 7f671e2ad000-7f671e2b1000 r--p 00010000 00:20 39219670 /usr/lib64/libk5crypto.so.3.1 7f671e2b1000-7f671e2b3000 r--p 00013000 00:20 39219670 /usr/lib64/libk5crypto.so.3.1 7f671e2b3000-7f671e2b4000 rw-p 00015000 00:20 39219670 /usr/lib64/libk5crypto.so.3.1 7f671e2b4000-7f671e2b7000 r--p 00000000 00:20 39455014 /usr/lib64/libpcre2-8.so.0.11.0 7f671e2b7000-7f671e338000 r-xp 00003000 00:20 39455014 /usr/lib64/libpcre2-8.so.0.11.0 7f671e338000-7f671e364000 r--p 00084000 00:20 39455014 /usr/lib64/libpcre2-8.so.0.11.0 7f671e364000-7f671e365000 ---p 000b0000 00:20 39455014 /usr/lib64/libpcre2-8.so.0.11.0 7f671e365000-7f671e366000 r--p 000b0000 00:20 39455014 /usr/lib64/libpcre2-8.so.0.11.0 7f671e366000-7f671e367000 rw-p 000b1000 00:20 39455014 /usr/lib64/libpcre2-8.so.0.11.0 7f671e367000-7f671e36a000 r--p 00000000 00:20 39054551 /usr/lib64/liblz4.so.1.9.3 7f671e36a000-7f671e385000 r-xp 00003000 00:20 39054551 /usr/lib64/liblz4.so.1.9.3 7f671e385000-7f671e388000 r--p 0001e000 00:20 39054551 /usr/lib64/liblz4.so.1.9.3 7f671e388000-7f671e389000 r--p 00020000 00:20 39054551 /usr/lib64/liblz4.so.1.9.3 7f671e389000-7f671e38a000 rw-p 00021000 00:20 39054551 /usr/lib64/liblz4.so.1.9.3 7f671e38a000-7f671e390000 r--p 00000000 00:20 39053974 /usr/lib64/libzstd.so.1.5.2 7f671e390000-7f671e42b000 r-xp 00006000 00:20 39053974 /usr/lib64/libzstd.so.1.5.2 7f671e42b000-7f671e43a000 r--p 000a1000 00:20 39053974 /usr/lib64/libzstd.so.1.5.2 7f671e43a000-7f671e43b000 ---p 000b0000 00:20 39053974 /usr/lib64/libzstd.so.1.5.2 7f671e43b000-7f671e43c000 r--p 000b0000 00:20 39053974 /usr/lib64/libzstd.so.1.5.2 7f671e43c000-7f671e43d000 rw-p 000b1000 00:20 39053974 /usr/lib64/libzstd.so.1.5.2 7f671e43d000-7f671e43f000 rw-p 00000000 00:00 0 7f671e43f000-7f671e443000 r--p 00000000 00:20 39054546 /usr/lib64/liblzma.so.5.2.5 7f671e443000-7f671e464000 r-xp 00004000 00:20 39054546 /usr/lib64/liblzma.so.5.2.5 7f671e464000-7f671e46f000 r--p 00025000 00:20 39054546 /usr/lib64/liblzma.so.5.2.5 7f671e46f000-7f671e470000 ---p 00030000 00:20 39054546 /usr/lib64/liblzma.so.5.2.5 7f671e470000-7f671e471000 r--p 00030000 00:20 39054546 /usr/lib64/liblzma.so.5.2.5 7f671e471000-7f671e472000 rw-p 00031000 00:20 39054546 /usr/lib64/liblzma.so.5.2.5 7f671e472000-7f671e481000 r--p 00000000 00:20 39055729 /usr/lib64/libgcrypt.so.20.3.4 7f671e481000-7f671e568000 r-xp 0000f000 00:20 39055729 /usr/lib64/libgcrypt.so.20.3.4 7f671e568000-7f671e5a7000 r--p 000f6000 00:20 39055729 /usr/lib64/libgcrypt.so.20.3.4 7f671e5a7000-7f671e5aa000 r--p 00134000 00:20 39055729 /usr/lib64/libgcrypt.so.20.3.4 7f671e5aa000-7f671e5b0000 rw-p 00137000 00:20 39055729 /usr/lib64/libgcrypt.so.20.3.4 7f671e5b0000-7f671e5b1000 rw-p 00000000 00:00 0 7f671e5b1000-7f671e5b4000 r--p 00000000 00:20 39055044 /usr/lib64/libcap.so.2.64 7f671e5b4000-7f671e5b9000 r-xp 00003000 00:20 39055044 /usr/lib64/libcap.so.2.64 7f671e5b9000-7f671e5bb000 r--p 00008000 00:20 39055044 /usr/lib64/libcap.so.2.64 7f671e5bb000-7f671e5bc000 ---p 0000a000 00:20 39055044 /usr/lib64/libcap.so.2.64 7f671e5bc000-7f671e5bd000 r--p 0000a000 00:20 39055044 /usr/lib64/libcap.so.2.64 7f671e5bd000-7f671e5be000 rw-p 0000b000 00:20 39055044 /usr/lib64/libcap.so.2.64 7f671e5be000-7f671e5c0000 r--p 00000000 00:20 39054900 /usr/lib64/libeconf.so.0.4.6 7f671e5c0000-7f671e5c5000 r-xp 00002000 00:20 39054900 /usr/lib64/libeconf.so.0.4.6 7f671e5c5000-7f671e5c7000 r--p 00007000 00:20 39054900 /usr/lib64/libeconf.so.0.4.6 7f671e5c7000-7f671e5c8000 r--p 00008000 00:20 39054900 /usr/lib64/libeconf.so.0.4.6 7f671e5c8000-7f671e5c9000 rw-p 00009000 00:20 39054900 /usr/lib64/libeconf.so.0.4.6 7f671e5c9000-7f671e5ca000 rw-p 00000000 00:00 0 7f671e5ca000-7f671e609000 r--p 00000000 00:20 39471716 /usr/lib64/libc.so.6 7f671e609000-7f671e78e000 r-xp 0003f000 00:20 39471716 /usr/lib64/libc.so.6 7f671e78e000-7f671e7e2000 r--p 001c4000 00:20 39471716 /usr/lib64/libc.so.6 7f671e7e2000-7f671e7e3000 ---p 00218000 00:20 39471716 /usr/lib64/libc.so.6 7f671e7e3000-7f671e7e6000 r--p 00218000 00:20 39471716 /usr/lib64/libc.so.6 7f671e7e6000-7f671e7ef000 rw-p 0021b000 00:20 39471716 /usr/lib64/libc.so.6 7f671e7ef000-7f671e7fc000 rw-p 00000000 00:00 0 7f671e7fc000-7f671e7fe000 r--p 00000000 00:20 38770934 /usr/lib64/libcom_err.so.2.1 7f671e7fe000-7f671e7ff000 r-xp 00002000 00:20 38770934 /usr/lib64/libcom_err.so.2.1 7f671e7ff000-7f671e800000 r--p 00003000 00:20 38770934 /usr/lib64/libcom_err.so.2.1 7f671e800000-7f671e801000 r--p 00003000 00:20 38770934 /usr/lib64/libcom_err.so.2.1 7f671e801000-7f671e802000 rw-p 00004000 00:20 38770934 /usr/lib64/libcom_err.so.2.1 7f671e802000-7f671e804000 rw-p 00000000 00:00 0 7f671e804000-7f671e828000 r--p 00000000 00:20 39219675 /usr/lib64/libkrb5.so.3.3 7f671e828000-7f671e886000 r-xp 00024000 00:20 39219675 /usr/lib64/libkrb5.so.3.3 7f671e886000-7f671e8c3000 r--p 00082000 00:20 39219675 /usr/lib64/libkrb5.so.3.3 7f671e8c3000-7f671e8c4000 ---p 000bf000 00:20 39219675 /usr/lib64/libkrb5.so.3.3 7f671e8c4000-7f671e8d2000 r--p 000bf000 00:20 39219675 /usr/lib64/libkrb5.so.3.3 7f671e8d2000-7f671e8d4000 rw-p 000cd000 00:20 39219675 /usr/lib64/libkrb5.so.3.3 7f671e8d4000-7f671e8e1000 r--p 00000000 00:20 39219668 /usr/lib64/libgssapi_krb5.so.2.2 7f671e8e1000-7f671e918000 r-xp 0000d000 00:20 39219668 /usr/lib64/libgssapi_krb5.so.2.2 7f671e918000-7f671e924000 r--p 00044000 00:20 39219668 /usr/lib64/libgssapi_krb5.so.2.2 7f671e924000-7f671e925000 ---p 00050000 00:20 39219668 /usr/lib64/libgssapi_krb5.so.2.2 7f671e925000-7f671e927000 r--p 00050000 00:20 39219668 /usr/lib64/libgssapi_krb5.so.2.2 7f671e927000-7f671e928000 rw-p 00052000 00:20 39219668 /usr/lib64/libgssapi_krb5.so.2.2 7f671e928000-7f671e930000 r--p 00000000 00:20 39471740 /usr/lib64/libselinux.so.1 7f671e930000-7f671e94b000 r-xp 00008000 00:20 39471740 /usr/lib64/libselinux.so.1 7f671e94b000-7f671e953000 r--p 00023000 00:20 39471740 /usr/lib64/libselinux.so.1 7f671e953000-7f671e954000 ---p 0002b000 00:20 39471740 /usr/lib64/libselinux.so.1 7f671e954000-7f671e955000 r--p 0002b000 00:20 39471740 /usr/lib64/libselinux.so.1 7f671e955000-7f671e956000 rw-p 0002c000 00:20 39471740 /usr/lib64/libselinux.so.1 7f671e956000-7f671e958000 rw-p 00000000 00:00 0 7f671e958000-7f671e95a000 r--p 00000000 00:20 39054986 /usr/lib64/libcrypt.so.1.1.0 7f671e95a000-7f671e970000 r-xp 00002000 00:20 39054986 /usr/lib64/libcrypt.so.1.1.0 7f671e970000-7f671e98a000 r--p 00018000 00:20 39054986 /usr/lib64/libcrypt.so.1.1.0 7f671e98a000-7f671e98b000 r--p 00031000 00:20 39054986 /usr/lib64/libcrypt.so.1.1.0 7f671e98b000-7f671e98c000 rw-p 00032000 00:20 39054986 /usr/lib64/libcrypt.so.1.1.0 7f671e98c000-7f671e994000 rw-p 00000000 00:00 0 7f671e994000-7f671e997000 r--p 00000000 00:20 39053979 /usr/lib64/libz.so.1.2.11 7f671e997000-7f671e9a9000 r-xp 00003000 00:20 39053979 /usr/lib64/libz.so.1.2.11 7f671e9a9000-7f671e9af000 r--p 00015000 00:20 39053979 /usr/lib64/libz.so.1.2.11 7f671e9af000-7f671e9b0000 ---p 0001b000 00:20 39053979 /usr/lib64/libz.so.1.2.11 7f671e9b0000-7f671e9b1000 r--p 0001b000 00:20 39053979 /usr/lib64/libz.so.1.2.11 7f671e9b1000-7f671e9b2000 rw-p 0001c000 00:20 39053979 /usr/lib64/libz.so.1.2.11 7f671e9b2000-7f671ea58000 r--p 00000000 00:20 39362770 /usr/lib64/libcrypto.so.1.1 7f671ea58000-7f671ec17000 r-xp 000a6000 00:20 39362770 /usr/lib64/libcrypto.so.1.1 7f671ec17000-7f671eca9000 r--p 00265000 00:20 39362770 /usr/lib64/libcrypto.so.1.1 7f671eca9000-7f671ecaa000 ---p 002f7000 00:20 39362770 /usr/lib64/libcrypto.so.1.1 7f671ecaa000-7f671ecd6000 r--p 002f7000 00:20 39362770 /usr/lib64/libcrypto.so.1.1 7f671ecd6000-7f671ece6000 rw-p 00323000 00:20 39362770 /usr/lib64/libcrypto.so.1.1 7f671ece6000-7f671ecec000 rw-p 00000000 00:00 0 7f671ecec000-7f671ed03000 r--p 00000000 00:20 39445620 /usr/lib64/libsystemd.so.0.34.0 7f671ed03000-7f671ed83000 r-xp 00017000 00:20 39445620 /usr/lib64/libsystemd.so.0.34.0 7f671ed83000-7f671edb2000 r--p 00097000 00:20 39445620 /usr/lib64/libsystemd.so.0.34.0 7f671edb2000-7f671edbe000 r--p 000c5000 00:20 39445620 /usr/lib64/libsystemd.so.0.34.0 7f671edbe000-7f671edbf000 rw-p 000d1000 00:20 39445620 /usr/lib64/libsystemd.so.0.34.0 7f671edbf000-7f671edc0000 rw-p 00000000 00:00 0 7f671edc0000-7f671edc3000 r--p 00000000 00:20 39363195 /usr/lib64/libpam.so.0.85.1 7f671edc3000-7f671edcc000 r-xp 00003000 00:20 39363195 /usr/lib64/libpam.so.0.85.1 7f671edcc000-7f671edd0000 r--p 0000c000 00:20 39363195 /usr/lib64/libpam.so.0.85.1 7f671edd0000-7f671edd1000 r--p 0000f000 00:20 39363195 /usr/lib64/libpam.so.0.85.1 7f671edd1000-7f671edd2000 rw-p 00010000 00:20 39363195 /usr/lib64/libpam.so.0.85.1 7f671edd2000-7f671edd6000 r--p 00000000 00:20 39055104 /usr/lib64/libaudit.so.1.0.0 7f671edd6000-7f671eddd000 r-xp 00004000 00:20 39055104 /usr/lib64/libaudit.so.1.0.0 7f671eddd000-7f671eded000 r--p 0000b000 00:20 39055104 /usr/lib64/libaudit.so.1.0.0 7f671eded000-7f671edee000 ---p 0001b000 00:20 39055104 /usr/lib64/libaudit.so.1.0.0 7f671edee000-7f671edef000 r--p 0001b000 00:20 39055104 /usr/lib64/libaudit.so.1.0.0 7f671edef000-7f671edf0000 rw-p 0001c000 00:20 39055104 /usr/lib64/libaudit.so.1.0.0 7f671edf0000-7f671edfc000 rw-p 00000000 00:00 0 7f671ee23000-7f671ee25000 rw-p 00000000 00:00 0 7f671ee25000-7f671ee29000 r--p 00000000 00:20 39471713 /usr/lib64/ld-linux-x86-64.so.2 7f671ee29000-7f671ee51000 r-xp 00004000 00:20 39471713 /usr/lib64/ld-linux-x86-64.so.2 7f671ee51000-7f671ee5c000 r--p 0002c000 00:20 39471713 /usr/lib64/ld-linux-x86-64.so.2 7f671ee5d000-7f671ee5f000 r--p 00037000 00:20 39471713 /usr/lib64/ld-linux-x86-64.so.2 7f671ee5f000-7f671ee61000 rw-p 00039000 00:20 39471713 /usr/lib64/ld-linux-x86-64.so.2 7fff5be29000-7fff5be4a000 rw-p 00000000 00:00 0 [stack] 7fff5bfd8000-7fff5bfdc000 r--p 00000000 00:00 0 [vvar] 7fff5bfdc000-7fff5bfde000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
stat /usr/lib64/libaudit.so.1.0.0 File: /usr/lib64/libaudit.so.1.0.0 Size: 116800 Blocks: 232 IO Block: 4096 regular file Device: 0,35 Inode: 39055104 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2022-07-21 16:08:53.425994779 -0500 Modify: 2022-06-14 17:47:30.000000000 -0500 Change: 2022-06-16 20:33:03.839130266 -0500 Birth: 2022-06-16 20:33:03.835130188 -0500
cat /proc/self/mountinfo 23 92 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:11 - proc proc rw 24 92 0:22 / /sys rw,nosuid,nodev,noexec,relatime shared:2 - sysfs sysfs rw 25 92 0:5 / /dev rw,nosuid shared:7 - devtmpfs devtmpfs rw,size=4096k,nr_inodes=1048576,mode=755,inode64 26 24 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:3 - securityfs securityfs rw 27 25 0:23 / /dev/shm rw,nosuid,nodev shared:8 - tmpfs tmpfs rw,inode64 28 25 0:24 / /dev/pts rw,nosuid,noexec,relatime shared:9 - devpts devpts rw,gid=5,mode=620,ptmxmode=000 29 92 0:25 / /run rw,nosuid,nodev shared:10 - tmpfs tmpfs rw,size=1107468k,nr_inodes=819200,mode=755,inode64 30 24 0:26 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:4 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot 31 24 0:27 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:5 - pstore pstore rw 32 24 0:28 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:6 - bpf bpf rw,mode=700 92 1 0:32 /.snapshots/21/snapshot / rw,relatime shared:1 - btrfs /dev/sda2 rw,space_cache,subvolid=2243,subvol=/.snapshots/21/snapshot 34 23 0:30 / /proc/sys/fs/binfmt_misc rw,relatime shared:13 - autofs systemd-1 rw,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13948 35 25 0:20 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:14 - mqueue mqueue rw 36 24 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:15 - debugfs debugfs rw 37 25 0:36 / /dev/hugepages rw,relatime shared:16 - hugetlbfs hugetlbfs rw,pagesize=2M 38 24 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:17 - tracefs tracefs rw 33 24 0:29 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:12 - configfs configfs rw 39 24 0:37 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:18 - fusectl fusectl rw 82 92 0:32 /.snapshots /.snapshots rw,relatime shared:36 - btrfs /dev/sda2 rw,space_cache,subvolid=276,subvol=/.snapshots 102 92 0:32 /boot/grub2/i386-pc /boot/grub2/i386-pc rw,relatime shared:42 - btrfs /dev/sda2 rw,space_cache,subvolid=257,subvol=/boot/grub2/i386-pc 81 92 0:32 /boot/grub2/x86_64-efi /boot/grub2/x86_64-efi rw,relatime shared:44 - btrfs /dev/sda2 rw,space_cache,subvolid=258,subvol=/boot/grub2/x86_64-efi 85 92 0:32 /home /home rw,relatime shared:46 - btrfs /dev/sda2 rw,space_cache,subvolid=259,subvol=/home 86 92 0:32 /opt /opt rw,relatime shared:48 - btrfs /dev/sda2 rw,space_cache,subvolid=260,subvol=/opt 87 92 0:32 /tmp /tmp rw,relatime shared:50 - btrfs /dev/sda2 rw,space_cache,subvolid=262,subvol=/tmp 89 92 0:32 /var/lib/mailman /var/lib/mailman rw,relatime shared:52 - btrfs /dev/sda2 rw,space_cache,subvolid=265,subvol=/var/lib/mailman 95 92 0:32 /var/lib/named /var/lib/named rw,relatime shared:54 - btrfs /dev/sda2 rw,space_cache,subvolid=266,subvol=/var/lib/named 96 92 0:32 /var/lib/pgsql /var/lib/pgsql rw,relatime shared:56 - btrfs /dev/sda2 rw,space_cache,subvolid=267,subvol=/var/lib/pgsql 98 92 0:32 /var/lib/machines /var/lib/machines rw,relatime shared:58 - btrfs /dev/sda2 rw,space_cache,subvolid=1182,subvol=/var/lib/machines 94 92 0:32 /var/crash /var/crash rw,relatime shared:60 - btrfs /dev/sda2 rw,space_cache,subvolid=264,subvol=/var/crash 93 92 0:32 /var/opt /var/opt rw,relatime shared:62 - btrfs /dev/sda2 rw,space_cache,subvolid=269,subvol=/var/opt 100 92 0:32 /var/tmp /var/tmp rw,relatime shared:64 - btrfs /dev/sda2 rw,space_cache,subvolid=271,subvol=/var/tmp 99 92 0:32 /srv /srv rw,relatime shared:66 - btrfs /dev/sda2 rw,space_cache,subvolid=261,subvol=/srv 88 92 0:32 /usr/local /usr/local rw,relatime shared:68 - btrfs /dev/sda2 rw,space_cache,subvolid=263,subvol=/usr/local 91 92 0:32 /var/spool /var/spool rw,relatime shared:70 - btrfs /dev/sda2 rw,space_cache,subvolid=270,subvol=/var/spool 101 92 0:32 /var/log /var/log rw,relatime shared:72 - btrfs /dev/sda2 rw,space_cache,subvolid=268,subvol=/var/log 997 29 0:63 / /run/user/0 rw,nosuid,nodev,relatime shared:530 - tmpfs tmpfs rw,size=553732k,nr_inodes=138433,mode=700,inode64
cat /proc/1270/mountinfo 23 92 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:11 - proc proc rw 24 92 0:22 / /sys rw,nosuid,nodev,noexec,relatime shared:2 - sysfs sysfs rw 25 92 0:5 / /dev rw,nosuid shared:7 - devtmpfs devtmpfs rw,size=4096k,nr_inodes=1048576,mode=755,inode64 26 24 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:3 - securityfs securityfs rw 27 25 0:23 / /dev/shm rw,nosuid,nodev shared:8 - tmpfs tmpfs rw,inode64 28 25 0:24 / /dev/pts rw,nosuid,noexec,relatime shared:9 - devpts devpts rw,gid=5,mode=620,ptmxmode=000 29 92 0:25 / /run rw,nosuid,nodev shared:10 - tmpfs tmpfs rw,size=1107468k,nr_inodes=819200,mode=755,inode64 30 24 0:26 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:4 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot 31 24 0:27 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:5 - pstore pstore rw 32 24 0:28 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:6 - bpf bpf rw,mode=700 92 1 0:32 /.snapshots/21/snapshot / rw,relatime shared:1 - btrfs /dev/sda2 rw,space_cache,subvolid=2243,subvol=/.snapshots/21/snapshot 34 23 0:30 / /proc/sys/fs/binfmt_misc rw,relatime shared:13 - autofs systemd-1 rw,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13948 35 25 0:20 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:14 - mqueue mqueue rw 36 24 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:15 - debugfs debugfs rw 37 25 0:36 / /dev/hugepages rw,relatime shared:16 - hugetlbfs hugetlbfs rw,pagesize=2M 38 24 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:17 - tracefs tracefs rw 33 24 0:29 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:12 - configfs configfs rw 39 24 0:37 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:18 - fusectl fusectl rw 82 92 0:32 /.snapshots /.snapshots rw,relatime shared:36 - btrfs /dev/sda2 rw,space_cache,subvolid=276,subvol=/.snapshots 102 92 0:32 /boot/grub2/i386-pc /boot/grub2/i386-pc rw,relatime shared:42 - btrfs /dev/sda2 rw,space_cache,subvolid=257,subvol=/boot/grub2/i386-pc 81 92 0:32 /boot/grub2/x86_64-efi /boot/grub2/x86_64-efi rw,relatime shared:44 - btrfs /dev/sda2 rw,space_cache,subvolid=258,subvol=/boot/grub2/x86_64-efi 85 92 0:32 /home /home rw,relatime shared:46 - btrfs /dev/sda2 rw,space_cache,subvolid=259,subvol=/home 86 92 0:32 /opt /opt rw,relatime shared:48 - btrfs /dev/sda2 rw,space_cache,subvolid=260,subvol=/opt 87 92 0:32 /tmp /tmp rw,relatime shared:50 - btrfs /dev/sda2 rw,space_cache,subvolid=262,subvol=/tmp 89 92 0:32 /var/lib/mailman /var/lib/mailman rw,relatime shared:52 - btrfs /dev/sda2 rw,space_cache,subvolid=265,subvol=/var/lib/mailman 95 92 0:32 /var/lib/named /var/lib/named rw,relatime shared:54 - btrfs /dev/sda2 rw,space_cache,subvolid=266,subvol=/var/lib/named 96 92 0:32 /var/lib/pgsql /var/lib/pgsql rw,relatime shared:56 - btrfs /dev/sda2 rw,space_cache,subvolid=267,subvol=/var/lib/pgsql 98 92 0:32 /var/lib/machines /var/lib/machines rw,relatime shared:58 - btrfs /dev/sda2 rw,space_cache,subvolid=1182,subvol=/var/lib/machines 94 92 0:32 /var/crash /var/crash rw,relatime shared:60 - btrfs /dev/sda2 rw,space_cache,subvolid=264,subvol=/var/crash 93 92 0:32 /var/opt /var/opt rw,relatime shared:62 - btrfs /dev/sda2 rw,space_cache,subvolid=269,subvol=/var/opt 100 92 0:32 /var/tmp /var/tmp rw,relatime shared:64 - btrfs /dev/sda2 rw,space_cache,subvolid=271,subvol=/var/tmp 99 92 0:32 /srv /srv rw,relatime shared:66 - btrfs /dev/sda2 rw,space_cache,subvolid=261,subvol=/srv 88 92 0:32 /usr/local /usr/local rw,relatime shared:68 - btrfs /dev/sda2 rw,space_cache,subvolid=263,subvol=/usr/local 91 92 0:32 /var/spool /var/spool rw,relatime shared:70 - btrfs /dev/sda2 rw,space_cache,subvolid=270,subvol=/var/spool 101 92 0:32 /var/log /var/log rw,relatime shared:72 - btrfs /dev/sda2 rw,space_cache,subvolid=268,subvol=/var/log 997 29 0:63 / /run/user/0 rw,nosuid,nodev,relatime shared:530 - tmpfs tmpfs rw,size=553732k,nr_inodes=138433,mode=700,inode64
REgards,
--Moby
On 7/22/2022 00:51, Andrei Borzenkov wrote:
On 22.07.2022 00:52, Moby wrote:
Thanks Robert.
The PIDs reported by zypper ps change after each reboot. I rebooted the machine again and ran zypper ps --debugFile zypper-dbg1
zypper ps output on the screen shows PID of 1270 for sshd using /usr/lib64/libaudit.so.1.0.0 (path dev=0,35). That file is not deleted.
grep'ing zypper-dbg1 for 1270, and then looking for the line containing libaudit.so.1.0.0 shows:
fmem^@tREG^@n/usr/lib64/libaudit.so.1.0.0 (path dev=0,35)^@
See the LSOF FAQ (https://github.com/lsof-org/lsof/blob/bcc125b7c2a160da019dd8089b999bc6d34866...):
10.4.1 What do ``path dev=xxx'' and ``path inode=yyy'' mean in the NAME column of Linux ``mem'' file types? ... When device and inode mismatches occur, lsof suppresses the reporting of link count and size. ...
and the lack of link count is interpreted by zypper as indication that file was deleted.
Show full output of
lsof -p 1270 cat /proc/1270/maps stat /usr/lib64/libaudit.so.1.0.0 cat /proc/self/mountinfo cat /proc/1270/mountinfo
On 22.07.2022 20:54, Moby wrote:
From what I gathered from the link provided by Andrei (https://github.com/lsof-org/lsof/blob/bcc125b7c2a160da019dd8089b999bc6d34866...), zypper ps may have issues if the device or inode or both of a file do not match. I chased one of the files (/usr/lib64/libaudit.so.1.0.0) and find that the device and inode match between /proc/1270/maps and stat.
No, they do not.
cat /proc/1270/maps ... 7f671edd2000-7f671edd6000 r--p 00000000 00:20>> 39055104 /usr/lib64/libaudit.so.1.0.0 ...
stat /usr/lib64/libaudit.so.1.0.0 File: /usr/lib64/libaudit.so.1.0.0 Size: 116800 Blocks: 232 IO Block: 4096 regular file Device: 0,35 Inode: 39055104 Links: 1
So this is device 0,32 in /proc/.../maps and device 0,35 in stat. Which kernel and root filesystem are you using (more precisely - what filesystem where /usr/lib64 is located)? Are you using vanilla or upstream kernel instead of openSUSE kernel? bor@tw:~> uname -a Linux tw 5.18.9-2-vanilla #1 SMP PREEMPT_DYNAMIC Wed Jul 6 05:57:32 UTC 2022 (a7c5f9c) x86_64 x86_64 x86_64 GNU/Linux bor@tw:~> sudo zypper ps The following running processes use deleted files: PID | PPID | UID | User | Command | Service | Files -----+------+------+------------+-------------------------------+-----------------------+----------------------------------------------------------------------------------------------------- 1 | 0 | 0 | root | systemd | | /usr/lib/systemd/libsystemd-shared-251.so (path dev=0,36) | | | | | | /usr/lib64/libc.so.6 (path dev=0,36) ... bor@tw:~> sudo grep /usr/lib/systemd/libsystemd-shared-251.so /proc/1/maps 7fa29f5b4000-7fa29f623000 r--p 00000000 00:20 13359340 /usr/lib/systemd/libsystemd-shared-251.so ... bor@tw:~> stat /usr/lib/systemd/libsystemd-shared-251.so File: /usr/lib/systemd/libsystemd-shared-251.so Size: 3262064 Blocks: 6376 IO Block: 4096 regular file Device: 0,36 Inode: 13359340 Links: 1 ...
Thanks Andrei, I guess my eyes are not what they used to be! On 7/22/2022 15:37, Andrei Borzenkov wrote:
On 22.07.2022 20:54, Moby wrote:
From what I gathered from the link provided by Andrei (https://github.com/lsof-org/lsof/blob/bcc125b7c2a160da019dd8089b999bc6d34866...), zypper ps may have issues if the device or inode or both of a file do not match. I chased one of the files (/usr/lib64/libaudit.so.1.0.0) and find that the device and inode match between /proc/1270/maps and stat.
No, they do not.
cat /proc/1270/maps ... 7f671edd2000-7f671edd6000 r--p 00000000 00:20>> 39055104 /usr/lib64/libaudit.so.1.0.0 ... stat /usr/lib64/libaudit.so.1.0.0 File: /usr/lib64/libaudit.so.1.0.0 Size: 116800 Blocks: 232 IO Block: 4096 regular file Device: 0,35 Inode: 39055104 Links: 1 So this is device 0,32 in /proc/.../maps and device 0,35 in stat.
Yes, you are correct, /proc/1270/maps shows the file as 0,32 while stat shows 0,35
Which kernel and root filesystem are you using (more precisely - what filesystem where /usr/lib64 is located)?
It is a vanilla kernel: uname -a Linux esilinux2.ad.techspace.cc 5.18.11-1-vanilla #1 SMP PREEMPT_DYNAMIC Fri Jul 15 05:36:11 UTC 2022 (4fcb983) x86_64 x86_64 x86_64 GNU/Linux root filesystem is same as /usr/lib64, it is btrfs. df -Th /usr/lib64 Filesystem Type Size Used Avail Use% Mounted on /dev/sda2 btrfs 595G 160G 434G 27% /
bor@tw:~> sudo zypper ps The following running processes use deleted files:
PID | PPID | UID | User | Command | Service | Files -----+------+------+------------+-------------------------------+-----------------------+----------------------------------------------------------------------------------------------------- 1 | 0 | 0 | root | systemd | | /usr/lib/systemd/libsystemd-shared-251.so (path dev=0,36) | | | | | | /usr/lib64/libc.so.6 (path dev=0,36) ... bor@tw:~> sudo grep /usr/lib/systemd/libsystemd-shared-251.so /proc/1/maps 7fa29f5b4000-7fa29f623000 r--p 00000000 00:20 13359340 /usr/lib/systemd/libsystemd-shared-251.so ... bor@tw:~> stat /usr/lib/systemd/libsystemd-shared-251.so File: /usr/lib/systemd/libsystemd-shared-251.so Size: 3262064 Blocks: 6376 IO Block: 4096 regular file Device: 0,36 Inode: 13359340 Links: 1 ...
On 22.07.2022 23:55, Moby wrote:
Thanks Andrei, I guess my eyes are not what they used to be!
On 7/22/2022 15:37, Andrei Borzenkov wrote:
On 22.07.2022 20:54, Moby wrote:
From what I gathered from the link provided by Andrei (https://github.com/lsof-org/lsof/blob/bcc125b7c2a160da019dd8089b999bc6d34866...), zypper ps may have issues if the device or inode or both of a file do not match. I chased one of the files (/usr/lib64/libaudit.so.1.0.0) and find that the device and inode match between /proc/1270/maps and stat.
No, they do not.
cat /proc/1270/maps ... 7f671edd2000-7f671edd6000 r--p 00000000 00:20>> 39055104 /usr/lib64/libaudit.so.1.0.0 ... stat /usr/lib64/libaudit.so.1.0.0 File: /usr/lib64/libaudit.so.1.0.0 Size: 116800 Blocks: 232 IO Block: 4096 regular file Device: 0,35 Inode: 39055104 Links: 1 So this is device 0,32 in /proc/.../maps and device 0,35 in stat.
Yes, you are correct, /proc/1270/maps shows the file as 0,32 while stat shows 0,35
Which kernel and root filesystem are you using (more precisely - what filesystem where /usr/lib64 is located)?
It is a vanilla kernel:
uname -a Linux esilinux2.ad.techspace.cc 5.18.11-1-vanilla #1 SMP PREEMPT_DYNAMIC Fri Jul 15 05:36:11 UTC 2022 (4fcb983) x86_64 x86_64 x86_64 GNU/Linux
And the reason you never mentioned it in the first place is ... ? When you post to openSUSE list everyone expects that you are using openSUSE packages. If you are using unmodified upstream version you either tell it or ask upstream. This is expected behavior and there is no way it could "have started a couple of weeks ago" unless you have been using openSUSE kernel before.
Thanks Andrei. You make valid points. In my defense: 1/ This Tumbleweed machine always updates only via zypper ref; zypper dup. I did not ever specify to use a specific kernel. 2/ I had assumed (Wrongly I know now from what you mention) that is was Tumbleweed distro that was switching to a vanilla kernel. 3/ I did not know this was expected behavior. This behavior did indeed start after a few updates ago, the updates again were via zypper ref; zypper dup 4/ Would zypper dup not put me back on openSUSE packages, including the kernel? Doing a zypper ref; zypper dup, I do not see any kernel packages being offered. I will browse the openSUSE Tumbleweed repo and see what kernel it offers, though now I am a bit concerned as to how this kernel got on the machine to begin with. Regards, --Moby On 7/22/2022 16:06, Andrei Borzenkov wrote:
On 22.07.2022 23:55, Moby wrote:
Thanks Andrei, I guess my eyes are not what they used to be!
On 7/22/2022 15:37, Andrei Borzenkov wrote:
On 22.07.2022 20:54, Moby wrote:
From what I gathered from the link provided by Andrei (https://github.com/lsof-org/lsof/blob/bcc125b7c2a160da019dd8089b999bc6d34866...), zypper ps may have issues if the device or inode or both of a file do not match. I chased one of the files (/usr/lib64/libaudit.so.1.0.0) and find that the device and inode match between /proc/1270/maps and stat.
No, they do not.
cat /proc/1270/maps ... 7f671edd2000-7f671edd6000 r--p 00000000 00:20>> 39055104 /usr/lib64/libaudit.so.1.0.0 ... stat /usr/lib64/libaudit.so.1.0.0 File: /usr/lib64/libaudit.so.1.0.0 Size: 116800 Blocks: 232 IO Block: 4096 regular file Device: 0,35 Inode: 39055104 Links: 1 So this is device 0,32 in /proc/.../maps and device 0,35 in stat. Yes, you are correct, /proc/1270/maps shows the file as 0,32 while stat shows 0,35
Which kernel and root filesystem are you using (more precisely - what filesystem where /usr/lib64 is located)? It is a vanilla kernel:
uname -a Linux esilinux2.ad.techspace.cc 5.18.11-1-vanilla #1 SMP PREEMPT_DYNAMIC Fri Jul 15 05:36:11 UTC 2022 (4fcb983) x86_64 x86_64 x86_64 GNU/Linux
And the reason you never mentioned it in the first place is ... ? When you post to openSUSE list everyone expects that you are using openSUSE packages. If you are using unmodified upstream version you either tell it or ask upstream.
This is expected behavior and there is no way it could "have started a couple of weeks ago" unless you have been using openSUSE kernel before.
It was exactly what Andrei said it was. Switching from kernel-vanilla to kernel-default resolved the issue. I did a few zypper ref; zypper dup and kernal-vanilla was not offered or get installed. I have no idea how the machine ended up with kernel-vanilla, but the issue of zypper ps showing deleted files in use, even after reboots, is now resolved. Regards, --Moby On 7/22/2022 21:59, Moby wrote:
Thanks Andrei.
You make valid points. In my defense:
1/ This Tumbleweed machine always updates only via zypper ref; zypper dup. I did not ever specify to use a specific kernel.
2/ I had assumed (Wrongly I know now from what you mention) that is was Tumbleweed distro that was switching to a vanilla kernel.
3/ I did not know this was expected behavior. This behavior did indeed start after a few updates ago, the updates again were via zypper ref; zypper dup
4/ Would zypper dup not put me back on openSUSE packages, including the kernel? Doing a zypper ref; zypper dup, I do not see any kernel packages being offered.
I will browse the openSUSE Tumbleweed repo and see what kernel it offers, though now I am a bit concerned as to how this kernel got on the machine to begin with.
Regards,
--Moby
On 7/22/2022 16:06, Andrei Borzenkov wrote:
On 22.07.2022 23:55, Moby wrote:
Thanks Andrei, I guess my eyes are not what they used to be!
On 7/22/2022 15:37, Andrei Borzenkov wrote:
On 22.07.2022 20:54, Moby wrote:
From what I gathered from the link provided by Andrei (https://github.com/lsof-org/lsof/blob/bcc125b7c2a160da019dd8089b999bc6d34866...),
zypper ps may have issues if the device or inode or both of a file do not match. I chased one of the files (/usr/lib64/libaudit.so.1.0.0) and find that the device and inode match between /proc/1270/maps and stat.
No, they do not.
cat /proc/1270/maps ... 7f671edd2000-7f671edd6000 r--p 00000000 00:20>> 39055104 /usr/lib64/libaudit.so.1.0.0 ... stat /usr/lib64/libaudit.so.1.0.0 File: /usr/lib64/libaudit.so.1.0.0 Size: 116800 Blocks: 232 IO Block: 4096 regular file Device: 0,35 Inode: 39055104 Links: 1 So this is device 0,32 in /proc/.../maps and device 0,35 in stat. Yes, you are correct, /proc/1270/maps shows the file as 0,32 while stat shows 0,35
Which kernel and root filesystem are you using (more precisely - what filesystem where /usr/lib64 is located)? It is a vanilla kernel:
uname -a Linux esilinux2.ad.techspace.cc 5.18.11-1-vanilla #1 SMP PREEMPT_DYNAMIC Fri Jul 15 05:36:11 UTC 2022 (4fcb983) x86_64 x86_64 x86_64 GNU/Linux
And the reason you never mentioned it in the first place is ... ? When you post to openSUSE list everyone expects that you are using openSUSE packages. If you are using unmodified upstream version you either tell it or ask upstream.
This is expected behavior and there is no way it could "have started a couple of weeks ago" unless you have been using openSUSE kernel before.
On 23.07.2022 07:50, Moby wrote:
It was exactly what Andrei said it was. Switching from kernel-vanilla to kernel-default resolved the issue. I did a few zypper ref; zypper dup and kernal-vanilla was not offered or get installed. I have no idea how the machine ended up with kernel-vanilla, but the issue of zypper ps showing deleted files in use, even after reboots, is now resolved.
Actually it probably still warrants bug report against zypper. When file is deleted, lsof reports file type as DEL (even on kernel without SUSE specific patches). If file was just renamed then lsof will report the new file name. May be zypper should filter out files without explicit link count. Without btrfs the only possibility to get into this situation is to mount something over original mount point. sleep 166769 root mem REG 8,4 2112342 /mnt/sub1/libc.so.6 (path dev=0,64, inode=259) It is questionable, whether this should be interpreted as "file was deleted". If original, "hidden", file is deleted, output with the same "overmount" is sleep 166769 root DEL REG 8,4 2112342 /mnt/sub1/libc.so.6 Notice file type "DEL", not "mem".
On 23/07/2022 06.50, Moby wrote: ...
I have no idea how the machine ended up with kernel-vanilla, but the issue of zypper ps showing deleted files in use, even after reboots, is now resolved.
Check repository list (zypper lr --details). There are logs you should check before they get rotated out: /var/log/zypper.log /var/log/zypp/history /var/log/YaST2/y2log* -- Cheers / Saludos, Carlos E. R. (from openSUSE 15.3 (Legolas))
Thanks Carlos. Looking at /var/log/zypper.log, I see: 2022-05-26 21:38:25|install|kernel-vanilla|5.17.9-1.1|x86_64||openSUSE_Tumbleweed|21f308ace2c0a9e56898741d54ef52062750aa5c665f685e579d9cb7b57ac7c8| # 2022-05-26 21:39:09 kernel-default-base-5.17.9-1.1.24.15.x86_64.rpm installed ok # Additional rpm output: # dracut: Executing: /usr/bin/dracut -f /boot/initrd-5.17.9-1-default 5.17.9-1-default Looks like kernel-vanilla was installed on 2022-05-26. Looking at zypper lr, I see nothing to show where the vanilla kernel could have come from. Regards, --Moby On 7/23/2022 12:49, Carlos E. R. wrote:
On 23/07/2022 06.50, Moby wrote:
...
I have no idea how the machine ended up with kernel-vanilla, but the issue of zypper ps showing deleted files in use, even after reboots, is now resolved.
Check repository list (zypper lr --details).
There are logs you should check before they get rotated out:
/var/log/zypper.log /var/log/zypp/history
/var/log/YaST2/y2log*
On 23/07/2022 21.30, Moby wrote:
Thanks Carlos.
Looking at /var/log/zypper.log, I see:
2022-05-26 21:38:25|install|kernel-vanilla|5.17.9-1.1|x86_64||openSUSE_Tumbleweed|21f308ace2c0a9e56898741d54ef52062750aa5c665f685e579d9cb7b57ac7c8|
# 2022-05-26 21:39:09 kernel-default-base-5.17.9-1.1.24.15.x86_64.rpm installed ok # Additional rpm output: # dracut: Executing: /usr/bin/dracut -f /boot/initrd-5.17.9-1-default 5.17.9-1-default
Looks like kernel-vanilla was installed on 2022-05-26.
Looking at zypper lr, I see nothing to show where the vanilla kernel could have come from.
You also have to verify the URLs, which is why I said "... --details" -- Cheers / Saludos, Carlos E. R. (from openSUSE 15.3 (Legolas))
If I am correct in reading the log entry below, I presume it shows that kernel-vanilla was installed from the openSUSE_Tumbleweed repo. zypper lr --details shows that repo at http://download.opensuse.org/tumbleweed/repo/oss/ In there I do see kernel-vanilla. I suppose that is the repo where zypper dup picked up kernel-vanilla from at some point in the past? Regards, --Moby On 7/23/2022 15:37, Carlos E. R. wrote:
On 23/07/2022 21.30, Moby wrote:
Thanks Carlos.
Looking at /var/log/zypper.log, I see:
2022-05-26 21:38:25|install|kernel-vanilla|5.17.9-1.1|x86_64||openSUSE_Tumbleweed|21f308ace2c0a9e56898741d54ef52062750aa5c665f685e579d9cb7b57ac7c8|
# 2022-05-26 21:39:09 kernel-default-base-5.17.9-1.1.24.15.x86_64.rpm installed ok # Additional rpm output: # dracut: Executing: /usr/bin/dracut -f /boot/initrd-5.17.9-1-default 5.17.9-1-default
Looks like kernel-vanilla was installed on 2022-05-26.
Looking at zypper lr, I see nothing to show where the vanilla kernel could have come from.
You also have to verify the URLs, which is why I said "... --details"
On 23/07/2022 23.36, Moby wrote:
If I am correct in reading the log entry below, I presume it shows that kernel-vanilla was installed from the openSUSE_Tumbleweed repo. zypper lr --details shows that repo at http://download.opensuse.org/tumbleweed/repo/oss/
In there I do see kernel-vanilla. I suppose that is the repo where zypper dup picked up kernel-vanilla from at some point in the past?
It must be. My knowledge ends about here :-) -- Cheers / Saludos, Carlos E. R. (from openSUSE 15.3 (Legolas))
participants (5)
-
Andrei Borzenkov -
Carlos E. R. -
Carlos E. R. -
Moby -
Robert Webb