[opensuse-support] [15.1] nvme device names change while system running
[Leap 15.1, xfce, kernel 5.2.11-1.g6385110-default] I have a machine with 2 nvme drives, 1 in the built-in motherboard m.2 slot (grub2 boots 15.1,"SX8200NP"), other in an m.2 slot on a pci-e expansion card (boots 15.0,"SX7000NP"). The dev names of the 2 nvmes seem to keep changing within the same boot. Here's an example from the logs with the mb nvme first called nvme0 by the kernel, then nvme1 by btrfs, then nvme0 by both smart and btrfs. But the device according to yast is still nvme1. Confusing, to say the least. I need someone smarter than me (which is just about anyone) to tell me what is going on here. What have I done :( Devices according to yast2 (partitioner): Device: /dev/nvme1n1 Device Path: pci-0000:02:00.0-nvme-1 Device ID 1: nvme-ADATA_SX8200NP_2I4520013773 Device: /dev/nvme0n1 Device Path: pci-0000:05:00.0-nvme-1 Device ID 1: nvme-ADATA_SX7000NP_2H4220054168 according to the journal, in chronological order: kernel: nvme nvme0: pci function 0000:02:00.0 kernel: nvme nvme1: pci function 0000:05:00.0 BTRFS: device label openSUSE-15 devid 1 transid 778057 /dev/nvme0n1p3 BTRFS: device label openSUSE-15.1 devid 1 transid 37777 /dev/nvme1n1p4 (note: if not apparent, this below is / on SX8200NP) kernel: BTRFS info (device nvme1n1p4): disk space caching is enabled kernel: BTRFS info (device nvme1n1p4): has skinny extents kernel: BTRFS info (device nvme1n1p4): enabling ssd optimizations kernel: BTRFS info (device nvme1n1p4): disk space caching is enabled smartd[1494]: Device: /dev/nvme0, opened smartd[1494]: Device: /dev/nvme0, ADATA SX8200NP, S/N:2I4520013773, FW:SVN139B smartd[1494]: Device: /dev/nvme1, opened smartd[1494]: Device: /dev/nvme1, ADATA SX7000NP, S/N:2H4220054168, FW:CB1.1.1 (note: if not apparent, this below is / on SX7000NP and the journal entry appears to be triggered by snapper. Why is snapper looking at something other than the running system? Is it confused, too?) kernel: BTRFS info (device nvme0n1p3): disk space caching is enabled kernel: BTRFS info (device nvme0n1p3): has skinny extents kernel: BTRFS info (device nvme0n1p3): enabling ssd optimizations kernel: BTRFS info (device nvme0n1p3): disk space caching is enabled kernel: BTRFS info (device nvme0n1p3): has skinny extents kernel: BTRFS info (device nvme0n1p3): enabling ssd optimizations Thanks. Ralph -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
02.09.2019 16:13, Ralph пишет:
[Leap 15.1, xfce, kernel 5.2.11-1.g6385110-default]
I have a machine with 2 nvme drives, 1 in the built-in motherboard m.2 slot (grub2 boots 15.1,"SX8200NP"), other in an m.2 slot on a pci-e expansion card (boots 15.0,"SX7000NP"). The dev names of the 2 nvmes seem to keep changing within the same boot. Here's an example from the logs
Show full logs from boot (journalctl -b).
with the mb nvme first called nvme0 by the kernel, then nvme1 by btrfs, then nvme0 by both smart and btrfs. But the device according to yast is still nvme1. Confusing, to say the least. I need someone smarter than me (which is just about anyone) to tell me what is going on here. What have I done :(
Devices according to yast2 (partitioner):
Device: /dev/nvme1n1 Device Path: pci-0000:02:00.0-nvme-1 Device ID 1: nvme-ADATA_SX8200NP_2I4520013773
Device: /dev/nvme0n1 Device Path: pci-0000:05:00.0-nvme-1 Device ID 1: nvme-ADATA_SX7000NP_2H4220054168
according to the journal, in chronological order:
kernel: nvme nvme0: pci function 0000:02:00.0 kernel: nvme nvme1: pci function 0000:05:00.0
BTRFS: device label openSUSE-15 devid 1 transid 778057 /dev/nvme0n1p3 BTRFS: device label openSUSE-15.1 devid 1 transid 37777 /dev/nvme1n1p4
(note: if not apparent, this below is / on SX8200NP) kernel: BTRFS info (device nvme1n1p4): disk space caching is enabled kernel: BTRFS info (device nvme1n1p4): has skinny extents kernel: BTRFS info (device nvme1n1p4): enabling ssd optimizations kernel: BTRFS info (device nvme1n1p4): disk space caching is enabled
smartd[1494]: Device: /dev/nvme0, opened smartd[1494]: Device: /dev/nvme0, ADATA SX8200NP, S/N:2I4520013773, FW:SVN139B
smartd[1494]: Device: /dev/nvme1, opened smartd[1494]: Device: /dev/nvme1, ADATA SX7000NP, S/N:2H4220054168, FW:CB1.1.1
(note: if not apparent, this below is / on SX7000NP and the journal entry appears to be triggered by snapper. Why is snapper looking at something other than the running system? Is it confused, too?)
snapper is mentioned nowhere in lines you posted.
kernel: BTRFS info (device nvme0n1p3): disk space caching is enabled kernel: BTRFS info (device nvme0n1p3): has skinny extents kernel: BTRFS info (device nvme0n1p3): enabling ssd optimizations kernel: BTRFS info (device nvme0n1p3): disk space caching is enabled kernel: BTRFS info (device nvme0n1p3): has skinny extents kernel: BTRFS info (device nvme0n1p3): enabling ssd optimizations
Thanks.
Ralph
-- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
Andrei, thanks for the response and excuse the delay. My replies inline below... On Mon, 2 Sep 2019 20:53:08 +0300 Andrei Borzenkov <arvidjaar@gmail.com> wrote:
02.09.2019 16:13, Ralph пишет:
[Leap 15.1, xfce, kernel 5.2.11-1.g6385110-default]
I have a machine with 2 nvme drives, 1 in the built-in motherboard m.2 slot (grub2 boots 15.1,"SX8200NP"), other in an m.2 slot on a pci-e expansion card (boots 15.0,"SX7000NP"). The dev names of the 2 nvmes seem to keep changing within the same boot. Here's an example from the logs
Show full logs from boot (journalctl -b).
Original journal under discussion is gone because apparently the default of a new 15.1 install is to not write journals to permanent storage. Bad idea, imho. Anyway, fixed now. Current log is at: https://paste.opensuse.org/9a901053 with same nvme device number problem as before. Note that journal is truncated at point of first user info as I log some file accesses and etc and I would need to edit it for privacy before posting beyond this point. If you need anything beyond this point in the journal, advise. Note that yast2 info on the nvme devices is the same now as given below.
with the mb nvme first called nvme0 by the kernel, then nvme1 by btrfs, then nvme0 by both smart and btrfs. But the device according to yast is still nvme1. Confusing, to say the least. I need someone smarter than me (which is just about anyone) to tell me what is going on here. What have I done :(
Devices according to yast2 (partitioner):
Device: /dev/nvme1n1 Device Path: pci-0000:02:00.0-nvme-1 Device ID 1: nvme-ADATA_SX8200NP_2I4520013773
Device: /dev/nvme0n1 Device Path: pci-0000:05:00.0-nvme-1 Device ID 1: nvme-ADATA_SX7000NP_2H4220054168
according to the journal, in chronological order:
kernel: nvme nvme0: pci function 0000:02:00.0 kernel: nvme nvme1: pci function 0000:05:00.0
BTRFS: device label openSUSE-15 devid 1 transid 778057 /dev/nvme0n1p3 BTRFS: device label openSUSE-15.1 devid 1 transid 37777 /dev/nvme1n1p4
(note: if not apparent, this below is / on SX8200NP) kernel: BTRFS info (device nvme1n1p4): disk space caching is enabled kernel: BTRFS info (device nvme1n1p4): has skinny extents kernel: BTRFS info (device nvme1n1p4): enabling ssd optimizations kernel: BTRFS info (device nvme1n1p4): disk space caching is enabled
smartd[1494]: Device: /dev/nvme0, opened smartd[1494]: Device: /dev/nvme0, ADATA SX8200NP, S/N:2I4520013773, FW:SVN139B
smartd[1494]: Device: /dev/nvme1, opened smartd[1494]: Device: /dev/nvme1, ADATA SX7000NP, S/N:2H4220054168, FW:CB1.1.1
(note: if not apparent, this below is / on SX7000NP and the journal entry appears to be triggered by snapper. Why is snapper looking at something other than the running system? Is it confused, too?)
snapper is mentioned nowhere in lines you posted.
In the original journal of that day, snapper process was wrapped around the journal entry and appeared to be triggering those entries. Not evident in the current journal.
kernel: BTRFS info (device nvme0n1p3): disk space caching is enabled kernel: BTRFS info (device nvme0n1p3): has skinny extents kernel: BTRFS info (device nvme0n1p3): enabling ssd optimizations kernel: BTRFS info (device nvme0n1p3): disk space caching is enabled kernel: BTRFS info (device nvme0n1p3): has skinny extents kernel: BTRFS info (device nvme0n1p3): enabling ssd optimizations
Thanks.
Ralph
-- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
On Fri, Sep 06, 2019 at 09:17:21AM -0500, Ralph wrote:
Show full logs from boot (journalctl -b).
Original journal under discussion is gone because apparently the default of a new 15.1 install is to not write journals to permanent storage. Bad idea, imho. Anyway, fixed now.
Non-persistence of the journal has been default for a while in openSUSE and other distros. So far this has bitten me/colleagues twice on 15.0 as well Linux Mint 19.1, Debian 10, RHEL 7 and Debian 9 - buster(deb10) just two nights ago after a machine needed its RAM reseating and we went looking for 'previous signs of wobbliness'[TM] to find half the uptime had been lost from the ring buffer (verbose debug was on) and no trace of earlier boots. I don't know the rationale for non-persistence, or install scripts not doing 'mkdir /var/log/journal'; but from experience we've only noticed after problems. 'journalctl -b -1' has been a really useful feature of systemd/journald, for me. Daniel -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
On 06/09/2019 17.10, Daniel Morris wrote:
On Fri, Sep 06, 2019 at 09:17:21AM -0500, Ralph wrote:
Show full logs from boot (journalctl -b).
Original journal under discussion is gone because apparently the default of a new 15.1 install is to not write journals to permanent storage. Bad idea, imho. Anyway, fixed now.
Non-persistence of the journal has been default for a while in openSUSE and other distros.
However, you may find that syslog has saved the log to /var/log/messages. I'm not fully sure if this is by default, or that I installed rsyslog manually (it is in my notes to install it if it not selected for install already). But AFAIR, it is installed by default. The only problem in this case is locating boots and halts.
<0.5> 2019-09-02 02:16:57 Telcontar kernel - - - [307305.944438] XFS (dm-0): Unmounting Filesystem 2019-09-02 02:16:58+02:00 - Halting the system now =========================================== uptime: 02:16:58 up 6 days 14:28, 1 user, load average: 3.12, 1.50, 1.01 2019-09-02 12:14:14+02:00 - Booting the system now ================================================================================ Linux Telcontar 4.12.14-lp151.28.13-default #1 SMP Wed Aug 7 07:20:16 UTC 2019 (0c09ad2) x86_64 x86_64 x86_64 GNU/Linux
I have a service that adds those two lines at halt/boot so that can easiliy detect them. You will have to search for the line that contains: "Command line: BOOT_IMAG" instead.
<0.6> 2019-09-02 12:14:15 Telcontar kernel - - - [ 0.000000] microcode: microcode updated early to revision 0xa0b, date = 2010-09-28 <0.5> 2019-09-02 12:14:15 Telcontar kernel - - - [ 0.000000] Linux version 4.12.14-lp151.28.13-default (geeko@buildhost) (gcc version 7.4.0 (SUSE Linux) ) #1 SMP Wed Aug 7 07:20:16 UT C 2019 (0c09ad2) <0.6> 2019-09-02 12:14:15 Telcontar kernel - - - [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.12.14-lp151.28.13-default root=UUID=ac173013-18ad-4c4e-921e-fd2ecfb56495 resume=/dev/d isk/by-label/ssd-swap splash=verbose
-- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
On Fri, 6 Sep 2019 20:25:55 +0200 "Carlos E. R." <robin.listas@telefonica.net> wrote:
On 06/09/2019 17.10, Daniel Morris wrote:
On Fri, Sep 06, 2019 at 09:17:21AM -0500, Ralph wrote:
Show full logs from boot (journalctl -b).
Original journal under discussion is gone because apparently the default of a new 15.1 install is to not write journals to permanent storage. Bad idea, imho. Anyway, fixed now.
Non-persistence of the journal has been default for a while in openSUSE and other distros.
However, you may find that syslog has saved the log to /var/log/messages. I'm not fully sure if this is by default, or that I installed rsyslog manually (it is in my notes to install it if it not selected for install already). But AFAIR, it is installed by default.
Indeed it is there. And archives too. Thanks Carlos. I am clearly blind (or senile). So, can you also please tell me why this system can't keep the nvme dev numbers straight? :) Ralph -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
On 07/09/2019 05.38, Ralph wrote:
On Fri, 6 Sep 2019 20:25:55 +0200 "Carlos E. R." <> wrote:
On 06/09/2019 17.10, Daniel Morris wrote:
On Fri, Sep 06, 2019 at 09:17:21AM -0500, Ralph wrote:
Show full logs from boot (journalctl -b).
Original journal under discussion is gone because apparently the default of a new 15.1 install is to not write journals to permanent storage. Bad idea, imho. Anyway, fixed now.
Non-persistence of the journal has been default for a while in openSUSE and other distros.
However, you may find that syslog has saved the log to /var/log/messages. I'm not fully sure if this is by default, or that I installed rsyslog manually (it is in my notes to install it if it not selected for install already). But AFAIR, it is installed by default.
Indeed it is there. And archives too. Thanks Carlos. I am clearly blind (or senile).
So, can you also please tell me why this system can't keep the nvme dev numbers straight? :)
Nope :-) But, you can post the data Andrei asked for from that file instead. Just make a local copy on your home to edit out the other days, leaving only the day of interest, then upload to susepaste.org for a month and post here the link. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
06.09.2019 17:17, Ralph пишет:
Andrei, thanks for the response and excuse the delay. My replies inline below...
On Mon, 2 Sep 2019 20:53:08 +0300 Andrei Borzenkov <arvidjaar@gmail.com> wrote:
02.09.2019 16:13, Ralph пишет:
[Leap 15.1, xfce, kernel 5.2.11-1.g6385110-default]
I have a machine with 2 nvme drives, 1 in the built-in motherboard m.2 slot (grub2 boots 15.1,"SX8200NP"), other in an m.2 slot on a pci-e expansion card (boots 15.0,"SX7000NP"). The dev names of the 2 nvmes seem to keep changing within the same boot. Here's an example from the logs
Show full logs from boot (journalctl -b).
Original journal under discussion is gone because apparently the default of a new 15.1 install is to not write journals to permanent storage. Bad idea, imho. Anyway, fixed now. Current log is at: https://paste.opensuse.org/9a901053 with same nvme device number problem as before. Note that journal is truncated at point of first user info as I log some file accesses and etc and I would need to edit it for privacy before posting beyond this point. If you need anything beyond this point in the journal, advise.
Unfortunately this does not really allows to identify nvme at various point in times. I'm still not sure what the actual problem is.
Note that yast2 info on the nvme devices is the same now as given below.
with the mb nvme first called nvme0 by the kernel, then nvme1 by btrfs, then nvme0 by both smart and btrfs. But the device according to yast is still nvme1. Confusing, to say the least. I need someone
Try showing output of lspci -nn ls -l /sys/block blkid nvme list nvme list-ns /dev/nvme0 nvme list-ns /dev/nvme1 -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
participants (4)
-
Andrei Borzenkov -
Carlos E. R. -
Daniel Morris -
Ralph