[Bug 619030] New: sata_sil24 pci passthrough fails after upgrade from 11.2 -> 11.3
http://bugzilla.novell.com/show_bug.cgi?id=619030 http://bugzilla.novell.com/show_bug.cgi?id=619030#c0 Summary: sata_sil24 pci passthrough fails after upgrade from 11.2 -> 11.3 Classification: openSUSE Product: openSUSE 11.3 Version: Factory Platform: x86-64 OS/Version: openSUSE 11.3 Status: NEW Severity: Major Priority: P5 - None Component: Xen AssignedTo: jdouglas@novell.com ReportedBy: 0.bugs.only.0@gmail.com QAContact: qa@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.6) Gecko/20100626 SUSE/3.6.6-1.1 Firefox/3.6.6 i've upgraded a test Xen server host from opensuse 11.2 -> 11.3. currently, i've, uname -a Linux devtest 2.6.34-5-xen #1 SMP 2010-06-29 02:39:08 +0200 x86_64 x86_64 x86_64 GNU/Linux lsb_release -a LSB Version: core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch Distributor ID: SUSE LINUX Description: openSUSE 11.3 (x86_64) Release: 11.3 Codename: n/a with, rpm -qa | grep -i ^xen xen-4.0.0_21091_05-67.2.x86_64 xen-libs-4.0.0_21091_05-67.2.x86_64 xen-tools-4.0.0_21091_05-67.2.x86_64 on this box, I pass through three PCI cards to DomUs, 02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01) 04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 02) 05:07.0 RAID bus controller: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller (rev 02) with, cat /boot/brub/menu.lst, ... title Xen4 root (hd0,0) kernel /xen.gz loglvl=all loglvl_guest=all apic_verbosity=debug dom0_mem=1024M ... iommu=soft dom0_max_vcpus=4 dom0_vcpus_pin=true sched=credit cpuidle=1 cpufreq=xen clocksource=acpi numa=on module /vmlinuz-xen root=/dev/VG0/ROOT rootfstype=ext4 rootflags=journal_checksum resume=/dev/VG0/SWAP ... pciback.permissive pciback.hide=(05:07.0)(02:00.0)(04:00.0) iommu=soft noirqdebug clocksource=xen mce=off module /initrd-xen ... and, cat /etc/modprobe.d/99-local.conf ... alias scsi_hostadapter2 sata_sil24 ... alias eth1 r8169 alias eth2 r8169 ... install r8169 /sbin/modprobe pciback ; /sbin/modprobe --first-time --ignore-install r8169 install sata_sil24 /sbin/modprobe pciback ; /sbin/modprobe --first-time --ignore-install sata_sil24 under 11.2, all three PCI cards are properly passed-thru dmesg | grep seiz [ 16.161302] pciback 0000:02:00.0: seizing device [ 16.161460] pciback 0000:04:00.0: seizing device [ 16.161962] pciback 0000:05:07.0: seizing device and their respective DomU can see/use the cards as expected. i.e., works perfectly. under 11.3, however, only the Ethernet cards are seized ... dmesg | grep seiz [ 14.142426] pciback 0000:02:00.0: seizing device [ 14.142512] pciback 0000:04:00.0: seizing device i.e, the sata_sil24 card is not. as expected, DomU can't see/use the sata_sil24 card. @ logs in the 11.3 case, xm dmesg | egrep "02:00.0|04:00.0|05:07.0|sata_sil24" (XEN) [2010-07-01 01:44:14] PCI add device 05:07.0 (XEN) [2010-07-01 01:44:27] PCI add device 02:00.0 (XEN) [2010-07-01 01:44:27] PCI add device 04:00.0 dmesg | egrep "02:00.0|04:00.0|05:07.0|sata_sil24" [ 0.000000] Command line: root=/dev/VG0/ROOT rootfstype=ext4 rootflags=journal_checksum resume=/dev/VG0/SWAP showopts splash=silent selinux=0 SELINUX_INIT=NO apparmor=0 elevator=cfq vga=0x31a console=tty0 console=xvc0,57600 pciback.permissive pciback.hide=(05:07.0)(02:00.0)(04:00.0) iommu=soft noirqdebug clocksource=xen mce=off [ 0.000000] Kernel command line: root=/dev/VG0/ROOT rootfstype=ext4 rootflags=journal_checksum resume=/dev/VG0/SWAP showopts splash=silent selinux=0 SELINUX_INIT=NO apparmor=0 elevator=cfq vga=0x31a console=tty0 console=xvc0,57600 pciback.permissive pciback.hide=(05:07.0)(02:00.0)(04:00.0) iommu=soft noirqdebug clocksource=xen mce=off [ 0.195383] pci 0000:02:00.0: reg 10: [io 0xb800-0xb8ff] [ 0.195404] pci 0000:02:00.0: reg 18: [mem 0xfbcff000-0xfbcfffff 64bit] [ 0.195426] pci 0000:02:00.0: reg 30: [mem 0xfbcc0000-0xfbcdffff pref] [ 0.195474] pci 0000:02:00.0: supports D1 D2 [ 0.195475] pci 0000:02:00.0: PME# supported from D1 D2 D3hot D3cold [ 0.195480] pci 0000:02:00.0: PME# disabled [ 0.195505] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' [ 0.195805] pci 0000:04:00.0: reg 10: [io 0xd800-0xd8ff] [ 0.195826] pci 0000:04:00.0: reg 18: [mem 0xfafff000-0xfaffffff 64bit pref] [ 0.195840] pci 0000:04:00.0: reg 20: [mem 0xfafe0000-0xfafeffff 64bit pref] [ 0.195847] pci 0000:04:00.0: reg 30: [mem 0xfbef0000-0xfbefffff pref] [ 0.195886] pci 0000:04:00.0: supports D1 D2 [ 0.195888] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.195892] pci 0000:04:00.0: PME# disabled [ 0.200135] pci 0000:05:07.0: reg 10: [mem 0xfbfffc00-0xfbfffc7f 64bit] [ 0.200153] pci 0000:05:07.0: reg 18: [mem 0xfbff0000-0xfbff7fff 64bit] [ 0.200163] pci 0000:05:07.0: reg 20: [io 0xec00-0xec0f] [ 0.200181] pci 0000:05:07.0: reg 30: [mem 0xfbf00000-0xfbf7ffff pref] [ 0.200224] pci 0000:05:07.0: supports D1 D2 [ 2.620898] sata_sil24 0000:05:07.0: version 1.1 [ 2.620933] sata_sil24 0000:05:07.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21 [ 2.621549] scsi8 : sata_sil24 [ 2.621705] scsi9 : sata_sil24 [ 2.621904] scsi10 : sata_sil24 [ 2.622054] scsi11 : sata_sil24 [ 16.161302] pciback 0000:02:00.0: seizing device [ 16.161460] pciback 0000:04:00.0: seizing device [ 16.161668] pciback 0000:04:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18 [ 16.161808] pciback 0000:04:00.0: PCI INT A disabled [ 16.161984] pciback 0000:02:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18 [ 16.162127] pciback 0000:02:00.0: PCI INT A disabled [ 57.677589] pciback: vpci: 0000:02:00.0: assign to virtual slot 0 [ 57.684327] pciback: vpci: 0000:04:00.0: assign to virtual slot 1 [ 67.298144] pciback 0000:02:00.0: enabling device (0000 -> 0003) [ 67.298167] pciback 0000:02:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18 [ 67.298886] pciback 0000:02:00.0: setting latency timer to 64 [ 67.299518] pciback 0000:02:00.0: get owner: 1 [ 67.299564] pciback 0000:02:00.0: irq 55 for dom1 MSI/MSI-X [ 67.314695] pciback 0000:04:00.0: enabling device (0000 -> 0003) [ 67.314702] pciback 0000:04:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18 [ 67.314983] pciback 0000:04:00.0: setting latency timer to 64 [ 67.316312] pciback 0000:04:00.0: get owner: 1 [ 67.316336] pciback 0000:04:00.0: irq 54 for dom1 MSI/MSI-X [ 78.359249] pciback 0000:02:00.0: Driver tried to write to a read-only configuration space field at offset 0x68, size 2. This may be harmless, but if you have problems with your device: shows no oops. so, unclear if it's a bug, a req'd config change, etc. if add'l info is req'd -- happy to provide it. Reproducible: Always Steps to Reproduce: 1. 2. 3. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=619030 http://bugzilla.novell.com/show_bug.cgi?id=619030#c1 --- Comment #1 from mail ignored <0.bugs.only.0@gmail.com> 2010-07-01 19:29:11 PDT --- reading at, "PCI-Paththrough with PCI-Card does not work anymore with RHEL5.4 " https://bugzilla.redhat.com/show_bug.cgi?id=521346#c27 with the setup above, after boot, dmesg | grep -i seiz [ 16.220175] pciback 0000:02:00.0: seizing device [ 16.220479] pciback 0000:04:00.0: seizing device xm pci-list-assignable-devices 0000:04:00.0 0000:02:00.0 then, once booted, @shell, echo -n 0000:05:07.0 > /sys/bus/pci/drivers/sata_sil24/unbind echo -n 0000:05:07.0 > /sys/bus/pci/drivers/pciback/new_slot echo -n 0000:05:07.0 > /sys/bus/pci/drivers/pciback/bind checking, dmesg | grep -i seiz [ 16.220175] pciback 0000:02:00.0: seizing device [ 16.220479] pciback 0000:04:00.0: seizing device [ 3886.476835] pciback 0000:05:07.0: seizing device xm pci-list-assignable-devices 0000:05:07.0 0000:04:00.0 0000:02:00.0 so the sata_sil24 device is *assignable* -- just not getting taken care of @ boot. testing per that thread -- just in case -- instead of as spec'd @, http://wiki.xensource.com/xenwiki/XenPCIpassthrough "If you're using Linux 2.6.31 or newer dom0 kernel based on the Novell/SLES/OpenSuse Xenlinux forward-ported patches, then you use this syntax:" pciback.permissive pciback.hide=(00:1d.7)(00:1a.0)(00:1a.1)(00:1a.7)(00:1b.0) pci=resource_alignment=00:1a.7;00:1d.7 adding to /etc/modprobe.d/99-local.conf options pciback hide=(0000:05:07.0)(0000:02:00.0)(0000:04:00.0) unfrotunately, same cirumstance -- not seized @ boot. but assignable after start using the "echo -n ..." cmds, above. now, the 'why not' ... and what's changes since 11.2 -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=619030 http://bugzilla.novell.com/show_bug.cgi?id=619030#c2 --- Comment #2 from mail ignored <0.bugs.only.0@gmail.com> 2010-07-01 20:43:58 PDT --- creating a simple startup script, cat /etc/init.d/xen.setup ... # Required-Start: $syslog $remote_fs xend xendomains ... echo -n 0000:05:07.0 > /sys/bus/pci/drivers/sata_sil24/unbind echo -n 0000:05:07.0 > /sys/bus/pci/drivers/pciback/new_slot echo -n 0000:05:07.0 > /sys/bus/pci/drivers/pciback/bind ... after boot, dmesg | egrep -i "seiz|pciback|05:07" [ 0.200140] pci 0000:05:07.0: reg 10: [mem 0xfbfffc00-0xfbfffc7f 64bit] [ 0.200158] pci 0000:05:07.0: reg 18: [mem 0xfbff0000-0xfbff7fff 64bit] [ 0.200168] pci 0000:05:07.0: reg 20: [io 0xec00-0xec0f] [ 0.200186] pci 0000:05:07.0: reg 30: [mem 0xfbf00000-0xfbf7ffff pref] [ 0.200229] pci 0000:05:07.0: supports D1 D2 [ 2.377663] sata_sil24 0000:05:07.0: version 1.1 [ 2.377698] sata_sil24 0000:05:07.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21 [ 16.256616] pciback 0000:02:00.0: seizing device [ 16.256897] pciback 0000:04:00.0: seizing device [ 16.257299] pciback 0000:04:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18 [ 16.257561] pciback 0000:04:00.0: PCI INT A disabled [ 16.257864] pciback 0000:02:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18 [ 16.258121] pciback 0000:02:00.0: PCI INT A disabled [ 51.004236] sata_sil24 0000:05:07.0: PCI INT A disabled [ 51.004368] pciback 0000:05:07.0: seizing device [ 51.004416] pciback 0000:05:07.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21 [ 51.004421] pciback 0000:05:07.0: PCI INT A disabled sata_sil24 is clearly being loaded early, then NOT unloaded -- unless/until handled via late_binding in /etc/init.d/xen.setup to prevent/cease the early load, echo "alias sata_sil24 off" >> /etc/modprobe.d/99-local.conf mkinitrd chkconfig xen.setup off shutdown -r now then, dmesg | grep -i seiz [ 7.877260] pciback 0000:02:00.0: seizing device [ 7.877401] pciback 0000:04:00.0: seizing device [ 7.877564] pciback 0000:05:07.0: seizing device dmesg | egrep -i "seiz|pciback|05:07" [ 0.200138] pci 0000:05:07.0: reg 10: [mem 0xfbfffc00-0xfbfffc7f 64bit] [ 0.200156] pci 0000:05:07.0: reg 18: [mem 0xfbff0000-0xfbff7fff 64bit] [ 0.200166] pci 0000:05:07.0: reg 20: [io 0xec00-0xec0f] [ 0.200184] pci 0000:05:07.0: reg 30: [mem 0xfbf00000-0xfbf7ffff pref] [ 0.200227] pci 0000:05:07.0: supports D1 D2 [ 7.877260] pciback 0000:02:00.0: seizing device [ 7.877401] pciback 0000:04:00.0: seizing device [ 7.877564] pciback 0000:05:07.0: seizing device [ 7.877791] pciback 0000:05:07.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21 [ 7.877922] pciback 0000:05:07.0: PCI INT A disabled [ 7.878092] pciback 0000:04:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18 [ 7.878216] pciback 0000:04:00.0: PCI INT A disabled [ 7.878372] pciback 0000:02:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18 [ 7.878497] pciback 0000:02:00.0: PCI INT A disabled seems to do the trick. for some/whatever reason, install sata_sil24 /sbin/modprobe pciback ; /sbin/modprobe --first-time --ignore-install sata_sil24 in /etc/modprobe.d/99-local.conf, sufficient/functional under 11.2, is not under 11.3 -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=619030
http://bugzilla.novell.com/show_bug.cgi?id=619030#c
Jan Beulich
http://bugzilla.novell.com/show_bug.cgi?id=619030
http://bugzilla.novell.com/show_bug.cgi?id=619030#c3
Jan Beulich
http://bugzilla.novell.com/show_bug.cgi?id=619030 http://bugzilla.novell.com/show_bug.cgi?id=619030#c4 mail ignored <0.bugs.only.0@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED InfoProvider|0.bugs.only.0@gmail.com | --- Comment #4 from mail ignored <0.bugs.only.0@gmail.com> 2010-07-05 09:34:12 PDT --- i no longer have the prior initrd. i simply documented the change in behavior that came with a straightforward upgrade from a (long-working) 11.2 setup to a (failed) 11.3 setup. in addition I provided an apparently functional workaround. there is no 'other action I expect'. if the prior way of doing things, per http://wiki.xensource.com/xenwiki/Assign_hardware_to_DomU_with_PCIBack_as_mo..., is considered fragile, np. a pointer to (other) documentation that says that may be helpful to others who experience this as well. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=619030
http://bugzilla.novell.com/show_bug.cgi?id=619030#c5
Jan Beulich
a pointer to (other) documentation that says that may be helpful to others who experience this as well.
I'm afraid there's not much we could point you to. The wiki page you refer to certainly implies a number of things (in particular that, if necessary, pciback.ko as well as the .conf file holding the modprobe directives need to be in the initrd, which I suspect may not be the case for you - otherwise I wouldn't be able to understand why this wouldn't take any effect). Can you verify that (or can you alternatively attach your 11.3 initrd)? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=619030 http://bugzilla.novell.com/show_bug.cgi?id=619030#c6 mail ignored <0.bugs.only.0@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED InfoProvider|0.bugs.only.0@gmail.com | --- Comment #6 from mail ignored <0.bugs.only.0@gmail.com> 2010-07-06 08:18:11 PDT --- grep ^INITRD_MODULES /etc/sysconfig/kernel INITRD_MODULES="raid0 ... cpuid pciback" mkinitrd cd /home/workspace/initrd rm -rf /home/workspace/initrd/* gunzip < /boot/initrd-2.6.34.1-1-xen | cpio -i --make-directories find . | grep pciback.ko ./lib/modules/2.6.34.1-1-xen/kernel/drivers/xen/pciback/pciback.ko cat etc/modprobe.d/99-local.conf | grep -i sata alias scsi_hostadapter2 sata_sil24 install sata_sil24 /sbin/modprobe pciback ; /sbin/modprobe --first-time --ignore-install sata_sil24 shutdown -r now dmesg | grep -i seiz [ 2.238296] pciback 0000:02:00.0: seizing device [ 2.238389] pciback 0000:04:00.0: seizing device [ 2.238460] pciback 0000:05:07.0: seizing device grep ^INITRD_MODULES /etc/sysconfig/kernel INITRD_MODULES="raid0 ... cpuid" mkinitrd cd /home/workspace/initrd rm -rf /home/workspace/initrd/* gunzip < /boot/initrd-2.6.34.1-1-xen | cpio -i --make-directories find . | grep pciback.ko (empty) shutdown -r now dmesg | grep -i seiz [ 19.490757] pciback 0000:02:00.0: seizing device [ 19.491052] pciback 0000:04:00.0: seizing device A missing pciback.ko seems to be the culprit. I no longer have 11.2 to check its initrd. I *can* state that my 11.2's '/etc/sysconfig/kernel' did NOT manually add 'pciback' to "INITRD_MODULES=...". In 11.3, it seems to be required for my case. pciback is clearly getting loaded during the 11.3 boot process. just not early enough -- unless spec'd in the initrd (note the timestamps @ 'seize') -- to deal with the sata_sil24 card/driver. fine for the eth NICs, however. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=619030
http://bugzilla.novell.com/show_bug.cgi?id=619030#c7
Jan Beulich
https://bugzilla.novell.com/show_bug.cgi?id=619030
https://bugzilla.novell.com/show_bug.cgi?id=619030#c8
--- Comment #8 from Jan Beulich
https://bugzilla.novell.com/show_bug.cgi?id=619030
https://bugzilla.novell.com/show_bug.cgi?id=619030#c9
Michal Marek
https://bugzilla.novell.com/show_bug.cgi?id=619030
https://bugzilla.novell.com/show_bug.cgi?id=619030#c10
Jan Beulich
participants (1)
-
bugzilla_noreply＠novell.com