[Bug 463596] New: PCI Pass-Through no longer working for XEN DomU in 11.1
https://bugzilla.novell.com/show_bug.cgi?id=463596 User wolf@womaro.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c1 Summary: PCI Pass-Through no longer working for XEN DomU in 11.1 Product: openSUSE 11.1 Version: Final Platform: 64bit OS/Version: openSUSE 11.1 Status: NEW Severity: Major Priority: P5 - None Component: Xen AssignedTo: cgriffin@novell.com ReportedBy: wolf@womaro.ch QAContact: qa@suse.de CC: wolf@womaro.ch Found By: Customer After an DomU upgrade from OpenSuSE 10.3 to OpenSuSE 11.1 the PCI Pass-Through for the DomU stopped to work. The XEN-Dom0 in both cases is OpenSuSE 11.1, Kernel "2.6.27.7-9-xen #1 SMP 2008-12-04 18:10:04 +0100 x86_64". I've attached both Boot logs - 2.6.22 XEN-DomU kernel works perfectly, 2.6.27 XEN-DomU kernel doesn't. The 2.6.27 kernel log is from the last boot with a factory kernel - the "vanilla XEN kernel" of 11.1 behaves exactly the same. The only change between the two DomU is the kernel & initrd (installed via rpm -i from the commandline) - nothing else was changed, neither in the Dom0 nor in the DomU. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User wolf@womaro.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c1 --- Comment #1 from Wolf Geldmacher <wolf@womaro.ch> 2009-01-05 09:41:55 MST --- Created an attachment (id=263228) --> (https://bugzilla.novell.com/attachment.cgi?id=263228) DomU kernel logs 2.6.22 / 2.6.27 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User frank.mueller@rhoen.de added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c2 frank mueller <frank.mueller@rhoen.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |frank.mueller@rhoen.de --- Comment #2 from frank mueller <frank.mueller@rhoen.de> 2009-01-07 15:04:57 MST --- Wolf, am i right, the problem is that you dont get interrupts in your domU ? Can you post output from domU cat /proc/interrupt. I am experiencing the same with xen from 11.0 and xen from 11.1, no interrupts passed to domU. Thanks Frank -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User frank.mueller@rhoen.de added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c3 --- Comment #3 from frank mueller <frank.mueller@rhoen.de> 2009-01-07 15:33:53 MST --- I have opened BUG 464252 before i have seen this bugreport. Looks this is the same issue like i have described in 464252. booting the domu with 2.6.22.5-31-xenpae kernel brings interrupts back to domu, booting with xen kernel from 11.1: no interrupt in domU. Frank -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User lbendixs@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c4 --- Comment #4 from Lynn Bendixsen <lbendixs@novell.com> 2009-01-12 09:24:18 MST --- *** Bug 464252 has been marked as a duplicate of this bug. *** https://bugzilla.novell.com/show_bug.cgi?id=464252 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 Lynn Bendixsen <lbendixs@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lbendixs@novell.com AssignedTo|cgriffin@novell.com |jbeulich@novell.com -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c5 Jan Beulich <jbeulich@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |wolf@womaro.ch --- Comment #5 from Jan Beulich <jbeulich@novell.com> 2009-01-13 02:02:19 MST --- There are several "Failed to obtain physical IRQ ..." messages in the 2.6.22 based log as well, so I'd want to clarify what device(s) you successfully passed through in 2.6.22 that you can't pass through anymore in 2.6.27. I also understand that the logs you provided are from the DomU only. We'd need to see hypervisor and Dom0 logs, too. All of that should perhaps be done after a (presumably 2.6.27.10 based) kernel update became available, as a few fixes to the interrupt handling logic were done, and with SLE11 testing (mostly the same code base) we haven't got reports about any issues in this area recently. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User wolf@womaro.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c6 --- Comment #6 from Wolf Geldmacher <wolf@womaro.ch> 2009-01-16 14:15:42 MST --- @Jan: There are "Failed to obtain IRQ" messages in the log from 2.6.22 - but that does not prevent the IRQs from being obtained in the DomU. Devices are the USB controllers on the motherboard (all of them and fully). I'm currently back on runnning the working 2.6.22, but will retry (and at that time also provide the additional logs you requested) when 2.6.27.10 is available. Any idea on when this will be? @Frank: Sorry for not responding - was afk and had to / still have to keep the DomU up for production. Will retry with the new kernel though and then also post /proc/interrupts. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c7 --- Comment #7 from Jan Beulich <jbeulich@novell.com> 2009-01-19 01:17:08 MST --- (In reply to comment #6)
There are "Failed to obtain IRQ" messages in the log from 2.6.22 - but that does not prevent the IRQs from being obtained in the DomU.
Hmm, that doesn't make much sense. Are you sure the driver doesn't use some polling mode? Anyway, it'll be clarified once we see /proc/interrupts for both kernel versions...
I'm currently back on runnning the working 2.6.22, but will retry (and at that time also provide the additional logs you requested) when 2.6.27.10 is available. Any idea on when this will be?
No, sorry. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User wolf@womaro.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c8 Wolf Geldmacher <wolf@womaro.ch> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|wolf@womaro.ch | --- Comment #8 from Wolf Geldmacher <wolf@womaro.ch> 2009-02-02 03:13:21 MST --- Created an attachment (id=269189) --> (https://bugzilla.novell.com/attachment.cgi?id=269189) Collected logs as requested I just re-tested with 2.6.27.10 from the factory - the behaviour is unchanged, i.e. no interrupts passed through to the DomU. Attached the logs as a gzipped tar archive: logs/configs -- Dom0 configuration files used to delegate PCI ressources to DomU; DomU XEN config logs/typescript-xenguest* -- "dmesg" output and "cat /proc/interrupts" output from freshly booted DomU logs/xenhostlogs/typescript-xenhost* -- output of "lspci -vv" and "cat /proc/interrupts" from Dom0 logs/xenhostlogs/2.6.XX.yy/* -- copy of logs from /var/log/xen that had modification timestamps close to the time of test. Again: The *only* thing changed for the test is the DomU kernel (packages kernel-xen, kernel-xen-base and kernel-xen-extra); Booting with the 2.6.22 kernel allows access to the hardware, booting with 2.6.27 does not. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c9 Jan Beulich <jbeulich@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED --- Comment #9 from Jan Beulich <jbeulich@novell.com> 2009-02-03 03:11:08 MST --- So, the 2.6.22 log indicates that only IRQ23 was successfully obtained, and only that IRQ actually does see activity. All other IRQs are also in use by devices in Dom0, which apparently results in them being unavailable for the DomU (although the intention is for the two domains to be permitted to share them). Judging is going to be impossible until you also provide hypervisor logs for both cases. Additionally providing the Dom0 kernel logs certainly wouldn't hurt. However, based on the IRQ setup failing with -ENOSYS I think I spotted a case that indeed isn't properly handled in the newer kernel. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c10 --- Comment #10 from Jan Beulich <jbeulich@novell.com> 2009-02-03 03:25:38 MST --- Created an attachment (id=269535) --> (https://bugzilla.novell.com/attachment.cgi?id=269535) potential fix -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c11 Jan Beulich <jbeulich@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P2 - High Status|ASSIGNED |NEEDINFO Info Provider| |wolf@womaro.ch QAContact|qa@suse.de |jdouglas@novell.com --- Comment #11 from Jan Beulich <jbeulich@novell.com> 2009-02-03 03:31:31 MST --- Could you try out this patch (i.e. can you build yourself a test kernel, which needs to be installed in the DomU)? Otherwise we'll have to see how quickly we can get you one. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User wolf@womaro.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c12 --- Comment #12 from Wolf Geldmacher <wolf@womaro.ch> 2009-02-03 10:32:07 MST --- Created an attachment (id=269717) --> (https://bugzilla.novell.com/attachment.cgi?id=269717) Bootlogs of patched and old kernel Here are the boot logs of the patched 2.6.27 kernel vs. the 2.6.22 one. 2.6.27 seems to at least get some interrupts now but then goes into a loop. As for "the hypervisor log": If you could point out, what exactly you need (file name or command to run) I'd gladly send it. Thanks for your support! -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User wolf@womaro.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c13 Wolf Geldmacher <wolf@womaro.ch> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED Info Provider|wolf@womaro.ch | --- Comment #13 from Wolf Geldmacher <wolf@womaro.ch> 2009-02-03 10:46:34 MST --- Created an attachment (id=269721) --> (https://bugzilla.novell.com/attachment.cgi?id=269721) Additional logs from Dom0 Pls find attached two more logs: - dmesg-dom0: output of dmesg command for Dom0 - messages-dom0: relevant content of /var/log/messages ( Xen instance 7/8 is the patched kernel - 7 surprised me; the log in the previous attachment is the boot of 8. Xen instance 9 is the good 'ol 2.6.22 kernel) Setting back the "NEEDINFO" attribute now - pls notify if you need anything else. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c14 --- Comment #14 from Jan Beulich <jbeulich@novell.com> 2009-02-04 04:00:54 MST --- (In reply to comment #12)
2.6.27 seems to at least get some interrupts now but then goes into a loop.
Not any better than for 2.6.22, i.e. it still doesn't really get control of these IRQs, the setup just appears to succeed to the driver.
As for "the hypervisor log": If you could point out, what exactly you need (file name or command to run) I'd gladly send it.
It may be sufficient to use 'xm dmesg' for this here, but in order to be certain to capture the complete log you'll have to use a serial console. We will need that log in order to (hopefully) determine the reason for the "Failed to obtain physical IRQ ..." messages. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c15 --- Comment #15 from Jan Beulich <jbeulich@novell.com> 2009-02-04 04:08:13 MST --- (In reply to comment #13) Could you clarify also what you found surprising with instance 7? Its trace in dom0's log is basically identical to that of 8. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c16 Jan Beulich <jbeulich@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #269535|0 |1 is obsolete| | --- Comment #16 from Jan Beulich <jbeulich@novell.com> 2009-02-04 04:25:04 MST --- Created an attachment (id=269962) --> (https://bugzilla.novell.com/attachment.cgi?id=269962) potential fix The endless loop is caused by two of the devices assigned to the DomU sharing IRQ 23. I had realized the issue yesterday already, but forgot to put the updated patch here. This should get .27 on par with .22, but in order to determine why the other IRQs cannot be grabbed by the DomU we'll need (possibly only as a first step) the hypervisor logs as indicated above. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 Jan Beulich <jbeulich@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |wolf@womaro.ch -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User wolf@womaro.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c17 --- Comment #17 from Wolf Geldmacher <wolf@womaro.ch> 2009-02-04 06:20:07 MST --- (In reply to comment #15) I didn't find anything surprising - I got suprised by the boot going into the loop, had to kill the Domu, start script and then restart the DomU. This is why there are two DomU boots in the log. (In reply to comment #14) 'xm dmesg' is no problem - will provide output next time round. Attaching a serial console would be some trouble, though: The mobo does not have the DB9 connectors - would have to whip up some temporary null-modem contraption, if this is really required. (In reply to comment #16) Will test this on Friday and send the logs - no access to the machine until then. Thanks again for your support. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User wolf@womaro.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c18 Wolf Geldmacher <wolf@womaro.ch> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED Info Provider|wolf@womaro.ch | --- Comment #18 from Wolf Geldmacher <wolf@womaro.ch> 2009-02-06 00:26:30 MST --- Created an attachment (id=270683) --> (https://bugzilla.novell.com/attachment.cgi?id=270683) Logs for 2.6.27.10 patched Hi Jan, with the second patch applied 2.6.27 now works like 2.6.22 did! The DomU can use the PCI hardware again. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User wolf@womaro.ch added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c19 --- Comment #19 from Wolf Geldmacher <wolf@womaro.ch> 2009-02-06 00:43:52 MST --- (In reply to comment #16) I understand that - although my immediate problem seems solved - you are not happy with the "shared interrupts not being passed through to DomU". If I can provide any additional information for you to solve that issue (which I would assume could be an issue in either the Dom0 or the DomU kernel or in the coordination inbetween as it also appears in with the 2.6.22 DomU), I'm willing to help. I may be a good idea, though, to create a new bugzilla entry (lower prio, related but not identical issue) for this. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=463596#c20 Jan Beulich <jbeulich@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |RESOLVED Resolution| |FIXED --- Comment #20 from Jan Beulich <jbeulich@novell.com> 2009-02-06 01:26:25 MST --- (In reply to comment #19)
I may be a good idea, though, to create a new bugzilla entry (lower prio, related but not identical issue) for this.
I agree to that, and it all depends on you (and with the state things are I really don't think the controllers not getting their interrupts set up really work - you may just happen to only use ports hanging off the two ones that do): If you enter a new bug for this, please provide either the full serial log or, if that's difficult to achieve, 'xm dmesg' output from right after the system got up (without any VMs started) and from after the VM in question was started. The patch for the problem here was committed yesterday, but I can't tell when you an update kernel for 11.1 would have it. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=463596 Wolf Geldmacher <wolf@womaro.ch> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |473217 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com