[opensuse-virtual] correct IOMMU setup for AMD processor & xen pci passthrough?
hi, i've an AMD processor, cat /proc/cpuinfo | egrep "vendor|model|flags" vendor_id : AuthenticAMD model : 4 model name : AMD Phenom(tm) II X4 920 Processor flags : fpu de tsc msr pae mce cx8 apic mtrr mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow constant_tsc rep_good nopl pni cx16 popcnt lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch i've openSUSE 11.1, running latest SL111 kernel, uname -ri 2.6.27.29-7-xen x86_64 with OBS Virtualization:Xen, xm dmesg | grep "Xen version" (XEN) Xen version 3.4.1_19711_01-9.1 (abuild@) (gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) ) Wed Aug 5 01:48:18 UTC 2009 i'm doing hardware passthrough of a sata PCI card to DomU, and want to ensure iommu is (un)set properly. reading @, "AMD64 specific boot options" http://www.kernel.org/doc/Documentation/x86/x86_64/boot-options.txt it seems i should (?) use, iommu=off,noagp,noaperture @ grub. however, comparing startup, (1) WITH "iommu=off,noagp,noaperture" @ grub/menu.lst dmesg | grep -i pciback pciback 0000:04:07.0: seizing device pciback 0000:04:07.0: enabling device (019d -> 019f) pciback 0000:04:07.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21 pciback 0000:04:07.0: PCI INT A disabled pciback: vpci: 0000:04:07.0: assign to virtual slot 0 pciback 0000:04:07.0: enabling device (0080 -> 0083) pciback 0000:04:07.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21 (2) WITHOUT "iommu=off,noagp,noaperture" @ grub/menu.lst dmesg | grep pciback pciback 0000:04:07.0: seizing device pciback 0000:04:07.0: enabling device (019d -> 019f) pciback 0000:04:07.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21 pciback 0000:04:07.0: PCI INT A disabled i can see that there's a difference -- namely the, pciback: vpci: 0000:04:07.0: assign to virtual slot 0 but in both cases, @ DomU, the card is correctly passed through, lspci 00:00.0 RAID bus controller: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller (rev 02) and, afaict, works fine. IS the aformentionted difference relevant? given my config, what is the recommended IOMMU config for kernel cmd line? thanks! -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-virtual+help@opensuse.org
I'm pretty sure (85%) that iommu is off by default in xen 3.4, but even it for some reason it's not, it would be a xen hypervisor option rather than a kernel option that controls it. In order to enable it in xen 3.3, for example, you need to pass iommu=1 on the hypervisor line, otherwise it is off. Jason
On 8/13/2009 at 12:08 PM, in message <94f2e81e0908131108s18872e2ajf093f78cab1a0871@mail.gmail.com>, PGNet Dev <pgnet.dev+osvm@gmail.com> wrote: hi,
i've an AMD processor,
cat /proc/cpuinfo | egrep "vendor|model|flags" vendor_id : AuthenticAMD model : 4 model name : AMD Phenom(tm) II X4 920 Processor flags : fpu de tsc msr pae mce cx8 apic mtrr mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow constant_tsc rep_good nopl pni cx16 popcnt lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch
i've openSUSE 11.1, running latest SL111 kernel,
uname -ri 2.6.27.29-7-xen x86_64
with OBS Virtualization:Xen,
xm dmesg | grep "Xen version" (XEN) Xen version 3.4.1_19711_01-9.1 (abuild@) (gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) ) Wed Aug 5 01:48:18 UTC 2009
i'm doing hardware passthrough of a sata PCI card to DomU, and want to ensure iommu is (un)set properly. reading @,
"AMD64 specific boot options" http://www.kernel.org/doc/Documentation/x86/x86_64/boot-options.txt
it seems i should (?) use,
iommu=off,noagp,noaperture
@ grub. however, comparing startup,
(1) WITH "iommu=off,noagp,noaperture" @ grub/menu.lst
dmesg | grep -i pciback pciback 0000:04:07.0: seizing device pciback 0000:04:07.0: enabling device (019d -> 019f) pciback 0000:04:07.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21 pciback 0000:04:07.0: PCI INT A disabled pciback: vpci: 0000:04:07.0: assign to virtual slot 0 pciback 0000:04:07.0: enabling device (0080 -> 0083) pciback 0000:04:07.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
(2) WITHOUT "iommu=off,noagp,noaperture" @ grub/menu.lst
dmesg | grep pciback pciback 0000:04:07.0: seizing device pciback 0000:04:07.0: enabling device (019d -> 019f) pciback 0000:04:07.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21 pciback 0000:04:07.0: PCI INT A disabled
i can see that there's a difference -- namely the,
pciback: vpci: 0000:04:07.0: assign to virtual slot 0
but in both cases, @ DomU, the card is correctly passed through,
lspci 00:00.0 RAID bus controller: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller (rev 02)
and, afaict, works fine.
IS the aformentionted difference relevant? given my config, what is the recommended IOMMU config for kernel cmd line?
thanks!
-- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-virtual+help@opensuse.org
hi jason, On Thu, Aug 13, 2009 at 1:16 PM, Jason Douglas<jdouglas@novell.com> wrote:
I'm pretty sure (85%) that iommu is off by default in xen 3.4, but even it for some reason it's not, it would be a xen hypervisor option rather than a kernel option that controls it. In order to enable it in xen 3.3, for example, you need to pass iommu=1 on the hypervisor line, otherwise it is off.
your comment abt hypervisor-not-kernel-line caught me off guard, as reading @, http://www.kernel.org/doc/Documentation/x86/x86_64/boot-options.txt those are certainly _kernel_ options. digging, i found this, "The Xen Hypervisor and its IO Subsystem" www.mulix.org/lectures/xen-iommu/xen-io.pdf & now understand that there's a _slew_ of IOMMU approaches, varying by arch and kernel flavor. given that i've an AMD cpu, reading @, "IOMMU" http://forums.amd.com/devblog/blogpost.cfm?threadid=99513&catid=317 "IOMMU for XEN" http://forums.amd.com/devblog/blogpost.cfm?threadid=104671&catid=317 "AMD I/O Virtualization Technology (IOMMU) Specification" http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/3443... i note, finally, "At the moment of writing, domain level device isolation is the case most interesting for the XEN community. As explained above, this means IOMMU support in hypervisor only while leaving the guest kernel untouched." aha. iiuc, them "correct" config, in xen use, is (?), grub hypervisor --> ... iommu=1 ... grub kernel --> ... iommu=off ... with that config, @ Dom0 dmesg | egrep -i "back|mmu" Command line: root=LABEL=DOM0_ROOT resume=LABEL=DOM0_SWAP showopts splash=silent vga=0x31a console=tty0 console=xvc0,57600 elevator=cfq reassigndev=0000:04:07.0 iommu=off Kernel command line: root=LABEL=DOM0_ROOT resume=LABEL=DOM0_SWAP showopts splash=silent vga=0x31a console=tty0 console=xvc0,57600 elevator=cfq reassigndev=0000:04:07.0 iommu=off pciback 0000:04:07.0: seizing device pciback 0000:04:07.0: enabling device (019d -> 019f) pciback 0000:04:07.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21 pciback 0000:04:07.0: PCI INT A disabled and, xm dmesg | egrep -i "back|mmu" (XEN) Command line: dom0_mem=768M loglvl=all loglvl_guest=all vga=gfx-1280x1024x32 console=vga,com1 com1=57600,8n1 cpufreq=xen:performance cpuidle iommu=1 also, @ DomU, lspci 00:00.0 RAID bus controller: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller (rev 02) and no apparent errors anywhere, which is good. still the results all look similar to the output before. and, tbh, i haven't found a stmt of what the _default_ states are of iommu @ kernel-xen & hypervisor. varying options can, of course, tell me that ... that said, the goal here, of course, is full per-domain DMA protection/isolation. any hints as to how one *verifies* functional DMA isolation between DomUs, given apparently working (?) iommu-in-Xen? richard -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-virtual+help@opensuse.org
participants (2)
-
Jason Douglas -
PGNet Dev