how to emulate aarch64/ppc64le with qemu on private instance?
Hi, I'm trying to build aarch64 and ppc64le packages on a private instance. I have configured the project configuration like openSUSE:Factory:RISCV (as Andreas mentioned in another thread). I've copied the configuration to home:cschneemann:armtest on build.opensuse.org and I can see the same error in the buildlog as on my instance. ### [ 8s] setting hostarch to aarch64 [ 8s] using io_uring on x86_64 [ 9s] booting kvm... [ 9s] ### VM INTERACTION START ### [ 9s] Using UART console [ 9s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none -cpu host -M pc,accel=kvm,usb=off,dump-guest-core=off,vmport=off -sandbox on -bios /usr/share/qemu/qboot.rom -object rng-random,filename=/dev/random,id=rng0 -device virtio-rng-pci,rng=rng0 -object iothread,id=io0 -runas qemu -net none -kernel /var/cache/obs/worker/root_2/.mount/boot/kernel -initrd /var/cache/obs/worker/root_2/.mount/boot/initrd -append root=/dev/disk/by-id/virtio-0 rootfstype=ext3 rootflags=data=writeback,nobarrier,commit=150,noatime kvmclock mitigations=off elevator=noop nmi_watchdog=0 rw rd.driver.pre=binfmt_misc oops=panic panic=1 quiet console=ttyS0 init=/.build/build -m 8192 -drive file=/var/cache/obs/worker/root_2/root,format=raw,if=none,id=disk,cache=unsafe,aio=io_uring -device virtio-blk-pci,iothread=io0,drive=disk,serial=0 -drive file=/var/cache/obs/worker/root_2/swap,format=raw,if=none,id=swap,cache=unsafe,aio=io_uring -device virtio-blk-pci,iothread=io0,drive=swap,serial=1 -serial stdio -chardev socket,id=monitor,server=on,wait=off,path=/var/cache/obs/worker/root_2/root.qemu/monitor -mon chardev=monitor,mode=readline -smp 8 [ 9s] qemu: linux kernel too old to load a ram disk ### What is needed to prepare a project in an obs instance to be able to build packages for other architectures than x86_64 emulated with qemu? My project configuration: ## %if "%_repository" == "standard" %ifarch aarch64 %define qemu_user_space_build 1 Target: aarch64-suse-linux %endif %endif %if "%qemu_user_space_build" == "1" Hostarch: x86_64 Preinstall: qemu-linux-user Macros: %qemu_user_space_build 1 :Macros ExportFilter: ^(kernel-obs-build|qemu-linux-user).*\.aarch64\.rpm$ ExportFilter: ^qemu-linux-user-aarch64.*\.aarch64\.rpm$ . %endif # %qemu_user_space_build ## I would provide the packages for the distribution by download on demand. Is this possible? Regards, Christian -- Christian Schneemann Linux Consultant & Trainer Tel.: +49-175-7250665 Mail: schneemann@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
You also need to aggregate copies of the kernel-obs-build and qemu-linux-user package suitable for the worker host architecture (ie. x86_64) to the aarch64 and ppc64le repositories with the matching ExportFilter directive: ExportFilter: ^(kernel-obs-build|qemu-linux-user).*\.x86_64\.rpm$ . aarch64 ppc64le -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."
Am 01.07.22 um 12:57 schrieb Andreas Schwab:
You also need to aggregate copies of the kernel-obs-build and qemu-linux-user package suitable for the worker host architecture (ie. x86_64) to the aarch64 and ppc64le repositories with the matching ExportFilter directive:
ExportFilter: ^(kernel-obs-build|qemu-linux-user).*\.x86_64\.rpm$ . aarch64 ppc64le
Thanks, I have exchanged my wrong lines with your corrected one, but still the same error. I (try to) build with openSUSE:Leap:15.3/standard repository. If I look into the buildconfig for the kernel-obs-build package I see this: <bdep name="kernel-obs-build" vminstall="1" notmeta="1" version="5.3.18" release="57.3" arch="aarch64" hdrmd5="f88414509bcf3e114002479eb8320bf1" project="SUSE:SLE-15-SP3:GA" repository="pool"/> So the kernel-obs-build package is taken from SUSE:SLE-15-SP3:GA and architecture aarch64. I think the exported package for x86_64 should be used and that's why I still see the error? Regards, Christian -- Christian Schneemann Linux Consultant & Trainer Tel.: +49-175-7250665 Mail: schneemann@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
ExportFilter directives are rather hard to work with, they are only evaluated when the scheduler picks up a package. The qemu repository is not needed, you should change the aggregate to use "standard" as the target repostory. Then you are missing the imports of kernel-obs-build to standard/ppc64le and standard/aarch64. You need to run `osc wipebinaries --all home:cschneemann:armtest kernel-obs-build.qemu', to retrigger the scheduler to evaluate the ExportFilter directive. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."
Hi Andreas and others, Am 01.07.22 um 18:45 schrieb Andreas Schwab:
ExportFilter directives are rather hard to work with, they are only evaluated when the scheduler picks up a package.
The qemu repository is not needed, you should change the aggregate to use "standard" as the target repostory. Then you are missing the imports of kernel-obs-build to standard/ppc64le and standard/aarch64. You need to run `osc wipebinaries --all home:cschneemann:armtest kernel-obs-build.qemu', to retrigger the scheduler to evaluate the ExportFilter directive.
I got it working. THANK YOU! :) Thanks! Christian -- Christian Schneemann Linux Consultant & Trainer Tel.: +49-175-7250665 Mail: schneemann@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Hi, Am 04.07.22 um 20:45 schrieb Christian Schneemann:
Hi Andreas and others,
Am 01.07.22 um 18:45 schrieb Andreas Schwab:
ExportFilter directives are rather hard to work with, they are only evaluated when the scheduler picks up a package.
The qemu repository is not needed, you should change the aggregate to use "standard" as the target repostory. Then you are missing the imports of kernel-obs-build to standard/ppc64le and standard/aarch64. You need to run `osc wipebinaries --all home:cschneemann:armtest kernel-obs-build.qemu', to retrigger the scheduler to evaluate the ExportFilter directive.
I got it working. THANK YOU! :)
Seems that I still face some errors. Building for aarch64 is working fine, but for ppc64le I still get kernel Panics after VM start. Building for aarch64 gives me this buildlog parts with some "debug" echos in the buildscripts: ### check_use_emulator is running [ 7s] we are not building native [ 7s] initvm.arch is initvm.x86_64 [ 7s] all tests are fine [ 7s] initvmscript to use is: initvm.x86_64 [ 7s] booting kvm... [ 7s] ### VM INTERACTION START ### [ 7s] qemu_bin: /usr/bin/qemu-kvm [ 7s] Using virtio-serial support [ 7s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none -cpu host -M pc,accel=kvm,usb=off,dump-guest-core=off,vmport=off -sandbox on -bios /usr/share/qemu/qboot.rom... [ 11s] ### VM INTERACTION END ### [ 11s] 2nd stage started in virtual machine [ 11s] machine type: aarch64 ### So hostarch is x86_64 and machine type in the vm is set to aarch64, everything is fine. Some project for ppc64le gives me this: ### [ 8s] Detected virtio-serial support [ 8s] check_use_emulator is running [ 8s] we are not building native [ 8s] initvm.arch is initvm.x86_64 [ 8s] all tests are fine [ 8s] initvmscript to use is: initvm.x86_64 [ 8s] booting kvm... [ 8s] ### VM INTERACTION START ### [ 8s] qemu_bin: /usr/bin/qemu-kvm [ 8s] Using virtio-serial support [ 8s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none -cpu host -M pc,accel=kvm,usb=off,dump-guest-core=off,vmport=off -sandbox on -bios /usr/share/qemu/qboot.rom -object rng-random,filename=/dev/random,id=rng0 -device virtio-rn g-pci,rng=rng0 -object iothread,id=io0 -runas qemu -net none -kernel /srv/obs/worker//root_1/.mount/boot/kernel -initrd /srv/obs/worker//root_1/.mount/boot/initrd -append root=/dev/disk/by-id/virtio-0 rootfstype=ext3 rootflags=data=writeback,nob arrier,commit=150,noatime kvmclock mitigations=off elevator=noop nmi_watchdog=0 rw rd.driver.pre=binfmt_misc oops=panic panic=1 quiet console=hvc0 init=/.build/initvm.x86_64 -m 12240 -drive file=/srv/obs/worker//root_1/root,format=raw,if=none,id =disk,cache=unsafe,aio=io_uring -device virtio-blk-pci,iothread=io0,drive=disk,serial=0 -drive file=/srv/obs/worker//root_1/swap,format=raw,if=none,id=swap,cache=unsafe,aio=io_uring -device virtio-blk-pci,iothread=io0,drive=swap,serial=1 -device virtio-serial,max_ports=2 -device virtconsole,chardev=virtiocon0 -chardev stdio,id=virtiocon0 -chardev socket,id=monitor,server=on,wait=off,path=/srv/obs/worker//root_1/root.qemu/monitor -mon chardev=monitor,mode=readline -smp 8 [ 11s] execve of /.build/build: Exec format error [ 11s] [ 2.370049][ T1] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 [ 11s] [ 2.370253][ T1] CPU: 1 PID: 1 Comm: initvm.x86_64 Not tainted 5.18.9-2-default #1 openSUSE Tumbleweed 07ef3bbedc2497eb5acf4866fc116699e805cd [ 11s] [ 2.370533][ T1] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.1 11/11/2019 [ 11s] [ 2.370648][ T1] Call Trace: [ 11s] [ 2.370685][ T1] <TASK> [ 11s] [ 2.370718][ T1] dump_stack_lvl+0x44/0x59 [ 11s] [ 2.370820][ T1] panic+0x10b/0x2c1 [ 11s] [ 2.370865][ T1] do_exit.cold+0x15/0x15 [ 11s] [ 2.370930][ T1] do_group_exit+0x2d/0x90 [ 11s] [ 2.371025][ T1] __x64_sys_exit_group+0x14/0x20 [ 11s] [ 2.371088][ T1] do_syscall_64+0x58/0x80 [ 11s] [ 2.371171][ T1] ? do_syscall_64+0x67/0x80 [ 11s] [ 2.371234][ T1] ? exc_page_fault+0x66/0x150 [ 11s] [ 2.371315][ T1] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 11s] [ 2.371394][ T1] RIP: 0033:0x432fa9 [ 11s] [ 2.371435][ T1] Code: 00 41 b8 3c 00 00 00 eb 19 0f 1f 84 00 00 00 00 00 48 89 d7 44 89 c0 0f 05 48 3d 00 f0 ff ff 77 1b f4 48 89 44 89 c8 0f 05 <48> 3d 00 f0 ff ff 76 df 89 c6 f7 de 64 41 89 32 eb d5 89 c6 f7 de [ 11s] [ 2.371636][ T1] RSP: 002b:00007ffdc6311a58 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 [ 11s] [ 2.371739][ T1] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 0000000000432fa9 [ 11s] [ 2.371827][ T1] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 0000000000000001 [ 11s] [ 2.371915][ T1] RBP: 00000000004aa800 R08: 000000000000003c R09: 00000000000000e7 [ 11s] [ 2.372003][ T1] R10: ffffffffffffffd0 R11: 0000000000000246 R12: 0000000000000001 [ 11s] [ 2.372091][ T1] R13: 00000000006b71c0 R14: 0000000000401b20 R15: 0000000000000000 [ 11s] [ 2.372181][ T1] </TASK> [ 11s] [ 2.372414][ T1] Kernel Offset: 0x1fa00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 11s] [ 2.372556][ T1] Rebooting in 1 seconds.. Is something special needed for ppc64le? Or is there a bug for ppc64le? My project configuration: ## %if "%_repository" == "standard" %ifarch aarch64 %define qemu_user_space_build 1 Target: aarch64-suse-linux %endif %ifarch ppc64le %define qemu_user_space_build 1 Target: powerpc64le-suse-linux %endif %endif %if "%qemu_user_space_build" == "1" Hostarch: x86_64 Preinstall: qemu-linux-user Macros: %qemu_user_space_build 1 :Macros ExportFilter: ^(kernel-obs-build|qemu-linux-user).*\.aarch64\.rpm$ ExportFilter: ^(kernel-obs-build|qemu-linux-user).*\.ppc64le\.rpm$ %endif # %qemu_user_space_build ExportFilter: kernel-obs-build.*\.x86_64\.rpm$ . aarch64 ppc64le ExportFilter: qemu-linux-user.*\.x86_64\.rpm$ . aarch64 ppc64le ## Regards, Christian -- Christian Schneemann Linux Consultant & Trainer Tel.: +49-175-7250665 Mail: schneemann@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On Jul 13 2022, Christian Schneemann wrote:
[ 11s] execve of /.build/build: Exec format error
That looks like something is missing in the qemu linux-user setup. For example, check if the binfmt_misc setup is working. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different."
Am 13.07.22 um 16:19 schrieb Andreas Schwab:
On Jul 13 2022, Christian Schneemann wrote:
[ 11s] execve of /.build/build: Exec format error
That looks like something is missing in the qemu linux-user setup. For example, check if the binfmt_misc setup is working.
But how to check that? Isn't this configuration part of the initvm.<arch> call? And this is a compiled binary. With the scripts I was able to simply add echos to check if a code was running and to see the values in the variables. Regards, Christian -- Christian Schneemann Linux Consultant & Trainer Tel.: +49-175-7250665 Mail: schneemann@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
participants (3)
-
Andreas Schwab -
Andreas Schwab -
Christian Schneemann