[Bug 1231231] New: Rootless podman inside distrobox not working
https://bugzilla.suse.com/show_bug.cgi?id=1231231 Bug ID: 1231231 Summary: Rootless podman inside distrobox not working Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Containers Assignee: containers-bugowner@suse.de Reporter: martin.sirringhaus@suse.com QA Contact: qa-bugs@suse.de Target Milestone: --- Found By: --- Blocker: --- Following the upstream guide on how to use podman inside distrobox (https://distrobox.it/useful_tips/#using-podman-inside-a-distrobox) I can't get rootless podman to work, even though the upstream guide is actually using Tumbleweed. I only seem to be able to get `sudo podman` to work (albeit with a warning): ❯ sudo podman run --rm -ti alpine WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path /libpod_parent/conmon: write /sys/fs/cgroup/libpod_parent/cgroup.subtree_control: operation not supported / # Rootless podman however fails: ❯ podman run --rm -ti alpine ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": could not find any running process: no such process ❯ And if I try to run `podman system migrate`, podman segfaults: ❯ podman system migrate stopped 8acf10be6dbf4336260113fd2b54b45bcdf803970cd84b21d4933a5b5eccd2ad panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x55e5f4a8f49b] goroutine 1 [running]: github.com/containers/podman/v5/libpod.(*storageService).UnmountContainerImage(0x0, {0xc00054be80?, 0x0?}, 0x0) /home/abuild/rpmbuild/BUILD/podman-5.2.3/libpod/storage.go:235 +0x3b github.com/containers/podman/v5/libpod.(*Container).unmount(0xc000880fa0, 0x80?) /home/abuild/rpmbuild/BUILD/podman-5.2.3/libpod/container_internal.go:2467 +0x3a github.com/containers/podman/v5/libpod.(*Container).cleanupStorage(0xc000880fa0) /home/abuild/rpmbuild/BUILD/podman-5.2.3/libpod/container_internal.go:2007 +0x505 github.com/containers/podman/v5/libpod.(*Container).cleanup(0xc000880fa0, {0x55e5f55fcc50, 0x55e5f6399840}) /home/abuild/rpmbuild/BUILD/podman-5.2.3/libpod/container_internal.go:2097 +0x5bb github.com/containers/podman/v5/libpod.(*Container).waitForConmonToExitAndSave(0xc000880fa0) /home/abuild/rpmbuild/BUILD/podman-5.2.3/libpod/container_internal.go:1512 +0x4c5 github.com/containers/podman/v5/libpod.(*Container).stop(0xc000880fa0, 0xa) /home/abuild/rpmbuild/BUILD/podman-5.2.3/libpod/container_internal.go:1454 +0x5d2 github.com/containers/podman/v5/libpod.(*Container).StopWithTimeout(0xc000880fa0?, 0xa?) /home/abuild/rpmbuild/BUILD/podman-5.2.3/libpod/container_api.go:282 +0xd7 github.com/containers/podman/v5/libpod.(*Container).Stop(...) /home/abuild/rpmbuild/BUILD/podman-5.2.3/libpod/container_api.go:252 github.com/containers/podman/v5/libpod.(*Runtime).Migrate(0xc0005848c0, {0x0, 0x0}) /home/abuild/rpmbuild/BUILD/podman-5.2.3/libpod/runtime_migrate_linux.go:76 +0x2e5 github.com/containers/podman/v5/pkg/domain/infra/abi.(*ContainerEngine).Migrate(0x0?, {0x0?, 0x0?}, {{0x0?, 0x8453589700000000?}}) /home/abuild/rpmbuild/BUILD/podman-5.2.3/pkg/domain/infra/abi/system.go:298 +0x25 github.com/containers/podman/v5/cmd/podman/system.migrate(0x55e5f62a12e0?, {0x55e5f6399840?, 0x0?, 0x0?}) /home/abuild/rpmbuild/BUILD/podman-5.2.3/cmd/podman/system/migrate.go:56 +0x63 github.com/spf13/cobra.(*Command).execute(0x55e5f62a12e0, {0xc000040470, 0x0, 0x0}) /home/abuild/rpmbuild/BUILD/podman-5.2.3/vendor/github.com/spf13/cobra/command.go:989 +0xa91 github.com/spf13/cobra.(*Command).ExecuteC(0x55e5f628b420) /home/abuild/rpmbuild/BUILD/podman-5.2.3/vendor/github.com/spf13/cobra/command.go:1117 +0x3ff github.com/spf13/cobra.(*Command).Execute(...) /home/abuild/rpmbuild/BUILD/podman-5.2.3/vendor/github.com/spf13/cobra/command.go:1041 github.com/spf13/cobra.(*Command).ExecuteContext(...) /home/abuild/rpmbuild/BUILD/podman-5.2.3/vendor/github.com/spf13/cobra/command.go:1034 main.Execute() /home/abuild/rpmbuild/BUILD/podman-5.2.3/cmd/podman/root.go:116 +0xb4 main.main() /home/abuild/rpmbuild/BUILD/podman-5.2.3/cmd/podman/main.go:61 +0x4b2 -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1231231 https://bugzilla.suse.com/show_bug.cgi?id=1231231#c1 Alexandre Vicenzi <alexandre.vicenzi@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|containers-bugowner@suse.de |alexandre.vicenzi@suse.com CC| |alexandre.vicenzi@suse.com, | |containers-bugowner@suse.de Status|NEW |CONFIRMED --- Comment #1 from Alexandre Vicenzi <alexandre.vicenzi@suse.com> --- I can confirm that rootless Podman fails inside distrobox. However, I was not able to reproduce the segfault. I do get an error when I try to run podman system migrate: ERRO[0000] Unable to clean up network for container cbffab8d8d94d173fe49bf666f9fbda2733a68008e4c09349bf8b59a8f7159db: "unmounting network namespace for container cbffab8d8d94d173fe49bf666f9fbda2733a68008e4c09349bf8b59a8f7159db: failed to unmount NS: at /run/user/1000/netns/netns-d6f9f70d-3639-577e-fd35-8fb1244f34d1: operation not permitted" -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1231231 https://bugzilla.suse.com/show_bug.cgi?id=1231231#c2 Alexandre Vicenzi <alexandre.vicenzi@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(dfaggioli@suse.co | |m) CC| |dfaggioli@suse.com --- Comment #2 from Alexandre Vicenzi <alexandre.vicenzi@suse.com> --- Launching the container with root privileges allows Podman to work in rootless mode inside a container. Is the following command a valid solution or workaround for you? distrobox create \ --root \ --image registry.opensuse.org/opensuse/distrobox:latest \ --additional-packages "podman crun" \ --unshare-all I tested this on a Fedora and Tumbleweed host, with Fedora and openSUSE container and I'm able to reproduce the issue every time. To me, this indicates a problem with the upstream documentation or scripts used by distrobox and not Podman or our container images. Dario, are you able to help? -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1231231 https://bugzilla.suse.com/show_bug.cgi?id=1231231#c3 Danish Prakash <danish.prakash@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |danish.prakash@suse.com --- Comment #3 from Danish Prakash <danish.prakash@suse.com> --- I don't know if it was supposed to work without `--root` or not--rootless podman within rootless podman works btw--but I suspect it's broken due to conflicts with state files between containers on the host and distrobox's. I'll take a deeper look into this next week and report back with an update, but for now, the workaround is to use the `--root` flag as Alexandre suggested. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1231231 https://bugzilla.suse.com/show_bug.cgi?id=1231231#c4 Alexandre Vicenzi <alexandre.vicenzi@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(dfaggioli@suse.co | |m) | --- Comment #4 from Alexandre Vicenzi <alexandre.vicenzi@suse.com> --- (In reply to Danish Prakash from comment #3)
I don't know if it was supposed to work without `--root` or not--rootless podman within rootless podman works btw--but I suspect it's broken due to conflicts with state files between containers on the host and distrobox's. I'll take a deeper look into this next week and report back with an update, but for now, the workaround is to use the `--root` flag as Alexandre suggested.
This article [1] covers how to run Podman in Podman (PINP), and all scenarios are supported, such as: - Rootful Podman in rootful Podman - Rootless Podman in rootful Podman - Rootful Podman in rootless Podman - Rootless Podman in rootless Podman The container however needs to run in privileged mode, or if privileged is not desired, disable some security options. With privileged:
podman run --privileged quay.io/podman/stable podman run alpine echo hello
This can be achieved with `distrobox create --root`, it is the easiest approach to run PINP. Without privileges, setting proper security flags is necessary.
podman run --security-opt label=disable --user podman --device /dev/fuse quay.io/podman/stable podman run alpine echo hello
This could be done with `distrobox create --additional-flags`. I tried to replicate non-privileged commands, but this caused `distrobox enter` command to fail during container initialization due to permission issues while distrobox sets up the container. My conclusion for now is that the Distrobox documentation needs to be updated to reflect the need for --root flag. I'll submit a PR to distrobox updating the documentation. [1]: https://www.redhat.com/en/blog/podman-inside-container -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1231231 https://bugzilla.suse.com/show_bug.cgi?id=1231231#c5 Alexandre Vicenzi <alexandre.vicenzi@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CONFIRMED |IN_PROGRESS --- Comment #5 from Alexandre Vicenzi <alexandre.vicenzi@suse.com> --- I submitted a PR upstream, which you can see at https://github.com/89luca89/distrobox/pull/1597. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1231231 https://bugzilla.suse.com/show_bug.cgi?id=1231231#c6 --- Comment #6 from Martin Sirringhaus <martin.sirringhaus@suse.com> --- distrobox already creates the containers in privileged mode: ❯ podman container inspect c364fe996b19 | grep -i privileged "io.podman.annotations.privileged": "TRUE", "--privileged", "io.podman.annotations.privileged": "TRUE", "Privileged": true, So rootless inside rootless should work, I think. Another colleague asked the maintainer, who responded with:
I think is something about opensuse, on ubuntu the guide works. After a quick test looks like if you do chmod +s /usr/bin/newuidmap /usr/bin/newgidmap works I guess a setcap problem -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1231231 https://bugzilla.suse.com/show_bug.cgi?id=1231231#c7 Alexandre Vicenzi <alexandre.vicenzi@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(danish.prakash@su | |se.com) --- Comment #7 from Alexandre Vicenzi <alexandre.vicenzi@suse.com> --- (In reply to Martin Sirringhaus from comment #6)
distrobox already creates the containers in privileged mode:
❯ podman container inspect c364fe996b19 | grep -i privileged "io.podman.annotations.privileged": "TRUE", "--privileged", "io.podman.annotations.privileged": "TRUE", "Privileged": true,
So rootless inside rootless should work, I think.
Another colleague asked the maintainer, who responded with:
I think is something about opensuse, on ubuntu the guide works. After a quick test looks like if you do chmod +s /usr/bin/newuidmap /usr/bin/newgidmap works I guess a setcap problem
Ubuntu newuidmap has +s set, while Tumbleweed and Fedora do not.
root@ubuntu:~# ls -lah /usr/bin/newuidmap -rwsr-xr-x 1 root root 69K May 30 14:52 /usr/bin/newuidmap
alexandre@tumbleweed $ ls -lah /usr/bin/newuidmap -rwxr-xr-x 1 root root 37K Jul 8 13:13 /usr/bin/newuidmap
alexandre@fedora-41:~$ ls -lah /usr/bin/newuidmap -rwxr-xr-x. 1 root root 43K Oct 10 02:00 /usr/bin/newuidmap
This is something to be discussed with Shadow maintainers perhaps, not sure if Podman can do something about it. Danish, anything we might be able to do on our side? -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com