[Bug 1233690] New: Docker containers are not reachable from network
https://bugzilla.suse.com/show_bug.cgi?id=1233690 Bug ID: 1233690 Summary: Docker containers are not reachable from network Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: Other OS: Other Status: NEW Severity: Major Priority: P5 - None Component: MicroOS Assignee: forgotten_u0-bnvADNc@user.net Reporter: seifert@alesak.net QA Contact: qa-bugs@suse.de Target Milestone: --- Found By: --- Blocker: --- After regular OS update from 20241114 to 20241115 or later makes containers unreachable. Containers seams to be normally bind to correct interfaces. As it is on production server I was not able to investigate for very long time, reverting to previous OS snapshot resolved the problem. Installed current version of "openSUSE-MicroOS.x86_64-ContainerHost-kvm-and-xen.qcow2" into VM but unable to reproduce the issue, everything seams to work normally. The only symptom in journal is high occurrence of similar lines: Nov 24 02:23:46 backup1 dockerd[1340]: time="2024-11-24T02:23:46.068350205Z" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.20.0.66:42800" dns-server="udp:213.186.33.99:53" error="read udp 172.20.0.66:42800->213.186.33.99:53: i/o timeout" question=";certbot.\tIN\t AAAA" spanID=850c3ffdfcec0675 traceID=e1693dcfdf213f90555090b6285ac906 Also there is a difference in IPTABLES: NOT working: Chain FORWARD (policy DROP) target prot opt source destination DOCKER-USER all -- anywhere anywhere DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere working: Chain FORWARD (policy DROP) target prot opt source destination DOCKER-USER all -- anywhere anywhere DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1233690 Ales Seifert <seifert@alesak.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Docker containers are not |Docker containers are not |reachable from network |reachable from network | |after OS update from | |20241114 to 20241115 -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1233690 https://bugzilla.suse.com/show_bug.cgi?id=1233690#c1 JDA <jda82@vicious-gaming.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jda82@vicious-gaming.de --- Comment #1 from JDA <jda82@vicious-gaming.de> --- I am also affected by this issue. This seems to be caused by docker being unable to set all required IPTables-Rules, especially in the FORWARD table. If i save the rules before the upgrade and restore after, everything works fine (until i change something). If i just restart the machine after an upgrade, rules are missing and the DROP-Counter of the FORWARD-Chain increases. I run a non-trivial Docker setup with multiple networks, especially a "ingress" external network for traefik, which is used in most of my docker-compose stacks. After a failed upgrade most of the FORWARD rules for interfaces other than docker0 are missing. This does not seem to be caused by docker or runc but by iptables / nftables /xtables. After downgrading those everything started working again. Working versions for me: - iptables-1.8.10-3.1 - iptables-backend-nft-1.8.10-3.1 - nftables-1.1.1-1.1 - libnftables1-1.1.1-1.1 - libxtables12-1.8.10-3.1 - xtables-plugins-1.8.10-3.1 This is on tumbleweed 20241125 with docker-stable 24.0.9_ce-2.1 and runc 1.2.2-1.1 -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1233690 Ales Seifert <seifert@alesak.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|forgotten_u0-bnvADNc@user.n |containers-bugowner@suse.de |et | Component|MicroOS |Containers -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1233690 https://bugzilla.suse.com/show_bug.cgi?id=1233690#c5 Ales Seifert <seifert@alesak.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(seifert@alesak.ne | |t) | --- Comment #5 from Ales Seifert <seifert@alesak.net> --- working docker version: Client: Version: 26.1.5-ce API version: 1.45 Go version: go1.21.13 Git commit: 411e817ddf71 Built: Wed Oct 16 22:24:52 2024 OS/Arch: linux/amd64 Context: default Server: Engine: Version: 26.1.5-ce API version: 1.45 (minimum version 1.24) Go version: go1.21.13 Git commit: 411e817ddf71 Built: Wed Oct 16 22:24:52 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.7.23 GitCommit: 57f17b0a6295a39009d861b89e3b3b87b005ca27 runc: Version: 1.2.1 GitCommit: v1.2.1-0-gd7735e388ef5 docker-init: Version: 0.2.0_catatonit GitCommit: working docker info: Client: Version: 26.1.5-ce Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: 0.17.1 Path: /usr/lib/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: 2.30.3 Path: /usr/lib/docker/cli-plugins/docker-compose Server: Containers: 17 Running: 17 Paused: 0 Stopped: 0 Images: 15 Server Version: 26.1.5-ce Storage Driver: overlay2 Backing Filesystem: btrfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: oci runc io.containerd.runc.v2 Default Runtime: runc Init Binary: docker-init containerd version: 57f17b0a6295a39009d861b89e3b3b87b005ca27 runc version: v1.2.1-0-gd7735e388ef5 init version: Security Options: seccomp Profile: builtin cgroupns Kernel Version: 6.11.7-1-default Operating System: openSUSE MicroOS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 15.52GiB Name: backup1 ID: ea496780-21e3-4f87-8a85-d8f9e3852b1d Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Default Address Pools: Base: 172.19.0.0/16, Size: 26 working docker network inspect bridge: [ { "Name": "bridge", "Id": "f789ae3dd0c196dd9423e4af6c212269de84d314696f55e00ea3af3b1749a2a0", "Created": "2024-11-24T04:19:30.545091227Z", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "172.16.0.0/16", "Gateway": "172.16.0.1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": {}, "Options": { "com.docker.network.bridge.default_bridge": "true", "com.docker.network.bridge.enable_icc": "true", "com.docker.network.bridge.enable_ip_masquerade": "true", "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0", "com.docker.network.bridge.name": "docker0", "com.docker.network.driver.mtu": "1500" }, "Labels": {} } ] working iptables --version: iptables v1.8.10 (nf_tables) working iptables -L: Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy DROP) target prot opt source destination DOCKER-USER all -- anywhere anywhere DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain DOCKER (6 references) target prot opt source destination ACCEPT tcp -- anywhere 172.19.0.2 tcp dpt:etlservicemgr ACCEPT tcp -- anywhere 172.19.1.2 tcp dpt:9096 ACCEPT tcp -- anywhere 172.19.1.2 tcp dpt:9095 ACCEPT tcp -- anywhere 172.19.1.5 tcp dpt:bacula-sd ACCEPT tcp -- anywhere 172.19.1.6 tcp dpt:bacula-dir ACCEPT tcp -- anywhere 172.19.0.66 tcp dpt:svcloud ACCEPT tcp -- anywhere 172.19.0.66 tcp dpt:https ACCEPT udp -- anywhere 172.19.0.66 udp dpt:https ACCEPT tcp -- anywhere 172.19.0.66 tcp dpt:http ACCEPT tcp -- anywhere 172.19.1.70 tcp dpt:http ACCEPT tcp -- anywhere 172.19.1.134 tcp dpt:http Chain DOCKER-ISOLATION-STAGE-1 (1 references) target prot opt source destination DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere RETURN all -- anywhere anywhere Chain DOCKER-ISOLATION-STAGE-2 (6 references) target prot opt source destination DROP all -- anywhere anywhere DROP all -- anywhere anywhere DROP all -- anywhere anywhere DROP all -- anywhere anywhere DROP all -- anywhere anywhere DROP all -- anywhere anywhere RETURN all -- anywhere anywhere Chain DOCKER-USER (1 references) target prot opt source destination RETURN all -- anywhere anywhere python311-nftables-1.1.1-1.1.noarch iptables-backend-nft-1.8.10-3.1.x86_64 xtables-plugins-1.8.10-3.1.x86_64 iptables-1.8.10-3.1.x86_64 nftables-1.1.1-1.1.x86_64 libxtables12-1.8.10-3.1.x86_64 libnftables1-1.1.1-1.1.x86_64 docker-26.1.5_ce-8.1.x86_64 NOT working docker version: Client: Version: 26.1.5-ce API version: 1.45 Go version: go1.21.13 Git commit: 411e817ddf71 Built: Tue Nov 12 06:34:28 2024 OS/Arch: linux/amd64 Context: default Server: Engine: Version: 26.1.5-ce API version: 1.45 (minimum version 1.24) Go version: go1.21.13 Git commit: 411e817ddf71 Built: Tue Nov 12 06:34:28 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.7.23 GitCommit: 57f17b0a6295a39009d861b89e3b3b87b005ca27 runc: Version: 1.2.2 GitCommit: v1.2.2-0-g7cb363254b69 docker-init: Version: 0.2.0_catatonit GitCommit: NOT working docker info: Client: Version: 26.1.5-ce Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: 0.17.1 Path: /usr/lib/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: 2.30.3 Path: /usr/lib/docker/cli-plugins/docker-compose Server: Containers: 17 Running: 17 Paused: 0 Stopped: 0 Images: 15 Server Version: 26.1.5-ce Storage Driver: overlay2 Backing Filesystem: btrfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 oci runc Default Runtime: runc Init Binary: docker-init containerd version: 57f17b0a6295a39009d861b89e3b3b87b005ca27 runc version: v1.2.2-0-g7cb363254b69 init version: Security Options: seccomp Profile: builtin cgroupns Kernel Version: 6.11.8-1-default Operating System: openSUSE MicroOS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 15.52GiB Name: backup1 ID: ea496780-21e3-4f87-8a85-d8f9e3852b1d Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Default Address Pools: Base: 172.19.0.0/16, Size: 26 NOT working docker network inspect bridge: [ { "Name": "bridge", "Id": "d720be83bd73fd3969dd3ee7d1378bcd4a91d224dbc49188fd48557a3b24dd0c", "Created": "2024-11-30T01:50:41.360946672Z", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "172.16.0.0/16", "Gateway": "172.16.0.1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": {}, "Options": { "com.docker.network.bridge.default_bridge": "true", "com.docker.network.bridge.enable_icc": "true", "com.docker.network.bridge.enable_ip_masquerade": "true", "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0", "com.docker.network.bridge.name": "docker0", "com.docker.network.driver.mtu": "1500" }, "Labels": {} } ] NOT working iptables --version: iptables v1.8.11 (nf_tables) NOT working iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy DROP) target prot opt source destination DOCKER-USER all -- anywhere anywhere DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain DOCKER (1 references) target prot opt source destination ACCEPT tcp -- anywhere 172.19.0.2 tcp dpt:etlservicemgr ACCEPT tcp -- anywhere 172.19.0.66 tcp dpt:svcloud ACCEPT tcp -- anywhere 172.19.0.66 tcp dpt:https ACCEPT udp -- anywhere 172.19.0.66 udp dpt:https ACCEPT tcp -- anywhere 172.19.0.66 tcp dpt:http ACCEPT tcp -- anywhere 172.19.0.131 tcp dpt:bacula-sd ACCEPT tcp -- anywhere 172.19.0.132 tcp dpt:9096 ACCEPT tcp -- anywhere 172.19.0.132 tcp dpt:9095 ACCEPT tcp -- anywhere 172.19.0.134 tcp dpt:bacula-dir ACCEPT tcp -- anywhere 172.19.0.198 tcp dpt:http ACCEPT tcp -- anywhere 172.19.1.6 tcp dpt:http Chain DOCKER-ISOLATION-STAGE-1 (1 references) target prot opt source destination DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere RETURN all -- anywhere anywhere Chain DOCKER-ISOLATION-STAGE-2 (1 references) target prot opt source destination DROP all -- anywhere anywhere RETURN all -- anywhere anywhere Chain DOCKER-USER (1 references) target prot opt source destination RETURN all -- anywhere anywhere python311-nftables-1.1.1-1.2.noarch iptables-backend-nft-1.8.11-1.1.x86_64 xtables-plugins-1.8.11-1.1.x86_64 iptables-1.8.11-1.1.x86_64 nftables-1.1.1-1.2.x86_64 libxtables12-1.8.11-1.1.x86_64 libnftables1-1.1.1-1.2.x86_64 docker-26.1.5_ce-9.1.x86_64 Unfortunately I cannot reproduce it on fresh current MicroOS installation, only on our two production servers. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1233690 https://bugzilla.suse.com/show_bug.cgi?id=1233690#c6 --- Comment #6 from Ales Seifert <seifert@alesak.net> --- Created attachment 878796 --> https://bugzilla.suse.com/attachment.cgi?id=878796&action=edit Working environment info -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1233690 https://bugzilla.suse.com/show_bug.cgi?id=1233690#c7 --- Comment #7 from Ales Seifert <seifert@alesak.net> --- Created attachment 878797 --> https://bugzilla.suse.com/attachment.cgi?id=878797&action=edit NOT Working environment info -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1233690 https://bugzilla.suse.com/show_bug.cgi?id=1233690#c14 --- Comment #14 from OBSbugzilla Bot <bwiedemann+obsbugzillabot@suse.com> --- This is an autogenerated message for OBS integration: This bug (1233690) was mentioned in https://build.opensuse.org/request/show/1227830 Factory / iptables -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com