[Bug 1233690] Docker containers are not reachable from network after OS update from 20241114 to 20241115

30 Nov 2024

      https://bugzilla.suse.com/show_bug.cgi?id=1233690
https://bugzilla.suse.com/show_bug.cgi?id=1233690#c5

Ales Seifert <seifert@alesak.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|needinfo?(seifert@alesak.ne |
                   |t)                          |

--- Comment #5 from Ales Seifert <seifert@alesak.net> ---
working docker version:

Client:
 Version:           26.1.5-ce
 API version:       1.45
 Go version:        go1.21.13
 Git commit:        411e817ddf71
 Built:             Wed Oct 16 22:24:52 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          26.1.5-ce
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       411e817ddf71
  Built:            Wed Oct 16 22:24:52 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.23
  GitCommit:        57f17b0a6295a39009d861b89e3b3b87b005ca27
 runc:
  Version:          1.2.1
  GitCommit:        v1.2.1-0-gd7735e388ef5
 docker-init:
  Version:          0.2.0_catatonit
  GitCommit:        

working docker info:

Client:
 Version:    26.1.5-ce
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.17.1
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.30.3
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 17
  Running: 17
  Paused: 0
  Stopped: 0
 Images: 15
 Server Version: 26.1.5-ce
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: oci runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 57f17b0a6295a39009d861b89e3b3b87b005ca27
 runc version: v1.2.1-0-gd7735e388ef5
 init version: 
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.11.7-1-default
 Operating System: openSUSE MicroOS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.52GiB
 Name: backup1
 ID: ea496780-21e3-4f87-8a85-d8f9e3852b1d
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.19.0.0/16, Size: 26

working docker network inspect bridge:

[
    {
        "Name": "bridge",
        "Id":
"f789ae3dd0c196dd9423e4af6c212269de84d314696f55e00ea3af3b1749a2a0",
        "Created": "2024-11-24T04:19:30.545091227Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.16.0.0/16",
                    "Gateway": "172.16.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

working iptables --version:

iptables v1.8.10 (nf_tables)

working iptables -L:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate
RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate
RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate
RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate
RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate
RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate
RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (6 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.19.0.2           tcp
dpt:etlservicemgr
ACCEPT     tcp  --  anywhere             172.19.1.2           tcp dpt:9096
ACCEPT     tcp  --  anywhere             172.19.1.2           tcp dpt:9095
ACCEPT     tcp  --  anywhere             172.19.1.5           tcp dpt:bacula-sd
ACCEPT     tcp  --  anywhere             172.19.1.6           tcp
dpt:bacula-dir
ACCEPT     tcp  --  anywhere             172.19.0.66          tcp dpt:svcloud
ACCEPT     tcp  --  anywhere             172.19.0.66          tcp dpt:https
ACCEPT     udp  --  anywhere             172.19.0.66          udp dpt:https
ACCEPT     tcp  --  anywhere             172.19.0.66          tcp dpt:http
ACCEPT     tcp  --  anywhere             172.19.1.70          tcp dpt:http
ACCEPT     tcp  --  anywhere             172.19.1.134         tcp dpt:http

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (6 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

python311-nftables-1.1.1-1.1.noarch
iptables-backend-nft-1.8.10-3.1.x86_64
xtables-plugins-1.8.10-3.1.x86_64
iptables-1.8.10-3.1.x86_64
nftables-1.1.1-1.1.x86_64
libxtables12-1.8.10-3.1.x86_64
libnftables1-1.1.1-1.1.x86_64
docker-26.1.5_ce-8.1.x86_64

NOT working docker version:

Client:
 Version:           26.1.5-ce
 API version:       1.45
 Go version:        go1.21.13
 Git commit:        411e817ddf71
 Built:             Tue Nov 12 06:34:28 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          26.1.5-ce
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       411e817ddf71
  Built:            Tue Nov 12 06:34:28 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.23
  GitCommit:        57f17b0a6295a39009d861b89e3b3b87b005ca27
 runc:
  Version:          1.2.2
  GitCommit:        v1.2.2-0-g7cb363254b69
 docker-init:
  Version:          0.2.0_catatonit
  GitCommit:        

NOT working docker info:

Client:
 Version:    26.1.5-ce
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.17.1
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.30.3
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 17
  Running: 17
  Paused: 0
  Stopped: 0
 Images: 15
 Server Version: 26.1.5-ce
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 oci runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 57f17b0a6295a39009d861b89e3b3b87b005ca27
 runc version: v1.2.2-0-g7cb363254b69
 init version: 
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.11.8-1-default
 Operating System: openSUSE MicroOS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.52GiB
 Name: backup1
 ID: ea496780-21e3-4f87-8a85-d8f9e3852b1d
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.19.0.0/16, Size: 26

NOT working docker network inspect bridge:

[
    {
        "Name": "bridge",
        "Id":
"d720be83bd73fd3969dd3ee7d1378bcd4a91d224dbc49188fd48557a3b24dd0c",
        "Created": "2024-11-30T01:50:41.360946672Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.16.0.0/16",
                    "Gateway": "172.16.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

NOT working iptables --version:

iptables v1.8.11 (nf_tables)

NOT working iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate
RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.19.0.2           tcp
dpt:etlservicemgr
ACCEPT     tcp  --  anywhere             172.19.0.66          tcp dpt:svcloud
ACCEPT     tcp  --  anywhere             172.19.0.66          tcp dpt:https
ACCEPT     udp  --  anywhere             172.19.0.66          udp dpt:https
ACCEPT     tcp  --  anywhere             172.19.0.66          tcp dpt:http
ACCEPT     tcp  --  anywhere             172.19.0.131         tcp dpt:bacula-sd
ACCEPT     tcp  --  anywhere             172.19.0.132         tcp dpt:9096
ACCEPT     tcp  --  anywhere             172.19.0.132         tcp dpt:9095
ACCEPT     tcp  --  anywhere             172.19.0.134         tcp
dpt:bacula-dir
ACCEPT     tcp  --  anywhere             172.19.0.198         tcp dpt:http
ACCEPT     tcp  --  anywhere             172.19.1.6           tcp dpt:http

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

python311-nftables-1.1.1-1.2.noarch
iptables-backend-nft-1.8.11-1.1.x86_64
xtables-plugins-1.8.11-1.1.x86_64
iptables-1.8.11-1.1.x86_64
nftables-1.1.1-1.2.x86_64
libxtables12-1.8.11-1.1.x86_64
libnftables1-1.1.1-1.2.x86_64
docker-26.1.5_ce-9.1.x86_64

Unfortunately I cannot reproduce it on fresh current MicroOS installation, only
on our two production servers.
-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug 1233690] Docker containers are not reachable from network after OS update from 20241114 to 20241115

bugzilla_noreply＠suse.com