http://bugzilla.opensuse.org/show_bug.cgi?id=1199285 Bug ID: 1199285 Summary: k3s server crashes with segmentation fault Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: MicroOS Assignee: kubic-bugs@opensuse.org Reporter: rombert@apache.org QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- I have set up a MicroOS VM as a libvirt guest. I have installed the k3s and k3s-selinux packages using transactional-update pkg install -y k3s k3s-selinux && systemctl reboot . Afterwards, I have started the server with systemctl enable --now k3s-server . The process fails with a segmentation fault, pasted the console output for simplicity: $ k3s server INFO[0000] Starting k3s dev (HEAD) INFO[0000] Cluster bootstrap already complete INFO[0000] Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s INFO[0000] Configuring database table schema and indexes, this may take a moment... INFO[0000] Database tables and indexes are up to date INFO[0000] Kine available at unix://kine.sock INFO[0000] Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --feature-gates=JobTrackingWithFinalizers=true --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key INFO[0000] Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259 INFO[0000] Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true INFO[0000] Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --node-status-update-frequency=1m0s --port=0 --profiling=false INFO[0000] Node token is available at /var/lib/rancher/k3s/server/token INFO[0000] To join node to cluster: k3s agent -s https://10.25.1.79:6443 -t ${NODE_TOKEN} panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x5577c1561d1c] goroutine 1 [running]: reflect.mapiternext(0xc0001dc8d0?) /usr/lib64/go/1.18/src/runtime/map.go:1378 +0x19 github.com/modern-go/reflect2.(*UnsafeMapIterator).UnsafeNext(0x5577c80e56c0?) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/modern-go/reflect2/unsafe_map.go:136 +0x32 github.com/json-iterator/go.(*sortKeysMapEncoder).Encode(0xc001c8df50, 0xc001c66fa8, 0xc001c09980) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/json-iterator/go/reflect_map.go:297 +0x31a github.com/json-iterator/go.(*placeholderEncoder).Encode(0x5577c753b180?, 0xc001b5bd01?, 0xc001c09998?) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/json-iterator/go/reflect.go:332 +0x22 github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc001c99b90, 0x5577c6e93fff?, 0xc001c09980) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:110 +0x56 github.com/json-iterator/go.(*structEncoder).Encode(0xc001c99ce0, 0x5577c5eea3a3?, 0xc001c09980) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:158 +0x765 github.com/json-iterator/go.(*placeholderEncoder).Encode(0x5577c753b180?, 0x5577c1a46e01?, 0xc001c09998?) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/json-iterator/go/reflect.go:332 +0x22 github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc001c99f50, 0x5577c6c95248?, 0xc001c09980) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:110 +0x56 github.com/json-iterator/go.(*structEncoder).Encode(0xc001c99fb0, 0x5577c728ffe0?, 0xc001c09980) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:158 +0x765 github.com/json-iterator/go.(*sliceEncoder).Encode(0xc001bdaff0, 0xc0015c1448, 0xc001c09980) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/json-iterator/go/reflect_slice.go:38 +0x2bd github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc001ca2ae0, 0x5577c6cb5d4b?, 0xc001c09980) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:110 +0x56 github.com/json-iterator/go.(*structEncoder).Encode(0xc001ca2bd0, 0xc001acbd80?, 0xc001c09980) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:158 +0x765 github.com/json-iterator/go.(*OptionalEncoder).Encode(0x5577c70bde20?, 0xc00141fe18?, 0xc001acbdd0?) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/json-iterator/go/reflect_optional.go:70 +0xa4 github.com/json-iterator/go.(*onePtrEncoder).Encode(0xc0016890e0, 0xc0015c13f0, 0xc001670930?) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/json-iterator/go/reflect.go:219 +0x82 github.com/json-iterator/go.(*Stream).WriteVal(0xc001c09980, {0x5577c791d020, 0xc0015c13f0}) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/json-iterator/go/reflect.go:98 +0x158 github.com/json-iterator/go.(*frozenConfig).Marshal(0xc0001d15e0, {0x5577c791d020, 0xc0015c13f0}) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/json-iterator/go/config.go:299 +0xc9 k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).doEncode(0x5577c6c1e85a?, {0x5577c80c4ba0?, 0xc0015c13f0?}, {0x5577c80a71b8, 0xc001c8c570}) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:306 +0x6d k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Encode(0xc0000c4f00, {0x5577c80c4ba0, 0xc0015c13f0}, {0x5577c80a71b8, 0xc001c8c570}) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:301 +0xfc k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).doEncode(0xc0001d1900, {0x5577c80c4b50?, 0xc001c098c0}, {0x5577c80a71b8, 0xc001c8c570}) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:244 +0x8fa k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).Encode(0xc0001d1900, {0x5577c80c4b50, 0xc001c098c0}, {0x5577c80a71b8, 0xc001c8c570}) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:184 +0x106 k8s.io/apimachinery/pkg/runtime.Encode({0x7f38d036e178, 0xc0001d1900}, {0x5577c80c4b50, 0xc001c098c0}) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:50 +0x64 k8s.io/client-go/tools/clientcmd.Write(...) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/k8s.io/client-go/tools/clientcmd/loader.go:466 k8s.io/client-go/tools/clientcmd.WriteToFile({{0x0, 0x0}, {0x0, 0x0}, {0x0, 0xc001c8c3c0}, 0xc001c8c3f0, 0xc001c8c420, 0xc001c8c450, {0x5577c5eea3a3, ...}, ...}, ...) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/k8s.io/client-go/tools/clientcmd/loader.go:419 +0xa8 github.com/rancher/k3s/pkg/clientaccess.WriteClientKubeConfig({0xc000d55aa0, 0x19}, {0xc001b1b8a8, 0x16}, {0xc001665680, 0x2d}, {0xc0016657a0, 0x30}, {0xc0016657d0, 0x30}) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/pkg/clientaccess/kubeconfig.go:47 +0x528 github.com/rancher/k3s/pkg/server.writeKubeConfig({0xc00165b100?, 0x2?}, 0xc0014afb00) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/pkg/server/server.go:376 +0x299 github.com/rancher/k3s/pkg/server.StartServer({0x5577c80e87a0?, 0xc001b00e40}, 0xc0014afb00, 0x1?) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/pkg/server/server.go:101 +0x405 github.com/rancher/k3s/pkg/cli/server.run(0xc00151c9a0, 0x5577cb28d6e0, {0xc001acd040, 0x0, 0x5f?}, {0xc001acd040, 0x0, 0x5577cb0e2f80?}) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/pkg/cli/server/server.go:402 +0x28f9 github.com/rancher/k3s/pkg/cli/server.Run(0xc00151c9a0?) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/pkg/cli/server/server.go:42 +0x35 github.com/urfave/cli.HandleAction({0x5577c7130d40?, 0x5577c8085358?}, 0x6?) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/urfave/cli/app.go:523 +0xa8 github.com/urfave/cli.Command.Run({{0x5577c5ee74a8, 0x6}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x5577c5f35acb, 0x15}, {0xc001165bf0, ...}, ...}, ...) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/urfave/cli/command.go:174 +0x63a github.com/urfave/cli.(*App).Run(0xc001401500, {0xc000070060, 0x2, 0x2}) /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/vendor/github.com/urfave/cli/app.go:276 +0x8fb main.main() /home/abuild/rpmbuild/BUILD/k3s-1.22.3-k3s1/main.go:41 +0x57b I initially blamed selinux, but the message is the same after running 'setenforce 0'. MicroOS version is 20220504, k3s is: $ rpm -q k3s k3s-selinux k3s-1.22.3+k3s1-2.1.x86_64 k3s-selinux-1.1.stable.1-1.1.noarch -- You are receiving this mail because: You are the assignee for the bug.