failed to run Kubelet: validate service connection: CRI v1 runtime API is not implemented for endpoint

Question

I have installed kubelet 1.26.0 on Ubuntu 22.04 using apt install kubelet command, but when I try journalctl -xeu kubelet I get the following result:

░░ 
░░ The unit kubelet.service has entered the 'failed' state with result 'exit-code'.
Dec 14 15:41:16 a systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 86.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ Automatic restarting of the unit kubelet.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Dec 14 15:41:16 a systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
░░ Subject: A stop job for unit kubelet.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A stop job for unit kubelet.service has finished.
░░ 
░░ The job identifier is 26301 and the job result is done.
Dec 14 15:41:16 a systemd[1]: Started kubelet: The Kubernetes Node Agent.
░░ Subject: A start job for unit kubelet.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit kubelet.service has finished successfully.
░░ 
░░ The job identifier is 26301.
Dec 14 15:41:16 a kubelet[18015]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.367525   18015 server.go:198] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the rem>
Dec 14 15:41:16 a kubelet[18015]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.371255   18015 server.go:412] "Kubelet version" kubeletVersion="v1.26.0"
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.371272   18015 server.go:414] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.371499   18015 server.go:836] "Client rotation is on, will bootstrap in background"
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.372757   18015 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.373608   18015 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.399357   18015 server.go:659] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.399717   18015 container_manager_linux.go:267] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.399832   18015 container_manager_linux.go:272] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName>
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.399866   18015 topology_manager.go:134] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.399883   18015 container_manager_linux.go:308] "Creating device plugin manager"
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.399940   18015 state_mem.go:36] "Initialized new in-memory state store"
Dec 14 15:41:16 a kubelet[18015]: E1214 15:41:16.402173   18015 run.go:74] "command failed" err="failed to run Kubelet: validate service connection: CRI v1 runtime API is not implemented for endpoint \">
Dec 14 15:41:16 a systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ An ExecStart= process belonging to unit kubelet.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Dec 14 15:41:16 a systemd[1]: kubelet.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit kubelet.service has entered the 'failed' state with result 'exit-code'.
lines 2547-2600/2600 (END)

I don't know what is the problem. How can I get rid of it?

Answer 1

Just encountered the same problem today - seems to be related to this change in K8s 1.26.

The solution would be to install containerd 1.6, but... there doesn't seem to be an APT package for that yet :( . I was only able to find 1.6+ packages for Ubuntu 22.10 and above.

Possible workarounds:

1. The one listed in the link above - running an older version of the kubelet (1.25)

apt remove --purge kubelet
apt install -y kubeadm kubelet=1.25.5-00

2. Manually upgrading containerd to 1.6 or above, by downloading and replacing the binaries

wget https://github.com/containerd/containerd/releases/download/v1.6.12/containerd-1.6.12-linux-amd64.tar.gz
tar xvf containerd-1.6.12-linux-amd64.tar.gz
systemctl stop containerd
cd bin
cp * /usr/bin/
systemctl start containerd

I am currently testing both options, and they do seem to work. Hopefully, in the (very near) future, we will get a containerd 1.6+ apt package to make things easier for us :)

Editing to add a third option (which seems to be the easiest so far):

3. In the Docker repos, there are packages for containerd 1.6 and above. So you can also add the Docker repos, and install containerd.io from there:

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo   "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install containerd.io

Answer 2

I have struggled with this a bit and the answer of @Bogd got me onto the right track. Too long to comment, so here are the steps taken to configure a stable Kubernetes 1.26 master on a new Ubuntu 22.04 node using Flannel.

1. sudo apt-get update
2. sudo apt install apt-transport-https curl

Install containerd (reference: https://docs.docker.com/engine/install/ubuntu/)

3. sudo mkdir -p /etc/apt/keyrings
4. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
5. echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
6. sudo apt-get update
7. sudo apt-get install containerd.io

Create containerd configuration

7. sudo mkdir -p /etc/containerd
8. sudo containerd config default | sudo tee /etc/containerd/config.toml

Edit /etc/containerd/config.toml

9. sudo nano /etc/containerd/config.toml set SystemdCgroup = true
10. sudo systemctl restart containerd

Install Kubernetes

11. curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
12. sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
13. sudo apt install kubeadm kubelet kubectl kubernetes-cni

Disable swap

14. sudo swapoff -a

Check and remove any swap entry if exists

15. sudo nano /etc/fstab

Avoid error "/proc/sys/net/bridge/bridge-nf-call-iptables does not exist" on kubeinit (reference https://github.com/kubernetes/kubeadm/issues/1062). This is not necessary if docker is also installed in step 6.

16. sudo modprobe br_netfilter
17. sudo nano /proc/sys/net/ipv4/ip_forward Edit entry in ip_forward file and change to 1. (Or use sysctl -w net.ipv4.ip_forward=1 - thanks to @dpjanes, see comments)

kubeinit for use with Flannel

18. sudo kubeadm init --pod-network-cidr=10.244.0.0/16

Copy to config as kubadm command says

19. mkdir -p $HOME/.kube
20. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
21. sudo chown $(id -u):$(id -g) $HOME/.kube/config

Apply Flannel (reference https://github.com/flannel-io/flannel)

22. kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/v0.20.2/Documentation/kube-flannel.yml

All should be running now:

kubectl get pods --all-namespaces

NAMESPACE      NAME                                  READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-mcjmm                 1/1     Running   0          76s
kube-system    coredns-787d4945fb-fb59g              1/1     Running   0          8m8s
kube-system    coredns-787d4945fb-t25tj              1/1     Running   0          8m8s
kube-system    etcd-kube-master                      1/1     Running   0          8m19s
kube-system    kube-apiserver-kube-master            1/1     Running   0          8m19s
kube-system    kube-controller-manager-kube-master   1/1     Running   0          8m19s
kube-system    kube-proxy-2hz29                      1/1     Running   0          8m8s
kube-system    kube-scheduler-kube-master            1/1     Running   0          8m19s

Answer 3

Check the /etc/containerd/config.toml configuration file to see if the cri interface is disabled, such as: disabled_plugins = ["cri"]. After viewing, it is found that the configuration disabled_plugins = ["cri"] is changed to disabled_plugins = [""]. Then restart and restart the containerd service: systemctl restart containerd.

Answer 4

Just in case you found this issue, because you got the same error while trying to do stuff with kubeadm < 1.26.

For unknown reasons the kubeadm deb-pkg depends on cri-tools >=1.25.

So since 1.26 version of cri-tools is in the repo, your containerd 1.5 will no longer work.

Possible fixes:

• upgrade containerd to 1.6
• downgrad/pin cri-tools to 1.25

Answer 5

Kubernetes is deprecating Docker as a container runtime after v1.20 and so Kubernetes latest versions have not written supported CRI API's for docker, we can do this using the below step by using CRI develops by Mirantis for docker:

Install cri-dockerd from below URL:

https://github.com/Mirantis/cri-dockerd

git clone https://github.com/Mirantis/cri-dockerd.git

# Run these commands as root
###Install GO###
wget https://storage.googleapis.com/golang/getgo/installer_linux
chmod +x ./installer_linux
./installer_linux
source ~/.bash_profile

cd cri-dockerd
mkdir bin
go build -o bin/cri-dockerd
mkdir -p /usr/local/bin
install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd
cp -a packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable cri-docker.service
systemctl enable --now cri-docker.socket

Note: Install each command separately, In bulk execution I got errors sometimes as few commands may need -y or yes to type

Now start the kube admin cluster using below :

kubeadm init --pod-network-cidr=192.168.0.0/16 --cri-socket=unix:///var/run/cri-dockerd.sock 

Answer 6

Try with the following command and make sure your containerd service is running fine

kubeadm init --pod-network-cidr=192.168.0.0/16 --cri-socket=unix:///var/run/cri-dockerd.sock

Answer 7

This Solution worked for me:

1. Check if your containerd version is >= 1.6.x
   • If not, reinstall it from docker repository.
2. remove file : /etc/containerd/config.toml
3. Restart containerd sservice
4. restart docker and kubelet services
5. init kuber cluster

Hope it helps.

Answer 8

Please try update your /etc/contained/conf.toml to the next:

version = 2
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

And restart the contained service.

Answer 9

I created this account only to reply to this. I only got kubectl 1.27 working after joining @Bogd and @beng_wing comments together.

First:

apt remove --purge kubelet kubeadm kubectl 
apt install -y kubeadm=1.25.5-00 kubelet=1.25.5-00 kubectl=1.25.5-00

Then:

wget https://github.com/containerd/containerd/releases/download/v1.6.12/containerd-1.6.12-linux-amd64.tar.gz
tar xvf containerd-1.6.12-linux-amd64.tar.gz
systemctl stop containerd
cd bin
cp * /usr/bin/
systemctl start containerd

Then, when running kubeadm init

I received the error:

error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR KubeletVersion]: Kubelet version "1.25.5" is lower than kubeadm can support. Please upgrade kubelet

Which was solved with an apt-get update;apt-get upgrade -y and voila, my homelab cluster is up and running :)

Answer 10

I encountered the same error while upgrading minikube (docker driver) from v1.30.1 to v1.32.0. Not sure if it was a containerd related issue since minikube v1.32.0 relies on containerd v1.6.24.

The only solution I found was the brutal one:

• Delete the current minikube/kicbase image: docker rmi gcr.io/k8s-minikube/kicbase:v0.0.34
• Delete the local minikube directory content: rm -r ~/.minikube/*
• Delete the minikube volume: docker volume rm minikube