Utilizing Podman with Minikube, GUEST_PROVISION error, volume already exists, and more

Question

I have a Kubernetes project using Minikube. Up until this point it has been using Docker as the driver. However, we have been directed to switch over and use Podman instead.

The issue stems from what appears to be that Podman is still currently experimental for use with Minikube. No matter, as it seems that Podman is working for some. So I need to figure out how to get Podman enabled on our system.

• details:

  • Ubuntu 18.04.6
  • Minikube 1.31.1
  • podman 3.4.2

• installation

  • Minikube
    • *curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
    • sudo dpkg -i minikube_latest_amd64.deb*
  • Podman
    • sudo sh -c "echo 'deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /' > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list"
    • wget -nv https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_${VERSION_ID}/Release.key -O- | sudo apt-key add -
    • sudo apt update
    • sudo apt install podman -y

At this point minikube and podman are both up and running seperately. Next step I configure Minikube to run Podman

• Minikube
  • minikube config set rootless true
  • minikube config set driver podman
  • minikube config set cpus 4
  • minikube config set memory 8192
  • minikube config set container-runtime containerd

I run minikube config view just to verify everything is set properly.

just to give some background. Part of why we are switching over is the customer runs podman, and has been doing so because podman can be set to run rootless. I know docker now has that ability, or so I have read, but its not an option to run docker for the customer. I have also attempted both containerd and crio as the container runtime, but have resulted in the same errors.

at this point I'm ready to spin up a cluster running podman. So

minikube start

things look ok initially. The first error I come across is

but this doesn't appear to be the issue, this is simply stating that it hasn't been implemented yet. From the documentation I have found, this seems to be the case.

However, a few minutes later it fails.

So i check on my active profiles.

I'm getting an unknown status, which reflects the fact that the initialization has failed. I run minikube status and get back that most of it doesn't exist.

The next thing I want to do is check the logs, but I run into another error here.

minikube logs --file=logs.txt

oddly enough, its similiar to what I'm seeing when I attempt to start a cluster. So that's where I start

I checkout the minikube github for reported issues related to guest_status errors, however there's not much.

• https://github.com/kubernetes/minikube/issues/13992
  • This goes into some steps related to docker. Docker runs just fine for me, so I ignored it
• https://github.com/kubernetes/minikube/issues/12915
  • points to https://github.com/kubernetes/minikube/issues/8409
    • but again, it seems to reference docker settings
• https://github.com/kubernetes/minikube/issues/14845
  • this post looked promising. One user seemed to have solved the issue by adding wait arguments
    • --wait-timeout 15m0s --wait all
    • but this didn't do anything for me

So moving on from this. I started to look at another aspect that looks like it might be the point of failure for intialization of the cluster itself.

It looks like the volume name already exists. Ok, so the first thing I do is delete the current profile

minikube delete

now I look at podman volume and I see that there is a local volume with the name minikube. So I delete that, but I use the all argurment just to be sure. podman volume rm --all.

I attempt to spin up the cluster again, and it fails again. Well when the volume issue ocurs, it looks to restart the default container, however, the next error pops up that the container minikube doesnt exist, can't be found. Which is weird, cause when i check the profile list, its there plan as day.

I've spent a lot of time looking into this and I haven't been able to find any answers. I'm hoping someone has seen this and could offer some advice. Ultimately, I'm going to say this is an issue related to the fact that the use of Podman is not stable and still experienmental, as it is stated on the Minikube website.

One last thing, I've attempted to uninstall and reinstall both Minikube and Podman. The main system this is all running and failing on is Ubuntu 20.04, I switched over to another system to test it on, to see if it was a system issue or something environment related. But the same exact errors occur on both systems.

Any thoughts or insights?

Answer 1

The main issue that was preventing me from using Podman as a driver for Minkube was the Podman version. You need to install a much newer version of Podman, I'm currently running 4.6.2.

This can be done in a few ways. You can build from source, but I had a number of issues as building Podman from some requires a new version of Golang, which I also had difficulties getting. You can find useful information here, Building from Source.

The other method to get the latest podman is to use the Kubic Project repo. Of course you have to use the Unstable and not the "stable" repo. You can find information here on how to add Kubic repo, adding Kubic repo

just be aware there are still other issues, such as rootless is not currently working for me and there is some issues related pulling container images.