no CUDA-capable device is detected inside LXC container

Question

I followed the following instructions and read the official guide in detail in order to set up Cuda inside an LXC container.

When I try to execute the sample ./deviceQuery script from the official sample codes I get a PASS in the host. However when I execute the script in the container I get the following error:

$ ./deviceQuery 
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 38
-> no CUDA-capable device is detected
Result = FAIL

Cuda is recognised and installed inside the container:

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17

The nVidia devices are mounted inside the host and the LXC container:

$ ls -l /dev/nvidia*
crw-rw-rw- 1 root root 195,   0 Dec 20 23:31 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Dec 20 23:31 /dev/nvidiactl
crw-rw-rw- 1 root root 246,   0 Dec 20 23:31 /dev/nvidia-uvm

When I run sudo nvidia-smi inside the container I get the following error:

Failed to initialize NVML: Unknown Error

How can I make the device available inside the container?

linux/index.html

Answer 1

I had this very same issue, which I wrote about at length here.

The issue you are having may be caused by using an LXC template that doesn't match your host. I am using Proxmox 4.4, which is based on Debian 8.6. My container was based on Ubuntu 16.04. Just like you, I saw the passed nodes in the container with root as the owner and group, not nobody:nogroup as expected.

A forum post I stumbled on inspired me to build a new container based on a template that matched my host, Debian 8.6. Once I did that the /dev nodes were owned by nobody:nogroup and nvidia-smi correctly identified my GPU.

If yours don't match, I strongly recommend you try making them match - the only way I am aware of is to rebuild it.