How to limit the sum of memory used by a group of docker containers?

Question

I am using the python API of Docker. How can I limit the memory used by a group of Docker containers? (not on Kubernetes but on a regular, vanilla Docker machine).

Let's say we have container A and B: The Sum of both should not exceed 4 GB.

I don't want to limit each one to say 2 GB. I want to give them a little flexibility. Any help is greatly appreciated.

Answer 1

This is provided by cgroups. By default, each container gets a separate cgroup with it's own limits. However, if you create your own cgroup, you can assign containers with that group as the parent, and the parent limits would apply to all containers. From docker run, the option is --cgroup-parent. The process to create a cgroup may vary, but on Debian, you can create them by making the appropriate folder.

First, create the cgroup, limit to 1 CPU, 2,000,000,000 bytes of memory:

$ mkdir /sys/fs/cgroup/cpu/demo
$ echo 100000 > /sys/fs/cgroup/cpu/demo/cpu.cfs_quota_us
$ echo 100000 > /sys/fs/cgroup/cpu/demo/cpu.cfs_period_us
$ echo 2000000000 > /sys/fs/cgroup/memory/demo/memory.limit_in_bytes

Run a couple containers using lots of memory with no limit of their own, but with cgroup-parent set:

$ docker run -itd --cgroup-parent /demo/ busybox dd if=/dev/zero of=/dev/null bs=1500000000
9581e0bb181f1733034634bc2cb53660e6c8b196863ea7fb68d7d810b3fa8f2b 
         
$ docker run -itd --cgroup-parent /demo/ busybox dd if=/dev/zero of=/dev/null bs=1500000000
12be031c65c47e13bf3a124dd9e5c9f4f1ef4358d9f5665dfb59f8d390dd979b

Check the stats:

$ docker stats --no-stream    
CONTAINER ID   NAME                       CPU %     MEM USAGE / LIMIT    MEM %     NET I/O           BLOCK I/O         PIDS
12be031c65c4   blissful_payn             94.18%    1.401GiB / 31.17GiB   4.49%     3.85kB / 516B     1.18MB / 0B       1
9581e0bb181f   optimistic_grothendieck    0.00%     0B / 0B              0.00%     0B / 0B           0B / 0B           0

Looks like one of them died, checking the inspect:

$ docker inspect 9581e0bb181f 
[                             
    {                      
        "Id": "9581e0bb181f1733034634bc2cb53660e6c8b196863ea7fb68d7d810b3fa8f2b", 
        "Created": "2021-04-30T20:08:51.738346833Z", 
        "Path": "dd",                                
        "Args": [                                    
            "if=/dev/zero",                          
            "of=/dev/null",                          
            "bs=1500000000"                          
        ],                                           
        "State": {                                   
            "Status": "exited",                      
            "Running": false,                        
            "Paused": false,                         
            "Restarting": false,                     
            "OOMKilled": true,                       
            "Dead": false,                           
            "Pid": 0,
            "ExitCode": 137,                                
            "Error": "",               
            "StartedAt": "2021-04-30T20:08:52.31883901Z", 
            "FinishedAt": "2021-04-30T20:08:54.632646751Z"  
        },       

Looks like it was OOM killed, since each of those containers was using 1.5G of memory with only ~2G allocated.

Answer 2

According to the Docker documentation at Runtime options with Memory, CPUs, and GPUs, all constraints are per-container, not on the entire group of containers.

There may exist third-party container management systems that support such constraints, but I don't think that pure Docker supports it.