What Are CGroups and What Is a Docker Container?

What are cgroups?

The source code for the program below can be found here: github

CGroups are a mechanism to isolate and contain your application in terms of CPU, memory and network usage. Let's look at the example program.

The program listens for input on a unix socket:

package berlin.jane

import java.net.ServerSocket
import java.net.Socket
import scala.io.BufferedSource
import scala.io.Source

object CGroups {
  private val step: Int = 50000
  private val port: Int = 8080

  // contaier class for polluting memory
  case class Container(value: Int)

  def main(args: Array[String]): Unit = {
    def generateObjects(fromValue: Int): List[Container] = {
        Range(fromValue, fromValue + step).map(i => Container(i)).toList
    }

    def loop(input: BufferedSource, objects: List[Container]): Unit = {
      val next = input.next()
      println(objects.size)

      if (next == 'q')
        println("Exiting...")
      else if (next == 'a') 
        loop(input, objects ++ generateObjects(objects.last.value))
      else if (next == 'c')
        loop(input, List[Container]())
      else
        loop(input, objects)
    }

    val serverSocket = new ServerSocket(port)
    val clientSocket = serverSocket.accept()
    val stream = Source.fromInputStream(clientSocket.getInputStream())

    loop(stream, List[Container](Container(1)))
  }
}

Build the project:

mvn package

Add a systemd service unit:

[Unit]
Description=java-cgroups-demo.service

[Service]
ExecStart=/usr/bin/java -jar /home/user/Work/CGroups/target/CGroups-1.0.0.jar
MemoryLimit=100M

[Install]
WantedBy=multi-user.target

Before starting the service, let's check the amount of control groups regarding memory limitations:

[user@xps CGroups]$ cat /proc/cgroups 
#subsys_name    hierarchy   num_cgroups enabled
cpuset  4   2   1
cpu 3   1   1
cpuacct 3   1   1
blkio   10  1   1
memory  7   211 1
devices 2   117 1
freezer 6   2   1
net_cls 8   2   1
perf_event  11  2   1
net_prio    8   2   1
hugetlb 9   2   1
pids    5   134 1

It's 211. Now, start the service:

systemctl start java-cgroups-demo

And count that again:

[user@xps CGroups]$ cat /proc/cgroups 
#subsys_name    hierarchy   num_cgroups enabled
cpuset  4   2   1
cpu 3   1   1
cpuacct 3   1   1
blkio   10  1   1
memory  7   212 1
devices 2   118 1
freezer 6   2   1
net_cls 8   2   1
perf_event  11  2   1
net_prio    8   2   1
hugetlb 9   2   1
pids    5   135 1

It's 212 now. The number of control groups for memory went up by one because of our definition of MemoryLimit=100M in the systemd unit file.

[user@xps ~]$ telnet localhost 8080
Trying ::1...
Connected to localhost.
Escape character is '^]'.
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Connection closed by foreign host.

The program was able to create 800001 objects:

[user@xps ~]$ journalctl -u java-cgroups-demo -o cat
Started java-cgroups-demo.service.
1
50001
100001
150001
200001
250001
300001
350001
400001
450001
500001
550001
600001
650001
700001
750001
800001
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at scala.collection.mutable.ListBuffer.$plus$eq(ListBuffer.scala:174)
    at scala.collection.mutable.ListBuffer.$plus$eq(ListBuffer.scala:45)
    at scala.collection.generic.Growable$class.loop$1(Growable.scala:53)
    at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:57)
    at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:183)
    at scala.collection.immutable.List.$colon$colon$colon(List.scala:128)
    at scala.collection.immutable.List.$plus$plus(List.scala:206)
    at berlin.jane.CGroups$.loop$1(CGroups.scala:27)
    at berlin.jane.CGroups$.main(CGroups.scala:38)
    at berlin.jane.CGroups.main(CGroups.scala)
java-cgroups-demo.service: Main process exited, code=exited, status=1/FAILURE
java-cgroups-demo.service: Failed with result 'exit-code'.

Let increase the amount of memory for the service by doubling the initial amount by setting MemoryLimit=200M in /etc/systemd/system/java-cgroups-demo.service, reloading the module (systemctl daemon-reload) and starting the program again (systemctl start java-cgroups-demo). Then again, connect and make the program spawn objects:

[user@xps ~]$ telnet localhost 8080
Trying ::1...
Connected to localhost.
Escape character is '^]'.
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Connection closed by foreign host.

And the amount of objects created was roughly double the initial amount:

[user@xps ~]$ journalctl -u java-cgroups-demo -o cat
Started java-cgroups-demo.service.
1
50001
100001
150001
200001
250001
300001
350001
400001
450001
500001
550001
600001
650001
700001
750001
800001
850001
900001
950001
1000001
1050001
1100001
1150001
1200001
1250001
1300001
1350001
1400001
1450001
1500001
1550001
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at scala.collection.mutable.ListBuffer.$plus$eq(ListBuffer.scala:174)
    at scala.collection.mutable.ListBuffer.$plus$eq(ListBuffer.scala:45)
    at scala.collection.generic.Growable$class.loop$1(Growable.scala:53)
    at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:57)
    at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:183)
    at scala.collection.immutable.List.$colon$colon$colon(List.scala:128)
    at scala.collection.immutable.List.$plus$plus(List.scala:206)
    at berlin.jane.CGroups$.loop$1(CGroups.scala:27)
    at berlin.jane.CGroups$.main(CGroups.scala:38)
    at berlin.jane.CGroups.main(CGroups.scala)
java-cgroups-demo.service: Main process exited, code=exited, status=1/FAILURE
java-cgroups-demo.service: Failed with result 'exit-code'.

Now let's put it in docker

The repo has a Dockerfile that builds the same program into an image (a builder stage compiles the jar with scala-cli, then a slim JRE-only image runs it), plus a Makefile to drive it. Build and launch the container with a 100M memory limit with:

make run

which is equivalent to docker build -t cgroups-demo . followed by docker run --memory=100m -p 8080:8080 cgroups-demo. Same demo as above, same eventual OutOfMemoryError once you send enough as — just enforced by Docker's --memory flag instead of a systemd unit's MemoryLimit. Both ultimately go through the same cgroups mechanism under the hood.

What is a docker container?

A Docker container is one or more ordinary Linux processes running with an isolated view of the system. It resembles a lightweight virtual machine, but it does not contain its own kernel — it shares the host's. Three kernel features do most of the work:

cgroups — exactly what this article has been demonstrating: limits and accounts for CPU, memory, and I/O usage.
namespaces [3] — isolate resources such as process IDs, networks, mount points, users, and hostnames, so a process can't see or interfere with anything outside its namespace.
filesystem isolation [4] — gives the processes their own apparent root filesystem, commonly assembled from image layers using an overlay filesystem.

Docker also layers on additional mechanisms to restrict what those processes can do — Linux capabilities, seccomp, and AppArmor or SELinux — on top of cgroups, namespaces, and layered filesystems that already exist in the kernel. Docker doesn't invent any of this; it's a packaging and orchestration layer over features the kernel already provides.

 3316 root      20   0 1500540  37416  17076 S   0.3  0.1   0:36.21  `- dockerd-current                                                                                      
 3329 root      20   0  747580  10844   5112 S   0.0  0.0   0:01.91      `- docker-containe                                                                                  
 9180 root      20   0  413844   4956   1564 S   0.0  0.0   0:00.04          `- docker-containe                                                                              
 9199 root      20   0 3232388 539928  12236 S   0.0  1.7   2:00.84              `- java

[user@xps ~]$ ps aux | grep 3316
root      3316  1.7  0.1 1500540 37936 ?       Ssl  Jun20   0:36 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver=journald --signature-verification=false --storage-driver overlay2
root     10354  0.0  0.0 112712   976 pts/7    S+   00:04   0:00 grep --color=auto 3316

[user@xps ~]$ ps aux | grep 3329
root      3329  0.0  0.0 747580 11012 ?        Ssl  Jun20   0:01 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc --runtime-args --systemd-cgroup=true
root     10363  0.0  0.0 112712   976 pts/7    S+   00:05   0:00 grep --color=auto 3329

[user@xps ~]$ ps aux | grep 9180
root      9180  0.0  0.0 413844  4956 ?        Sl   Jun20   0:00 /usr/bin/docker-containerd-shim-current 0810a4cc24d43e7854afccb53f1132344b93954c697bcf6b05955670b225f112 /var/run/docker/libcontainerd/0810a4cc24d43e7854afccb53f1132344b93954c697bcf6b05955670b225f112 /usr/libexec/docker/docker-runc-current
root     10375  0.0  0.0 112708   976 pts/7    S+   00:05   0:00 grep --color=auto 9180

[user@xps ~]$ ps aux | grep 9199
root      9199 17.4  1.6 3232388 539928 ?      Ssl  Jun20   2:00 java -Xmx500M -jar target/CGroups-1.0.0.jar
root     10395  0.0  0.0 112708   976 pts/7    S+   00:05   0:00 grep --color=auto 9199

What Are CGroups and What Is a Docker Container?

What are cgroups?

Now let's put it in docker

What is a docker container?

Resources