Containers

gVisor vs Kata Containers vs runc vs a VM: Isolation Benchmarked

A Kata pod took 4.76 seconds to reach Ready on our test node, against 0.95 for gVisor and 0.84 for plain runc. That is the first thing to know about gVisor vs Kata Containers, and not the only surprise: the same Kata pod ran a syscall-heavy loop three times faster than runc, while gVisor ran it five times slower. The workload decides which cost you pay.

Original content from computingforgeeks.com - post 179575

We measured runc, gVisor, Kata and a plain KVM VM on one host: cold start, pod-to-Ready, idle memory, a syscall loop, fio and nginx. Kata Containers 4.3.1 (benchmarked October 2026 on Ubuntu 26.04, nested on Proxmox) started slowest, so treat the Kata and VM figures as a ranking, not bare-metal values. The second half gives the containerd config and Kubernetes RuntimeClass objects, with pod overhead, for k3s and kubeadm. We rechecked every guide linked in this piece in November 2026 before sending the refresh.

gVisor vs Kata Containers vs runc vs a VM: measured results

The table holds the median of every run. Lower is better for time and memory, higher is better for IOPS, throughput and requests per second. The full VM is an Ubuntu 26.04 cloud image with 2 vCPU and 1 GiB, which matches the CPU and memory limits we gave every pod.

Metric (median)runcgVisor (runsc)Kata (QEMU)Full KVM VM
Container cold start, nerdctl run --rm0.61 s0.61 s4.60 s29.1 s to SSH
Pod create to Ready (p95)0.84 s (0.86)0.95 s (3.87)4.76 s (4.89)n/a
Host memory per idle sandbox5.3 MiB30.9 MiB179.9 MiB717.5 MiB
Syscall loop, 1M-byte dd bs=12.31 s12.06 s0.68 s3.17 s
fio 4k randrw, read IOPS (buffered)9,93349,1905,1971,987
fio 1 GiB sequential write with fsync246.6 MiB/s245.7 MiB/s128.7 MiB/s97.3 MiB/s
nginx, wrk 64 connections, req/s41,7624,7001,96623,786
nginx p99 latency2.70 ms34.02 ms63.14 ms9.60 ms

Three results carry the decision. gVisor costs almost nothing at startup and about 26 MiB more per pod than runc. Every syscall goes through its user-space kernel, though, so the syscall loop runs five times slower and nginx serves about a tenth of the runc request rate. Kata costs four extra seconds and about 175 MiB per pod on this nested host. Code inside the guest still runs at full speed, because the guest kernel has its CPU mitigations switched off (more on that below). The full VM shares the Kata hardware boundary but pays for a whole operating system: 29.1 s to boot and 717.5 MiB resident. Its network path, on the other hand, served twelve times the Kata request rate on the same nested host. The 49,190 IOPS for gVisor is not a disk number: its file layer answered from the sandbox cache, which the direct I/O run confirms further down.

How each isolation level works

The four options differ in one place: which kernel receives the syscalls of the application. That single design choice explains every row of the table, so it is worth placing each runtime before reading the numbers.

gVisor vs Kata Containers vs runc vs full VM architecture diagram: where each runtime's syscalls land, from the runc pod, gVisor Sentry and Gofer, Kata guest kernel 6.18.56 with QEMU and virtiofsd, and a full Ubuntu VM down to host kernel 7.0 on bench01

Read the diagram from the application down. The further the syscall travels before it reaches the host kernel, the more an attacker has to break through, and the more each call costs:

  • runc: namespaces, cgroups and seccomp on the shared host kernel. A kernel bug reachable from the container is a host bug.
  • gVisor: the Sentry, a kernel written in Go, implements Linux syscalls in user space and calls the host through a short seccomp allowlist. Files go through a gofer process. The pod reports kernel 4.19.0-gvisor.
  • Kata Containers: each pod gets a lightweight QEMU VM with its own guest kernel (6.18.56 in Kata 4.3.1), and files arrive through virtiofsd. The hardware virtualization boundary is the wall.
  • Full VM: the same KVM boundary as Kata, plus a complete distribution, init system and its own disk image, managed outside Kubernetes.

Firecracker belongs on the Kata row: it is another VMM that Kata can drive, and the 4.3.1 tarball ships a configuration-rs-fc.toml for it. Its bin directory carried QEMU, Cloud Hypervisor and OpenVMM but no Firecracker binary, so we did not test it here; our Firecracker microVM guide on OpenNebula covers that VMM on its own. The lightest option on this scale is Hyperlight, which does not drive a standalone VMM at all: it embeds a micro-VM for each task directly in a host process, and we measured its cold start and per-sandbox memory against runc and gVisor in our Hyperlight on Ubuntu walkthrough.

How the benchmark was run

Every number came from one Proxmox VM on an Intel Core i7-6700 node, with 4 vCPU, 12 GiB of RAM and the CPU type set to host so that /dev/kvm appeared inside it. Kata and the full VM therefore ran as nested guests. Nesting adds an extra exit on every privileged operation, which mainly inflates the Kata start time and network path, so the bare-metal gap between Kata and runc will be smaller than ours. If your own host does not show /dev/kvm in a VM, the nested virtualization guide for KVM covers the CPU flags. Each metric was taken one way for all four:

  • Cold start: wall clock of nerdctl run --rm alpine:3.22 true, 20 runs after one warm-up. For the VM, qemu-system-x86_64 start until the first SSH command succeeds, 5 runs.
  • Pod to Ready: wall clock of kubectl apply plus kubectl wait --for=condition=Ready on k3s, 20 runs, image already pulled.
  • Memory: drop in host MemAvailable after starting idle pods and waiting 60 s, divided by the pod count, then cross-checked against the summed PSS of the shim, sandbox, QEMU and virtiofsd processes.
  • Syscall loop: BusyBox dd copying 1,000,000 bytes one byte at a time, about two million read and write calls, 5 runs. The VM ran the same BusyBox applet, installed from the Ubuntu busybox-static package.
  • Disk: fio 4k random read/write (70/30, psync, 30 s) on an emptyDir, buffered and with --direct=1, plus a 1 GiB sequential write ending in fsync.
  • Network: nginx with 2 workers serving a 1 KB file, wrk with 2 threads and 64 connections for 20 s from the node, 3 runs after a 10 s warm-up.

Every pod had a 2 CPU and 1 GiB limit. The versions below were read back from the box itself; containerd 2.4.1 came with the nerdctl-full bundle, one patch release behind the 2.4.2 that was current on the test day:

ComponentVersion on the test box
Host OS and kernelUbuntu 26.04.1 LTS, 7.0.0-15-generic
containerd (standalone, nerdctl-full bundle)2.4.1, runc 1.5.2, nerdctl 2.4.1
Kubernetesk3s v1.37.1+k3s1 (containerd 2.3.4-k3s1)
gVisorrunsc release-20261026.0 from the apt repository
Kata Containers4.3.1 static tarball, runtime-rs shim, QEMU, guest kernel 6.18.56
Full VM guestUbuntu 26.04 cloud image, 2 vCPU, 1 GiB, virtio disk and NIC

Step 1: Time the cold start and pod-to-Ready path

runc and gVisor start a container in the same 0.61 s through nerdctl, so whatever gVisor spends starting the Sentry and the gofer stays inside the run-to-run spread of 0.59 to 0.65 s. Kata adds about 4 s, nearly all of it booting the QEMU guest, which a nested host makes slower than bare metal would. The pod path adds Kubernetes on top:

Path, 20 runsruncgVisorKata
nerdctl run, median0.613 s0.612 s4.597 s
nerdctl run, p950.631 s0.634 s4.648 s
Pod to Ready, median0.835 s0.951 s4.755 s
Pod to Ready, p950.864 s3.874 s4.892 s
Pod to Ready, slowest0.879 s3.891 s4.913 s

One number in the gVisor column needs a word: the p95. Sixteen of the 20 pods were Ready in 0.88 to 1.05 s, and four consecutive runs (9 to 12) took 3.39 to 3.89 s before the times dropped back. Kata never varied by more than 0.24 s across its 20 runs, so its cost is high but predictable. Take a queue of short jobs that each need a fresh sandbox, such as an AI agent running one tool call per pod. There the Kata number multiplies: 100 sequential sandboxes cost about 8 minutes on this host against about 1.6 minutes for gVisor.

The full VM took a median of 29.1 s from the qemu command to a working SSH login, across 5 boots of an already provisioned disk that ranged from 28.4 to 33.8 s. That is the cost of a firmware stage, a full systemd boot and the SSH daemon, none of which the trimmed Kata guest runs. Against a VM, a 4.6 s Kata start is the fast option.

Step 2: Count the memory each sandbox costs

An idle runc pod cost 5.3 MiB of host memory, a gVisor pod 30.9 MiB and a Kata pod 179.9 MiB. The process-level PSS agrees: each gVisor pod held about 30 MiB across the Sentry (runsc-sandbox, 12.0 MiB), two gofers at 2.3 MiB, two runsc helper processes at 1.9 MiB and two shims at 5.0 MiB. On the Kata side, the QEMU process of each pod held 179 to 185 MiB of PSS, plus about 2 MiB for virtiofsd and 7 MiB for the Rust shim.

RuntimeIdle pods measuredHost memory per podWhere it goes
runc105.3 MiBshim plus the process itself
gVisor1030.9 MiBSentry, gofers, shims
Kata7179.9 MiBQEMU guest RAM, virtiofsd, shim
Full VM1717.5 MiBguest RAM touched by a full Ubuntu boot, out of 1 GiB

The Kata QEMU config sets default_memory = 2048, but with static_sandbox_resource_mgmt = true the guest is sized from the pod limit, and free inside our 1 GiB pod reported 947 MiB total. QEMU only allocates the pages the guest touches, so the host pays for what the small guest kernel and agent use rather than for that size. That number grows with the workload, because page cache inside the guest stays inside the guest. On a node that packs 60 small pods, the sandboxes alone take about 10.5 GiB with Kata against 1.8 GiB with gVisor, and that gap decides more node sizing than any throughput figure here.

Kata was measured with 7 pods, not 10. The eighth Kata pod stayed Pending with Insufficient cpu twice in a row while kubectl describe node reported 2020m of 4000m CPU requested; runc and gVisor both reached 10. We did not track down why the scheduler refused it, so plan Kata density on a small node with a test of your own rather than from the request arithmetic. The raw log is in the error section below.

Step 3: Load it with syscalls, disk and HTTP

This is where gVisor performance overhead shows up, and where Kata surprised us. The syscall loop is BusyBox dd moving one byte per call, run inside each pod:

kubectl exec b-gvisor -- sh -c 'time dd if=/dev/zero of=/dev/null bs=1 count=1000000'

Five runs per runtime gave these wall times, in order:

RuntimeRun 1Run 2Run 3Run 4Run 5
runc2.29 s2.31 s2.29 s2.41 s2.69 s
gVisor11.60 s12.56 s12.06 s11.83 s19.41 s
Kata0.67 s0.68 s0.69 s0.68 s0.69 s
Full VM3.29 s2.81 s2.92 s3.37 s3.17 s

A five-fold gVisor cost is the expected price of intercepting every call in the Sentry. Kata beating runc by 3.4 times is not a Kata speed trick. The host kernel on this Skylake CPU runs IBRS and page table isolation for Spectre, retbleed and Meltdown, and each syscall pays for them. A bare dd on the host itself took 2.28 to 2.72 s, the same as runc. The Kata guest kernel reports itself as unprotected:

kubectl exec mit-kata -- sh -c 'cd /sys/devices/system/cpu/vulnerabilities && for f in retbleed spectre_v2 mds meltdown spec_store_bypass; do echo "$f:$(cat $f)"; done'

Inside the Kata guest, retbleed, Spectre v2, MDS and speculative store bypass all read Vulnerable, and only Meltdown keeps PTI:

retbleed:Vulnerable
spectre_v2:Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Not affected; BHI: Vulnerable
mds:Vulnerable; SMT Host state unknown
meltdown:Mitigation: PTI
spec_store_bypass:Vulnerable

That is a deliberate trade: the guest holds one pod, so protecting processes inside it from each other buys little, while the VM boundary and the mitigations on the host protect everything else. The full VM is the control: its stock Ubuntu kernel keeps IBRS and PTI switched on inside the guest, and it ran the same loop in 3.17 s, close to runc rather than to Kata. So the speed comes from the missing mitigations, not from virtualization. It also means a multi-process workload inside one Kata pod has no speculative-execution protection between its own processes. We only measured this i7-6700. On newer CPUs where the kernel can use enhanced IBRS instead of IBRS, the host pays less per syscall, so expect runc to close part of the gap there.

Disk results need the mount type next to them, because each runtime reaches the emptyDir differently. runc sees the host ext4 directly. gVisor mounts it as 9p with cache=remote_revalidating and directfs, and Kata as virtiofs:

fio testruncgVisorKataFull VM
4k randrw buffered, read / write IOPS9,933 / 4,26749,190 / 21,0975,197 / 2,2241,987 / 847
4k randrw --direct=1, read / write IOPS3,122 / 1,33550,026 / 21,4604,864 / 2,0801,767 / 758
1 GiB sequential write, fsync at end246.6 MiB/s245.7 MiB/s128.7 MiB/s97.3 MiB/s

Random I/O under gVisor did not change when we asked for direct I/O, which tells you the file cache in the Sentry answered it and O_DIRECT never reached the host disk. Only the sequential write with its closing fsync forces data down, and there gVisor and runc both land at 246 MiB/s, so the gVisor disk path is not the bottleneck for large writes. The Kata virtiofs path halved sequential write throughput. Its random direct I/O beat the runc result, which points to virtiofsd and the host page cache sitting between O_DIRECT in the guest and the disk. The full VM is slowest on disk. Its qcow2 overlay on the host ext4 adds a translation layer, and we opened it with cache=none, so every write in the guest is a direct write on the host. If your workload depends on direct I/O semantics, such as a database that calls O_DIRECT for its buffer pool, test it on the runtime you pick rather than trusting a random-I/O number.

Network is the weakest row for both sandboxes. nginx under runc answered 41,762 requests per second with a 2.70 ms p99. The gVisor netstack, which reimplements TCP/IP in user space, dropped that to 4,700 req/s and 34 ms. Kata reached only 1,966 req/s at 63 ms. The full VM rules out nesting as the whole explanation: on the same nested host, through a plain tap device on a Linux bridge, it served 23,786 req/s at a 9.60 ms p99, twelve times Kata. The difference sits in the Kata pod networking, which the 4.3.1 config sets to internetworking_model = "tcfilter" (tc rules mirroring the pod veth onto a tap device for the guest) with a single network_queues = 1 queue. We did not isolate which part costs the most. The gVisor loss comes from its user-space netstack, which runs on the host kernel and is not affected by nesting.

Step 4: Install runsc and Kata next to containerd

Both sandboxes install as shims that containerd finds on the PATH, so nothing replaces runc. gVisor publishes an apt repository. Add its key and source, then install the runsc package, which brings runsc, containerd-shim-runsc-v1 and the helper binaries under /usr/bin/gvisor-bin/:

curl -fsSL https://gvisor.dev/archive.key | sudo gpg --dearmor -o /usr/share/keyrings/gvisor-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/gvisor-archive-keyring.gpg] https://storage.googleapis.com/gvisor/releases release main" | sudo tee /etc/apt/sources.list.d/gvisor.list > /dev/null
sudo apt-get update && sudo apt-get install -y runsc

Confirm the version and that the binary can start a sandbox on its own, before containerd is involved:

runsc --version
sudo runsc do uname -r

A working install prints the release tag and the fake kernel version that the Sentry reports:

runsc version release-20261026.0
spec: 1.2.1
4.19.0-gvisor

Kata needs /dev/kvm on the node, so check that before downloading anything. Kata 4.x ships as a zstd tarball, kata-static-<version>-amd64.tar.zst, where 3.x guides use a .tar.xz file and older ones an x86_64 name, neither of which exists for this line. The release URL carries the version, so read the latest tag from the Kata Containers releases API (it returned 4.3.1 on our test day):

ls -l /dev/kvm
sudo apt-get install -y zstd
KATA_VERSION=$(curl -s https://api.github.com/repos/kata-containers/kata-containers/releases/latest | grep '"tag_name"' | cut -d '"' -f 4)
curl -fsSLO https://github.com/kata-containers/kata-containers/releases/download/${KATA_VERSION}/kata-static-${KATA_VERSION}-amd64.tar.zst
sudo tar --zstd -xf kata-static-${KATA_VERSION}-amd64.tar.zst -C /

In 4.x the Rust runtime is the default and the Go runtime is deprecated: the main kata-static tarball has no kata-runtime command and no shim in /opt/kata/bin, because the Go pieces ship in a separate kata-go-static tarball. The shim containerd needs is the Rust one under runtime-rs, and it reads /opt/kata/share/defaults/kata-containers/runtime-rs/configuration.toml, which points at the QEMU config. Link it onto the PATH and check it:

sudo ln -sf /opt/kata/runtime-rs/bin/containerd-shim-kata-v2 /usr/local/bin/containerd-shim-kata-v2
containerd-shim-kata-v2 --version

The shim identifies itself as the Rust build with the release and commit:

Kata Containers containerd shim (Rust): id: io.containerd.kata.v2, version: 4.3.1, commit: 03cf2ec06a5d302f23af811744f6bcea49988bfb

A quick smoke test through standalone containerd shows the two kernels side by side. Use the runsc binary path for gVisor with nerdctl, for the reason given in the error section:

sudo nerdctl run --rm --runtime /usr/bin/runsc alpine:3.22 uname -r
sudo nerdctl run --rm --runtime io.containerd.kata.v2 alpine:3.22 uname -r

The first line comes from the Sentry and the second from the Kata guest kernel, while the host still runs 7.0. nerdctl also prints a harmless cannot set cgroup manager to "systemd" warning for the Kata run, trimmed here:

4.19.0-gvisor
6.18.56

If the Kata line fails, check /dev/kvm again before anything else, because Kata cannot boot its QEMU guest without it and no containerd setting replaces it.

Step 5: Register the runtimes in containerd for kubeadm or k3s

Kubernetes picks a runtime by handler name, and the handler is a key under the CRI plugin in containerd’s config. On a kubeadm node, put the two runtimes in a drop-in file rather than editing the main config. containerd 2.4 writes config version 4 and its default config imports /etc/containerd/conf.d/*.toml, but a node with no config.toml at all reads no drop-ins. Check for the imports line first:

grep -n "^imports" /etc/containerd/config.toml

If the file is missing or the grep prints nothing, generate the default config, which carries the imports line. On an existing kubeadm node, add the line to your current file instead so you keep settings such as SystemdCgroup. For a fresh node:

containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo mkdir -p /etc/containerd/conf.d

Create the drop-in:

sudo vim /etc/containerd/conf.d/sandbox-runtimes.toml

Add the two runtimes. The pod_annotations line lets Kata read its per-pod annotations, and privileged_without_host_devices stops a privileged pod from passing every host device into the guest:

version = 4

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.kata]
  runtime_type = "io.containerd.kata.v2"
  privileged_without_host_devices = true
  pod_annotations = ["io.katacontainers.*"]

Restart containerd and read the merged config back:

sudo systemctl restart containerd
sudo containerd config dump | grep -A1 -E "runtimes\.(runsc|kata)\]"

Both handlers should appear with their runtime type. If they are missing, the imports line is missing too:

        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.kata]
          runtime_type = 'io.containerd.kata.v2'
--
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runsc]
          runtime_type = 'io.containerd.runsc.v1'

k3s runs its own containerd and does not auto-detect runsc or Kata, so it needs a template. Create it before installing k3s (on a running node, restart k3s afterwards):

sudo mkdir -p /var/lib/rancher/k3s/agent/etc/containerd
sudo vim /var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.tmpl

The first line pulls in k3s’s generated base config, and the rest is the same pair of runtimes:

{{ template "base" . }}

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.kata]
  runtime_type = "io.containerd.kata.v2"
  privileged_without_host_devices = true
  pod_annotations = ["io.katacontainers.*"]

We installed the same k3s release used for the benchmark. The k3s guide for Ubuntu 26.04 covers the rest of a k3s setup, and the kubeadm guide for Ubuntu 26.04 covers the containerd side of a kubeadm cluster:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.1+k3s1 sh -   # https://github.com/k3s-io/k3s/releases
sudo grep -A3 -E "runtimes.(runsc|kata)\]" /var/lib/rancher/k3s/agent/etc/containerd/config.toml

The rendered config.toml should carry both blocks from the template:

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.kata]
  runtime_type = "io.containerd.kata.v2"
  privileged_without_host_devices = true
  pod_annotations = ["io.katacontainers.*"]

We restarted k3s afterwards with sudo systemctl restart k3s and both blocks survived the regenerated config, so the template is the place to keep them.

Step 6: Create the Kubernetes RuntimeClass for gVisor and Kata

A RuntimeClass maps a name that pods use to a containerd handler. Its overhead field adds the cost of the sandbox itself to the requests of every pod, so the scheduler and the pod cgroup account for it. We set the overhead from the memory measured in Step 2. gVisor used 30.9 MiB per idle pod, so 64Mi leaves room for Sentry growth under load. For Kata we kept the 320Mi and 250m that the kata-deploy chart from the Kata project sets for QEMU. That covers the 180 MiB we measured idle and the growth of the VMM when the guest gets busy. Create the manifest:

vim runtimeclasses.yaml

The handler values must match the runtime names in the containerd config, not the RuntimeClass names:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
overhead:
  podFixed:
    memory: "64Mi"
    cpu: "100m"
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata
handler: kata
overhead:
  podFixed:
    memory: "320Mi"
    cpu: "250m"

Apply it and list the two classes:

kubectl apply -f runtimeclasses.yaml
kubectl get runtimeclass gvisor kata

k3s also lists the runtimes it detects on its own (crun, the WebAssembly shims and NVIDIA), but gvisor and kata only appear because of this file:

NAME     HANDLER   AGE
gvisor   runsc     24s
kata     kata      24s

A pod that names a RuntimeClass whose handler is missing from containerd does not fall back to runc, so a typo here shows up as a pod that never starts rather than as an unsandboxed one.

Step 7: Run a pod in each sandbox and check the overhead

A pod opts in with one field, runtimeClassName. Everything else in the spec stays the same, which is what makes RuntimeClass the clean way to sandbox only the workloads that need it. Create a test pod:

vim sandbox-test.yaml

This one prints the kernel it sees and the first lines of its kernel log, then sleeps:

apiVersion: v1
kind: Pod
metadata:
  name: sandbox-test
spec:
  runtimeClassName: gvisor
  containers:
  - name: shell
    image: docker.io/library/alpine:3.22
    command: ["sh", "-c", "uname -r; dmesg | head -3; sleep 3600"]
    resources:
      requests:
        cpu: 100m
        memory: 64Mi
      limits:
        memory: 256Mi

Apply it, wait for Ready and read the log:

kubectl apply -f sandbox-test.yaml
kubectl wait --for=condition=Ready pod/sandbox-test --timeout=180s
kubectl logs sandbox-test

gVisor answers with its own kernel version and its well-known joke boot messages, which is the quickest proof the pod is not on the host kernel:

4.19.0-gvisor
[   0.000000] Starting gVisor...
[   0.382965] Reading process obituaries...
[   0.880296] DeFUSEing fork bombs...

For Kata, create the same pod with runtimeClassName: kata and the name kata-test. Its log shows 6.18.56 and a real Linux boot line. Then check that admission copied the overhead into the pod spec:

kubectl get pod sandbox-test -o jsonpath='{.spec.overhead}'; echo
kubectl get pod kata-test -o jsonpath='{.spec.overhead}'; echo

Each pod carries the overhead of its class:

{"cpu":"100m","memory":"64Mi"}
{"cpu":"250m","memory":"320Mi"}

The node view shows what the scheduler charges. The gVisor pod asked for 100m and 64Mi and is billed 200m and 128Mi. The Kata pod asked for the same and is billed 350m and 384Mi, and its memory limit grew from 256Mi to 576Mi:

kubectl describe node bench01 | grep -E "kata-test|sandbox-test"

The columns are CPU requests, CPU limits, memory requests and memory limits:

  default                     kata-test                                  350m (8%)     0 (0%)      384Mi (3%)       576Mi (4%)     15s
  default                     sandbox-test                               200m (5%)     0 (0%)      128Mi (1%)       320Mi (2%)     24s

That raised limit matters for Kata. With sandbox_cgroup_only = true, the default in the 4.3.1 QEMU config, QEMU itself lives in the pod cgroup. The kata-deploy chart notes that it raised its QEMU overhead from 160Mi to 320Mi because the lower value let QEMU get OOM-killed on pods with small limits. The Kubernetes pod overhead documentation covers how the scheduler and kubelet apply the field. To keep untrusted namespaces on the sandbox, pair the RuntimeClass with admission policy; our Pod Security Standards guide covers the namespace side.

Errors we hit in the lab

Each of these came up on the test box, with the exact text it printed.

gVisor: “sandbox no longer running and its exit status is unavailable”

Running a gVisor container through standalone containerd with its own shim, ctr run --runtime io.containerd.runsc.v1 or nerdctl run --runtime io.containerd.runsc.v1, printed nothing and exited with 128. With shim debug logging on, the shim log held the error, and the first runsc call the shim made took 5.0 s to return. The application itself ran, and the Sentry boot log recorded application exiting with exit status 0, but its stdout never reached the terminal. It never happened through Kubernetes: the k3s CRI path ran the same shim cleanly for every pod in this benchmark. With nerdctl, run runsc as an OCI binary under the runc shim instead, which is also how Docker uses it:

sudo nerdctl run --rm --runtime /usr/bin/runsc alpine:3.22 uname -r

That form printed 4.19.0-gvisor and exited 0, and it is the path behind the gVisor cold-start numbers above.

nerdctl -d with gVisor: “failed to start io pipe copy … opening binary:///usr/local/bin/nerdctl”

Detached nerdctl containers on the runsc shim failed with failed to create shim task: failed to start io pipe copy: gvisor-containerd-shim: opening binary:///usr/local/bin/nerdctl?_NERDCTL_INTERNAL_LOGGING=...: no such file or directory. nerdctl hands containerd a binary:// log URI for its logging driver, and the gVisor shim tries to open it as a file path. The same --runtime /usr/bin/runsc form avoids it.

Kata 4.x: “kata-runtime: command not found”

After unpacking the 4.3.1 tarball, linking /opt/kata/bin/containerd-shim-kata-v2 and /opt/kata/bin/kata-runtime as older guides show produced bash: line 19: kata-runtime: command not found, because neither file is in the main 4.x tarball (the deprecated Go runtime moved to kata-go-static). The top-level configuration.toml is also a dangling link to a configuration-qemu.toml that is no longer shipped. Use /opt/kata/runtime-rs/bin/containerd-shim-kata-v2 as in Step 4; the Rust shim reads the runtime-rs config directory and needs no kata-runtime binary.

Kata pod Pending: “0/1 nodes are available: 1 Insufficient cpu”

Creating idle Kata pods one at a time, each requesting 10m CPU plus the 250m overhead, the eighth one never scheduled. The pod events read 0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. while the node’s allocated resources showed cpu 2020m (50%). With seven Kata pods still running, runc probe pods requesting 240m and 250m both scheduled on the same node, so the node had room for ordinary pods of that size. We reproduced it twice and did not find the cause, so treat Kata pod density on a small node as something to test, and check kubectl describe pod events whenever a sandboxed pod sits in Pending.

containerd 2.4: “Configuration migrated from version 3”

A config.toml that starts with version = 3, which is what most current guides for containerd 2.x show, still loads on 2.4 but logs Configuration migrated from version 3, use `containerd config migrate` to avoid migration. The CRI plugin keys are unchanged between 3 and 4, so set version = 4 in new files or run containerd config migrate to rewrite an old one.

Pick the runtime by workload, including AI agent sandboxes

The measurements point to one runtime per kind of workload, and in most clusters the right answer is runc by default with a RuntimeClass for the pods that run code you did not write:

  • Trusted services you build and scan: runc. It won every network and startup row, and the kernel attack surface is the price you already accept on every node.
  • Short-lived untrusted code, such as AI agent tool calls, notebook cells or CI jobs from forks: gVisor. A sandbox is Ready in under a second and costs about 31 MiB, so per-task sandboxes stay cheap, and these jobs rarely live on syscall-heavy or network-heavy paths.
  • Untrusted code that is compute-bound, syscall-heavy or needs a real Linux kernel: Kata. It ran the syscall loop fastest of all and runs kernel features gVisor does not implement. Plan for 4 to 5 s starts and about 180 MiB per pod, and keep a warm pool if agents need sandboxes on demand.
  • Long-lived tenants that need their own OS, kernel modules or a separate lifecycle: a full VM. Kubernetes cannot schedule it, and it is the slowest to boot, but nothing else gives a tenant root on its own kernel.

Network-heavy untrusted services are the hard case: gVisor kept a tenth of the runc request rate and Kata less than a twentieth on this host, while a plain VM kept more than half. If an untrusted service must serve real traffic, a VM-per-tenant or a bare-metal Kata test is worth the effort before you commit, because neither sandbox came close on this box. For how these runtimes sit under containerd and CRI-O in the first place, the Docker vs CRI-O vs containerd comparison and the guide to working with containerd in Kubernetes fill in the layer below RuntimeClass.

Keep reading

Best UI Applications for Managing Docker Containers Containers Best UI Applications for Managing Docker Containers Install Docker and Run Containers on Ubuntu 24.04|22.04 Containers Install Docker and Run Containers on Ubuntu 24.04|22.04 Install UniFi OS Server on Ubuntu 24.04 LTS Containers Install UniFi OS Server on Ubuntu 24.04 LTS OpenCloud vs ownCloud vs Nextcloud: Tested on Identical VMs Cloud OpenCloud vs ownCloud vs Nextcloud: Tested on Identical VMs Kiri Container Runtime on Linux: Install and Benchmark vs Docker Containers Kiri Container Runtime on Linux: Install and Benchmark vs Docker Setup Red Hat Quay Registry on Linux using Podman Containers Setup Red Hat Quay Registry on Linux using Podman

Leave a Comment

Press ESC to close