Containers

CKS Practice Questions: Hands-On Kubernetes Security Tasks

Half of what the CKS checks lives where kubectl cannot see it. A profile loaded into the node kernel, a syscall filter on disk under the kubelet, an alert in a systemd journal and a line in an audit log are all on that list. That is why CKS practice questions that only show a YAML answer leave you short. You also need to know which machine to SSH into, which file to edit there, and what the failure looks like when you get it wrong.

Original content from computingforgeeks.com - post 187745

This set has seven tasks, one for each tool the security specialist outline leans on hardest: NetworkPolicy, Pod Security Admission, AppArmor, seccomp, Trivy image scanning, Falco runtime alerts and API server audit logging. Each task reads the way a CKS task does, so you can try it yourself first. Our solution follows with the exact manifest, the real output and the failed first attempts. I ran every task on Kubernetes 1.37 in November 2026 on a fresh two node kubeadm cluster, and the errors quoted here are the ones it gave us.

What a CKS task expects from you

The Certified Kubernetes Security Specialist is a performance test, the same format as the CKA. You get a terminal, several clusters, and a list of tasks, and a script grades the state you leave behind. The Linux Foundation CKS instructions page sets the ground rules, and these are the ones that shape how you should practise:

  • 2 hours for 15 to 20 tasks
  • Pass mark of 67 percent, one point higher than the CKA
  • Kubernetes v1.37 in the environment today
  • A passed CKA required before you can book it
  • Docs allowed: kubernetes.io docs and blog, Falco, etcd, Cilium, Istio, bom and ingress-nginx docs

Read that last line carefully, because the Trivy and AppArmor docs are not on the allowed list. The Falco docs are, so you can look up a field name for an output format. For Trivy, plan on trivy image --help unless a task links a page in its own quick reference box, which the allowed resources page also permits. That is why the Trivy step below uses only flags that the help text shows. The outline itself splits into six domains, and the table maps each one to the step in this guide that drills it.

DomainWeightDrilled here
Cluster Setup15%Step 1 (NetworkPolicy)
Cluster Hardening15%Step 7 (audit logs touch API access)
System Hardening10%Steps 3 and 4 (AppArmor, seccomp)
Minimize Microservice Vulnerabilities20%Step 2 (Pod Security Admission)
Supply Chain Security20%Step 5 (Trivy)
Monitoring, Logging and Runtime Security20%Steps 6 and 7 (Falco, audit logs)

Every CKS task also starts on a node you have to reach first, usually with ssh to a hostname the task names. Half of the tasks below finish on the worker node, not on the machine where kubectl runs, so the habit to build is checking the prompt before you type. If you have not done the CKA labs yet, our CKA practice questions cover the cluster administration side that the CKS assumes you already know, and our CKAD practice questions cover the application side.

The lab cluster behind these answers

We built two Ubuntu 24.04 VMs in Google Cloud, both e2-standard-2 with 8 GB of RAM, and joined them with kubeadm. The control plane is cks-cp and the worker is cks-node1. Ubuntu was a deliberate pick, because it ships AppArmor enabled and a recent kernel, which the AppArmor task and Falco’s modern eBPF probe both need. The pieces we used:

  • Kubernetes 1.37.2 from pkgs.k8s.io, held with apt-mark
  • containerd 2.2.1 with the systemd cgroup driver
  • Calico 3.33.0, pod CIDR 192.168.0.0/16
  • Falco 0.45.2 and Trivy 0.76.2 for steps 5 and 6

Our diagram shows where each control sits. The API server admits pods and writes the audit log on the control plane, while AppArmor, seccomp, Falco and Calico all do their work on the worker, next to the containers they police.

CKS practice cluster diagram: kube-apiserver with Pod Security Admission writing audit.log on cks-cp, kubelet on cks-node1 feeding AppArmor, Calico and Falco

Our first snag came before any task. With no NetworkPolicy in place at all, a pod still could not resolve DNS, and the message was the same one a deny policy produces, which made it very confusing:

;; connection timed out; no servers could be reached

command terminated with exit code 1

Blame the cloud firewall here, not Kubernetes. The default Calico manifest wraps pod traffic between nodes in IPIP, which is IP protocol 4, and the Google Cloud default-allow-internal rule only passes TCP, UDP and ICMP. Pods on the same node talked fine, pods across nodes did not. One firewall rule fixed it:

gcloud compute firewall-rules create cks-lab-ipip --network=default --allow=4 --source-ranges=10.128.0.0/9

Prove that DNS works on a clean cluster before you write a single policy, otherwise you will blame your YAML for a fault in the network underneath. The same trap exists on AWS security groups and on any homelab firewall that filters by protocol. One more cloud detail: Google Cloud VMs report a long internal FQDN as their hostname, and kubeadm registered our nodes under it on the first try. We reset and ran kubeadm init and kubeadm join again with --node-name, so that nodeName: cks-node1 in the later manifests matches a real node.

Step 1: Default deny a namespace, then allow one client

You are asked to block all ingress and egress traffic in namespace payments by default. Pods in payments must still resolve DNS, and only pods labelled role=gateway in namespace edge may reach pods labelled app=api, on TCP port 80. Our setup had an api nginx pod with a Service, a client pod in payments, a gateway pod in edge and an unlabelled intruder pod in a third namespace called other. To build the same playground, save this as setup.yaml:

vim setup.yaml

It holds the three namespaces, four pods and the Service in one file:

apiVersion: v1
kind: Namespace
metadata:
  name: payments
---
apiVersion: v1
kind: Namespace
metadata:
  name: edge
---
apiVersion: v1
kind: Namespace
metadata:
  name: other
---
apiVersion: v1
kind: Pod
metadata:
  name: api
  namespace: payments
  labels:
    app: api
spec:
  containers:
  - name: nginx
    image: nginx:1.29-alpine
    ports:
    - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: payments
spec:
  selector:
    app: api
  ports:
  - port: 80
    targetPort: 80
---
apiVersion: v1
kind: Pod
metadata:
  name: gateway
  namespace: edge
  labels:
    role: gateway
spec:
  containers:
  - name: busybox
    image: busybox:1.37
    command: ["sleep", "3600"]
---
apiVersion: v1
kind: Pod
metadata:
  name: intruder
  namespace: other
spec:
  containers:
  - name: busybox
    image: busybox:1.37
    command: ["sleep", "3600"]
---
apiVersion: v1
kind: Pod
metadata:
  name: client
  namespace: payments
  labels:
    app: client
spec:
  containers:
  - name: busybox
    image: busybox:1.37
    command: ["sleep", "3600"]

Apply it with kubectl apply -f setup.yaml and wait until all four pods are Running before you add any policy.

A default deny is one policy with an empty pod selector and both policy types listed, so create that file first.

vim default-deny.yaml

Add the policy. Listing Egress under policyTypes with no egress rules is what blocks outbound traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Apply it, then test name resolution from the client pod:

kubectl apply -f default-deny.yaml
kubectl -n payments exec client -- nslookup kubernetes.default.svc.cluster.local

DNS dies with the rest of the traffic, because CoreDNS sits in kube-system and egress to it is now blocked too:

;; connection timed out; no servers could be reached

command terminated with exit code 1

People skip this part of the task when they read too fast. The fix is a second egress policy for every pod in the namespace that allows port 53 on both UDP and TCP to the kube-dns pods. Our file was allow-dns.yaml:

vim allow-dns.yaml

Both selectors sit in the same list item, so a packet must match the kube-system namespace and the kube-dns label together.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

After applying it the same nslookup returned Address: 10.96.0.1 for the kubernetes Service. One small trap from our run: busybox nslookup ignores the search list in resolv.conf, so the short name kubernetes.default returns NXDOMAIN even when DNS is fine. Test with the full name ending in svc.cluster.local. Now the ingress rule, in allow-gateway.yaml:

vim allow-gateway.yaml

Again the two selectors share one from item. Splitting them into two items (two dashes) would turn the rule into an OR and let every pod in edge through.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-gateway-to-api
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: edge
      podSelector:
        matchLabels:
          role: gateway
    ports:
    - protocol: TCP
      port: 80

Apply it and test from all three clients. A grader checks the deny paths as well as the allow path, so you should too:

kubectl apply -f allow-gateway.yaml
kubectl -n edge exec gateway -- wget -qO- -T 3 http://api.payments.svc.cluster.local
kubectl -n other exec intruder -- wget -qO- -T 3 http://api.payments.svc.cluster.local
kubectl -n payments exec client -- wget -qO- -T 3 http://api.payments.svc.cluster.local

Only the gateway gets the nginx welcome page, while the intruder and the client both time out.

NetworkPolicy test: gateway pod in edge gets the nginx page while the intruder and payments client time out

Why is the client blocked when it sits in the same namespace as the API pod? Because the default deny covers it too, and the allow rule names only the edge gateway. A timeout rather than a connection refused is the sign that the policy dropped the packet. kubectl describe networkpolicy prints the rule in plain words (“Allowing ingress traffic … From NamespaceSelector … PodSelector”) and is the fastest way to check you did not split the selectors by accident.

Step 2: Enforce the restricted Pod Security Standard

In this task, namespace team-a must reject any pod that does not meet the restricted Pod Security Standard, and users must see a warning as well. Pod Security Admission is built into the API server, so the whole job is namespace labels. The labels follow one pattern, pod-security.kubernetes.io/<mode>, where the mode is enforce, audit or warn, as the Pod Security Admission page documents.

Before labelling a namespace that already runs workloads, do a server side dry run. To have something for the dry run to report, we created the namespace with kubectl create namespace team-a and left a privileged nginx pod called legacy in it on purpose:

vim legacy.yaml

Nothing restricts the namespace yet, so this pod is admitted and starts normally after kubectl apply -f legacy.yaml.

apiVersion: v1
kind: Pod
metadata:
  name: legacy
  namespace: team-a
spec:
  containers:
  - name: nginx
    image: nginx:1.29-alpine
    securityContext:
      privileged: true

Now run the label as a server side dry run.

kubectl label --dry-run=server --overwrite ns team-a pod-security.kubernetes.io/enforce=restricted

Every existing pod that would violate the level is listed with its reasons, and nothing changes yet.

Warning: existing pods in namespace "team-a" violate the new PodSecurity enforce level "restricted:latest"
Warning: legacy: privileged, allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile
namespace/team-a labeled (server dry run)

Apply the real labels next. We set the enforce version to latest and added the warn mode the task asked for:

kubectl label --overwrite ns team-a \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest \
  pod-security.kubernetes.io/warn=restricted

Then test it the way the grader will, with a plain pod that sets nothing:

kubectl -n team-a run bad --image=nginx:1.29-alpine

Every field you have to set is named in the rejection, which makes it the best checklist you have for writing the compliant pod.

Pod Security Admission restricted labels on team-a and the Forbidden error listing the securityContext fields a pod must set

One thing the capture also shows: the legacy pod kept running after the label went on. Enforcement only applies to pods created from then on, so if a task says “make sure no pod in the namespace violates the policy”, you also have to delete or fix the old pods yourself. The compliant manifest below uses the unprivileged nginx image, which listens on 8080 and runs as UID 101:

vim good-pod.yaml

Put runAsNonRoot and the seccomp profile at pod level, and the capability drop and privilege escalation flag on the container:

apiVersion: v1
kind: Pod
metadata:
  name: good
  namespace: team-a
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 101
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: nginx
    image: nginxinc/nginx-unprivileged:1.29-alpine
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]

Our good pod was admitted and reached Running in 5 seconds. We also tried the same manifest with the stock nginx:1.29-alpine image. It was admitted, then went into CrashLoopBackOff, because its startup needs root to create cache directories, and the log said so plainly: mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied). Pick an image built to run as non-root before you fight the manifest.

Step 3: Confine a pod with an AppArmor profile

You are told that a profile called k8s-deny-write exists on node cks-node1, and you must load it, run a pod on that node that uses it, and prove the container cannot write files. AppArmor profiles live in the node kernel, so the first half of this task happens over SSH on the worker. The profile file we used is the deny write example from the Kubernetes AppArmor tutorial, saved under the task’s name:

sudo vim /etc/apparmor.d/k8s-deny-write

Whatever follows the word profile is the name your pod refers to, and it does not have to match the file name.

#include <tunables/global>

profile k8s-deny-write flags=(attach_disconnected) {
  #include <abstractions/base>

  file,

  # Deny all file writes.
  deny /** w,
}

Here we hit the failure first, on purpose, because it is the one people meet in practice. We created the pod with the profile name misspelled as k8s-deny-writes, and the pod sat in CreateContainerError. Its events gave the reason clearly:

Warning  Failed  6s (x3 over 8s)  kubelet  spec.containers{busybox}: Error: failed to get container spec opts: failed to generate apparmor spec opts: apparmor profile not found k8s-deny-writes

You get the same message when the name is right but the profile was never loaded, which is the more common case. On the worker, load it and confirm the kernel knows it:

sudo apparmor_parser -q /etc/apparmor.d/k8s-deny-write
sudo aa-status | grep k8s

Back on the control plane, the pod sets the profile through securityContext.appArmorProfile, the field that went stable in Kubernetes 1.31 and replaced the old annotation. Pin the pod to the node that has the profile:

vim aa-pod.yaml

We used nodeName here because the profile exists on one node only. A nodeSelector on a label works just as well if you prefer it:

apiVersion: v1
kind: Pod
metadata:
  name: aa-deny-write
spec:
  nodeName: cks-node1
  securityContext:
    appArmorProfile:
      type: Localhost
      localhostProfile: k8s-deny-write
  containers:
  - name: busybox
    image: busybox:1.37
    command: ["sleep", "3600"]

Apply it, then try a write and read the profile that the first process in the container runs under:

kubectl apply -f aa-pod.yaml
kubectl exec aa-deny-write -- touch /tmp/test
kubectl exec aa-deny-write -- cat /proc/1/attr/current

Writing fails with Permission denied, and the process reports the profile in enforce mode.

AppArmor k8s-deny-write profile in enforce mode blocking touch and shell writes inside the pod with Permission denied

Both lines matter. Permission denied on its own could come from file ownership, while k8s-deny-write (enforce) proves the denial came from AppArmor. A profile loaded with apparmor_parser does not survive a reboot unless the file stays in /etc/apparmor.d, so leave it there. Our AppArmor cheat sheet covers complain mode and aa-logprof for building profiles of your own.

Step 4: Block a syscall with a seccomp profile

Your next task is a seccomp profile on cks-node1 that blocks the mkdir syscall, plus a pod that runs with it. Seccomp profiles are JSON files that the kubelet reads from /var/lib/kubelet/seccomp, and the path you give in the pod is relative to that directory. On the worker:

sudo mkdir -p /var/lib/kubelet/seccomp/profiles
sudo vim /var/lib/kubelet/seccomp/profiles/no-mkdir.json

Everything is allowed by default, and the two syscalls that create directories return an error. List both names, since a C library can implement the mkdir command with either mkdir or mkdirat.

{
  "defaultAction": "SCMP_ACT_ALLOW",
  "syscalls": [
    {
      "names": ["mkdir", "mkdirat"],
      "action": "SCMP_ACT_ERRNO"
    }
  ]
}

In your pod, the profile is profiles/no-mkdir.json, relative to the seccomp directory.

vim seccomp-pod.yaml

Paste the pod spec. Putting the profile at pod level applies it to every container in the pod:

apiVersion: v1
kind: Pod
metadata:
  name: sec-no-mkdir
spec:
  nodeName: cks-node1
  securityContext:
    seccompProfile:
      type: Localhost
      localhostProfile: profiles/no-mkdir.json
  containers:
  - name: busybox
    image: busybox:1.37
    command: ["sleep", "3600"]

Get the path wrong and the pod lands in CreateContainerError, the same status as the AppArmor mistake. Our test pod pointed at profiles/not-there.json, and the event spelled out the full path the kubelet tried, which tells you at once whether the file is missing or the relative path is wrong:

Error: failed to create containerd container: cannot load seccomp profile "/var/lib/kubelet/seccomp/profiles/not-there.json": open /var/lib/kubelet/seccomp/profiles/not-there.json: no such file or directory

With the correct path, test the block, test that other writes still work, and read the seccomp mode from the process status:

kubectl apply -f seccomp-pod.yaml
kubectl exec sec-no-mkdir -- mkdir /tmp/x
kubectl exec sec-no-mkdir -- touch /tmp/y
kubectl exec sec-no-mkdir -- grep Seccomp /proc/1/status

We ran the same checks against a pod with RuntimeDefault and one with no profile at all, so the three modes sit side by side:

seccomp Localhost profile blocking mkdir with Operation not permitted, and Seccomp mode 2 versus 0 for RuntimeDefault and unconfined pods

Seccomp mode 2 means a filter is attached, and you can see it on both the custom profile and RuntimeDefault. The unconfined pod shows 0, which is what every pod gets on a default kubeadm cluster unless you set a profile, which the restricted Pod Security level from step 2 forces you to do. (A kubelet started with seccompDefault: true applies RuntimeDefault to every pod instead.) The error text is Operation not permitted, not Permission denied, and that difference tells you seccomp blocked the call rather than AppArmor or file permissions.

Step 5: Find and remove images with critical CVEs

Namespace scan runs three Deployments, and you must scan every image they use with Trivy, then scale any Deployment whose image has a CRITICAL vulnerability down to zero. If your control plane lacks Trivy, add the Aqua Security apt repository and install it together with jq, which the scan loop below uses:

curl -fsSL https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo gpg --dearmor --yes -o /usr/share/keyrings/trivy.gpg && echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" | sudo tee /etc/apt/sources.list.d/trivy.list
sudo apt-get update
sudo apt-get install -y trivy jq

Ours installed Trivy 0.76.2. Our three Deployments came from one file, scan.yaml:

vim scan.yaml

One old image, one current image and a busybox tool pod give the scan three different results to sort:

apiVersion: v1
kind: Namespace
metadata:
  name: scan
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: old-web
  namespace: scan
spec:
  replicas: 1
  selector:
    matchLabels: {app: old-web}
  template:
    metadata:
      labels: {app: old-web}
    spec:
      containers:
      - name: nginx
        image: nginx:1.19
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: new-web
  namespace: scan
spec:
  replicas: 1
  selector:
    matchLabels: {app: new-web}
  template:
    metadata:
      labels: {app: new-web}
    spec:
      containers:
      - name: nginx
        image: nginx:1.29-alpine
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tools
  namespace: scan
spec:
  replicas: 1
  selector:
    matchLabels: {app: tools}
  template:
    metadata:
      labels: {app: tools}
    spec:
      containers:
      - name: busybox
        image: busybox:1.36
        command: ["sleep", "3600"]

After kubectl apply -f scan.yaml, list the images per Deployment first, because a pod list shows the same image once per replica:

kubectl -n scan get deploy -o jsonpath="{range .items[*]}{.metadata.name}{\"\t\"}{.spec.template.spec.containers[*].image}{\"\n\"}{end}"

Three Deployments, three images, one of them clearly old:

new-web	nginx:1.29-alpine
old-web	nginx:1.19
tools	busybox:1.36

A full table for nginx:1.19 runs to hundreds of lines, so filter by severity and count. Our first scan took 53 seconds because Trivy downloads its vulnerability database on first use, and later scans finished in a few seconds. Budget for that first download when the clock is running. The loop below also needs jq, which we installed alongside Trivy, so run which jq first on any host you did not build yourself. This loop scans each image, prints the CRITICAL count, and scales the offending Deployment in one pass:

kubectl -n scan get deploy -o jsonpath="{range .items[*]}{.metadata.name}{\"\t\"}{.spec.template.spec.containers[*].image}{\"\n\"}{end}" | while read d i; do cnt=$(trivy image -q --severity CRITICAL -f json $i | jq "[.Results[]?.Vulnerabilities // [] | length] | add // 0"); echo "$d $i $cnt"; [ "$cnt" -gt 0 ] && kubectl -n scan scale deploy/$d --replicas=0; done

Two Deployments went to zero, and the counts explain why:

Trivy CRITICAL counts per image, 42 for nginx 1.19 and 1 for nginx 1.29-alpine, with both Deployments scaled to zero

nginx:1.19 carried 42 CRITICAL findings on its Debian 10 base, which nobody would argue with. The surprise was nginx:1.29-alpine with one: a brotli library fix that was out but not yet in that image build. A task that says “critical” means any critical, so it goes. Then read the busybox line closely. Its count is 0, but the full Trivy report for busybox:1.36 shows a dash under Vulnerabilities, which the legend defines as “Not scanned”. Trivy found no package database in that image, so the zero means it could not look, not that the image is clean. Our older Trivy scanning guide covers filesystem and repository scans too.

Step 6: Catch a shell in a container with Falco

Falco already runs on cks-node1 in this task. You need to change the “Terminal shell in container” rule so its alert prints in the format time,user,process,container id, trigger it, and find the pod behind the alert. We installed Falco 0.45.2 from the falcosecurity apt repository with FALCO_DRIVER_CHOICE=modern_ebpf, which needs no kernel headers, and the package enabled falco-modern-bpf.service by itself. The Falco package docs list the other unit names for the kmod driver. To build the same setup on your worker, these were our install commands:

curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | sudo gpg --dearmor --yes -o /usr/share/keyrings/falco-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] https://download.falco.org/packages/deb stable main" | sudo tee /etc/apt/sources.list.d/falcosecurity.list
sudo apt-get update
sudo FALCO_FRONTEND=noninteractive FALCO_DRIVER_CHOICE=modern_ebpf apt-get install -y falco

The package starts the service as soon as it installs, so the rule work below can begin right away.

Never edit /etc/falco/falco_rules.yaml, because a package upgrade overwrites it. Overrides go in falco_rules.local.yaml, which Falco loads after the stock rules:

sudo vim /etc/falco/falco_rules.local.yaml

An override block tells Falco which keys of the existing rule to replace. We also raised the priority to WARNING so the alert stands out in the journal:

- rule: Terminal shell in container
  output: "%evt.time,%user.name,%proc.name,%container.id,%k8s.ns.name/%k8s.pod.name"
  priority: WARNING
  override:
    output: replace
    priority: replace

We got there on the third try, and both failures took the Falco service down into a restart loop. The first version had no override block, so Falco read it as a brand new rule with the same name and refused it because a new rule needs a condition. The second had a typo, %user.nam, and failed to compile. Validate before you restart, every time:

sudo falco -V /etc/falco/falco_rules.yaml -V /etc/falco/falco_rules.local.yaml
sudo systemctl restart falco-modern-bpf.service

For reference, this is how the missing override looked in the journal, with the restart counter climbing beneath it:

LOAD_ERR_YAML_VALIDATE (Error validating internal structure of YAML file): Item has no mapping for key 'condition'

With valid rules loaded, you need a target pod on the worker. Ours was a plain nginx pod called shelly, pinned to the node where Falco runs:

vim shelly.yaml

Falco only sees syscalls on its own node, so nodeName matters here:

apiVersion: v1
kind: Pod
metadata:
  name: shelly
  namespace: default
spec:
  nodeName: cks-node1
  containers:
  - name: nginx
    image: nginx:1.29-alpine

Apply it, then open an interactive shell from the control plane with kubectl exec -it shelly -- sh and exit again. The rule only fires on a shell with a terminal attached, so a shell without a terminal, such as our earlier kubectl exec shelly -- sh -c "id; hostname", did not trigger it. On the worker, read the alert, then map the container ID it prints to a pod. Use the ID from your own alert, since ours below will not exist on your cluster:

sudo journalctl -u falco-modern-bpf.service --since "-1min" --no-pager -o cat | grep -E "Warning"
sudo crictl ps --id 5aeae8d32363

Our four comma separated fields lead the alert, followed by the namespace and pod, then the container fields Falco adds on its own.

Falco rules validated, custom Terminal shell in container alert with time, user, process and container ID, and crictl mapping it to pod shelly

Falco 0.45 appends its own container_id=... k8s_pod_name=... fields after the custom output, so your fields sit right after Falco’s own timestamp and priority prefix, ahead of everything Falco adds. We also added the namespace and pod to the format for our own convenience; if a task names its fields exactly, write exactly those and nothing more. Our crictl was missing on the worker, because the 1.37 kubeadm package no longer pulls in cri-tools, so we installed it with apt. Cluster nodes you are handed normally have it, but a homelab built from the kubeadm packages alone may not. For a wider look at what an eBPF sensor records next to auditd, our test of what a Linux EDR agent sees runs Falco on the same kind of host.

Step 7: Turn on API server audit logging

Your last task is to enable audit logging on the cluster. Log Secret access at Metadata level so the secret values never reach the log, log every request on Pods in namespace payments at RequestResponse level, and nothing else. Keep 2 backups, 7 days, 50 MB per file. The API server is a static pod, so this is an edit to its manifest on the control plane, and a wrong edit takes the API down. Copy the manifest outside the manifests directory first:

sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.bak
sudo mkdir -p /etc/kubernetes/audit /var/log/kubernetes/audit
sudo vim /etc/kubernetes/audit/policy.yaml

Rules match top to bottom and the first match wins, so the specific rules go first and the catch all None goes last. Note that resources: ["pods"] does not cover subresources such as pods/exec or pods/log; a task that asks you to audit exec calls needs those listed too.

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- RequestReceived
rules:
- level: RequestResponse
  namespaces: ["payments"]
  resources:
  - group: ""
    resources: ["pods"]
- level: Metadata
  resources:
  - group: ""
    resources: ["secrets"]
- level: None

Now edit the API server manifest. It needs four things: the flags, a volume mount for the policy, a volume mount for the log directory, and the matching hostPath volumes:

sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml

Our first edit added only the flags, and it is worth seeing what that does. The API server container exited and kubectl answered The connection to the server 10.132.0.8:6443 was refused. So how do you read an error when kubectl itself is gone? Run sudo crictl ps -a --name kube-apiserver to find the exited container, then crictl logs on its ID, which printed this (install cri-tools first if your control plane lacks crictl, as our worker did in step 6).

E1111 11:43:23.759469       1 run.go:72] "command failed" err="loading audit policy file: failed to read file path \"/etc/kubernetes/audit/policy.yaml\": open /etc/kubernetes/audit/policy.yaml: no such file or directory"

Our policy file existed on the host, but the container could not see it without a mount. Compare your edit with the backup using sudo diff -u /root/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml, which printed the complete working change.

@@ -12,6 +12,11 @@
   containers:
   - command:
     - kube-apiserver
+    - --audit-policy-file=/etc/kubernetes/audit/policy.yaml
+    - --audit-log-path=/var/log/kubernetes/audit/audit.log
+    - --audit-log-maxage=7
+    - --audit-log-maxbackup=2
+    - --audit-log-maxsize=50
     - --advertise-address=10.132.0.8
     - --allow-privileged=true
     - --authorization-mode=Node,RBAC
@@ -79,6 +84,11 @@
       periodSeconds: 10
       timeoutSeconds: 15
     volumeMounts:
+    - mountPath: /etc/kubernetes/audit/policy.yaml
+      name: audit-policy
+      readOnly: true
+    - mountPath: /var/log/kubernetes/audit
+      name: audit-log
     - mountPath: /etc/ssl/certs
       name: ca-certs
       readOnly: true
@@ -102,6 +112,14 @@
       type: RuntimeDefault
   volumes:
   - hostPath:
+      path: /etc/kubernetes/audit/policy.yaml
+      type: File
+    name: audit-policy
+  - hostPath:
+      path: /var/log/kubernetes/audit
+      type: DirectoryOrCreate
+    name: audit-log
+  - hostPath:
       path: /etc/ssl/certs
       type: DirectoryOrCreate
     name: ca-certs

Once saved, the manifest was picked up by the kubelet and the API server answered again in under a minute. To check the policy, create a secret in payments, read it back, run a pod there and delete the secret, so the log has events of each kind:

kubectl -n payments create secret generic db-pass --from-literal=password=S3cr3t-lab
kubectl -n payments get secret db-pass -o jsonpath="{.data.password}" | base64 -d; echo
kubectl -n payments run audit-demo --image=busybox:1.37 --restart=Never -- sleep 60
kubectl -n payments delete secret db-pass

Now query the log with jq. This filter keeps only secret events and prints who did what at which level:

sudo jq -c 'select(.objectRef.resource=="secrets") | {verb, user: .user.username, ns: .objectRef.namespace, name: .objectRef.name, level}' /var/log/kubernetes/audit/audit.log | tail

Each secret action is there at Metadata level, and the pod create in payments is the only RequestResponse event:

Kubernetes audit log queried with jq: secret events at Metadata level, pod create at RequestResponse and zero matches for the secret value

The capture ends with two counts, the name of the secret and its value, taken straight from the log file:

sudo grep -c db-pass /var/log/kubernetes/audit/audit.log; sudo grep -c S3cr3t /var/log/kubernetes/audit/audit.log

Of all these checks, that grep matters most for a real cluster. The secret name appears 4 times in the log, and the secret value appears 0 times, because Metadata level records who touched what without the request or response body. Write the same policy with Request on secrets and the base64 value lands in the log and in every rotated backup, readable by anyone with root on the control plane. The whole log held 6 lines and 8.7 KB after our test, which shows how much the closing None rule cuts out. RBAC decides who may read those secrets in the first place, and our guide to Kubernetes RBAC roles and ServiceAccounts covers that half.

A hardening checklist to run after the drills

All seven tasks map onto checks you can run against any cluster you look after, not only a practice one. After our run we turned them into a short audit list:

  1. Every app namespace has a default deny NetworkPolicy plus a DNS egress rule
  2. Namespaces carry an enforce label, restricted where the workload allows it
  3. No running pod shows Seccomp mode 0 in its process status
  4. Custom AppArmor profiles sit in /etc/apparmor.d so they load at boot
  5. Image scans count “Not scanned” results as failures, not passes
  6. Falco rule overrides pass falco -V before any restart
  7. The audit policy logs secrets at Metadata and nothing at Request or above

Items 3 and 5 are the ones that fail quietly. A pod without a seccomp profile and an image Trivy could not read both look fine until you check, so they are worth scripting. Admission policy goes further than Pod Security Admission can, and our comparison of OPA Gatekeeper and Kyverno shows the two engines that cover registry allow lists and image rules. For isolation beyond seccomp and AppArmor, the gVisor vs Kata Containers benchmark measures what a sandboxed RuntimeClass costs. The same gVisor RuntimeClass is the base layer for sandboxing untrusted AI agent workloads with gVisor, where each agent runs in its own locked-down pod. Once these seven tasks come without the page open, run them again on a fresh cluster with a timer, aiming for 6 to 8 minutes each, which is the pace 15 to 20 tasks in 2 hours demands.

Keep reading

Best UI Applications for Managing Docker Containers Containers Best UI Applications for Managing Docker Containers Install Docker and Run Containers on Ubuntu 24.04|22.04 Containers Install Docker and Run Containers on Ubuntu 24.04|22.04 Install UniFi OS Server on Ubuntu 24.04 LTS Containers Install UniFi OS Server on Ubuntu 24.04 LTS OpenCloud vs ownCloud vs Nextcloud: Tested on Identical VMs Cloud OpenCloud vs ownCloud vs Nextcloud: Tested on Identical VMs Kiri Container Runtime on Linux: Install and Benchmark vs Docker Containers Kiri Container Runtime on Linux: Install and Benchmark vs Docker How to reset ArgoCD admin user password Containers How to reset ArgoCD admin user password

Leave a Comment

Press ESC to close