The previous post explained how agents in separate microVMs passed secrets to each other through the CPU cache, why no kind of VM stops that on its own, and what a layered defence looks like. It stayed at the level of what to do. This post is the how: one Linux host running KVM guests (Firecracker or QEMU), taken from the default configuration to one where agents in different trust domains share as little hardware as the machine allows, with the commands, the files they touch, and how to check that each step did something.

📖 Terms in this post

Trust domain. A group of workloads allowed to see each other’s data. Every step here is about keeping different domains apart; agents inside one domain can share freely.
SMT sibling. One of the two hardware threads on a single core (Intel calls SMT Hyper-Threading). Siblings share the L1 and L2 caches, the TLB, branch predictors and execution ports. (Simultaneous multithreading)
L3 domain. The set of CPUs that share one last-level cache: usually a socket on Intel, a core complex (CCX) on AMD EPYC. Two VMs in different L3 domains do not share a cache at all.
cgroup v2 and cpuset. Linux control groups put processes into a tree and apply limits to each branch. The cpuset controller limits which CPUs and memory nodes a branch may use. (Linux cgroup v2)
Slice. A systemd unit that is a branch of that tree. Services and VMs started “in” a slice inherit its limits. (systemd.slice)
resctrl, CAT, MBA, CMT and MBM. resctrl is the Linux filesystem for the processor’s cache and memory-bandwidth controls: Cache Allocation Technology (CAT) reserves cache ways, Memory Bandwidth Allocation (MBA) throttles memory traffic, and Cache Monitoring (CMT) and Memory Bandwidth Monitoring (MBM) report usage per group. AMD exposes the same controls under the same interface. (Linux resctrl)
Capacity bitmask (CBM). A resctrl allocation written as a bitmask, one bit per cache way. 0x0f means “the lowest four ways”.
KSM. Kernel Samepage Merging, which keeps one shared copy of identical memory pages across VMs. (Linux KSM)
PMU and RAPL. The performance monitoring unit counts hardware events such as cache misses; RAPL reports energy use. Both make excellent receivers for a covert channel. (perf_event_open(2) · Linux powercap)
Jailer. Firecracker’s launcher, which drops privileges and places each microVM in its own cgroups, namespaces and chroot before it starts. (Firecracker jailer)

🗺️ Step 0: map the machine

You cannot keep tenants off shared hardware without knowing what is shared. Three questions: which CPUs are SMT siblings, which CPUs share an L3, and which belong to which NUMA node.

# One line per logical CPU: core, socket, NUMA node, and the cache IDs it uses.
lscpu -e=CPU,CORE,SOCKET,NODE,CACHE

# Which logical CPUs share CPU 0's core, and CPU 0's last-level cache.
cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list
cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list

# Every distinct L3 domain on the host.
cat /sys/devices/system/cpu/cpu*/cache/index3/shared_cpu_list | sort -u

On a two-socket server you might see two L3 domains, 0-15,32-47 and 16-31,48-63, where CPU n and CPU n+32 are siblings. On an AMD EPYC part there are many more, one per core complex, often eight cores each. That second case is good news: small L3 domains mean you can give each trust domain its own cache without buying a whole socket per tenant. hwloc’s lstopo draws the same thing as a picture, which is worth printing and taping to the wall.

Domain	L3 domain	Physical cores	Logical CPUs (SMT off)
host housekeeping	0	0–1	0–1
tenant A	0	2–7	2–7
tenant B	1	16–23	16–23
tenant C	1	24–31	24–31

In this example A has an L3 domain to itself apart from the host, while B and C share one. B and C are the pair that need step 5; A mostly does not.

🧵 Step 1: stop sharing cores between domains

Two tenants on sibling threads share nearly everything inside the core. This one step removes the L1, L2, TLB, branch-predictor and port-contention channels, and it is the prerequisite for most of what follows (TLBleed went straight through a CAT-partitioned cache via the TLB).

To make it permanent, add nosmt to the kernel command line, or use mitigations=auto,nosmt, which turns on every CPU-bug mitigation and disables SMT whenever the CPU needs that to be fully mitigated.

The cheaper option, if you want to keep SMT’s throughput, is core scheduling: tasks carry a cookie, and the kernel only lets tasks with the same cookie run on sibling threads at the same time. util-linux ships a coresched command for it:

# Start a microVM with a fresh cookie: its threads may share a core with
# each other, never with anything else.
coresched new -- firecracker --api-sock /run/fc-a.sock

# Give an already running VMM (and all its threads) a new cookie.
coresched new -t tgid --dest "$VMM_PID"

# Check.
coresched get --source "$VMM_PID"

Each VM gets its own cookie, so a VM’s two vCPUs can sit on one core’s siblings, but two different VMs never do. Three limits:

📌 Step 2: pin each domain to its own cores

Without pinning, the scheduler moves vCPU threads wherever there is room, so over a day every tenant runs on every core and leaves state behind for whoever comes next. Pinning fixes each domain to the cores in your table for the life of the VM, as AWS Nitro does for fixed-performance instances.

With cgroup v2 cpusets, give each domain its own branch of the tree and make it a partition root. Its CPUs are then exclusive: no cgroup outside the partition may use them.

On a systemd host, do this with slice units, not by making directories under /sys/fs/cgroup yourself. systemd owns the cgroup tree, and its delegation rules say plainly not to create your own cgroups below the root: “all you do is ask for trouble”. Two details shape the units:

Write tenant-b.slice (CPUs 16-23) and tenant-c.slice (24-31) the same way, with AllowedMemoryNodes= set to the NUMA node of their socket. Then load and check them:

systemctl daemon-reload
systemctl start tenant-a.slice tenant-b.slice tenant-c.slice
cat /sys/fs/cgroup/tenant.slice/tenant-a.slice/cpuset.cpus.partition         # "root"; "root invalid (<reason>)" says why not
cat /sys/fs/cgroup/tenant.slice/tenant-a.slice/cpuset.cpus.effective         # 2-7

CPUSetPartition= arrived in systemd 261, and CPUSetPartition=isolated also turns off load balancing inside the partition. On older systemd, AllowedCPUs= alone gives the same placement, but nothing enforces exclusivity, so make sure no two slices list the same CPU. Always read cpuset.cpus.effective afterwards: where the cpuset controller is not available to a unit, as in a user session where it is not delegated, systemd starts the unit anyway and leaves its CPUs unrestricted.

A partition already keeps every other cgroup off its CPUs. To also keep the host’s own work on the housekeeping cores rather than spread over whatever is left, confine the system slices and the default interrupt affinity:

set-property persists across reboots; add --runtime while you experiment. irqaffinity does not move “managed” interrupts, such as NVMe queues, which the kernel assigns itself. isolcpus=managed_irq,2-31 asks the kernel to keep those off the tenant CPUs. That is best-effort: it only works for device queues whose interrupt mask also includes a housekeeping CPU.

For Firecracker, run the jailer as a transient service in the slice, and point it at the cgroup systemd made for that service:

systemd-run --slice=tenant-a.slice --unit=fc-vm-a -- \
  jailer --id vm-a --exec-file /usr/bin/firecracker --uid 1001 --gid 1001 \
    --cgroup-version 2 --parent-cgroup tenant.slice/tenant-a.slice/fc-vm-a.service \
    -- --api-sock /run/firecracker.socket

With --cgroup-version 2 and no --cgroup values, the jailer creates no cgroup of its own; it only moves the process into the --parent-cgroup path if it exists, which here is where systemd already put it. Do not leave --cgroup-version out: it defaults to 1, cgroup v1, and the jailer then tries to build a v1 hierarchy.

If tenants share a domain’s cores (several VMs inside tenant A), pin each vCPU thread to its own core with taskset -pc so they do not take turns either. Firecracker names its vCPU threads fc_vcpu 0, fc_vcpu 1 and so on, which makes them easy to find under /proc/<pid>/task/*/comm.

On Kubernetes the kubelet does the per-node part. The static CPU manager policy gives Guaranteed pods with whole-number CPU requests exclusive CPUs; full-pcpus-only=true makes it hand out whole physical cores so two pods never share siblings, strict-cpu-reservation=true keeps every pod, not only Guaranteed ones, off the reserved system CPUs, and prefer-align-cpus-by-uncorecache (GA in 1.36) tries to give each container CPUs from a single L3 domain. That last one is best-effort: if it cannot align, the pod is still admitted with ordinary packing. (Kubernetes CPU management policies · Node resource managers)

Some options cannot be combined, and the kubelet refuses to start if you ask for an incompatible set, so check its log after changing them.

That keeps pods off each other’s cores, not off each other’s L3. Pods from different trust domains still need separate nodes, enforced with taints and anti-affinity as in layer 1 of the previous post.

🔁 Step 3: turn off memory deduplication

KSM is what makes Flush+Reload and Flip Feng Shui work across VMs. Turning it off costs memory and nothing else.

echo 0 only stops new merging and leaves existing shared pages shared, so use 2. Make it stick across reboots with a tmpfiles rule, and disable any tuning daemon that would turn it back on:

KSM only merges memory a process has marked as mergeable. QEMU marks guest memory that way by default, and -machine mem-merge=off stops it. Firecracker’s production host guide still asks you to turn KSM off host-wide, and it is the safer rule: it does not depend on every process on the host behaving.

🧹 Step 4: keep the CPU-bug mitigations on, and flush on switch

This step is about the transient-execution bugs (Spectre, MDS, L1TF and the rest) rather than plain cache sharing, but the two travel together: a covert channel between cooperating agents uses the same receivers, and an unmitigated bug turns a covert channel into a side channel against an agent that is not cooperating.

Anything reading Vulnerable, or SMT vulnerable while SMT is on, needs attention. On recent kernels and Intel CPUs, old_microcode reads Vulnerable when the CPU booted with microcode older than the kernel’s table of released versions. Update it first, through your distribution’s microcode package. The status is set at boot, so a runtime microcode load does not change it; reboot to see the new value. (Linux: old microcode)

Recent kernels group the mitigations by attack vector so you can say what the machine is for instead of listing bugs. The defaults already cover guest-to-guest and guest-to-host; what you add for a multi-tenant agent host is full cross-thread protection:

Do not use no_guest_guest or no_cross_thread on a host that runs more than one tenant. (Attack vector controls)

With steps 1 and 2 in place, tenants rarely share a core at all, so these flushes are cheap insurance rather than a hot path.

🧱 Step 5: partition the last-level cache with resctrl

Tenants B and C from the table still share an L3. CAT lets you give each of them its own cache ways, so one can only evict its own lines. Prime+Probe across the boundary stops working, because the sender can no longer push the receiver’s lines out.

grep -o -w 'rdt_a\|cat_l3\|cdp_l3\|mba\|cqm_llc\|cqm_mbm_total' /proc/cpuinfo | sort -u
mount -t resctrl resctrl /sys/fs/resctrl

cat /sys/fs/resctrl/info/L3/cbm_mask      # all ways, e.g. 7ff = 11 ways
cat /sys/fs/resctrl/info/L3/num_closids   # how many groups you can have, often 16 or fewer
cat /sys/fs/resctrl/info/L3/min_cbm_bits
cat /sys/fs/resctrl/info/L3/bit_usage     # who uses which ways today
cat /sys/fs/resctrl/info/L3/sparse_masks  # 0 means each mask must be one contiguous run

Some ways are also written by devices, not only by CPUs. On Intel Xeon this is DDIO, where network cards write incoming packets straight into the L3, the mechanism NetCAT abused. DDIO uses two ways by default, and on the 11-way Skylake server measured in Farshin et al. (USENIX ATC 2020) those were the two leftmost ways, 0x600. resctrl does not show DDIO’s ways on Intel, so check your platform’s documentation. On AMD CPUs with Smart Data Cache Injection Allocation Enforcement (SDCIAE), recent kernels add resctrl io_alloc and io_alloc_cbm files that show and set the ways device traffic goes to (enabling it uses up the highest group ID), and bit_usage marks them H (hardware only) or X (hardware and a group). Either way, keep tenants off the device ways and leave them to the host.

Every task starts in the root group, which by default owns every way. Shrink it first. With an 11-way cache (7ff) and the two DDIO ways at 600, one possible split for L3 domain 1 is:

cd /sys/fs/resctrl

# Root group: the host keeps the two DDIO ways plus one more.
echo "L3:0=7ff;1=700" > schemata

mkdir tenant-b tenant-c
echo "L3:0=7ff;1=00f" > tenant-b/schemata    # ways 0-3
echo "L3:0=7ff;1=0f0" > tenant-c/schemata    # ways 4-7
cat */schemata schemata                      # check: no two masks overlap on domain 1

Every group has a mask for every L3 instance, so the 0=7ff entries are there only because each group has to have some mask on domain 0. Neither tenant runs there, so it does not matter. It does stop you using resctrl’s exclusive mode, which makes the kernel refuse overlapping masks but checks every instance at once: to use it, partition domain 0 the same way, so no two groups overlap anywhere. Either way, what keeps B and C apart is that their masks on domain 1 share no bits.

Then put the tenants in their groups. Since step 2 pinned each tenant to fixed cores, the simplest way is by CPU: a task that is still in the root group uses the allocation of the group that owns the CPU it is running on.

The other way is by task, writing thread IDs to the group’s tasks file (several at once, separated by commas). Each ID moves one thread, not a whole process. New threads start in the group of the thread that created them, so a VMM started inside the group is covered, but one moved in after it started needs every existing thread in /proc/<pid>/task/ moved.

Memory bandwidth is the next shared thing down, and MBA can throttle it in the same schemata file. On Intel the value is a percentage (MB:1=40 caps a group at about 40% on domain 1). On AMD it is an absolute limit in multiples of 1/8 GB/s (MB:1=16 is 2 GB/s). Treat it as lowering the bandwidth of a memory-bus contention channel, not closing it: MBA is approximate throttling, not isolation, and it does nothing about DRAM row-buffer timing (what is still open).

🔋 Step 6: take away the free receivers

A covert channel needs a receiver that can measure something. Some measurements are much easier with help from the host, so do not hand it out.

Do not bother trapping rdtsc. It is slow, and a counting thread rebuilds a fine-grained clock anyway (Fantastic Timers).

⚖️ Step 7: fixed quotas for what is still shared

The CPU-contention and storage channels in OpenAI’s report did not need a shared cache at all: one agent’s load changed the other’s speed. Hard caps make each tenant’s performance independent of its neighbours.

systemd writes these to the cgroup’s cpu.max (here 600000 100000: six CPUs’ worth in every 100 ms period) and io.max. Sizes are in units of 1,000, so 200M is 200,000,000 bytes per second. The systemd documentation spells out the property that matters here: the I/O limits are “not work-conserving”, so a tenant cannot borrow idle capacity even when the disk is free.

With step 2’s exclusive cores, cpu.max mostly matters inside a domain. io.max matters more: the “acoustic modem” channel in the report was bursts of writes to shared storage. Better still, give each tenant its own disk or volume, and its own instances of every shared service (DNS resolver, HTTP proxy, package mirror, TLS cache), as in layer 3.

🔬 Step 8: check that it worked

Configuration that nobody checks drifts. Three kinds of check, from cheapest to most convincing.

Audit the configuration

A short script that prints the state of every step, suitable for a cron job or a node health check:

#!/bin/sh
# Report the covert-channel hardening state of this host.
echo "smt:          $(cat /sys/devices/system/cpu/smt/control)"
echo "ksm run:      $(cat /sys/kernel/mm/ksm/run)  shared pages: $(cat /sys/kernel/mm/ksm/pages_shared)"
echo "cmdline:      $(tr ' ' '\n' < /proc/cmdline | grep -E '^(mitigations|nosmt|spectre_v2_user|l1d_flush|kvm|irqaffinity)' | tr '\n' ' ')"
echo "perf paranoid: $(cat /proc/sys/kernel/perf_event_paranoid)"
grep -i -l -r 'vulnerable' /sys/devices/system/cpu/vulnerabilities/ 2>/dev/null | sed 's/^/VULNERABLE:   /'
for g in /sys/fs/cgroup/tenant.slice/tenant-*.slice; do
  [ -d "$g" ] && echo "$(basename "$g"): cpus=$(cat "$g/cpuset.cpus.effective") partition=$(cat "$g/cpuset.cpus.partition")"
done
if [ -d /sys/fs/resctrl/info ]; then
  echo "L3 bit usage: $(cat /sys/fs/resctrl/info/L3/bit_usage)"
else
  echo "resctrl:      not mounted"
fi

Attack your own host

The only convincing test is a channel that worked before and does not work after. Use published research code between two of your own test VMs, placed the way real tenants would be:

Measure the bit error rate and bits per second before you start, then after each step. What you should expect to see:

If a row does not match, the step is not doing what you think: a thread outside the cpuset, a resctrl group missing a CPU, a QEMU guest still marked mergeable. That is the point of running it.

Watch for it in production

After	L1 / SMT channel	LLC Prime+Probe	Flush+Reload	DRAM and memory bus
default host	works	works	works if KSM merged a page	works
+ steps 1–2 (no shared cores, pinned)	gone	works	works if KSM merged a page	works
+ step 3 (KSM off)	gone	works	gone	works
+ step 5 (CAT, separate ways)	gone	largely blocked	gone	works; MBA slows bus contention only
tenants in separate L3 domains	gone	gone	gone	works

The same resctrl filesystem reports per-group cache occupancy and memory traffic on CPUs that support monitoring (cqm_llc and cqm_mbm_total in /proc/cpuinfo), with no performance counters needed. Tasks counted through cpus_list are reported in that group too.

LLC-loads and LLC-load-misses are generic names that not every CPU supports; perf list shows what yours offers.

A covert channel looks like a regular signal: cache misses or memory traffic switching on and off at a fixed period, or a small working set with a miss rate that ordinary code of that size never has. Sample every few milliseconds and look for strong periodicity (an autocorrelation peak, or a sharp line in a Fourier transform). Treat it as a tripwire, as layer 5 of the previous post did: a patient sender can add jitter and hide in noise, and detectors trained on known channels miss new ones.

🔒 What is closed

“Closed” here means closed between trust domains, once the step is in place and the audit in step 8 says it still is. Each holds only while the steps before it do: cache partitioning means little if two tenants still share a core.

Together these remove every channel the previous post’s table marks as needing a shared core, a shared page, or shared cache ways. What remains is the package-wide and off-chip part of that table.

🧾 What is still open

None of these needs a shared core or shared cache ways, and every one of them still works for a patient sender with error correction. That is why the steps above are a way of lowering the bandwidth between agents of the same trust level, and why agents of different trust levels belong on different hosts.