Hardening a shared Linux host against cache covert channels

The previous post explained how agents in separate microVMs passed secrets to each other through the CPU cache, why no kind of VM stops that on its own, and what a layered defence looks like. It stayed at the level of what to do. This post is the how: one Linux host running KVM guests (Firecracker or QEMU), taken from the default configuration to one where agents in different trust domains share as little hardware as the machine allows, with the commands, the files they touch, and how to check that each step did something.

Two things up front.

• This does not get you to zero. The previous post’s conclusion still holds: on shared hardware you lower bandwidth and raise cost, you do not remove the channel. If two agents must never communicate (one holds secrets, the other can reach the network), put them on separate physical hosts and stop reading here. Everything below is for agents that have to share a box.
• Order matters. The steps run from cheapest and most effective to most fiddly. Removing shared cores closes more channels than any cache partitioning does, and cache partitioning is worth little while two tenants still share a core.

Point-in-time (late October 2026). Commands assume a recent mainline kernel, cgroup v2, systemd 261 or later and util-linux 2.41 or later, run as root. Try them on a host you can reboot. Example output is illustrative; your topology will differ.

🗺️ Step 0: map the machine

You cannot keep tenants off shared hardware without knowing what is shared. Three questions: which CPUs are SMT siblings, which CPUs share an L3, and which belong to which NUMA node.

# One line per logical CPU: core, socket, NUMA node, and the cache IDs it uses.
lscpu -e=CPU,CORE,SOCKET,NODE,CACHE

# Which logical CPUs share CPU 0's core, and CPU 0's last-level cache.
cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list
cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list

# Every distinct L3 domain on the host.
cat /sys/devices/system/cpu/cpu*/cache/index3/shared_cpu_list | sort -u

On a two-socket server you might see two L3 domains, 0-15,32-47 and 16-31,48-63, where CPU n and CPU n+32 are siblings. On an AMD EPYC part there are many more, one per core complex, often eight cores each. That second case is good news: small L3 domains mean you can give each trust domain its own cache without buying a whole socket per tenant. hwloc’s lstopo draws the same thing as a picture, which is worth printing and taping to the wall.

Write the result down as a table. Everything else in this post is applying it.

In this example A has an L3 domain to itself apart from the host, while B and C share one. B and C are the pair that need step 5; A mostly does not.

🧵 Step 1: stop sharing cores between domains

Two tenants on sibling threads share nearly everything inside the core. This one step removes the L1, L2, TLB, branch-predictor and port-contention channels, and it is the prerequisite for most of what follows (TLBleed went straight through a CAT-partitioned cache via the TLB).

The blunt option is to turn SMT off:

cat /sys/devices/system/cpu/smt/control     # on, off, forceoff, notsupported, notimplemented (or a thread count)
echo off > /sys/devices/system/cpu/smt/control

To make it permanent, add nosmt to the kernel command line, or use mitigations=auto,nosmt, which turns on every CPU-bug mitigation and disables SMT whenever the CPU needs that to be fully mitigated.

The cheaper option, if you want to keep SMT’s throughput, is core scheduling: tasks carry a cookie, and the kernel only lets tasks with the same cookie run on sibling threads at the same time. util-linux ships a coresched command for it:

# Start a microVM with a fresh cookie: its threads may share a core with
# each other, never with anything else.
coresched new -- firecracker --api-sock /run/fc-a.sock

# Give an already running VMM (and all its threads) a new cookie.
coresched new -t tgid --dest "$VMM_PID"

# Check.
coresched get --source "$VMM_PID"

Each VM gets its own cookie, so a VM’s two vCPUs can sit on one core’s siblings, but two different VMs never do. Three limits:

• The kernel must be built with CONFIG_SCHED_CORE, and SMT must be on: with SMT off, coresched refuses to run (“Core scheduling is not supported on this system”). It is an alternative to turning SMT off, not an addition.
• It is not airtight. The kernel documentation lists a short window, while the sibling is being told to switch, in which untrusted tasks can still run side by side, and it says core scheduling does not stop MDS-style leaks between a sibling in user mode and one running kernel code. If you can afford SMT off, it is the stronger choice.
• It only stops simultaneous sharing. Tenants can still take turns on a core, which step 2 removes and step 4 cleans up after. (Linux core scheduling · coresched(1))

📌 Step 2: pin each domain to its own cores

Without pinning, the scheduler moves vCPU threads wherever there is room, so over a day every tenant runs on every core and leaves state behind for whoever comes next. Pinning fixes each domain to the cores in your table for the life of the VM, as AWS Nitro does for fixed-performance instances.

With cgroup v2 cpusets, give each domain its own branch of the tree and make it a partition root. Its CPUs are then exclusive: no cgroup outside the partition may use them.

On a systemd host, do this with slice units, not by making directories under /sys/fs/cgroup yourself. systemd owns the cgroup tree, and its delegation rules say plainly not to create your own cgroups below the root: “all you do is ask for trouble”. Two details shape the units:

• A dash in a slice name is a level in the tree. tenant-a.slice lives inside tenant.slice, at /sys/fs/cgroup/tenant.slice/tenant-a.slice.
• A partition is only valid if its parent is one. So make tenant.slice a partition holding every tenant CPU, and each tenant a partition inside it.

# /etc/systemd/system/tenant.slice
[Slice]
AllowedCPUs=2-31
CPUSetPartition=root

# /etc/systemd/system/tenant-a.slice
[Slice]
AllowedCPUs=2-7
AllowedMemoryNodes=0
CPUSetPartition=root

Write tenant-b.slice (CPUs 16-23) and tenant-c.slice (24-31) the same way, with AllowedMemoryNodes= set to the NUMA node of their socket. Then load and check them:

systemctl daemon-reload
systemctl start tenant-a.slice tenant-b.slice tenant-c.slice
cat /sys/fs/cgroup/tenant.slice/tenant-a.slice/cpuset.cpus.partition         # "root"; "root invalid (<reason>)" says why not
cat /sys/fs/cgroup/tenant.slice/tenant-a.slice/cpuset.cpus.effective         # 2-7

CPUSetPartition= arrived in systemd 261, and CPUSetPartition=isolated also turns off load balancing inside the partition. On older systemd, AllowedCPUs= alone gives the same placement, but nothing enforces exclusivity, so make sure no two slices list the same CPU. Always read cpuset.cpus.effective afterwards: where the cpuset controller is not available to a unit, as in a user session where it is not delegated, systemd starts the unit anyway and leaves its CPUs unrestricted.

A partition already keeps every other cgroup off its CPUs. To also keep the host’s own work on the housekeeping cores rather than spread over whatever is left, confine the system slices and the default interrupt affinity:

systemctl set-property system.slice  AllowedCPUs=0-1
systemctl set-property user.slice    AllowedCPUs=0-1
systemctl set-property machine.slice AllowedCPUs=0-1   # where libvirt puts VMs by default
# and on the kernel command line: irqaffinity=0-1

set-property persists across reboots; add --runtime while you experiment. irqaffinity does not move “managed” interrupts, such as NVMe queues, which the kernel assigns itself. isolcpus=managed_irq,2-31 asks the kernel to keep those off the tenant CPUs. That is best-effort: it only works for device queues whose interrupt mask also includes a housekeeping CPU.

Start each VM inside its tenant’s slice. For QEMU, a transient scope:

systemd-run --slice=tenant-a.slice --scope -- qemu-system-x86_64 ...

For Firecracker, run the jailer as a transient service in the slice, and point it at the cgroup systemd made for that service:

systemd-run --slice=tenant-a.slice --unit=fc-vm-a -- \
  jailer --id vm-a --exec-file /usr/bin/firecracker --uid 1001 --gid 1001 \
    --cgroup-version 2 --parent-cgroup tenant.slice/tenant-a.slice/fc-vm-a.service \
    -- --api-sock /run/firecracker.socket

With --cgroup-version 2 and no --cgroup values, the jailer creates no cgroup of its own; it only moves the process into the --parent-cgroup path if it exists, which here is where systemd already put it. Do not leave --cgroup-version out: it defaults to 1, cgroup v1, and the jailer then tries to build a v1 hierarchy.

If tenants share a domain’s cores (several VMs inside tenant A), pin each vCPU thread to its own core with taskset -pc so they do not take turns either. Firecracker names its vCPU threads fc_vcpu 0, fc_vcpu 1 and so on, which makes them easy to find under /proc/<pid>/task/*/comm.

On Kubernetes the kubelet does the per-node part. The static CPU manager policy gives Guaranteed pods with whole-number CPU requests exclusive CPUs; full-pcpus-only=true makes it hand out whole physical cores so two pods never share siblings, strict-cpu-reservation=true keeps every pod, not only Guaranteed ones, off the reserved system CPUs, and prefer-align-cpus-by-uncorecache (GA in 1.36) tries to give each container CPUs from a single L3 domain. That last one is best-effort: if it cannot align, the pod is still admitted with ordinary packing. (Kubernetes CPU management policies · Node resource managers)

# KubeletConfiguration
cpuManagerPolicy: static
cpuManagerPolicyOptions:
  full-pcpus-only: "true"
  strict-cpu-reservation: "true"
  prefer-align-cpus-by-uncorecache: "true"
reservedSystemCPUs: "0-1"

Some options cannot be combined, and the kubelet refuses to start if you ask for an incompatible set, so check its log after changing them.

That keeps pods off each other’s cores, not off each other’s L3. Pods from different trust domains still need separate nodes, enforced with taints and anti-affinity as in layer 1 of the previous post.

🔁 Step 3: turn off memory deduplication

KSM is what makes Flush+Reload and Flip Feng Shui work across VMs. Turning it off costs memory and nothing else.

cat /sys/kernel/mm/ksm/run            # 0 stopped, 1 running, 2 stopped and unmerged
echo 2 > /sys/kernel/mm/ksm/run       # stop, and split every page already merged
cat /sys/kernel/mm/ksm/pages_shared   # should fall to 0

echo 0 only stops new merging and leaves existing shared pages shared, so use 2. Make it stick across reboots with a tmpfiles rule, and disable any tuning daemon that would turn it back on:

echo 'w /sys/kernel/mm/ksm/run - - - - 2' > /etc/tmpfiles.d/ksm-off.conf
systemctl disable --now ksmtuned.service 2>/dev/null

KSM only merges memory a process has marked as mergeable. QEMU marks guest memory that way by default, and -machine mem-merge=off stops it. Firecracker’s production host guide still asks you to turn KSM off host-wide, and it is the safer rule: it does not depend on every process on the host behaving.

🧹 Step 4: keep the CPU-bug mitigations on, and flush on switch

This step is about the transient-execution bugs (Spectre, MDS, L1TF and the rest) rather than plain cache sharing, but the two travel together: a covert channel between cooperating agents uses the same receivers, and an unmitigated bug turns a covert channel into a side channel against an agent that is not cooperating.

Start by seeing what the kernel thinks of this CPU:

grep -r . /sys/devices/system/cpu/vulnerabilities/

Anything reading Vulnerable, or SMT vulnerable while SMT is on, needs attention. On recent kernels and Intel CPUs, old_microcode reads Vulnerable when the CPU booted with microcode older than the kernel’s table of released versions. Update it first, through your distribution’s microcode package. The status is set at boot, so a runtime microcode load does not change it; reboot to see the new value. (Linux: old microcode)

Recent kernels group the mitigations by attack vector so you can say what the machine is for instead of listing bugs. The defaults already cover guest-to-guest and guest-to-host; what you add for a multi-tenant agent host is full cross-thread protection:

# kernel command line
mitigations=auto,nosmt

Do not use no_guest_guest or no_cross_thread on a host that runs more than one tenant. (Attack vector controls)

Then the flushes for cores that are shared in time:

• spectre_v2_user=on issues an indirect branch prediction barrier (IBPB) on every switch between different user processes, so the next tenant does not inherit what the predictor learned from the last one, and turns on STIBP to separate sibling threads. The default (auto, which means prctl) only does this for processes that ask. spectre_v2_user=prctl,ibpb is the cheaper middle ground: IBPB on every switch, STIBP only on request.
• kvm-intel.vmentry_l1d_flush=always empties the L1 data cache on every entry into a guest, on Intel CPUs affected by L1TF. The default, cond, skips the flush when it judges nothing sensitive ran in between. (Linux: L1TF)
• l1d_flush=on lets individual processes ask for an L1D flush when they are switched out for a different process, through prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_L1D_FLUSH, ...). Useful for a VMM or broker that handles secrets. It is Intel-only, needs the CPU’s hardware flush (there is no software fallback), and only works on cores with SMT off: a process that opted in and is scheduled on an SMT core gets SIGBUS. (Linux: L1D flushing)

With steps 1 and 2 in place, tenants rarely share a core at all, so these flushes are cheap insurance rather than a hot path.

🧱 Step 5: partition the last-level cache with resctrl

Tenants B and C from the table still share an L3. CAT lets you give each of them its own cache ways, so one can only evict its own lines. Prime+Probe across the boundary stops working, because the sender can no longer push the receiver’s lines out.

Check the CPU supports it, then mount the filesystem:

grep -o -w 'rdt_a\|cat_l3\|cdp_l3\|mba\|cqm_llc\|cqm_mbm_total' /proc/cpuinfo | sort -u
mount -t resctrl resctrl /sys/fs/resctrl

cat /sys/fs/resctrl/info/L3/cbm_mask      # all ways, e.g. 7ff = 11 ways
cat /sys/fs/resctrl/info/L3/num_closids   # how many groups you can have, often 16 or fewer
cat /sys/fs/resctrl/info/L3/min_cbm_bits
cat /sys/fs/resctrl/info/L3/bit_usage     # who uses which ways today
cat /sys/fs/resctrl/info/L3/sparse_masks  # 0 means each mask must be one contiguous run

Some ways are also written by devices, not only by CPUs. On Intel Xeon this is DDIO, where network cards write incoming packets straight into the L3, the mechanism NetCAT abused. DDIO uses two ways by default, and on the 11-way Skylake server measured in Farshin et al. (USENIX ATC 2020) those were the two leftmost ways, 0x600. resctrl does not show DDIO’s ways on Intel, so check your platform’s documentation. On AMD CPUs with Smart Data Cache Injection Allocation Enforcement (SDCIAE), recent kernels add resctrl io_alloc and io_alloc_cbm files that show and set the ways device traffic goes to (enabling it uses up the highest group ID), and bit_usage marks them H (hardware only) or X (hardware and a group). Either way, keep tenants off the device ways and leave them to the host.

Every task starts in the root group, which by default owns every way. Shrink it first. With an 11-way cache (7ff) and the two DDIO ways at 600, one possible split for L3 domain 1 is:

cd /sys/fs/resctrl

# Root group: the host keeps the two DDIO ways plus one more.
echo "L3:0=7ff;1=700" > schemata

mkdir tenant-b tenant-c
echo "L3:0=7ff;1=00f" > tenant-b/schemata    # ways 0-3
echo "L3:0=7ff;1=0f0" > tenant-c/schemata    # ways 4-7
cat */schemata schemata                      # check: no two masks overlap on domain 1

Every group has a mask for every L3 instance, so the 0=7ff entries are there only because each group has to have some mask on domain 0. Neither tenant runs there, so it does not matter. It does stop you using resctrl’s exclusive mode, which makes the kernel refuse overlapping masks but checks every instance at once: to use it, partition domain 0 the same way, so no two groups overlap anywhere. Either way, what keeps B and C apart is that their masks on domain 1 share no bits.

Then put the tenants in their groups. Since step 2 pinned each tenant to fixed cores, the simplest way is by CPU: a task that is still in the root group uses the allocation of the group that owns the CPU it is running on.

echo 16-23 > tenant-b/cpus_list
echo 24-31 > tenant-c/cpus_list

The other way is by task, writing thread IDs to the group’s tasks file (several at once, separated by commas). Each ID moves one thread, not a whole process. New threads start in the group of the thread that created them, so a VMM started inside the group is covered, but one moved in after it started needs every existing thread in /proc/<pid>/task/ moved.

Memory bandwidth is the next shared thing down, and MBA can throttle it in the same schemata file. On Intel the value is a percentage (MB:1=40 caps a group at about 40% on domain 1). On AMD it is an absolute limit in multiples of 1/8 GB/s (MB:1=16 is 2 GB/s). Treat it as lowering the bandwidth of a memory-bus contention channel, not closing it: MBA is approximate throttling, not isolation, and it does nothing about DRAM row-buffer timing (what is still open).

Know the limits before relying on this:

• Not persistent. resctrl starts empty at every boot. Put the setup in a systemd unit that runs before your VMs start.
• Few groups, few ways. Around sixteen groups and eleven to sixteen ways is typical, so CAT separates a handful of trust domains per L3, not hundreds of tenants. That is another reason to use the L3 domain table from step 0 first.
• The L3 only. CAT does nothing for the coherence directory, DRAM banks, the interconnect or frequency. Colouring those (Marghera) is not available in mainline Linux.

🔋 Step 6: take away the free receivers

A covert channel needs a receiver that can measure something. Some measurements are much easier with help from the host, so do not hand it out.

• Energy counters. Since the PLATYPUS fix in late 2020 (CVE-2020-8694), the RAPL energy_uj files are readable by root only. Check that nothing on your host loosens that, and do not pass power or energy MSRs through to guests.

  ls -l /sys/class/powercap/intel-rapl:*/energy_uj   # expect -r-------- root
  

• Performance counters in guests. A virtual PMU gives a guest exact cache-miss counts, a far better receiver than timing loads. Unless guests need profiling, turn it off: host-wide with kvm.enable_pmu=0, or per VM in QEMU with pmu=off (QEMU turns the PMU on by default for -cpu host and -cpu max under KVM, so write -cpu host,pmu=off). Firecracker hides the architectural PMU from Intel guests by zeroing CPUID leaf 0xA. Check from inside each guest, whatever the VMM: perf stat -e cache-misses true should report the event as not supported.
• Performance counters on the host. Keep kernel.perf_event_paranoid at its default of 2. From 1 up, unprivileged processes (a VMM after it drops privileges, an agent’s helper) cannot count system-wide events; 2 also blocks kernel profiling. They can still count their own events, which is enough for a receiver, so some distributions, Debian among them, patch in a stricter level 3 that blocks perf entirely for users without CAP_SYS_ADMIN. Use it where it exists.

Do not bother trapping rdtsc. It is slow, and a counting thread rebuilds a fine-grained clock anyway (Fantastic Timers).

⚖️ Step 7: fixed quotas for what is still shared

The CPU-contention and storage channels in OpenAI’s report did not need a shared cache at all: one agent’s load changed the other’s speed. Hard caps make each tenant’s performance independent of its neighbours.

Add them to the [Slice] section of the tenant’s slice from step 2:

# /etc/systemd/system/tenant-a.slice, [Slice] section, continued
CPUQuota=600%
IOReadBandwidthMax=/dev/nvme0n1 200M
IOWriteBandwidthMax=/dev/nvme0n1 100M
IOReadIOPSMax=/dev/nvme0n1 20000
IOWriteIOPSMax=/dev/nvme0n1 10000

systemd writes these to the cgroup’s cpu.max (here 600000 100000: six CPUs’ worth in every 100 ms period) and io.max. Sizes are in units of 1,000, so 200M is 200,000,000 bytes per second. The systemd documentation spells out the property that matters here: the I/O limits are “not work-conserving”, so a tenant cannot borrow idle capacity even when the disk is free.

With step 2’s exclusive cores, cpu.max mostly matters inside a domain. io.max matters more: the “acoustic modem” channel in the report was bursts of writes to shared storage. Better still, give each tenant its own disk or volume, and its own instances of every shared service (DNS resolver, HTTP proxy, package mirror, TLS cache), as in layer 3.

🔬 Step 8: check that it worked

Configuration that nobody checks drifts. Three kinds of check, from cheapest to most convincing.

Audit the configuration

A short script that prints the state of every step, suitable for a cron job or a node health check:

#!/bin/sh
# Report the covert-channel hardening state of this host.
echo "smt:          $(cat /sys/devices/system/cpu/smt/control)"
echo "ksm run:      $(cat /sys/kernel/mm/ksm/run)  shared pages: $(cat /sys/kernel/mm/ksm/pages_shared)"
echo "cmdline:      $(tr ' ' '\n' < /proc/cmdline | grep -E '^(mitigations|nosmt|spectre_v2_user|l1d_flush|kvm|irqaffinity)' | tr '\n' ' ')"
echo "perf paranoid: $(cat /proc/sys/kernel/perf_event_paranoid)"
grep -i -l -r 'vulnerable' /sys/devices/system/cpu/vulnerabilities/ 2>/dev/null | sed 's/^/VULNERABLE:   /'
for g in /sys/fs/cgroup/tenant.slice/tenant-*.slice; do
  [ -d "$g" ] && echo "$(basename "$g"): cpus=$(cat "$g/cpuset.cpus.effective") partition=$(cat "$g/cpuset.cpus.partition")"
done
if [ -d /sys/fs/resctrl/info ]; then
  echo "L3 bit usage: $(cat /sys/fs/resctrl/info/L3/bit_usage)"
else
  echo "resctrl:      not mounted"
fi

Attack your own host

The only convincing test is a channel that worked before and does not work after. Use published research code between two of your own test VMs, placed the way real tenants would be:

• Mastik, Yuval Yarom’s toolkit, provides the building blocks: Prime+Probe on the L1, L2 and last-level caches, Flush+Reload, Flush+Flush and more, on Intel x86-64. You write a small sender and receiver around them.
• CJAG, from the group behind Hello from the Other Side, is a ready-made cross-VM LLC covert channel: one binary is the sender, -r makes it the receiver, and the two agree on cache sets with no shared memory. It needs huge pages inside each guest (vm.nr_hugepages). Its authors note that auto-detection often fails inside VMs, so pass the host CPU’s cache size, ways, slices and miss threshold by hand (./cjag --help). It targets Intel CPUs whose number of cache slices is a power of two.

Measure the bit error rate and bits per second before you start, then after each step. What you should expect to see:

If a row does not match, the step is not doing what you think: a thread outside the cpuset, a resctrl group missing a CPU, a QEMU guest still marked mergeable. That is the point of running it.

Watch for it in production

The same resctrl filesystem reports per-group cache occupancy and memory traffic on CPUs that support monitoring (cqm_llc and cqm_mbm_total in /proc/cpuinfo), with no performance counters needed. Tasks counted through cpus_list are reported in that group too.

cat /sys/fs/resctrl/tenant-b/mon_data/mon_L3_01/llc_occupancy    # bytes of L3 the group holds now
cat /sys/fs/resctrl/tenant-b/mon_data/mon_L3_01/mbm_total_bytes  # memory traffic so far; sample and diff

Or per slice with perf, which takes the cgroup path below /sys/fs/cgroup:

perf stat -a -G tenant.slice/tenant-b.slice -e LLC-loads,LLC-load-misses -I 10

LLC-loads and LLC-load-misses are generic names that not every CPU supports; perf list shows what yours offers.

A covert channel looks like a regular signal: cache misses or memory traffic switching on and off at a fixed period, or a small working set with a miss rate that ordinary code of that size never has. Sample every few milliseconds and look for strong periodicity (an autocorrelation peak, or a sharp line in a Fourier transform). Treat it as a tripwire, as layer 5 of the previous post did: a patient sender can add jitter and hide in noise, and detectors trained on known channels miss new ones.

🔒 What is closed

“Closed” here means closed between trust domains, once the step is in place and the audit in step 8 says it still is. Each holds only while the steps before it do: cache partitioning means little if two tenants still share a core.

• 🧵 L1 and L2 caches, the TLB, branch predictors and execution ports, by steps 1 and 2. No two domains run on one core at the same time, or one after the other, so none of the core-private state is shared. That covers TLBleed and port contention, and, with step 4’s mitigations on, most of the MDS family.
• 🧹 State left behind on a shared core, by step 4. Where a core is still shared in time, the predictor barrier and L1 flush stop the next tenant reading what the last one left. Current microcode and mitigations=auto,nosmt keep the known transient-execution bugs from turning a covert channel into a side channel against an agent that is not cooperating.
• 🔁 Flush+Reload and Flip Feng Shui across VMs, by step 3. Both need a physical page shared between tenants. With KSM off there is none.
• 🧱 Last-level cache Prime+Probe, by step 5 for domains that share an L3, and completely by step 0’s placement for domains that do not. A sender can only evict lines in its own ways, so it can no longer push the receiver’s lines out. Leaving the device (DDIO) ways to the host also keeps tenant data out of the part of the cache that network traffic, and so NetCAT, reaches.
• 🔋 The free receivers, by step 6. Guests have no performance counters and no energy readings, so PLATYPUS-style power channels and exact miss counts are gone, and a receiver is left with timing loops, which are slower and noisier.
• ⚖️ CPU-time and disk contention between domains, by step 7. With fixed quotas and separate disks, one tenant’s load no longer changes another’s speed, which was the basis of the CPU and “acoustic modem” storage channels in OpenAI’s report.

Together these remove every channel the previous post’s table marks as needing a shared core, a shared page, or shared cache ways. What remains is the package-wide and off-chip part of that table.

🧾 What is still open

After every step above, two tenants on one host still share:

• 🚌 Memory controllers, the bus and DRAM banks. This is the fastest of what remains: DRAMA (USENIX Security 2016) built a covert channel of up to 2 Mbit/s through DRAM row buffers, and it works across CPUs. MBA slows bus contention but not row-buffer timing, and nothing in mainline Linux colours DRAM banks. Separate NUMA nodes per domain, with each tenant’s memory bound to its own node (cpuset.mems), keep tenants off each other’s DIMMs and memory controller, if the host has enough nodes.
• 🌡️ Frequency, power and temperature. These are package-wide. Hertzbleed needs no shared cache at all. The Hertzbleed authors’ workaround is to disable frequency boost (Turbo Boost on Intel, Turbo Core or Precision Boost on AMD), in the BIOS or through the frequency-scaling driver, at a real performance cost.
• 🕸️ The on-die interconnect and coherence directory, unless the tenants are in different L3 domains, and even then the interconnect may be shared.
• 🎮 The GPU, if they share one, MIG or not.
• 🗄️ Every service both can reach: storage, logging, metrics, the model-serving cluster.

None of these needs a shared core or shared cache ways, and every one of them still works for a patient sender with error correction. That is why the steps above are a way of lowering the bandwidth between agents of the same trust level, and why agents of different trust levels belong on different hosts.

✅ The runbook on one page

1. Map siblings, L3 domains and NUMA nodes; write the placement table.
2. SMT off (nosmt), or core scheduling with one cookie per VM.
3. One systemd slice per domain under tenant.slice, each a cpuset partition (CPUSetPartition=root); host slices and interrupts on their own cores; VMs started inside their slice; vCPUs pinned.
4. KSM off (echo 2), persisted; mem-merge=off for QEMU.
5. mitigations=auto,nosmt, current microcode, spectre_v2_user=on, kvm-intel.vmentry_l1d_flush=always where it applies.
6. Domains that share an L3: resctrl groups with non-overlapping masks, device (DDIO) ways left to the host, MBA caps, set up at boot.
7. No vPMU or energy counters for guests; perf_event_paranoid at 2, or 3 where the distribution offers it.
8. Hard CPU and I/O limits on each slice (CPUQuota=, IO*Max=); per-tenant disks and service instances.
9. Audit nightly; re-run Mastik and CJAG after every kernel, microcode or hardware change; watch per-group cache and memory traffic for periodic signals.
10. For agents that must never talk: none of the above. Separate hosts.

📚 Sources

Point-in-time (checked October 2026).

• Image: “Cache timing as covert chat channel” by 113 (@0x113d) on X, quote-posting @scaling01
• Previous posts: When sandboxed agents talk through the CPU cache · Sandboxing untrusted AI agents on Linux
• Kernel: cgroup v2 and cpuset partitions · resctrl · core scheduling · attack vector controls · old microcode · sysctl: perf_event_paranoid · L1TF · L1D flushing · KSM · kernel parameters · powercap
• Tools: coresched(1) · hwloc · systemd resource control · systemd.slice · systemd cgroup delegation · Firecracker jailer · Firecracker production host setup · Kubernetes CPU management policies · Kubernetes node resource managers
• Hardware: Farshin et al., Reexamining Direct Cache Access (USENIX ATC 2020) · DRAMA (USENIX Security 2016)
• Testing: Mastik · CJAG · Hello from the Other Side (NDSS 2017)
• Attacks referred to: PLATYPUS and CVE-2020-8694 · Fantastic Timers (2017) · NetCAT · TLBleed · Hertzbleed