Lab 20: Local GPU Passthrough with HAMi

AdvancedDuration: about 120 minutesEnvironment: Linux host / QEMU-KVM-libvirt / Ubuntu VM / NVIDIA GPU passthroughCost: local machine onlyVerified: 2026-10-14By: @Utkarsh56016

This lab builds the tested local HAMi GPU passthrough environment from a Linux workstation, a QEMU/KVM/libvirt Ubuntu VM, and one NVIDIA GPU assigned to that VM. The Kubernetes, container runtime, Calico, Helm, and HAMi layers all live inside the guest. The host keeps its desktop on another display path and gets the NVIDIA device back when the VM shuts down.

The lab is phase-gated. Validate each layer before adding the next one: host feasibility, CPU-only VM, qcow2 rollback, managed PCI passthrough, guest NVIDIA driver, container runtime GPU access, kubeadm, Calico, plain CUDA, HAMi, fractional workload, and host recovery.

note

This is a reproducible local lab path for one tested workstation and GPU. It is not a universal VFIO/libvirt troubleshooting guide for every laptop, workstation, firmware, or IOMMU topology.

Purpose and Tested Environment

Use this lab when you want a local, resettable place to test HAMi fractional GPU allocation against a real NVIDIA GPU without installing Kubernetes or HAMi on the host operating system.

Component	Tested value
Host VM stack	QEMU/KVM, system libvirt, OVMF/UEFI
VM/domain name	hami-lab-v2
Guest OS	Ubuntu 24.04.5 LTS
Guest kernel	6.8.0-142-generic
GPU	NVIDIA GeForce RTX 3050 Laptop GPU
GPU memory	4096 MiB
Guest NVIDIA driver	595.91.07
containerd	2.2.1
NVIDIA Toolkit	1.20.1
Kubernetes	v1.36.5
Calico	v3.32.2
Measured V2 Pod CIDR	192.168.0.0/16
Helm	v3.22.0
HAMi chart/app	2.10.0
RuntimeClass	nvidia
GPU node label	gpu=on

The VM/GPU boundary matters because it keeps host state recoverable. The host runs libvirt and owns the base qcow2 files. The Ubuntu guest owns the NVIDIA driver, container runtime, Kubernetes, Calico, and HAMi stack. If a guest-layer experiment goes wrong, the active qcow2 overlay can be discarded without rewriting the preserved base disk.

Set Local Variables

Set these values in the shell where a step uses them. Host-only variables drive libvirt and disk commands; GUEST_IP and POD_CIDR are also used inside the VM during kubeadm and Calico setup. The commands below use variables so they are easier to copy, while the output blocks show the values from the tested environment.

export DOMAIN=hami-lab-v2
export GUEST_USER=ubuntu # replace with your VM user
export GUEST_IP=192.168.122.242 # replace with your VM IP
export POD_CIDR=10.244.0.0/16
export ISO="$HOME/Downloads/ubuntu-24.04.5-live-server-amd64.iso"
export BASE=/var/lib/libvirt/images/hami-lab-v2-base.qcow2
export WORK=/var/lib/libvirt/images/hami-lab-v2-work.qcow2

The full GPU and HAMi run used the V2 cluster's 192.168.0.0/16 Pod CIDR. That range contains the default libvirt network 192.168.122.0/24. During the additional process-path tests, Calico treated the libvirt resolver as in-pool traffic and did not apply normal egress masquerading, so CoreDNS timed out while forwarding external DNS queries to 192.168.122.1:53.

The runnable configuration above uses the non-overlapping 10.244.0.0/16 range. It was validated separately with Kubernetes v1.36.5, Calico v3.32.2, the unchanged CoreDNS forwarder, internal and external DNS, and verified HTTPS egress from a Pod. That check did not repeat the GPU or HAMi workload tests. The GPU and HAMi output blocks below remain evidence from the original 192.168.0.0/16 run.

Host Feasibility Checks

Start with read-only host checks. Do not detach the GPU yet.

Run on the host:

lscpu | awk -F: '/Model name|Virtualization/ {gsub(/^[[:space:]]+/, "", $2); print $1 ": " $2}'
test -e /dev/kvm && echo "/dev/kvm present"
virsh -c qemu:///system dominfo "$DOMAIN" >/dev/null 2>&1 || \
  echo "$DOMAIN domain name is free"

Output from the tested host:

Model name: AMD Ryzen 5 5600H with Radeon Graphics
Virtualization: AMD-V
/dev/kvm present
hami-lab-v2 domain name is free

Check the VM tooling and install media:

qemu-system-x86_64 --version
qemu-img --version
virt-install --version
ls /usr/share/edk2/x64/OVMF_CODE.4m.fd
test -f "$ISO" && echo "Ubuntu ISO is present"

Output from the tested host:

QEMU emulator version 11.1.1
qemu-img version 11.1.1
virt-install version 5.1.0
/usr/share/edk2/x64/OVMF_CODE.4m.fd
Ubuntu ISO is present

Identify the GPU, its audio function, and every device in the GPU's IOMMU group:

GPU_GROUP="$(basename "$(readlink /sys/bus/pci/devices/0000:01:00.0/iommu_group)")"
printf 'GPU IOMMU group: %s\n' "$GPU_GROUP"
for dev in /sys/kernel/iommu_groups/"$GPU_GROUP"/devices/*; do
  pci="${dev##*/}"
  printf '%s ' "$pci"
  lspci -nn -s "${pci#0000:}"
  lspci -nnk -s "${pci#0000:}" | awk '/Kernel driver in use/ {print "  " $0}'
done

Output from the tested host:

GPU IOMMU group: 11
0000:01:00.0 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA107M [GeForce RTX 3050 Mobile] [10de:25a2]
    Kernel driver in use: nvidia
0000:01:00.1 01:00.1 Audio device [0403]: NVIDIA Corporation GA107 High Definition Audio Controller [10de:2291]
    Kernel driver in use: snd_hda_intel

Verify that the host display does not depend on the NVIDIA GPU:

echo "$XDG_SESSION_TYPE"
echo "$XDG_CURRENT_DESKTOP"
for node in /sys/class/drm/card*-*; do
  printf '%s ' "$(basename "$node")"
  cat "$node/status"
done
glxinfo -B | grep -E 'OpenGL vendor|OpenGL renderer'

Output from the tested host:

wayland
niri
card0-HDMI-A-1 disconnected
card2-DP-1 disconnected
card2-eDP-1 connected
OpenGL vendor: AMD
OpenGL renderer: AMD Radeon Graphics

Gate: every member of the IOMMU group moves together. Continue only if every listed device can safely be detached from the host and the host has a separate working display path.

Fresh VM Creation and qcow2 Rollback Boundary

Create the VM without the GPU first. This proves the guest can boot, accept SSH, and shut down before passthrough adds complexity.

Create a fresh base disk:

sudo qemu-img create -f qcow2 \
  "$BASE" \
  50G

Output:

Formatting '/var/lib/libvirt/images/hami-lab-v2-base.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=53687091200 lazy_refcounts=off refcount_bits=16

Create the CPU-only Ubuntu VM:

sudo virt-install \
  --connect qemu:///system \
  --name "$DOMAIN" \
  --memory 8192 \
  --vcpus 4 \
  --cpu host-passthrough \
  --machine q35 \
  --boot uefi \
  --disk path="$BASE",format=qcow2,bus=virtio \
  --cdrom "$ISO" \
  --os-variant ubuntu24.04 \
  --network network=default,model=virtio \
  --graphics spice \
  --video virtio \
  --console pty,target_type=serial \
  --autoconsole graphical

Complete the Ubuntu Server 24.04.5 LTS installer in the graphical console that opens. With --graphics spice, this is a SPICE-backed console. Install OpenSSH during setup, then validate the guest:

ssh "${GUEST_USER}@${GUEST_IP}" \
  'hostnamectl --static; lsb_release -ds; uname -r; systemctl is-active ssh'

Output:

hami-lab-v2
Ubuntu 24.04.5 LTS
6.8.0-142-generic
active

Shut the VM down before changing its disk chain:

ssh "${GUEST_USER}@${GUEST_IP}" 'sudo poweroff'
for attempt in $(seq 1 60); do
  DOMAIN_STATE="$(virsh -c qemu:///system domstate "$DOMAIN")"
  [ "$DOMAIN_STATE" = "shut off" ] && break
  sleep 2
done
[ "$DOMAIN_STATE" = "shut off" ] || {
  echo "Refusing disk changes: $DOMAIN did not shut off within 120 seconds" >&2
  exit 1
}
printf '%s\n' "$DOMAIN_STATE"

Output:

shut off

Create a disposable work overlay from the installed base disk:

sudo qemu-img create -f qcow2 -F qcow2 -b "$BASE" "$WORK"
qemu-img info --backing-chain "$WORK"

Output:

Formatting '/var/lib/libvirt/images/hami-lab-v2-work.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=53687091200 backing_file=/var/lib/libvirt/images/hami-lab-v2-base.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
backing file: /var/lib/libvirt/images/hami-lab-v2-base.qcow2
backing file format: qcow2
corrupt: false

Switch the shut-off domain from the base disk to the overlay:

DOMAIN_STATE="$(virsh -c qemu:///system domstate "$DOMAIN")"
printf '%s\n' "$DOMAIN_STATE"
[ "$DOMAIN_STATE" = "shut off" ] || {
  echo "Refusing disk changes: $DOMAIN is $DOMAIN_STATE" >&2
  exit 1
}
virsh -c qemu:///system dumpxml "$DOMAIN" > hami-lab-v2-before-overlay.xml
virsh -c qemu:///system detach-disk "$DOMAIN" vda --config
virsh -c qemu:///system attach-disk "$DOMAIN" "$WORK" vda \
  --config \
  --type disk \
  --driver qemu \
  --subdriver qcow2 \
  --targetbus virtio
virsh -c qemu:///system domblklist "$DOMAIN"

Output:

shut off
Device detached successfully
Device attached successfully
Target   Source
vda      /var/lib/libvirt/images/hami-lab-v2-work.qcow2
sda      -

Boot from the active overlay:

virsh -c qemu:///system start "$DOMAIN"
virsh -c qemu:///system domstate "$DOMAIN"
virsh -c qemu:///system domblklist "$DOMAIN"

Output:

Domain 'hami-lab-v2' started
running
vda      /var/lib/libvirt/images/hami-lab-v2-work.qcow2
sda      -

Create a marker inside the overlay-backed guest:

ssh "${GUEST_USER}@${GUEST_IP}" \
  'cat > ~/v2-rollback-marker.txt <<EOF
V2 rollback marker
created_utc=2026-09-30T11:35:18Z
hostname=hami-lab-v2
disk_expected=hami-lab-v2-work.qcow2
EOF
cat ~/v2-rollback-marker.txt'

Output:

V2 rollback marker
created_utc=2026-09-30T11:35:18Z
hostname=hami-lab-v2
disk_expected=hami-lab-v2-work.qcow2

Shut the VM down and wait until libvirt reports shut off before touching the overlay:

ssh "${GUEST_USER}@${GUEST_IP}" 'sudo poweroff'
for attempt in $(seq 1 60); do
  DOMAIN_STATE="$(virsh -c qemu:///system domstate "$DOMAIN")"
  [ "$DOMAIN_STATE" = "shut off" ] && break
  sleep 2
done
[ "$DOMAIN_STATE" = "shut off" ] || {
  echo "Refusing overlay reset: $DOMAIN did not shut off within 120 seconds" >&2
  exit 1
}
printf '%s\n' "$DOMAIN_STATE"

Output:

shut off

Move the old overlay aside and recreate a clean one at the same active path:

sudo mv "$WORK" /var/lib/libvirt/images/hami-lab-v2-work-with-marker.qcow2
sudo qemu-img create -f qcow2 -F qcow2 -b "$BASE" "$WORK"
qemu-img info --backing-chain "$WORK"
virsh -c qemu:///system domblklist "$DOMAIN"

Output:

Formatting '/var/lib/libvirt/images/hami-lab-v2-work.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=53687091200 backing_file=/var/lib/libvirt/images/hami-lab-v2-base.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
backing file: /var/lib/libvirt/images/hami-lab-v2-base.qcow2
vda      /var/lib/libvirt/images/hami-lab-v2-work.qcow2

Boot the clean overlay and prove the marker is gone:

virsh -c qemu:///system start "$DOMAIN"
ssh "${GUEST_USER}@${GUEST_IP}" \
  'test ! -e ~/v2-rollback-marker.txt && echo "PASS: marker is absent after overlay reset"'

Output:

Domain 'hami-lab-v2' started
PASS: marker is absent after overlay reset

Gate: continue only after rollback works. This boundary is what makes later Kubernetes and HAMi changes easy to discard.

Managed RTX VGA/HDA Passthrough Using virsh

The tested passthrough pair is the RTX VGA function 0000:01:00.0 and its HDA audio function 0000:01:00.1. They stay in the host drivers while the VM is shut off. With managed='yes', libvirt moves them to vfio-pci when the VM starts and returns them to host drivers when the VM shuts down.

Stop the observed host holders before starting the passthrough VM. Fan-control services are machine-specific and are not part of this generic sequence. If your host requires you to stop one manually, record its unit name and restore and verify it before declaring host recovery complete.

systemctl --user stop pipewire-pulse.service pipewire-pulse.socket pipewire.service pipewire.socket wireplumber.service
sudo systemctl stop nvidia-persistenced

for unit in pipewire.socket pipewire.service pipewire-pulse.socket pipewire-pulse.service wireplumber.service; do
  printf '%s ' "$unit"
  systemctl --user is-active "$unit" || true
done
printf 'nvidia-persistenced '
systemctl is-active nvidia-persistenced || true

Observed output from the tested host:

pipewire.socket inactive
pipewire.service inactive
pipewire-pulse.socket inactive
pipewire-pulse.service inactive
wireplumber.service inactive
nvidia-persistenced inactive

On the tested laptop, the host-specific nbfc_service fan-control unit was also stopped and observed inactive. This is a historical host observation, not a required step for other systems.

Create the GPU and audio hostdev XML files, then attach them to the shut-off VM. The same XML is also kept in rtx-gpu.xml and rtx-audio.xml for reference.

cat >rtx-gpu.xml <<'EOF'
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
  </source>
</hostdev>
EOF

cat >rtx-audio.xml <<'EOF'
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
  </source>
</hostdev>
EOF

virsh -c qemu:///system domstate "$DOMAIN"
virsh -c qemu:///system attach-device "$DOMAIN" ./rtx-gpu.xml --config
virsh -c qemu:///system attach-device "$DOMAIN" ./rtx-audio.xml --config

Output:

shut off
Device attached successfully
Device attached successfully

Start the VM and inspect host ownership:

virsh -c qemu:///system start "$DOMAIN"
lspci -nnk -s 01:00.0 | grep 'Kernel driver in use'
lspci -nnk -s 01:00.1 | grep 'Kernel driver in use'
nvidia-smi

Output:

Domain 'hami-lab-v2' started
Kernel driver in use: vfio-pci
Kernel driver in use: vfio-pci
Failed to initialize NVML: No supported GPUs were found

Inside the guest, confirm the PCI devices are visible:

ssh "${GUEST_USER}@${GUEST_IP}" \
  "lspci -nnk | grep -A3 -E '07:00.0|08:00.0'"

Output:

07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA107M [GeForce RTX 3050 Mobile] [10de:25a2]
  Kernel driver in use: nouveau
08:00.0 Audio device [0403]: NVIDIA Corporation Device [10de:2291]
  Kernel driver in use: snd_hda_intel

Gate: the guest must see both NVIDIA PCI functions before installing the guest NVIDIA driver.

Guest NVIDIA Driver Validation

Inside the guest, inspect the recommended driver and install it:

ubuntu-drivers devices | grep recommended
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y nvidia-driver-595-open
sudo reboot

Output:

recommended: nvidia-driver-595-open
nvidia-driver-595-open 595.91.07-0ubuntu0.24.04.1 installed

After reboot, validate the guest driver:

nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv,noheader

Output:

NVIDIA GeForce RTX 3050 Laptop GPU, 595.91.07, 4096 MiB

Check the guest PCI binding:

lspci -nnk -s 07:00.0 | grep 'Kernel driver in use'
lsmod | grep -E '^nvidia|^nouveau' || true

Output:

Kernel driver in use: nvidia
nvidia_uvm
nvidia_drm
nvidia_modeset
nvidia

Gate: nvidia-smi must work in the guest before container runtime work begins.

containerd and NVIDIA Container Toolkit Validation

Install containerd, runc, and the NVIDIA Container Toolkit inside the guest. The tested package versions were containerd 2.2.1-0ubuntu1~24.04.3, runc 1.3.4-0ubuntu1~24.04.1, and NVIDIA Container Toolkit 1.20.1-1.

Install the packages before configuring the runtime:

sudo apt-get update
sudo apt-get install -y ca-certificates curl gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey |
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list |
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y \
  containerd=2.2.1-0ubuntu1~24.04.3 \
  runc=1.3.4-0ubuntu1~24.04.1 \
  nvidia-container-toolkit=1.20.1-1

Configure the NVIDIA runtime, enable systemd cgroups for both runc and the nvidia runtime handler, then restart containerd:

sudo nvidia-ctk runtime configure --runtime=containerd
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' \
  /etc/containerd/config.toml \
  /etc/containerd/conf.d/99-nvidia.toml
sudo systemctl restart containerd

Validate the runtime versions and cgroup settings:

containerd --version
runc --version
nvidia-ctk --version
systemctl is-active containerd
grep -R 'SystemdCgroup = true' /etc/containerd/config.toml /etc/containerd/conf.d/99-nvidia.toml

Output:

containerd github.com/containerd/containerd/v2 2.2.1
runc version 1.3.4-0ubuntu1~24.04.1
NVIDIA Container Toolkit CLI version 1.20.1
active
/etc/containerd/config.toml:            SystemdCgroup = true
/etc/containerd/conf.d/99-nvidia.toml:            SystemdCgroup = true

Confirm the NVIDIA runtime drop-in:

grep -R 'BinaryName.*nvidia-container-runtime' /etc/containerd

Output:

BinaryName = "/usr/bin/nvidia-container-runtime"

Run a plain container GPU smoke test before Kubernetes:

sudo ctr run --rm --gpus 0 \
  docker.io/nvidia/cuda:12.4.1-base-ubuntu22.04 \
  hami-v2-nvidia-smi \
  nvidia-smi --query-gpu=name,memory.total --format=csv,noheader

Output:

NVIDIA GeForce RTX 3050 Laptop GPU, 4096 MiB

Gate: prove the guest container runtime can reach the GPU before kubeadm. If this layer fails, HAMi will not be able to fix it later.

Manual kubeadm Bootstrap

Prepare the node for Kubernetes:

sudo swapoff -a
sudo sed -i.bak '/ swap /s/^/# kubeadm disabled: /' /etc/fstab
cat <<'EOF' | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
cat <<'EOF' | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system

Output:

net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1

Install Kubernetes packages from the v1.36 repository and hold them:

sudo apt-get update
sudo apt-get install -y ca-certificates curl gpg apt-transport-https
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.36/deb/Release.key |
  sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /' |
  sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubeadm=1.36.5-1.1 kubelet=1.36.5-1.1 kubectl=1.36.5-1.1 cri-tools=1.36.0-1.1
sudo apt-mark hold kubeadm kubelet kubectl
kubeadm version -o short
kubelet --version
kubectl version --client -o jsonpath='{.clientVersion.gitVersion}{"\n"}'

Output:

kubeadm: v1.36.5
kubelet: Kubernetes v1.36.5
kubectl client gitVersion: v1.36.5

Point crictl at containerd:

cat <<'EOF' | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
pull-image-on-create: false
EOF
sudo crictl info | grep -E 'RuntimeName|RuntimeVersion|RuntimeApiVersion'

Output:

RuntimeName: containerd
RuntimeVersion: 2.2.1
RuntimeApiVersion: v1

Bootstrap the control plane:

sudo kubeadm init \
  --kubernetes-version v1.36.5 \
  --apiserver-advertise-address "${GUEST_IP}" \
  --pod-network-cidr "${POD_CIDR}" \
  --cri-socket unix:///run/containerd/containerd.sock

Output:

Your Kubernetes control-plane has initialized successfully!

Configure kubectl for the guest user:

mkdir -p "$HOME/.kube"
sudo cp /etc/kubernetes/admin.conf "$HOME/.kube/config"
sudo chown "$(id -u):$(id -g)" "$HOME/.kube/config"
kubectl get nodes -o wide

Output:

NAME          STATUS     ROLES           VERSION   INTERNAL-IP       OS-IMAGE
hami-lab-v2   NotReady   control-plane   v1.36.5   192.168.122.242   Ubuntu 24.04.5 LTS

Check why the node is not ready yet:

kubectl describe node hami-lab-v2 | grep -A2 'KubeletNotReady'

Output:

KubeletNotReady: container runtime network not ready
NetworkPluginNotReady: cni plugin not initialized

Gate: this NotReady state is expected before CNI. Continue only if the control plane pods are running and /livez plus /readyz pass.

Manual Calico Install

Download the pinned Calico manifest and verify its checksum:

CALICO_MANIFEST="$HOME/calico-v3.32.2.yaml"
curl -L \
  https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/calico.yaml \
  -o "$CALICO_MANIFEST"
(cd "$HOME" && printf '%s  %s\n' \
  'a8c828a06a87c629a282ebbc424895b77f3a030251993e41ea400a743675bb02' \
  'calico-v3.32.2.yaml' | sha256sum -c -)

Output:

calico-v3.32.2.yaml: OK

Set Calico's IPv4 pool to the same Pod CIDR used by kubeadm. The upstream manifest leaves this option commented; setting it explicitly keeps kubeadm and Calico aligned.

python3 - "$POD_CIDR" <<'PY'
from pathlib import Path
import re
import sys

cidr = sys.argv[1]
path = Path.home() / "calico-v3.32.2.yaml"
text = path.read_text()
new = f'''            - name: CALICO_IPV4POOL_CIDR
              value: "{cidr}"
'''
text, count = re.subn(
    r'(?m)^            # - name: CALICO_IPV4POOL_CIDR\n            #   value: "[^"]+"\n',
    new,
    text,
    count=1,
)
if count != 1:
    raise SystemExit("CALICO_IPV4POOL_CIDR block not found")
path.write_text(text)
PY

Dry-run the manifest before applying it:

kubectl apply --dry-run=server -f "$CALICO_MANIFEST"

Output:

daemonset.apps/calico-node created (server dry run)
deployment.apps/calico-kube-controllers created (server dry run)

Apply Calico:

kubectl apply -f "$CALICO_MANIFEST"
kubectl -n kube-system rollout status daemonset/calico-node --timeout=5m
kubectl -n kube-system rollout status deployment/calico-kube-controllers --timeout=5m
kubectl wait node hami-lab-v2 --for=condition=Ready --timeout=5m

Output:

daemonset/calico-node successfully rolled out
deployment/calico-kube-controllers successfully rolled out
node/hami-lab-v2 condition met

Inspect the cluster:

kubectl get nodes
kubectl -n kube-system get pods

Output:

NAME          STATUS   ROLES           VERSION
hami-lab-v2   Ready    control-plane   v1.36.5

calico-kube-controllers   1/1 Running
calico-node               1/1 Running
coredns                   1/1 Running
etcd-hami-lab-v2          1/1 Running
kube-apiserver-hami-lab-v2 1/1 Running
kube-controller-manager   1/1 Running
kube-proxy                1/1 Running
kube-scheduler-hami-lab-v2 1/1 Running

Validate DNS from a temporary pod. The toleration is required because this is a single-node control-plane cluster:

kubectl run v2-calico-dns-smoke \
  --image=busybox:1.36.1 \
  --restart=Never \
  --overrides='{"spec":{"tolerations":[{"key":"node-role.kubernetes.io/control-plane","operator":"Exists","effect":"NoSchedule"}]}}' \
  -- nslookup kubernetes.default.svc.cluster.local
kubectl logs v2-calico-dns-smoke
kubectl delete pod v2-calico-dns-smoke

Output:

Server: 10.96.0.10
Name: kubernetes.default.svc.cluster.local
Address: 10.96.0.1
pod "v2-calico-dns-smoke" deleted

Gate: node readiness, Calico, CoreDNS, API health, and guest GPU visibility must all pass before GPU workloads.

Plain CUDA Kubernetes Workload Before HAMi

Create the RuntimeClass for the named NVIDIA runtime handler:

kubectl apply -f - <<'EOF'
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia
EOF
kubectl get runtimeclass nvidia -o custom-columns='NAME:.metadata.name,HANDLER:.handler'

Output:

runtimeclass.node.k8s.io/nvidia created
NAME     HANDLER
nvidia   nvidia

Install the lab-adapted NVIDIA device plugin. It uses the stock nvcr.io/nvidia/k8s-device-plugin:v0.20.1 image, adds runtimeClassName: nvidia, and tolerates the control-plane taint:

kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      runtimeClassName: nvidia
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - image: nvcr.io/nvidia/k8s-device-plugin:v0.20.1
          name: nvidia-device-plugin-ctr
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: device-plugin
              mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins
EOF
kubectl -n kube-system rollout status daemonset/nvidia-device-plugin-daemonset --timeout=5m

Output:

runtimeclass.node.k8s.io/nvidia unchanged
daemonset.apps/nvidia-device-plugin-daemonset created
daemon set "nvidia-device-plugin-daemonset" successfully rolled out

Wait for the node to advertise one physical GPU:

kubectl get node hami-lab-v2 \
  -o jsonpath='capacity={.status.capacity.nvidia\.com/gpu} allocatable={.status.allocatable.nvidia\.com/gpu}{"\n"}'

Output:

capacity=1 allocatable=1

Run a plain CUDA workload before HAMi:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: v2-plain-cuda-vectoradd
spec:
  restartPolicy: Never
  runtimeClassName: nvidia
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
  containers:
    - name: vectoradd
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
      resources:
        limits:
          nvidia.com/gpu: 1
EOF
kubectl wait pod/v2-plain-cuda-vectoradd --for=jsonpath='{.status.phase}'=Succeeded --timeout=10m
kubectl logs v2-plain-cuda-vectoradd

Output:

pod/v2-plain-cuda-vectoradd created
pod/v2-plain-cuda-vectoradd condition met
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

This proves Kubernetes, the NVIDIA runtime handler, and the stock device plugin work before HAMi. It is not a HAMi result.

Remove the stock plugin before installing HAMi:

kubectl -n kube-system delete ds nvidia-device-plugin-daemonset --ignore-not-found
if kubectl -n kube-system get pod -l name=nvidia-device-plugin-ds --no-headers 2>/dev/null | grep -q .; then
  kubectl -n kube-system wait --for=delete pod \
    -l name=nvidia-device-plugin-ds --timeout=5m >/dev/null
fi
kubectl get runtimeclass nvidia

Output:

daemonset.apps "nvidia-device-plugin-daemonset" deleted
NAME     HANDLER   AGE
nvidia   nvidia

HAMi Helm Install with Inspected Values

HAMi needs the existing RuntimeClass, the gpu=on label, and tolerations for this single-node control-plane cluster.

Label the node:

kubectl label node hami-lab-v2 gpu=on
kubectl get node hami-lab-v2 -o jsonpath='gpu-label={.metadata.labels.gpu}{"\n"}'

Output:

node/hami-lab-v2 labeled
gpu-label=on

Install Helm and add the HAMi repo:

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
HELM_BUILDKITE_APT_KEY_ID="DDF78C3E6EBB2D2CC223C95C62BA89D07698DBC6"
curl -fsSL https://packages.buildkite.com/helm-linux/helm-debian/gpgkey -o /tmp/helm.gpgkey
gpg --show-keys --with-colons /tmp/helm.gpgkey |
  awk -F: '$1 == "fpr" { print $10 }' |
  grep -Fx "$HELM_BUILDKITE_APT_KEY_ID"
gpg --dearmor /tmp/helm.gpgkey |
  sudo tee /usr/share/keyrings/helm.gpg >/dev/null
echo "deb [signed-by=/usr/share/keyrings/helm.gpg] https://packages.buildkite.com/helm-linux/helm-debian/any/ any main" |
  sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install -y helm
helm version --short
helm repo add hami https://project-hami.github.io/HAMi
helm repo update
helm search repo hami/hami --versions | grep 2.10.0

Output:

v3.22.0
"hami" has been added to your repositories
Update Complete. Happy Helming!
hami/hami 2.10.0 2.10.0

Create /tmp/hami-v2-values.yaml. The same content is available in examples/20-local-gpu-passthrough/hami-values.yaml.

cat >/tmp/hami-v2-values.yaml <<'EOF'
devicePlugin:
  runtimeClassName: nvidia
  createRuntimeClass: false
  nvidiaDriverRoot: /
  nvidiaNodeSelector:
    gpu: "on"
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule

scheduler:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
  patch:
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
  kubeScheduler:
    image:
      tag: v1.36.5
EOF

Render the chart before installing it:

helm template hami hami/hami \
  --namespace kube-system \
  --version 2.10.0 \
  -f /tmp/hami-v2-values.yaml \
  | grep -E 'runtimeClassName: nvidia|gpu: "on"|tag: v1.36.5' \
  | sort -u

Output:

gpu: "on"
runtimeClassName: nvidia
tag: v1.36.5

Install HAMi:

helm install hami hami/hami \
  --namespace kube-system \
  --version 2.10.0 \
  -f /tmp/hami-v2-values.yaml \
  --wait \
  --timeout 10m

Output:

NAME: hami
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
Resource name: nvidia.com/gpu

Verify HAMi:

helm -n kube-system list --filter '^hami$'
kubectl -n kube-system get ds hami-device-plugin
kubectl -n kube-system get deploy hami-scheduler
kubectl get node hami-lab-v2 \
  -o jsonpath='capacity={.status.capacity.nvidia\.com/gpu} allocatable={.status.allocatable.nvidia\.com/gpu}{"\n"}'

Output:

hami  kube-system  1  deployed  hami-2.10.0  2.10.0
hami-device-plugin  desired=1 current=1 ready=1 available=1
hami-scheduler      ready=1/1 available=1
capacity=10 allocatable=10

Inspect the HAMi node registration:

kubectl get node hami-lab-v2 \
  -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}' |
  python3 -c 'import json, sys
gpu = json.load(sys.stdin)[0]
for key in ("type", "count", "devmem", "devcore", "health"):
    print(f"{key}={gpu[key]}")'

Output:

type=NVIDIA GeForce RTX 3050 Laptop GPU
count=10
devmem=4096
devcore=100
health=true

Gate: HAMi pods must be healthy and the node must advertise nvidia.com/gpu capacity and allocatable as 10.

Fractional HAMi Workload

Run a fractional CUDA sample through HAMi:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: v2-hami-fractional-vectoradd
spec:
  restartPolicy: Never
  runtimeClassName: nvidia
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
  containers:
    - name: vectoradd
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
      resources:
        limits:
          nvidia.com/gpu: 1
          nvidia.com/gpumem: 2048
          nvidia.com/gpucores: 50
EOF
kubectl wait pod/v2-hami-fractional-vectoradd --for=jsonpath='{.status.phase}'=Succeeded --timeout=10m
kubectl logs v2-hami-fractional-vectoradd

Output:

pod/v2-hami-fractional-vectoradd created
pod/v2-hami-fractional-vectoradd condition met
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

Inspect how HAMi scheduled the pod:

kubectl get pod v2-hami-fractional-vectoradd \
  -o jsonpath='scheduler={.spec.schedulerName}{"\n"}runtimeClass={.spec.runtimeClassName}{"\n"}allocated={.metadata.annotations.hami\.io/vgpu-devices-allocated}{"\n"}node={.metadata.annotations.hami\.io/vgpu-node}{"\n"}'

Output:

scheduler=hami-scheduler
runtimeClass=nvidia
allocated=GPU-55c8d7db-73a6-bf9a-e47f-05680f2e16f8,NVIDIA,2048,50:;
node=hami-lab-v2

This validates the HAMi workload path:

Main-Process Memory-Slice Proof

Memory enforcement depends on the process path. Use the container main process as the primary proof, because it starts with the environment HAMi injects for the workload. Do not use a later login shell or ad hoc kubectl exec path as the main evidence.

This lab intentionally separates the process paths:

Process path	Status in the tested environment
Startup, child, kubectl exec non-login, and su non-login paths	Reported 2048 MiB and denied the 3072 MiB allocation.
Login shell and SSH-started process	Lost the HAMi allocation environment and exposed the full 4096 MiB.

The login-shell and SSH-started paths also allowed the 3072 MiB allocation after falling back to /tmp/cudevshr.cache. They are confirmed limitations in this environment, not alternate success paths. The procedure below keeps the workload's main process as the reproducible enforcement proof.

Run nvidia-smi and vectorAdd from the container's main command:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: v2-hami-process-path-check
spec:
  restartPolicy: Never
  runtimeClassName: nvidia
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
  containers:
    - name: check
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
      command: ["/bin/sh", "-lc"]
      args:
        - |
          echo "===== CONTAINER NVIDIA-SMI ====="
          nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
          echo
          echo "===== CONTAINER VECTORADD ====="
          /cuda-samples/vectorAdd
      resources:
        limits:
          nvidia.com/gpu: 1
          nvidia.com/gpumem: 2048
          nvidia.com/gpucores: 50
EOF
kubectl wait pod/v2-hami-process-path-check --for=jsonpath='{.status.phase}'=Succeeded --timeout=10m
kubectl logs v2-hami-process-path-check

Output:

pod/v2-hami-process-path-check created
pod/v2-hami-process-path-check condition met
===== CONTAINER NVIDIA-SMI =====
NVIDIA GeForce RTX 3050 Laptop GPU, 2048 MiB

===== CONTAINER VECTORADD =====
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

Inspect annotations and events:

kubectl get pod v2-hami-process-path-check \
  -o jsonpath='phase={.status.phase}{"\n"}scheduler={.spec.schedulerName}{"\n"}allocated={.metadata.annotations.hami\.io/vgpu-devices-allocated}{"\n"}'
kubectl describe pod v2-hami-process-path-check | grep -E 'Scheduled|FilteringSucceed|BindingSucceed'

Output:

phase=Succeeded
scheduler=hami-scheduler
allocated=GPU-55c8d7db-73a6-bf9a-e47f-05680f2e16f8,NVIDIA,2048,50:;
Scheduled by hami-scheduler
FilteringSucceed: find fit node(hami-lab-v2)
BindingSucceed: Successfully binding node [hami-lab-v2]

Gate: the public memory-slice proof is the main process reporting 2048 MiB and then completing vectorAdd.

Check Memory Over-allocation Behavior

The vectorAdd check proves that the fractional workload can run. This next check asks the CUDA Driver API to allocate below, at, and above the requested 2048 MiB slice. It runs the allocation probe as the container's main process and writes, synchronizes, and frees successful allocations.

This is a main-process check only. Do not use a login shell, SSH session, or later kubectl exec process as equivalent evidence: those paths can have a different runtime environment.

Create the probe on the guest. The tested Ubuntu guest already provided cc; stop here and install a compiler if cc --version fails. The probe loads the CUDA Driver API dynamically, then reports the return code from each operation:

cc --version

cat > cuda-mem-probe.c <<'EOF'
#include <dlfcn.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>

typedef int CUresult;
typedef int CUdevice;
typedef void *CUcontext;
typedef unsigned long long CUdeviceptr;
#define CUDA_SUCCESS 0

int main(int argc, char **argv) {
  if (argc != 2) return 2;
  size_t mib = strtoull(argv[1], NULL, 10);
  size_t bytes = mib * 1024ULL * 1024ULL;
  void *cuda = dlopen("libcuda.so.1", RTLD_NOW | RTLD_GLOBAL);
  if (!cuda) { fprintf(stderr, "dlopen libcuda.so.1 failed: %s\n", dlerror()); return 2; }
  CUresult (*cuInit)(unsigned int) = dlsym(cuda, "cuInit");
  CUresult (*cuDeviceGet)(CUdevice *, int) = dlsym(cuda, "cuDeviceGet");
  CUresult (*cuCtxCreate)(CUcontext *, unsigned int, CUdevice) = dlsym(cuda, "cuCtxCreate_v2");
  CUresult (*cuMemAlloc)(CUdeviceptr *, size_t) = dlsym(cuda, "cuMemAlloc_v2");
  CUresult (*cuMemsetD8)(CUdeviceptr, unsigned char, size_t) = dlsym(cuda, "cuMemsetD8_v2");
  CUresult (*cuCtxSynchronize)(void) = dlsym(cuda, "cuCtxSynchronize");
  CUresult (*cuMemFree)(CUdeviceptr) = dlsym(cuda, "cuMemFree_v2");
  CUresult (*cuCtxDestroy)(CUcontext) = dlsym(cuda, "cuCtxDestroy_v2");
  if (!cuInit || !cuDeviceGet || !cuCtxCreate || !cuMemAlloc || !cuMemsetD8 || !cuCtxSynchronize || !cuMemFree || !cuCtxDestroy) return 2;
  CUdevice dev; CUcontext ctx; CUdeviceptr ptr = 0;
  CUresult rc = cuInit(0); if (rc) { printf("probe requested_mib=%zu cuInit_rc=%d\n", mib, rc); return 1; }
  rc = cuDeviceGet(&dev, 0); if (rc) { printf("probe requested_mib=%zu cuDeviceGet_rc=%d\n", mib, rc); return 1; }
  rc = cuCtxCreate(&ctx, 0, dev); if (rc) { printf("probe requested_mib=%zu cuCtxCreate_rc=%d\n", mib, rc); return 1; }
  rc = cuMemAlloc(&ptr, bytes); printf("probe requested_mib=%zu cuMemAlloc_rc=%d\n", mib, rc);
  if (rc == CUDA_SUCCESS) {
    CUresult set_rc = cuMemsetD8(ptr, 0, bytes);
    CUresult sync_rc = cuCtxSynchronize();
    CUresult free_rc = cuMemFree(ptr);
    printf("probe requested_mib=%zu cuMemsetD8_rc=%d cuCtxSynchronize_rc=%d cuMemFree_rc=%d\n", mib, set_rc, sync_rc, free_rc);
    if (set_rc || sync_rc || free_rc) rc = 1;
  }
  CUresult destroy_rc = cuCtxDestroy(ctx);
  printf("probe requested_mib=%zu cuCtxDestroy_rc=%d\n", mib, destroy_rc);
  return (rc == CUDA_SUCCESS && destroy_rc == CUDA_SUCCESS) ? 0 : 1;
}
EOF

cc cuda-mem-probe.c -O2 -Wall -Wextra -Werror -o cuda-mem-probe -ldl
kubectl create configmap hami-memory-probe \
  --from-file=cuda-mem-probe=./cuda-mem-probe

Run one disposable Pod for each size. This manifest uses imagePullPolicy: Never because the earlier plain CUDA and fractional workload steps already cached nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0 on the node. Do not continue if those steps did not pull and run that exact image successfully.

The helper validates the scheduler, runtime class, HAMi allocation, visible memory, probe result, Pod phase, and container exit code before deleting a successful gate. If a gate fails, it retains the Pod, prints diagnostics, returns non-zero, and prevents the later allocation tests from running:

probe_diagnostics() {
  local pod="$1"

  echo "Probe gate failed; retaining ${pod} for diagnosis" >&2
  kubectl get "pod/${pod}" -o wide >&2 || true
  kubectl describe "pod/${pod}" >&2 || true
  kubectl logs "${pod}" >&2 || true
}

run_probe() {
  local requested_mib="$1"
  local expected_phase="$2"
  local expected_exit="$3"
  local expected_alloc_rc="$4"
  local pod="hami-memory-probe-${requested_mib}"
  local existing_pod metadata logs scheduler runtime_class allocation phase exit_code

  if ! existing_pod="$(kubectl get "pod/${pod}" --ignore-not-found -o name)"; then
    echo "Kubernetes API query failed while checking ${pod}" >&2
    return 1
  fi
  if [ -n "$existing_pod" ]; then
    echo "Refusing to reuse existing Pod ${pod}" >&2
    probe_diagnostics "$pod"
    return 1
  fi

  kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: ${pod}
spec:
  restartPolicy: Never
  schedulerName: hami-scheduler
  runtimeClassName: nvidia
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
  volumes:
    - name: probe
      configMap:
        name: hami-memory-probe
        defaultMode: 0755
  containers:
    - name: probe
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
      imagePullPolicy: Never
      command: ["/bin/sh", "-ec"]
      args:
        - |
          nvidia-smi --query-gpu=name,driver_version,memory.total,memory.used --format=csv,noheader
          exec /probe/cuda-mem-probe ${requested_mib}
      volumeMounts:
        - name: probe
          mountPath: /probe
          readOnly: true
      resources:
        limits:
          nvidia.com/gpu: 1
          nvidia.com/gpumem: 2048
          nvidia.com/gpucores: 50
EOF
  if [ "$?" -ne 0 ]; then
    probe_diagnostics "$pod"
    return 1
  fi

  if ! kubectl wait "pod/${pod}" \
    --for=jsonpath='{.status.phase}'="${expected_phase}" --timeout=5m; then
    probe_diagnostics "$pod"
    return 1
  fi

  if ! metadata="$(kubectl get "pod/${pod}" \
    -o jsonpath='{.spec.schedulerName}{"\t"}{.spec.runtimeClassName}{"\t"}{.metadata.annotations.hami\.io/vgpu-devices-allocated}{"\t"}{.status.phase}{"\t"}{.status.containerStatuses[0].state.terminated.exitCode}')"; then
    probe_diagnostics "$pod"
    return 1
  fi
  IFS=$'\t' read -r scheduler runtime_class allocation phase exit_code <<<"$metadata"

  if ! logs="$(kubectl logs "${pod}")"; then
    probe_diagnostics "$pod"
    return 1
  fi

  printf 'scheduler=%s\nruntimeClass=%s\nallocated=%s\nphase=%s\n' \
    "$scheduler" "$runtime_class" "$allocation" "$phase"
  printf '%s\n' "$logs"

  if [ "$scheduler" != "hami-scheduler" ] || \
    [ "$runtime_class" != "nvidia" ] || \
    [ "$phase" != "$expected_phase" ] || \
    [ "$exit_code" != "$expected_exit" ] || \
    ! printf '%s\n' "$allocation" | grep -Eq ',NVIDIA,2048,50:;$' || \
    ! printf '%s\n' "$logs" | grep -Fq ', 2048 MiB,' || \
    ! printf '%s\n' "$logs" | grep -Fq "probe requested_mib=${requested_mib} cuMemAlloc_rc=${expected_alloc_rc}" || \
    ! printf '%s\n' "$logs" | grep -Fq "probe requested_mib=${requested_mib} cuCtxDestroy_rc=0"; then
    probe_diagnostics "$pod"
    return 1
  fi

  if [ "$expected_alloc_rc" = "0" ]; then
    if ! printf '%s\n' "$logs" | grep -Fq \
      "probe requested_mib=${requested_mib} cuMemsetD8_rc=0 cuCtxSynchronize_rc=0 cuMemFree_rc=0"; then
      probe_diagnostics "$pod"
      return 1
    fi
  elif ! printf '%s\n' "$logs" | grep -Fq \
    'Device 0 OOM 3221225472 / 2147483648'; then
    probe_diagnostics "$pod"
    return 1
  fi

  if ! kubectl delete "pod/${pod}" --wait=true; then
    probe_diagnostics "$pod"
    return 1
  fi
}

run_probe 512 Succeeded 0 0 || exit 1
run_probe 2048 Succeeded 0 0 || exit 1
run_probe 3072 Failed 1 2 || exit 1

In the tested run, each Pod received a 2048 MiB / 50-core HAMi allocation and reported 2048 MiB of visible memory. The results were:

Requested allocation	Pod result	Probe result
512 MiB	Succeeded	Allocation, write, synchronization, and free returned 0.
2048 MiB	Succeeded	Allocation, write, synchronization, and free returned 0.
3072 MiB	Failed	cuMemAlloc_rc=2; HAMi logged Device 0 OOM 3221225472 / 2147483648.

Output excerpts from the three runs:

NVIDIA GeForce RTX 3050 Laptop GPU, 595.91.07, 2048 MiB, 0 MiB
probe requested_mib=512 cuMemAlloc_rc=0
probe requested_mib=512 cuMemsetD8_rc=0 cuCtxSynchronize_rc=0 cuMemFree_rc=0
probe requested_mib=512 cuCtxDestroy_rc=0

NVIDIA GeForce RTX 3050 Laptop GPU, 595.91.07, 2048 MiB, 0 MiB
probe requested_mib=2048 cuMemAlloc_rc=0
probe requested_mib=2048 cuMemsetD8_rc=0 cuCtxSynchronize_rc=0 cuMemFree_rc=0
probe requested_mib=2048 cuCtxDestroy_rc=0

NVIDIA GeForce RTX 3050 Laptop GPU, 595.91.07, 2048 MiB, 0 MiB
[HAMI-core ERROR (pid:1 thread=123130178780032 allocator.c:52)]: Device 0 OOM 3221225472 / 2147483648
probe requested_mib=3072 cuMemAlloc_rc=2
probe requested_mib=3072 cuCtxDestroy_rc=0

The 3072 MiB result is the expected over-allocation behavior for this 2048 MiB slice. The allocator compared the 3072 MiB request with the configured 2048 MiB limit, rather than exhausting the physical 4096 MiB GPU.

Remove the temporary probe artifacts when you are done:

kubectl delete configmap hami-memory-probe
rm -f cuda-mem-probe.c cuda-mem-probe

Clean Shutdown and Full Host Recovery

Before shutting down the VM, confirm the expected ownership while it is running:

lspci -nnk -s 01:00.0 | grep 'Kernel driver in use'
lspci -nnk -s 01:00.1 | grep 'Kernel driver in use'
nvidia-smi

Output:

Kernel driver in use: vfio-pci
Kernel driver in use: vfio-pci
Failed to initialize NVML: No supported GPUs were found

Shut the guest down cleanly and wait for libvirt:

ssh "${GUEST_USER}@${GUEST_IP}" 'sudo poweroff'
for attempt in $(seq 1 60); do
  DOMAIN_STATE="$(virsh -c qemu:///system domstate "$DOMAIN")"
  [ "$DOMAIN_STATE" = "shut off" ] && break
  sleep 2
done
[ "$DOMAIN_STATE" = "shut off" ] || {
  echo "Host recovery blocked: $DOMAIN did not shut off within 120 seconds" >&2
  exit 1
}
printf '%s\n' "$DOMAIN_STATE"

Output:

shut off

Verify host PCI ownership:

lspci -nnk -s 01:00.0 | grep 'Kernel driver in use'
lspci -nnk -s 01:00.1 | grep 'Kernel driver in use'

Output:

Kernel driver in use: nvidia
Kernel driver in use: snd_hda_intel

Restart the generic services that were stopped before passthrough. If you manually stopped a machine-specific service such as fan control, restore it separately using its documented unit name and verify that it is active before considering recovery complete.

if systemctl list-unit-files nvidia-persistenced.service --no-legend 2>/dev/null | grep -q .; then
  if ! sudo systemctl restart nvidia-persistenced || \
    ! sudo systemctl is-active --quiet nvidia-persistenced; then
    echo "Warning: optional nvidia-persistenced service is not active" >&2
  fi
fi

AUDIO_RECOVERED=false
if systemctl --user restart pipewire.socket pipewire.service pipewire-pulse.socket pipewire-pulse.service wireplumber.service && \
  systemctl --user is-active --quiet pipewire.socket pipewire.service pipewire-pulse.socket pipewire-pulse.service wireplumber.service; then
  AUDIO_RECOVERED=true
else
  echo "Host recovery failed: one or more audio services are not active" >&2
fi

if [ "$AUDIO_RECOVERED" != true ]; then
  exit 1
fi
systemctl --user is-active pipewire.socket pipewire.service pipewire-pulse.socket pipewire-pulse.service wireplumber.service

Output:

active
active
active
active
active

Confirm PulseAudio compatibility on PipeWire and the returned NVIDIA HDA device:

pactl info | grep -E 'Server Name|Default Sink'
pactl list short cards | awk '{print $2}' | grep -E 'alsa_card\.pci-0000_(01_00\.1|06_00\.[16])'

Output:

Server Name: PulseAudio (on PipeWire 1.6.8)
Default Sink: alsa_output.pci-0000_06_00.1.pro-output-3
alsa_card.pci-0000_01_00.1
alsa_card.pci-0000_06_00.1
alsa_card.pci-0000_06_00.6

Finally, verify the host GPU:

nvidia-smi

Output:

NVIDIA-SMI 615.71.09
NVIDIA GeForce RTX 3050 Laptop GPU
Memory-Usage: 1MiB / 4096MiB
No running processes found

Troubleshooting Notes

• If the VM will not start after hostdev attachment, recheck host holders with fuser -v /dev/nvidia* /dev/snd/* and confirm the host display is not using the NVIDIA GPU.
• If the node stays NotReady after kubeadm, do not continue to HAMi. Install and validate CNI first.
• If a workload stays Pending on this single-node control-plane lab, check for the node-role.kubernetes.io/control-plane:NoSchedule taint and add the toleration shown in the manifests above.
• If nvidia-smi works in the guest but not inside containers, fix the containerd NVIDIA runtime layer before installing Kubernetes.
• If the main-process memory proof reports the full 4096 MiB, inspect the HAMi allocation annotation, RuntimeClass, and resource limits before drawing a conclusion about memory enforcement.
• Login shells and later process paths can behave differently from the container main process. Treat them as advanced debugging paths, not as the primary proof for this lab.