Skip to main content
Version: v2.11.0

Install HAMi on Tencent Kubernetes Engine

This guide covers the TKE-specific configuration for installing HAMi on NVIDIA GPU nodes in Tencent Kubernetes Engine (TKE). For everything else, follow Prerequisites and Online Installation from Helm.

This guide uses the following environment:

ComponentVersion
Node typeTKE native node (GN7.2XLARGE32, 1 × Tesla T4)
Operating systemTencentOS Server 3.1
Kubernetesv1.34.1-tke.8
Container runtimecontainerd 1.6.9-tke.9
NVIDIA driver580.126.20, CUDA 13.0.2, installed by TKE
HAMi2.11.0

Prepare the GPU node pool​

When creating the node pool, select a GPU instance type and a GPU driver version. TKE installs the NVIDIA driver and NVIDIA Container Toolkit on the node, so GPU Operator is not required. Keep qGPU sharing disabled; this guide was verified without it.

TKE also sets nvidia-container-runtime as the binary of containerd's default runc runtime. Every container therefore uses the NVIDIA runtime, and HAMi does not need a RuntimeClass. To confirm, run on the GPU node:

grep -A1 'runtimes.runc.options' /etc/containerd/config.toml

The output should contain:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
BinaryName = "nvidia-container-runtime"

Because TKE installs the driver directly on the host, with nvidia-smi in /usr/bin and the libraries in /usr/lib64, set HAMi's devicePlugin.nvidiaDriverRoot to /. See Prepare the HAMi values.

Disable TKE's NVIDIA Device Plugin​

TKE deploys its own NVIDIA Device Plugin as the kube-system/nvidia-device-plugin-daemonset DaemonSet. It runs on nodes labeled nvidia-device-enable=enable, a label TKE adds to GPU nodes. It registers nvidia.com/gpu through the same kubelet socket as HAMi, so only one of them can be active on a node. See Pods see the whole GPU for what happens when both run.

In the TKE console, edit the GPU node pool and set the following Labels:

KeyValuePurpose
nvidia-device-enabledisableStops TKE's Device Plugin on the node
gpuonSelects the node for HAMi's Device Plugin

Select Apply this Label/Annotation/Taint update to existing nodes so that nodes already in the pool are updated.

Alternatively, label an individual node with kubectl:

kubectl label node <node-name> nvidia-device-enable=disable gpu=on --overwrite

Verify the labels and confirm that TKE's DaemonSet no longer schedules Pods on the node:

kubectl get nodes -L gpu,nvidia-device-enable
kubectl -n kube-system get ds nvidia-device-plugin-daemonset
NAME STATUS ROLES AGE VERSION GPU NVIDIA-DEVICE-ENABLE
10.100.0.13 Ready <none> 59m v1.34.1-tke.8 on disable

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
nvidia-device-plugin-daemonset 0 0 0 0 0 nvidia-device-enable=enable 66m

Prepare the HAMi values​

TKE nodes have no public IP by default, so unless the VPC provides a NAT gateway they can pull images only from Tencent Cloud internal endpoints. The docker.io/projecthami/hami image still pulls normally, because TKE configures mirror.ccs.tencentyun.com as the containerd mirror for docker.io. The chart's default kube-scheduler and certgen images, however, come from registry.k8s.io, which such nodes cannot reach.

When the kube-scheduler image pull fails, the hami-scheduler Pod reports an error similar to:

Failed to pull image "registry.k8s.io/kube-scheduler:v1.34.1": ... i/o timeout

If the certgen image pull fails during installation, inspect the hami-admission-create Job and its Pod. Helm waits for this pre-install Job to complete, so the hami-scheduler Pod may not exist yet.

For kube-scheduler, use TKE's ccr.ccs.tencentyun.com/tkeimages/hyperkube image instead. It contains the kube-scheduler binary, and its tags match TKE's Kubernetes versions. Find the version in the VERSION column:

kubectl get nodes

Save the following as hami-tke-values.yaml, replacing the kube-scheduler tag with your cluster version and the registry and repository path placeholders:

scheduler:
kubeScheduler:
image:
registry: ccr.ccs.tencentyun.com
repository: tkeimages/hyperkube
tag: v1.34.1-tke.8
patch:
imageNew:
# Before installation, run on a machine with access to both registries:
# skopeo login <internal-registry>
# skopeo copy --all docker://registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.9 docker://<internal-registry>/<image-namespace>/kube-webhook-certgen:v1.6.9
registry: <internal-registry>
repository: <image-namespace>/kube-webhook-certgen
tag: v1.6.9
# If pull credentials are required, use an existing image pull Secret in kube-system:
# pullSecrets: [<image-pull-secret>]
devicePlugin:
nvidiaDriverRoot: /

Install HAMi​

Follow Online Installation from Helm and add the values file to the installation command:

helm install hami hami-charts/hami -n kube-system --version 2.11.0 -f hami-tke-values.yaml

After hami-device-plugin and hami-scheduler are running, confirm that HAMi registered the GPU. With the default devices.nvidia.deviceSplitCount of 10, a single T4 is reported as 10 nvidia.com/gpu resources:

kubectl get node <node-name> -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'
10

Troubleshooting​

Pods see the whole GPU​

If TKE's Device Plugin runs on a node alongside HAMi, the last plugin to register takes over nvidia.com/gpu. When TKE's plugin wins, the node's nvidia.com/gpu allocatable drops from the HAMi value (for example, 10) to the physical GPU count. New Pods start without errors, but HAMi's limits no longer apply.

To recover:

  1. Set nvidia-device-enable=disable on the node as described in Disable TKE's NVIDIA Device Plugin, and wait for the nvidia-device-plugin-daemonset Pod on that node to be deleted.

  2. Restart HAMi's Device Plugin so it registers again:

    kubectl -n kube-system rollout restart ds/hami-device-plugin
    kubectl -n kube-system rollout status ds/hami-device-plugin
  3. Confirm that the node's nvidia.com/gpu allocatable is back to the HAMi value.

  4. Recreate the Pods that started while TKE's plugin was active.

NUMA error in Device Plugin logs​

On the verified T4 instance, hami-device-plugin logs the following error every registration cycle:

E0916 16:14:55.021458 83601 register.go:168] "failed to get numa information from sysfs" idx=0
I0916 16:14:55.021480 83601 register.go:204] Registered device id=0, memory=15360MB, type=NVIDIA-Tesla T4, numa=0, health=true

HAMi reads the GPU's NUMA node from /sys/bus/pci/devices/<bus-id>/numa_node. The kernel writes -1 there when the firmware exposes no proximity domain (ACPI _PXM) for the device, which is the normal case on a single-NUMA virtual machine such as this instance type. HAMi cannot determine a node, logs the error, and falls back to numa=0. Check the value on the GPU node:

nvidia-smi --query-gpu=pci.bus_id --format=csv,noheader |
tr 'A-F' 'a-f' | sed 's/^0000//' |
while read -r id; do echo "$id $(cat /sys/bus/pci/devices/$id/numa_node)"; done
0000:00:08.0 -1

The instance has only one NUMA node, node0, so the fallback matches the actual topology. The device still registers with health=true, and GPU workloads run normally. You can ignore this error on single-NUMA instances.

CNCFHAMi is a CNCF Incubating project