Install HAMi on Alibaba Cloud Container Service for Kubernetes

This guide covers the ACK-specific configuration for installing HAMi on NVIDIA GPU nodes in Alibaba Cloud Container Service for Kubernetes (ACK). For everything else, follow Prerequisites and Online Installation from Helm.

Prepare the GPU node pool

When creating the node pool, select a GPU instance type. ACK installs the NVIDIA driver and NVIDIA Container Toolkit on the node, so GPU Operator is not required. ACK's shared GPU scheduling components are not required either.

ACK also sets nvidia-container-runtime as the binary of containerd's default runc runtime. Every container therefore uses the NVIDIA runtime, and HAMi does not need a RuntimeClass. To confirm, run on the GPU node:

grep -A1 'runtimes.runc.options' /etc/containerd/config.toml

The output should contain:

          [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc.options]
            BinaryName = "nvidia-container-runtime"

Because ACK installs the driver directly on the host, with nvidia-smi in /usr/bin and the libraries in /lib/x86_64-linux-gnu, set HAMi's devicePlugin.nvidiaDriverRoot to /. See Prepare the HAMi values.

Disable ACK's NVIDIA Device Plugin

ACK installs the ack-nvidia-device-plugin add-on by default. It runs as the kube-system/ack-nvidia-device-plugin DaemonSet on nodes labeled ack.node.gpu.schedule=default, a label ACK adds to GPU nodes. It registers nvidia.com/gpu through the same kubelet socket as HAMi's Device Plugin, so only one of them can be active on a node. See Pods see the whole GPU for what happens when both run.

Use one of the following methods.

Method 1: Label the GPU nodes

This method disables ACK's Device Plugin on the labeled nodes only. Manage the labels through the node pool: edit the GPU node pool in the ACK console and add the following Node Labels:

Key	Value	Purpose
ack.node.gpu.schedule	disabled	Stops ACK's Device Plugin on the node
gpu	on	Selects the node for HAMi's Device Plugin

The disabled value is described in ACK's Schedule GPUs with DRA guide.

To label a single node with kubectl instead, use it only on a new node, or on a drained node with no running GPU workloads:

kubectl label node <node-name> ack.node.gpu.schedule=disabled gpu=on --overwrite

Verify the labels and confirm that ACK's DaemonSet no longer schedules Pods on the node:

kubectl get nodes -L gpu,ack.node.gpu.schedule
kubectl -n kube-system get ds ack-nvidia-device-plugin

NAME                      STATUS   ROLES    AGE   VERSION            GPU   ACK.NODE.GPU.SCHEDULE
cn-hangzhou.10.207.93.6   Ready    <none>   51m   v1.36.2-aliyun.1   on    disabled

NAME                       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                   AGE
ack-nvidia-device-plugin   0         0         0       0            0           ack.node.gpu.schedule=default   57m

Method 2: Uninstall the add-on

This method disables ACK's Device Plugin for the whole cluster. Uninstall the ack-nvidia-device-plugin add-on on the Add-ons page of the cluster in the ACK console. Then label the GPU nodes for HAMi's Device Plugin:

kubectl label node <node-name> gpu=on

Copy the HAMi images to Container Registry

Nodes in Chinese mainland regions cannot reach Docker Hub, so the hami and kube-webhook-certgen images fail to pull. The chart's default kube-scheduler image comes from registry.cn-hangzhou.aliyuncs.com and pulls normally.

Copy the two Docker Hub images to an Alibaba Cloud Container Registry (ACR) instance in the same region as the cluster, and make the hami and kube-webhook-certgen repositories public so that the nodes can pull them without credentials. Use either edition:

• ACR Enterprise Edition can sync the images from Docker Hub. See Create an ACR Enterprise Edition instance and Subscribe to images from overseas sources.
• ACR Personal Edition requires you to push the images yourself. See Push and pull images by using a Personal Edition instance.

To push the images yourself, log in to the instance's public endpoint from a machine that can reach Docker Hub, and copy the images. --all copies the image for every platform:

skopeo login --username=<username> registry.cn-<personal-instance-region>.aliyuncs.com
skopeo copy --all docker://docker.io/projecthami/hami:v2.10.0 \
  docker://registry.cn-<personal-instance-region>.aliyuncs.com/<namespace>/hami:v2.10.0
skopeo copy --all docker://docker.io/liangjw/kube-webhook-certgen:v1.1.1 \
  docker://registry.cn-<personal-instance-region>.aliyuncs.com/<namespace>/kube-webhook-certgen:v1.1.1

Grant the HAMi scheduler access to DRA resources

The kube-scheduler container in the hami-scheduler Pod is bound to the built-in system:kube-scheduler ClusterRole. On ACK, this ClusterRole has no rules for the resource.k8s.io API group. kube-scheduler v1.36 watches the DRA resources in that group, so it logs the following error and never starts leader election:

E1010 16:36:37.676013       1 reflector.go:227] "Failed to watch" err="failed to list *v1.ResourceClaim: resourceclaims.resource.k8s.io is forbidden: User \"system:serviceaccount:kube-system:hami-scheduler\" cannot list resource \"resourceclaims\" in API group \"resource.k8s.io\" at the cluster scope"

The hami-scheduler Pod stays Running, but its vgpu-scheduler-extender container repeats Scheduler extender has not become leader yet, and Pods that request vGPU resources stay Pending.

Before installing HAMi, save the following as hami-scheduler-dra-rbac.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: hami-scheduler-dra
rules:
  - apiGroups: ["resource.k8s.io"]
    resources: ["resourceclaims", "resourceslices", "deviceclasses"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hami-scheduler-dra
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: hami-scheduler-dra
subjects:
  - kind: ServiceAccount
    name: hami-scheduler
    namespace: kube-system

Apply it:

kubectl apply -f hami-scheduler-dra-rbac.yaml

The binding refers to the ServiceAccount of a release named hami. For another release name, the ServiceAccount is <release-name>-hami-scheduler.

Prepare the HAMi values

Point the HAMi images to your ACR repositories. Use the instance's VPC endpoint, which the nodes reach without a NAT gateway. Set each image separately, because global.imageRegistry also changes the registry of the kube-scheduler image.

Save the following as hami-ack-values.yaml, replacing <personal-instance-region> and <namespace>. For Enterprise Edition, set registry to the VPC endpoint of the Enterprise Edition instance:

scheduler:
  extender:
    image:
      registry: registry-vpc.cn-<personal-instance-region>.aliyuncs.com
      repository: <namespace>/hami
  patch:
    imageNew:
      registry: registry-vpc.cn-<personal-instance-region>.aliyuncs.com
      repository: <namespace>/kube-webhook-certgen
devicePlugin:
  nvidiaDriverRoot: /
  image:
    registry: registry-vpc.cn-<personal-instance-region>.aliyuncs.com
    repository: <namespace>/hami
  monitor:
    image:
      registry: registry-vpc.cn-<personal-instance-region>.aliyuncs.com
      repository: <namespace>/hami

Install HAMi

Follow Online Installation from Helm and add the values file to the installation command:

helm install hami hami-charts/hami -n kube-system --version 2.10.0 -f hami-ack-values.yaml

After hami-device-plugin and hami-scheduler are running, confirm that the HAMi scheduler holds its leader lease:

kubectl -n kube-system get lease hami-scheduler

NAME             HOLDER                                                                 AGE
hami-scheduler   hami-scheduler-566cc89c9d-68f6n_8ed04e0d-8e3d-457a-a866-76fc43de07f5   38s

Then confirm that HAMi registered the GPU. With the default devicePlugin.deviceSplitCount of 10, a single GPU is reported as 10 nvidia.com/gpu resources:

kubectl get node <node-name> -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'

10

Troubleshooting

Pods see the whole GPU

If ACK's Device Plugin runs on a node alongside HAMi, the last Device Plugin to register takes over nvidia.com/gpu. When ACK's Device Plugin wins, the node's nvidia.com/gpu allocatable drops from the HAMi value (for example, 10) to the physical GPU count. New Pods start without errors, but HAMi's limits no longer apply.

To recover:

1. Disable ACK's Device Plugin on the node as described in Disable ACK's NVIDIA Device Plugin, and wait for the ack-nvidia-device-plugin Pod on that node to be deleted.

2. Restart HAMi's Device Plugin so it registers again:

   kubectl -n kube-system rollout restart ds/hami-device-plugin
   kubectl -n kube-system rollout status ds/hami-device-plugin

3. Confirm that the node's nvidia.com/gpu allocatable is back to the HAMi value.

4. Recreate the Pods that started while ACK's Device Plugin was active.

NUMA error in Device Plugin logs

On an ecs.gn6i-c4g1.xlarge instance, which has a single NUMA node, hami-device-plugin logs the following error every registration cycle:

E1010 16:47:54.742604   36382 register.go:168] "failed to get numa information from sysfs" idx=0
I1010 16:47:54.742629   36382 register.go:204] Registered device id=0, memory=15360MB, type=NVIDIA-Tesla T4, numa=0, health=true

HAMi reads the GPU's NUMA node from /sys/bus/pci/devices/<bus-id>/numa_node. On this instance, the kernel reports -1 for every PCI device, including the GPU. HAMi cannot determine the NUMA node, logs the error, and falls back to numa=0, which matches the actual topology. The device still registers with health=true, and GPU workloads run normally. You can ignore this error on single-NUMA instances.