Lab 15: Run SGLang on HAMi GPU Shares

IntermediateDuration: about 45 minutesEnvironment: Kubernetes cluster with NVIDIA GPUsVerified: 2026-07-23By: @rudrakshkarpe, @shivaylamba

This lab demonstrates how to install HAMi on a Kubernetes cluster that already has NVIDIA GPUs, and use HAMi to schedule an SGLang inference service. Upon completion, you will have an OpenAI-compatible model service that can be verified through /v1/models and /v1/chat/completions, with HAMi enforcing a software-based GPU memory quota and compute throttle inside the Pod.

This guide is modeled after Lab 6: Run vLLM on HAMi GPU Shares. The steps are not tied to a specific cloud vendor. As long as your Kubernetes cluster has available NVIDIA GPUs, NVIDIA drivers, and container runtime support, you can reproduce the same setup.

Learning Objectives

• Verify that an existing GPU Kubernetes cluster meets the prerequisites for HAMi and SGLang
• Install HAMi scheduler and device plugin
• Add labels required by the HAMi DaemonSet to GPU nodes
• Run SGLang using HAMi's nvidia.com/gpu, nvidia.com/gpumem, and nvidia.com/gpucores resources
• Test the SGLang OpenAI-compatible API via port forwarding
• Confirm that nvidia.com/gpumem is enforced inside the SGLang Pod

Lab Overview

HAMi + SGLang Lab Flowchart

Deployment Architecture

HAMi + SGLang Deployment Architecture

Prerequisites

You need to prepare in advance:

• A working Kubernetes cluster
• At least 1 NVIDIA GPU node with more than 25,000 MiB of usable VRAM (this guide uses a single NVIDIA H100 80GB). For an A10, lower every nvidia.com/gpumem request and limit below the memory reported by nvidia-smi.
• kubectl connected to the cluster
• helm 3.x
• GPU nodes with NVIDIA drivers and NVIDIA Container Toolkit / runtime support installed
• The cluster can pull the SGLang image (lmsysorg/sglang) and download model weights (Hugging Face Hub or a mirror)

Kind note: If you use kind with GPU passthrough, install and configure nvidia-container-toolkit inside the kind node and set default_runtime_name = "nvidia" for containerd before installing HAMi. On managed GPU clusters this is usually already handled by the vendor GPU operator / device plugin stack.

Example Cluster State

Below is the verification cluster used for this guide: one kind control-plane node backed by a host NVIDIA H100 80GB HBM3.

kubectl get nodes -o wide

NAME                      STATUS   ROLES           AGE   VERSION   INTERNAL-IP   OS-IMAGE                       CONTAINER-RUNTIME
hami-demo-control-plane   Ready    control-plane   2m    v1.36.1   172.19.0.2    Debian GNU/Linux 13 (trixie)   containerd://2.3.1

Host GPU:

nvidia-smi --query-gpu=index,name,memory.total --format=csv

index, name, memory.total [MiB]
0, NVIDIA H100 80GB HBM3, 81559 MiB

After HAMi is installed (see steps below), each physical GPU is registered as 10 schedulable shares:

kubectl get nodes -o 'custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

NAME                      GPU
hami-demo-control-plane   10

HAMi component status:

kubectl get pods -n kube-system -l app.kubernetes.io/instance=hami -o wide

NAME                              READY   STATUS    NODE
hami-device-plugin-...            2/2     Running   hami-demo-control-plane
hami-scheduler-...                2/2     Running   hami-demo-control-plane

Step 1: Check the GPU Cluster

Confirm that Kubernetes can see the GPU nodes:

kubectl get nodes -o wide
kubectl describe node | grep -A8 -E "Capacity:|Allocatable:" | grep -E "nvidia.com/gpu|cpu:|memory:"

If the cluster already has a vendor NVIDIA device plugin installed, you may already see nvidia.com/gpu. HAMi's device plugin registers the same resource, so the two device plugins must not run on the same GPU nodes. Keep the NVIDIA drivers, Container Toolkit, and runtime installed, but disable or remove the vendor device-plugin DaemonSet before installing HAMi.

First identify how the existing plugin was installed. You will need the original Helm release, Operator configuration, managed add-on setting, or source manifest to restore it later:

kubectl get daemonsets --all-namespaces | grep -E 'nvidia.*device-plugin'

VENDOR_PLUGIN_NAMESPACE=<namespace>
VENDOR_PLUGIN_DAEMONSET=<daemonset-name>
kubectl delete daemonset "${VENDOR_PLUGIN_DAEMONSET}" \
  -n "${VENDOR_PLUGIN_NAMESPACE}"

If a GPU Operator or managed Kubernetes add-on owns the DaemonSet, disable that component through its operator/add-on configuration instead; otherwise its controller may recreate the DaemonSet. Do not uninstall the host driver or NVIDIA container runtime.

HAMi's device plugin matches gpu=on when managed node selectors are enabled. Record whether the label already existed, then label the GPU node:

GPU_NODE=<gpu-node-name>
GPU_LABEL_WAS_PRESENT="$(kubectl get node "${GPU_NODE}" -o go-template='{{if .metadata.labels}}{{if index .metadata.labels "gpu"}}true{{else}}false{{end}}{{else}}false{{end}}')" || {
  echo "Failed to record whether the gpu label exists" >&2
  exit 1
}
GPU_LABEL_BEFORE="$(kubectl get node "${GPU_NODE}" -o jsonpath='{.metadata.labels.gpu}')" || {
  echo "Failed to record the current gpu label" >&2
  exit 1
}
kubectl label node "${GPU_NODE}" gpu=on --overwrite

Step 2: Install HAMi

Add the HAMi Helm repo:

helm repo add hami-charts https://project-hami.github.io/HAMi
helm repo update hami-charts

Create a values file (save as hami-values.yaml):

global:
  managedNodeSelectorEnable: true
  managedNodeSelector:
    gpu: "on"

devicePlugin:
  # This is the chart default, shown explicitly because the lab verifies 10 shares.
  deviceSplitCount: 10

scheduler:
  leaderElect: false

Key configuration details:

Configuration	Description
global.managedNodeSelector.gpu: "on"	Only schedule the HAMi device plugin to GPU nodes labeled gpu=on.
devicePlugin.deviceSplitCount: 10	Register each physical GPU as 10 vGPUs. This matches the current chart default and is explicit because the lab verifies the resulting value.
scheduler.leaderElect: false	Single-replica lab scheduler; avoids extender leader-election waits.

Install HAMi:

helm upgrade --install hami hami-charts/hami \
  -n kube-system \
  -f hami-values.yaml \
  --version 2.9.0

On some ACK Kubernetes 1.36 clusters, the built-in kube-scheduler in HAMi also needs DRA-related RBAC. If the scheduler logs show resource.k8s.io permission errors, apply the helper used in Lab 6:

RBAC_COMMIT="90ac82510bfabe05894dd8037078c59f02a51553"
RBAC_SHA256="e0a77f99422230ccc8958aac0d04694347769279ec26a9d4a5ff729f89efe3d9"
RBAC_MANIFEST="hami-scheduler-dra-rbac.yaml"
curl -fsSLo "${RBAC_MANIFEST}" \
  "https://raw.githubusercontent.com/Project-HAMi/website/${RBAC_COMMIT}/tutorials/labs/hami-vllm/hami-scheduler-dra-rbac.yaml"
printf '%s  %s\n' "${RBAC_SHA256}" "${RBAC_MANIFEST}" | sha256sum -c -
kubectl apply -f "${RBAC_MANIFEST}"

Wait for components to be running:

kubectl rollout status deployment/hami-scheduler -n kube-system
kubectl rollout status daemonset/hami-device-plugin -n kube-system

Expected result:

deployment "hami-scheduler" successfully rolled out
daemon set "hami-device-plugin" successfully rolled out

Step 3: Verify HAMi Resources

kubectl get pods -n kube-system -l app.kubernetes.io/instance=hami -o wide
kubectl get nodes -o 'custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

You should see hami-scheduler and hami-device-plugin Running, and each GPU node advertising nvidia.com/gpu equal to deviceSplitCount (10 in this lab).

If the device plugin is not Ready, check labels and DaemonSet node selector:

kubectl get nodes -L gpu
kubectl get ds hami-device-plugin -n kube-system -o wide
kubectl logs -n kube-system -l app.kubernetes.io/instance=hami -c device-plugin --tail=50

Step 4: Deploy SGLang with HAMi Resources

This lab deploys Qwen3-1.7B with SGLang. The Pod requests one HAMi GPU share, 25000 MiB GPU memory, and 30% GPU cores so multiple inference workloads can still share a large GPU.

Apply the manifests (self-contained):

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: sglang
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang-qwen3-17b
  namespace: sglang
  labels:
    app.kubernetes.io/name: sglang-qwen3-17b
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: sglang-qwen3-17b
  template:
    metadata:
      labels:
        app.kubernetes.io/name: sglang-qwen3-17b
      annotations:
        hami.io/node-scheduler-policy: binpack
        hami.io/gpu-scheduler-policy: binpack
    spec:
      schedulerName: hami-scheduler
      containers:
        - name: sglang
          image: lmsysorg/sglang:v0.5.7
          imagePullPolicy: IfNotPresent
          command:
            - python3
            - -m
            - sglang.launch_server
            - --model-path=Qwen/Qwen3-1.7B
            - --host=0.0.0.0
            - --port=30000
            - --mem-fraction-static=0.7
            - --context-length=8192
            - --attention-backend=triton
          ports:
            - name: http
              containerPort: 30000
          resources:
            requests:
              cpu: "2"
              memory: 8Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "25000"
              nvidia.com/gpucores: "30"
            limits:
              cpu: "8"
              memory: 32Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "25000"
              nvidia.com/gpucores: "30"
          readinessProbe:
            httpGet:
              path: /health
              port: 30000
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 90
          volumeMounts:
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 8Gi
---
apiVersion: v1
kind: Service
metadata:
  name: sglang-qwen3-17b
  namespace: sglang
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: sglang-qwen3-17b
  ports:
    - name: http
      port: 8001
      targetPort: http
EOF

Key points:

Configuration	Description
schedulerName: hami-scheduler	Explicitly delegate scheduling to HAMi.
nvidia.com/gpu: "1"	Request 1 HAMi GPU device share.
nvidia.com/gpumem: "25000"	Software-enforced CUDA memory quota in MiB, also reported through intercepted NVML calls.
nvidia.com/gpucores: "30"	Apply a software compute throttle targeting 30% SM usage.
hami.io/*-scheduler-policy: binpack	Prefer packing workloads onto the same physical GPU.
--attention-backend=triton	Uses the Triton attention backend verified on the H100 test cluster; choose a backend supported by your SGLang version and GPU architecture.
/dev/shm Memory emptyDir	SGLang benefits from a larger shared-memory mount.

Wait for SGLang to become Ready (first start downloads the model and captures CUDA graphs):

kubectl rollout status deployment/sglang-qwen3-17b -n sglang --timeout=30m
kubectl get pods -n sglang -o wide

Example output:

NAME                               READY   STATUS    NODE
sglang-qwen3-17b-6d894b9655-ppxrw  1/1     Running   hami-demo-control-plane

Check HAMi scheduling events:

kubectl describe pod -n sglang -l app.kubernetes.io/name=sglang-qwen3-17b \
  | grep -E "hami-scheduler|Filtering|Binding" -A2

You should see events such as:

Successfully assigned sglang/... to hami-demo-control-plane
FilteringSucceed ... find fit node(hami-demo-control-plane)...
BindingSucceed ... Successfully binding node [hami-demo-control-plane] ...

Step 5: Expose the SGLang Service

For local verification, use port forwarding:

kubectl -n sglang port-forward svc/sglang-qwen3-17b 8001:8001

In another terminal:

curl http://127.0.0.1:8001/v1/models

On cloud clusters you can also create a LoadBalancer Service or Ingress in front of sglang-qwen3-17b. Verify with port-forward first, then wire public exposure.

Step 6: Test Inference

List models:

curl -s http://127.0.0.1:8001/v1/models | python3 -m json.tool

Example output from the verification cluster:

{
  "object": "list",
  "data": [
    {
      "id": "Qwen/Qwen3-1.7B",
      "object": "model",
      "owned_by": "sglang",
      "max_model_len": 8192
    }
  ]
}

Send a chat completion request:

curl -s http://127.0.0.1:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-1.7B",
    "messages": [
      {"role": "user", "content": "Explain in one sentence how HAMi and SGLang work together."}
    ],
    "max_tokens": 128,
    "temperature": 0.2,
    "chat_template_kwargs": {"enable_thinking": false}
  }' | python3 -m json.tool

If the response contains choices[0].message.content, the SGLang inference service is working.

Qwen3 models may emit a thinking/reasoning channel depending on the chat template. Passing "chat_template_kwargs": {"enable_thinking": false} keeps the demo answer concise.

Step 7: Check GPU Allocation

Confirm the Pod used HAMi scheduling and the expected resource limits:

POD=$(kubectl get pod -n sglang -l app.kubernetes.io/name=sglang-qwen3-17b -o jsonpath='{.items[0].metadata.name}')
kubectl get pod -n sglang ${POD} \
  -o jsonpath='{.spec.schedulerName}{"\n"}{.spec.containers[0].resources.limits}{"\n"}'

Expected:

hami-scheduler
... nvidia.com/gpu:1 nvidia.com/gpumem:25000 nvidia.com/gpucores:30 ...

Check environment variables injected by HAMi:

kubectl exec -n sglang ${POD} -- env | grep -E 'CUDA_DEVICE|NVIDIA_VISIBLE'

Example from the verification cluster:

NVIDIA_VISIBLE_DEVICES=GPU-04b76a6c-da10-342f-e9f5-5f5684eacb86
CUDA_DEVICE_MEMORY_LIMIT_0=25000m
CUDA_DEVICE_SM_LIMIT=30

Finally, check nvidia-smi inside the container:

kubectl exec -n sglang ${POD} -- nvidia-smi

When gpumem is in effect, the total GPU memory visible inside the container is close to 25000 MiB, not the full physical card (81559 MiB on the H100 verification host):

| GPU  Name                 ... | Memory-Usage          |
| NVIDIA H100 80GB HBM3     ... | 18213MiB / 25000MiB   |

On the host, the same card still reports the full capacity:

nvidia-smi --query-gpu=memory.total,memory.used --format=csv

memory.total [MiB], memory.used [MiB]
81559 MiB, 18560 MiB

That contrast confirms that HAMi's NVML interception exposes the configured quota. It is supporting evidence, but it does not by itself prove that an over-quota CUDA allocation is rejected.

Optionally verify CUDA-level enforcement from a separate process. Set GPU_QUOTA_MIB to the nvidia.com/gpumem value used in your manifest. The probe requests 1024 MiB more than that quota, so it remains valid if you lower the quota for another GPU:

GPU_QUOTA_MIB=25000
OVER_QUOTA_MIB=$((GPU_QUOTA_MIB + 1024))
kubectl exec -i -n sglang ${POD} -- \
  env OVER_QUOTA_MIB="${OVER_QUOTA_MIB}" python3 - <<'PY'
import os
import torch

allocation_mib = int(os.environ["OVER_QUOTA_MIB"])
try:
    torch.empty(allocation_mib * 1024**2 // 4, dtype=torch.float32, device="cuda")
except RuntimeError as exc:
    if "out of memory" not in str(exc).lower():
        raise
    print("PASS: over-quota CUDA allocation returned out of memory")
else:
    raise SystemExit("FAIL: over-quota CUDA allocation unexpectedly succeeded")
PY

Expected output includes PASS: over-quota CUDA allocation returned out of memory. The failed request should not retain memory or change the serving process. Confirm that the live server remains healthy:

curl --fail --silent http://127.0.0.1:8001/health
curl --fail --silent http://127.0.0.1:8001/v1/models | python3 -m json.tool

HAMi enforces the memory quota by intercepting CUDA allocation calls; this is software enforcement, not a hardware partition such as MIG. Compute limiting is likewise a software throttle.

Troubleshooting

Symptom	What to Check
hami-device-plugin CrashLoop / FailedPostStartHook	NVIDIA Container Toolkit must inject driver libs into Pods. On kind, configure the nvidia runtime inside the node.
hami-device-plugin not Ready	Node missing gpu=on, or node selector mismatch.
SGLang Pod Pending	kubectl describe pod for HAMi filter/bind events; confirm gpumem does not exceed physical GPU memory.
Image pull fails / disk full	lmsysorg/sglang images are large. Prune unused images or kind load docker-image from a pre-pulled host image.
SGLang starts slowly	First-time Hugging Face download + CUDA graph capture can take several minutes. Watch Pod logs.
/v1/models empty / connection refused	Wait until readiness probe passes; use kubectl logs and kubectl port-forward to the Pod port 30000 if the Service is wrong.
In-pod nvidia-smi still shows full GPU memory	Pod is not on the HAMi memory-limit path. Recheck schedulerName, resource limits, and HAMi webhook/scheduler events.

Common troubleshooting commands:

kubectl get pods -A -o wide
kubectl describe pod -n sglang -l app.kubernetes.io/name=sglang-qwen3-17b
kubectl logs -n sglang -l app.kubernetes.io/name=sglang-qwen3-17b --tail=100
kubectl logs -n kube-system deploy/hami-scheduler --tail=100
kubectl get ds hami-device-plugin -n kube-system -o wide

Cleanup

kubectl delete namespace sglang --ignore-not-found

If this cluster is only used for this lab, you can also uninstall HAMi:

helm uninstall hami -n kube-system

Restore the vendor device plugin after HAMi is removed by reversing the method used to disable it. For a self-managed plugin installed from a manifest, apply the original version-pinned source manifest, not a live kubectl get -o yaml export:

kubectl apply -f <original-version-pinned-device-plugin-manifest>

For a Helm release, GPU Operator, or managed add-on, restore it through that same management mechanism. Confirm that exactly one device-plugin DaemonSet owns nvidia.com/gpu.

Restore the original gpu label state. Run this in the same shell that recorded GPU_LABEL_WAS_PRESENT and GPU_LABEL_BEFORE:

if [ "${GPU_LABEL_WAS_PRESENT}" = true ]; then
  kubectl label node "${GPU_NODE}" "gpu=${GPU_LABEL_BEFORE}" --overwrite
else
  kubectl label node "${GPU_NODE}" gpu-
fi

Verification Results

Claim	Evidence
HAMi has taken over GPU scheduling	SGLang Pod uses schedulerName: hami-scheduler and requests nvidia.com/gpu, nvidia.com/gpumem, nvidia.com/gpucores.
GPU node runs HAMi device plugin	hami-device-plugin is Ready and advertises nvidia.com/gpu=10.
SGLang runs on HAMi resources	Pod Ready; HAMi injects CUDA_DEVICE_MEMORY_LIMIT_0=25000m and CUDA_DEVICE_SM_LIMIT=30.
Memory quota is visible in-container	In-pod nvidia-smi shows ... / 25000MiB while host still shows 81559 MiB.
Over-quota allocation is rejected	A PyTorch CUDA allocation 1024 MiB above the configured quota returns an out-of-memory error.
Inference service is accessible	/v1/models returns Qwen/Qwen3-1.7B; chat endpoint returns content.

Next Steps

• Increase replicas and observe how HAMi packs multiple SGLang Pods with binpack.
• Lower nvidia.com/gpumem / nvidia.com/gpucores further and co-locate another small workload on the same GPU.
• Deliver the model from an OCI registry instead of Hugging Face: the companion KitOps ModelKit lab PR replaces the runtime download with a kitops-init initContainer.
• For memory isolation and small-slice sharing patterns, see Lab 3: GPU Partitioning.