Skip to main content
Version: Next

Performance and fault isolation of sharing methods

When multiple inference instances share a GPU, the comparison covers inference performance and the effect of a failure on other instances. The baselines run one or two vLLM instances directly. The other configurations are HAMi and NVIDIA MPS-based sharing, measured on throughput, response latency, and fault isolation.

The performance tests run Qwen3-4B on A100 GPUs. The fault tests use a CUDA test program. Fixed-cache performance tests use legacy MPS v2, standard performance tests use GPU Fractioning (which uses MPS v3), and fault tests use MPS v3 directly.

Test objectives​

  • At the same total concurrency per GPU, compare throughput and latency with one and two inference instances. Then compare two instances with HAMi or MPS enabled.
  • Check whether healthy processes on the same GPU and on another GPU continue to compute correctly when a CUDA process fails.

In the performance tables, "native" means running vLLM directly, with HAMi and MPS disabled. The single-instance configuration runs one inference instance per GPU. The two-instance configuration runs two independent CUDA processes per GPU. The native pair provides a baseline with the same instance count as the sharing configurations.

Each configuration handles the same total concurrency per GPU. For example, at total concurrency 8, a single instance handles 8 requests, while each instance in a pair handles 4.

Terms​

TermMeaning and use in this report
CUDA kernelA function that runs on the GPU. One inference operation calls multiple kernels.
Graph / CUDA GraphRecords a set of GPU operations and their dependencies for repeated execution, reducing the CPU overhead of submitting operations individually. In this report, Graph means vLLM's default execution mode, which can use CUDA Graphs.
EagerSubmits computation as execution proceeds. These tests enable it with --enforce-eager, which disables CUDA Graphs and vLLM's compilation integration.
MPS (Multi-Process Service)NVIDIA's service for sharing a GPU across CUDA processes through a server. Legacy MPS v2 and MPS v3 refer to different control interface protocols.
GPU FractioningA GPU sharing component used with KAI Scheduler. These tests use the MPS v3 sharing service in release 0.1.8.
KV cache and prefix cachingThe KV cache stores attention keys and values that the model has already computed. Prefix caching reuses cached keys and values for prompt prefixes shared across requests.
TTFT (Time to First Token)The time from sending a request to receiving its first token. Lower is better.
TPOT (Time per Output Token)The average time a request takes to generate each subsequent token. Lower is better.
p50 and p99At p50, 50% of requests finish within the reported time. At p99, 99% do. This report averages the same percentile across instances.

Test results​

  • In the fixed-cache Graph test, native single-instance throughput is 628.57, native-pair throughput is 335.51, HAMi 50% pair throughput is 335.87 (+0.11% relative to the native pair), and legacy MPS v2 throughput is 449.36 tokens/s/GPU.
  • Across the five concurrency levels in the standard Graph test, throughput differs from the native pair by −0.66% to +2.19% for HAMi 50% and −0.10% to +0.20% for HAMi unlimited. GPU Fractioning is 25.14% to 37.60% higher.
  • HAMi 50% throughput is 4.46% to 7.94% below the native pair across the five concurrency levels in the standard Eager test, and 2.22% below it in the fixed-cache Eager test.
  • Invalid GPU memory writes, trap instructions, and forced process termination during computation are each tested twice. With native processes or HAMi, healthy processes on the same GPU keep producing correct output. With either the shared or static MPS v3 configuration, healthy processes on the same GPU are affected.

In the fixed-cache test, native single, native pair, and legacy MPS are on set A, and HAMi 50% is on set C. In the standard test, the native pair is on set B, HAMi 50% is on set C, and HAMi unlimited and GPU Fractioning are on set A. A percentage across sets includes both the card difference and the method difference. See Test conditions.

HAMi 2.10 applies the compute limit on kernel launch. After Eager turns Graph off, every kernel launch runs this check, which then decides whether to rate-limit according to the compute quota and whether compute limiting is enabled. Graph replay calls cuGraphLaunch. This version forwards the call directly to the CUDA driver, and the driver executes the recorded compute Graph, so Graph replay bypasses the HAMi rate-limiting path above.

Inference performance​

"HAMi 50%" means a 50% compute quota and a 20 GiB memory quota per instance. The compute quotas configured for the two instances on the same GPU add up to 100%.

Fixed-cache comparison: 6 GiB per instance​

Total concurrency is 8 per GPU. Each instance has a fixed 6 GiB KV cache, and prefix caching is disabled. Maximum model length is 4096, and maximum sequence count is 32. HAMi has a 50% compute quota per instance, and MPS uses legacy v2. Each configuration is measured three times in one deployment.

Graph and Eager throughput in the fixed-cache test. Bars are the mean of three trials, and the line on each bar is one sample standard deviation.

Bars are the mean throughput of three trials. The line on each bar runs from one sample standard deviation below the mean to one above. A longer line means the three trials differ more. Throughput is measured in output tokens/s/GPU. Higher is better.

Throughput is the mean ± sample standard deviation across three trials. Latency is in ms.

ModeConfigurationOutput tokens/s/GPUTTFT p50TTFT p99TPOT p50TPOT p99
GraphNative single628.57 ± 0.30335.46678.1911.4212.02
GraphNative pair335.51 ± 0.02563.53738.6121.7122.40
GraphHAMi pair (50%)335.87 ± 0.06558.37729.4521.7122.39
GraphLegacy MPS pair449.36 ± 13.92413.50538.9816.8217.40
EagerNative single193.95 ± 1.17331.43722.7040.0340.82
EagerNative pair185.80 ± 0.73440.87659.3841.5242.73
EagerHAMi pair (50%)181.67 ± 1.32426.37671.9742.5443.68
EagerLegacy MPS pair188.51 ± 0.73311.23500.7041.3742.55

Throughput and latency calculations​

The figure and performance tables use the sum of instance throughput: calculate output tokens per second for each instance, sum the rates, and divide by the number of GPUs.

The whole-test calculation divides the total output tokens from all instances by the time from the earliest start to the latest finish, then by the number of GPUs. The two calculations differ when instance measurement intervals overlap only partially. Legacy MPS results are:

ModeSum of instance throughput (tokens/s/GPU)Whole-test throughput (tokens/s/GPU)
Graph449.36399.28
Eager188.51187.29

With the whole-test calculation, Graph throughput is 334.50 for the native pair and 334.97 tokens/s/GPU for HAMi 50%. Use the same calculation when comparing configurations.

For latency, average p50 and p99 separately across instances, then average each result across the three trials.

Fixed-cache data · Reference data without compute limiting · HAMi 50% results by trial.

Standard test: automatic cache allocation​

The tests run vLLM through Dynamo and use the official run-sweep.sh for load generation. They use automatic KV cache allocation and default prefix caching, with one trial per configuration.

At concurrency 2 and 4 per GPU, the prefix cache read rate is 99.61% for the native single instance and 0% for every two-instance configuration. At concurrency 8, the rates are 49.81% and 0%, respectively. Rates for all five concurrency levels are in the standard-test CSV. The single-instance and two-instance gap includes this difference in cache reuse. The fixed-cache test has prefix caching off.

Total concurrency per GPU is 2, 4, 8, 16, and 32. All configurations except native single run two instances per GPU. Both HAMi configurations have a 20 GiB memory quota per instance. Throughput is measured in output tokens/s/GPU. Bold marks the highest value at each concurrency level. Parentheses show the difference relative to the native pair.

Graph and Eager throughput in the standard test. Each line is one configuration, and each point is one measurement of that configuration at one concurrency level.

Each line is one configuration, and each point is one trial measured for that configuration at one concurrency level, including GPU Fractioning and HAMi unlimited. Higher is better.

Graph (default)​

Concurrency/GPUNative singleNative pairHAMi 50%HAMi unlimitedGPU Fractioning
2225.5097.6699.80 (+2.19%)97.61 (−0.05%)134.37 (+37.60%)
4426.82186.77186.92 (+0.08%)186.76 (0.00%)255.64 (+36.88%)
8693.67335.67336.52 (+0.25%)335.62 (−0.02%)438.59 (+30.66%)
161038.10565.45561.73 (−0.66%)564.90 (−0.10%)718.28 (+27.03%)
321117.88817.12817.89 (+0.10%)818.71 (+0.20%)1022.54 (+25.14%)

Eager​

Concurrency/GPUNative singleNative pairHAMi 50%HAMi unlimitedGPU Fractioning
251.2651.8447.76 (−7.88%)48.11 (−7.20%)50.34 (−2.91%)
4103.10101.9993.89 (−7.94%)94.42 (−7.42%)99.66 (−2.29%)
8200.28194.53180.49 (−7.21%)182.01 (−6.44%)189.54 (−2.57%)
16377.41361.61341.28 (−5.62%)343.67 (−4.96%)361.11 (−0.14%)
32653.43630.84602.68 (−4.46%)606.86 (−3.80%)636.66 (+0.92%)

Throughput differences across the five concurrency levels:

  • HAMi 50% (Graph): −0.66% to +2.19% relative to the native pair.
  • GPU Fractioning: 25.14% to 37.60% above the native pair in Graph, and −2.91% to +0.92% in Eager.
  • HAMi (Eager): 4.46% to 7.94% below the native pair with the 50% quota, and 3.80% to 7.42% below it without compute limiting.

Response latency: concurrency 8 per GPU​

The table shows the mean of the per-instance p99 values, in ms.

ConfigurationGraph first tokenGraph subsequent tokensEager first tokenEager subsequent tokens
Native single665.7611.95730.7140.39
Native pair730.3522.44669.5540.83
HAMi 50%714.9922.41588.0444.52
HAMi unlimited739.0922.43599.8943.90
GPU Fractioning427.7417.48595.7942.67

Relative to the native pair, HAMi 50% has lower first-token p99 latency and higher subsequent-token p99 latency in Eager mode.

Fault isolation results​

A faulting process and a healthy process run on the same GPU, with a healthy control process on another GPU. Healthy processes replay a CUDA Graph and check their output. Each case runs twice, with observation for 3 seconds before the fault and at least 6 seconds afterward.

The fault tests use HAMi without compute limiting, with a 20 GiB memory quota per instance. MPS v3 is tested in shared and static-partition configurations. The shared configuration enables memacct and context-share.

2/2 means the healthy process on the same GPU produced correct results in both trials. 0/2 means it was affected in both trials.

Injected eventNative processesHAMi processesMPS v3 sharedMPS v3 tested static configuration
Excess memory allocation2/22/22/22/2
Invalid GPU memory write2/22/20/20/2
GPU trap instruction2/22/20/20/2
SIGKILL during computation2/22/20/20/2
Safe MPS client terminationN/AN/A2/22/2

All 36 control trials on the other GPU pass. The excess-memory case requests twice the visible capacity, and all allocation requests are rejected. Fault counts, error codes, and maximum sample gaps (CSV).

The two clients in the static configuration each report 54 streaming multiprocessors (SMs), with different partition IDs. See MPS fault handling and static partitioning for the mechanisms.

Test conditions​

Hardware: 2× A100-SXM4-40GB. Software: Ubuntu 24.04.5, NVIDIA driver 615.71.10, Dynamo 1.6.1, vLLM 0.31.0, and AIPerf 0.13.0. The model is Qwen3-4B in BF16. Each instance uses one GPU. Input length is 2,048 tokens, and output length is 256 tokens.

Sets A, B, and C are each two A100-SXM4-40GB GPUs, and each letter is a different physical pair. The same letter is the same pair. Cards of the same model differ in performance, so a percentage taken across sets includes both that card difference and the sharing-method difference. The set and collection date for each test are below.

TestConfigurationGPU setCollection date (UTC)
Fixed cacheNative singleA2026-11-03
Fixed cacheNative pairA2026-11-03
Fixed cacheLegacy MPSA2026-11-03
Fixed cacheHAMi unlimitedA2026-11-03
Fixed cacheHAMi 50%C2026-11-05
StandardNative singleA2026-11-02
StandardNative pairB2026-11-04
StandardHAMi 50%C2026-11-05
StandardHAMi unlimitedA2026-11-02
StandardGPU FractioningA2026-11-02
FaultNative, HAMi unlimited, MPS v3 shared, MPS v3 staticB2026-11-04

The standard-test deployment configurations are below. Legacy MPS v2 settings for the fixed-cache tests and MPS v3 settings for the fault tests are in their respective sections.

DeploymentInstances/GPUMemory configurationExecution configuration
Native single1Full-device memory visibilityIndependent CUDA process
Native pair2Engine memory utilization 0.45 per instanceIndependent CUDA processes without HAMi or MPS
HAMi 2.10.0 pair220 GiB quota per instance50% compute quota per instance
HAMi 2.10.0 pair (unlimited)220 GiB quota per instanceCompute limiting disabled
KAI 0.18.6 + GPU Fractioning 0.1.8 pair220 GiB quota per instanceMPS v3 sm-sharing without static SM partitioning

These tests use the shared MPS service from the GPU Fractioning v0.1.8 release. The NVIDIA report uses the KAI-Scheduler and GPU Fractioning revisions maintained by nv-kmcgill53. Both branches are named kmcgill/sm-sharing-main. Each container gets its own MPS namespace, with compute and memory quotas.

Reproduction scripts​

Run the tests · Source code · All results

Raw data: Standard tests CSV · No compute limit CSV.

Conclusion​

Whether a sharing method is worth recommending depends on the Pareto frontier of NVIDIA containerized GPU virtualization. That frontier has not been measured, and this report DOES NOT recommend any method.

CNCFHAMi is a CNCF Incubating project