Skip to main content
Version: Next

Troubleshoot Tecorigin scheduling

The node has the resource but never appears in the scheduler​

A Tecorigin node is only registered when the register annotation parses and at least one card is healthy. Check the annotation first:

kubectl get node <node-name> -o jsonpath='{.metadata.annotations.hami\.io/node-tecorigin-register}'
SymptomCauseFix
Annotation missingThe device plugin is not running, or advertises a different nameDeploy the vendor plugin and confirm tecorigin.com/gpucore is in the node's allocatable resources
want 9 fields, got 7The plugin omits index and modePublish nine fields per card; the 7-field form leaves every card at index 0 and is rejected
invalid core count N for card "..."count is 0 or above 4Report the real core count of a 1–4-core card
invalid memory ... Devmem is MiB, not bytesThe plugin wrote a byte countWrite MiB
no healthy tecorigin card found on nodeEvery card reports health=falseFix the card, then update the annotation so the health check re-reads it

An annotation that is present but produces no registration also leaves Pods pending with node unregistered in the scheduler log.

Note: Both node unregistered and no healthy tecorigin card found on node are logged at V(5). The HAMi chart defaults to -v=4, so these messages are not visible by default. You can set the scheduler log level to -v=5 to view them, or inspect the scheduling rejection reason directly. The register annotation and hami.io/node-tecorigin-cores are fingerprinted on each health check, so correcting either one re-reads the card list; no scheduler restart is needed.

The Pod stays pending​

HAMi reports the reason in the Pod's allocation reason, as <failed>/<total> <Reason>. The Tecorigin backend emits four:

ReasonMeaning
CardNotHealthThe card is marked unhealthy in the register annotation
CardTimeSlicingExhaustedThe card already holds as many containers as it has cores
CardUuidMismatchtecorigin.com/use-gpuuuid or tecorigin.com/nouse-gpuuuid excludes this card
CardInsufficientCoreNot enough free cores, or serving the request would need more cards than the request's own minimum

For CardInsufficientCore, look at the free-core count in the scheduler log:

kubectl logs -n {Release Namespace} -l app.kubernetes.io/component=hami-scheduler -c vgpu-scheduler-extender --tail=200 | grep "tecorigin fit failed"

Note: When all GPUs on the node have zero free cores, the scheduler returns CardInsufficientCore without logging tecorigin fit failed. In this case the grep above will find no matches; you need to check the card allocation status directly.

The log prints want, the aggregated freeCores from eligible candidate cards, candidate count, and maxCards. freeCores sums only cards that pass health, UUID, container-count and free-core filters. maxCards is a card count, not a core count: it equals ceil(want / core count of the largest eligible card), the minimal number of cards required for this request. Any split requiring more cards will be rejected.

To tell the failure causes apart: first compare freeCores against want. If freeCores < want, the pool of eligible candidate cards does not have enough total free cores. If freeCores >= want, then compare want with the largest card’s core count.

  • When want ≤ largest card core count: maxCards = 1, failure means no single eligible card has enough free cores.
  • When want > largest card core count: the request must span multiple cards; the node is rejected if the required split exceeds the minimal-card limit.

Verify what a running container received​

kubectl get pod <pod-name> -o jsonpath='{.metadata.annotations.hami\.io/tecorigin-devices-allocated}'

Each entry is <card-serial>,<type>,<memory-MiB>,<core-mask>. The mask tells you which cores of that card were allocated:

MaskCores on a 4-core card
1core 0
5cores 0 and 2
15all four cores

A mask with gaps is expected: the runtime accepts an arbitrary set of cores, so HAMi hands out the lowest free ones instead of waiting for a contiguous run.

Trace a core index back to a card​

TECO_VISIBLE_DEVICES inside the container holds global core indexes. To map an index to a card, read the optional core-id map the plugin may publish:

kubectl get node <node-name> -o jsonpath='{.metadata.annotations.hami\.io/node-tecorigin-cores}'
{"3CP100241200184":[0,1,2,3],"3CP100241200185":[4,5,6,7]}

Index 5 belongs to card 3CP100241200185, local core 1. The map never affects scheduling; it exists only for this kind of lookup. Without it, the index range still identifies the card when cards report equal core counts and are ordered by serial.

Requests do not produce the expected split​

Placement is deterministic: fewest cards first, then the set whose free cores add up most tightly to the request, then the lowest-indexed cards of each size. A 6-core request therefore lands on two cards with three free cores each rather than on a four-core and a three-core card, because 3 + 3 wastes nothing. The cores are then taken from the roomiest selected card first, so a set with four and three free cores gives 4 + 2 and leaves one core on the second card.

If a whole card was expected, confirm the request matches the card's core count: 4 on 4-core cards, 3 on 3-core cards. A 4 on a node of 3-core cards is split and cannot be exclusive.

CNCFHAMi is a CNCF Incubating project