Skip to content
Gradient Notes logoGradient Notes
Infrastructure

GPU scheduling for small teams

Four GPUs, nine engineers and a lot of idle time. How a simple queue and a few house rules doubled our utilisation.

Daniel Reyes2 min read

Close-up of a graphics processor mounted on a circuit board
Nick Stathas · CC BY-SA 4.0

When we bought our first four GPUs, we assigned them the obvious way: one per person who asked first. Three months later, monitoring showed average utilisation of 23%. Meanwhile, people waited days for a free card. The hardware was not the bottleneck; the allocation was.

Measure before you schedule

We exported per-GPU utilisation and memory every 30 seconds and looked at a week of data. The pattern was clear: notebooks held GPUs for days while running for minutes. Long training runs ran at night but nobody could start them reliably because a card was "reserved".

A queue, not a calendar

We replaced the spreadsheet of reservations with a job queue. We chose a lightweight scheduler rather than a full Kubernetes stack, because for four to sixteen GPUs the operational overhead of a cluster manager is hard to justify.

The rules we settled on:

  • Interactive sessions are capped at four hours and one GPU, and are pre-empted after 30 minutes idle.
  • Batch jobs go through the queue with a declared GPU count and time limit.
  • Long runs checkpoint every 30 minutes so they can be pre-empted and resumed.
  • Nights and weekends favour batch, weekdays favour interactive work.
submit.shbash
# Submit a fine-tuning run: 2 GPUs, max 10 hours, resumable
queue submit \
  --gpus 2 \
  --time 10h \
  --checkpoint-dir /shared/ckpt/$USER/run-42 \
  -- python train.py --config configs/lora.yaml --resume auto

Make idle time visible

We post a daily summary to the team channel: utilisation per GPU, queue wait times, and the top three idle reservations. Nobody likes appearing on that list twice. Social pressure turned out to be the cheapest scheduler feature.

Results

Utilisation went from 23% to 58% in the first month, and median queue wait for a single GPU dropped below 20 minutes. We postponed buying more hardware by two quarters.

Keep reading