NVIDIA H100 80GB HBM3 · median of 10 runs after 3 warm-ups · measured 2026-09-28T20:13:23+00:00
Linnet 0.1.0, PyTorch 2.14.0, JAX 0.11.2, transformers 5.17.0, diffusers 0.40.0, ONNX Runtime 1.30.0, Python 3.12.3, driver 580.126.09
Linnet against the stack it replaces
Each side in its fastest configuration, on the same GPU and checkpoint. A speed-up is how many times the other's speed; an export is measured against the original checkpoint in the same engine.
- Linnet, CUDA graphs10.4 msdiffusers, compiled10.4 ms
Each row also carries its distance from reference: the largest absolute difference between its output and diffusers (eager)'s on the same input. bf16 outputs differ by rounding (about 0.1 on logits near 16), so a small number is expected; it is there so that a fast wrong answer cannot look like a win.
Latency
ms, lower is better; the best bar is red.
Throughput
/s, higher is better; the best bar is red.
Load time
s, lower is better; the best bar is red.
Peak GPU memory
What the driver reports the process holding at its peak, in GiB; lower is better; the best bar is red.
Every number
| Method | Latency | Throughput | Load time | Peak GPU memory | Against the stack it replaces | Distance from reference | Notes |
|---|---|---|---|---|---|---|---|
| PyTorch | |||||||
| diffusers | 22.0 ms | 66.2/s | 6.71 s | 8.95 GiB | 0 (the reference) | 64x64 latent (512-pixel image), bf16; throughput at batch 8 | |
| diffusers, compiled | 10.4 ms | 110/s | 4.89 s | 7.40 GiB | 0.0332 | 64x64 latent (512-pixel image), bf16; throughput at batch 8 | |
| Linnet, generated PyTorch | 27.5 ms | 39.4/s | 0.87 s | 12.2 GiB | 0.38× diffusers, compiled | 0.0313 | 64x64 latent (512-pixel image), bf16; throughput at batch 8 |
| Linnet, inductor | 10.9 ms | 111/s | 0.47 s | 4.82 GiB | 0.95× diffusers, compiled | 0.0332 | 64x64 latent (512-pixel image), bf16; throughput at batch 8 |
| Linnet, CUDA graphs | 10.4 ms | 112/s | 0.45 s | 5.53 GiB | 1.00× diffusers, compiled | 0.0332 | 64x64 latent (512-pixel image), bf16; throughput at batch 8 |
| JAX | |||||||
| Linnet, XLA (StableHLO) | 7.47 ms | 175/s | 0.66 s | 3.30 GiB | 0.0449 | 64x64 latent (512-pixel image), bf16; throughput at batch 8; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions | |
| Linnet, XLA (generated JAX) | 7.41 ms | 173/s | 0.50 s | 3.30 GiB | 0.0527 | 64x64 latent (512-pixel image), bf16; throughput at batch 8; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions | |
| ONNX Runtime and TensorRT | |||||||
| Linnet, ONNX Runtime f32 | 24.7 ms | 49.1/s | 0.27 s | 26.3 GiB | 0.0397 | f32; ONNX Runtime's CUDA execution provider; first calls build the sessions | |
| Linnet, ONNX Runtime f16 | 21.0 ms | 69.0/s | 0.24 s | 13.0 GiB | 0.038 | f16; ONNX Runtime's CUDA execution provider; first calls build the sessions | |
| Linnet, ONNX Runtime bf16 | 28.1 ms | 42.1/s | 0.24 s | 21.2 GiB | 0.0566 | bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions | |
| Linnet, TensorRT f32 | 13.7 ms | 71.2/s | 0.24 s | 11.5 GiB | 0.0398 | f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions | |
| Linnet, TensorRT f16 | 5.41 ms | 199/s | 0.24 s | 8.58 GiB | 0.0432 | f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions | |
| Linnet, TensorRT bf16 | 12.5 ms | 78.1/s | 0.24 s | 10.5 GiB | 0.0352 | bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions | |