NVIDIA H100 80GB HBM3 · median of 10 runs after 3 warm-ups · measured 2026-09-28T17:57:04+00:00
Linnet 0.1.0, PyTorch 2.14.0, JAX 0.11.2, transformers 5.17.0, diffusers 0.40.0, ONNX Runtime 1.30.0, Python 3.12.3, driver 580.126.09
Linnet against the stack it replaces
Each side in its fastest configuration, on the same GPU and checkpoint. A speed-up is how many times the other's speed; an export is measured against the original checkpoint in the same engine.
vs the reference, in PyTorch2.61× fasterlatency
- Linnet, CUDA graphs0.32 mstransformers, compiled0.84 ms
vs KerasHub, in JAX8.91× fasterlatency
- Linnet, XLA (generated JAX)0.44 msKerasHub3.96 ms
vs torch.onnx, on ONNX Runtime0.84× slowerlatency
- Linnet, ONNX Runtime f321.21 mstorch.onnx export1.02 ms
vs torch.onnx, behind Triton1.00× slowerlatency
- Triton, Linnet's ONNX2.15 msTriton, torch.onnx export2.15 ms
Each row also carries its distance from reference: the largest absolute difference between its output and transformers (eager)'s on the same input. bf16 outputs differ by rounding (about 0.1 on logits near 16), so a small number is expected; it is there so that a fast wrong answer cannot look like a win.
Latency
ms, lower is better; the best bar is red.
Throughput
/s, higher is better; the best bar is red.
Load time
s, lower is better; the best bar is red.
Peak GPU memory
What the driver reports the process holding at its peak, in GiB; lower is better; the best bar is red.
Every number
| Method | Latency | Throughput | Load time | Peak GPU memory | Against the stack it replaces | Distance from reference | Notes |
|---|---|---|---|---|---|---|---|
| PyTorch | |||||||
| transformers | 1.91 ms | 26,734/s | 2.19 s | 0.86 GiB | 0 (the reference) | unpadded batches of exactly 128 tokens (this card takes no padding mask) | |
| transformers, compiled | 0.84 ms | 50,395/s | 1.49 s | 0.87 GiB | 0.0469 | unpadded batches of exactly 128 tokens (this card takes no padding mask) | |
| sentence-transformers | 3.21 ms | 6,431/s | 2.44 s | 0.80 GiB | not compared | natural text tokenizing to 65 tokens, identical across the batch (no padding wasted); not the fixed-length random-token workload the other rows use, since this is the library's own string-in interface | |
| Linnet, generated PyTorch | 1.03 ms | 49,772/s | 0.62 s | 0.84 GiB | 0.82× transformers, compiled | 0.0625 | unpadded batches of exactly 128 tokens; bf16 on cuda |
| Linnet, inductor | 0.79 ms | 70,709/s | 0.63 s | 0.82 GiB | 1.07× transformers, compiled | 0.0625 | unpadded batches of exactly 128 tokens; bf16 on cuda |
| Linnet, CUDA graphs | 0.32 ms | 74,093/s | 0.57 s | 0.95 GiB | 2.61× transformers, compiled | 0.0625 | unpadded batches of exactly 128 tokens; bf16 on cuda |
| JAX | |||||||
| KerasHub | 3.96 ms | 11,661/s | 6.72 s | 0.25 GiB | 0.0469 | bf16, under jax.jit; unpadded batches of exactly 128 tokens; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions | |
| Linnet, XLA (StableHLO) | 0.52 ms | 38,961/s | 0.33 s | 0.34 GiB | 7.58× KerasHub | 0.0469 | unpadded batches of exactly 128 tokens; bf16 on cuda; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions |
| Linnet, XLA (generated JAX) | 0.44 ms | 39,720/s | 0.29 s | 0.31 GiB | 8.91× KerasHub | 0.0469 | unpadded batches of exactly 128 tokens; bf16 on cuda; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions |
| ONNX Runtime and TensorRT | |||||||
| torch.onnx export | 1.02 ms | 19,156/s | 12.4 s | 1.45 GiB | 0.0629 | the reference model's own ONNX export, f32 like Linnet's; unpadded batches of exactly 128 tokens | |
| Linnet, ONNX Runtime f32 | 1.21 ms | 20,250/s | 0.29 s | 1.46 GiB | 0.84× torch.onnx export | 0.0629 | f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length |
| Linnet, ONNX Runtime f16 | 1.05 ms | 34,378/s | 0.29 s | 1.11 GiB | 0.0645 | f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length | |
| Linnet, ONNX Runtime bf16 | 1.05 ms | 34,780/s | 0.28 s | 1.11 GiB | 0.0352 | bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length | |
| Linnet, TensorRT f32 | 0.35 ms | 34,138/s | 0.28 s | 2.50 GiB | 0.0638 | f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length | |
| Linnet, TensorRT f16 | 0.23 ms | 80,632/s | 0.28 s | 2.41 GiB | 0.0586 | f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length | |
| Linnet, TensorRT bf16 | 0.24 ms | 74,959/s | 0.27 s | 2.41 GiB | 0.0469 | bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length | |
| Triton Inference Server | |||||||
| Triton, torch.onnx export | 2.15 ms | 1,107/s | 6.18 s | 1.52 GiB | 0.0629 | over HTTP; f32; throughput with 4 requests of batch 64 in flight | |
| Triton, Linnet's ONNX | 2.15 ms | 1,111/s | 10.0 s | 1.55 GiB | 1.00× Triton, torch.onnx export | 0.0629 | over HTTP; f32; throughput with 4 requests of batch 64 in flight |
| Triton, Linnet Python backend | 1.39 ms | 1,100/s | 6.03 s | 2.38 GiB | 0.0625 | over HTTP; bf16, CUDA graphs; throughput with 4 requests of batch 64 in flight | |