NVIDIA H100 80GB HBM3 · median of 10 runs after 3 warm-ups · measured 2026-09-28T20:02:51+00:00

Linnet 0.1.0, PyTorch 2.14.0, JAX 0.11.2, transformers 5.17.0, diffusers 0.40.0, ONNX Runtime 1.30.0, Python 3.12.3, driver 580.126.09

Linnet against the stack it replaces

Each side in its fastest configuration, on the same GPU and checkpoint. A speed-up is how many times the other's speed; an export is measured against the original checkpoint in the same engine.

vs the reference, in PyTorch1.04× fasterencode

• Linnet, CUDA graphs27.7 mstransformers, compiled28.9 ms

Best in the chartLinnetReference

Each row also carries its distance from reference: the largest absolute difference between its output and transformers (eager)'s on the same input. bf16 outputs differ by rounding (about 0.1 on logits near 16), so a small number is expected; it is there so that a fast wrong answer cannot look like a win.

Encode

ms, lower is better; the best bar is red.

transformers (eager)39.0 msthe referenceDistance from reference: 0 (the reference)

transformers (torch.compile)28.9 ms1.3x the reference's speedDistance from reference: 1.0e-4

Linnet torch (generated source)41.6 ms0.70× the speed of transformers, compiledDistance from reference: 8.2e-5

Linnet torch (torch.compile, inductor)28.2 ms1.03× the speed of transformers, compiledDistance from reference: 9.7e-5

Linnet torch (CUDA graphs)27.7 ms1.04× the speed of transformers, compiledDistance from reference: 9.7e-5

Linnet JAX (XLA, StableHLO)13.7 ms2.8x the reference's speedDistance from reference: 0.00123peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)13.8 ms2.8x the reference's speedDistance from reference: 0.00128peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet ONNX f32 -> ONNX Runtime (CUDA)22.8 ms1.7x the reference's speedDistance from reference: 0.00232f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; the image encoder

Linnet ONNX f16 -> ONNX Runtime (CUDA)20.9 ms1.9x the reference's speedDistance from reference: 0.00284f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; the image encoder

Linnet ONNX bf16 -> ONNX Runtime (CUDA)22.9 ms1.7x the reference's speedDistance from reference: 0.0276bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; the image encoder

Linnet ONNX f32 -> ONNX Runtime (TensorRT)14.2 ms2.7x the reference's speedDistance from reference: 0.00123f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; the image encoder

Linnet ONNX f16 -> ONNX Runtime (TensorRT)9.64 ms4.0x the reference's speedDistance from reference: 0.00798f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; the image encoder

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)9.76 ms4.0x the reference's speedDistance from reference: 0.0298bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; the image encoder

Load time

s, lower is better; the best bar is red.

transformers (eager)3.17 sthe referenceDistance from reference: 0 (the reference)

transformers (torch.compile)0.65 s4.9x the reference's speedDistance from reference: 1.0e-4

Linnet torch (generated source)0.50 s1.31× the speed of transformers, compiledDistance from reference: 8.2e-5

Linnet torch (torch.compile, inductor)0.45 s1.43× the speed of transformers, compiledDistance from reference: 9.7e-5

Linnet torch (CUDA graphs)0.50 s1.30× the speed of transformers, compiledDistance from reference: 9.7e-5

Linnet JAX (XLA, StableHLO)0.57 s5.6x the reference's speedDistance from reference: 0.00123peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)0.82 s3.9x the reference's speedDistance from reference: 0.00128peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet ONNX f32 -> ONNX Runtime (CUDA)0.31 s10x the reference's speedDistance from reference: 0.00232f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; the image encoder

Linnet ONNX f16 -> ONNX Runtime (CUDA)0.36 s8.9x the reference's speedDistance from reference: 0.00284f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; the image encoder

Linnet ONNX bf16 -> ONNX Runtime (CUDA)0.30 s11x the reference's speedDistance from reference: 0.0276bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; the image encoder

Linnet ONNX f32 -> ONNX Runtime (TensorRT)0.30 s11x the reference's speedDistance from reference: 0.00123f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; the image encoder

Linnet ONNX f16 -> ONNX Runtime (TensorRT)0.32 s10x the reference's speedDistance from reference: 0.00798f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; the image encoder

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)0.32 s10x the reference's speedDistance from reference: 0.0298bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; the image encoder

Peak GPU memory

What the driver reports the process holding at its peak, in GiB; lower is better; the best bar is red.

transformers (eager)2.99 GiB (3,064 MiB)Distance from reference: 0 (the reference)

transformers (torch.compile)2.34 GiB (2,394 MiB)Distance from reference: 1.0e-4

Linnet torch (generated source)2.90 GiB (2,968 MiB)Distance from reference: 8.2e-5

Linnet torch (torch.compile, inductor)2.20 GiB (2,250 MiB)Distance from reference: 9.7e-5

Linnet torch (CUDA graphs)2.34 GiB (2,392 MiB)Distance from reference: 9.7e-5

Linnet JAX (XLA, StableHLO)2.78 GiB (2,848 MiB)Distance from reference: 0.00123peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)2.78 GiB (2,848 MiB)Distance from reference: 0.00128peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet ONNX f32 -> ONNX Runtime (CUDA)8.58 GiB (8,782 MiB)Distance from reference: 0.00232f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; the image encoder

Linnet ONNX f16 -> ONNX Runtime (CUDA)7.81 GiB (8,002 MiB)Distance from reference: 0.00284f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; the image encoder

Linnet ONNX bf16 -> ONNX Runtime (CUDA)7.67 GiB (7,854 MiB)Distance from reference: 0.0276bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; the image encoder

Linnet ONNX f32 -> ONNX Runtime (TensorRT)4.39 GiB (4,492 MiB)Distance from reference: 0.00123f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; the image encoder

Linnet ONNX f16 -> ONNX Runtime (TensorRT)4.11 GiB (4,206 MiB)Distance from reference: 0.00798f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; the image encoder

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)3.81 GiB (3,904 MiB)Distance from reference: 0.0298bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; the image encoder

Every number

Method	Encode	Load time	Peak GPU memory	Against the stack it replaces	Distance from reference	Notes
PyTorch
transformers	39.0 ms	3.17 s	2.99 GiB		0 (the reference)
transformers, compiled	28.9 ms	0.65 s	2.34 GiB		1.0e-4
Linnet, generated PyTorch	41.6 ms	0.50 s	2.90 GiB	0.70× transformers, compiled	8.2e-5
Linnet, inductor	28.2 ms	0.45 s	2.20 GiB	1.03× transformers, compiled	9.7e-5
Linnet, CUDA graphs	27.7 ms	0.50 s	2.34 GiB	1.04× transformers, compiled	9.7e-5
JAX
Linnet, XLA (StableHLO)	13.7 ms	0.57 s	2.78 GiB		0.00123	peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
Linnet, XLA (generated JAX)	13.8 ms	0.82 s	2.78 GiB		0.00128	peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
ONNX Runtime and TensorRT
Linnet, ONNX Runtime f32	22.8 ms	0.31 s	8.58 GiB		0.00232	f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; the image encoder
Linnet, ONNX Runtime f16	20.9 ms	0.36 s	7.81 GiB		0.00284	f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; the image encoder
Linnet, ONNX Runtime bf16	22.9 ms	0.30 s	7.67 GiB		0.0276	bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; the image encoder
Linnet, TensorRT f32	14.2 ms	0.30 s	4.39 GiB		0.00123	f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; the image encoder
Linnet, TensorRT f16	9.64 ms	0.32 s	4.11 GiB		0.00798	f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; the image encoder
Linnet, TensorRT bf16	9.76 ms	0.32 s	3.81 GiB		0.0298	bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; the image encoder