NVIDIA H100 80GB HBM3 · median of 10 runs after 3 warm-ups · measured 2026-09-28T19:24:12+00:00

Linnet 0.1.0, PyTorch 2.14.0, JAX 0.11.2, transformers 5.17.0, diffusers 0.40.0, ONNX Runtime 1.30.0, Python 3.12.3, driver 580.126.09

Linnet against the stack it replaces

Each side in its fastest configuration, on the same GPU and checkpoint. A speed-up is how many times the other's speed; an export is measured against the original checkpoint in the same engine.

vs the reference, in PyTorch2.04× fasterlatency

• Linnet, CUDA graphs0.93 mstransformers, compiled1.89 ms

vs torch.onnx, on ONNX Runtime1.01× fasterlatency

• Linnet, ONNX Runtime f322.00 mstorch.onnx export2.01 ms

vs torch.onnx, behind Triton1.01× fasterlatency

• Triton, Linnet's ONNX4.00 msTriton, torch.onnx export4.04 ms

Best in the chartLinnetReference

Each row also carries its distance from reference: the largest absolute difference between its output and transformers (eager)'s on the same input. bf16 outputs differ by rounding (about 0.1 on logits near 16), so a small number is expected; it is there so that a fast wrong answer cannot look like a win.

Latency

ms, lower is better; the best bar is red.

transformers (eager)2.87 msthe referenceDistance from reference: 0 (the reference)

transformers (torch.compile)1.89 ms1.5x the reference's speedDistance from reference: 0.0313

Linnet torch (generated source)2.21 ms0.86× the speed of transformers, compiledDistance from reference: 0.0156

Linnet torch (torch.compile, inductor)1.64 ms1.15× the speed of transformers, compiledDistance from reference: 0.0313

Linnet torch (CUDA graphs)0.93 ms2.04× the speed of transformers, compiledDistance from reference: 0.0313

Linnet JAX (XLA, StableHLO)1.15 ms2.5x the reference's speedDistance from reference: 0.0313bf16, like the torch rows; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)1.11 ms2.6x the reference's speedDistance from reference: 0.0234bf16, like the torch rows; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

transformers -> torch.onnx -> ONNX Runtime2.01 ms1.4x the reference's speedDistance from reference: 0.0446the reference model's own ONNX export, f32 like Linnet's;

Linnet ONNX f32 -> ONNX Runtime (CUDA)2.00 ms1.01× the speed of torch.onnx exportDistance from reference: 0.0452f32; ONNX Runtime's CUDA execution provider; first calls build the sessions

Linnet ONNX f16 -> ONNX Runtime (CUDA)1.94 ms1.5x the reference's speedDistance from reference: 0.0469f16; ONNX Runtime's CUDA execution provider; first calls build the sessions

Linnet ONNX bf16 -> ONNX Runtime (CUDA)1.74 ms1.6x the reference's speedDistance from reference: 0.0313bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions

Linnet ONNX f32 -> ONNX Runtime (TensorRT)0.90 ms3.2x the reference's speedDistance from reference: 0.0456f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions

Linnet ONNX f16 -> ONNX Runtime (TensorRT)0.64 ms4.5x the reference's speedDistance from reference: 0.043f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)0.67 ms4.3x the reference's speedDistance from reference: 0.0313bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions

Triton Inference Server (ONNX Runtime backend, torch.onnx export)4.04 ms0.7x the reference's speedDistance from reference: 0.0293over HTTP; f32; throughput with 4 requests of batch 32 in flight

Triton Inference Server (ONNX Runtime backend, Linnet ONNX)4.00 ms1.01× the speed of Triton, torch.onnx exportDistance from reference: 0.0299over HTTP; f32; throughput with 4 requests of batch 32 in flight

Triton Inference Server (Python backend, Linnet torch)3.19 ms0.9x the reference's speedDistance from reference: 0.0313over HTTP; bf16, CUDA graphs; throughput with 4 requests of batch 32 in flight

Throughput

/s, higher is better; the best bar is red.

transformers (eager)6,030/sthe referenceDistance from reference: 0 (the reference)

transformers (torch.compile)8,317/s1.4x the reference's speedDistance from reference: 0.0313

Linnet torch (generated source)7,997/s0.96× the speed of transformers, compiledDistance from reference: 0.0156

Linnet torch (torch.compile, inductor)9,536/s1.15× the speed of transformers, compiledDistance from reference: 0.0313

Linnet torch (CUDA graphs)9,937/s1.19× the speed of transformers, compiledDistance from reference: 0.0313

Linnet JAX (XLA, StableHLO)6,716/s1.1x the reference's speedDistance from reference: 0.0313bf16, like the torch rows; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)9,054/s1.5x the reference's speedDistance from reference: 0.0234bf16, like the torch rows; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

transformers -> torch.onnx -> ONNX Runtime2,664/s0.4x the reference's speedDistance from reference: 0.0446the reference model's own ONNX export, f32 like Linnet's;

Linnet ONNX f32 -> ONNX Runtime (CUDA)2,752/s1.03× the speed of torch.onnx exportDistance from reference: 0.0452f32; ONNX Runtime's CUDA execution provider; first calls build the sessions

Linnet ONNX f16 -> ONNX Runtime (CUDA)4,923/s0.8x the reference's speedDistance from reference: 0.0469f16; ONNX Runtime's CUDA execution provider; first calls build the sessions

Linnet ONNX bf16 -> ONNX Runtime (CUDA)4,443/s0.7x the reference's speedDistance from reference: 0.0313bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions

Linnet ONNX f32 -> ONNX Runtime (TensorRT)4,968/s0.8x the reference's speedDistance from reference: 0.0456f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions

Linnet ONNX f16 -> ONNX Runtime (TensorRT)11,447/s1.9x the reference's speedDistance from reference: 0.043f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)10,896/s1.8x the reference's speedDistance from reference: 0.0313bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions

Triton Inference Server (ONNX Runtime backend, torch.onnx export)1,314/s0.2x the reference's speedDistance from reference: 0.0293over HTTP; f32; throughput with 4 requests of batch 32 in flight

Triton Inference Server (ONNX Runtime backend, Linnet ONNX)1,235/s0.94× the speed of Triton, torch.onnx exportDistance from reference: 0.0299over HTTP; f32; throughput with 4 requests of batch 32 in flight

Triton Inference Server (Python backend, Linnet torch)1,225/s0.2x the reference's speedDistance from reference: 0.0313over HTTP; bf16, CUDA graphs; throughput with 4 requests of batch 32 in flight

Load time

s, lower is better; the best bar is red.

transformers (eager)7.93 sthe referenceDistance from reference: 0 (the reference)

transformers (torch.compile)4.07 s1.9x the reference's speedDistance from reference: 0.0313

Linnet torch (generated source)2.38 s1.71× the speed of transformers, compiledDistance from reference: 0.0156

Linnet torch (torch.compile, inductor)2.33 s1.75× the speed of transformers, compiledDistance from reference: 0.0313

Linnet torch (CUDA graphs)2.28 s1.79× the speed of transformers, compiledDistance from reference: 0.0313

Linnet JAX (XLA, StableHLO)0.73 s11x the reference's speedDistance from reference: 0.0313bf16, like the torch rows; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)1.02 s7.8x the reference's speedDistance from reference: 0.0234bf16, like the torch rows; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

transformers -> torch.onnx -> ONNX Runtime19.7 s0.4x the reference's speedDistance from reference: 0.0446the reference model's own ONNX export, f32 like Linnet's;

Linnet ONNX f32 -> ONNX Runtime (CUDA)0.29 s67× the speed of torch.onnx exportDistance from reference: 0.0452f32; ONNX Runtime's CUDA execution provider; first calls build the sessions

Linnet ONNX f16 -> ONNX Runtime (CUDA)0.31 s26x the reference's speedDistance from reference: 0.0469f16; ONNX Runtime's CUDA execution provider; first calls build the sessions

Linnet ONNX bf16 -> ONNX Runtime (CUDA)0.34 s23x the reference's speedDistance from reference: 0.0313bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions

Linnet ONNX f32 -> ONNX Runtime (TensorRT)0.31 s26x the reference's speedDistance from reference: 0.0456f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions

Linnet ONNX f16 -> ONNX Runtime (TensorRT)0.32 s25x the reference's speedDistance from reference: 0.043f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)0.34 s23x the reference's speedDistance from reference: 0.0313bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions

Triton Inference Server (ONNX Runtime backend, torch.onnx export)9.03 s0.9x the reference's speedDistance from reference: 0.0293over HTTP; f32; throughput with 4 requests of batch 32 in flight

Triton Inference Server (ONNX Runtime backend, Linnet ONNX)17.1 s0.53× the speed of Triton, torch.onnx exportDistance from reference: 0.0299over HTTP; f32; throughput with 4 requests of batch 32 in flight

Triton Inference Server (Python backend, Linnet torch)6.03 s1.3x the reference's speedDistance from reference: 0.0313over HTTP; bf16, CUDA graphs; throughput with 4 requests of batch 32 in flight

Peak GPU memory

What the driver reports the process holding at its peak, in GiB; lower is better; the best bar is red.

transformers (eager)1.26 GiB (1,290 MiB)Distance from reference: 0 (the reference)

transformers (torch.compile)1.28 GiB (1,312 MiB)Distance from reference: 0.0313

Linnet torch (generated source)1.24 GiB (1,270 MiB)Distance from reference: 0.0156

Linnet torch (torch.compile, inductor)1.22 GiB (1,252 MiB)Distance from reference: 0.0313

Linnet torch (CUDA graphs)1.42 GiB (1,452 MiB)Distance from reference: 0.0313

Linnet JAX (XLA, StableHLO)1.06 GiB (1,085 MiB)Distance from reference: 0.0313bf16, like the torch rows; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)1.04 GiB (1,067 MiB)Distance from reference: 0.0234bf16, like the torch rows; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

transformers -> torch.onnx -> ONNX Runtime2.34 GiB (2,394 MiB)Distance from reference: 0.0446the reference model's own ONNX export, f32 like Linnet's;

Linnet ONNX f32 -> ONNX Runtime (CUDA)2.39 GiB (2,444 MiB)Distance from reference: 0.0452f32; ONNX Runtime's CUDA execution provider; first calls build the sessions

Linnet ONNX f16 -> ONNX Runtime (CUDA)1.64 GiB (1,684 MiB)Distance from reference: 0.0469f16; ONNX Runtime's CUDA execution provider; first calls build the sessions

Linnet ONNX bf16 -> ONNX Runtime (CUDA)1.65 GiB (1,694 MiB)Distance from reference: 0.0313bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions

Linnet ONNX f32 -> ONNX Runtime (TensorRT)3.13 GiB (3,202 MiB)Distance from reference: 0.0456f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions

Linnet ONNX f16 -> ONNX Runtime (TensorRT)2.89 GiB (2,960 MiB)Distance from reference: 0.043f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)2.89 GiB (2,962 MiB)Distance from reference: 0.0313bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions

Triton Inference Server (ONNX Runtime backend, torch.onnx export)2.38 GiB (2,437 MiB)Distance from reference: 0.0293over HTTP; f32; throughput with 4 requests of batch 32 in flight

Triton Inference Server (ONNX Runtime backend, Linnet ONNX)2.38 GiB (2,433 MiB)Distance from reference: 0.0299over HTTP; f32; throughput with 4 requests of batch 32 in flight

Triton Inference Server (Python backend, Linnet torch)3.22 GiB (3,302 MiB)Distance from reference: 0.0313over HTTP; bf16, CUDA graphs; throughput with 4 requests of batch 32 in flight

Every number

Method	Latency	Throughput	Load time	Peak GPU memory	Against the stack it replaces	Distance from reference	Notes
PyTorch
transformers	2.87 ms	6,030/s	7.93 s	1.26 GiB		0 (the reference)
transformers, compiled	1.89 ms	8,317/s	4.07 s	1.28 GiB		0.0313
Linnet, generated PyTorch	2.21 ms	7,997/s	2.38 s	1.24 GiB	0.86× transformers, compiled	0.0156
Linnet, inductor	1.64 ms	9,536/s	2.33 s	1.22 GiB	1.15× transformers, compiled	0.0313
Linnet, CUDA graphs	0.93 ms	9,937/s	2.28 s	1.42 GiB	2.04× transformers, compiled	0.0313
JAX
Linnet, XLA (StableHLO)	1.15 ms	6,716/s	0.73 s	1.06 GiB		0.0313	bf16, like the torch rows; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
Linnet, XLA (generated JAX)	1.11 ms	9,054/s	1.02 s	1.04 GiB		0.0234	bf16, like the torch rows; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
ONNX Runtime and TensorRT
torch.onnx export	2.01 ms	2,664/s	19.7 s	2.34 GiB		0.0446	the reference model's own ONNX export, f32 like Linnet's;
Linnet, ONNX Runtime f32	2.00 ms	2,752/s	0.29 s	2.39 GiB	1.01× torch.onnx export	0.0452	f32; ONNX Runtime's CUDA execution provider; first calls build the sessions
Linnet, ONNX Runtime f16	1.94 ms	4,923/s	0.31 s	1.64 GiB		0.0469	f16; ONNX Runtime's CUDA execution provider; first calls build the sessions
Linnet, ONNX Runtime bf16	1.74 ms	4,443/s	0.34 s	1.65 GiB		0.0313	bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions
Linnet, TensorRT f32	0.90 ms	4,968/s	0.31 s	3.13 GiB		0.0456	f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions
Linnet, TensorRT f16	0.64 ms	11,447/s	0.32 s	2.89 GiB		0.043	f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions
Linnet, TensorRT bf16	0.67 ms	10,896/s	0.34 s	2.89 GiB		0.0313	bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions
Triton Inference Server
Triton, torch.onnx export	4.04 ms	1,314/s	9.03 s	2.38 GiB		0.0293	over HTTP; f32; throughput with 4 requests of batch 32 in flight
Triton, Linnet's ONNX	4.00 ms	1,235/s	17.1 s	2.38 GiB	1.01× Triton, torch.onnx export	0.0299	over HTTP; f32; throughput with 4 requests of batch 32 in flight
Triton, Linnet Python backend	3.19 ms	1,225/s	6.03 s	3.22 GiB		0.0313	over HTTP; bf16, CUDA graphs; throughput with 4 requests of batch 32 in flight