NVIDIA H100 80GB HBM3 · median of 10 runs after 3 warm-ups · measured 2026-09-28T18:17:48+00:00

Linnet 0.1.0, PyTorch 2.14.0, JAX 0.11.2, transformers 5.17.0, diffusers 0.40.0, ONNX Runtime 1.30.0, Python 3.12.3, driver 580.126.09

Linnet against the stack it replaces

Each side in its fastest configuration, on the same GPU and checkpoint. A speed-up is how many times the other's speed; an export is measured against the original checkpoint in the same engine.

vs the reference, in PyTorch2.14× fasterlatency

• Linnet, CUDA graphs0.73 mstransformers, compiled1.57 ms

vs KerasHub, in JAX5.32× fasterlatency

• Linnet, XLA (generated JAX)0.93 msKerasHub4.95 ms

vs torch.onnx, on ONNX Runtime1.60× fasterlatency

• Linnet, ONNX Runtime f321.53 mstorch.onnx export2.44 ms

vs torch.onnx, behind Triton1.12× fasterlatency

• Triton, Linnet's ONNX3.32 msTriton, torch.onnx export3.73 ms

Best in the chartLinnetReference

Each row also carries its distance from reference: the largest absolute difference between its output and transformers (eager)'s on the same input. bf16 outputs differ by rounding (about 0.1 on logits near 16), so a small number is expected; it is there so that a fast wrong answer cannot look like a win.

Latency

ms, lower is better; the best bar is red.

transformers (eager)3.52 msthe referenceDistance from reference: 0 (the reference)unpadded batches of exactly 128 tokens (this card takes no padding mask)

transformers (torch.compile)1.57 ms2.2x the reference's speedDistance from reference: 0.125unpadded batches of exactly 128 tokens (this card takes no padding mask)

Linnet torch (generated source)1.92 ms0.82× the speed of transformers, compiledDistance from reference: 0.156unpadded batches of exactly 128 tokens; bf16 on cuda

Linnet torch (torch.compile, inductor)1.38 ms1.14× the speed of transformers, compiledDistance from reference: 0.125unpadded batches of exactly 128 tokens; bf16 on cuda

Linnet torch (CUDA graphs)0.73 ms2.14× the speed of transformers, compiledDistance from reference: 0.125unpadded batches of exactly 128 tokens; bf16 on cuda

KerasHub (JAX)4.95 ms0.7x the reference's speedDistance from reference: 0.117bf16, under jax.jit; unpadded batches of exactly 128 tokens; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, StableHLO)1.18 ms4.20× the speed of KerasHubDistance from reference: 0.156unpadded batches of exactly 128 tokens; bf16 on cuda; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)0.93 ms5.32× the speed of KerasHubDistance from reference: 0.219unpadded batches of exactly 128 tokens; bf16 on cuda; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

transformers -> torch.onnx -> ONNX Runtime2.44 ms1.4x the reference's speedDistance from reference: 0.0883the reference model's own ONNX export, f32 like Linnet's; unpadded batches of exactly 128 tokens

Linnet ONNX f32 -> ONNX Runtime (CUDA)1.53 ms1.60× the speed of torch.onnx exportDistance from reference: 0.0883f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX f16 -> ONNX Runtime (CUDA)1.23 ms2.9x the reference's speedDistance from reference: 0.0879f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX bf16 -> ONNX Runtime (CUDA)1.23 ms2.9x the reference's speedDistance from reference: 0.125bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX f32 -> ONNX Runtime (TensorRT)0.75 ms4.7x the reference's speedDistance from reference: 0.0876f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX f16 -> ONNX Runtime (TensorRT)0.53 ms6.7x the reference's speedDistance from reference: 0.0977f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)0.56 ms6.3x the reference's speedDistance from reference: 0.125bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Triton Inference Server (ONNX Runtime backend, torch.onnx export)3.73 ms0.9x the reference's speedDistance from reference: 0.0883over HTTP; f32; throughput with 4 requests of batch 64 in flight

Triton Inference Server (ONNX Runtime backend, Linnet ONNX)3.32 ms1.12× the speed of Triton, torch.onnx exportDistance from reference: 0.0883over HTTP; f32; throughput with 4 requests of batch 64 in flight

Triton Inference Server (Python backend, Linnet torch)2.68 ms1.3x the reference's speedDistance from reference: 0.125over HTTP; bf16, CUDA graphs; throughput with 4 requests of batch 64 in flight

Throughput

/s, higher is better; the best bar is red.

transformers (eager)10,857/sthe referenceDistance from reference: 0 (the reference)unpadded batches of exactly 128 tokens (this card takes no padding mask)

transformers (torch.compile)14,513/s1.3x the reference's speedDistance from reference: 0.125unpadded batches of exactly 128 tokens (this card takes no padding mask)

Linnet torch (generated source)13,836/s0.95× the speed of transformers, compiledDistance from reference: 0.156unpadded batches of exactly 128 tokens; bf16 on cuda

Linnet torch (torch.compile, inductor)17,223/s1.19× the speed of transformers, compiledDistance from reference: 0.125unpadded batches of exactly 128 tokens; bf16 on cuda

Linnet torch (CUDA graphs)18,181/s1.25× the speed of transformers, compiledDistance from reference: 0.125unpadded batches of exactly 128 tokens; bf16 on cuda

KerasHub (JAX)5,082/s0.5x the reference's speedDistance from reference: 0.117bf16, under jax.jit; unpadded batches of exactly 128 tokens; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, StableHLO)12,433/s2.45× the speed of KerasHubDistance from reference: 0.156unpadded batches of exactly 128 tokens; bf16 on cuda; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)11,985/s2.36× the speed of KerasHubDistance from reference: 0.219unpadded batches of exactly 128 tokens; bf16 on cuda; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

transformers -> torch.onnx -> ONNX Runtime4,955/s0.5x the reference's speedDistance from reference: 0.0883the reference model's own ONNX export, f32 like Linnet's; unpadded batches of exactly 128 tokens

Linnet ONNX f32 -> ONNX Runtime (CUDA)5,276/s1.06× the speed of torch.onnx exportDistance from reference: 0.0883f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX f16 -> ONNX Runtime (CUDA)9,664/s0.9x the reference's speedDistance from reference: 0.0879f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX bf16 -> ONNX Runtime (CUDA)9,497/s0.9x the reference's speedDistance from reference: 0.125bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX f32 -> ONNX Runtime (TensorRT)9,050/s0.8x the reference's speedDistance from reference: 0.0876f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX f16 -> ONNX Runtime (TensorRT)19,752/s1.8x the reference's speedDistance from reference: 0.0977f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)18,619/s1.7x the reference's speedDistance from reference: 0.125bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Triton Inference Server (ONNX Runtime backend, torch.onnx export)566/s0.05x the reference's speedDistance from reference: 0.0883over HTTP; f32; throughput with 4 requests of batch 64 in flight

Triton Inference Server (ONNX Runtime backend, Linnet ONNX)502/s0.89× the speed of Triton, torch.onnx exportDistance from reference: 0.0883over HTTP; f32; throughput with 4 requests of batch 64 in flight

Triton Inference Server (Python backend, Linnet torch)540/s0.05x the reference's speedDistance from reference: 0.125over HTTP; bf16, CUDA graphs; throughput with 4 requests of batch 64 in flight

Load time

s, lower is better; the best bar is red.

transformers (eager)15.6 sthe referenceDistance from reference: 0 (the reference)unpadded batches of exactly 128 tokens (this card takes no padding mask)

transformers (torch.compile)1.55 s10x the reference's speedDistance from reference: 0.125unpadded batches of exactly 128 tokens (this card takes no padding mask)

Linnet torch (generated source)1.14 s1.36× the speed of transformers, compiledDistance from reference: 0.156unpadded batches of exactly 128 tokens; bf16 on cuda

Linnet torch (torch.compile, inductor)1.11 s1.40× the speed of transformers, compiledDistance from reference: 0.125unpadded batches of exactly 128 tokens; bf16 on cuda

Linnet torch (CUDA graphs)1.14 s1.35× the speed of transformers, compiledDistance from reference: 0.125unpadded batches of exactly 128 tokens; bf16 on cuda

KerasHub (JAX)5.99 s2.6x the reference's speedDistance from reference: 0.117bf16, under jax.jit; unpadded batches of exactly 128 tokens; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, StableHLO)0.54 s11× the speed of KerasHubDistance from reference: 0.156unpadded batches of exactly 128 tokens; bf16 on cuda; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)0.57 s10× the speed of KerasHubDistance from reference: 0.219unpadded batches of exactly 128 tokens; bf16 on cuda; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

transformers -> torch.onnx -> ONNX Runtime18.6 s0.8x the reference's speedDistance from reference: 0.0883the reference model's own ONNX export, f32 like Linnet's; unpadded batches of exactly 128 tokens

Linnet ONNX f32 -> ONNX Runtime (CUDA)0.28 s65× the speed of torch.onnx exportDistance from reference: 0.0883f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX f16 -> ONNX Runtime (CUDA)0.29 s54x the reference's speedDistance from reference: 0.0879f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX bf16 -> ONNX Runtime (CUDA)0.37 s42x the reference's speedDistance from reference: 0.125bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX f32 -> ONNX Runtime (TensorRT)0.27 s57x the reference's speedDistance from reference: 0.0876f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX f16 -> ONNX Runtime (TensorRT)0.27 s57x the reference's speedDistance from reference: 0.0977f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)0.27 s58x the reference's speedDistance from reference: 0.125bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Triton Inference Server (ONNX Runtime backend, torch.onnx export)10.2 s1.5x the reference's speedDistance from reference: 0.0883over HTTP; f32; throughput with 4 requests of batch 64 in flight

Triton Inference Server (ONNX Runtime backend, Linnet ONNX)17.0 s0.60× the speed of Triton, torch.onnx exportDistance from reference: 0.0883over HTTP; f32; throughput with 4 requests of batch 64 in flight

Triton Inference Server (Python backend, Linnet torch)6.02 s2.6x the reference's speedDistance from reference: 0.125over HTTP; bf16, CUDA graphs; throughput with 4 requests of batch 64 in flight

Peak GPU memory

What the driver reports the process holding at its peak, in GiB; lower is better; the best bar is red.

transformers (eager)1.16 GiB (1,190 MiB)Distance from reference: 0 (the reference)unpadded batches of exactly 128 tokens (this card takes no padding mask)

transformers (torch.compile)1.14 GiB (1,170 MiB)Distance from reference: 0.125unpadded batches of exactly 128 tokens (this card takes no padding mask)

Linnet torch (generated source)1.11 GiB (1,136 MiB)Distance from reference: 0.156unpadded batches of exactly 128 tokens; bf16 on cuda

Linnet torch (torch.compile, inductor)1.06 GiB (1,088 MiB)Distance from reference: 0.125unpadded batches of exactly 128 tokens; bf16 on cuda

Linnet torch (CUDA graphs)1.15 GiB (1,176 MiB)Distance from reference: 0.125unpadded batches of exactly 128 tokens; bf16 on cuda

KerasHub (JAX)0.55 GiB (567 MiB)Distance from reference: 0.117bf16, under jax.jit; unpadded batches of exactly 128 tokens; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, StableHLO)0.77 GiB (785 MiB)Distance from reference: 0.156unpadded batches of exactly 128 tokens; bf16 on cuda; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)0.75 GiB (771 MiB)Distance from reference: 0.219unpadded batches of exactly 128 tokens; bf16 on cuda; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

transformers -> torch.onnx -> ONNX Runtime2.81 GiB (2,874 MiB)Distance from reference: 0.0883the reference model's own ONNX export, f32 like Linnet's; unpadded batches of exactly 128 tokens

Linnet ONNX f32 -> ONNX Runtime (CUDA)2.64 GiB (2,702 MiB)Distance from reference: 0.0883f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX f16 -> ONNX Runtime (CUDA)1.76 GiB (1,804 MiB)Distance from reference: 0.0879f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX bf16 -> ONNX Runtime (CUDA)1.76 GiB (1,804 MiB)Distance from reference: 0.125bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX f32 -> ONNX Runtime (TensorRT)3.33 GiB (3,410 MiB)Distance from reference: 0.0876f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX f16 -> ONNX Runtime (TensorRT)3.06 GiB (3,130 MiB)Distance from reference: 0.0977f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)3.05 GiB (3,128 MiB)Distance from reference: 0.125bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length

Triton Inference Server (ONNX Runtime backend, torch.onnx export)3.50 GiB (3,585 MiB)Distance from reference: 0.0883over HTTP; f32; throughput with 4 requests of batch 64 in flight

Triton Inference Server (ONNX Runtime backend, Linnet ONNX)3.87 GiB (3,961 MiB)Distance from reference: 0.0883over HTTP; f32; throughput with 4 requests of batch 64 in flight

Triton Inference Server (Python backend, Linnet torch)2.87 GiB (2,936 MiB)Distance from reference: 0.125over HTTP; bf16, CUDA graphs; throughput with 4 requests of batch 64 in flight

Every number

Method	Latency	Throughput	Load time	Peak GPU memory	Against the stack it replaces	Distance from reference	Notes
PyTorch
transformers	3.52 ms	10,857/s	15.6 s	1.16 GiB		0 (the reference)	unpadded batches of exactly 128 tokens (this card takes no padding mask)
transformers, compiled	1.57 ms	14,513/s	1.55 s	1.14 GiB		0.125	unpadded batches of exactly 128 tokens (this card takes no padding mask)
Linnet, generated PyTorch	1.92 ms	13,836/s	1.14 s	1.11 GiB	0.82× transformers, compiled	0.156	unpadded batches of exactly 128 tokens; bf16 on cuda
Linnet, inductor	1.38 ms	17,223/s	1.11 s	1.06 GiB	1.14× transformers, compiled	0.125	unpadded batches of exactly 128 tokens; bf16 on cuda
Linnet, CUDA graphs	0.73 ms	18,181/s	1.14 s	1.15 GiB	2.14× transformers, compiled	0.125	unpadded batches of exactly 128 tokens; bf16 on cuda
JAX
KerasHub	4.95 ms	5,082/s	5.99 s	0.55 GiB		0.117	bf16, under jax.jit; unpadded batches of exactly 128 tokens; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
Linnet, XLA (StableHLO)	1.18 ms	12,433/s	0.54 s	0.77 GiB	4.20× KerasHub	0.156	unpadded batches of exactly 128 tokens; bf16 on cuda; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
Linnet, XLA (generated JAX)	0.93 ms	11,985/s	0.57 s	0.75 GiB	5.32× KerasHub	0.219	unpadded batches of exactly 128 tokens; bf16 on cuda; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
ONNX Runtime and TensorRT
torch.onnx export	2.44 ms	4,955/s	18.6 s	2.81 GiB		0.0883	the reference model's own ONNX export, f32 like Linnet's; unpadded batches of exactly 128 tokens
Linnet, ONNX Runtime f32	1.53 ms	5,276/s	0.28 s	2.64 GiB	1.60× torch.onnx export	0.0883	f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length
Linnet, ONNX Runtime f16	1.23 ms	9,664/s	0.29 s	1.76 GiB		0.0879	f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length
Linnet, ONNX Runtime bf16	1.23 ms	9,497/s	0.37 s	1.76 GiB		0.125	bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; unpadded batches of exactly the sequence length
Linnet, TensorRT f32	0.75 ms	9,050/s	0.27 s	3.33 GiB		0.0876	f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length
Linnet, TensorRT f16	0.53 ms	19,752/s	0.27 s	3.06 GiB		0.0977	f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length
Linnet, TensorRT bf16	0.56 ms	18,619/s	0.27 s	3.05 GiB		0.125	bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; unpadded batches of exactly the sequence length
Triton Inference Server
Triton, torch.onnx export	3.73 ms	566/s	10.2 s	3.50 GiB		0.0883	over HTTP; f32; throughput with 4 requests of batch 64 in flight
Triton, Linnet's ONNX	3.32 ms	502/s	17.0 s	3.87 GiB	1.12× Triton, torch.onnx export	0.0883	over HTTP; f32; throughput with 4 requests of batch 64 in flight
Triton, Linnet Python backend	2.68 ms	540/s	6.02 s	2.87 GiB		0.125	over HTTP; bf16, CUDA graphs; throughput with 4 requests of batch 64 in flight