NVIDIA H100 80GB HBM3 · median of 10 runs after 3 warm-ups · measured 2026-09-28T19:33:54+00:00

Linnet 0.1.0, PyTorch 2.14.0, JAX 0.11.2, transformers 5.17.0, diffusers 0.40.0, ONNX Runtime 1.30.0, Python 3.12.3, driver 580.126.09

Linnet against the stack it replaces

Each side in its fastest configuration, on the same GPU and checkpoint. A speed-up is how many times the other's speed; an export is measured against the original checkpoint in the same engine.

vs the reference, in PyTorch2.32× fastertranscribe

• Linnet, CUDA graphs22.3 mstransformers, compiled51.8 ms

Best in the chartLinnetReference

Each row also carries its distance from reference: the largest absolute difference between its output and transformers (eager)'s on the same input. bf16 outputs differ by rounding (about 0.1 on logits near 16), so a small number is expected; it is there so that a fast wrong answer cannot look like a win.

Encode

ms, lower is better; the best bar is red.

transformers (eager)2.53 msthe referenceDistance from reference: 0 (the reference)

transformers (torch.compile, static cache)2.18 ms1.2x the reference's speedDistance from reference: 8.0e-4

Linnet torch (generated source)2.36 ms0.92× the speed of transformers, compiledDistance from reference: 0.0789transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (torch.compile, inductor)1.99 ms1.09× the speed of transformers, compiledDistance from reference: 0.0791transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (CUDA graphs)1.81 ms1.20× the speed of transformers, compiledDistance from reference: 0.0791transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet JAX (XLA, StableHLO)1.02 ms2.5x the reference's speedDistance from reference: 0.651transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)1.01 ms2.5x the reference's speedDistance from reference: 0.682transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet ONNX f32 -> ONNX Runtime (CUDA)1.70 ms1.5x the reference's speedDistance from reference: 0.751f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (CUDA)1.64 ms1.5x the reference's speedDistance from reference: 0.718f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (CUDA)1.26 ms2.0x the reference's speedDistance from reference: 8.53bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f32 -> ONNX Runtime (TensorRT)0.79 ms3.2x the reference's speedDistance from reference: 0.784f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (TensorRT)0.41 ms6.1x the reference's speedDistance from reference: 1.35f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)0.45 ms5.6x the reference's speedDistance from reference: 5.79bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Transcribe

ms, lower is better; the best bar is red.

transformers (eager)63.1 msthe referenceDistance from reference: 0 (the reference)

transformers (torch.compile, static cache)51.8 ms1.2x the reference's speedDistance from reference: 8.0e-4

Linnet torch (generated source)58.7 ms0.88× the speed of transformers, compiledDistance from reference: 0.0789transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (torch.compile, inductor)34.3 ms1.51× the speed of transformers, compiledDistance from reference: 0.0791transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (CUDA graphs)22.3 ms2.32× the speed of transformers, compiledDistance from reference: 0.0791transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet JAX (XLA, StableHLO)16.4 ms3.9x the reference's speedDistance from reference: 0.651transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)16.6 ms3.8x the reference's speedDistance from reference: 0.682transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet ONNX f32 -> ONNX Runtime (CUDA)16.2 ms3.9x the reference's speedDistance from reference: 0.751f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (CUDA)16.5 ms3.8x the reference's speedDistance from reference: 0.718f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (CUDA)17.3 ms3.7x the reference's speedDistance from reference: 8.53bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f32 -> ONNX Runtime (TensorRT)24.3 ms2.6x the reference's speedDistance from reference: 0.784f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (TensorRT)20.4 ms3.1x the reference's speedDistance from reference: 1.35f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)20.4 ms3.1x the reference's speedDistance from reference: 5.79bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Load time

s, lower is better; the best bar is red.

transformers (eager)1.91 sthe referenceDistance from reference: 0 (the reference)

transformers (torch.compile, static cache)0.67 s2.9x the reference's speedDistance from reference: 8.0e-4

Linnet torch (generated source)0.31 s2.19× the speed of transformers, compiledDistance from reference: 0.0789transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (torch.compile, inductor)0.30 s2.26× the speed of transformers, compiledDistance from reference: 0.0791transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (CUDA graphs)0.30 s2.22× the speed of transformers, compiledDistance from reference: 0.0791transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet JAX (XLA, StableHLO)0.22 s8.6x the reference's speedDistance from reference: 0.651transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)0.22 s8.9x the reference's speedDistance from reference: 0.682transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet ONNX f32 -> ONNX Runtime (CUDA)0.24 s8.1x the reference's speedDistance from reference: 0.751f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (CUDA)0.23 s8.2x the reference's speedDistance from reference: 0.718f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (CUDA)0.24 s7.9x the reference's speedDistance from reference: 8.53bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f32 -> ONNX Runtime (TensorRT)0.15 s12x the reference's speedDistance from reference: 0.784f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (TensorRT)0.25 s7.8x the reference's speedDistance from reference: 1.35f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)0.15 s13x the reference's speedDistance from reference: 5.79bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Peak GPU memory

What the driver reports the process holding at its peak, in GiB; lower is better; the best bar is red.

transformers (eager)0.91 GiB (934 MiB)Distance from reference: 0 (the reference)

transformers (torch.compile, static cache)1.09 GiB (1,120 MiB)Distance from reference: 8.0e-4

Linnet torch (generated source)1.04 GiB (1,064 MiB)Distance from reference: 0.0789transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (torch.compile, inductor)1.04 GiB (1,062 MiB)Distance from reference: 0.0791transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (CUDA graphs)1.16 GiB (1,184 MiB)Distance from reference: 0.0791transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet JAX (XLA, StableHLO)0.42 GiB (432 MiB)Distance from reference: 0.651transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)0.42 GiB (432 MiB)Distance from reference: 0.682transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet ONNX f32 -> ONNX Runtime (CUDA)1.80 GiB (1,846 MiB)Distance from reference: 0.751f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (CUDA)1.63 GiB (1,674 MiB)Distance from reference: 0.718f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (CUDA)1.52 GiB (1,552 MiB)Distance from reference: 8.53bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f32 -> ONNX Runtime (TensorRT)3.34 GiB (3,418 MiB)Distance from reference: 0.784f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (TensorRT)3.07 GiB (3,142 MiB)Distance from reference: 1.35f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)3.01 GiB (3,080 MiB)Distance from reference: 5.79bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Every number

Method	Encode	Transcribe	Load time	Peak GPU memory	Against the stack it replaces	Distance from reference	Notes
PyTorch
transformers	2.53 ms	63.1 ms	1.91 s	0.91 GiB		0 (the reference)
transformers, compiled	2.18 ms	51.8 ms	0.67 s	1.09 GiB		8.0e-4
Linnet, generated PyTorch	2.36 ms	58.7 ms	0.31 s	1.04 GiB	0.88× transformers, compiled	0.0789	transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
Linnet, inductor	1.99 ms	34.3 ms	0.30 s	1.04 GiB	1.51× transformers, compiled	0.0791	transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
Linnet, CUDA graphs	1.81 ms	22.3 ms	0.30 s	1.16 GiB	2.32× transformers, compiled	0.0791	transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
JAX
Linnet, XLA (StableHLO)	1.02 ms	16.4 ms	0.22 s	0.42 GiB		0.651	transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
Linnet, XLA (generated JAX)	1.01 ms	16.6 ms	0.22 s	0.42 GiB		0.682	transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
ONNX Runtime and TensorRT
Linnet, ONNX Runtime f32	1.70 ms	16.2 ms	0.24 s	1.80 GiB		0.751	f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
Linnet, ONNX Runtime f16	1.64 ms	16.5 ms	0.23 s	1.63 GiB		0.718	f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
Linnet, ONNX Runtime bf16	1.26 ms	17.3 ms	0.24 s	1.52 GiB		8.53	bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
Linnet, TensorRT f32	0.79 ms	24.3 ms	0.15 s	3.34 GiB		0.784	f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
Linnet, TensorRT f16	0.41 ms	20.4 ms	0.25 s	3.07 GiB		1.35	f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
Linnet, TensorRT bf16	0.45 ms	20.4 ms	0.15 s	3.01 GiB		5.79	bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each