NVIDIA H100 80GB HBM3 · median of 10 runs after 3 warm-ups · measured 2026-09-28T19:56:30+00:00

Linnet 0.1.0, PyTorch 2.14.0, JAX 0.11.2, transformers 5.17.0, diffusers 0.40.0, ONNX Runtime 1.30.0, Python 3.12.3, driver 580.126.09

Linnet against the stack it replaces

Each side in its fastest configuration, on the same GPU and checkpoint. A speed-up is how many times the other's speed; an export is measured against the original checkpoint in the same engine.

vs the reference, in PyTorch2.85× fastertranscribe

• Linnet, CUDA graphs67.4 mstransformers, compiled192 ms

Best in the chartLinnetReference

Each row also carries its distance from reference: the largest absolute difference between its output and transformers (eager)'s on the same input. bf16 outputs differ by rounding (about 0.1 on logits near 16), so a small number is expected; it is there so that a fast wrong answer cannot look like a win.

Encode

ms, lower is better; the best bar is red.

transformers (eager)9.21 msthe referenceDistance from reference: 0 (the reference)

transformers (torch.compile, static cache)7.27 ms1.3x the reference's speedDistance from reference: 4.91

Linnet torch (generated source)7.34 ms0.99× the speed of transformers, compiledDistance from reference: 1.13transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (torch.compile, inductor)6.58 ms1.11× the speed of transformers, compiledDistance from reference: 4.57transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (CUDA graphs)5.63 ms1.29× the speed of transformers, compiledDistance from reference: 4.57transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet JAX (XLA, StableHLO)11.7 ms0.8x the reference's speedDistance from reference: 3.29transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)7.32 ms1.3x the reference's speedDistance from reference: 16.3transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet ONNX f32 -> ONNX Runtime (CUDA)25.3 ms0.4x the reference's speedDistance from reference: 4.81f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (CUDA)28.8 ms0.3x the reference's speedDistance from reference: 0.636f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (CUDA)21.0 ms0.4x the reference's speedDistance from reference: 15bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f32 -> ONNX Runtime (TensorRT)25.0 ms0.4x the reference's speedDistance from reference: 4.81f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (TensorRT)5.42 ms1.7x the reference's speedDistance from reference: 3.99f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)6.13 ms1.5x the reference's speedDistance from reference: 14.2bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Transcribe

ms, lower is better; the best bar is red.

transformers (eager)307 msthe referenceDistance from reference: 0 (the reference)

transformers (torch.compile, static cache)192 ms1.6x the reference's speedDistance from reference: 4.91

Linnet torch (generated source)338 ms0.57× the speed of transformers, compiledDistance from reference: 1.13transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (torch.compile, inductor)159 ms1.21× the speed of transformers, compiledDistance from reference: 4.57transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (CUDA graphs)67.4 ms2.85× the speed of transformers, compiledDistance from reference: 4.57transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet JAX (XLA, StableHLO)96.5 ms3.2x the reference's speedDistance from reference: 3.29transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)108 ms2.9x the reference's speedDistance from reference: 16.3transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet ONNX f32 -> ONNX Runtime (CUDA)158 ms1.9x the reference's speedDistance from reference: 4.81f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (CUDA)154 ms2.0x the reference's speedDistance from reference: 0.636f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (CUDA)160 ms1.9x the reference's speedDistance from reference: 15bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f32 -> ONNX Runtime (TensorRT)250 ms1.2x the reference's speedDistance from reference: 4.81f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (TensorRT)194 ms1.6x the reference's speedDistance from reference: 3.99f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)202 ms1.5x the reference's speedDistance from reference: 14.2bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Load time

s, lower is better; the best bar is red.

transformers (eager)8.98 sthe referenceDistance from reference: 0 (the reference)

transformers (torch.compile, static cache)1.84 s4.9x the reference's speedDistance from reference: 4.91

Linnet torch (generated source)2.07 s0.89× the speed of transformers, compiledDistance from reference: 1.13transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (torch.compile, inductor)2.01 s0.92× the speed of transformers, compiledDistance from reference: 4.57transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (CUDA graphs)2.00 s0.92× the speed of transformers, compiledDistance from reference: 4.57transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet JAX (XLA, StableHLO)2.30 s3.9x the reference's speedDistance from reference: 3.29transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)2.13 s4.2x the reference's speedDistance from reference: 16.3transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet ONNX f32 -> ONNX Runtime (CUDA)0.25 s36x the reference's speedDistance from reference: 4.81f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (CUDA)0.24 s37x the reference's speedDistance from reference: 0.636f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (CUDA)0.24 s37x the reference's speedDistance from reference: 15bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f32 -> ONNX Runtime (TensorRT)0.25 s37x the reference's speedDistance from reference: 4.81f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (TensorRT)0.24 s37x the reference's speedDistance from reference: 3.99f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)0.24 s37x the reference's speedDistance from reference: 14.2bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Peak GPU memory

What the driver reports the process holding at its peak, in GiB; lower is better; the best bar is red.

transformers (eager)4.10 GiB (4,196 MiB)Distance from reference: 0 (the reference)

transformers (torch.compile, static cache)4.37 GiB (4,472 MiB)Distance from reference: 4.91

Linnet torch (generated source)5.32 GiB (5,448 MiB)Distance from reference: 1.13transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (torch.compile, inductor)5.36 GiB (5,492 MiB)Distance from reference: 4.57transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet torch (CUDA graphs)5.50 GiB (5,632 MiB)Distance from reference: 4.57transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet JAX (XLA, StableHLO)3.61 GiB (3,699 MiB)Distance from reference: 3.29transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet JAX (XLA, generated source)3.53 GiB (3,616 MiB)Distance from reference: 16.3transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions

Linnet ONNX f32 -> ONNX Runtime (CUDA)11.7 GiB (12,022 MiB)Distance from reference: 4.81f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (CUDA)7.57 GiB (7,756 MiB)Distance from reference: 0.636f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (CUDA)8.66 GiB (8,864 MiB)Distance from reference: 15bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f32 -> ONNX Runtime (TensorRT)18.0 GiB (18,440 MiB)Distance from reference: 4.81f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX f16 -> ONNX Runtime (TensorRT)10.9 GiB (11,168 MiB)Distance from reference: 3.99f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Linnet ONNX bf16 -> ONNX Runtime (TensorRT)10.7 GiB (10,936 MiB)Distance from reference: 14.2bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each

Every number

Method	Encode	Transcribe	Load time	Peak GPU memory	Against the stack it replaces	Distance from reference	Notes
PyTorch
transformers	9.21 ms	307 ms	8.98 s	4.10 GiB		0 (the reference)
transformers, compiled	7.27 ms	192 ms	1.84 s	4.37 GiB		4.91
Linnet, generated PyTorch	7.34 ms	338 ms	2.07 s	5.32 GiB	0.57× transformers, compiled	1.13	transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
Linnet, inductor	6.58 ms	159 ms	2.01 s	5.36 GiB	1.21× transformers, compiled	4.57	transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
Linnet, CUDA graphs	5.63 ms	67.4 ms	2.00 s	5.50 GiB	2.85× transformers, compiled	4.57	transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
JAX
Linnet, XLA (StableHLO)	11.7 ms	96.5 ms	2.30 s	3.61 GiB		3.29	transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
Linnet, XLA (generated JAX)	7.32 ms	108 ms	2.13 s	3.53 GiB		16.3	transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
ONNX Runtime and TensorRT
Linnet, ONNX Runtime f32	25.3 ms	158 ms	0.25 s	11.7 GiB		4.81	f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
Linnet, ONNX Runtime f16	28.8 ms	154 ms	0.24 s	7.57 GiB		0.636	f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
Linnet, ONNX Runtime bf16	21.0 ms	160 ms	0.24 s	8.66 GiB		15	bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
Linnet, TensorRT f32	25.0 ms	250 ms	0.25 s	18.0 GiB		4.81	f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
Linnet, TensorRT f16	5.42 ms	194 ms	0.24 s	10.9 GiB		3.99	f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each
Linnet, TensorRT bf16	6.13 ms	202 ms	0.24 s	10.7 GiB		14.2	bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each