Nestmodels for Linnet
Models / qwen3-4b

Qwen3 4B

A 4B-parameter Qwen3 decoder with grouped-query attention, per-head RMS normalization of the query and key projections, unbiased projections, and an output head tied to the token embedding.

4B parametersqwen3Apache-2.0text-generationdecoder-onlygrouped-query-attentionqk-norm
Hugging FaceGitHubarXiv Registry entry

NVIDIA H100 80GB HBM3 · 512 prompt tokens, 128 new · batch 1 · median of 10 runs after 3 warm-ups · measured 2026-09-28T17:48:44+00:00

Linnet 0.1.0, PyTorch 2.14.0, JAX 0.11.2, transformers 5.17.0, diffusers 0.40.0, ONNX Runtime 1.30.0, Python 3.12.3, driver 580.126.09

Linnet against the stack it replaces

Each side in its fastest configuration, on the same GPU and checkpoint. A speed-up is how many times the other's speed; an export is measured against the original checkpoint in the same engine.

vs vLLM, one request1.11× fasterdecode speed
  • Linnet, XLA (generated JAX)255 tok/svLLM230 tok/s
vs the reference, in PyTorch2.00× fasterdecode speed
  • Linnet, CUDA graphs247 tok/stransformers, compiled123 tok/s
vs KerasHub, in JAX1.12× fasterdecode speed
  • Linnet, XLA (generated JAX)255 tok/sKerasHub227 tok/s
exported to vLLM, SGLang, TGI+2.4% the same speeddecode speed
  • vLLM on Linnet's export236 tok/svLLM230 tok/s
vs vLLM, serving1.02× fasterserving throughput
  • linnet.serve, CUDA graphs8,196 tok/svLLM8,048 tok/s
exported, serving+1.4% the same speedserving throughput
  • vLLM on Linnet's export8,158 tok/svLLM8,048 tok/s
vs vLLM, behind Triton3.74× fasterserving throughput
  • Triton, linnet.serve backend7,786 tok/sTriton, vLLM backend2,082 tok/s

Each row also carries its distance from reference: the largest absolute difference between its output and transformers (eager)'s on the same input. bf16 outputs differ by rounding (about 0.1 on logits near 16), so a small number is expected; it is there so that a fast wrong answer cannot look like a win.

Time to first token

ms, lower is better; the best bar is red.

PyTorchJAXONNX Runtime and TensorRTLLM engines010203040mstransformers20.7 mstransformers, compiled13.2 msLinnet, generated PyTorch14.5 msLinnet, inductor8.71 msLinnet, CUDA graphs7.64 msKerasHub24.6 msLinnet, XLA (StableHLO)14.6 msLinnet, XLA (generated JAX)13.2 msLinnet, ONNX Runtime f3238.6 msLinnet, ONNX Runtime f1630.1 msLinnet, ONNX Runtime bf1632.1 msLinnet, TensorRT f3235.9 msLinnet, TensorRT f1619.5 msLinnet, TensorRT bf1619.3 msvLLM11.5 msvLLM on Linnet's export7.47 msllama.cpp on Linnet's GGUF17.5 ms

Decode speed

tok/s, higher is better; the best bar is red.

PyTorchJAXONNX Runtime and TensorRTLLM engines0100200tok/stransformers50.0 tok/stransformers, compiled123 tok/sLinnet, generated PyTorch72.2 tok/sLinnet, inductor168 tok/sLinnet, CUDA graphs247 tok/sKerasHub227 tok/sLinnet, XLA (StableHLO)254 tok/sLinnet, XLA (generated JAX)255 tok/sLinnet, ONNX Runtime f3259.4 tok/sLinnet, ONNX Runtime f1668.7 tok/sLinnet, ONNX Runtime bf1667.4 tok/sLinnet, TensorRT f3245.6 tok/sLinnet, TensorRT f1672.2 tok/sLinnet, TensorRT bf1669.7 tok/svLLM230 tok/svLLM on Linnet's export236 tok/sllama.cpp on Linnet's GGUF250 tok/s

Serving throughput

tok/s, higher is better; the best bar is red.

Serving many requests0250050007500tok/svLLM8,048 tok/slinnet.serve, CUDA graphs8,196 tok/slinnet.serve, XLA6,786 tok/slinnet.serve, ONNX Runtime4,203 tok/svLLM on Linnet's export8,158 tok/sTriton, vLLM backend2,082 tok/sTriton, linnet.serve backend7,786 tok/stransformers, batched850 tok/sKerasHub, static batches384 tok/s

Serving time to first token

ms, lower is better; the best bar is red.

Serving many requests0200040006000mslinnet.serve, CUDA graphs1,666 mslinnet.serve, XLA2,136 mslinnet.serve, ONNX Runtime3,644 msTriton, vLLM backend5,981 msTriton, linnet.serve backend1,747 ms

Load time

s, lower is better; the best bar is red.

PyTorchJAXONNX Runtime and TensorRTLLM enginesServing many requests0200400600stransformers5.74 stransformers, compiled2.71 sLinnet, generated PyTorch1.90 sLinnet, inductor1.79 sLinnet, CUDA graphs1.83 sKerasHub17.0 sLinnet, XLA (StableHLO)5.04 sLinnet, XLA (generated JAX)4.85 sLinnet, ONNX Runtime f320.42 sLinnet, ONNX Runtime f160.45 sLinnet, ONNX Runtime bf160.42 sLinnet, TensorRT f320.42 sLinnet, TensorRT f160.41 sLinnet, TensorRT bf160.43 svLLM95.5 svLLM on Linnet's export68.4 sllama.cpp on Linnet's GGUF29.8 svLLM58.2 slinnet.serve, CUDA graphs5.11 slinnet.serve, XLA8.40 slinnet.serve, ONNX Runtime0.26 svLLM on Linnet's export50.2 sTriton, vLLM backend72.1 sTriton, linnet.serve backend659 stransformers, batched2.31 sKerasHub, static batches17.6 s

Peak GPU memory

What the driver reports the process holding at its peak, in GiB; lower is better; the best bar is red. Not drawn, since theirs is a setting rather than a need: vLLM, vLLM, transformers, batched, Triton, vLLM backend, vLLM on Linnet's export, vLLM on Linnet's export (in the table).

PyTorchJAXONNX Runtime and TensorRTLLM enginesServing many requests02040GiBtransformers8.58 GiBtransformers, compiled8.69 GiBLinnet, generated PyTorch9.14 GiBLinnet, inductor12.4 GiBLinnet, CUDA graphs12.5 GiBKerasHub10.0 GiBLinnet, XLA (StableHLO)9.19 GiBLinnet, XLA (generated JAX)9.19 GiBLinnet, ONNX Runtime f3220.3 GiBLinnet, ONNX Runtime f1610.6 GiBLinnet, ONNX Runtime bf1610.00 GiBLinnet, TensorRT f3244.8 GiBLinnet, TensorRT f1623.6 GiBLinnet, TensorRT bf1623.1 GiBllama.cpp on Linnet's GGUF8.55 GiBlinnet.serve, CUDA graphs18.1 GiBlinnet.serve, XLA15.2 GiBlinnet.serve, ONNX Runtime30.9 GiBTriton, linnet.serve backend18.6 GiBKerasHub, static batches21.1 GiB

Every number

MethodTime to first tokenDecode speedServing throughputServing time to first tokenLoad timePeak GPU memoryAgainst the stack it replacesDistance from referenceNotes
PyTorch
transformers20.7 ms50.0 tok/s––5.74 s8.58 GiB0 (the reference)
transformers, compiled13.2 ms123 tok/s––2.71 s8.69 GiB0.125
Linnet, generated PyTorch14.5 ms72.2 tok/s––1.90 s9.14 GiB0.59× transformers, compiled0.0938KV cache compiled for 768 positions
Linnet, inductor8.71 ms168 tok/s––1.79 s12.4 GiB1.36× transformers, compiled0.188KV cache compiled for 768 positions
Linnet, CUDA graphs7.64 ms247 tok/s––1.83 s12.5 GiB2.00× transformers, compiled0.188KV cache compiled for 768 positions
JAX
KerasHub24.6 ms227 tok/s––17.0 s10.0 GiB0.125peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
Linnet, XLA (StableHLO)14.6 ms254 tok/s––5.04 s9.19 GiB1.12× KerasHub0.0938KV cache compiled for 768 positions; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
Linnet, XLA (generated JAX)13.2 ms255 tok/s––4.85 s9.19 GiB1.12× KerasHub0.188KV cache compiled for 768 positions; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
ONNX Runtime and TensorRT
Linnet, ONNX Runtime f3238.6 ms59.4 tok/s––0.42 s20.3 GiB0.155f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; logits copied to the host each step for the argmax
Linnet, ONNX Runtime f1630.1 ms68.7 tok/s––0.45 s10.6 GiB0.672f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; logits copied to the host each step for the argmax
Linnet, ONNX Runtime bf1632.1 ms67.4 tok/s––0.42 s10.00 GiB0.5bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; logits copied to the host each step for the argmax
Linnet, TensorRT f3235.9 ms45.6 tok/s––0.42 s44.8 GiB0.153f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; logits copied to the host each step for the argmax
Linnet, TensorRT f1619.5 ms72.2 tok/s––0.41 s23.6 GiB11.4f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; logits copied to the host each step for the argmax
Linnet, TensorRT bf1619.3 ms69.7 tok/s––0.43 s23.1 GiB0.406bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; logits copied to the host each step for the argmax
LLM engines
vLLM11.5 ms230 tok/s––95.5 s68.0 GiBnot comparedreserves a KV-cache pool up front (gpu_memory_utilization 0.85), so its memory is a setting, not a need
vLLM on Linnet's export7.47 ms236 tok/s––68.4 s68.0 GiB+2.4% from vLLMnot comparedlinnet.hf.export (8 s), then vLLM on the exported checkpoint; reserves a KV-cache pool up front (gpu_memory_utilization 0.85), so its memory is a setting, not a need
llama.cpp on Linnet's GGUF17.5 ms250 tok/s––29.8 s8.55 GiBnot comparedlinnet.gguf.export, then llama-bench with every layer on the GPU; the first token is the prompt at llama-bench's prompt rate
Serving many requests
vLLM––8,048 tok/s–58.2 s68.4 GiBnot compared256 requests of 128-512 prompt tokens and 128 new tokens, at most 64 in flight; KV-cache pool at gpu_memory_utilization 0.85
linnet.serve, CUDA graphs––8,196 tok/s1,666 ms5.11 s18.1 GiB1.02× vLLMnot compared256 requests of 128-512 prompt tokens and 128 new tokens, at most 64 in flight; 64 fixed cache rows of 640 positions
linnet.serve, XLA––6,786 tok/s2,136 ms8.40 s15.2 GiB0.84× vLLMnot compared256 requests of 128-512 prompt tokens and 128 new tokens, at most 64 in flight; 64 fixed cache rows of 640 positions; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions
linnet.serve, ONNX Runtime––4,203 tok/s3,644 ms0.26 s30.9 GiB0.52× vLLMnot compared256 requests of 128-512 prompt tokens and 128 new tokens, at most 64 in flight; 64 fixed cache rows of 640 positions
vLLM on Linnet's export––8,158 tok/s–50.2 s68.4 GiB+1.4% from vLLMnot comparedlinnet.hf.export (0 s), then vLLM (offline, continuous batching) on the exported checkpoint; 256 requests of 128-512 prompt tokens and 128 new tokens, at most 64 in flight; KV-cache pool at gpu_memory_utilization 0.85
Triton, vLLM backend––2,082 tok/s5,981 ms72.1 s67.9 GiBnot compared256 requests of 128-512 prompt tokens and 128 new tokens, at most 64 in flight; over HTTP, streamed, first tokens timed at the client
Triton, linnet.serve backend––7,786 tok/s1,747 ms659 s18.6 GiB3.74× Triton, vLLM backendnot compared256 requests of 128-512 prompt tokens and 128 new tokens, at most 64 in flight; over HTTP, streamed, first tokens timed at the client
transformers, batched––850 tok/s–2.31 s79.1 GiBnot compared256 requests of 128-512 prompt tokens and 128 new tokens, at most 64 in flight; paged|sdpa attention; reserves a paged KV-cache pool up front, so its memory is a setting, not a need; no per-request timestamps
KerasHub, static batches––384 tok/s–17.6 s21.1 GiBnot compared256 requests of 128-512 prompt tokens and 128 new tokens, at most 64 in flight; every batch runs to the longest possible prompt plus the new tokens; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions