Nestmodels for Linnet
Models / qwen2.5-0.5b-instruct

Qwen2.5 0.5B Instruct

A 0.5B-parameter Qwen2 decoder with grouped-query attention, biased query/key/value projections, and an output head tied to the token embedding, instruction-tuned.

494.5M parametersqwen2Apache-2.0text-generationdecoder-onlygrouped-query-attentionchat
Hugging FaceGitHubarXiv Registry entry

A 0.5B-parameter decoder with the Qwen2 architecture, instruction-tuned. 24 layers of width 896, grouped-query attention (14 query heads, 2 key/value heads, head dimension 64), rotary positions with base 1,000,000, RMS normalization with epsilon 1e-6, and a SwiGLU MLP of width 4864. The query, key, and value projections carry a bias; the output projection and the MLP do not. The output head is tied to the token embedding.

The Linnet source is a self-contained Qwen2 package (src/lib.linnet, src/attention.linnet, src/rope.linnet), adapted from the Llama-style decoder the other Nest models use. std.nn.linear::Linear already carries an optional bias, so the query/key/value bias needs no structural change -- only bindings.json entries for q_proj.bias, k_proj.bias, and v_proj.bias, left unbound (and so none) for o_proj and the MLP projections, which the checkpoint has none of. bindings.json maps both embedding.weight and lm_head.weight to the checkpoint's single model.embed_tokens.weight (tie_word_embeddings in config.json), which is how tied embeddings are expressed without a language feature for them. The epsilon Qwen2.5 uses for RMSNorm (1e-6) is tighter than the stdlib RmsNorm block's default (1e-5), so this model defines its own RmsNorm block that calls std.nn.norm::rms_norm directly with the checkpoint's epsilon.

Loading ​

python
from linnet import nest

model = nest.load("qwen2.5-0.5b-instruct", backend="torch", numerics="fast", compile="inductor")
logits = model(tokens)                      # Tensor[1, S; i32] -> Tensor[1, S, 151936; bf16]

The weights are the published model.safetensors (bf16); bindings.json maps Linnet parameter paths to its tensor names. Every name, shape, and dtype is checked before anything runs.

Entries ​

Entry
forward<B, S>(tokens)logits for a whole sequence
next_token<B, S>(tokens)logits for the last position
decode(token, pos)one token through the KV caches (Batch = 1, MaxSeq = 32768)
prefill<S>(tokens, pos)a whole prompt through the KV caches in one pass; logits after its last token, decode continues at pos + S
prefill_slots<M, S>(tokens, slots, lengths)M requests' prompts into rows slots of the caches in one pass (each padded to S, its first lengths[m] tokens real), as they join a batch being served
prefill_packed<P>(tokens, rows, positions, segments, last)several requests' prompts packed end to end into one pass of P tokens, each token given its cache row, its position, and its prompt: no padding between prompts, and each prompt sees only itself
decode_rows(tokens, positions)one token for every row of the caches, each at its own position: the step linnet.serve takes for continuous batching
prefill_paged<P, Rows, Pages>(tokens, positions, rows, slots, last, table)for serving from pages: prompts packed end to end, each token written at its place in the pool and attending over its row's pages up to its position (chunks, shared prefixes)
decode_paged<Rows, Pages>(tokens, positions, table)decode_rows for serving from pages: loaded with Batch = 1, the caches' one row is a pool of MaxSeq positions in pages of PageSize (64), and row b's positions lie in the pages table[b] lists
step_paged<P, Rows, Pages>(...)prefill_paged and decode_paged in one pass
generate<Steps>(token, pos)greedy decoding in the graph
sample<Steps>(token, pos, key, temperature)sampling with std.random
generate_until<MaxNew>(token, pos, eos)decoding until an end token

Shards ​

Shards (1 unless bound) splits the model across devices: one shard holds Heads / Shards query heads, KvHeads / Shards key and value heads and Inner / Shards hidden units, with the KV caches to match, and the attention's and the MLP's output projections end in std.nn.parallel::all_reduce, the sum over the shards. lm_head holds Vocab / Shards rows of the vocabulary, and std.nn.parallel::all_gather sets the shards' slices of the logits side by side. On one device the sum and the gather are the value itself. linnet.torch.load(..., tensor_parallel=mesh) binds Shards to the mesh size and gives each process its part of the checkpoint. The training entries train split as well: each split computation reads its input through std.nn.parallel::shared, and loss_packed and log_probs_packed run over the vocabulary's parts (std.nn.loss::split_cross_entropy, split_token_log_probs).

Provenance ​

The Linnet forward pass matches transformers logits on the final position to within 2e-3 absolute in f32, with the top-1 next token agreeing, on a prompt run through the model's chat template. The decode entry's cached logits match a full forward pass over the same extended sequence to the same tolerance.

Backends ​

The same source and checkpoint in each framework; numerics and compile are the loaders' options.

PyTorch ​

python
from linnet import nest

model = nest.load("qwen2.5-0.5b-instruct", backend="torch", numerics="fast", compile="inductor")
logits = model(tokens)

JAX ​

python
from linnet import nest

f = nest.load("qwen2.5-0.5b-instruct", backend="jax")          # StableHLO compiled by XLA
logits = f(tokens)

Flax NNX ​

python
from linnet import nest

model = nest.load("qwen2.5-0.5b-instruct", backend="nnx")
logits = model(tokens)

Command line ​

bash
python -m linnet.nest pull qwen2.5-0.5b-instruct
linnet stablehlo --root Model --entry forward --bind Vocab=151936 --bind H=896 --bind Heads=14 --bind KvHeads=2 --bind Inner=4864 --bind Layers=24 --bind Batch=1 --bind MaxSeq=32768 --bind T=bf16 --bind B=1 --bind S=8 <model dir>/src/lib.linnet