Nestmodels for Linnet
Models / mistral-7b-instruct-v0.3

Mistral 7B Instruct v0.3

A 7B-parameter Llama-architecture decoder with grouped-query attention, a 32768-token vocabulary and context, and an untied output head, instruction-tuned.

7.2B parametersllamaApache-2.0text-generationdecoder-onlygrouped-query-attentionchat
Hugging FaceGitHubarXiv Registry entry

A 7.25B-parameter decoder with the Llama architecture, instruction-tuned: grouped-query attention (32 query heads, 8 key/value heads, head dimension 128), rotary positions with base 1000000, RMS normalization, a SwiGLU MLP, and an untied output head, with a KV cache for token-by-token decoding. The context is 32768 tokens; v0.3 has no sliding window (config.json sets sliding_window to null), so attention is full rather than windowed, and its vocabulary of 32768 tokens is larger than v0.2's.

The Linnet source is the same Llama package the other decoders in this zoo use (tinyllama-1.1b-chat, smollm2-1.7b-instruct), with THETA set to this model's rotary base and the generics set to its width, head counts, and depth. Unlike those two, the output head is not tied to the embedding: bindings.json maps lm_head.weight to the checkpoint's own separate lm_head.weight tensor.

Loading ​

python
from linnet import nest

model = nest.load("mistral-7b-instruct-v0.3", backend="torch", numerics="fast")
logits = model(tokens)                      # Tensor[1, S; i32] -> Tensor[1, S, 32768; bf16]

The weights are the published, sharded model-0000N-of-00003.safetensors (bf16); bindings.json maps Linnet parameter paths to their tensor names. Every name, shape, and dtype is checked before anything runs.

Entries ​

Entry
forward<B, S>(tokens)logits for a whole sequence
next_token<B, S>(tokens)logits for the last position
decode(token, pos)one token through the KV caches (Batch = 1, MaxSeq = 32768)
prefill<S>(tokens, pos)a whole prompt through the KV caches in one pass; logits after its last token, decode continues at pos + S
prefill_slots<M, S>(tokens, slots, lengths)M requests' prompts into rows slots of the caches in one pass (each padded to S, its first lengths[m] tokens real), as they join a batch being served
prefill_packed<P>(tokens, rows, positions, segments, last)several requests' prompts packed end to end into one pass of P tokens, each token given its cache row, its position, and its prompt: no padding between prompts, and each prompt sees only itself
decode_rows(tokens, positions)one token for every row of the caches, each at its own position: the step linnet.serve takes for continuous batching
prefill_paged<P, Rows, Pages>(tokens, positions, rows, slots, last, table)for serving from pages: prompts packed end to end, each token written at its place in the pool and attending over its row's pages up to its position (chunks, shared prefixes)
decode_paged<Rows, Pages>(tokens, positions, table)decode_rows for serving from pages: loaded with Batch = 1, the caches' one row is a pool of MaxSeq positions in pages of PageSize (64), and row b's positions lie in the pages table[b] lists
step_paged<P, Rows, Pages>(...)prefill_paged and decode_paged in one pass
generate<Steps>(token, pos)greedy decoding in the graph
sample<Steps>(token, pos, key, temperature)sampling with std.random
generate_until<MaxNew>(token, pos, eos)decoding until an end token

Shards ​

Shards (1 unless bound) splits the model across devices: one shard holds Heads / Shards query heads, KvHeads / Shards key and value heads and Inner / Shards hidden units, with the KV caches to match, and the attention's and the MLP's output projections end in std.nn.parallel::all_reduce, the sum over the shards. lm_head holds Vocab / Shards rows of the vocabulary, and std.nn.parallel::all_gather sets the shards' slices of the logits side by side. On one device the sum and the gather are the value itself. linnet.torch.load(..., tensor_parallel=mesh) binds Shards to the mesh size and gives each process its part of the checkpoint. The training entries train split as well: each split computation reads its input through std.nn.parallel::shared, and loss_packed and log_probs_packed run over the vocabulary's parts (std.nn.loss::split_cross_entropy, split_token_log_probs).

Provenance ​

Validation ​

Checked against transformers (MistralForCausalLM) on CPU, torch.no_grad(), under the project's memory discipline for multi-billion-parameter models.

  • Truncated, f32, tight tolerance -- done. Both sides built from only the first 2 of 32 layers' real weights (generics={"Layers": 2, "T": "f32"} on the Linnet side, num_hidden_layers=2 on the reference, both reading the same 21 tensors, cast to f32), prompt "The capital of France is". Max absolute difference on the logits: 4.3e-6; the last-position argmax agrees. (A first attempt left the Linnet side at its native bf16 against the f32 reference and saw a 6e-2 max absolute difference with the same argmax match -- bf16 storage-rounding noise consistent with this zoo's other bf16-only checkpoints, not an architecture bug; the all-f32 comparison above is the one that isolates the architecture.)
  • Whole model, bf16, loose tolerance -- deferred, not run here. Loading all 32 layers in bf16 on both sides needs roughly 14 GB of RSS per side, which this pass would hold on the shared CPU machine this card was written on. By instruction, that measurement is deferred to a batched GPU validation run instead of being run here; treat it as not yet checked until that run reports a top-1 token agreement and a bf16-scale (order 1e-2) logits max absolute difference for this model.

Backends ​

The same source and checkpoint in each framework; numerics and compile are the loaders' options.

PyTorch ​

python
from linnet import nest

model = nest.load("mistral-7b-instruct-v0.3", backend="torch", numerics="fast", compile="inductor")
logits = model(tokens)

JAX ​

python
from linnet import nest

f = nest.load("mistral-7b-instruct-v0.3", backend="jax")          # StableHLO compiled by XLA
logits = f(tokens)

Flax NNX ​

python
from linnet import nest

model = nest.load("mistral-7b-instruct-v0.3", backend="nnx")
logits = model(tokens)

Command line ​

bash
python -m linnet.nest pull mistral-7b-instruct-v0.3
linnet stablehlo --root Model --entry forward --bind Vocab=32768 --bind H=4096 --bind Heads=32 --bind KvHeads=8 --bind Inner=14336 --bind Layers=32 --bind Batch=1 --bind MaxSeq=32768 --bind T=bf16 --bind B=1 --bind S=8 <model dir>/src/lib.linnet