Nestmodels for Linnet
Models / qwen3.8-27b

Qwen3.8 27B

A 27B-parameter hybrid decoder: three Gated DeltaNet (linear attention) layers for every gated full-attention layer, zero-centered RMS norms, SwiGLU, and an untied output head; the text model of Qwen3.8-27B, without its vision tower.

26.9B parametersqwen3_5Apache-2.0text-generationdecoder-onlylinear-attentiongated-deltanethybrid
Hugging FaceGitHub Registry entry

The text model of Qwen3.8-27B, a 27B-parameter hybrid decoder of the Qwen3.5 architecture (qwen3_5), without its vision tower or its multi-token-prediction layer. Its 64 layers come in fours: three of linear attention (Gated DeltaNet) and one of gated full attention, each followed by a SwiGLU MLP of width 17408.

  • Gated DeltaNet (src/deltanet.linnet): one projection gives the queries, keys and values (16 query/key heads, 48 value heads, width 128 each); a depthwise causal convolution of width 4 and silu run over them; queries and keys are L2-normalized. Each value head keeps a [128, 128] state that the gated delta rule (std.nn.recurrent) decays by exp(-exp(A_log) * softplus(a + dt_bias)) and writes at rate sigmoid(b). The heads' outputs are RMS-normalized and gated by silu(z).
  • Gated attention (src/attention.linnet): 24 query heads, 4 key/value heads, head dimension 256, a per-head RMS norm on queries and keys, rotary positions on the first quarter of each head (base 10,000,000), and the output scaled by sigmoid of a gate q_proj gives each head.
  • Norms (src/norm.linnet): RMS norms scale by 1 + w (zero-centered), in f32.

The full-attention layers keep key/value caches; the linear-attention layers keep each row's convolution history and its recurrent state in f32. prefill/decode serve one batch at one position, prefill_slots/ decode_rows a server's rows each at its own length (fixed rows; the card has no paged entries yet).

bindings.json maps groups.<g>.linear.<i> to checkpoint layer 4g + i and groups.<g>.full to layer 4g + 3.

Backends ​

The same source and checkpoint in each framework; numerics and compile are the loaders' options.

PyTorch ​

python
from linnet import nest

model = nest.load("qwen3.8-27b", backend="torch", numerics="fast", compile="inductor")
logits = model(tokens)

JAX ​

python
from linnet import nest

f = nest.load("qwen3.8-27b", backend="jax")          # StableHLO compiled by XLA
logits = f(tokens)

Flax NNX ​

python
from linnet import nest

model = nest.load("qwen3.8-27b", backend="nnx")
logits = model(tokens)

Command line ​

bash
python -m linnet.nest pull qwen3.8-27b
linnet stablehlo --root Model --entry forward --bind Vocab=248320 --bind H=5120 --bind Heads=24 --bind KvHeads=4 --bind HeadDim=256 --bind KeyHeads=16 --bind ValueHeads=48 --bind KeyDim=128 --bind ValueDim=128 --bind Conv=10240 --bind Width=4 --bind Inner=17408 --bind Groups=16 --bind Batch=1 --bind MaxSeq=262144 --bind T=bf16 --bind B=1 --bind S=8 <model dir>/src/lib.linnet