Nestmodels for Linnet
Models / whisper-large-v3

Whisper large-v3

OpenAI's largest multilingual speech recognition model: a convolutional stem over a 128-bin log-mel spectrogram, 32 encoder layers, and a 32-layer decoder that cross-attends to them, at width 1280.

1.5B parameterswhisperApache-2.0automatic-speech-recognitionaudioencoder-decodercross-attentionmultilingual
Hugging FaceGitHubarXiv Registry entry
models/whisper-large-v38 files · 148.6 kB in the registry · GitHub
README.md9.5 kB
md
# Whisper large-v3

OpenAI's largest multilingual speech recognition model, and the biggest
re-parameterization of `whisper-tiny`'s architecture: an encoder that reads a
30-second window of audio as a **128-channel** log-mel spectrogram of 3000
frames, halves the frame rate with two convolutions, and runs **32** pre-norm
transformer layers of width **1280** with **20** attention heads over the
1500 positions that remain. The decoder is an ordinary causal language model
of the same width and 32 layers over a **51866-token** multilingual
vocabulary, with a cross-attention over those encoder states inserted
between its self-attention and its MLP, and an output head tied to its token
embedding.

Whisper large-v3's only architectural difference from `whisper-tiny` is the
mel filterbank: 128 bins instead of 80, per its `config.json`
(`num_mel_bins`). Width, head count, depth, and inner size differ as any two
sizes of a family do. The decoder's learned position table stays
`[448, 1280]` -- `max_target_positions` is 448 in both checkpoints, so
`MaxTokens` is unchanged -- and the tied head grows only because the
vocabulary does: large-v3 adds one token over tiny's 51865, all other
special-token ids besides that addition are shifted by it (`<|transcribe|>`
is 50360 here, not 50359). Everything else -- the stem, the attention block
serving both self- and cross-attention, the missing `k_proj.bias` -- is
`whisper-tiny`'s source with new generic values; see that card's README for
how each piece is built.

## The convolutional stem

The stem is two 1-D convolutions of kernel three over the mel channels, the
first at stride 1 and the second at stride 2, each followed by GELU. The
standard library has no `conv1d` -- `std.nn.conv::conv2d` takes a square
kernel -- so the source carries its own hand-built 1-D convolution, the way
`whisper-tiny`'s does: an index expression may not do arithmetic, so the tap
positions are a tensor made from `iota` and the padded signal is gathered
through it, then the convolution finishes as one `linear` against the
checkpoint's `[Cout, Cin, K]` kernel flattened over (channel, tap).

## The key projection has no bias

In both stacks, queries, values and the output projection carry a bias and
keys do not, exactly as in `whisper-tiny`. `k_proj.bias` is left out of
`bindings.json`, `Linear`'s bias is an optional parameter, and the absent
optional makes the `none` branch of `std.nn.linear::linear` run.

## Entries

Transcription encodes once and decodes many times, so the two stacks are
separate entries rather than one forward pass, as in `whisper-tiny`.

```python
from linnet import nest

model = nest.load("whisper-large-v3", backend="torch")
states = model.run_entry("encode", [mel])            # [B, 128, 3000] -> [B, 1500, 1280]
logits = model.run_entry("decode", [tokens, states])  # -> [B, S, 51866]
```

`mel` is what `WhisperFeatureExtractor` produces for this checkpoint
(`input_features`): a log-mel spectrogram of 128 bins and 3000 frames,
padded to 30 seconds -- note that this is *not* the 80-bin extractor
`whisper-tiny` and earlier Whisper sizes use. `tokens` begins with Whisper's
prompt of special tokens, for example
`<|startoftranscript|> <|en|> <|transcribe|> <|notimestamps|>`, which the
Hub repository's tokenizer supplies (`large-v3`'s ids for these differ
slightly from smaller Whisper checkpoints', since one extra vocabulary token
shifts everything after it; do not reuse another checkpoint's hardcoded
ids).

| Entry | |
| --- | --- |
| `encode<B>(mel)` | encoder states for a 30-second window |
| `decode<B, S, A>(tokens, audio)` | logits for every token position |

`decode` recomputes the whole prefix each step; the cached entries below
do not. The encoder attends over all 1500 positions, including the silence
a shorter clip was padded with, exactly as the reference encoder does;
`std.nn.attention::attention` takes an unbatched `Tensor[Q, K; bool]` mask,
so a per-sequence padding mask could not be expressed even if the reference
used one.

## Transcribing with caches

`listen`, `prefill`, and `step` transcribe without recomputing: each
decoder layer keeps the keys and values of every token so far (`MaxTokens`
positions) and those of the encoder's states (1500 positions), sized by the
`Batch` generic (1 by default).

```python
model.run_entry("listen", [mel])                    # encode; fill the cross-attention caches
logits = model.run_entry("prefill", [prompt])       # the prompt's tokens; logits after it
logits = model.run_entry("step", [token, position]) # one more token at `position`
```

| Entry | |
| --- | --- |
| `listen(mel)` | encoder states, and every decoder layer's cross-attention keys and values |
| `prefill<S>(tokens)` | the prompt from position 0 into the self-attention caches; logits after its last token |
| `step(token, pos)` | one token at `pos` over the caches; its logits |

Greedy transcription of the benchmark's clip gives the same tokens through
these entries as through `decode` over the whole prefix on PyTorch (CUDA
graphs), and the same on ONNX Runtime. JAX in the card's f16 rounds one
near-tie the other way and drops the transcript's opening quotation mark and
a comma; in f32 it gives the same text.

## Numerics

Whisper's `config.json` asks for plain `gelu`, the error-function form. The
source uses `std.nn.activations::gelu_erf`, which carries the same
Abramowitz-and-Stegun polynomial `whisper-tiny` hand-rolled before that
stdlib op existed (accurate to within 5e-7 of `F.gelu` in f32), not the tanh
approximation (`std.nn.activations::gelu`). Under `numerics="fast"` --
the default, what `nest.load` uses -- `gelu_erf` is not even evaluated as
that polynomial: `linnet explain` shows both GELU ops as registered stdlib
`op`s with a fused candidate, so the backend calls `torch.nn.functional.gelu`
directly, with `approximate='none'` for `gelu_erf` and `approximate='tanh'`
for `gelu`. A hand-rolled private `fn`, which is what `whisper-tiny` used
before `gelu_erf` existed, has no such registration and always falls back to
evaluating the canonical polynomial regardless of tier; moving the same math
into a stdlib `op` is not just deduplication, it lets `fast` numerics select
the exact PyTorch kernel instead.

`large-v3` has eight times `whisper-tiny`'s depth (32 encoder layers against
4) to compound whatever an activation gets wrong, so getting the form right
matters more here, not less. To measure what getting it wrong would have
cost, both GELU forms were run through the **same** truncated model (below)
and compared to the same truncated reference, real weights, f32:

| encoder states, truncated 2 encoder / 2 decoder layers | max abs diff |
| --- | --- |
| `gelu_erf` (shipped), mel from `N(0, 1)` | 1.9e-5 |
| `gelu_erf` (shipped), mel from `U(-1, 1)` | 1.5e-5 |
| tanh `gelu` (ablation, not shipped), mel from `N(0, 1)` | 5.5e-2 |
| tanh `gelu` (ablation, not shipped), mel from `U(-1, 1)` | 6.9e-2 |

At only 2 of the 32 encoder layers, the tanh approximation is already
roughly 2,900-4,600x worse than `gelu_erf` on this checkpoint, and on the
decoder side the ablation's end-to-end logits reach 2.6e-1 and 2.8e-1 max
abs diff and its **argmax disagrees with the reference at one of the two
inputs** -- a wrong transcription, not just a rounding difference, from two
layers alone. `gelu_erf` is the correct choice and the only one shipped.

### Validation

`check` only proves shapes and exports. Numeric validation for a 1.5B model
follows the two-pass scheme for large checkpoints: this card reports the
first pass only -- **truncated, f32, real weights, tight tolerance** -- and
defers the second pass, the whole model in bf16 for full-depth agreement, to
a batched GPU run alongside the rest of the registry's large models.

Both sides -- `WhisperForConditionalGeneration` and the Linnet model -- were
built with `EncoderLayers = DecoderLayers = 2`, so both read only the first
two layers of each stack out of the real `openai/whisper-large-v3`
checkpoint. In f32 on CPU, with a fixed random mel and the five-token prefix
above:

| | max abs diff |
| --- | --- |
| encoder states, mel from `N(0, 1)` | 1.9e-5 |
| encoder states, mel from `U(-1, 1)`, a real log-mel's range | 1.5e-5 |
| decoder logits, end to end, mel from `N(0, 1)` | 4.3e-6 |
| decoder logits, end to end, mel from `U(-1, 1)` | 4.2e-6 |
| decoder logits, on the reference's own encoder states, mel from `N(0, 1)` | 4.1e-6 |
| decoder logits, on the reference's own encoder states, mel from `U(-1, 1)` | 4.3e-6 |

The argmax agrees at every position in every case, well inside the usual
2e-3 tolerance -- tighter even than `whisper-tiny`'s own reported four-layer
figures (1.2e-4 to 2.4e-4 on encoder states), consistent with `gelu_erf`
costing nothing here (the fused kernel runs, not the polynomial) and with
only 2 of 32 layers having had the chance to accumulate ordinary f32
rounding.

Peak RSS for the whole pass (truncated reference, released, then the shipped
model, released, then the tanh-gelu ablation) was 1.5 GiB.

**The whole-model bf16 pass is deferred.** Per the current plan for this
registry, large-model whole-checkpoint agreement (full 32+32 depth, top-1
token agreement, logits max abs diff against bf16 rounding) runs once on a
GPU in a batch with every other large model's measurements, rather than on
this CPU machine per model. Only the truncated f32 pass above ran here.

## Provenance

- Weights: [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3), Apache-2.0.
- Code: [openai/whisper](https://github.com/openai/whisper).
- Paper: [Robust Speech Recognition via Large-Scale Weak Supervision](https://arxiv.org/abs/2212.04356).

Open on GitHub

bench.json7.8 kB
json
{
  "date": "2026-09-28T19:56:30+00:00",
  "environment": {
    "python": "3.12.3",
    "machine": "x86_64",
    "torch": "2.14.0",
    "jax": "0.11.2",
    "transformers": "5.17.0",
    "diffusers": "0.40.0",
    "onnxruntime": "1.30.0",
    "linnet": "0.1.0",
    "gpu": "NVIDIA H100 80GB HBM3",
    "gpus": "1",
    "driver": "580.126.09"
  },
  "workload": {
    "device": "cuda",
    "prompt_tokens": 512,
    "new_tokens": 128,
    "batch": 1,
    "warmup": 3,
    "iters": 10,
    "seed": 0,
    "offload_gib": 8.0,
    "serve_requests": 256,
    "serve_concurrency": 64,
    "serve_prompt_min": 128,
    "serve_prompt_max": 512,
    "serve_new": 128
  },
  "methods": [
    {
      "method": "transformers (eager)",
      "kind": "reference",
      "metrics": {
        "encode_ms": 9.206102229654789,
        "transcribe_ms": 307.3985744267702,
        "load_s": 8.97533887065947,
        "peak_vram_mib": 4196.0
      },
      "max_abs_diff": 0.0,
      "notes": "",
      "error": null,
      "key": "transformers-eager"
    },
    {
      "method": "transformers (torch.compile, static cache)",
      "kind": "reference",
      "metrics": {
        "encode_ms": 7.272509858012199,
        "transcribe_ms": 192.2719506546855,
        "load_s": 1.8416809886693954,
        "peak_vram_mib": 4472.0
      },
      "max_abs_diff": 4.90625,
      "notes": "",
      "error": null,
      "key": "transformers-compile"
    },
    {
      "method": "Linnet torch (generated source)",
      "kind": "linnet",
      "metrics": {
        "encode_ms": 7.340836804360151,
        "transcribe_ms": 338.20221992209554,
        "load_s": 2.0714575983583927,
        "peak_vram_mib": 5448.0
      },
      "max_abs_diff": 1.125,
      "notes": "transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each",
      "error": null,
      "key": "linnet-torch"
    },
    {
      "method": "Linnet torch (CUDA graphs)",
      "kind": "linnet",
      "metrics": {
        "encode_ms": 5.629597697407007,
        "transcribe_ms": 67.41623114794493,
        "load_s": 2.004842147231102,
        "peak_vram_mib": 5632.0
      },
      "max_abs_diff": 4.5703125,
      "notes": "transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each",
      "error": null,
      "key": "linnet-cudagraphs"
    },
    {
      "method": "Linnet torch (torch.compile, inductor)",
      "kind": "linnet",
      "metrics": {
        "encode_ms": 6.58007524907589,
        "transcribe_ms": 159.25538539886475,
        "load_s": 2.0097391046583652,
        "peak_vram_mib": 5492.0
      },
      "max_abs_diff": 4.5703125,
      "notes": "transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each",
      "error": null,
      "key": "linnet-inductor"
    },
    {
      "method": "Linnet JAX (XLA, StableHLO)",
      "kind": "linnet",
      "metrics": {
        "encode_ms": 11.697333306074142,
        "transcribe_ms": 96.54302895069122,
        "load_s": 2.2979840338230133,
        "peak_vram_mib": 3698.5,
        "driver_vram_mib": 4748.9
      },
      "max_abs_diff": 3.2890625,
      "notes": "transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions",
      "error": null,
      "key": "linnet-jax"
    },
    {
      "method": "Linnet JAX (XLA, generated source)",
      "kind": "linnet",
      "metrics": {
        "encode_ms": 7.322512567043304,
        "transcribe_ms": 107.80552588403225,
        "load_s": 2.1348913572728634,
        "peak_vram_mib": 3615.7,
        "driver_vram_mib": 4744.9
      },
      "max_abs_diff": 16.263671875,
      "notes": "transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each; peak memory is JAX's allocator peak; the driver shows its pool, which grows in whole regions",
      "error": null,
      "key": "linnet-jax-source"
    },
    {
      "method": "Linnet ONNX f32 -> ONNX Runtime (CUDA)",
      "kind": "linnet",
      "metrics": {
        "encode_ms": 25.25244653224945,
        "transcribe_ms": 158.34852773696184,
        "load_s": 0.24869217164814472,
        "peak_vram_mib": 12022.0
      },
      "max_abs_diff": 4.8056793212890625,
      "notes": "f32; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each",
      "error": null,
      "key": "linnet-onnx-f32"
    },
    {
      "method": "Linnet ONNX f32 -> ONNX Runtime (TensorRT)",
      "kind": "linnet",
      "metrics": {
        "encode_ms": 25.001354981213808,
        "transcribe_ms": 250.2225381322205,
        "load_s": 0.24531798344105482,
        "peak_vram_mib": 18440.0
      },
      "max_abs_diff": 4.8056793212890625,
      "notes": "f32; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each",
      "error": null,
      "key": "linnet-onnx-f32-trt"
    },
    {
      "method": "Linnet ONNX f16 -> ONNX Runtime (CUDA)",
      "kind": "linnet",
      "metrics": {
        "encode_ms": 28.81768671795726,
        "transcribe_ms": 153.90245476737618,
        "load_s": 0.24280712846666574,
        "peak_vram_mib": 7756.0
      },
      "max_abs_diff": 0.636474609375,
      "notes": "f16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each",
      "error": null,
      "key": "linnet-onnx-f16"
    },
    {
      "method": "Linnet ONNX f16 -> ONNX Runtime (TensorRT)",
      "kind": "linnet",
      "metrics": {
        "encode_ms": 5.420684814453125,
        "transcribe_ms": 193.60361387953162,
        "load_s": 0.24492099788039923,
        "peak_vram_mib": 11168.0
      },
      "max_abs_diff": 3.9921875,
      "notes": "f16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each",
      "error": null,
      "key": "linnet-onnx-f16-trt"
    },
    {
      "method": "Linnet ONNX bf16 -> ONNX Runtime (CUDA)",
      "kind": "linnet",
      "metrics": {
        "encode_ms": 21.007349248975515,
        "transcribe_ms": 159.69179524108768,
        "load_s": 0.24099965393543243,
        "peak_vram_mib": 8864.0
      },
      "max_abs_diff": 15.03515625,
      "notes": "bf16; ONNX Runtime's CUDA execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each",
      "error": null,
      "key": "linnet-onnx-bf16"
    },
    {
      "method": "Linnet ONNX bf16 -> ONNX Runtime (TensorRT)",
      "kind": "linnet",
      "metrics": {
        "encode_ms": 6.134360563009977,
        "transcribe_ms": 202.09650788456202,
        "load_s": 0.2431494677439332,
        "peak_vram_mib": 10936.0
      },
      "max_abs_diff": 14.21875,
      "notes": "bf16; ONNX Runtime's TensorRT execution provider; first calls build the sessions; transcribed with the card's KV caches: `listen` encodes and fills the cross-attention caches, `prefill` feeds the prompt, `step` one token each",
      "error": null,
      "key": "linnet-onnx-bf16-trt"
    }
  ],
  "reference": "transformers-eager"
}

Open on GitHub

bindings.json115.5 kB
json
{
  "conv1_weight": "model.encoder.conv1.weight",
  "conv1_bias": "model.encoder.conv1.bias",
  "conv2_weight": "model.encoder.conv2.weight",
  "conv2_bias": "model.encoder.conv2.bias",
  "encoder_positions": "model.encoder.embed_positions.weight",
  "encoder_layers.0.self_attn_layer_norm.weight": "model.encoder.layers.0.self_attn_layer_norm.weight",
  "encoder_layers.0.self_attn_layer_norm.bias": "model.encoder.layers.0.self_attn_layer_norm.bias",
  "encoder_layers.0.self_attn.q_proj.weight": "model.encoder.layers.0.self_attn.q_proj.weight",
  "encoder_layers.0.self_attn.q_proj.bias": "model.encoder.layers.0.self_attn.q_proj.bias",
  "encoder_layers.0.self_attn.k_proj.weight": "model.encoder.layers.0.self_attn.k_proj.weight",
  "encoder_layers.0.self_attn.v_proj.weight": "model.encoder.layers.0.self_attn.v_proj.weight",
  "encoder_layers.0.self_attn.v_proj.bias": "model.encoder.layers.0.self_attn.v_proj.bias",
  "encoder_layers.0.self_attn.out_proj.weight": "model.encoder.layers.0.self_attn.out_proj.weight",
  "encoder_layers.0.self_attn.out_proj.bias": "model.encoder.layers.0.self_attn.out_proj.bias",
  "encoder_layers.0.final_layer_norm.weight": "model.encoder.layers.0.final_layer_norm.weight",
  "encoder_layers.0.final_layer_norm.bias": "model.encoder.layers.0.final_layer_norm.bias",
  "encoder_layers.0.mlp.fc1.weight": "model.encoder.layers.0.fc1.weight",
  "encoder_layers.0.mlp.fc1.bias": "model.encoder.layers.0.fc1.bias",
  "encoder_layers.0.mlp.fc2.weight": "model.encoder.layers.0.fc2.weight",
  "encoder_layers.0.mlp.fc2.bias": "model.encoder.layers.0.fc2.bias",
  "encoder_layers.1.self_attn_layer_norm.weight": "model.encoder.layers.1.self_attn_layer_norm.weight",
  "encoder_layers.1.self_attn_layer_norm.bias": "model.encoder.layers.1.self_attn_layer_norm.bias",
  "encoder_layers.1.self_attn.q_proj.weight": "model.encoder.layers.1.self_attn.q_proj.weight",
  "encoder_layers.1.self_attn.q_proj.bias": "model.encoder.layers.1.self_attn.q_proj.bias",
  "encoder_layers.1.self_attn.k_proj.weight": "model.encoder.layers.1.self_attn.k_proj.weight",
  "encoder_layers.1.self_attn.v_proj.weight": "model.encoder.layers.1.self_attn.v_proj.weight",
  "encoder_layers.1.self_attn.v_proj.bias": "model.encoder.layers.1.self_attn.v_proj.bias",
  "encoder_layers.1.self_attn.out_proj.weight": "model.encoder.layers.1.self_attn.out_proj.weight",
  "encoder_layers.1.self_attn.out_proj.bias": "model.encoder.layers.1.self_attn.out_proj.bias",
  "encoder_layers.1.final_layer_norm.weight": "model.encoder.layers.1.final_layer_norm.weight",
  "encoder_layers.1.final_layer_norm.bias": "model.encoder.layers.1.final_layer_norm.bias",
  "encoder_layers.1.mlp.fc1.weight": "model.encoder.layers.1.fc1.weight",
  "encoder_layers.1.mlp.fc1.bias": "model.encoder.layers.1.fc1.bias",
  "encoder_layers.1.mlp.fc2.weight": "model.encoder.layers.1.fc2.weight",
  "encoder_layers.1.mlp.fc2.bias": "model.encoder.layers.1.fc2.bias",
  "encoder_layers.2.self_attn_layer_norm.weight": "model.encoder.layers.2.self_attn_layer_norm.weight",
  "encoder_layers.2.self_attn_layer_norm.bias": "model.encoder.layers.2.self_attn_layer_norm.bias",
  "encoder_layers.2.self_attn.q_proj.weight": "model.encoder.layers.2.self_attn.q_proj.weight",
  "encoder_layers.2.self_attn.q_proj.bias": "model.encoder.layers.2.self_attn.q_proj.bias",
  "encoder_layers.2.self_attn.k_proj.weight": "model.encoder.layers.2.self_attn.k_proj.weight",
  "encoder_layers.2.self_attn.v_proj.weight": "model.encoder.layers.2.self_attn.v_proj.weight",
  "encoder_layers.2.self_attn.v_proj.bias": "model.encoder.layers.2.self_attn.v_proj.bias",
  "encoder_layers.2.self_attn.out_proj.weight": "model.encoder.layers.2.self_attn.out_proj.weight",
  "encoder_layers.2.self_attn.out_proj.bias": "model.encoder.layers.2.self_attn.out_proj.bias",
  "encoder_layers.2.final_layer_norm.weight": "model.encoder.layers.2.final_layer_norm.weight",
  "encoder_layers.2.final_layer_norm.bias": "model.encoder.layers.2.final_layer_norm.bias",
  "encoder_layers.2.mlp.fc1.weight": "model.encoder.layers.2.fc1.weight",
  "encoder_layers.2.mlp.fc1.bias": "model.encoder.layers.2.fc1.bias",
  "encoder_layers.2.mlp.fc2.weight": "model.encoder.layers.2.fc2.weight",
  "encoder_layers.2.mlp.fc2.bias": "model.encoder.layers.2.fc2.bias",
  "encoder_layers.3.self_attn_layer_norm.weight": "model.encoder.layers.3.self_attn_layer_norm.weight",
  "encoder_layers.3.self_attn_layer_norm.bias": "model.encoder.layers.3.self_attn_layer_norm.bias",
  "encoder_layers.3.self_attn.q_proj.weight": "model.encoder.layers.3.self_attn.q_proj.weight",
  "encoder_layers.3.self_attn.q_proj.bias": "model.encoder.layers.3.self_attn.q_proj.bias",
  "encoder_layers.3.self_attn.k_proj.weight": "model.encoder.layers.3.self_attn.k_proj.weight",
  "encoder_layers.3.self_attn.v_proj.weight": "model.encoder.layers.3.self_attn.v_proj.weight",
  "encoder_layers.3.self_attn.v_proj.bias": "model.encoder.layers.3.self_attn.v_proj.bias",
  "encoder_layers.3.self_attn.out_proj.weight": "model.encoder.layers.3.self_attn.out_proj.weight",
  "encoder_layers.3.self_attn.out_proj.bias": "model.encoder.layers.3.self_attn.out_proj.bias",
  "encoder_layers.3.final_layer_norm.weight": "model.encoder.layers.3.final_layer_norm.weight",
  "encoder_layers.3.final_layer_norm.bias": "model.encoder.layers.3.final_layer_norm.bias",
  "encoder_layers.3.mlp.fc1.weight": "model.encoder.layers.3.fc1.weight",
  "encoder_layers.3.mlp.fc1.bias": "model.encoder.layers.3.fc1.bias",
  "encoder_layers.3.mlp.fc2.weight": "model.encoder.layers.3.fc2.weight",
  "encoder_layers.3.mlp.fc2.bias": "model.encoder.layers.3.fc2.bias",
  "encoder_layers.4.self_attn_layer_norm.weight": "model.encoder.layers.4.self_attn_layer_norm.weight",
  "encoder_layers.4.self_attn_layer_norm.bias": "model.encoder.layers.4.self_attn_layer_norm.bias",
  "encoder_layers.4.self_attn.q_proj.weight": "model.encoder.layers.4.self_attn.q_proj.weight",
  "encoder_layers.4.self_attn.q_proj.bias": "model.encoder.layers.4.self_attn.q_proj.bias",
  "encoder_layers.4.self_attn.k_proj.weight": "model.encoder.layers.4.self_attn.k_proj.weight",
  "encoder_layers.4.self_attn.v_proj.weight": "model.encoder.layers.4.self_attn.v_proj.weight",
  "encoder_layers.4.self_attn.v_proj.bias": "model.encoder.layers.4.self_attn.v_proj.bias",
  "encoder_layers.4.self_attn.out_proj.weight": "model.encoder.layers.4.self_attn.out_proj.weight",
  "encoder_layers.4.self_attn.out_proj.bias": "model.encoder.layers.4.self_attn.out_proj.bias",
  "encoder_layers.4.final_layer_norm.weight": "model.encoder.layers.4.final_layer_norm.weight",
  "encoder_layers.4.final_layer_norm.bias": "model.encoder.layers.4.final_layer_norm.bias",
  "encoder_layers.4.mlp.fc1.weight": "model.encoder.layers.4.fc1.weight",
  "encoder_layers.4.mlp.fc1.bias": "model.encoder.layers.4.fc1.bias",
  "encoder_layers.4.mlp.fc2.weight": "model.encoder.layers.4.fc2.weight",
  "encoder_layers.4.mlp.fc2.bias": "model.encoder.layers.4.fc2.bias",
  "encoder_layers.5.self_attn_layer_norm.weight": "model.encoder.layers.5.self_attn_layer_norm.weight",
  "encoder_layers.5.self_attn_layer_norm.bias": "model.encoder.layers.5.self_attn_layer_norm.bias",
  "encoder_layers.5.self_attn.q_proj.weight": "model.encoder.layers.5.self_attn.q_proj.weight",
  "encoder_layers.5.self_attn.q_proj.bias": "model.encoder.layers.5.self_attn.q_proj.bias",
  "encoder_layers.5.self_attn.k_proj.weight": "model.encoder.layers.5.self_attn.k_proj.weight",
  "encoder_layers.5.self_attn.v_proj.weight": "model.encoder.layers.5.self_attn.v_proj.weight",
  "encoder_layers.5.self_attn.v_proj.bias": "model.encoder.layers.5.self_attn.v_proj.bias",
  "encoder_layers.5.self_attn.out_proj.weight": "model.encoder.layers.5.self_attn.out_proj.weight",
  "encoder_layers.5.self_attn.out_proj.bias": "model.encoder.layers.5.self_attn.out_proj.bias",
  "encoder_layers.5.final_layer_norm.weight": "model.encoder.layers.5.final_layer_norm.weight",
  "encoder_layers.5.final_layer_norm.bias": "model.encoder.layers.5.final_layer_norm.bias",
  "encoder_layers.5.mlp.fc1.weight": "model.encoder.layers.5.fc1.weight",
  "encoder_layers.5.mlp.fc1.bias": "model.encoder.layers.5.fc1.bias",
  "encoder_layers.5.mlp.fc2.weight": "model.encoder.layers.5.fc2.weight",
  "encoder_layers.5.mlp.fc2.bias": "model.encoder.layers.5.fc2.bias",
  "encoder_layers.6.self_attn_layer_norm.weight": "model.encoder.layers.6.self_attn_layer_norm.weight",
  "encoder_layers.6.self_attn_layer_norm.bias": "model.encoder.layers.6.self_attn_layer_norm.bias",
  "encoder_layers.6.self_attn.q_proj.weight": "model.encoder.layers.6.self_attn.q_proj.weight",
  "encoder_layers.6.self_attn.q_proj.bias": "model.encoder.layers.6.self_attn.q_proj.bias",
  "encoder_layers.6.self_attn.k_proj.weight": "model.encoder.layers.6.self_attn.k_proj.weight",
  "encoder_layers.6.self_attn.v_proj.weight": "model.encoder.layers.6.self_attn.v_proj.weight",
  "encoder_layers.6.self_attn.v_proj.bias": "model.encoder.layers.6.self_attn.v_proj.bias",
  "encoder_layers.6.self_attn.out_proj.weight": "model.encoder.layers.6.self_attn.out_proj.weight",
  "encoder_layers.6.self_attn.out_proj.bias": "model.encoder.layers.6.self_attn.out_proj.bias",
  "encoder_layers.6.final_layer_norm.weight": "model.encoder.layers.6.final_layer_norm.weight",
  "encoder_layers.6.final_layer_norm.bias": "model.encoder.layers.6.final_layer_norm.bias",
  "encoder_layers.6.mlp.fc1.weight": "model.encoder.layers.6.fc1.weight",
  "encoder_layers.6.mlp.fc1.bias": "model.encoder.layers.6.fc1.bias",
  "encoder_layers.6.mlp.fc2.weight": "model.encoder.layers.6.fc2.weight",
  "encoder_layers.6.mlp.fc2.bias": "model.encoder.layers.6.fc2.bias",
  "encoder_layers.7.self_attn_layer_norm.weight": "model.encoder.layers.7.self_attn_layer_norm.weight",
  "encoder_layers.7.self_attn_layer_norm.bias": "model.encoder.layers.7.self_attn_layer_norm.bias",
  "encoder_layers.7.self_attn.q_proj.weight": "model.encoder.layers.7.self_attn.q_proj.weight",
  "encoder_layers.7.self_attn.q_proj.bias": "model.encoder.layers.7.self_attn.q_proj.bias",
  "encoder_layers.7.self_attn.k_proj.weight": "model.encoder.layers.7.self_attn.k_proj.weight",
  "encoder_layers.7.self_attn.v_proj.weight": "model.encoder.layers.7.self_attn.v_proj.weight",
  "encoder_layers.7.self_attn.v_proj.bias": "model.encoder.layers.7.self_attn.v_proj.bias",
  "encoder_layers.7.self_attn.out_proj.weight": "model.encoder.layers.7.self_attn.out_proj.weight",
  "encoder_layers.7.self_attn.out_proj.bias": "model.encoder.layers.7.self_attn.out_proj.bias",
  "encoder_layers.7.final_layer_norm.weight": "model.encoder.layers.7.final_layer_norm.weight",
  "encoder_layers.7.final_layer_norm.bias": "model.encoder.layers.7.final_layer_norm.bias",
  "encoder_layers.7.mlp.fc1.weight": "model.encoder.layers.7.fc1.weight",
  "encoder_layers.7.mlp.fc1.bias": "model.encoder.layers.7.fc1.bias",
  "encoder_layers.7.mlp.fc2.weight": "model.encoder.layers.7.fc2.weight",
  "encoder_layers.7.mlp.fc2.bias": "model.encoder.layers.7.fc2.bias",
  "encoder_layers.8.self_attn_layer_norm.weight": "model.encoder.layers.8.self_attn_layer_norm.weight",
  "encoder_layers.8.self_attn_layer_norm.bias": "model.encoder.layers.8.self_attn_layer_norm.bias",
  "encoder_layers.8.self_attn.q_proj.weight": "model.encoder.layers.8.self_attn.q_proj.weight",
  "encoder_layers.8.self_attn.q_proj.bias": "model.encoder.layers.8.self_attn.q_proj.bias",
  "encoder_layers.8.self_attn.k_proj.weight": "model.encoder.layers.8.self_attn.k_proj.weight",
  "encoder_layers.8.self_attn.v_proj.weight": "model.encoder.layers.8.self_attn.v_proj.weight",
  "encoder_layers.8.self_attn.v_proj.bias": "model.encoder.layers.8.self_attn.v_proj.bias",
  "encoder_layers.8.self_attn.out_proj.weight": "model.encoder.layers.8.self_attn.out_proj.weight",
  "encoder_layers.8.self_attn.out_proj.bias": "model.encoder.layers.8.self_attn.out_proj.bias",
  "encoder_layers.8.final_layer_norm.weight": "model.encoder.layers.8.final_layer_norm.weight",
  "encoder_layers.8.final_layer_norm.bias": "model.encoder.layers.8.final_layer_norm.bias",
  "encoder_layers.8.mlp.fc1.weight": "model.encoder.layers.8.fc1.weight",
  "encoder_layers.8.mlp.fc1.bias": "model.encoder.layers.8.fc1.bias",
  "encoder_layers.8.mlp.fc2.weight": "model.encoder.layers.8.fc2.weight",
  "encoder_layers.8.mlp.fc2.bias": "model.encoder.layers.8.fc2.bias",
  "encoder_layers.9.self_attn_layer_norm.weight": "model.encoder.layers.9.self_attn_layer_norm.weight",
  "encoder_layers.9.self_attn_layer_norm.bias": "model.encoder.layers.9.self_attn_layer_norm.bias",
  "encoder_layers.9.self_attn.q_proj.weight": "model.encoder.layers.9.self_attn.q_proj.weight",
  "encoder_layers.9.self_attn.q_proj.bias": "model.encoder.layers.9.self_attn.q_proj.bias",
  "encoder_layers.9.self_attn.k_proj.weight": "model.encoder.layers.9.self_attn.k_proj.weight",
  "encoder_layers.9.self_attn.v_proj.weight": "model.encoder.layers.9.self_attn.v_proj.weight",
  "encoder_layers.9.self_attn.v_proj.bias": "model.encoder.layers.9.self_attn.v_proj.bias",
  "encoder_layers.9.self_attn.out_proj.weight": "model.encoder.layers.9.self_attn.out_proj.weight",
  "encoder_layers.9.self_attn.out_proj.bias": "model.encoder.layers.9.self_attn.out_proj.bias",
  "encoder_layers.9.final_layer_norm.weight": "model.encoder.layers.9.final_layer_norm.weight",
  "encoder_layers.9.final_layer_norm.bias": "model.encoder.layers.9.final_layer_norm.bias",
  "encoder_layers.9.mlp.fc1.weight": "model.encoder.layers.9.fc1.weight",
  "encoder_layers.9.mlp.fc1.bias": "model.encoder.layers.9.fc1.bias",
  "encoder_layers.9.mlp.fc2.weight": "model.encoder.layers.9.fc2.weight",
  "encoder_layers.9.mlp.fc2.bias": "model.encoder.layers.9.fc2.bias",
  "encoder_layers.10.self_attn_layer_norm.weight": "model.encoder.layers.10.self_attn_layer_norm.weight",
  "encoder_layers.10.self_attn_layer_norm.bias": "model.encoder.layers.10.self_attn_layer_norm.bias",
  "encoder_layers.10.self_attn.q_proj.weight": "model.encoder.layers.10.self_attn.q_proj.weight",
  "encoder_layers.10.self_attn.q_proj.bias": "model.encoder.layers.10.self_attn.q_proj.bias",
  "encoder_layers.10.self_attn.k_proj.weight": "model.encoder.layers.10.self_attn.k_proj.weight",
  "encoder_layers.10.self_attn.v_proj.weight": "model.encoder.layers.10.self_attn.v_proj.weight",
  "encoder_layers.10.self_attn.v_proj.bias": "model.encoder.layers.10.self_attn.v_proj.bias",
  "encoder_layers.10.self_attn.out_proj.weight": "model.encoder.layers.10.self_attn.out_proj.weight",
  "encoder_layers.10.self_attn.out_proj.bias": "model.encoder.layers.10.self_attn.out_proj.bias",
  "encoder_layers.10.final_layer_norm.weight": "model.encoder.layers.10.final_layer_norm.weight",
  "encoder_layers.10.final_layer_norm.bias": "model.encoder.layers.10.final_layer_norm.bias",
  "encoder_layers.10.mlp.fc1.weight": "model.encoder.layers.10.fc1.weight",
  "encoder_layers.10.mlp.fc1.bias": "model.encoder.layers.10.fc1.bias",
  "encoder_layers.10.mlp.fc2.weight": "model.encoder.layers.10.fc2.weight",
  "encoder_layers.10.mlp.fc2.bias": "model.encoder.layers.10.fc2.bias",
  "encoder_layers.11.self_attn_layer_norm.weight": "model.encoder.layers.11.self_attn_layer_norm.weight",
  "encoder_layers.11.self_attn_layer_norm.bias": "model.encoder.layers.11.self_attn_layer_norm.bias",
  "encoder_layers.11.self_attn.q_proj.weight": "model.encoder.layers.11.self_attn.q_proj.weight",
  "encoder_layers.11.self_attn.q_proj.bias": "model.encoder.layers.11.self_attn.q_proj.bias",
  "encoder_layers.11.self_attn.k_proj.weight": "model.encoder.layers.11.self_attn.k_proj.weight",
  "encoder_layers.11.self_attn.v_proj.weight": "model.encoder.layers.11.self_attn.v_proj.weight",
  "encoder_layers.11.self_attn.v_proj.bias": "model.encoder.layers.11.self_attn.v_proj.bias",
  "encoder_layers.11.self_attn.out_proj.weight": "model.encoder.layers.11.self_attn.out_proj.weight",
  "encoder_layers.11.self_attn.out_proj.bias": "model.encoder.layers.11.self_attn.out_proj.bias",
  "encoder_layers.11.final_layer_norm.weight": "model.encoder.layers.11.final_layer_norm.weight",
  "encoder_layers.11.final_layer_norm.bias": "model.encoder.layers.11.final_layer_norm.bias",
  "encoder_layers.11.mlp.fc1.weight": "model.encoder.layers.11.fc1.weight",
  "encoder_layers.11.mlp.fc1.bias": "model.encoder.layers.11.fc1.bias",
  "encoder_layers.11.mlp.fc2.weight": "model.encoder.layers.11.fc2.weight",
  "encoder_layers.11.mlp.fc2.bias": "model.encoder.layers.11.fc2.bias",
  "encoder_layers.12.self_attn_layer_norm.weight": "model.encoder.layers.12.self_attn_layer_norm.weight",
  "encoder_layers.12.self_attn_layer_norm.bias": "model.encoder.layers.12.self_attn_layer_norm.bias",
  "encoder_layers.12.self_attn.q_proj.weight": "model.encoder.layers.12.self_attn.q_proj.weight",
  "encoder_layers.12.self_attn.q_proj.bias": "model.encoder.layers.12.self_attn.q_proj.bias",
  "encoder_layers.12.self_attn.k_proj.weight": "model.encoder.layers.12.self_attn.k_proj.weight",
  "encoder_layers.12.self_attn.v_proj.weight": "model.encoder.layers.12.self_attn.v_proj.weight",
  "encoder_layers.12.self_attn.v_proj.bias": "model.encoder.layers.12.self_attn.v_proj.bias",
  "encoder_layers.12.self_attn.out_proj.weight": "model.encoder.layers.12.self_attn.out_proj.weight",
  "encoder_layers.12.self_attn.out_proj.bias": "model.encoder.layers.12.self_attn.out_proj.bias",
  "encoder_layers.12.final_layer_norm.weight": "model.encoder.layers.12.final_layer_norm.weight",
  "encoder_layers.12.final_layer_norm.bias": "model.encoder.layers.12.final_layer_norm.bias",
  "encoder_layers.12.mlp.fc1.weight": "model.encoder.layers.12.fc1.weight",
  "encoder_layers.12.mlp.fc1.bias": "model.encoder.layers.12.fc1.bias",
  "encoder_layers.12.mlp.fc2.weight": "model.encoder.layers.12.fc2.weight",
  "encoder_layers.12.mlp.fc2.bias": "model.encoder.layers.12.fc2.bias",
  "encoder_layers.13.self_attn_layer_norm.weight": "model.encoder.layers.13.self_attn_layer_norm.weight",
  "encoder_layers.13.self_attn_layer_norm.bias": "model.encoder.layers.13.self_attn_layer_norm.bias",
  "encoder_layers.13.self_attn.q_proj.weight": "model.encoder.layers.13.self_attn.q_proj.weight",
  "encoder_layers.13.self_attn.q_proj.bias": "model.encoder.layers.13.self_attn.q_proj.bias",
  "encoder_layers.13.self_attn.k_proj.weight": "model.encoder.layers.13.self_attn.k_proj.weight",
  "encoder_layers.13.self_attn.v_proj.weight": "model.encoder.layers.13.self_attn.v_proj.weight",
  "encoder_layers.13.self_attn.v_proj.bias": "model.encoder.layers.13.self_attn.v_proj.bias",
  "encoder_layers.13.self_attn.out_proj.weight": "model.encoder.layers.13.self_attn.out_proj.weight",
  "encoder_layers.13.self_attn.out_proj.bias": "model.encoder.layers.13.self_attn.out_proj.bias",
  "encoder_layers.13.final_layer_norm.weight": "model.encoder.layers.13.final_layer_norm.weight",
  "encoder_layers.13.final_layer_norm.bias": "model.encoder.layers.13.final_layer_norm.bias",
  "encoder_layers.13.mlp.fc1.weight": "model.encoder.layers.13.fc1.weight",
  "encoder_layers.13.mlp.fc1.bias": "model.encoder.layers.13.fc1.bias",
  "encoder_layers.13.mlp.fc2.weight": "model.encoder.layers.13.fc2.weight",
  "encoder_layers.13.mlp.fc2.bias": "model.encoder.layers.13.fc2.bias",
  "encoder_layers.14.self_attn_layer_norm.weight": "model.encoder.layers.14.self_attn_layer_norm.weight",
  "encoder_layers.14.self_attn_layer_norm.bias": "model.encoder.layers.14.self_attn_layer_norm.bias",
  "encoder_layers.14.self_attn.q_proj.weight": "model.encoder.layers.14.self_attn.q_proj.weight",
  "encoder_layers.14.self_attn.q_proj.bias": "model.encoder.layers.14.self_attn.q_proj.bias",
  "encoder_layers.14.self_attn.k_proj.weight": "model.encoder.layers.14.self_attn.k_proj.weight",
  "encoder_layers.14.self_attn.v_proj.weight": "model.encoder.layers.14.self_attn.v_proj.weight",
  "encoder_layers.14.self_attn.v_proj.bias": "model.encoder.layers.14.self_attn.v_proj.bias",
  "encoder_layers.14.self_attn.out_proj.weight": "model.encoder.layers.14.self_attn.out_proj.weight",
  "encoder_layers.14.self_attn.out_proj.bias": "model.encoder.layers.14.self_attn.out_proj.bias",
  "encoder_layers.14.final_layer_norm.weight": "model.encoder.layers.14.final_layer_norm.weight",
  "encoder_layers.14.final_layer_norm.bias": "model.encoder.layers.14.final_layer_norm.bias",
  "encoder_layers.14.mlp.fc1.weight": "model.encoder.layers.14.fc1.weight",
  "encoder_layers.14.mlp.fc1.bias": "model.encoder.layers.14.fc1.bias",
  "encoder_layers.14.mlp.fc2.weight": "model.encoder.layers.14.fc2.weight",
  "encoder_layers.14.mlp.fc2.bias": "model.encoder.layers.14.fc2.bias",
  "encoder_layers.15.self_attn_layer_norm.weight": "model.encoder.layers.15.self_attn_layer_norm.weight",
  "encoder_layers.15.self_attn_layer_norm.bias": "model.encoder.layers.15.self_attn_layer_norm.bias",
  "encoder_layers.15.self_attn.q_proj.weight": "model.encoder.layers.15.self_attn.q_proj.weight",
  "encoder_layers.15.self_attn.q_proj.bias": "model.encoder.layers.15.self_attn.q_proj.bias",
  "encoder_layers.15.self_attn.k_proj.weight": "model.encoder.layers.15.self_attn.k_proj.weight",
  "encoder_layers.15.self_attn.v_proj.weight": "model.encoder.layers.15.self_attn.v_proj.weight",
  "encoder_layers.15.self_attn.v_proj.bias": "model.encoder.layers.15.self_attn.v_proj.bias",
  "encoder_layers.15.self_attn.out_proj.weight": "model.encoder.layers.15.self_attn.out_proj.weight",
  "encoder_layers.15.self_attn.out_proj.bias": "model.encoder.layers.15.self_attn.out_proj.bias",
  "encoder_layers.15.final_layer_norm.weight": "model.encoder.layers.15.final_layer_norm.weight",
  "encoder_layers.15.final_layer_norm.bias": "model.encoder.layers.15.final_layer_norm.bias",
  "encoder_layers.15.mlp.fc1.weight": "model.encoder.layers.15.fc1.weight",
  "encoder_layers.15.mlp.fc1.bias": "model.encoder.layers.15.fc1.bias",
  "encoder_layers.15.mlp.fc2.weight": "model.encoder.layers.15.fc2.weight",
  "encoder_layers.15.mlp.fc2.bias": "model.encoder.layers.15.fc2.bias",
  "encoder_layers.16.self_attn_layer_norm.weight": "model.encoder.layers.16.self_attn_layer_norm.weight",
  "encoder_layers.16.self_attn_layer_norm.bias": "model.encoder.layers.16.self_attn_layer_norm.bias",
  "encoder_layers.16.self_attn.q_proj.weight": "model.encoder.layers.16.self_attn.q_proj.weight",
  "encoder_layers.16.self_attn.q_proj.bias": "model.encoder.layers.16.self_attn.q_proj.bias",
  "encoder_layers.16.self_attn.k_proj.weight": "model.encoder.layers.16.self_attn.k_proj.weight",
  "encoder_layers.16.self_attn.v_proj.weight": "model.encoder.layers.16.self_attn.v_proj.weight",
  "encoder_layers.16.self_attn.v_proj.bias": "model.encoder.layers.16.self_attn.v_proj.bias",
  "encoder_layers.16.self_attn.out_proj.weight": "model.encoder.layers.16.self_attn.out_proj.weight",
  "encoder_layers.16.self_attn.out_proj.bias": "model.encoder.layers.16.self_attn.out_proj.bias",
  "encoder_layers.16.final_layer_norm.weight": "model.encoder.layers.16.final_layer_norm.weight",
  "encoder_layers.16.final_layer_norm.bias": "model.encoder.layers.16.final_layer_norm.bias",
  "encoder_layers.16.mlp.fc1.weight": "model.encoder.layers.16.fc1.weight",
  "encoder_layers.16.mlp.fc1.bias": "model.encoder.layers.16.fc1.bias",
  "encoder_layers.16.mlp.fc2.weight": "model.encoder.layers.16.fc2.weight",
  "encoder_layers.16.mlp.fc2.bias": "model.encoder.layers.16.fc2.bias",
  "encoder_layers.17.self_attn_layer_norm.weight": "model.encoder.layers.17.self_attn_layer_norm.weight",
  "encoder_layers.17.self_attn_layer_norm.bias": "model.encoder.layers.17.self_attn_layer_norm.bias",
  "encoder_layers.17.self_attn.q_proj.weight": "model.encoder.layers.17.self_attn.q_proj.weight",
  "encoder_layers.17.self_attn.q_proj.bias": "model.encoder.layers.17.self_attn.q_proj.bias",
  "encoder_layers.17.self_attn.k_proj.weight": "model.encoder.layers.17.self_attn.k_proj.weight",
  "encoder_layers.17.self_attn.v_proj.weight": "model.encoder.layers.17.self_attn.v_proj.weight",
  "encoder_layers.17.self_attn.v_proj.bias": "model.encoder.layers.17.self_attn.v_proj.bias",
  "encoder_layers.17.self_attn.out_proj.weight": "model.encoder.layers.17.self_attn.out_proj.weight",
  "encoder_layers.17.self_attn.out_proj.bias": "model.encoder.layers.17.self_attn.out_proj.bias",
  "encoder_layers.17.final_layer_norm.weight": "model.encoder.layers.17.final_layer_norm.weight",
  "encoder_layers.17.final_layer_norm.bias": "model.encoder.layers.17.final_layer_norm.bias",
  "encoder_layers.17.mlp.fc1.weight": "model.encoder.layers.17.fc1.weight",
  "encoder_layers.17.mlp.fc1.bias": "model.encoder.layers.17.fc1.bias",
  "encoder_layers.17.mlp.fc2.weight": "model.encoder.layers.17.fc2.weight",
  "encoder_layers.17.mlp.fc2.bias": "model.encoder.layers.17.fc2.bias",
  "encoder_layers.18.self_attn_layer_norm.weight": "model.encoder.layers.18.self_attn_layer_norm.weight",
  "encoder_layers.18.self_attn_layer_norm.bias": "model.encoder.layers.18.self_attn_layer_norm.bias",
  "encoder_layers.18.self_attn.q_proj.weight": "model.encoder.layers.18.self_attn.q_proj.weight",
  "encoder_layers.18.self_attn.q_proj.bias": "model.encoder.layers.18.self_attn.q_proj.bias",
  "encoder_layers.18.self_attn.k_proj.weight": "model.encoder.layers.18.self_attn.k_proj.weight",
  "encoder_layers.18.self_attn.v_proj.weight": "model.encoder.layers.18.self_attn.v_proj.weight",
  "encoder_layers.18.self_attn.v_proj.bias": "model.encoder.layers.18.self_attn.v_proj.bias",
  "encoder_layers.18.self_attn.out_proj.weight": "model.encoder.layers.18.self_attn.out_proj.weight",
  "encoder_layers.18.self_attn.out_proj.bias": "model.encoder.layers.18.self_attn.out_proj.bias",
  "encoder_layers.18.final_layer_norm.weight": "model.encoder.layers.18.final_layer_norm.weight",
  "encoder_layers.18.final_layer_norm.bias": "model.encoder.layers.18.final_layer_norm.bias",
  "encoder_layers.18.mlp.fc1.weight": "model.encoder.layers.18.fc1.weight",
  "encoder_layers.18.mlp.fc1.bias": "model.encoder.layers.18.fc1.bias",
  "encoder_layers.18.mlp.fc2.weight": "model.encoder.layers.18.fc2.weight",
  "encoder_layers.18.mlp.fc2.bias": "model.encoder.layers.18.fc2.bias",
  "encoder_layers.19.self_attn_layer_norm.weight": "model.encoder.layers.19.self_attn_layer_norm.weight",
  "encoder_layers.19.self_attn_layer_norm.bias": "model.encoder.layers.19.self_attn_layer_norm.bias",
  "encoder_layers.19.self_attn.q_proj.weight": "model.encoder.layers.19.self_attn.q_proj.weight",
  "encoder_layers.19.self_attn.q_proj.bias": "model.encoder.layers.19.self_attn.q_proj.bias",
  "encoder_layers.19.self_attn.k_proj.weight": "model.encoder.layers.19.self_attn.k_proj.weight",
  "encoder_layers.19.self_attn.v_proj.weight": "model.encoder.layers.19.self_attn.v_proj.weight",
  "encoder_layers.19.self_attn.v_proj.bias": "model.encoder.layers.19.self_attn.v_proj.bias",
  "encoder_layers.19.self_attn.out_proj.weight": "model.encoder.layers.19.self_attn.out_proj.weight",
  "encoder_layers.19.self_attn.out_proj.bias": "model.encoder.layers.19.self_attn.out_proj.bias",
  "encoder_layers.19.final_layer_norm.weight": "model.encoder.layers.19.final_layer_norm.weight",
  "encoder_layers.19.final_layer_norm.bias": "model.encoder.layers.19.final_layer_norm.bias",
  "encoder_layers.19.mlp.fc1.weight": "model.encoder.layers.19.fc1.weight",
  "encoder_layers.19.mlp.fc1.bias": "model.encoder.layers.19.fc1.bias",
  "encoder_layers.19.mlp.fc2.weight": "model.encoder.layers.19.fc2.weight",
  "encoder_layers.19.mlp.fc2.bias": "model.encoder.layers.19.fc2.bias",
  "encoder_layers.20.self_attn_layer_norm.weight": "model.encoder.layers.20.self_attn_layer_norm.weight",
  "encoder_layers.20.self_attn_layer_norm.bias": "model.encoder.layers.20.self_attn_layer_norm.bias",
  "encoder_layers.20.self_attn.q_proj.weight": "model.encoder.layers.20.self_attn.q_proj.weight",
  "encoder_layers.20.self_attn.q_proj.bias": "model.encoder.layers.20.self_attn.q_proj.bias",
  "encoder_layers.20.self_attn.k_proj.weight": "model.encoder.layers.20.self_attn.k_proj.weight",
  "encoder_layers.20.self_attn.v_proj.weight": "model.encoder.layers.20.self_attn.v_proj.weight",
  "encoder_layers.20.self_attn.v_proj.bias": "model.encoder.layers.20.self_attn.v_proj.bias",
  "encoder_layers.20.self_attn.out_proj.weight": "model.encoder.layers.20.self_attn.out_proj.weight",
  "encoder_layers.20.self_attn.out_proj.bias": "model.encoder.layers.20.self_attn.out_proj.bias",
  "encoder_layers.20.final_layer_norm.weight": "model.encoder.layers.20.final_layer_norm.weight",
  "encoder_layers.20.final_layer_norm.bias": "model.encoder.layers.20.final_layer_norm.bias",
  "encoder_layers.20.mlp.fc1.weight": "model.encoder.layers.20.fc1.weight",
  "encoder_layers.20.mlp.fc1.bias": "model.encoder.layers.20.fc1.bias",
  "encoder_layers.20.mlp.fc2.weight": "model.encoder.layers.20.fc2.weight",
  "encoder_layers.20.mlp.fc2.bias": "model.encoder.layers.20.fc2.bias",
  "encoder_layers.21.self_attn_layer_norm.weight": "model.encoder.layers.21.self_attn_layer_norm.weight",
  "encoder_layers.21.self_attn_layer_norm.bias": "model.encoder.layers.21.self_attn_layer_norm.bias",
  "encoder_layers.21.self_attn.q_proj.weight": "model.encoder.layers.21.self_attn.q_proj.weight",
  "encoder_layers.21.self_attn.q_proj.bias": "model.encoder.layers.21.self_attn.q_proj.bias",
  "encoder_layers.21.self_attn.k_proj.weight": "model.encoder.layers.21.self_attn.k_proj.weight",
  "encoder_layers.21.self_attn.v_proj.weight": "model.encoder.layers.21.self_attn.v_proj.weight",
  "encoder_layers.21.self_attn.v_proj.bias": "model.encoder.layers.21.self_attn.v_proj.bias",
  "encoder_layers.21.self_attn.out_proj.weight": "model.encoder.layers.21.self_attn.out_proj.weight",
  "encoder_layers.21.self_attn.out_proj.bias": "model.encoder.layers.21.self_attn.out_proj.bias",
  "encoder_layers.21.final_layer_norm.weight": "model.encoder.layers.21.final_layer_norm.weight",
  "encoder_layers.21.final_layer_norm.bias": "model.encoder.layers.21.final_layer_norm.bias",
  "encoder_layers.21.mlp.fc1.weight": "model.encoder.layers.21.fc1.weight",
  "encoder_layers.21.mlp.fc1.bias": "model.encoder.layers.21.fc1.bias",
  "encoder_layers.21.mlp.fc2.weight": "model.encoder.layers.21.fc2.weight",
  "encoder_layers.21.mlp.fc2.bias": "model.encoder.layers.21.fc2.bias",
  "encoder_layers.22.self_attn_layer_norm.weight": "model.encoder.layers.22.self_attn_layer_norm.weight",
  "encoder_layers.22.self_attn_layer_norm.bias": "model.encoder.layers.22.self_attn_layer_norm.bias",
  "encoder_layers.22.self_attn.q_proj.weight": "model.encoder.layers.22.self_attn.q_proj.weight",
  "encoder_layers.22.self_attn.q_proj.bias": "model.encoder.layers.22.self_attn.q_proj.bias",
  "encoder_layers.22.self_attn.k_proj.weight": "model.encoder.layers.22.self_attn.k_proj.weight",
  "encoder_layers.22.self_attn.v_proj.weight": "model.encoder.layers.22.self_attn.v_proj.weight",
  "encoder_layers.22.self_attn.v_proj.bias": "model.encoder.layers.22.self_attn.v_proj.bias",
  "encoder_layers.22.self_attn.out_proj.weight": "model.encoder.layers.22.self_attn.out_proj.weight",
  "encoder_layers.22.self_attn.out_proj.bias": "model.encoder.layers.22.self_attn.out_proj.bias",
  "encoder_layers.22.final_layer_norm.weight": "model.encoder.layers.22.final_layer_norm.weight",
  "encoder_layers.22.final_layer_norm.bias": "model.encoder.layers.22.final_layer_norm.bias",
  "encoder_layers.22.mlp.fc1.weight": "model.encoder.layers.22.fc1.weight",
  "encoder_layers.22.mlp.fc1.bias": "model.encoder.layers.22.fc1.bias",
  "encoder_layers.22.mlp.fc2.weight": "model.encoder.layers.22.fc2.weight",
  "encoder_layers.22.mlp.fc2.bias": "model.encoder.layers.22.fc2.bias",
  "encoder_layers.23.self_attn_layer_norm.weight": "model.encoder.layers.23.self_attn_layer_norm.weight",
  "encoder_layers.23.self_attn_layer_norm.bias": "model.encoder.layers.23.self_attn_layer_norm.bias",
  "encoder_layers.23.self_attn.q_proj.weight": "model.encoder.layers.23.self_attn.q_proj.weight",
  "encoder_layers.23.self_attn.q_proj.bias": "model.encoder.layers.23.self_attn.q_proj.bias",
  "encoder_layers.23.self_attn.k_proj.weight": "model.encoder.layers.23.self_attn.k_proj.weight",
  "encoder_layers.23.self_attn.v_proj.weight": "model.encoder.layers.23.self_attn.v_proj.weight",
  "encoder_layers.23.self_attn.v_proj.bias": "model.encoder.layers.23.self_attn.v_proj.bias",
  "encoder_layers.23.self_attn.out_proj.weight": "model.encoder.layers.23.self_attn.out_proj.weight",
  "encoder_layers.23.self_attn.out_proj.bias": "model.encoder.layers.23.self_attn.out_proj.bias",
  "encoder_layers.23.final_layer_norm.weight": "model.encoder.layers.23.final_layer_norm.weight",
  "encoder_layers.23.final_layer_norm.bias": "model.encoder.layers.23.final_layer_norm.bias",
  "encoder_layers.23.mlp.fc1.weight": "model.encoder.layers.23.fc1.weight",
  "encoder_layers.23.mlp.fc1.bias": "model.encoder.layers.23.fc1.bias",
  "encoder_layers.23.mlp.fc2.weight": "model.encoder.layers.23.fc2.weight",
  "encoder_layers.23.mlp.fc2.bias": "model.encoder.layers.23.fc2.bias",
  "encoder_layers.24.self_attn_layer_norm.weight": "model.encoder.layers.24.self_attn_layer_norm.weight",
  "encoder_layers.24.self_attn_layer_norm.bias": "model.encoder.layers.24.self_attn_layer_norm.bias",
  "encoder_layers.24.self_attn.q_proj.weight": "model.encoder.layers.24.self_attn.q_proj.weight",
  "encoder_layers.24.self_attn.q_proj.bias": "model.encoder.layers.24.self_attn.q_proj.bias",
  "encoder_layers.24.self_attn.k_proj.weight": "model.encoder.layers.24.self_attn.k_proj.weight",
  "encoder_layers.24.self_attn.v_proj.weight": "model.encoder.layers.24.self_attn.v_proj.weight",
  "encoder_layers.24.self_attn.v_proj.bias": "model.encoder.layers.24.self_attn.v_proj.bias",
  "encoder_layers.24.self_attn.out_proj.weight": "model.encoder.layers.24.self_attn.out_proj.weight",
  "encoder_layers.24.self_attn.out_proj.bias": "model.encoder.layers.24.self_attn.out_proj.bias",
  "encoder_layers.24.final_layer_norm.weight": "model.encoder.layers.24.final_layer_norm.weight",
  "encoder_layers.24.final_layer_norm.bias": "model.encoder.layers.24.final_layer_norm.bias",
  "encoder_layers.24.mlp.fc1.weight": "model.encoder.layers.24.fc1.weight",
  "encoder_layers.24.mlp.fc1.bias": "model.encoder.layers.24.fc1.bias",
  "encoder_layers.24.mlp.fc2.weight": "model.encoder.layers.24.fc2.weight",
  "encoder_layers.24.mlp.fc2.bias": "model.encoder.layers.24.fc2.bias",
  "encoder_layers.25.self_attn_layer_norm.weight": "model.encoder.layers.25.self_attn_layer_norm.weight",
  "encoder_layers.25.self_attn_layer_norm.bias": "model.encoder.layers.25.self_attn_layer_norm.bias",
  "encoder_layers.25.self_attn.q_proj.weight": "model.encoder.layers.25.self_attn.q_proj.weight",
  "encoder_layers.25.self_attn.q_proj.bias": "model.encoder.layers.25.self_attn.q_proj.bias",
  "encoder_layers.25.self_attn.k_proj.weight": "model.encoder.layers.25.self_attn.k_proj.weight",
  "encoder_layers.25.self_attn.v_proj.weight": "model.encoder.layers.25.self_attn.v_proj.weight",
  "encoder_layers.25.self_attn.v_proj.bias": "model.encoder.layers.25.self_attn.v_proj.bias",
  "encoder_layers.25.self_attn.out_proj.weight": "model.encoder.layers.25.self_attn.out_proj.weight",
  "encoder_layers.25.self_attn.out_proj.bias": "model.encoder.layers.25.self_attn.out_proj.bias",
  "encoder_layers.25.final_layer_norm.weight": "model.encoder.layers.25.final_layer_norm.weight",
  "encoder_layers.25.final_layer_norm.bias": "model.encoder.layers.25.final_layer_norm.bias",
  "encoder_layers.25.mlp.fc1.weight": "model.encoder.layers.25.fc1.weight",
  "encoder_layers.25.mlp.fc1.bias": "model.encoder.layers.25.fc1.bias",
  "encoder_layers.25.mlp.fc2.weight": "model.encoder.layers.25.fc2.weight",
  "encoder_layers.25.mlp.fc2.bias": "model.encoder.layers.25.fc2.bias",
  "encoder_layers.26.self_attn_layer_norm.weight": "model.encoder.layers.26.self_attn_layer_norm.weight",
  "encoder_layers.26.self_attn_layer_norm.bias": "model.encoder.layers.26.self_attn_layer_norm.bias",
  "encoder_layers.26.self_attn.q_proj.weight": "model.encoder.layers.26.self_attn.q_proj.weight",
  "encoder_layers.26.self_attn.q_proj.bias": "model.encoder.layers.26.self_attn.q_proj.bias",
  "encoder_layers.26.self_attn.k_proj.weight": "model.encoder.layers.26.self_attn.k_proj.weight",
  "encoder_layers.26.self_attn.v_proj.weight": "model.encoder.layers.26.self_attn.v_proj.weight",
  "encoder_layers.26.self_attn.v_proj.bias": "model.encoder.layers.26.self_attn.v_proj.bias",
  "encoder_layers.26.self_attn.out_proj.weight": "model.encoder.layers.26.self_attn.out_proj.weight",
  "encoder_layers.26.self_attn.out_proj.bias": "model.encoder.layers.26.self_attn.out_proj.bias",
  "encoder_layers.26.final_layer_norm.weight": "model.encoder.layers.26.final_layer_norm.weight",
  "encoder_layers.26.final_layer_norm.bias": "model.encoder.layers.26.final_layer_norm.bias",
  "encoder_layers.26.mlp.fc1.weight": "model.encoder.layers.26.fc1.weight",
  "encoder_layers.26.mlp.fc1.bias": "model.encoder.layers.26.fc1.bias",
  "encoder_layers.26.mlp.fc2.weight": "model.encoder.layers.26.fc2.weight",
  "encoder_layers.26.mlp.fc2.bias": "model.encoder.layers.26.fc2.bias",
  "encoder_layers.27.self_attn_layer_norm.weight": "model.encoder.layers.27.self_attn_layer_norm.weight",
  "encoder_layers.27.self_attn_layer_norm.bias": "model.encoder.layers.27.self_attn_layer_norm.bias",
  "encoder_layers.27.self_attn.q_proj.weight": "model.encoder.layers.27.self_attn.q_proj.weight",
  "encoder_layers.27.self_attn.q_proj.bias": "model.encoder.layers.27.self_attn.q_proj.bias",
  "encoder_layers.27.self_attn.k_proj.weight": "model.encoder.layers.27.self_attn.k_proj.weight",
  "encoder_layers.27.self_attn.v_proj.weight": "model.encoder.layers.27.self_attn.v_proj.weight",
  "encoder_layers.27.self_attn.v_proj.bias": "model.encoder.layers.27.self_attn.v_proj.bias",
  "encoder_layers.27.self_attn.out_proj.weight": "model.encoder.layers.27.self_attn.out_proj.weight",
  "encoder_layers.27.self_attn.out_proj.bias": "model.encoder.layers.27.self_attn.out_proj.bias",
  "encoder_layers.27.final_layer_norm.weight": "model.encoder.layers.27.final_layer_norm.weight",
  "encoder_layers.27.final_layer_norm.bias": "model.encoder.layers.27.final_layer_norm.bias",
  "encoder_layers.27.mlp.fc1.weight": "model.encoder.layers.27.fc1.weight",
  "encoder_layers.27.mlp.fc1.bias": "model.encoder.layers.27.fc1.bias",
  "encoder_layers.27.mlp.fc2.weight": "model.encoder.layers.27.fc2.weight",
  "encoder_layers.27.mlp.fc2.bias": "model.encoder.layers.27.fc2.bias",
  "encoder_layers.28.self_attn_layer_norm.weight": "model.encoder.layers.28.self_attn_layer_norm.weight",
  "encoder_layers.28.self_attn_layer_norm.bias": "model.encoder.layers.28.self_attn_layer_norm.bias",
  "encoder_layers.28.self_attn.q_proj.weight": "model.encoder.layers.28.self_attn.q_proj.weight",
  "encoder_layers.28.self_attn.q_proj.bias": "model.encoder.layers.28.self_attn.q_proj.bias",
  "encoder_layers.28.self_attn.k_proj.weight": "model.encoder.layers.28.self_attn.k_proj.weight",
  "encoder_layers.28.self_attn.v_proj.weight": "model.encoder.layers.28.self_attn.v_proj.weight",
  "encoder_layers.28.self_attn.v_proj.bias": "model.encoder.layers.28.self_attn.v_proj.bias",
  "encoder_layers.28.self_attn.out_proj.weight": "model.encoder.layers.28.self_attn.out_proj.weight",
  "encoder_layers.28.self_attn.out_proj.bias": "model.encoder.layers.28.self_attn.out_proj.bias",
  "encoder_layers.28.final_layer_norm.weight": "model.encoder.layers.28.final_layer_norm.weight",
  "encoder_layers.28.final_layer_norm.bias": "model.encoder.layers.28.final_layer_norm.bias",
  "encoder_layers.28.mlp.fc1.weight": "model.encoder.layers.28.fc1.weight",
  "encoder_layers.28.mlp.fc1.bias": "model.encoder.layers.28.fc1.bias",
  "encoder_layers.28.mlp.fc2.weight": "model.encoder.layers.28.fc2.weight",
  "encoder_layers.28.mlp.fc2.bias": "model.encoder.layers.28.fc2.bias",
  "encoder_layers.29.self_attn_layer_norm.weight": "model.encoder.layers.29.self_attn_layer_norm.weight",
  "encoder_layers.29.self_attn_layer_norm.bias": "model.encoder.layers.29.self_attn_layer_norm.bias",
  "encoder_layers.29.self_attn.q_proj.weight": "model.encoder.layers.29.self_attn.q_proj.weight",
  "encoder_layers.29.self_attn.q_proj.bias": "model.encoder.layers.29.self_attn.q_proj.bias",
  "encoder_layers.29.self_attn.k_proj.weight": "model.encoder.layers.29.self_attn.k_proj.weight",
  "encoder_layers.29.self_attn.v_proj.weight": "model.encoder.layers.29.self_attn.v_proj.weight",
  "encoder_layers.29.self_attn.v_proj.bias": "model.encoder.layers.29.self_attn.v_proj.bias",
  "encoder_layers.29.self_attn.out_proj.weight": "model.encoder.layers.29.self_attn.out_proj.weight",
  "encoder_layers.29.self_attn.out_proj.bias": "model.encoder.layers.29.self_attn.out_proj.bias",
  "encoder_layers.29.final_layer_norm.weight": "model.encoder.layers.29.final_layer_norm.weight",
  "encoder_layers.29.final_layer_norm.bias": "model.encoder.layers.29.final_layer_norm.bias",
  "encoder_layers.29.mlp.fc1.weight": "model.encoder.layers.29.fc1.weight",
  "encoder_layers.29.mlp.fc1.bias": "model.encoder.layers.29.fc1.bias",
  "encoder_layers.29.mlp.fc2.weight": "model.encoder.layers.29.fc2.weight",
  "encoder_layers.29.mlp.fc2.bias": "model.encoder.layers.29.fc2.bias",
  "encoder_layers.30.self_attn_layer_norm.weight": "model.encoder.layers.30.self_attn_layer_norm.weight",
  "encoder_layers.30.self_attn_layer_norm.bias": "model.encoder.layers.30.self_attn_layer_norm.bias",
  "encoder_layers.30.self_attn.q_proj.weight": "model.encoder.layers.30.self_attn.q_proj.weight",
  "encoder_layers.30.self_attn.q_proj.bias": "model.encoder.layers.30.self_attn.q_proj.bias",
  "encoder_layers.30.self_attn.k_proj.weight": "model.encoder.layers.30.self_attn.k_proj.weight",
  "encoder_layers.30.self_attn.v_proj.weight": "model.encoder.layers.30.self_attn.v_proj.weight",
  "encoder_layers.30.self_attn.v_proj.bias": "model.encoder.layers.30.self_attn.v_proj.bias",
  "encoder_layers.30.self_attn.out_proj.weight": "model.encoder.layers.30.self_attn.out_proj.weight",
  "encoder_layers.30.self_attn.out_proj.bias": "model.encoder.layers.30.self_attn.out_proj.bias",
  "encoder_layers.30.final_layer_norm.weight": "model.encoder.layers.30.final_layer_norm.weight",
  "encoder_layers.30.final_layer_norm.bias": "model.encoder.layers.30.final_layer_norm.bias",
  "encoder_layers.30.mlp.fc1.weight": "model.encoder.layers.30.fc1.weight",
  "encoder_layers.30.mlp.fc1.bias": "model.encoder.layers.30.fc1.bias",
  "encoder_layers.30.mlp.fc2.weight": "model.encoder.layers.30.fc2.weight",
  "encoder_layers.30.mlp.fc2.bias": "model.encoder.layers.30.fc2.bias",
  "encoder_layers.31.self_attn_layer_norm.weight": "model.encoder.layers.31.self_attn_layer_norm.weight",
  "encoder_layers.31.self_attn_layer_norm.bias": "model.encoder.layers.31.self_attn_layer_norm.bias",
  "encoder_layers.31.self_attn.q_proj.weight": "model.encoder.layers.31.self_attn.q_proj.weight",
  "encoder_layers.31.self_attn.q_proj.bias": "model.encoder.layers.31.self_attn.q_proj.bias",
  "encoder_layers.31.self_attn.k_proj.weight": "model.encoder.layers.31.self_attn.k_proj.weight",
  "encoder_layers.31.self_attn.v_proj.weight": "model.encoder.layers.31.self_attn.v_proj.weight",
  "encoder_layers.31.self_attn.v_proj.bias": "model.encoder.layers.31.self_attn.v_proj.bias",
  "encoder_layers.31.self_attn.out_proj.weight": "model.encoder.layers.31.self_attn.out_proj.weight",
  "encoder_layers.31.self_attn.out_proj.bias": "model.encoder.layers.31.self_attn.out_proj.bias",
  "encoder_layers.31.final_layer_norm.weight": "model.encoder.layers.31.final_layer_norm.weight",
  "encoder_layers.31.final_layer_norm.bias": "model.encoder.layers.31.final_layer_norm.bias",
  "encoder_layers.31.mlp.fc1.weight": "model.encoder.layers.31.fc1.weight",
  "encoder_layers.31.mlp.fc1.bias": "model.encoder.layers.31.fc1.bias",
  "encoder_layers.31.mlp.fc2.weight": "model.encoder.layers.31.fc2.weight",
  "encoder_layers.31.mlp.fc2.bias": "model.encoder.layers.31.fc2.bias",
  "encoder_norm.weight": "model.encoder.layer_norm.weight",
  "encoder_norm.bias": "model.encoder.layer_norm.bias",
  "token_embedding": "model.decoder.embed_tokens.weight",
  "decoder_positions": "model.decoder.embed_positions.weight",
  "decoder_layers.0.self_attn_layer_norm.weight": "model.decoder.layers.0.self_attn_layer_norm.weight",
  "decoder_layers.0.self_attn_layer_norm.bias": "model.decoder.layers.0.self_attn_layer_norm.bias",
  "decoder_layers.0.self_attn.q_proj.weight": "model.decoder.layers.0.self_attn.q_proj.weight",
  "decoder_layers.0.self_attn.q_proj.bias": "model.decoder.layers.0.self_attn.q_proj.bias",
  "decoder_layers.0.self_attn.k_proj.weight": "model.decoder.layers.0.self_attn.k_proj.weight",
  "decoder_layers.0.self_attn.v_proj.weight": "model.decoder.layers.0.self_attn.v_proj.weight",
  "decoder_layers.0.self_attn.v_proj.bias": "model.decoder.layers.0.self_attn.v_proj.bias",
  "decoder_layers.0.self_attn.out_proj.weight": "model.decoder.layers.0.self_attn.out_proj.weight",
  "decoder_layers.0.self_attn.out_proj.bias": "model.decoder.layers.0.self_attn.out_proj.bias",
  "decoder_layers.0.encoder_attn_layer_norm.weight": "model.decoder.layers.0.encoder_attn_layer_norm.weight",
  "decoder_layers.0.encoder_attn_layer_norm.bias": "model.decoder.layers.0.encoder_attn_layer_norm.bias",
  "decoder_layers.0.encoder_attn.q_proj.weight": "model.decoder.layers.0.encoder_attn.q_proj.weight",
  "decoder_layers.0.encoder_attn.q_proj.bias": "model.decoder.layers.0.encoder_attn.q_proj.bias",
  "decoder_layers.0.encoder_attn.k_proj.weight": "model.decoder.layers.0.encoder_attn.k_proj.weight",
  "decoder_layers.0.encoder_attn.v_proj.weight": "model.decoder.layers.0.encoder_attn.v_proj.weight",
  "decoder_layers.0.encoder_attn.v_proj.bias": "model.decoder.layers.0.encoder_attn.v_proj.bias",
  "decoder_layers.0.encoder_attn.out_proj.weight": "model.decoder.layers.0.encoder_attn.out_proj.weight",
  "decoder_layers.0.encoder_attn.out_proj.bias": "model.decoder.layers.0.encoder_attn.out_proj.bias",
  "decoder_layers.0.final_layer_norm.weight": "model.decoder.layers.0.final_layer_norm.weight",
  "decoder_layers.0.final_layer_norm.bias": "model.decoder.layers.0.final_layer_norm.bias",
  "decoder_layers.0.mlp.fc1.weight": "model.decoder.layers.0.fc1.weight",
  "decoder_layers.0.mlp.fc1.bias": "model.decoder.layers.0.fc1.bias",
  "decoder_layers.0.mlp.fc2.weight": "model.decoder.layers.0.fc2.weight",
  "decoder_layers.0.mlp.fc2.bias": "model.decoder.layers.0.fc2.bias",
  "decoder_layers.1.self_attn_layer_norm.weight": "model.decoder.layers.1.self_attn_layer_norm.weight",
  "decoder_layers.1.self_attn_layer_norm.bias": "model.decoder.layers.1.self_attn_layer_norm.bias",
  "decoder_layers.1.self_attn.q_proj.weight": "model.decoder.layers.1.self_attn.q_proj.weight",
  "decoder_layers.1.self_attn.q_proj.bias": "model.decoder.layers.1.self_attn.q_proj.bias",
  "decoder_layers.1.self_attn.k_proj.weight": "model.decoder.layers.1.self_attn.k_proj.weight",
  "decoder_layers.1.self_attn.v_proj.weight": "model.decoder.layers.1.self_attn.v_proj.weight",
  "decoder_layers.1.self_attn.v_proj.bias": "model.decoder.layers.1.self_attn.v_proj.bias",
  "decoder_layers.1.self_attn.out_proj.weight": "model.decoder.layers.1.self_attn.out_proj.weight",
  "decoder_layers.1.self_attn.out_proj.bias": "model.decoder.layers.1.self_attn.out_proj.bias",
  "decoder_layers.1.encoder_attn_layer_norm.weight": "model.decoder.layers.1.encoder_attn_layer_norm.weight",
  "decoder_layers.1.encoder_attn_layer_norm.bias": "model.decoder.layers.1.encoder_attn_layer_norm.bias",
  "decoder_layers.1.encoder_attn.q_proj.weight": "model.decoder.layers.1.encoder_attn.q_proj.weight",
  "decoder_layers.1.encoder_attn.q_proj.bias": "model.decoder.layers.1.encoder_attn.q_proj.bias",
  "decoder_layers.1.encoder_attn.k_proj.weight": "model.decoder.layers.1.encoder_attn.k_proj.weight",
  "decoder_layers.1.encoder_attn.v_proj.weight": "model.decoder.layers.1.encoder_attn.v_proj.weight",
  "decoder_layers.1.encoder_attn.v_proj.bias": "model.decoder.layers.1.encoder_attn.v_proj.bias",
  "decoder_layers.1.encoder_attn.out_proj.weight": "model.decoder.layers.1.encoder_attn.out_proj.weight",
  "decoder_layers.1.encoder_attn.out_proj.bias": "model.decoder.layers.1.encoder_attn.out_proj.bias",
  "decoder_layers.1.final_layer_norm.weight": "model.decoder.layers.1.final_layer_norm.weight",
  "decoder_layers.1.final_layer_norm.bias": "model.decoder.layers.1.final_layer_norm.bias",
  "decoder_layers.1.mlp.fc1.weight": "model.decoder.layers.1.fc1.weight",
  "decoder_layers.1.mlp.fc1.bias": "model.decoder.layers.1.fc1.bias",
  "decoder_layers.1.mlp.fc2.weight": "model.decoder.layers.1.fc2.weight",
  "decoder_layers.1.mlp.fc2.bias": "model.decoder.layers.1.fc2.bias",
  "decoder_layers.2.self_attn_layer_norm.weight": "model.decoder.layers.2.self_attn_layer_norm.weight",
  "decoder_layers.2.self_attn_layer_norm.bias": "model.decoder.layers.2.self_attn_layer_norm.bias",
  "decoder_layers.2.self_attn.q_proj.weight": "model.decoder.layers.2.self_attn.q_proj.weight",
  "decoder_layers.2.self_attn.q_proj.bias": "model.decoder.layers.2.self_attn.q_proj.bias",
  "decoder_layers.2.self_attn.k_proj.weight": "model.decoder.layers.2.self_attn.k_proj.weight",
  "decoder_layers.2.self_attn.v_proj.weight": "model.decoder.layers.2.self_attn.v_proj.weight",
  "decoder_layers.2.self_attn.v_proj.bias": "model.decoder.layers.2.self_attn.v_proj.bias",
  "decoder_layers.2.self_attn.out_proj.weight": "model.decoder.layers.2.self_attn.out_proj.weight",
  "decoder_layers.2.self_attn.out_proj.bias": "model.decoder.layers.2.self_attn.out_proj.bias",
  "decoder_layers.2.encoder_attn_layer_norm.weight": "model.decoder.layers.2.encoder_attn_layer_norm.weight",
  "decoder_layers.2.encoder_attn_layer_norm.bias": "model.decoder.layers.2.encoder_attn_layer_norm.bias",
  "decoder_layers.2.encoder_attn.q_proj.weight": "model.decoder.layers.2.encoder_attn.q_proj.weight",
  "decoder_layers.2.encoder_attn.q_proj.bias": "model.decoder.layers.2.encoder_attn.q_proj.bias",
  "decoder_layers.2.encoder_attn.k_proj.weight": "model.decoder.layers.2.encoder_attn.k_proj.weight",
  "decoder_layers.2.encoder_attn.v_proj.weight": "model.decoder.layers.2.encoder_attn.v_proj.weight",
  "decoder_layers.2.encoder_attn.v_proj.bias": "model.decoder.layers.2.encoder_attn.v_proj.bias",
  "decoder_layers.2.encoder_attn.out_proj.weight": "model.decoder.layers.2.encoder_attn.out_proj.weight",
  "decoder_layers.2.encoder_attn.out_proj.bias": "model.decoder.layers.2.encoder_attn.out_proj.bias",
  "decoder_layers.2.final_layer_norm.weight": "model.decoder.layers.2.final_layer_norm.weight",
  "decoder_layers.2.final_layer_norm.bias": "model.decoder.layers.2.final_layer_norm.bias",
  "decoder_layers.2.mlp.fc1.weight": "model.decoder.layers.2.fc1.weight",
  "decoder_layers.2.mlp.fc1.bias": "model.decoder.layers.2.fc1.bias",
  "decoder_layers.2.mlp.fc2.weight": "model.decoder.layers.2.fc2.weight",
  "decoder_layers.2.mlp.fc2.bias": "model.decoder.layers.2.fc2.bias",
  "decoder_layers.3.self_attn_layer_norm.weight": "model.decoder.layers.3.self_attn_layer_norm.weight",
  "decoder_layers.3.self_attn_layer_norm.bias": "model.decoder.layers.3.self_attn_layer_norm.bias",
  "decoder_layers.3.self_attn.q_proj.weight": "model.decoder.layers.3.self_attn.q_proj.weight",
  "decoder_layers.3.self_attn.q_proj.bias": "model.decoder.layers.3.self_attn.q_proj.bias",
  "decoder_layers.3.self_attn.k_proj.weight": "model.decoder.layers.3.self_attn.k_proj.weight",
  "decoder_layers.3.self_attn.v_proj.weight": "model.decoder.layers.3.self_attn.v_proj.weight",
  "decoder_layers.3.self_attn.v_proj.bias": "model.decoder.layers.3.self_attn.v_proj.bias",
  "decoder_layers.3.self_attn.out_proj.weight": "model.decoder.layers.3.self_attn.out_proj.weight",
  "decoder_layers.3.self_attn.out_proj.bias": "model.decoder.layers.3.self_attn.out_proj.bias",
  "decoder_layers.3.encoder_attn_layer_norm.weight": "model.decoder.layers.3.encoder_attn_layer_norm.weight",
  "decoder_layers.3.encoder_attn_layer_norm.bias": "model.decoder.layers.3.encoder_attn_layer_norm.bias",
  "decoder_layers.3.encoder_attn.q_proj.weight": "model.decoder.layers.3.encoder_attn.q_proj.weight",
  "decoder_layers.3.encoder_attn.q_proj.bias": "model.decoder.layers.3.encoder_attn.q_proj.bias",
  "decoder_layers.3.encoder_attn.k_proj.weight": "model.decoder.layers.3.encoder_attn.k_proj.weight",
  "decoder_layers.3.encoder_attn.v_proj.weight": "model.decoder.layers.3.encoder_attn.v_proj.weight",
  "decoder_layers.3.encoder_attn.v_proj.bias": "model.decoder.layers.3.encoder_attn.v_proj.bias",
  "decoder_layers.3.encoder_attn.out_proj.weight": "model.decoder.layers.3.encoder_attn.out_proj.weight",
  "decoder_layers.3.encoder_attn.out_proj.bias": "model.decoder.layers.3.encoder_attn.out_proj.bias",
  "decoder_layers.3.final_layer_norm.weight": "model.decoder.layers.3.final_layer_norm.weight",
  "decoder_layers.3.final_layer_norm.bias": "model.decoder.layers.3.final_layer_norm.bias",
  "decoder_layers.3.mlp.fc1.weight": "model.decoder.layers.3.fc1.weight",
  "decoder_layers.3.mlp.fc1.bias": "model.decoder.layers.3.fc1.bias",
  "decoder_layers.3.mlp.fc2.weight": "model.decoder.layers.3.fc2.weight",
  "decoder_layers.3.mlp.fc2.bias": "model.decoder.layers.3.fc2.bias",
  "decoder_layers.4.self_attn_layer_norm.weight": "model.decoder.layers.4.self_attn_layer_norm.weight",
  "decoder_layers.4.self_attn_layer_norm.bias": "model.decoder.layers.4.self_attn_layer_norm.bias",
  "decoder_layers.4.self_attn.q_proj.weight": "model.decoder.layers.4.self_attn.q_proj.weight",
  "decoder_layers.4.self_attn.q_proj.bias": "model.decoder.layers.4.self_attn.q_proj.bias",
  "decoder_layers.4.self_attn.k_proj.weight": "model.decoder.layers.4.self_attn.k_proj.weight",
  "decoder_layers.4.self_attn.v_proj.weight": "model.decoder.layers.4.self_attn.v_proj.weight",
  "decoder_layers.4.self_attn.v_proj.bias": "model.decoder.layers.4.self_attn.v_proj.bias",
  "decoder_layers.4.self_attn.out_proj.weight": "model.decoder.layers.4.self_attn.out_proj.weight",
  "decoder_layers.4.self_attn.out_proj.bias": "model.decoder.layers.4.self_attn.out_proj.bias",
  "decoder_layers.4.encoder_attn_layer_norm.weight": "model.decoder.layers.4.encoder_attn_layer_norm.weight",
  "decoder_layers.4.encoder_attn_layer_norm.bias": "model.decoder.layers.4.encoder_attn_layer_norm.bias",
  "decoder_layers.4.encoder_attn.q_proj.weight": "model.decoder.layers.4.encoder_attn.q_proj.weight",
  "decoder_layers.4.encoder_attn.q_proj.bias": "model.decoder.layers.4.encoder_attn.q_proj.bias",
  "decoder_layers.4.encoder_attn.k_proj.weight": "model.decoder.layers.4.encoder_attn.k_proj.weight",
  "decoder_layers.4.encoder_attn.v_proj.weight": "model.decoder.layers.4.encoder_attn.v_proj.weight",
  "decoder_layers.4.encoder_attn.v_proj.bias": "model.decoder.layers.4.encoder_attn.v_proj.bias",
  "decoder_layers.4.encoder_attn.out_proj.weight": "model.decoder.layers.4.encoder_attn.out_proj.weight",
  "decoder_layers.4.encoder_attn.out_proj.bias": "model.decoder.layers.4.encoder_attn.out_proj.bias",
  "decoder_layers.4.final_layer_norm.weight": "model.decoder.layers.4.final_layer_norm.weight",
  "decoder_layers.4.final_layer_norm.bias": "model.decoder.layers.4.final_layer_norm.bias",
  "decoder_layers.4.mlp.fc1.weight": "model.decoder.layers.4.fc1.weight",
  "decoder_layers.4.mlp.fc1.bias": "model.decoder.layers.4.fc1.bias",
  "decoder_layers.4.mlp.fc2.weight": "model.decoder.layers.4.fc2.weight",
  "decoder_layers.4.mlp.fc2.bias": "model.decoder.layers.4.fc2.bias",
  "decoder_layers.5.self_attn_layer_norm.weight": "model.decoder.layers.5.self_attn_layer_norm.weight",
  "decoder_layers.5.self_attn_layer_norm.bias": "model.decoder.layers.5.self_attn_layer_norm.bias",
  "decoder_layers.5.self_attn.q_proj.weight": "model.decoder.layers.5.self_attn.q_proj.weight",
  "decoder_layers.5.self_attn.q_proj.bias": "model.decoder.layers.5.self_attn.q_proj.bias",
  "decoder_layers.5.self_attn.k_proj.weight": "model.decoder.layers.5.self_attn.k_proj.weight",
  "decoder_layers.5.self_attn.v_proj.weight": "model.decoder.layers.5.self_attn.v_proj.weight",
  "decoder_layers.5.self_attn.v_proj.bias": "model.decoder.layers.5.self_attn.v_proj.bias",
  "decoder_layers.5.self_attn.out_proj.weight": "model.decoder.layers.5.self_attn.out_proj.weight",
  "decoder_layers.5.self_attn.out_proj.bias": "model.decoder.layers.5.self_attn.out_proj.bias",
  "decoder_layers.5.encoder_attn_layer_norm.weight": "model.decoder.layers.5.encoder_attn_layer_norm.weight",
  "decoder_layers.5.encoder_attn_layer_norm.bias": "model.decoder.layers.5.encoder_attn_layer_norm.bias",
  "decoder_layers.5.encoder_attn.q_proj.weight": "model.decoder.layers.5.encoder_attn.q_proj.weight",
  "decoder_layers.5.encoder_attn.q_proj.bias": "model.decoder.layers.5.encoder_attn.q_proj.bias",
  "decoder_layers.5.encoder_attn.k_proj.weight": "model.decoder.layers.5.encoder_attn.k_proj.weight",
  "decoder_layers.5.encoder_attn.v_proj.weight": "model.decoder.layers.5.encoder_attn.v_proj.weight",
  "decoder_layers.5.encoder_attn.v_proj.bias": "model.decoder.layers.5.encoder_attn.v_proj.bias",
  "decoder_layers.5.encoder_attn.out_proj.weight": "model.decoder.layers.5.encoder_attn.out_proj.weight",
  "decoder_layers.5.encoder_attn.out_proj.bias": "model.decoder.layers.5.encoder_attn.out_proj.bias",
  "decoder_layers.5.final_layer_norm.weight": "model.decoder.layers.5.final_layer_norm.weight",
  "decoder_layers.5.final_layer_norm.bias": "model.decoder.layers.5.final_layer_norm.bias",
  "decoder_layers.5.mlp.fc1.weight": "model.decoder.layers.5.fc1.weight",
  "decoder_layers.5.mlp.fc1.bias": "model.decoder.layers.5.fc1.bias",
  "decoder_layers.5.mlp.fc2.weight": "model.decoder.layers.5.fc2.weight",
  "decoder_layers.5.mlp.fc2.bias": "model.decoder.layers.5.fc2.bias",
  "decoder_layers.6.self_attn_layer_norm.weight": "model.decoder.layers.6.self_attn_layer_norm.weight",
  "decoder_layers.6.self_attn_layer_norm.bias": "model.decoder.layers.6.self_attn_layer_norm.bias",
  "decoder_layers.6.self_attn.q_proj.weight": "model.decoder.layers.6.self_attn.q_proj.weight",
  "decoder_layers.6.self_attn.q_proj.bias": "model.decoder.layers.6.self_attn.q_proj.bias",
  "decoder_layers.6.self_attn.k_proj.weight": "model.decoder.layers.6.self_attn.k_proj.weight",
  "decoder_layers.6.self_attn.v_proj.weight": "model.decoder.layers.6.self_attn.v_proj.weight",
  "decoder_layers.6.self_attn.v_proj.bias": "model.decoder.layers.6.self_attn.v_proj.bias",
  "decoder_layers.6.self_attn.out_proj.weight": "model.decoder.layers.6.self_attn.out_proj.weight",
  "decoder_layers.6.self_attn.out_proj.bias": "model.decoder.layers.6.self_attn.out_proj.bias",
  "decoder_layers.6.encoder_attn_layer_norm.weight": "model.decoder.layers.6.encoder_attn_layer_norm.weight",
  "decoder_layers.6.encoder_attn_layer_norm.bias": "model.decoder.layers.6.encoder_attn_layer_norm.bias",
  "decoder_layers.6.encoder_attn.q_proj.weight": "model.decoder.layers.6.encoder_attn.q_proj.weight",
  "decoder_layers.6.encoder_attn.q_proj.bias": "model.decoder.layers.6.encoder_attn.q_proj.bias",
  "decoder_layers.6.encoder_attn.k_proj.weight": "model.decoder.layers.6.encoder_attn.k_proj.weight",
  "decoder_layers.6.encoder_attn.v_proj.weight": "model.decoder.layers.6.encoder_attn.v_proj.weight",
  "decoder_layers.6.encoder_attn.v_proj.bias": "model.decoder.layers.6.encoder_attn.v_proj.bias",
  "decoder_layers.6.encoder_attn.out_proj.weight": "model.decoder.layers.6.encoder_attn.out_proj.weight",
  "decoder_layers.6.encoder_attn.out_proj.bias": "model.decoder.layers.6.encoder_attn.out_proj.bias",
  "decoder_layers.6.final_layer_norm.weight": "model.decoder.layers.6.final_layer_norm.weight",
  "decoder_layers.6.final_layer_norm.bias": "model.decoder.layers.6.final_layer_norm.bias",
  "decoder_layers.6.mlp.fc1.weight": "model.decoder.layers.6.fc1.weight",
  "decoder_layers.6.mlp.fc1.bias": "model.decoder.layers.6.fc1.bias",
  "decoder_layers.6.mlp.fc2.weight": "model.decoder.layers.6.fc2.weight",
  "decoder_layers.6.mlp.fc2.bias": "model.decoder.layers.6.fc2.bias",
  "decoder_layers.7.self_attn_layer_norm.weight": "model.decoder.layers.7.self_attn_layer_norm.weight",
  "decoder_layers.7.self_attn_layer_norm.bias": "model.decoder.layers.7.self_attn_layer_norm.bias",
  "decoder_layers.7.self_attn.q_proj.weight": "model.decoder.layers.7.self_attn.q_proj.weight",
  "decoder_layers.7.self_attn.q_proj.bias": "model.decoder.layers.7.self_attn.q_proj.bias",
  "decoder_layers.7.self_attn.k_proj.weight": "model.decoder.layers.7.self_attn.k_proj.weight",
  "decoder_layers.7.self_attn.v_proj.weight": "model.decoder.layers.7.self_attn.v_proj.weight",
  "decoder_layers.7.self_attn.v_proj.bias": "model.decoder.layers.7.self_attn.v_proj.bias",
  "decoder_layers.7.self_attn.out_proj.weight": "model.decoder.layers.7.self_attn.out_proj.weight",
  "decoder_layers.7.self_attn.out_proj.bias": "model.decoder.layers.7.self_attn.out_proj.bias",
  "decoder_layers.7.encoder_attn_layer_norm.weight": "model.decoder.layers.7.encoder_attn_layer_norm.weight",
  "decoder_layers.7.encoder_attn_layer_norm.bias": "model.decoder.layers.7.encoder_attn_layer_norm.bias",
  "decoder_layers.7.encoder_attn.q_proj.weight": "model.decoder.layers.7.encoder_attn.q_proj.weight",
  "decoder_layers.7.encoder_attn.q_proj.bias": "model.decoder.layers.7.encoder_attn.q_proj.bias",
  "decoder_layers.7.encoder_attn.k_proj.weight": "model.decoder.layers.7.encoder_attn.k_proj.weight",
  "decoder_layers.7.encoder_attn.v_proj.weight": "model.decoder.layers.7.encoder_attn.v_proj.weight",
  "decoder_layers.7.encoder_attn.v_proj.bias": "model.decoder.layers.7.encoder_attn.v_proj.bias",
  "decoder_layers.7.encoder_attn.out_proj.weight": "model.decoder.layers.7.encoder_attn.out_proj.weight",
  "decoder_layers.7.encoder_attn.out_proj.bias": "model.decoder.layers.7.encoder_attn.out_proj.bias",
  "decoder_layers.7.final_layer_norm.weight": "model.decoder.layers.7.final_layer_norm.weight",
  "decoder_layers.7.final_layer_norm.bias": "model.decoder.layers.7.final_layer_norm.bias",
  "decoder_layers.7.mlp.fc1.weight": "model.decoder.layers.7.fc1.weight",
  "decoder_layers.7.mlp.fc1.bias": "model.decoder.layers.7.fc1.bias",
  "decoder_layers.7.mlp.fc2.weight": "model.decoder.layers.7.fc2.weight",
  "decoder_layers.7.mlp.fc2.bias": "model.decoder.layers.7.fc2.bias",
  "decoder_layers.8.self_attn_layer_norm.weight": "model.decoder.layers.8.self_attn_layer_norm.weight",
  "decoder_layers.8.self_attn_layer_norm.bias": "model.decoder.layers.8.self_attn_layer_norm.bias",
  "decoder_layers.8.self_attn.q_proj.weight": "model.decoder.layers.8.self_attn.q_proj.weight",
  "decoder_layers.8.self_attn.q_proj.bias": "model.decoder.layers.8.self_attn.q_proj.bias",
  "decoder_layers.8.self_attn.k_proj.weight": "model.decoder.layers.8.self_attn.k_proj.weight",
  "decoder_layers.8.self_attn.v_proj.weight": "model.decoder.layers.8.self_attn.v_proj.weight",
  "decoder_layers.8.self_attn.v_proj.bias": "model.decoder.layers.8.self_attn.v_proj.bias",
  "decoder_layers.8.self_attn.out_proj.weight": "model.decoder.layers.8.self_attn.out_proj.weight",
  "decoder_layers.8.self_attn.out_proj.bias": "model.decoder.layers.8.self_attn.out_proj.bias",
  "decoder_layers.8.encoder_attn_layer_norm.weight": "model.decoder.layers.8.encoder_attn_layer_norm.weight",
  "decoder_layers.8.encoder_attn_layer_norm.bias": "model.decoder.layers.8.encoder_attn_layer_norm.bias",
  "decoder_layers.8.encoder_attn.q_proj.weight": "model.decoder.layers.8.encoder_attn.q_proj.weight",
  "decoder_layers.8.encoder_attn.q_proj.bias": "model.decoder.layers.8.encoder_attn.q_proj.bias",
  "decoder_layers.8.encoder_attn.k_proj.weight": "model.decoder.layers.8.encoder_attn.k_proj.weight",
  "decoder_layers.8.encoder_attn.v_proj.weight": "model.decoder.layers.8.encoder_attn.v_proj.weight",
  "decoder_layers.8.encoder_attn.v_proj.bias": "model.decoder.layers.8.encoder_attn.v_proj.bias",
  "decoder_layers.8.encoder_attn.out_proj.weight": "model.decoder.layers.8.encoder_attn.out_proj.weight",
  "decoder_layers.8.encoder_attn.out_proj.bias": "model.decoder.layers.8.encoder_attn.out_proj.bias",
  "decoder_layers.8.final_layer_norm.weight": "model.decoder.layers.8.final_layer_norm.weight",
  "decoder_layers.8.final_layer_norm.bias": "model.decoder.layers.8.final_layer_norm.bias",
  "decoder_layers.8.mlp.fc1.weight": "model.decoder.layers.8.fc1.weight",
  "decoder_layers.8.mlp.fc1.bias": "model.decoder.layers.8.fc1.bias",
  "decoder_layers.8.mlp.fc2.weight": "model.decoder.layers.8.fc2.weight",
  "decoder_layers.8.mlp.fc2.bias": "model.decoder.layers.8.fc2.bias",
  "decoder_layers.9.self_attn_layer_norm.weight": "model.decoder.layers.9.self_attn_layer_norm.weight",
  "decoder_layers.9.self_attn_layer_norm.bias": "model.decoder.layers.9.self_attn_layer_norm.bias",
  "decoder_layers.9.self_attn.q_proj.weight": "model.decoder.layers.9.self_attn.q_proj.weight",
  "decoder_layers.9.self_attn.q_proj.bias": "model.decoder.layers.9.self_attn.q_proj.bias",
  "decoder_layers.9.self_attn.k_proj.weight": "model.decoder.layers.9.self_attn.k_proj.weight",
  "decoder_layers.9.self_attn.v_proj.weight": "model.decoder.layers.9.self_attn.v_proj.weight",
  "decoder_layers.9.self_attn.v_proj.bias": "model.decoder.layers.9.self_attn.v_proj.bias",
  "decoder_layers.9.self_attn.out_proj.weight": "model.decoder.layers.9.self_attn.out_proj.weight",
  "decoder_layers.9.self_attn.out_proj.bias": "model.decoder.layers.9.self_attn.out_proj.bias",
  "decoder_layers.9.encoder_attn_layer_norm.weight": "model.decoder.layers.9.encoder_attn_layer_norm.weight",
  "decoder_layers.9.encoder_attn_layer_norm.bias": "model.decoder.layers.9.encoder_attn_layer_norm.bias",
  "decoder_layers.9.encoder_attn.q_proj.weight": "model.decoder.layers.9.encoder_attn.q_proj.weight",
  "decoder_layers.9.encoder_attn.q_proj.bias": "model.decoder.layers.9.encoder_attn.q_proj.bias",
  "decoder_layers.9.encoder_attn.k_proj.weight": "model.decoder.layers.9.encoder_attn.k_proj.weight",
  "decoder_layers.9.encoder_attn.v_proj.weight": "model.decoder.layers.9.encoder_attn.v_proj.weight",
  "decoder_layers.9.encoder_attn.v_proj.bias": "model.decoder.layers.9.encoder_attn.v_proj.bias",
  "decoder_layers.9.encoder_attn.out_proj.weight": "model.decoder.layers.9.encoder_attn.out_proj.weight",
  "decoder_layers.9.encoder_attn.out_proj.bias": "model.decoder.layers.9.encoder_attn.out_proj.bias",
  "decoder_layers.9.final_layer_norm.weight": "model.decoder.layers.9.final_layer_norm.weight",
  "decoder_layers.9.final_layer_norm.bias": "model.decoder.layers.9.final_layer_norm.bias",
  "decoder_layers.9.mlp.fc1.weight": "model.decoder.layers.9.fc1.weight",
  "decoder_layers.9.mlp.fc1.bias": "model.decoder.layers.9.fc1.bias",
  "decoder_layers.9.mlp.fc2.weight": "model.decoder.layers.9.fc2.weight",
  "decoder_layers.9.mlp.fc2.bias": "model.decoder.layers.9.fc2.bias",
  "decoder_layers.10.self_attn_layer_norm.weight": "model.decoder.layers.10.self_attn_layer_norm.weight",
  "decoder_layers.10.self_attn_layer_norm.bias": "model.decoder.layers.10.self_attn_layer_norm.bias",
  "decoder_layers.10.self_attn.q_proj.weight": "model.decoder.layers.10.self_attn.q_proj.weight",
  "decoder_layers.10.self_attn.q_proj.bias": "model.decoder.layers.10.self_attn.q_proj.bias",
  "decoder_layers.10.self_attn.k_proj.weight": "model.decoder.layers.10.self_attn.k_proj.weight",
  "decoder_layers.10.self_attn.v_proj.weight": "model.decoder.layers.10.self_attn.v_proj.weight",
  "decoder_layers.10.self_attn.v_proj.bias": "model.decoder.layers.10.self_attn.v_proj.bias",
  "decoder_layers.10.self_attn.out_proj.weight": "model.decoder.layers.10.self_attn.out_proj.weight",
  "decoder_layers.10.self_attn.out_proj.bias": "model.decoder.layers.10.self_attn.out_proj.bias",
  "decoder_layers.10.encoder_attn_layer_norm.weight": "model.decoder.layers.10.encoder_attn_layer_norm.weight",
  "decoder_layers.10.encoder_attn_layer_norm.bias": "model.decoder.layers.10.encoder_attn_layer_norm.bias",
  "decoder_layers.10.encoder_attn.q_proj.weight": "model.decoder.layers.10.encoder_attn.q_proj.weight",
  "decoder_layers.10.encoder_attn.q_proj.bias": "model.decoder.layers.10.encoder_attn.q_proj.bias",
  "decoder_layers.10.encoder_attn.k_proj.weight": "model.decoder.layers.10.encoder_attn.k_proj.weight",
  "decoder_layers.10.encoder_attn.v_proj.weight": "model.decoder.layers.10.encoder_attn.v_proj.weight",
  "decoder_layers.10.encoder_attn.v_proj.bias": "model.decoder.layers.10.encoder_attn.v_proj.bias",
  "decoder_layers.10.encoder_attn.out_proj.weight": "model.decoder.layers.10.encoder_attn.out_proj.weight",
  "decoder_layers.10.encoder_attn.out_proj.bias": "model.decoder.layers.10.encoder_attn.out_proj.bias",
  "decoder_layers.10.final_layer_norm.weight": "model.decoder.layers.10.final_layer_norm.weight",
  "decoder_layers.10.final_layer_norm.bias": "model.decoder.layers.10.final_layer_norm.bias",
  "decoder_layers.10.mlp.fc1.weight": "model.decoder.layers.10.fc1.weight",
  "decoder_layers.10.mlp.fc1.bias": "model.decoder.layers.10.fc1.bias",
  "decoder_layers.10.mlp.fc2.weight": "model.decoder.layers.10.fc2.weight",
  "decoder_layers.10.mlp.fc2.bias": "model.decoder.layers.10.fc2.bias",
  "decoder_layers.11.self_attn_layer_norm.weight": "model.decoder.layers.11.self_attn_layer_norm.weight",
  "decoder_layers.11.self_attn_layer_norm.bias": "model.decoder.layers.11.self_attn_layer_norm.bias",
  "decoder_layers.11.self_attn.q_proj.weight": "model.decoder.layers.11.self_attn.q_proj.weight",
  "decoder_layers.11.self_attn.q_proj.bias": "model.decoder.layers.11.self_attn.q_proj.bias",
  "decoder_layers.11.self_attn.k_proj.weight": "model.decoder.layers.11.self_attn.k_proj.weight",
  "decoder_layers.11.self_attn.v_proj.weight": "model.decoder.layers.11.self_attn.v_proj.weight",
  "decoder_layers.11.self_attn.v_proj.bias": "model.decoder.layers.11.self_attn.v_proj.bias",
  "decoder_layers.11.self_attn.out_proj.weight": "model.decoder.layers.11.self_attn.out_proj.weight",
  "decoder_layers.11.self_attn.out_proj.bias": "model.decoder.layers.11.self_attn.out_proj.bias",
  "decoder_layers.11.encoder_attn_layer_norm.weight": "model.decoder.layers.11.encoder_attn_layer_norm.weight",
  "decoder_layers.11.encoder_attn_layer_norm.bias": "model.decoder.layers.11.encoder_attn_layer_norm.bias",
  "decoder_layers.11.encoder_attn.q_proj.weight": "model.decoder.layers.11.encoder_attn.q_proj.weight",
  "decoder_layers.11.encoder_attn.q_proj.bias": "model.decoder.layers.11.encoder_attn.q_proj.bias",
  "decoder_layers.11.encoder_attn.k_proj.weight": "model.decoder.layers.11.encoder_attn.k_proj.weight",
  "decoder_layers.11.encoder_attn.v_proj.weight": "model.decoder.layers.11.encoder_attn.v_proj.weight",
  "decoder_layers.11.encoder_attn.v_proj.bias": "model.decoder.layers.11.encoder_attn.v_proj.bias",
  "decoder_layers.11.encoder_attn.out_proj.weight": "model.decoder.layers.11.encoder_attn.out_proj.weight",
  "decoder_layers.11.encoder_attn.out_proj.bias": "model.decoder.layers.11.encoder_attn.out_proj.bias",
  "decoder_layers.11.final_layer_norm.weight": "model.decoder.layers.11.final_layer_norm.weight",
  "decoder_layers.11.final_layer_norm.bias": "model.decoder.layers.11.final_layer_norm.bias",
  "decoder_layers.11.mlp.fc1.weight": "model.decoder.layers.11.fc1.weight",
  "decoder_layers.11.mlp.fc1.bias": "model.decoder.layers.11.fc1.bias",
  "decoder_layers.11.mlp.fc2.weight": "model.decoder.layers.11.fc2.weight",
  "decoder_layers.11.mlp.fc2.bias": "model.decoder.layers.11.fc2.bias",
  "decoder_layers.12.self_attn_layer_norm.weight": "model.decoder.layers.12.self_attn_layer_norm.weight",
  "decoder_layers.12.self_attn_layer_norm.bias": "model.decoder.layers.12.self_attn_layer_norm.bias",
  "decoder_layers.12.self_attn.q_proj.weight": "model.decoder.layers.12.self_attn.q_proj.weight",
  "decoder_layers.12.self_attn.q_proj.bias": "model.decoder.layers.12.self_attn.q_proj.bias",
  "decoder_layers.12.self_attn.k_proj.weight": "model.decoder.layers.12.self_attn.k_proj.weight",
  "decoder_layers.12.self_attn.v_proj.weight": "model.decoder.layers.12.self_attn.v_proj.weight",
  "decoder_layers.12.self_attn.v_proj.bias": "model.decoder.layers.12.self_attn.v_proj.bias",
  "decoder_layers.12.self_attn.out_proj.weight": "model.decoder.layers.12.self_attn.out_proj.weight",
  "decoder_layers.12.self_attn.out_proj.bias": "model.decoder.layers.12.self_attn.out_proj.bias",
  "decoder_layers.12.encoder_attn_layer_norm.weight": "model.decoder.layers.12.encoder_attn_layer_norm.weight",
  "decoder_layers.12.encoder_attn_layer_norm.bias": "model.decoder.layers.12.encoder_attn_layer_norm.bias",
  "decoder_layers.12.encoder_attn.q_proj.weight": "model.decoder.layers.12.encoder_attn.q_proj.weight",
  "decoder_layers.12.encoder_attn.q_proj.bias": "model.decoder.layers.12.encoder_attn.q_proj.bias",
  "decoder_layers.12.encoder_attn.k_proj.weight": "model.decoder.layers.12.encoder_attn.k_proj.weight",
  "decoder_layers.12.encoder_attn.v_proj.weight": "model.decoder.layers.12.encoder_attn.v_proj.weight",
  "decoder_layers.12.encoder_attn.v_proj.bias": "model.decoder.layers.12.encoder_attn.v_proj.bias",
  "decoder_layers.12.encoder_attn.out_proj.weight": "model.decoder.layers.12.encoder_attn.out_proj.weight",
  "decoder_layers.12.encoder_attn.out_proj.bias": "model.decoder.layers.12.encoder_attn.out_proj.bias",
  "decoder_layers.12.final_layer_norm.weight": "model.decoder.layers.12.final_layer_norm.weight",
  "decoder_layers.12.final_layer_norm.bias": "model.decoder.layers.12.final_layer_norm.bias",
  "decoder_layers.12.mlp.fc1.weight": "model.decoder.layers.12.fc1.weight",
  "decoder_layers.12.mlp.fc1.bias": "model.decoder.layers.12.fc1.bias",
  "decoder_layers.12.mlp.fc2.weight": "model.decoder.layers.12.fc2.weight",
  "decoder_layers.12.mlp.fc2.bias": "model.decoder.layers.12.fc2.bias",
  "decoder_layers.13.self_attn_layer_norm.weight": "model.decoder.layers.13.self_attn_layer_norm.weight",
  "decoder_layers.13.self_attn_layer_norm.bias": "model.decoder.layers.13.self_attn_layer_norm.bias",
  "decoder_layers.13.self_attn.q_proj.weight": "model.decoder.layers.13.self_attn.q_proj.weight",
  "decoder_layers.13.self_attn.q_proj.bias": "model.decoder.layers.13.self_attn.q_proj.bias",
  "decoder_layers.13.self_attn.k_proj.weight": "model.decoder.layers.13.self_attn.k_proj.weight",
  "decoder_layers.13.self_attn.v_proj.weight": "model.decoder.layers.13.self_attn.v_proj.weight",
  "decoder_layers.13.self_attn.v_proj.bias": "model.decoder.layers.13.self_attn.v_proj.bias",
  "decoder_layers.13.self_attn.out_proj.weight": "model.decoder.layers.13.self_attn.out_proj.weight",
  "decoder_layers.13.self_attn.out_proj.bias": "model.decoder.layers.13.self_attn.out_proj.bias",
  "decoder_layers.13.encoder_attn_layer_norm.weight": "model.decoder.layers.13.encoder_attn_layer_norm.weight",
  "decoder_layers.13.encoder_attn_layer_norm.bias": "model.decoder.layers.13.encoder_attn_layer_norm.bias",
  "decoder_layers.13.encoder_attn.q_proj.weight": "model.decoder.layers.13.encoder_attn.q_proj.weight",
  "decoder_layers.13.encoder_attn.q_proj.bias": "model.decoder.layers.13.encoder_attn.q_proj.bias",
  "decoder_layers.13.encoder_attn.k_proj.weight": "model.decoder.layers.13.encoder_attn.k_proj.weight",
  "decoder_layers.13.encoder_attn.v_proj.weight": "model.decoder.layers.13.encoder_attn.v_proj.weight",
  "decoder_layers.13.encoder_attn.v_proj.bias": "model.decoder.layers.13.encoder_attn.v_proj.bias",
  "decoder_layers.13.encoder_attn.out_proj.weight": "model.decoder.layers.13.encoder_attn.out_proj.weight",
  "decoder_layers.13.encoder_attn.out_proj.bias": "model.decoder.layers.13.encoder_attn.out_proj.bias",
  "decoder_layers.13.final_layer_norm.weight": "model.decoder.layers.13.final_layer_norm.weight",
  "decoder_layers.13.final_layer_norm.bias": "model.decoder.layers.13.final_layer_norm.bias",
  "decoder_layers.13.mlp.fc1.weight": "model.decoder.layers.13.fc1.weight",
  "decoder_layers.13.mlp.fc1.bias": "model.decoder.layers.13.fc1.bias",
  "decoder_layers.13.mlp.fc2.weight": "model.decoder.layers.13.fc2.weight",
  "decoder_layers.13.mlp.fc2.bias": "model.decoder.layers.13.fc2.bias",
  "decoder_layers.14.self_attn_layer_norm.weight": "model.decoder.layers.14.self_attn_layer_norm.weight",
  "decoder_layers.14.self_attn_layer_norm.bias": "model.decoder.layers.14.self_attn_layer_norm.bias",
  "decoder_layers.14.self_attn.q_proj.weight": "model.decoder.layers.14.self_attn.q_proj.weight",
  "decoder_layers.14.self_attn.q_proj.bias": "model.decoder.layers.14.self_attn.q_proj.bias",
  "decoder_layers.14.self_attn.k_proj.weight": "model.decoder.layers.14.self_attn.k_proj.weight",
  "decoder_layers.14.self_attn.v_proj.weight": "model.decoder.layers.14.self_attn.v_proj.weight",
  "decoder_layers.14.self_attn.v_proj.bias": "model.decoder.layers.14.self_attn.v_proj.bias",
  "decoder_layers.14.self_attn.out_proj.weight": "model.decoder.layers.14.self_attn.out_proj.weight",
  "decoder_layers.14.self_attn.out_proj.bias": "model.decoder.layers.14.self_attn.out_proj.bias",
  "decoder_layers.14.encoder_attn_layer_norm.weight": "model.decoder.layers.14.encoder_attn_layer_norm.weight",
  "decoder_layers.14.encoder_attn_layer_norm.bias": "model.decoder.layers.14.encoder_attn_layer_norm.bias",
  "decoder_layers.14.encoder_attn.q_proj.weight": "model.decoder.layers.14.encoder_attn.q_proj.weight",
  "decoder_layers.14.encoder_attn.q_proj.bias": "model.decoder.layers.14.encoder_attn.q_proj.bias",
  "decoder_layers.14.encoder_attn.k_proj.weight": "model.decoder.layers.14.encoder_attn.k_proj.weight",
  "decoder_layers.14.encoder_attn.v_proj.weight": "model.decoder.layers.14.encoder_attn.v_proj.weight",
  "decoder_layers.14.encoder_attn.v_proj.bias": "model.decoder.layers.14.encoder_attn.v_proj.bias",
  "decoder_layers.14.encoder_attn.out_proj.weight": "model.decoder.layers.14.encoder_attn.out_proj.weight",
  "decoder_layers.14.encoder_attn.out_proj.bias": "model.decoder.layers.14.encoder_attn.out_proj.bias",
  "decoder_layers.14.final_layer_norm.weight": "model.decoder.layers.14.final_layer_norm.weight",
  "decoder_layers.14.final_layer_norm.bias": "model.decoder.layers.14.final_layer_norm.bias",
  "decoder_layers.14.mlp.fc1.weight": "model.decoder.layers.14.fc1.weight",
  "decoder_layers.14.mlp.fc1.bias": "model.decoder.layers.14.fc1.bias",
  "decoder_layers.14.mlp.fc2.weight": "model.decoder.layers.14.fc2.weight",
  "decoder_layers.14.mlp.fc2.bias": "model.decoder.layers.14.fc2.bias",
  "decoder_layers.15.self_attn_layer_norm.weight": "model.decoder.layers.15.self_attn_layer_norm.weight",
  "decoder_layers.15.self_attn_layer_norm.bias": "model.decoder.layers.15.self_attn_layer_norm.bias",
  "decoder_layers.15.self_attn.q_proj.weight": "model.decoder.layers.15.self_attn.q_proj.weight",
  "decoder_layers.15.self_attn.q_proj.bias": "model.decoder.layers.15.self_attn.q_proj.bias",
  "decoder_layers.15.self_attn.k_proj.weight": "model.decoder.layers.15.self_attn.k_proj.weight",
  "decoder_layers.15.self_attn.v_proj.weight": "model.decoder.layers.15.self_attn.v_proj.weight",
  "decoder_layers.15.self_attn.v_proj.bias": "model.decoder.layers.15.self_attn.v_proj.bias",
  "decoder_layers.15.self_attn.out_proj.weight": "model.decoder.layers.15.self_attn.out_proj.weight",
  "decoder_layers.15.self_attn.out_proj.bias": "model.decoder.layers.15.self_attn.out_proj.bias",
  "decoder_layers.15.encoder_attn_layer_norm.weight": "model.decoder.layers.15.encoder_attn_layer_norm.weight",
  "decoder_layers.15.encoder_attn_layer_norm.bias": "model.decoder.layers.15.encoder_attn_layer_norm.bias",
  "decoder_layers.15.encoder_attn.q_proj.weight": "model.decoder.layers.15.encoder_attn.q_proj.weight",
  "decoder_layers.15.encoder_attn.q_proj.bias": "model.decoder.layers.15.encoder_attn.q_proj.bias",
  "decoder_layers.15.encoder_attn.k_proj.weight": "model.decoder.layers.15.encoder_attn.k_proj.weight",
  "decoder_layers.15.encoder_attn.v_proj.weight": "model.decoder.layers.15.encoder_attn.v_proj.weight",
  "decoder_layers.15.encoder_attn.v_proj.bias": "model.decoder.layers.15.encoder_attn.v_proj.bias",
  "decoder_layers.15.encoder_attn.out_proj.weight": "model.decoder.layers.15.encoder_attn.out_proj.weight",
  "decoder_layers.15.encoder_attn.out_proj.bias": "model.decoder.layers.15.encoder_attn.out_proj.bias",
  "decoder_layers.15.final_layer_norm.weight": "model.decoder.layers.15.final_layer_norm.weight",
  "decoder_layers.15.final_layer_norm.bias": "model.decoder.layers.15.final_layer_norm.bias",
  "decoder_layers.15.mlp.fc1.weight": "model.decoder.layers.15.fc1.weight",
  "decoder_layers.15.mlp.fc1.bias": "model.decoder.layers.15.fc1.bias",
  "decoder_layers.15.mlp.fc2.weight": "model.decoder.layers.15.fc2.weight",
  "decoder_layers.15.mlp.fc2.bias": "model.decoder.layers.15.fc2.bias",
  "decoder_layers.16.self_attn_layer_norm.weight": "model.decoder.layers.16.self_attn_layer_norm.weight",
  "decoder_layers.16.self_attn_layer_norm.bias": "model.decoder.layers.16.self_attn_layer_norm.bias",
  "decoder_layers.16.self_attn.q_proj.weight": "model.decoder.layers.16.self_attn.q_proj.weight",
  "decoder_layers.16.self_attn.q_proj.bias": "model.decoder.layers.16.self_attn.q_proj.bias",
  "decoder_layers.16.self_attn.k_proj.weight": "model.decoder.layers.16.self_attn.k_proj.weight",
  "decoder_layers.16.self_attn.v_proj.weight": "model.decoder.layers.16.self_attn.v_proj.weight",
  "decoder_layers.16.self_attn.v_proj.bias": "model.decoder.layers.16.self_attn.v_proj.bias",
  "decoder_layers.16.self_attn.out_proj.weight": "model.decoder.layers.16.self_attn.out_proj.weight",
  "decoder_layers.16.self_attn.out_proj.bias": "model.decoder.layers.16.self_attn.out_proj.bias",
  "decoder_layers.16.encoder_attn_layer_norm.weight": "model.decoder.layers.16.encoder_attn_layer_norm.weight",
  "decoder_layers.16.encoder_attn_layer_norm.bias": "model.decoder.layers.16.encoder_attn_layer_norm.bias",
  "decoder_layers.16.encoder_attn.q_proj.weight": "model.decoder.layers.16.encoder_attn.q_proj.weight",
  "decoder_layers.16.encoder_attn.q_proj.bias": "model.decoder.layers.16.encoder_attn.q_proj.bias",
  "decoder_layers.16.encoder_attn.k_proj.weight": "model.decoder.layers.16.encoder_attn.k_proj.weight",
  "decoder_layers.16.encoder_attn.v_proj.weight": "model.decoder.layers.16.encoder_attn.v_proj.weight",
  "decoder_layers.16.encoder_attn.v_proj.bias": "model.decoder.layers.16.encoder_attn.v_proj.bias",
  "decoder_layers.16.encoder_attn.out_proj.weight": "model.decoder.layers.16.encoder_attn.out_proj.weight",
  "decoder_layers.16.encoder_attn.out_proj.bias": "model.decoder.layers.16.encoder_attn.out_proj.bias",
  "decoder_layers.16.final_layer_norm.weight": "model.decoder.layers.16.final_layer_norm.weight",
  "decoder_layers.16.final_layer_norm.bias": "model.decoder.layers.16.final_layer_norm.bias",
  "decoder_layers.16.mlp.fc1.weight": "model.decoder.layers.16.fc1.weight",
  "decoder_layers.16.mlp.fc1.bias": "model.decoder.layers.16.fc1.bias",
  "decoder_layers.16.mlp.fc2.weight": "model.decoder.layers.16.fc2.weight",
  "decoder_layers.16.mlp.fc2.bias": "model.decoder.layers.16.fc2.bias",
  "decoder_layers.17.self_attn_layer_norm.weight": "model.decoder.layers.17.self_attn_layer_norm.weight",
  "decoder_layers.17.self_attn_layer_norm.bias": "model.decoder.layers.17.self_attn_layer_norm.bias",
  "decoder_layers.17.self_attn.q_proj.weight": "model.decoder.layers.17.self_attn.q_proj.weight",
  "decoder_layers.17.self_attn.q_proj.bias": "model.decoder.layers.17.self_attn.q_proj.bias",
  "decoder_layers.17.self_attn.k_proj.weight": "model.decoder.layers.17.self_attn.k_proj.weight",
  "decoder_layers.17.self_attn.v_proj.weight": "model.decoder.layers.17.self_attn.v_proj.weight",
  "decoder_layers.17.self_attn.v_proj.bias": "model.decoder.layers.17.self_attn.v_proj.bias",
  "decoder_layers.17.self_attn.out_proj.weight": "model.decoder.layers.17.self_attn.out_proj.weight",
  "decoder_layers.17.self_attn.out_proj.bias": "model.decoder.layers.17.self_attn.out_proj.bias",
  "decoder_layers.17.encoder_attn_layer_norm.weight": "model.decoder.layers.17.encoder_attn_layer_norm.weight",
  "decoder_layers.17.encoder_attn_layer_norm.bias": "model.decoder.layers.17.encoder_attn_layer_norm.bias",
  "decoder_layers.17.encoder_attn.q_proj.weight": "model.decoder.layers.17.encoder_attn.q_proj.weight",
  "decoder_layers.17.encoder_attn.q_proj.bias": "model.decoder.layers.17.encoder_attn.q_proj.bias",
  "decoder_layers.17.encoder_attn.k_proj.weight": "model.decoder.layers.17.encoder_attn.k_proj.weight",
  "decoder_layers.17.encoder_attn.v_proj.weight": "model.decoder.layers.17.encoder_attn.v_proj.weight",
  "decoder_layers.17.encoder_attn.v_proj.bias": "model.decoder.layers.17.encoder_attn.v_proj.bias",
  "decoder_layers.17.encoder_attn.out_proj.weight": "model.decoder.layers.17.encoder_attn.out_proj.weight",
  "decoder_layers.17.encoder_attn.out_proj.bias": "model.decoder.layers.17.encoder_attn.out_proj.bias",
  "decoder_layers.17.final_layer_norm.weight": "model.decoder.layers.17.final_layer_norm.weight",
  "decoder_layers.17.final_layer_norm.bias": "model.decoder.layers.17.final_layer_norm.bias",
  "decoder_layers.17.mlp.fc1.weight": "model.decoder.layers.17.fc1.weight",
  "decoder_layers.17.mlp.fc1.bias": "model.decoder.layers.17.fc1.bias",
  "decoder_layers.17.mlp.fc2.weight": "model.decoder.layers.17.fc2.weight",
  "decoder_layers.17.mlp.fc2.bias": "model.decoder.layers.17.fc2.bias",
  "decoder_layers.18.self_attn_layer_norm.weight": "model.decoder.layers.18.self_attn_layer_norm.weight",
  "decoder_layers.18.self_attn_layer_norm.bias": "model.decoder.layers.18.self_attn_layer_norm.bias",
  "decoder_layers.18.self_attn.q_proj.weight": "model.decoder.layers.18.self_attn.q_proj.weight",
  "decoder_layers.18.self_attn.q_proj.bias": "model.decoder.layers.18.self_attn.q_proj.bias",
  "decoder_layers.18.self_attn.k_proj.weight": "model.decoder.layers.18.self_attn.k_proj.weight",
  "decoder_layers.18.self_attn.v_proj.weight": "model.decoder.layers.18.self_attn.v_proj.weight",
  "decoder_layers.18.self_attn.v_proj.bias": "model.decoder.layers.18.self_attn.v_proj.bias",
  "decoder_layers.18.self_attn.out_proj.weight": "model.decoder.layers.18.self_attn.out_proj.weight",
  "decoder_layers.18.self_attn.out_proj.bias": "model.decoder.layers.18.self_attn.out_proj.bias",
  "decoder_layers.18.encoder_attn_layer_norm.weight": "model.decoder.layers.18.encoder_attn_layer_norm.weight",
  "decoder_layers.18.encoder_attn_layer_norm.bias": "model.decoder.layers.18.encoder_attn_layer_norm.bias",
  "decoder_layers.18.encoder_attn.q_proj.weight": "model.decoder.layers.18.encoder_attn.q_proj.weight",
  "decoder_layers.18.encoder_attn.q_proj.bias": "model.decoder.layers.18.encoder_attn.q_proj.bias",
  "decoder_layers.18.encoder_attn.k_proj.weight": "model.decoder.layers.18.encoder_attn.k_proj.weight",
  "decoder_layers.18.encoder_attn.v_proj.weight": "model.decoder.layers.18.encoder_attn.v_proj.weight",
  "decoder_layers.18.encoder_attn.v_proj.bias": "model.decoder.layers.18.encoder_attn.v_proj.bias",
  "decoder_layers.18.encoder_attn.out_proj.weight": "model.decoder.layers.18.encoder_attn.out_proj.weight",
  "decoder_layers.18.encoder_attn.out_proj.bias": "model.decoder.layers.18.encoder_attn.out_proj.bias",
  "decoder_layers.18.final_layer_norm.weight": "model.decoder.layers.18.final_layer_norm.weight",
  "decoder_layers.18.final_layer_norm.bias": "model.decoder.layers.18.final_layer_norm.bias",
  "decoder_layers.18.mlp.fc1.weight": "model.decoder.layers.18.fc1.weight",
  "decoder_layers.18.mlp.fc1.bias": "model.decoder.layers.18.fc1.bias",
  "decoder_layers.18.mlp.fc2.weight": "model.decoder.layers.18.fc2.weight",
  "decoder_layers.18.mlp.fc2.bias": "model.decoder.layers.18.fc2.bias",
  "decoder_layers.19.self_attn_layer_norm.weight": "model.decoder.layers.19.self_attn_layer_norm.weight",
  "decoder_layers.19.self_attn_layer_norm.bias": "model.decoder.layers.19.self_attn_layer_norm.bias",
  "decoder_layers.19.self_attn.q_proj.weight": "model.decoder.layers.19.self_attn.q_proj.weight",
  "decoder_layers.19.self_attn.q_proj.bias": "model.decoder.layers.19.self_attn.q_proj.bias",
  "decoder_layers.19.self_attn.k_proj.weight": "model.decoder.layers.19.self_attn.k_proj.weight",
  "decoder_layers.19.self_attn.v_proj.weight": "model.decoder.layers.19.self_attn.v_proj.weight",
  "decoder_layers.19.self_attn.v_proj.bias": "model.decoder.layers.19.self_attn.v_proj.bias",
  "decoder_layers.19.self_attn.out_proj.weight": "model.decoder.layers.19.self_attn.out_proj.weight",
  "decoder_layers.19.self_attn.out_proj.bias": "model.decoder.layers.19.self_attn.out_proj.bias",
  "decoder_layers.19.encoder_attn_layer_norm.weight": "model.decoder.layers.19.encoder_attn_layer_norm.weight",
  "decoder_layers.19.encoder_attn_layer_norm.bias": "model.decoder.layers.19.encoder_attn_layer_norm.bias",
  "decoder_layers.19.encoder_attn.q_proj.weight": "model.decoder.layers.19.encoder_attn.q_proj.weight",
  "decoder_layers.19.encoder_attn.q_proj.bias": "model.decoder.layers.19.encoder_attn.q_proj.bias",
  "decoder_layers.19.encoder_attn.k_proj.weight": "model.decoder.layers.19.encoder_attn.k_proj.weight",
  "decoder_layers.19.encoder_attn.v_proj.weight": "model.decoder.layers.19.encoder_attn.v_proj.weight",
  "decoder_layers.19.encoder_attn.v_proj.bias": "model.decoder.layers.19.encoder_attn.v_proj.bias",
  "decoder_layers.19.encoder_attn.out_proj.weight": "model.decoder.layers.19.encoder_attn.out_proj.weight",
  "decoder_layers.19.encoder_attn.out_proj.bias": "model.decoder.layers.19.encoder_attn.out_proj.bias",
  "decoder_layers.19.final_layer_norm.weight": "model.decoder.layers.19.final_layer_norm.weight",
  "decoder_layers.19.final_layer_norm.bias": "model.decoder.layers.19.final_layer_norm.bias",
  "decoder_layers.19.mlp.fc1.weight": "model.decoder.layers.19.fc1.weight",
  "decoder_layers.19.mlp.fc1.bias": "model.decoder.layers.19.fc1.bias",
  "decoder_layers.19.mlp.fc2.weight": "model.decoder.layers.19.fc2.weight",
  "decoder_layers.19.mlp.fc2.bias": "model.decoder.layers.19.fc2.bias",
  "decoder_layers.20.self_attn_layer_norm.weight": "model.decoder.layers.20.self_attn_layer_norm.weight",
  "decoder_layers.20.self_attn_layer_norm.bias": "model.decoder.layers.20.self_attn_layer_norm.bias",
  "decoder_layers.20.self_attn.q_proj.weight": "model.decoder.layers.20.self_attn.q_proj.weight",
  "decoder_layers.20.self_attn.q_proj.bias": "model.decoder.layers.20.self_attn.q_proj.bias",
  "decoder_layers.20.self_attn.k_proj.weight": "model.decoder.layers.20.self_attn.k_proj.weight",
  "decoder_layers.20.self_attn.v_proj.weight": "model.decoder.layers.20.self_attn.v_proj.weight",
  "decoder_layers.20.self_attn.v_proj.bias": "model.decoder.layers.20.self_attn.v_proj.bias",
  "decoder_layers.20.self_attn.out_proj.weight": "model.decoder.layers.20.self_attn.out_proj.weight",
  "decoder_layers.20.self_attn.out_proj.bias": "model.decoder.layers.20.self_attn.out_proj.bias",
  "decoder_layers.20.encoder_attn_layer_norm.weight": "model.decoder.layers.20.encoder_attn_layer_norm.weight",
  "decoder_layers.20.encoder_attn_layer_norm.bias": "model.decoder.layers.20.encoder_attn_layer_norm.bias",
  "decoder_layers.20.encoder_attn.q_proj.weight": "model.decoder.layers.20.encoder_attn.q_proj.weight",
  "decoder_layers.20.encoder_attn.q_proj.bias": "model.decoder.layers.20.encoder_attn.q_proj.bias",
  "decoder_layers.20.encoder_attn.k_proj.weight": "model.decoder.layers.20.encoder_attn.k_proj.weight",
  "decoder_layers.20.encoder_attn.v_proj.weight": "model.decoder.layers.20.encoder_attn.v_proj.weight",
  "decoder_layers.20.encoder_attn.v_proj.bias": "model.decoder.layers.20.encoder_attn.v_proj.bias",
  "decoder_layers.20.encoder_attn.out_proj.weight": "model.decoder.layers.20.encoder_attn.out_proj.weight",
  "decoder_layers.20.encoder_attn.out_proj.bias": "model.decoder.layers.20.encoder_attn.out_proj.bias",
  "decoder_layers.20.final_layer_norm.weight": "model.decoder.layers.20.final_layer_norm.weight",
  "decoder_layers.20.final_layer_norm.bias": "model.decoder.layers.20.final_layer_norm.bias",
  "decoder_layers.20.mlp.fc1.weight": "model.decoder.layers.20.fc1.weight",
  "decoder_layers.20.mlp.fc1.bias": "model.decoder.layers.20.fc1.bias",
  "decoder_layers.20.mlp.fc2.weight": "model.decoder.layers.20.fc2.weight",
  "decoder_layers.20.mlp.fc2.bias": "model.decoder.layers.20.fc2.bias",
  "decoder_layers.21.self_attn_layer_norm.weight": "model.decoder.layers.21.self_attn_layer_norm.weight",
  "decoder_layers.21.self_attn_layer_norm.bias": "model.decoder.layers.21.self_attn_layer_norm.bias",
  "decoder_layers.21.self_attn.q_proj.weight": "model.decoder.layers.21.self_attn.q_proj.weight",
  "decoder_layers.21.self_attn.q_proj.bias": "model.decoder.layers.21.self_attn.q_proj.bias",
  "decoder_layers.21.self_attn.k_proj.weight": "model.decoder.layers.21.self_attn.k_proj.weight",
  "decoder_layers.21.self_attn.v_proj.weight": "model.decoder.layers.21.self_attn.v_proj.weight",
  "decoder_layers.21.self_attn.v_proj.bias": "model.decoder.layers.21.self_attn.v_proj.bias",
  "decoder_layers.21.self_attn.out_proj.weight": "model.decoder.layers.21.self_attn.out_proj.weight",
  "decoder_layers.21.self_attn.out_proj.bias": "model.decoder.layers.21.self_attn.out_proj.bias",
  "decoder_layers.21.encoder_attn_layer_norm.weight": "model.decoder.layers.21.encoder_attn_layer_norm.weight",
  "decoder_layers.21.encoder_attn_layer_norm.bias": "model.decoder.layers.21.encoder_attn_layer_norm.bias",
  "decoder_layers.21.encoder_attn.q_proj.weight": "model.decoder.layers.21.encoder_attn.q_proj.weight",
  "decoder_layers.21.encoder_attn.q_proj.bias": "model.decoder.layers.21.encoder_attn.q_proj.bias",
  "decoder_layers.21.encoder_attn.k_proj.weight": "model.decoder.layers.21.encoder_attn.k_proj.weight",
  "decoder_layers.21.encoder_attn.v_proj.weight": "model.decoder.layers.21.encoder_attn.v_proj.weight",
  "decoder_layers.21.encoder_attn.v_proj.bias": "model.decoder.layers.21.encoder_attn.v_proj.bias",
  "decoder_layers.21.encoder_attn.out_proj.weight": "model.decoder.layers.21.encoder_attn.out_proj.weight",
  "decoder_layers.21.encoder_attn.out_proj.bias": "model.decoder.layers.21.encoder_attn.out_proj.bias",
  "decoder_layers.21.final_layer_norm.weight": "model.decoder.layers.21.final_layer_norm.weight",
  "decoder_layers.21.final_layer_norm.bias": "model.decoder.layers.21.final_layer_norm.bias",
  "decoder_layers.21.mlp.fc1.weight": "model.decoder.layers.21.fc1.weight",
  "decoder_layers.21.mlp.fc1.bias": "model.decoder.layers.21.fc1.bias",
  "decoder_layers.21.mlp.fc2.weight": "model.decoder.layers.21.fc2.weight",
  "decoder_layers.21.mlp.fc2.bias": "model.decoder.layers.21.fc2.bias",
  "decoder_layers.22.self_attn_layer_norm.weight": "model.decoder.layers.22.self_attn_layer_norm.weight",
  "decoder_layers.22.self_attn_layer_norm.bias": "model.decoder.layers.22.self_attn_layer_norm.bias",
  "decoder_layers.22.self_attn.q_proj.weight": "model.decoder.layers.22.self_attn.q_proj.weight",
  "decoder_layers.22.self_attn.q_proj.bias": "model.decoder.layers.22.self_attn.q_proj.bias",
  "decoder_layers.22.self_attn.k_proj.weight": "model.decoder.layers.22.self_attn.k_proj.weight",
  "decoder_layers.22.self_attn.v_proj.weight": "model.decoder.layers.22.self_attn.v_proj.weight",
  "decoder_layers.22.self_attn.v_proj.bias": "model.decoder.layers.22.self_attn.v_proj.bias",
  "decoder_layers.22.self_attn.out_proj.weight": "model.decoder.layers.22.self_attn.out_proj.weight",
  "decoder_layers.22.self_attn.out_proj.bias": "model.decoder.layers.22.self_attn.out_proj.bias",
  "decoder_layers.22.encoder_attn_layer_norm.weight": "model.decoder.layers.22.encoder_attn_layer_norm.weight",
  "decoder_layers.22.encoder_attn_layer_norm.bias": "model.decoder.layers.22.encoder_attn_layer_norm.bias",
  "decoder_layers.22.encoder_attn.q_proj.weight": "model.decoder.layers.22.encoder_attn.q_proj.weight",
  "decoder_layers.22.encoder_attn.q_proj.bias": "model.decoder.layers.22.encoder_attn.q_proj.bias",
  "decoder_layers.22.encoder_attn.k_proj.weight": "model.decoder.layers.22.encoder_attn.k_proj.weight",
  "decoder_layers.22.encoder_attn.v_proj.weight": "model.decoder.layers.22.encoder_attn.v_proj.weight",
  "decoder_layers.22.encoder_attn.v_proj.bias": "model.decoder.layers.22.encoder_attn.v_proj.bias",
  "decoder_layers.22.encoder_attn.out_proj.weight": "model.decoder.layers.22.encoder_attn.out_proj.weight",
  "decoder_layers.22.encoder_attn.out_proj.bias": "model.decoder.layers.22.encoder_attn.out_proj.bias",
  "decoder_layers.22.final_layer_norm.weight": "model.decoder.layers.22.final_layer_norm.weight",
  "decoder_layers.22.final_layer_norm.bias": "model.decoder.layers.22.final_layer_norm.bias",
  "decoder_layers.22.mlp.fc1.weight": "model.decoder.layers.22.fc1.weight",
  "decoder_layers.22.mlp.fc1.bias": "model.decoder.layers.22.fc1.bias",
  "decoder_layers.22.mlp.fc2.weight": "model.decoder.layers.22.fc2.weight",
  "decoder_layers.22.mlp.fc2.bias": "model.decoder.layers.22.fc2.bias",
  "decoder_layers.23.self_attn_layer_norm.weight": "model.decoder.layers.23.self_attn_layer_norm.weight",
  "decoder_layers.23.self_attn_layer_norm.bias": "model.decoder.layers.23.self_attn_layer_norm.bias",
  "decoder_layers.23.self_attn.q_proj.weight": "model.decoder.layers.23.self_attn.q_proj.weight",
  "decoder_layers.23.self_attn.q_proj.bias": "model.decoder.layers.23.self_attn.q_proj.bias",
  "decoder_layers.23.self_attn.k_proj.weight": "model.decoder.layers.23.self_attn.k_proj.weight",
  "decoder_layers.23.self_attn.v_proj.weight": "model.decoder.layers.23.self_attn.v_proj.weight",
  "decoder_layers.23.self_attn.v_proj.bias": "model.decoder.layers.23.self_attn.v_proj.bias",
  "decoder_layers.23.self_attn.out_proj.weight": "model.decoder.layers.23.self_attn.out_proj.weight",
  "decoder_layers.23.self_attn.out_proj.bias": "model.decoder.layers.23.self_attn.out_proj.bias",
  "decoder_layers.23.encoder_attn_layer_norm.weight": "model.decoder.layers.23.encoder_attn_layer_norm.weight",
  "decoder_layers.23.encoder_attn_layer_norm.bias": "model.decoder.layers.23.encoder_attn_layer_norm.bias",
  "decoder_layers.23.encoder_attn.q_proj.weight": "model.decoder.layers.23.encoder_attn.q_proj.weight",
  "decoder_layers.23.encoder_attn.q_proj.bias": "model.decoder.layers.23.encoder_attn.q_proj.bias",
  "decoder_layers.23.encoder_attn.k_proj.weight": "model.decoder.layers.23.encoder_attn.k_proj.weight",
  "decoder_layers.23.encoder_attn.v_proj.weight": "model.decoder.layers.23.encoder_attn.v_proj.weight",
  "decoder_layers.23.encoder_attn.v_proj.bias": "model.decoder.layers.23.encoder_attn.v_proj.bias",
  "decoder_layers.23.encoder_attn.out_proj.weight": "model.decoder.layers.23.encoder_attn.out_proj.weight",
  "decoder_layers.23.encoder_attn.out_proj.bias": "model.decoder.layers.23.encoder_attn.out_proj.bias",
  "decoder_layers.23.final_layer_norm.weight": "model.decoder.layers.23.final_layer_norm.weight",
  "decoder_layers.23.final_layer_norm.bias": "model.decoder.layers.23.final_layer_norm.bias",
  "decoder_layers.23.mlp.fc1.weight": "model.decoder.layers.23.fc1.weight",
  "decoder_layers.23.mlp.fc1.bias": "model.decoder.layers.23.fc1.bias",
  "decoder_layers.23.mlp.fc2.weight": "model.decoder.layers.23.fc2.weight",
  "decoder_layers.23.mlp.fc2.bias": "model.decoder.layers.23.fc2.bias",
  "decoder_layers.24.self_attn_layer_norm.weight": "model.decoder.layers.24.self_attn_layer_norm.weight",
  "decoder_layers.24.self_attn_layer_norm.bias": "model.decoder.layers.24.self_attn_layer_norm.bias",
  "decoder_layers.24.self_attn.q_proj.weight": "model.decoder.layers.24.self_attn.q_proj.weight",
  "decoder_layers.24.self_attn.q_proj.bias": "model.decoder.layers.24.self_attn.q_proj.bias",
  "decoder_layers.24.self_attn.k_proj.weight": "model.decoder.layers.24.self_attn.k_proj.weight",
  "decoder_layers.24.self_attn.v_proj.weight": "model.decoder.layers.24.self_attn.v_proj.weight",
  "decoder_layers.24.self_attn.v_proj.bias": "model.decoder.layers.24.self_attn.v_proj.bias",
  "decoder_layers.24.self_attn.out_proj.weight": "model.decoder.layers.24.self_attn.out_proj.weight",
  "decoder_layers.24.self_attn.out_proj.bias": "model.decoder.layers.24.self_attn.out_proj.bias",
  "decoder_layers.24.encoder_attn_layer_norm.weight": "model.decoder.layers.24.encoder_attn_layer_norm.weight",
  "decoder_layers.24.encoder_attn_layer_norm.bias": "model.decoder.layers.24.encoder_attn_layer_norm.bias",
  "decoder_layers.24.encoder_attn.q_proj.weight": "model.decoder.layers.24.encoder_attn.q_proj.weight",
  "decoder_layers.24.encoder_attn.q_proj.bias": "model.decoder.layers.24.encoder_attn.q_proj.bias",
  "decoder_layers.24.encoder_attn.k_proj.weight": "model.decoder.layers.24.encoder_attn.k_proj.weight",
  "decoder_layers.24.encoder_attn.v_proj.weight": "model.decoder.layers.24.encoder_attn.v_proj.weight",
  "decoder_layers.24.encoder_attn.v_proj.bias": "model.decoder.layers.24.encoder_attn.v_proj.bias",
  "decoder_layers.24.encoder_attn.out_proj.weight": "model.decoder.layers.24.encoder_attn.out_proj.weight",
  "decoder_layers.24.encoder_attn.out_proj.bias": "model.decoder.layers.24.encoder_attn.out_proj.bias",
  "decoder_layers.24.final_layer_norm.weight": "model.decoder.layers.24.final_layer_norm.weight",
  "decoder_layers.24.final_layer_norm.bias": "model.decoder.layers.24.final_layer_norm.bias",
  "decoder_layers.24.mlp.fc1.weight": "model.decoder.layers.24.fc1.weight",
  "decoder_layers.24.mlp.fc1.bias": "model.decoder.layers.24.fc1.bias",
  "decoder_layers.24.mlp.fc2.weight": "model.decoder.layers.24.fc2.weight",
  "decoder_layers.24.mlp.fc2.bias": "model.decoder.layers.24.fc2.bias",
  "decoder_layers.25.self_attn_layer_norm.weight": "model.decoder.layers.25.self_attn_layer_norm.weight",
  "decoder_layers.25.self_attn_layer_norm.bias": "model.decoder.layers.25.self_attn_layer_norm.bias",
  "decoder_layers.25.self_attn.q_proj.weight": "model.decoder.layers.25.self_attn.q_proj.weight",
  "decoder_layers.25.self_attn.q_proj.bias": "model.decoder.layers.25.self_attn.q_proj.bias",
  "decoder_layers.25.self_attn.k_proj.weight": "model.decoder.layers.25.self_attn.k_proj.weight",
  "decoder_layers.25.self_attn.v_proj.weight": "model.decoder.layers.25.self_attn.v_proj.weight",
  "decoder_layers.25.self_attn.v_proj.bias": "model.decoder.layers.25.self_attn.v_proj.bias",
  "decoder_layers.25.self_attn.out_proj.weight": "model.decoder.layers.25.self_attn.out_proj.weight",
  "decoder_layers.25.self_attn.out_proj.bias": "model.decoder.layers.25.self_attn.out_proj.bias",
  "decoder_layers.25.encoder_attn_layer_norm.weight": "model.decoder.layers.25.encoder_attn_layer_norm.weight",
  "decoder_layers.25.encoder_attn_layer_norm.bias": "model.decoder.layers.25.encoder_attn_layer_norm.bias",
  "decoder_layers.25.encoder_attn.q_proj.weight": "model.decoder.layers.25.encoder_attn.q_proj.weight",
  "decoder_layers.25.encoder_attn.q_proj.bias": "model.decoder.layers.25.encoder_attn.q_proj.bias",
  "decoder_layers.25.encoder_attn.k_proj.weight": "model.decoder.layers.25.encoder_attn.k_proj.weight",
  "decoder_layers.25.encoder_attn.v_proj.weight": "model.decoder.layers.25.encoder_attn.v_proj.weight",
  "decoder_layers.25.encoder_attn.v_proj.bias": "model.decoder.layers.25.encoder_attn.v_proj.bias",
  "decoder_layers.25.encoder_attn.out_proj.weight": "model.decoder.layers.25.encoder_attn.out_proj.weight",
  "decoder_layers.25.encoder_attn.out_proj.bias": "model.decoder.layers.25.encoder_attn.out_proj.bias",
  "decoder_layers.25.final_layer_norm.weight": "model.decoder.layers.25.final_layer_norm.weight",
  "decoder_layers.25.final_layer_norm.bias": "model.decoder.layers.25.final_layer_norm.bias",
  "decoder_layers.25.mlp.fc1.weight": "model.decoder.layers.25.fc1.weight",
  "decoder_layers.25.mlp.fc1.bias": "model.decoder.layers.25.fc1.bias",
  "decoder_layers.25.mlp.fc2.weight": "model.decoder.layers.25.fc2.weight",
  "decoder_layers.25.mlp.fc2.bias": "model.decoder.layers.25.fc2.bias",
  "decoder_layers.26.self_attn_layer_norm.weight": "model.decoder.layers.26.self_attn_layer_norm.weight",
  "decoder_layers.26.self_attn_layer_norm.bias": "model.decoder.layers.26.self_attn_layer_norm.bias",
  "decoder_layers.26.self_attn.q_proj.weight": "model.decoder.layers.26.self_attn.q_proj.weight",
  "decoder_layers.26.self_attn.q_proj.bias": "model.decoder.layers.26.self_attn.q_proj.bias",
  "decoder_layers.26.self_attn.k_proj.weight": "model.decoder.layers.26.self_attn.k_proj.weight",
  "decoder_layers.26.self_attn.v_proj.weight": "model.decoder.layers.26.self_attn.v_proj.weight",
  "decoder_layers.26.self_attn.v_proj.bias": "model.decoder.layers.26.self_attn.v_proj.bias",
  "decoder_layers.26.self_attn.out_proj.weight": "model.decoder.layers.26.self_attn.out_proj.weight",
  "decoder_layers.26.self_attn.out_proj.bias": "model.decoder.layers.26.self_attn.out_proj.bias",
  "decoder_layers.26.encoder_attn_layer_norm.weight": "model.decoder.layers.26.encoder_attn_layer_norm.weight",
  "decoder_layers.26.encoder_attn_layer_norm.bias": "model.decoder.layers.26.encoder_attn_layer_norm.bias",
  "decoder_layers.26.encoder_attn.q_proj.weight": "model.decoder.layers.26.encoder_attn.q_proj.weight",
  "decoder_layers.26.encoder_attn.q_proj.bias": "model.decoder.layers.26.encoder_attn.q_proj.bias",
  "decoder_layers.26.encoder_attn.k_proj.weight": "model.decoder.layers.26.encoder_attn.k_proj.weight",
  "decoder_layers.26.encoder_attn.v_proj.weight": "model.decoder.layers.26.encoder_attn.v_proj.weight",
  "decoder_layers.26.encoder_attn.v_proj.bias": "model.decoder.layers.26.encoder_attn.v_proj.bias",
  "decoder_layers.26.encoder_attn.out_proj.weight": "model.decoder.layers.26.encoder_attn.out_proj.weight",
  "decoder_layers.26.encoder_attn.out_proj.bias": "model.decoder.layers.26.encoder_attn.out_proj.bias",
  "decoder_layers.26.final_layer_norm.weight": "model.decoder.layers.26.final_layer_norm.weight",
  "decoder_layers.26.final_layer_norm.bias": "model.decoder.layers.26.final_layer_norm.bias",
  "decoder_layers.26.mlp.fc1.weight": "model.decoder.layers.26.fc1.weight",
  "decoder_layers.26.mlp.fc1.bias": "model.decoder.layers.26.fc1.bias",
  "decoder_layers.26.mlp.fc2.weight": "model.decoder.layers.26.fc2.weight",
  "decoder_layers.26.mlp.fc2.bias": "model.decoder.layers.26.fc2.bias",
  "decoder_layers.27.self_attn_layer_norm.weight": "model.decoder.layers.27.self_attn_layer_norm.weight",
  "decoder_layers.27.self_attn_layer_norm.bias": "model.decoder.layers.27.self_attn_layer_norm.bias",
  "decoder_layers.27.self_attn.q_proj.weight": "model.decoder.layers.27.self_attn.q_proj.weight",
  "decoder_layers.27.self_attn.q_proj.bias": "model.decoder.layers.27.self_attn.q_proj.bias",
  "decoder_layers.27.self_attn.k_proj.weight": "model.decoder.layers.27.self_attn.k_proj.weight",
  "decoder_layers.27.self_attn.v_proj.weight": "model.decoder.layers.27.self_attn.v_proj.weight",
  "decoder_layers.27.self_attn.v_proj.bias": "model.decoder.layers.27.self_attn.v_proj.bias",
  "decoder_layers.27.self_attn.out_proj.weight": "model.decoder.layers.27.self_attn.out_proj.weight",
  "decoder_layers.27.self_attn.out_proj.bias": "model.decoder.layers.27.self_attn.out_proj.bias",
  "decoder_layers.27.encoder_attn_layer_norm.weight": "model.decoder.layers.27.encoder_attn_layer_norm.weight",
  "decoder_layers.27.encoder_attn_layer_norm.bias": "model.decoder.layers.27.encoder_attn_layer_norm.bias",
  "decoder_layers.27.encoder_attn.q_proj.weight": "model.decoder.layers.27.encoder_attn.q_proj.weight",
  "decoder_layers.27.encoder_attn.q_proj.bias": "model.decoder.layers.27.encoder_attn.q_proj.bias",
  "decoder_layers.27.encoder_attn.k_proj.weight": "model.decoder.layers.27.encoder_attn.k_proj.weight",
  "decoder_layers.27.encoder_attn.v_proj.weight": "model.decoder.layers.27.encoder_attn.v_proj.weight",
  "decoder_layers.27.encoder_attn.v_proj.bias": "model.decoder.layers.27.encoder_attn.v_proj.bias",
  "decoder_layers.27.encoder_attn.out_proj.weight": "model.decoder.layers.27.encoder_attn.out_proj.weight",
  "decoder_layers.27.encoder_attn.out_proj.bias": "model.decoder.layers.27.encoder_attn.out_proj.bias",
  "decoder_layers.27.final_layer_norm.weight": "model.decoder.layers.27.final_layer_norm.weight",
  "decoder_layers.27.final_layer_norm.bias": "model.decoder.layers.27.final_layer_norm.bias",
  "decoder_layers.27.mlp.fc1.weight": "model.decoder.layers.27.fc1.weight",
  "decoder_layers.27.mlp.fc1.bias": "model.decoder.layers.27.fc1.bias",
  "decoder_layers.27.mlp.fc2.weight": "model.decoder.layers.27.fc2.weight",
  "decoder_layers.27.mlp.fc2.bias": "model.decoder.layers.27.fc2.bias",
  "decoder_layers.28.self_attn_layer_norm.weight": "model.decoder.layers.28.self_attn_layer_norm.weight",
  "decoder_layers.28.self_attn_layer_norm.bias": "model.decoder.layers.28.self_attn_layer_norm.bias",
  "decoder_layers.28.self_attn.q_proj.weight": "model.decoder.layers.28.self_attn.q_proj.weight",
  "decoder_layers.28.self_attn.q_proj.bias": "model.decoder.layers.28.self_attn.q_proj.bias",
  "decoder_layers.28.self_attn.k_proj.weight": "model.decoder.layers.28.self_attn.k_proj.weight",
  "decoder_layers.28.self_attn.v_proj.weight": "model.decoder.layers.28.self_attn.v_proj.weight",
  "decoder_layers.28.self_attn.v_proj.bias": "model.decoder.layers.28.self_attn.v_proj.bias",
  "decoder_layers.28.self_attn.out_proj.weight": "model.decoder.layers.28.self_attn.out_proj.weight",
  "decoder_layers.28.self_attn.out_proj.bias": "model.decoder.layers.28.self_attn.out_proj.bias",
  "decoder_layers.28.encoder_attn_layer_norm.weight": "model.decoder.layers.28.encoder_attn_layer_norm.weight",
  "decoder_layers.28.encoder_attn_layer_norm.bias": "model.decoder.layers.28.encoder_attn_layer_norm.bias",
  "decoder_layers.28.encoder_attn.q_proj.weight": "model.decoder.layers.28.encoder_attn.q_proj.weight",
  "decoder_layers.28.encoder_attn.q_proj.bias": "model.decoder.layers.28.encoder_attn.q_proj.bias",
  "decoder_layers.28.encoder_attn.k_proj.weight": "model.decoder.layers.28.encoder_attn.k_proj.weight",
  "decoder_layers.28.encoder_attn.v_proj.weight": "model.decoder.layers.28.encoder_attn.v_proj.weight",
  "decoder_layers.28.encoder_attn.v_proj.bias": "model.decoder.layers.28.encoder_attn.v_proj.bias",
  "decoder_layers.28.encoder_attn.out_proj.weight": "model.decoder.layers.28.encoder_attn.out_proj.weight",
  "decoder_layers.28.encoder_attn.out_proj.bias": "model.decoder.layers.28.encoder_attn.out_proj.bias",
  "decoder_layers.28.final_layer_norm.weight": "model.decoder.layers.28.final_layer_norm.weight",
  "decoder_layers.28.final_layer_norm.bias": "model.decoder.layers.28.final_layer_norm.bias",
  "decoder_layers.28.mlp.fc1.weight": "model.decoder.layers.28.fc1.weight",
  "decoder_layers.28.mlp.fc1.bias": "model.decoder.layers.28.fc1.bias",
  "decoder_layers.28.mlp.fc2.weight": "model.decoder.layers.28.fc2.weight",
  "decoder_layers.28.mlp.fc2.bias": "model.decoder.layers.28.fc2.bias",
  "decoder_layers.29.self_attn_layer_norm.weight": "model.decoder.layers.29.self_attn_layer_norm.weight",
  "decoder_layers.29.self_attn_layer_norm.bias": "model.decoder.layers.29.self_attn_layer_norm.bias",
  "decoder_layers.29.self_attn.q_proj.weight": "model.decoder.layers.29.self_attn.q_proj.weight",
  "decoder_layers.29.self_attn.q_proj.bias": "model.decoder.layers.29.self_attn.q_proj.bias",
  "decoder_layers.29.self_attn.k_proj.weight": "model.decoder.layers.29.self_attn.k_proj.weight",
  "decoder_layers.29.self_attn.v_proj.weight": "model.decoder.layers.29.self_attn.v_proj.weight",
  "decoder_layers.29.self_attn.v_proj.bias": "model.decoder.layers.29.self_attn.v_proj.bias",
  "decoder_layers.29.self_attn.out_proj.weight": "model.decoder.layers.29.self_attn.out_proj.weight",
  "decoder_layers.29.self_attn.out_proj.bias": "model.decoder.layers.29.self_attn.out_proj.bias",
  "decoder_layers.29.encoder_attn_layer_norm.weight": "model.decoder.layers.29.encoder_attn_layer_norm.weight",
  "decoder_layers.29.encoder_attn_layer_norm.bias": "model.decoder.layers.29.encoder_attn_layer_norm.bias",
  "decoder_layers.29.encoder_attn.q_proj.weight": "model.decoder.layers.29.encoder_attn.q_proj.weight",
  "decoder_layers.29.encoder_attn.q_proj.bias": "model.decoder.layers.29.encoder_attn.q_proj.bias",
  "decoder_layers.29.encoder_attn.k_proj.weight": "model.decoder.layers.29.encoder_attn.k_proj.weight",
  "decoder_layers.29.encoder_attn.v_proj.weight": "model.decoder.layers.29.encoder_attn.v_proj.weight",
  "decoder_layers.29.encoder_attn.v_proj.bias": "model.decoder.layers.29.encoder_attn.v_proj.bias",
  "decoder_layers.29.encoder_attn.out_proj.weight": "model.decoder.layers.29.encoder_attn.out_proj.weight",
  "decoder_layers.29.encoder_attn.out_proj.bias": "model.decoder.layers.29.encoder_attn.out_proj.bias",
  "decoder_layers.29.final_layer_norm.weight": "model.decoder.layers.29.final_layer_norm.weight",
  "decoder_layers.29.final_layer_norm.bias": "model.decoder.layers.29.final_layer_norm.bias",
  "decoder_layers.29.mlp.fc1.weight": "model.decoder.layers.29.fc1.weight",
  "decoder_layers.29.mlp.fc1.bias": "model.decoder.layers.29.fc1.bias",
  "decoder_layers.29.mlp.fc2.weight": "model.decoder.layers.29.fc2.weight",
  "decoder_layers.29.mlp.fc2.bias": "model.decoder.layers.29.fc2.bias",
  "decoder_layers.30.self_attn_layer_norm.weight": "model.decoder.layers.30.self_attn_layer_norm.weight",
  "decoder_layers.30.self_attn_layer_norm.bias": "model.decoder.layers.30.self_attn_layer_norm.bias",
  "decoder_layers.30.self_attn.q_proj.weight": "model.decoder.layers.30.self_attn.q_proj.weight",
  "decoder_layers.30.self_attn.q_proj.bias": "model.decoder.layers.30.self_attn.q_proj.bias",
  "decoder_layers.30.self_attn.k_proj.weight": "model.decoder.layers.30.self_attn.k_proj.weight",
  "decoder_layers.30.self_attn.v_proj.weight": "model.decoder.layers.30.self_attn.v_proj.weight",
  "decoder_layers.30.self_attn.v_proj.bias": "model.decoder.layers.30.self_attn.v_proj.bias",
  "decoder_layers.30.self_attn.out_proj.weight": "model.decoder.layers.30.self_attn.out_proj.weight",
  "decoder_layers.30.self_attn.out_proj.bias": "model.decoder.layers.30.self_attn.out_proj.bias",
  "decoder_layers.30.encoder_attn_layer_norm.weight": "model.decoder.layers.30.encoder_attn_layer_norm.weight",
  "decoder_layers.30.encoder_attn_layer_norm.bias": "model.decoder.layers.30.encoder_attn_layer_norm.bias",
  "decoder_layers.30.encoder_attn.q_proj.weight": "model.decoder.layers.30.encoder_attn.q_proj.weight",
  "decoder_layers.30.encoder_attn.q_proj.bias": "model.decoder.layers.30.encoder_attn.q_proj.bias",
  "decoder_layers.30.encoder_attn.k_proj.weight": "model.decoder.layers.30.encoder_attn.k_proj.weight",
  "decoder_layers.30.encoder_attn.v_proj.weight": "model.decoder.layers.30.encoder_attn.v_proj.weight",
  "decoder_layers.30.encoder_attn.v_proj.bias": "model.decoder.layers.30.encoder_attn.v_proj.bias",
  "decoder_layers.30.encoder_attn.out_proj.weight": "model.decoder.layers.30.encoder_attn.out_proj.weight",
  "decoder_layers.30.encoder_attn.out_proj.bias": "model.decoder.layers.30.encoder_attn.out_proj.bias",
  "decoder_layers.30.final_layer_norm.weight": "model.decoder.layers.30.final_layer_norm.weight",
  "decoder_layers.30.final_layer_norm.bias": "model.decoder.layers.30.final_layer_norm.bias",
  "decoder_layers.30.mlp.fc1.weight": "model.decoder.layers.30.fc1.weight",
  "decoder_layers.30.mlp.fc1.bias": "model.decoder.layers.30.fc1.bias",
  "decoder_layers.30.mlp.fc2.weight": "model.decoder.layers.30.fc2.weight",
  "decoder_layers.30.mlp.fc2.bias": "model.decoder.layers.30.fc2.bias",
  "decoder_layers.31.self_attn_layer_norm.weight": "model.decoder.layers.31.self_attn_layer_norm.weight",
  "decoder_layers.31.self_attn_layer_norm.bias": "model.decoder.layers.31.self_attn_layer_norm.bias",
  "decoder_layers.31.self_attn.q_proj.weight": "model.decoder.layers.31.self_attn.q_proj.weight",
  "decoder_layers.31.self_attn.q_proj.bias": "model.decoder.layers.31.self_attn.q_proj.bias",
  "decoder_layers.31.self_attn.k_proj.weight": "model.decoder.layers.31.self_attn.k_proj.weight",
  "decoder_layers.31.self_attn.v_proj.weight": "model.decoder.layers.31.self_attn.v_proj.weight",
  "decoder_layers.31.self_attn.v_proj.bias": "model.decoder.layers.31.self_attn.v_proj.bias",
  "decoder_layers.31.self_attn.out_proj.weight": "model.decoder.layers.31.self_attn.out_proj.weight",
  "decoder_layers.31.self_attn.out_proj.bias": "model.decoder.layers.31.self_attn.out_proj.bias",
  "decoder_layers.31.encoder_attn_layer_norm.weight": "model.decoder.layers.31.encoder_attn_layer_norm.weight",
  "decoder_layers.31.encoder_attn_layer_norm.bias": "model.decoder.layers.31.encoder_attn_layer_norm.bias",
  "decoder_layers.31.encoder_attn.q_proj.weight": "model.decoder.layers.31.encoder_attn.q_proj.weight",
  "decoder_layers.31.encoder_attn.q_proj.bias": "model.decoder.layers.31.encoder_attn.q_proj.bias",
  "decoder_layers.31.encoder_attn.k_proj.weight": "model.decoder.layers.31.encoder_attn.k_proj.weight",
  "decoder_layers.31.encoder_attn.v_proj.weight": "model.decoder.layers.31.encoder_attn.v_proj.weight",
  "decoder_layers.31.encoder_attn.v_proj.bias": "model.decoder.layers.31.encoder_attn.v_proj.bias",
  "decoder_layers.31.encoder_attn.out_proj.weight": "model.decoder.layers.31.encoder_attn.out_proj.weight",
  "decoder_layers.31.encoder_attn.out_proj.bias": "model.decoder.layers.31.encoder_attn.out_proj.bias",
  "decoder_layers.31.final_layer_norm.weight": "model.decoder.layers.31.final_layer_norm.weight",
  "decoder_layers.31.final_layer_norm.bias": "model.decoder.layers.31.final_layer_norm.bias",
  "decoder_layers.31.mlp.fc1.weight": "model.decoder.layers.31.fc1.weight",
  "decoder_layers.31.mlp.fc1.bias": "model.decoder.layers.31.fc1.bias",
  "decoder_layers.31.mlp.fc2.weight": "model.decoder.layers.31.fc2.weight",
  "decoder_layers.31.mlp.fc2.bias": "model.decoder.layers.31.fc2.bias",
  "decoder_norm.weight": "model.decoder.layer_norm.weight",
  "decoder_norm.bias": "model.decoder.layer_norm.bias"
}

Open on GitHub

linnet.toml62 B
toml
[package]
name = "whisper"
version = "0.1.0"
language = "0.1"

Open on GitHub

nest.toml960 B
toml
[model]
name = "whisper-large-v3"
title = "Whisper large-v3"
summary = "OpenAI's largest multilingual speech recognition model: a convolutional stem over a 128-bin log-mel spectrogram, 32 encoder layers, and a 32-layer decoder that cross-attends to them, at width 1280."
license = "Apache-2.0"
family = "whisper"
tags = ["automatic-speech-recognition", "audio", "encoder-decoder", "cross-attention", "multilingual"]

[links]
huggingface = "https://huggingface.co/openai/whisper-large-v3"
github = "https://github.com/openai/whisper"
arxiv = "https://arxiv.org/abs/2212.04356"

[source]
path = "src/lib.linnet"
root = "Model"
entry = "encode"

[generics]
Mels = 128
Frames = 3000
D = 1280
Heads = 20
Inner = 5120
EncoderLayers = 32
DecoderLayers = 32
Vocab = 51866
MaxTokens = 448
T = "f16"

[check]
B = 1

[weights]
repo = "openai/whisper-large-v3"
revision = "06f233fe06e710322aca913c1bc4249a0d71fce1"
files = ["model.safetensors"]
bindings = "bindings.json"

Open on GitHub

samples.json323 B
json
{
  "kind": "transcription",
  "audio": "hf-internal-testing/librispeech_asr_dummy, clean/validation[0]",
  "text": "\"Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.",
  "reference_text": "Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel."
}

Open on GitHub

src
lib.linnet14.6 kB
linnet
// Whisper: speech recognition as an encoder-decoder transformer. The encoder
// reads a log-mel spectrogram through two 1-D convolutions that halve the
// frame rate, adds the position table the checkpoint stores, and runs pre-norm
// encoder layers over the result. The decoder is an ordinary causal language
// model with a cross-attention over those encoder states inserted between its
// self-attention and its MLP, and an output head tied to the token embedding.
//
// The two are separate entries because transcription uses them separately:
// the encoder runs once over the 30 seconds of audio, the decoder runs once
// per token against the states it produced.
module whisper

use std.nn.activations::{gelu_erf}
use std.nn.attention::{attention, causal_mask}
use std.nn.cache::{write_at, write_span}
use std.nn.embedding::{embedding}
use std.nn.linear::{Linear, linear}
use std.nn.norm::{layer_norm}

// Whisper normalizes with `torch.nn.LayerNorm`'s default epsilon.
pub const EPS: f32 = 1e-5

pub block LayerNorm<D: Dim, T: Float> {
    param weight: Tensor[D; T]
    param bias: Tensor[D; T]

    pub fn forward<*S: Shape>(x: Tensor[*S, D; T]) -> Tensor[*S, D; T] {
        return layer_norm(x, weight, some(bias), EPS)
    }
}

// Zero padding on the position axis of a `[B, Positions, Channels]` signal.
// `P = 0` concatenates empty tensors, which every backend folds away.
fn pad1d<B: Dim, L: Dim, C: Dim, P: Dim, T: Float>(
    x: Tensor[B, L, C; T],
) -> Tensor[B, L + 2 * P, C; T] {
    let edge = fill<T>([B, P, C], 0.0)
    return concat(concat(edge, x, axis = 1), edge, axis = 1)
}

// A 1-D convolution, which the standard library does not have: `conv2d` takes
// a square kernel, and the stem's kernel is a line of three frames. The
// window geometry is built the way `std.nn.conv::conv2d` builds it, because an
// index expression may not do arithmetic: the tap positions are a tensor made
// from `iota` and the padded signal is gathered through it.
//
// Signals are `[B, Positions, Channels]` here, the layout the rest of the
// encoder wants, so the gathered window can be flattened over (channel, tap)
// and the convolution finished as one `linear` against the kernel flattened
// the same way. That matters: there is no `conv1d` op for a backend to
// recognize, so a five-axis product written out in index notation would stay
// a five-axis product, while `linear` selects a matrix multiply everywhere.
fn conv1d<
    B: Dim,
    L: Dim,
    Cin: Dim,
    Cout: Dim,
    K: Dim,
    Stride: Dim,
    Pad: Dim,
    T: Float,
>(
    x: Tensor[B, L, Cin; T],
    weight: Tensor[Cout, Cin, K; T],
    bias: Tensor[Cout; T],
) -> Tensor[B, (L + 2 * Pad - K) / Stride + 1, Cout; T]
where
    Stride > 0,
    K > 0
{
    let padded = pad1d<B, L, Cin, Pad, T>(x)
    // Window `o` starts at `o * Stride`, so tap `k` of it reads position
    // `o * Stride + k` of the padded signal.
    let taps[o, k] =
        iota<i32>((L + 2 * Pad - K) / Stride + 1)[o] * cast<i32>(Stride) + iota<i32>(K)[k]
    // The gather transposes as it reads, so that (channel, tap) flattens in
    // the order the checkpoint's `[Cout, Cin, K]` kernel flattens in.
    let windows[b, o, ci, k] = padded[b, taps[o, k], ci]
    return linear(
        reshape(windows, [B, (L + 2 * Pad - K) / Stride + 1, Cin * K]),
        reshape(weight, [Cout, Cin * K]),
        some(bias),
    )
}

// Attention over an explicit key/value source, which is what lets one block
// serve both self-attention (the source is the sequence itself) and
// cross-attention (the source is the encoder's states).
//
// Whisper projects queries, values, and the output with a bias but keys
// without one. The asymmetry is in the reference implementation and it is
// real: a key bias shifts every key by the same vector, which a softmax over
// dot products does not ignore, so leaving it out is a choice the checkpoint
// was trained with rather than a redundancy.
pub block Attention<D: Dim, Heads: Dim, T: Float>
where
    Heads > 0,
    D % Heads == 0
{
    sub q_proj: Linear<D, D, T>
    sub k_proj: Linear<D, D, T>
    sub v_proj: Linear<D, D, T>
    sub out_proj: Linear<D, D, T>

    pub fn forward<B: Dim, Q: Dim, Src: Dim>(
        x: Tensor[B, Q, D; T],
        source: Tensor[B, Src, D; T],
        mask: Tensor[Q, Src; bool]?,
    ) -> Tensor[B, Q, D; T] {
        return attend(x, keys(source), values(source), mask)
    }

    // The keys and the values of `source`, split into heads: what a cache
    // holds so that later tokens do not project them again.
    pub fn keys<B: Dim, N: Dim>(source: Tensor[B, N, D; T]) -> Tensor[B, Heads, N, D / Heads; T] {
        return heads<B, N, Heads, D / Heads, T>(k_proj.forward(source))
    }

    pub fn values<B: Dim, N: Dim>(
        source: Tensor[B, N, D; T],
    ) -> Tensor[B, Heads, N, D / Heads; T] {
        return heads<B, N, Heads, D / Heads, T>(v_proj.forward(source))
    }

    // `x`'s queries over keys and values already split into heads.
    pub fn attend<B: Dim, Q: Dim, K: Dim>(
        x: Tensor[B, Q, D; T],
        keys: Tensor[B, Heads, K, D / Heads; T],
        values: Tensor[B, Heads, K, D / Heads; T],
        mask: Tensor[Q, K; bool]?,
    ) -> Tensor[B, Q, D; T] {
        let mixed = attention(
            heads<B, Q, Heads, D / Heads, T>(q_proj.forward(x)),
            keys,
            values,
            rsqrt(cast<f32>(D / Heads)),
            mask,
        )
        return out_proj.forward(reshape(permute(mixed, [0, 2, 1, 3]), [B, Q, D]))
    }
}

fn heads<B: Dim, N: Dim, Heads: Dim, Dh: Dim, T: Float>(
    x: Tensor[B, N, Heads * Dh; T],
) -> Tensor[B, Heads, N, Dh; T] {
    return permute(reshape(x, [B, N, Heads, Dh]), [0, 2, 1, 3])
}

// The MLP of both stacks: one widening projection, GELU, one narrowing
// projection.
pub block Mlp<D: Dim, Inner: Dim, T: Float> {
    sub fc1: Linear<D, Inner, T>
    sub fc2: Linear<Inner, D, T>

    pub fn forward<*S: Shape>(x: Tensor[*S, D; T]) -> Tensor[*S, D; T] {
        return fc2.forward(gelu_erf(fc1.forward(x)))
    }
}

// Every audio position attends to every other: the spectrogram is padded to a
// fixed 30 seconds, and the reference encoder attends over the padding too.
pub block EncoderLayer<D: Dim, Heads: Dim, Inner: Dim, T: Float>
where
    Heads > 0,
    D % Heads == 0
{
    sub self_attn_layer_norm: LayerNorm<D, T>
    sub self_attn: Attention<D, Heads, T>
    sub final_layer_norm: LayerNorm<D, T>
    sub mlp: Mlp<D, Inner, T>

    pub fn forward<B: Dim, N: Dim>(x: Tensor[B, N, D; T]) -> Tensor[B, N, D; T] {
        let normed = self_attn_layer_norm.forward(x)
        let attended = x + self_attn.forward(normed, normed, none)
        return attended + mlp.forward(final_layer_norm.forward(attended))
    }
}

// Three residual steps: causal self-attention over the tokens decoded so far,
// then attention over the encoder's states, then the MLP. The cross-attention
// needs no mask, since every token may read the whole utterance.
//
// For decoding one token at a time the layer keeps what it would otherwise
// recompute: the keys and values of every token so far (`self_k`, `self_v`,
// `MaxTokens` positions) and those of the encoder's states (`cross_k`,
// `cross_v`, `Audio` positions), which `listen` fills once per utterance.
pub block DecoderLayer<
    D: Dim,
    Heads: Dim,
    Inner: Dim,
    Batch: Dim,
    MaxTokens: Dim,
    Audio: Dim,
    T: Float,
>
where
    Heads > 0,
    D % Heads == 0,
    MaxTokens > 0
{
    sub self_attn_layer_norm: LayerNorm<D, T>
    sub self_attn: Attention<D, Heads, T>
    sub encoder_attn_layer_norm: LayerNorm<D, T>
    sub encoder_attn: Attention<D, Heads, T>
    sub final_layer_norm: LayerNorm<D, T>
    sub mlp: Mlp<D, Inner, T>

    state self_k: Tensor[Batch, Heads, MaxTokens, D / Heads; T]
    state self_v: Tensor[Batch, Heads, MaxTokens, D / Heads; T]
    state cross_k: Tensor[Batch, Heads, Audio, D / Heads; T]
    state cross_v: Tensor[Batch, Heads, Audio, D / Heads; T]

    pub fn forward<B: Dim, S: Dim, A: Dim>(
        x: Tensor[B, S, D; T],
        audio: Tensor[B, A, D; T],
    ) -> Tensor[B, S, D; T] {
        let normed = self_attn_layer_norm.forward(x)
        let attended = x + self_attn.forward(normed, normed, some(causal_mask<S, S>()))
        let crossed =
            attended + encoder_attn.forward(encoder_attn_layer_norm.forward(attended), audio, none)
        return crossed + mlp.forward(final_layer_norm.forward(crossed))
    }

    // The encoder's states as this layer's cross-attention reads them.
    pub fn listen(audio: Tensor[Batch, Audio, D; T]) -> Tensor[Batch, Audio, D; T] {
        cross_k = encoder_attn.keys(audio)
        cross_v = encoder_attn.values(audio)
        return audio
    }

    // A prompt of `S` tokens from the first position: their keys and values
    // go into the caches, and they attend causally among themselves.
    pub fn prefill<S: Dim>(x: Tensor[Batch, S, D; T]) -> Tensor[Batch, S, D; T]
    where S > 0 {
        let normed = self_attn_layer_norm.forward(x)
        let k = self_attn.keys(normed)
        let v = self_attn.values(normed)
        self_k = write_span(self_k, k, 0)
        self_v = write_span(self_v, v, 0)
        let attended = x + self_attn.attend(normed, k, v, some(causal_mask<S, S>()))
        return cross(attended)
    }

    // One token at position `pos`, over the tokens before it in the cache.
    pub fn step(x: Tensor[Batch, 1, D; T], pos: i32) -> Tensor[Batch, 1, D; T] {
        let normed = self_attn_layer_norm.forward(x)
        self_k = write_at(self_k, self_attn.keys(normed), pos)
        self_v = write_at(self_v, self_attn.values(normed), pos)
        let slots = iota<i32>(MaxTokens)
        let seen[t] = slots[t] <= pos
        let attended =
            x + self_attn.attend(normed, self_k, self_v, some(reshape(seen, [1, MaxTokens])))
        return cross(attended)
    }

    // Cross-attention over the cached encoder states, then the MLP.
    fn cross<S: Dim>(attended: Tensor[Batch, S, D; T]) -> Tensor[Batch, S, D; T] {
        let crossed =
            attended +
            encoder_attn.attend(
                encoder_attn_layer_norm.forward(attended),
                cross_k,
                cross_v,
                none,
            )
        return crossed + mlp.forward(final_layer_norm.forward(crossed))
    }
}

pub block Model<
    Mels: Dim,
    Frames: Dim,
    D: Dim,
    Heads: Dim,
    Inner: Dim,
    EncoderLayers: Dim,
    DecoderLayers: Dim,
    Vocab: Dim,
    MaxTokens: Dim,
    T: Float = f32,
    Batch: Dim = 1,
>
where
    Heads > 0,
    D % Heads == 0,
    Frames > 0,
    MaxTokens > 0
{
    // The stem, with the `[Cout, Cin, K]` kernels the checkpoint stores.
    param conv1_weight: Tensor[D, Mels, 3; T]
    param conv1_bias: Tensor[D; T]
    param conv2_weight: Tensor[D, D, 3; T]
    param conv2_bias: Tensor[D; T]
    // Sinusoidal in origin, but the checkpoint stores it as a tensor like any
    // other, and its length follows from the second convolution's stride.
    param encoder_positions: Tensor[(Frames - 1) / 2 + 1, D; T]
    sub encoder_layers: [EncoderLayer<D, Heads, Inner, T>; EncoderLayers]
    sub encoder_norm: LayerNorm<D, T>

    param token_embedding: Tensor[Vocab, D; T]
    param decoder_positions: Tensor[MaxTokens, D; T]
    sub decoder_layers: [DecoderLayer<D, Heads, Inner, Batch, MaxTokens, (Frames - 1) / 2 +
        1, T>; DecoderLayers]
    sub decoder_norm: LayerNorm<D, T>

    // The spectrogram to the encoder's states: `Frames` mel frames become
    // `(Frames - 1) / 2 + 1` positions, the form the shape solver derives
    // from a stride-2 kernel of three with one frame of padding on each side.
    // The mel bins arrive as the channel axis, the layout the feature
    // extractor and the checkpoint's kernels use, and the permutation to
    // positions-first happens once here.
    pub entry encode<B: Dim>(
        mel: Tensor[B, Mels, Frames; T],
    ) -> Tensor[B, (Frames - 1) / 2 + 1, D; T] {
        return encoded(mel)
    }

    fn encoded<B: Dim>(mel: Tensor[B, Mels, Frames; T]) -> Tensor[B, (Frames - 1) / 2 + 1, D; T] {
        let signal = permute(mel, [0, 2, 1])
        let first = gelu_erf(
            conv1d<B, Frames, Mels, D, 3, 1, 1, T>(signal, conv1_weight, conv1_bias),
        )
        let second = gelu_erf(conv1d<B, Frames, D, D, 3, 2, 1, T>(first, conv2_weight, conv2_bias))
        var x = second + encoder_positions
        static for layer in encoder_layers {
            x = layer.forward(x)
        }
        return encoder_norm.forward(x)
    }

    // Tokens and encoder states to logits for every token position. The audio
    // length is its own generic rather than the encoder's output length, so a
    // shorter set of states is accepted as it is.
    pub entry decode<B: Dim, S: Dim, A: Dim>(
        tokens: Tensor[B, S; i32],
        audio: Tensor[B, A, D; T],
    ) -> Tensor[B, S, Vocab; T]
    where S <= MaxTokens {
        // Positions 0..S: the decoder is fed from the start of the transcript.
        var x = embedding(tokens, token_embedding) + embedding(iota<i32>(S), decoder_positions)
        static for layer in decoder_layers {
            x = layer.forward(x, audio)
        }
        // The head is the token embedding read as a projection; Whisper ties
        // the two, so the checkpoint has no separate output matrix.
        return linear(decoder_norm.forward(x), token_embedding)
    }

    // Transcribing with caches: `listen` encodes the spectrogram and gives
    // every decoder layer the keys and values of its states, `prefill`
    // feeds the prompt (start of transcript, language, task) and returns
    // the logits after it, and `step` feeds one more token at `pos`,
    // reading the caches instead of recomputing every token before it.
    pub entry listen(mel: Tensor[Batch, Mels, Frames; T]) -> Tensor[Batch, (Frames - 1) / 2 +
        1, D; T] {
        let audio = encoded(mel)
        static for layer in decoder_layers {
            let _ = layer.listen(audio)
        }
        return audio
    }

    pub entry prefill<S: Dim>(tokens: Tensor[Batch, S; i32]) -> Tensor[Batch, Vocab; T]
    where
        S > 0,
        S <= MaxTokens
    {
        var x = embedding(tokens, token_embedding) + embedding(iota<i32>(S), decoder_positions)
        static for layer in decoder_layers {
            x = layer.prefill(x)
        }
        return linear(decoder_norm.forward(x[:, S - 1, :]), token_embedding)
    }

    pub entry step(token: Tensor[Batch, 1; i32], pos: i32) -> Tensor[Batch, Vocab; T] {
        let position[d] = decoder_positions[cast<i64>(pos), d]
        var x = embedding(token, token_embedding) + reshape(position, [1, 1, D])
        static for layer in decoder_layers {
            x = layer.step(x, pos)
        }
        return linear(decoder_norm.forward(x[:, 0, :]), token_embedding)
    }
}

Open on GitHub

model.safetensorsHugging Face ↗