Add Spark 2.0-code model
Browse files- .gitattributes +2 -0
- README.md +96 -0
- avatar.png +3 -0
- banner.png +3 -0
- config.json +19 -0
- generate.py +47 -0
- model.safetensors +3 -0
- spark/__init__.py +0 -0
- spark/decode.py +325 -0
- spark/model.py +386 -0
- tokenizer.json +0 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
avatar.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
banner.png filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,96 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-nc-4.0
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
pipeline_tag: text-generation
|
| 6 |
+
tags:
|
| 7 |
+
- code
|
| 8 |
+
- python
|
| 9 |
+
- from-scratch
|
| 10 |
+
- gated-deltanet
|
| 11 |
+
- hybrid
|
| 12 |
+
- base-model
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
<p align="center"><img src="banner.png" alt="Spark 2.0-code-base" width="100%"></p>
|
| 16 |
+
|
| 17 |
+
# Spark 2.0-code-base (pretrained base model)
|
| 18 |
+
|
| 19 |
+
The **base model** behind Spark 2.0-code: **322M parameters**, trained completely from scratch on Python code by one
|
| 20 |
+
person on a single RTX 3090. It was pretrained on about **18.4B tokens** and has **no instruction tuning**.
|
| 21 |
+
|
| 22 |
+
**This is not a chat model.** It does not follow tasks. You give it the **beginning of Python code** (a function
|
| 23 |
+
signature with a docstring, an import block, a class header) and it **continues** it. For instructions in plain
|
| 24 |
+
English use **Spark 2.0-code** or **Spark 2.0-code (stage 1)**.
|
| 25 |
+
|
| 26 |
+
Why release a base model: to fine-tune it yourself, to study it, or to compare it with the instruction-tuned versions.
|
| 27 |
+
|
| 28 |
+
> **Honest summary.** It completes functions reasonably (HumanEval 34.8%), but it does not know when to stop: after
|
| 29 |
+
> finishing one function it usually goes on and invents more functions. Cut its output yourself (for example at the
|
| 30 |
+
> next top-level `def`/`class`), or use a stop condition.
|
| 31 |
+
|
| 32 |
+
## Quick start
|
| 33 |
+
|
| 34 |
+
Needs an NVIDIA GPU (CUDA). The model uses Gated DeltaNet layers from
|
| 35 |
+
[flash-linear-attention](https://github.com/fla-org/flash-linear-attention), which run on GPU only.
|
| 36 |
+
|
| 37 |
+
```bash
|
| 38 |
+
pip install torch flash-linear-attention triton tokenizers safetensors
|
| 39 |
+
python generate.py 'def is_prime(n):\n # Return True if n is prime.\n'
|
| 40 |
+
```
|
| 41 |
+
|
| 42 |
+
`generate.py` prints your prompt and the continuation (greedy, up to 300 tokens). There is no prompt template: the text
|
| 43 |
+
you give is simply continued.
|
| 44 |
+
|
| 45 |
+
## Model
|
| 46 |
+
|
| 47 |
+
| | |
|
| 48 |
+
|---|---|
|
| 49 |
+
| Parameters | 322M (embeddings tied) |
|
| 50 |
+
| Architecture | hybrid: Gated DeltaNet layers + full attention every 4th layer, 24 layers, d_model 1024 |
|
| 51 |
+
| Context | 4096 tokens |
|
| 52 |
+
| Vocabulary | 16 384 (own BPE tokenizer, Python-focused) |
|
| 53 |
+
| Pretraining | about 18.4B tokens of Python (35 070 steps), Muon + AdamW |
|
| 54 |
+
| Validation loss | 0.50 |
|
| 55 |
+
| Weights | bf16, `model.safetensors`, 0.65 GB |
|
| 56 |
+
|
| 57 |
+
## Results
|
| 58 |
+
|
| 59 |
+
My own harness, greedy decoding, code is run in a sandbox. HumanEval is completion-style, which is what a base model does.
|
| 60 |
+
|
| 61 |
+
| | **Base (this model)** | SFT stage 1 | SFT stage 2 |
|
| 62 |
+
|---|---|---|---|
|
| 63 |
+
| HumanEval pass@1 (164 tasks) | **34.8%** | 52.4% | 47.6% |
|
| 64 |
+
|
| 65 |
+
HumanEval and MBPP were kept out of the training data: documents sharing a 13-gram with any task or reference solution
|
| 66 |
+
were removed.
|
| 67 |
+
|
| 68 |
+
## Limitations
|
| 69 |
+
|
| 70 |
+
- Not a chat or instruction model; it continues text.
|
| 71 |
+
- Does not stop after a function; trim the output.
|
| 72 |
+
- It sometimes invents library functions and misses imports.
|
| 73 |
+
- It only knows Python.
|
| 74 |
+
- A language model: it can write insecure or wrong code. Review before running.
|
| 75 |
+
|
| 76 |
+
## Training data and license
|
| 77 |
+
|
| 78 |
+
**License: CC BY-NC 4.0** (non-commercial, with attribution). The pretraining data includes
|
| 79 |
+
**KodCode (CC BY-NC 4.0)**, so I do not offer the model for commercial use. For a commercial product, contact the
|
| 80 |
+
author first.
|
| 81 |
+
|
| 82 |
+
Licenses are as shown on the Hugging Face dataset pages when I checked.
|
| 83 |
+
|
| 84 |
+
| Source | License |
|
| 85 |
+
|---|---|
|
| 86 |
+
| swallow-code-v2 (Python from The Stack v2, rewritten and filtered) | Apache-2.0 |
|
| 87 |
+
| OpenCodeInstruct (only solutions that passed all unit tests) | CC BY 4.0 |
|
| 88 |
+
| KodCode-V1 | CC BY-NC 4.0 |
|
| 89 |
+
| opc-annealing-corpus (synthetic Python snippets) | ODC-By |
|
| 90 |
+
|
| 91 |
+
Some of these datasets were themselves generated by other language models, so their own terms may apply.
|
| 92 |
+
This card is my best effort, not legal advice.
|
| 93 |
+
|
| 94 |
+
## Author
|
| 95 |
+
|
| 96 |
+
Trained from scratch as a personal project. Feedback and bug reports are welcome.
|
avatar.png
ADDED
|
|
Git LFS Details
|
banner.png
ADDED
|
Git LFS Details
|
config.json
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"vocab_size": 16384,
|
| 3 |
+
"d_model": 1024,
|
| 4 |
+
"n_layers": 24,
|
| 5 |
+
"max_seq_len": 4096,
|
| 6 |
+
"attn_every": 4,
|
| 7 |
+
"n_heads": 8,
|
| 8 |
+
"n_kv_heads": 2,
|
| 9 |
+
"head_dim": 128,
|
| 10 |
+
"rope_theta": 50000.0,
|
| 11 |
+
"dn_k_heads": 4,
|
| 12 |
+
"dn_v_heads": 8,
|
| 13 |
+
"dn_head_dim": 128,
|
| 14 |
+
"dn_conv_size": 4,
|
| 15 |
+
"ffn_hidden": 2816,
|
| 16 |
+
"norm_eps": 1e-06,
|
| 17 |
+
"tie_embeddings": true,
|
| 18 |
+
"init_std": 0.02
|
| 19 |
+
}
|
generate.py
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Spark 2.0-code-base: продолжает код (это не чат). python generate.py 'def is_prime(n):\n # Return True if n is prime.\n'
|
| 3 |
+
|
| 4 |
+
Нужна видеокарта NVIDIA (CUDA) и пакеты: torch, flash-linear-attention (fla), triton, tokenizers, safetensors.
|
| 5 |
+
Это базовая модель: ей дают НАЧАЛО кода (заголовок функции, докстринг), она его продолжает.
|
| 6 |
+
Генерация жадная (temperature 0): один и тот же запрос всегда даёт один и тот же ответ.
|
| 7 |
+
"""
|
| 8 |
+
import json, sys
|
| 9 |
+
from pathlib import Path
|
| 10 |
+
|
| 11 |
+
import torch
|
| 12 |
+
from safetensors.torch import load_file
|
| 13 |
+
from tokenizers import Tokenizer
|
| 14 |
+
|
| 15 |
+
from spark.model import HybridLM, ModelConfig
|
| 16 |
+
from spark.decode import Cache, GraphedStep, generate
|
| 17 |
+
|
| 18 |
+
HERE = Path(__file__).resolve().parent
|
| 19 |
+
TEMPLATE = "### User\n{content}\n\n### Assistant\n"
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def load(device="cuda"):
|
| 23 |
+
cfg = ModelConfig(**json.loads((HERE / "config.json").read_text()))
|
| 24 |
+
model = HybridLM(cfg)
|
| 25 |
+
sd = load_file(str(HERE / "model.safetensors"))
|
| 26 |
+
sd["lm_head.weight"] = sd["embed.weight"] # веса общие (tie_embeddings)
|
| 27 |
+
model.load_state_dict(sd)
|
| 28 |
+
model = model.to(device, torch.bfloat16).eval()
|
| 29 |
+
tok = Tokenizer.from_file(str(HERE / "tokenizer.json"))
|
| 30 |
+
return model, tok
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def ask(model, tok, task, max_new=300):
|
| 34 |
+
eos = tok.token_to_id("<|endoftext|>")
|
| 35 |
+
cache = Cache(model, max_len=4096)
|
| 36 |
+
graphed = GraphedStep(model, cache)
|
| 37 |
+
ids = torch.tensor([tok.encode(task).ids], device="cuda")
|
| 38 |
+
out = []
|
| 39 |
+
with torch.no_grad():
|
| 40 |
+
generate(model, cache, graphed, ids, max_new, 0.0, 1.0, eos, 1.0, on_token=out.append)
|
| 41 |
+
return tok.decode(out)
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
if __name__ == "__main__":
|
| 45 |
+
task = " ".join(sys.argv[1:]) or "def is_prime(n):\n \"\"\"Return True if n is prime.\"\"\"\n"
|
| 46 |
+
model, tok = load()
|
| 47 |
+
print(task + ask(model, tok, task))
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:83652db9e99b36686c90050d6ec39502f0602ff590d1b388ef3625453ca70364
|
| 3 |
+
size 644855272
|
spark/__init__.py
ADDED
|
File without changes
|
spark/decode.py
ADDED
|
@@ -0,0 +1,325 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Пошаговое декодирование с кэшем состояний + CUDA-граф.
|
| 3 |
+
|
| 4 |
+
Зачем это нужно. Замер (scripts/13_bench_gen.py) показал странное: один
|
| 5 |
+
forward стоит 29 мс и при контексте в 1 токен, и при 1024. Профайлер
|
| 6 |
+
объяснил почему -- за forward запускается 1466 отдельных CUDA-ядер, а
|
| 7 |
+
считает GPU всего 2.47 мс. То есть 92% времени карта простаивает, ожидая,
|
| 8 |
+
пока CPU поставит ей очередную мелкую задачу.
|
| 9 |
+
|
| 10 |
+
Лечится это CUDA-графом: вся последовательность запусков записывается один
|
| 11 |
+
раз и потом воспроизводится одной командой. Но граф требует, чтобы формы
|
| 12 |
+
тензоров и адреса памяти не менялись от шага к шагу -- а при генерации без
|
| 13 |
+
кэша контекст растёт на каждом токене. Поэтому сначала кэш (он делает вход
|
| 14 |
+
всегда равным одному токену), и только потом граф.
|
| 15 |
+
|
| 16 |
+
Кэш держит:
|
| 17 |
+
* для слоёв внимания -- K и V в буфере фиксированного размера MAX_LEN,
|
| 18 |
+
заполняемом по индексу; маска отсекает ещё не записанные позиции;
|
| 19 |
+
* для слоёв DeltaNet -- состояние дельта-правила [B, H, D, D] и хвост
|
| 20 |
+
входа для причинной свёртки (conv_size-1 токенов).
|
| 21 |
+
|
| 22 |
+
model.py не тронут: здесь только чтение его весов.
|
| 23 |
+
"""
|
| 24 |
+
import torch
|
| 25 |
+
import torch.nn.functional as F
|
| 26 |
+
|
| 27 |
+
from fla.ops.gated_delta_rule import (chunk_gated_delta_rule,
|
| 28 |
+
fused_recurrent_gated_delta_rule)
|
| 29 |
+
|
| 30 |
+
from .model import apply_rope, build_rope_cache
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
class Cache:
|
| 34 |
+
"""Все состояния модели для одной генерации. Формы фиксированы навсегда."""
|
| 35 |
+
|
| 36 |
+
def __init__(self, model, batch=1, max_len=1024, device="cuda"):
|
| 37 |
+
cfg = model.cfg
|
| 38 |
+
self.cfg, self.max_len, self.B = cfg, max_len, batch
|
| 39 |
+
dt = next(model.parameters()).dtype
|
| 40 |
+
|
| 41 |
+
self.attn, self.dn = {}, {}
|
| 42 |
+
for i, blk in enumerate(model.blocks):
|
| 43 |
+
if blk.is_attn:
|
| 44 |
+
self.attn[i] = {
|
| 45 |
+
"k": torch.zeros(batch, cfg.n_kv_heads, max_len, cfg.head_dim,
|
| 46 |
+
device=device, dtype=dt),
|
| 47 |
+
"v": torch.zeros(batch, cfg.n_kv_heads, max_len, cfg.head_dim,
|
| 48 |
+
device=device, dtype=dt),
|
| 49 |
+
}
|
| 50 |
+
else:
|
| 51 |
+
m = blk.mixer
|
| 52 |
+
self.dn[i] = {
|
| 53 |
+
"state": torch.zeros(batch, m.hv, m.hd, m.hd,
|
| 54 |
+
device=device, dtype=torch.float32),
|
| 55 |
+
"conv": torch.zeros(batch, m.conv_dim, cfg.dn_conv_size - 1,
|
| 56 |
+
device=device, dtype=dt),
|
| 57 |
+
}
|
| 58 |
+
|
| 59 |
+
# позиция живёт в видеопамяти: граф читает её как данные, а не как
|
| 60 |
+
# число, вшитое в момент записи
|
| 61 |
+
self.pos = torch.zeros((), device=device, dtype=torch.long)
|
| 62 |
+
self.ar = torch.arange(max_len, device=device)
|
| 63 |
+
self.cos, self.sin = build_rope_cache(max_len, cfg.head_dim,
|
| 64 |
+
cfg.rope_theta, device, torch.float32)
|
| 65 |
+
|
| 66 |
+
def reset(self):
|
| 67 |
+
for c in self.attn.values():
|
| 68 |
+
c["k"].zero_(); c["v"].zero_()
|
| 69 |
+
for c in self.dn.values():
|
| 70 |
+
c["state"].zero_(); c["conv"].zero_()
|
| 71 |
+
self.pos.zero_()
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
# --------------------------------------------------------------------------- #
|
| 75 |
+
# Один шаг: T = 1
|
| 76 |
+
# --------------------------------------------------------------------------- #
|
| 77 |
+
|
| 78 |
+
def _attn_step(mix, h, cache, i):
|
| 79 |
+
B = h.shape[0]
|
| 80 |
+
c = cache.attn[i]
|
| 81 |
+
q = mix.q_norm(mix.q_proj(h).view(B, 1, mix.nh, mix.hd))
|
| 82 |
+
k = mix.k_norm(mix.k_proj(h).view(B, 1, mix.nkv, mix.hd))
|
| 83 |
+
v = mix.v_proj(h).view(B, 1, mix.nkv, mix.hd)
|
| 84 |
+
|
| 85 |
+
cos = cache.cos.index_select(0, cache.pos.view(1))
|
| 86 |
+
sin = cache.sin.index_select(0, cache.pos.view(1))
|
| 87 |
+
q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
|
| 88 |
+
|
| 89 |
+
# index_copy_ по тензору-индексу: адрес буфера постоянен, меняется
|
| 90 |
+
# только содержимое -- ровно то, что нужно графу
|
| 91 |
+
c["k"].index_copy_(2, cache.pos.view(1), k.transpose(1, 2))
|
| 92 |
+
c["v"].index_copy_(2, cache.pos.view(1), v.transpose(1, 2))
|
| 93 |
+
|
| 94 |
+
# смотрим на весь буфер, маской отсекая ещё не заполненный хвост
|
| 95 |
+
mask = (cache.ar <= cache.pos).view(1, 1, 1, -1)
|
| 96 |
+
o = F.scaled_dot_product_attention(q.transpose(1, 2), c["k"], c["v"],
|
| 97 |
+
attn_mask=mask, enable_gqa=True)
|
| 98 |
+
o = o.transpose(1, 2).reshape(B, 1, mix.nh * mix.hd)
|
| 99 |
+
return mix.o_proj(o * torch.sigmoid(mix.g_proj(h)))
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
def _dn_step(mix, h, cache, i):
|
| 103 |
+
B = h.shape[0]
|
| 104 |
+
c = cache.dn[i]
|
| 105 |
+
qkv = torch.cat([mix.q_proj(h), mix.k_proj(h), mix.v_proj(h)], dim=-1)
|
| 106 |
+
|
| 107 |
+
# причинная свёртка вручную: хвост прошлых входов + текущий вход
|
| 108 |
+
win = torch.cat([c["conv"], qkv.transpose(1, 2)], dim=-1)
|
| 109 |
+
conv = F.conv1d(win, mix.conv.weight, groups=mix.conv_dim)
|
| 110 |
+
c["conv"].copy_(win[..., 1:])
|
| 111 |
+
qkv = F.silu(conv).transpose(1, 2)
|
| 112 |
+
|
| 113 |
+
q, k, v = qkv.split([mix.k_dim, mix.k_dim, mix.v_dim], dim=-1)
|
| 114 |
+
q = q.view(B, 1, mix.hk, mix.hd).repeat_interleave(mix.rep, dim=2)
|
| 115 |
+
k = k.view(B, 1, mix.hk, mix.hd).repeat_interleave(mix.rep, dim=2)
|
| 116 |
+
v = v.view(B, 1, mix.hv, mix.hd)
|
| 117 |
+
|
| 118 |
+
beta = torch.sigmoid(mix.b_proj(h).float())
|
| 119 |
+
g = -mix.A_log.float().exp() * F.softplus(mix.a_proj(h).float()
|
| 120 |
+
+ mix.dt_bias.float())
|
| 121 |
+
|
| 122 |
+
o, new_state = fused_recurrent_gated_delta_rule(
|
| 123 |
+
q=q, k=k, v=v, g=g, beta=beta.to(v.dtype),
|
| 124 |
+
initial_state=c["state"], output_final_state=True,
|
| 125 |
+
use_qk_l2norm_in_kernel=True)
|
| 126 |
+
c["state"].copy_(new_state)
|
| 127 |
+
|
| 128 |
+
o = mix.o_norm(o).reshape(B, 1, mix.v_dim)
|
| 129 |
+
return mix.o_proj(o * torch.sigmoid(mix.g_proj(h)))
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
def step(model, cache, tok):
|
| 133 |
+
"""tok: [B, 1] -> логиты [B, vocab]. Позицию двигает вызывающий."""
|
| 134 |
+
x = model.embed(tok)
|
| 135 |
+
for i, blk in enumerate(model.blocks):
|
| 136 |
+
h = blk.norm1(x)
|
| 137 |
+
mix = _attn_step(blk.mixer, h, cache, i) if blk.is_attn \
|
| 138 |
+
else _dn_step(blk.mixer, h, cache, i)
|
| 139 |
+
x = x + mix
|
| 140 |
+
x = x + blk.ffn(blk.norm2(x))
|
| 141 |
+
return model.lm_head(model.norm_f(x))[:, -1]
|
| 142 |
+
|
| 143 |
+
|
| 144 |
+
# --------------------------------------------------------------------------- #
|
| 145 |
+
# Префилл: весь промпт разом, чтобы не гнать его по одному токену
|
| 146 |
+
# --------------------------------------------------------------------------- #
|
| 147 |
+
|
| 148 |
+
def _attn_prefill(mix, h, cache, i, start):
|
| 149 |
+
B, T, _ = h.shape
|
| 150 |
+
c = cache.attn[i]
|
| 151 |
+
q = mix.q_norm(mix.q_proj(h).view(B, T, mix.nh, mix.hd))
|
| 152 |
+
k = mix.k_norm(mix.k_proj(h).view(B, T, mix.nkv, mix.hd))
|
| 153 |
+
v = mix.v_proj(h).view(B, T, mix.nkv, mix.hd)
|
| 154 |
+
|
| 155 |
+
cos, sin = cache.cos[start:start + T], cache.sin[start:start + T]
|
| 156 |
+
q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
|
| 157 |
+
|
| 158 |
+
c["k"][:, :, start:start + T] = k.transpose(1, 2)
|
| 159 |
+
c["v"][:, :, start:start + T] = v.transpose(1, 2)
|
| 160 |
+
|
| 161 |
+
o = F.scaled_dot_product_attention(q.transpose(1, 2),
|
| 162 |
+
c["k"][:, :, :start + T],
|
| 163 |
+
c["v"][:, :, :start + T],
|
| 164 |
+
is_causal=True, enable_gqa=True)
|
| 165 |
+
o = o.transpose(1, 2).reshape(B, T, mix.nh * mix.hd)
|
| 166 |
+
return mix.o_proj(o * torch.sigmoid(mix.g_proj(h)))
|
| 167 |
+
|
| 168 |
+
|
| 169 |
+
def _dn_prefill(mix, h, cache, i):
|
| 170 |
+
B, T, _ = h.shape
|
| 171 |
+
c = cache.dn[i]
|
| 172 |
+
qkv = torch.cat([mix.q_proj(h), mix.k_proj(h), mix.v_proj(h)], dim=-1)
|
| 173 |
+
xt = qkv.transpose(1, 2)
|
| 174 |
+
conv = mix.conv(xt)[..., :T]
|
| 175 |
+
# хвост входа переносим в кэш: следующему шагу нужен именно он
|
| 176 |
+
tail = xt[..., -(mix.cfg.dn_conv_size - 1):]
|
| 177 |
+
c["conv"][..., -tail.shape[-1]:].copy_(tail)
|
| 178 |
+
qkv = F.silu(conv).transpose(1, 2)
|
| 179 |
+
|
| 180 |
+
q, k, v = qkv.split([mix.k_dim, mix.k_dim, mix.v_dim], dim=-1)
|
| 181 |
+
q = q.view(B, T, mix.hk, mix.hd).repeat_interleave(mix.rep, dim=2)
|
| 182 |
+
k = k.view(B, T, mix.hk, mix.hd).repeat_interleave(mix.rep, dim=2)
|
| 183 |
+
v = v.view(B, T, mix.hv, mix.hd)
|
| 184 |
+
|
| 185 |
+
beta = torch.sigmoid(mix.b_proj(h).float())
|
| 186 |
+
g = -mix.A_log.float().exp() * F.softplus(mix.a_proj(h).float()
|
| 187 |
+
+ mix.dt_bias.float())
|
| 188 |
+
|
| 189 |
+
o, new_state = chunk_gated_delta_rule(
|
| 190 |
+
q=q, k=k, v=v, g=g, beta=beta.to(v.dtype),
|
| 191 |
+
initial_state=c["state"], output_final_state=True,
|
| 192 |
+
use_qk_l2norm_in_kernel=True)
|
| 193 |
+
c["state"].copy_(new_state)
|
| 194 |
+
|
| 195 |
+
o = mix.o_norm(o).reshape(B, T, mix.v_dim)
|
| 196 |
+
return mix.o_proj(o * torch.sigmoid(mix.g_proj(h)))
|
| 197 |
+
|
| 198 |
+
|
| 199 |
+
@torch.no_grad()
|
| 200 |
+
def prefill(model, cache, ids):
|
| 201 |
+
"""ids: [B, T] -> логиты последней позиции [B, vocab]."""
|
| 202 |
+
start = int(cache.pos)
|
| 203 |
+
x = model.embed(ids)
|
| 204 |
+
for i, blk in enumerate(model.blocks):
|
| 205 |
+
h = blk.norm1(x)
|
| 206 |
+
mix = _attn_prefill(blk.mixer, h, cache, i, start) if blk.is_attn \
|
| 207 |
+
else _dn_prefill(blk.mixer, h, cache, i)
|
| 208 |
+
x = x + mix
|
| 209 |
+
x = x + blk.ffn(blk.norm2(x))
|
| 210 |
+
cache.pos.fill_(start + ids.shape[1] - 1) # позиция ПОСЛЕДНЕГО токена
|
| 211 |
+
return model.lm_head(model.norm_f(x))[:, -1]
|
| 212 |
+
|
| 213 |
+
|
| 214 |
+
# --------------------------------------------------------------------------- #
|
| 215 |
+
# CUDA-граф вокруг шага
|
| 216 |
+
# --------------------------------------------------------------------------- #
|
| 217 |
+
|
| 218 |
+
class GraphedStep:
|
| 219 |
+
"""
|
| 220 |
+
Записывает один шаг декодирования в CUDA-граф и потом воспроизводит его.
|
| 221 |
+
|
| 222 |
+
Граф работает только с теми буферами, что были при записи, поэтому вход
|
| 223 |
+
и выход -- постоянные тензоры: пишем в них, запускаем, читаем из них.
|
| 224 |
+
"""
|
| 225 |
+
|
| 226 |
+
def __init__(self, model, cache, warmup=3):
|
| 227 |
+
self.model, self.cache = model, cache
|
| 228 |
+
self.tok = torch.zeros(cache.B, 1, device="cuda", dtype=torch.long)
|
| 229 |
+
|
| 230 |
+
# прогрев на отдельном потоке: без него в граф попадут ленивые
|
| 231 |
+
# инициализации Triton-ядер, и запись сломается
|
| 232 |
+
s = torch.cuda.Stream()
|
| 233 |
+
s.wait_stream(torch.cuda.current_stream())
|
| 234 |
+
with torch.cuda.stream(s), torch.no_grad():
|
| 235 |
+
for _ in range(warmup):
|
| 236 |
+
step(model, cache, self.tok)
|
| 237 |
+
torch.cuda.current_stream().wait_stream(s)
|
| 238 |
+
|
| 239 |
+
self.graph = torch.cuda.CUDAGraph()
|
| 240 |
+
with torch.no_grad(), torch.cuda.graph(self.graph):
|
| 241 |
+
self.logits = step(model, cache, self.tok)
|
| 242 |
+
|
| 243 |
+
def __call__(self, tok):
|
| 244 |
+
self.tok.copy_(tok)
|
| 245 |
+
self.graph.replay()
|
| 246 |
+
return self.logits
|
| 247 |
+
|
| 248 |
+
|
| 249 |
+
# --------------------------------------------------------------------------- #
|
| 250 |
+
# Потоковая генерация поверх графа
|
| 251 |
+
# --------------------------------------------------------------------------- #
|
| 252 |
+
|
| 253 |
+
@torch.no_grad()
|
| 254 |
+
def generate(model, cache, graphed, prompt_ids, max_new, temp=0.4, top_p=0.95,
|
| 255 |
+
eos_id=0, rep_penalty=1.0, rep_window=256, structural=None,
|
| 256 |
+
on_token=None):
|
| 257 |
+
"""
|
| 258 |
+
Генерирует токены, вызывая on_token(id) по мере готовности.
|
| 259 |
+
|
| 260 |
+
Семплирование и штраф за повтор считаются ВНЕ графа: внутрь их не
|
| 261 |
+
затащить -- там нельзя ветвиться по значению и менять формы. Поэтому
|
| 262 |
+
они написаны так, чтобы стоить доли миллисекунды: при 4 мс на шаг
|
| 263 |
+
даже одна лишняя аллокация на токен заметна.
|
| 264 |
+
|
| 265 |
+
Штраф за повтор держит счётчики в векторе на [vocab] и правит их
|
| 266 |
+
инкрементально. Наивный вариант (собрать историю в тензор и позвать
|
| 267 |
+
torch.unique на каждом шаге) стоил около трети времени шага.
|
| 268 |
+
|
| 269 |
+
Возвращает (список токенов, причина остановки).
|
| 270 |
+
"""
|
| 271 |
+
cache.reset()
|
| 272 |
+
ids = prompt_ids[:, -(cache.max_len - max_new - 1):] # запас под ответ
|
| 273 |
+
logits = prefill(model, cache, ids)
|
| 274 |
+
dev = ids.device
|
| 275 |
+
|
| 276 |
+
hist = ids[0].tolist()
|
| 277 |
+
counts = torch.zeros(model.cfg.vocab_size, device=dev)
|
| 278 |
+
if rep_penalty > 1.0:
|
| 279 |
+
for t in hist[-rep_window:]:
|
| 280 |
+
counts[t] += 1
|
| 281 |
+
if structural is not None:
|
| 282 |
+
counts[structural] = 0 # отступы и переносы не штрафуем
|
| 283 |
+
|
| 284 |
+
out, why = [], "упёрся в лимит"
|
| 285 |
+
for _ in range(max_new):
|
| 286 |
+
lg = logits[0].float()
|
| 287 |
+
|
| 288 |
+
if rep_penalty > 1.0:
|
| 289 |
+
factor = rep_penalty ** counts # для counts=0 это ровно 1
|
| 290 |
+
lg = torch.where(lg > 0, lg / factor, lg * factor)
|
| 291 |
+
|
| 292 |
+
if temp <= 0:
|
| 293 |
+
nxt = int(lg.argmax())
|
| 294 |
+
else:
|
| 295 |
+
p = F.softmax(lg / temp, dim=-1)
|
| 296 |
+
if top_p < 1.0:
|
| 297 |
+
sp, si = torch.sort(p, descending=True)
|
| 298 |
+
sp[sp.cumsum(0) - sp > top_p] = 0
|
| 299 |
+
sp /= sp.sum()
|
| 300 |
+
nxt = int(si[torch.multinomial(sp, 1)])
|
| 301 |
+
else:
|
| 302 |
+
nxt = int(torch.multinomial(p, 1))
|
| 303 |
+
|
| 304 |
+
if nxt == eos_id:
|
| 305 |
+
why = "закончил сам (EOS)"
|
| 306 |
+
break
|
| 307 |
+
|
| 308 |
+
out.append(nxt); hist.append(nxt)
|
| 309 |
+
if rep_penalty > 1.0:
|
| 310 |
+
if structural is None or not bool(structural[nxt]):
|
| 311 |
+
counts[nxt] += 1
|
| 312 |
+
if len(hist) > rep_window: # токен вышел из окна
|
| 313 |
+
old = hist[-rep_window - 1]
|
| 314 |
+
if structural is None or not bool(structural[old]):
|
| 315 |
+
counts[old] -= 1
|
| 316 |
+
|
| 317 |
+
if on_token is not None:
|
| 318 |
+
on_token(nxt)
|
| 319 |
+
if int(cache.pos) + 1 >= cache.max_len - 1:
|
| 320 |
+
why = "кончился буфер контекста"
|
| 321 |
+
break
|
| 322 |
+
|
| 323 |
+
cache.pos += 1
|
| 324 |
+
logits = graphed(torch.tensor([[nxt]], device=dev))
|
| 325 |
+
return out, why
|
spark/model.py
ADDED
|
@@ -0,0 +1,386 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Гибридная языковая модель в стиле Qwen3.5 / Qwen3-Next.
|
| 3 |
+
|
| 4 |
+
Схема одного блока (как в обычном трансформере):
|
| 5 |
+
x -> [норма] -> [смеситель токенов] -> +x -> [норма] -> [FFN] -> +x
|
| 6 |
+
|
| 7 |
+
"Смеситель токенов" бывает двух видов, они чередуются 3:1
|
| 8 |
+
- Gated DeltaNet (линейное внимание, быстрое, 3 слоя из 4)
|
| 9 |
+
- Gated Attention (полное внимание, медленное но точное, каждый 4-й слой)
|
| 10 |
+
"""
|
| 11 |
+
|
| 12 |
+
import math
|
| 13 |
+
from dataclasses import dataclass, asdict
|
| 14 |
+
|
| 15 |
+
import torch
|
| 16 |
+
import torch.nn as nn
|
| 17 |
+
import torch.nn.functional as F
|
| 18 |
+
|
| 19 |
+
from fla.ops.gated_delta_rule import chunk_gated_delta_rule
|
| 20 |
+
from fla.ops.attn.parallel import parallel_attn
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
# --------------------------------------------------------------------------- #
|
| 24 |
+
# Конфигурация
|
| 25 |
+
# --------------------------------------------------------------------------- #
|
| 26 |
+
|
| 27 |
+
@dataclass
|
| 28 |
+
class ModelConfig:
|
| 29 |
+
vocab_size: int = 16384
|
| 30 |
+
d_model: int = 640
|
| 31 |
+
n_layers: int = 20
|
| 32 |
+
max_seq_len: int = 4096
|
| 33 |
+
|
| 34 |
+
# --- слои полного внимания ---
|
| 35 |
+
attn_every: int = 4 # каждый 4-й слой — полное внимание
|
| 36 |
+
n_heads: int = 8 # голов запроса
|
| 37 |
+
n_kv_heads: int = 2 # голов ключа/значения (GQA)
|
| 38 |
+
head_dim: int = 128 # развязан с d_model, как у Qwen3
|
| 39 |
+
rope_theta: float = 50000.0 # НЕ 1e6: у нас контекст 4096, не 40k
|
| 40 |
+
|
| 41 |
+
# --- слои Gated DeltaNet ---
|
| 42 |
+
dn_k_heads: int = 4
|
| 43 |
+
dn_v_heads: int = 8
|
| 44 |
+
dn_head_dim: int = 128
|
| 45 |
+
dn_conv_size: int = 4
|
| 46 |
+
|
| 47 |
+
# --- FFN ---
|
| 48 |
+
ffn_hidden: int = 1536
|
| 49 |
+
|
| 50 |
+
norm_eps: float = 1e-6
|
| 51 |
+
tie_embeddings: bool = True
|
| 52 |
+
init_std: float = 0.02
|
| 53 |
+
|
| 54 |
+
# пересчитывать активации в backward вместо хранения (экономит память,
|
| 55 |
+
# стоит ~30% лишних вычислений в прямом проходе)
|
| 56 |
+
grad_checkpoint: bool = False
|
| 57 |
+
|
| 58 |
+
def is_attn_layer(self, i: int) -> bool:
|
| 59 |
+
"""Каждый attn_every-й слой (последний в группе) — полное внимание."""
|
| 60 |
+
return (i + 1) % self.attn_every == 0
|
| 61 |
+
|
| 62 |
+
def to_dict(self):
|
| 63 |
+
return asdict(self)
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
# --------------------------------------------------------------------------- #
|
| 67 |
+
# Базовые кирпичики
|
| 68 |
+
# --------------------------------------------------------------------------- #
|
| 69 |
+
|
| 70 |
+
class RMSNorm(nn.Module):
|
| 71 |
+
"""Нормализация. Дешевле LayerNorm, стандарт во всех современных моделях."""
|
| 72 |
+
|
| 73 |
+
def __init__(self, dim: int, eps: float = 1e-6):
|
| 74 |
+
super().__init__()
|
| 75 |
+
self.eps = eps
|
| 76 |
+
self.weight = nn.Parameter(torch.ones(dim))
|
| 77 |
+
|
| 78 |
+
def forward(self, x):
|
| 79 |
+
dtype = x.dtype
|
| 80 |
+
x = x.float()
|
| 81 |
+
x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
|
| 82 |
+
return (x * self.weight.float()).to(dtype)
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
class SwiGLU(nn.Module):
|
| 86 |
+
"""FFN. Три матрицы вместо двух: одна работает воротами для другой."""
|
| 87 |
+
|
| 88 |
+
def __init__(self, d_model: int, hidden: int):
|
| 89 |
+
super().__init__()
|
| 90 |
+
self.gate = nn.Linear(d_model, hidden, bias=False)
|
| 91 |
+
self.up = nn.Linear(d_model, hidden, bias=False)
|
| 92 |
+
self.down = nn.Linear(hidden, d_model, bias=False)
|
| 93 |
+
|
| 94 |
+
def forward(self, x):
|
| 95 |
+
return self.down(F.silu(self.gate(x)) * self.up(x))
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
def build_rope_cache(seq_len: int, head_dim: int, theta: float, device, dtype):
|
| 99 |
+
"""Предпосчитанные синусы/косинусы для RoPE."""
|
| 100 |
+
inv_freq = 1.0 / (theta ** (torch.arange(0, head_dim, 2, device=device).float() / head_dim))
|
| 101 |
+
pos = torch.arange(seq_len, device=device).float()
|
| 102 |
+
freqs = torch.outer(pos, inv_freq) # [T, head_dim/2]
|
| 103 |
+
return freqs.cos().to(dtype), freqs.sin().to(dtype)
|
| 104 |
+
|
| 105 |
+
|
| 106 |
+
def apply_rope(x, cos, sin):
|
| 107 |
+
"""x: [B, T, H, D]. Поворачивает пары каналов на угол, зависящий от позиции."""
|
| 108 |
+
x1, x2 = x.float().chunk(2, dim=-1)
|
| 109 |
+
cos = cos[None, :, None, :]
|
| 110 |
+
sin = sin[None, :, None, :]
|
| 111 |
+
out = torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1)
|
| 112 |
+
return out.to(x.dtype)
|
| 113 |
+
|
| 114 |
+
|
| 115 |
+
# --------------------------------------------------------------------------- #
|
| 116 |
+
# Смеситель 1: полное внимание с воротами (каждый 4-й слой)
|
| 117 |
+
# --------------------------------------------------------------------------- #
|
| 118 |
+
|
| 119 |
+
class GatedAttention(nn.Module):
|
| 120 |
+
"""
|
| 121 |
+
Обычное причинное внимание + два отличия Qwen3:
|
| 122 |
+
* QK-Norm — нормализация Q и K, страхует обучение от разъезжания
|
| 123 |
+
* выходные ворота — сигмоида решает, сколько внимания пропустить дальше
|
| 124 |
+
"""
|
| 125 |
+
|
| 126 |
+
def __init__(self, cfg: ModelConfig):
|
| 127 |
+
super().__init__()
|
| 128 |
+
self.cfg = cfg
|
| 129 |
+
self.nh, self.nkv, self.hd = cfg.n_heads, cfg.n_kv_heads, cfg.head_dim
|
| 130 |
+
q_dim, kv_dim = self.nh * self.hd, self.nkv * self.hd
|
| 131 |
+
|
| 132 |
+
self.q_proj = nn.Linear(cfg.d_model, q_dim, bias=False)
|
| 133 |
+
self.k_proj = nn.Linear(cfg.d_model, kv_dim, bias=False)
|
| 134 |
+
self.v_proj = nn.Linear(cfg.d_model, kv_dim, bias=False)
|
| 135 |
+
self.o_proj = nn.Linear(q_dim, cfg.d_model, bias=False)
|
| 136 |
+
self.g_proj = nn.Linear(cfg.d_model, q_dim, bias=False) # ворота
|
| 137 |
+
|
| 138 |
+
self.q_norm = RMSNorm(self.hd, cfg.norm_eps) # QK-Norm
|
| 139 |
+
self.k_norm = RMSNorm(self.hd, cfg.norm_eps)
|
| 140 |
+
|
| 141 |
+
def forward(self, x, cos, sin, docs=None):
|
| 142 |
+
B, T, _ = x.shape
|
| 143 |
+
|
| 144 |
+
q = self.q_proj(x).view(B, T, self.nh, self.hd)
|
| 145 |
+
k = self.k_proj(x).view(B, T, self.nkv, self.hd)
|
| 146 |
+
v = self.v_proj(x).view(B, T, self.nkv, self.hd)
|
| 147 |
+
|
| 148 |
+
q, k = self.q_norm(q), self.k_norm(k)
|
| 149 |
+
q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
|
| 150 |
+
|
| 151 |
+
if docs is not None:
|
| 152 |
+
# маска документов: каждый токен видит только свой документ.
|
| 153 |
+
# Кернел fla сам пропускает блоки чужих документов
|
| 154 |
+
o = parallel_attn(q, k, v, cu_seqlens=docs[0])
|
| 155 |
+
o = o.reshape(B, T, self.nh * self.hd)
|
| 156 |
+
o = o * torch.sigmoid(self.g_proj(x))
|
| 157 |
+
return self.o_proj(o)
|
| 158 |
+
|
| 159 |
+
# SDPA ждёт [B, H, T, D]
|
| 160 |
+
q, k, v = (t.transpose(1, 2) for t in (q, k, v))
|
| 161 |
+
# kv-головы размножаем явно, а не через enable_gqa=True: в сборке
|
| 162 |
+
# PyTorch под Windows нет FlashAttention, и с enable_gqa SDPA падает
|
| 163 |
+
# на математический путь с полной матрицей TxT. С явным повтором
|
| 164 |
+
# включается memory-efficient кернел -- замерено 8.3 мс против 16.1
|
| 165 |
+
# на слой (fwd+bwd, T=4096), и сотни МБ памяти на слой не хранятся
|
| 166 |
+
if self.nkv != self.nh:
|
| 167 |
+
k = k.repeat_interleave(self.nh // self.nkv, dim=1)
|
| 168 |
+
v = v.repeat_interleave(self.nh // self.nkv, dim=1)
|
| 169 |
+
o = F.scaled_dot_product_attention(q, k, v, is_causal=True)
|
| 170 |
+
o = o.transpose(1, 2).reshape(B, T, self.nh * self.hd)
|
| 171 |
+
|
| 172 |
+
o = o * torch.sigmoid(self.g_proj(x)) # ворота
|
| 173 |
+
return self.o_proj(o)
|
| 174 |
+
|
| 175 |
+
|
| 176 |
+
# --------------------------------------------------------------------------- #
|
| 177 |
+
# Смеситель 2: Gated DeltaNet (линейное внимание, 3 слоя из 4)
|
| 178 |
+
# --------------------------------------------------------------------------- #
|
| 179 |
+
|
| 180 |
+
class GatedDeltaNet(nn.Module):
|
| 181 |
+
"""
|
| 182 |
+
Линейное внимание из Qwen3-Next. Вместо матрицы TxT держит в голове
|
| 183 |
+
компактное состояние фиксированного размера и обновляет его по дельта-правилу:
|
| 184 |
+
на каждом токене чуть-чуть забывает старое (затухание g) и дописывает новое
|
| 185 |
+
(сила записи beta). Отсюда O(T) вместо O(T^2).
|
| 186 |
+
|
| 187 |
+
Короткая причинная свёртка перед этим даёт слою локальный контекст,
|
| 188 |
+
который чистой рекуррентности даётся тяжело.
|
| 189 |
+
"""
|
| 190 |
+
|
| 191 |
+
def __init__(self, cfg: ModelConfig):
|
| 192 |
+
super().__init__()
|
| 193 |
+
self.cfg = cfg
|
| 194 |
+
self.hk, self.hv, self.hd = cfg.dn_k_heads, cfg.dn_v_heads, cfg.dn_head_dim
|
| 195 |
+
self.rep = self.hv // self.hk # во сколько раз v-голов больше
|
| 196 |
+
k_dim, v_dim = self.hk * self.hd, self.hv * self.hd
|
| 197 |
+
self.k_dim, self.v_dim = k_dim, v_dim
|
| 198 |
+
|
| 199 |
+
self.q_proj = nn.Linear(cfg.d_model, k_dim, bias=False)
|
| 200 |
+
self.k_proj = nn.Linear(cfg.d_model, k_dim, bias=False)
|
| 201 |
+
self.v_proj = nn.Linear(cfg.d_model, v_dim, bias=False)
|
| 202 |
+
self.b_proj = nn.Linear(cfg.d_model, self.hv, bias=False) # beta: сила записи
|
| 203 |
+
self.a_proj = nn.Linear(cfg.d_model, self.hv, bias=False) # g: скорость забывания
|
| 204 |
+
self.g_proj = nn.Linear(cfg.d_model, v_dim, bias=False) # выходные ворота
|
| 205 |
+
self.o_proj = nn.Linear(v_dim, cfg.d_model, bias=False)
|
| 206 |
+
|
| 207 |
+
# depthwise причинная свёртка по q|k|v
|
| 208 |
+
conv_dim = k_dim * 2 + v_dim
|
| 209 |
+
self.conv_dim = conv_dim
|
| 210 |
+
self.conv = nn.Conv1d(conv_dim, conv_dim, kernel_size=cfg.dn_conv_size,
|
| 211 |
+
groups=conv_dim, padding=cfg.dn_conv_size - 1, bias=False)
|
| 212 |
+
|
| 213 |
+
# обучаемые параметры затухания (инициализация как в Mamba2/Qwen3-Next)
|
| 214 |
+
self.A_log = nn.Parameter(torch.log(torch.empty(self.hv).uniform_(1, 16)))
|
| 215 |
+
dt = torch.exp(torch.rand(self.hv) * (math.log(0.1) - math.log(0.001)) + math.log(0.001))
|
| 216 |
+
dt = dt.clamp(min=1e-4)
|
| 217 |
+
self.dt_bias = nn.Parameter(dt + torch.log(-torch.expm1(-dt)))
|
| 218 |
+
|
| 219 |
+
self.o_norm = RMSNorm(self.hd, cfg.norm_eps)
|
| 220 |
+
|
| 221 |
+
def forward(self, x, cos=None, sin=None, docs=None):
|
| 222 |
+
B, T, _ = x.shape
|
| 223 |
+
|
| 224 |
+
qkv = torch.cat([self.q_proj(x), self.k_proj(x), self.v_proj(x)], dim=-1)
|
| 225 |
+
if docs is None:
|
| 226 |
+
qkv = self.conv(qkv.transpose(1, 2))[..., :T].transpose(1, 2) # причинная обрезка
|
| 227 |
+
else:
|
| 228 |
+
# та же свёртка, но сдвиги, залезающие в предыдущий документ,
|
| 229 |
+
# обнуляются. Иначе первые 3 токена документа видят хвост чужого,
|
| 230 |
+
# а внимание разносит эту утечку по всему документу
|
| 231 |
+
w = self.conv.weight[:, 0, :].to(qkv.dtype) # [C, K]
|
| 232 |
+
K = w.shape[1]
|
| 233 |
+
pos = docs[2] # позиция внутри документа
|
| 234 |
+
out = qkv * w[:, K - 1]
|
| 235 |
+
for j in range(1, K):
|
| 236 |
+
sh = F.pad(qkv, (0, 0, j, 0))[:, :T]
|
| 237 |
+
out = out + sh * (pos >= j)[None, :, None].to(qkv.dtype) * w[:, K - 1 - j]
|
| 238 |
+
qkv = out
|
| 239 |
+
qkv = F.silu(qkv)
|
| 240 |
+
q, k, v = qkv.split([self.k_dim, self.k_dim, self.v_dim], dim=-1)
|
| 241 |
+
|
| 242 |
+
q = q.view(B, T, self.hk, self.hd).repeat_interleave(self.rep, dim=2)
|
| 243 |
+
k = k.view(B, T, self.hk, self.hd).repeat_interleave(self.rep, dim=2)
|
| 244 |
+
v = v.view(B, T, self.hv, self.hd)
|
| 245 |
+
|
| 246 |
+
beta = torch.sigmoid(self.b_proj(x).float())
|
| 247 |
+
g = -self.A_log.float().exp() * F.softplus(self.a_proj(x).float() + self.dt_bias.float())
|
| 248 |
+
|
| 249 |
+
o, _ = chunk_gated_delta_rule(
|
| 250 |
+
q=q, k=k, v=v, g=g, beta=beta.to(v.dtype),
|
| 251 |
+
use_qk_l2norm_in_kernel=True, # L2-норма Q/K обязательна для устойчивости
|
| 252 |
+
# с маской документов состояние обнуляется на границе документа
|
| 253 |
+
cu_seqlens=None if docs is None else docs[0],
|
| 254 |
+
cu_seqlens_cpu=None if docs is None else docs[1],
|
| 255 |
+
)
|
| 256 |
+
|
| 257 |
+
o = self.o_norm(o).reshape(B, T, self.v_dim)
|
| 258 |
+
o = o * torch.sigmoid(self.g_proj(x))
|
| 259 |
+
return self.o_proj(o)
|
| 260 |
+
|
| 261 |
+
|
| 262 |
+
def make_doc_mask(x_cpu: torch.Tensor, eos_id: int = 0, device="cuda"):
|
| 263 |
+
"""
|
| 264 |
+
Границы документов для маски: x_cpu -- батч [B, T] на процессоре.
|
| 265 |
+
|
| 266 |
+
Батч склеивается в одну строку [1, B*T]; документ начинается после
|
| 267 |
+
каждого EOS и на стыке строк батча. Возвращает (cu_seqlens на карте,
|
| 268 |
+
то же на процессоре, позиция каждого токена внутри своего документа).
|
| 269 |
+
Копия на процессоре избавляет fla от синхронизации с картой.
|
| 270 |
+
"""
|
| 271 |
+
B, T = x_cpu.shape
|
| 272 |
+
N = B * T
|
| 273 |
+
flat = x_cpu.reshape(-1)
|
| 274 |
+
starts = (flat == eos_id).nonzero().flatten() + 1 # после EOS
|
| 275 |
+
starts = torch.cat([starts, torch.arange(0, N + 1, T)]) # стыки строк
|
| 276 |
+
cu = torch.unique(starts.clamp(max=N)).to(torch.int32)
|
| 277 |
+
lens = cu[1:] - cu[:-1]
|
| 278 |
+
pos = torch.arange(N, dtype=torch.int32) - torch.repeat_interleave(cu[:-1], lens)
|
| 279 |
+
return (cu.to(device, non_blocking=True), cu,
|
| 280 |
+
pos.to(device, non_blocking=True))
|
| 281 |
+
|
| 282 |
+
|
| 283 |
+
# --------------------------------------------------------------------------- #
|
| 284 |
+
# Блок и модель целиком
|
| 285 |
+
# --------------------------------------------------------------------------- #
|
| 286 |
+
|
| 287 |
+
class Block(nn.Module):
|
| 288 |
+
def __init__(self, cfg: ModelConfig, layer_idx: int):
|
| 289 |
+
super().__init__()
|
| 290 |
+
self.is_attn = cfg.is_attn_layer(layer_idx)
|
| 291 |
+
self.norm1 = RMSNorm(cfg.d_model, cfg.norm_eps)
|
| 292 |
+
self.mixer = GatedAttention(cfg) if self.is_attn else GatedDeltaNet(cfg)
|
| 293 |
+
self.norm2 = RMSNorm(cfg.d_model, cfg.norm_eps)
|
| 294 |
+
self.ffn = SwiGLU(cfg.d_model, cfg.ffn_hidden)
|
| 295 |
+
|
| 296 |
+
def forward(self, x, cos, sin, docs=None):
|
| 297 |
+
x = x + self.mixer(self.norm1(x), cos, sin, docs)
|
| 298 |
+
x = x + self.ffn(self.norm2(x))
|
| 299 |
+
return x
|
| 300 |
+
|
| 301 |
+
|
| 302 |
+
class HybridLM(nn.Module):
|
| 303 |
+
def __init__(self, cfg: ModelConfig):
|
| 304 |
+
super().__init__()
|
| 305 |
+
self.cfg = cfg
|
| 306 |
+
self.embed = nn.Embedding(cfg.vocab_size, cfg.d_model)
|
| 307 |
+
self.blocks = nn.ModuleList([Block(cfg, i) for i in range(cfg.n_layers)])
|
| 308 |
+
self.norm_f = RMSNorm(cfg.d_model, cfg.norm_eps)
|
| 309 |
+
self.lm_head = nn.Linear(cfg.d_model, cfg.vocab_size, bias=False)
|
| 310 |
+
if cfg.tie_embeddings:
|
| 311 |
+
self.lm_head.weight = self.embed.weight
|
| 312 |
+
|
| 313 |
+
self.apply(self._init)
|
| 314 |
+
# затухающая инициализация выходных проекций: чем глубже, тем тише вклад
|
| 315 |
+
for name, p in self.named_parameters():
|
| 316 |
+
if name.endswith("o_proj.weight") or name.endswith("down.weight"):
|
| 317 |
+
nn.init.normal_(p, std=cfg.init_std / math.sqrt(2 * cfg.n_layers))
|
| 318 |
+
|
| 319 |
+
self._rope = {}
|
| 320 |
+
|
| 321 |
+
def _init(self, m):
|
| 322 |
+
if isinstance(m, nn.Linear):
|
| 323 |
+
nn.init.normal_(m.weight, std=self.cfg.init_std)
|
| 324 |
+
elif isinstance(m, nn.Embedding):
|
| 325 |
+
nn.init.normal_(m.weight, std=self.cfg.init_std)
|
| 326 |
+
elif isinstance(m, nn.Conv1d):
|
| 327 |
+
nn.init.normal_(m.weight, std=self.cfg.init_std)
|
| 328 |
+
|
| 329 |
+
def _rope_for(self, T, device, dtype):
|
| 330 |
+
key = (T, device, dtype)
|
| 331 |
+
if key not in self._rope:
|
| 332 |
+
self._rope[key] = build_rope_cache(
|
| 333 |
+
T, self.cfg.head_dim, self.cfg.rope_theta, device, dtype)
|
| 334 |
+
return self._rope[key]
|
| 335 |
+
|
| 336 |
+
def forward(self, idx, targets=None, docs=None):
|
| 337 |
+
"""
|
| 338 |
+
docs -- маска документов для обучения, из make_doc_mask(): тогда
|
| 339 |
+
idx должен быть [1, N] (батч склеен в одну строку), а внимание и
|
| 340 |
+
состояние DeltaNet не переходят через границы документов.
|
| 341 |
+
"""
|
| 342 |
+
B, T = idx.shape
|
| 343 |
+
x = self.embed(idx)
|
| 344 |
+
cos, sin = self._rope_for(T, idx.device, torch.float32)
|
| 345 |
+
|
| 346 |
+
use_ckpt = self.cfg.grad_checkpoint and self.training
|
| 347 |
+
for blk in self.blocks:
|
| 348 |
+
if use_ckpt:
|
| 349 |
+
x = torch.utils.checkpoint.checkpoint(blk, x, cos, sin, docs, use_reentrant=False)
|
| 350 |
+
else:
|
| 351 |
+
x = blk(x, cos, sin, docs)
|
| 352 |
+
|
| 353 |
+
x = self.norm_f(x)
|
| 354 |
+
logits = self.lm_head(x)
|
| 355 |
+
|
| 356 |
+
loss = None
|
| 357 |
+
if targets is not None:
|
| 358 |
+
loss = F.cross_entropy(
|
| 359 |
+
logits.view(-1, logits.size(-1)).float(),
|
| 360 |
+
targets.reshape(-1),
|
| 361 |
+
ignore_index=-100,
|
| 362 |
+
)
|
| 363 |
+
return logits, loss
|
| 364 |
+
|
| 365 |
+
# ------------------------------------------------------------------ #
|
| 366 |
+
|
| 367 |
+
def param_counts(self):
|
| 368 |
+
emb = self.embed.weight.numel()
|
| 369 |
+
total = sum(p.numel() for p in self.parameters())
|
| 370 |
+
if self.cfg.tie_embeddings:
|
| 371 |
+
non_emb = total - emb
|
| 372 |
+
else:
|
| 373 |
+
non_emb = total - emb - self.lm_head.weight.numel()
|
| 374 |
+
return {"total": total, "embedding": emb, "non_embedding": non_emb}
|
| 375 |
+
|
| 376 |
+
def optim_groups(self, weight_decay: float):
|
| 377 |
+
"""Матрицы штрафуем весовым распадом, векторы (нормы, смещения) — нет."""
|
| 378 |
+
decay, no_decay = [], []
|
| 379 |
+
for n, p in self.named_parameters():
|
| 380 |
+
if not p.requires_grad:
|
| 381 |
+
continue
|
| 382 |
+
(decay if p.dim() >= 2 else no_decay).append(p)
|
| 383 |
+
return [
|
| 384 |
+
{"params": decay, "weight_decay": weight_decay},
|
| 385 |
+
{"params": no_decay, "weight_decay": 0.0},
|
| 386 |
+
]
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|