Walkie-Code-0.5B

Walkie-Code-0.5B is a ~501M parameter decoder-only language model focused on Python code generation. It is the flagship model from the LLM Walk-Through project — an educational and engineering effort to walk through modern LLM design from modular components to a full training pipeline.

This checkpoint is the DAPO RL best model after: 17B-token pretraining → KodCode SFT → DAPO reinforcement learning.

Note on config format: weights are exported in a Qwen3-compatible layout (model_type: qwen3) because Walkie shares the same structural primitives (RMSNorm, SwiGLU, GQA, QK-Norm). This is an export convention for Transformers/vLLM compatibility — the model is Walkie-Code-0.5B, not an official Qwen release.

Model Summary

Item	Value
Parameters	~501M
Architecture	24-layer decoder, hidden 1280, GQA 4:1 (20Q / 5KV)
Context length	4096 tokens
Vocab size	65536 (BPE)
FFN	SwiGLU, d_ffn=3456
Position encoding	RoPE (θ=5×10⁵)
Training stages	Pretrain → SFT → DAPO RL
Best RL method	DAPO (Direct Advantage Policy Optimization)

Benchmark Results

Evaluated on HumanEval, HumanEval+, MBPP, MBPP+ with n=8, temperature=0.2, top_p=0.95, sandbox execution.

Macro average (this checkpoint)

Metric	Score
pass@1	37.6%
pass@4	43.6%
pass@8	46.6%

Per-dataset pass@1

Dataset	pass@1
HumanEval+	34.1%
HumanEval	36.1%
MBPP	42.1%
MBPP+	38.3%

Comparison vs Qwen2.5-0.5B-Instruct (pass@1 macro)

Model	Macro pass@1
Qwen2.5-0.5B-Instruct	36.7%
Walkie-Code-0.5B (this)	38.4% (+1.7 pp)

Full training pipeline (macro pass@1)

Stage	pass@1	pass@8
SFT	33.7%	41.4%
DAPO (this checkpoint)	37.6%	46.6%

Training Details

Pretraining (~17B tokens, code-heavy ~70%)

• The Stack v2 Python, StarCoder Python Edu, FineWeb Edu, FineMath, OPC Annealing
• Two-stage WSD schedule (main 89% + anneal 11%)
• Muon + AdamW mixed optimizer, FlashAttention-2

SFT

• KodCode-V1-SFT-R1 (~246k samples), DeepSeek-R1 generated solutions
• Instruct/complete Python function generation, ~2.6 epochs

RL (DAPO)

• KodCode-V1-RL (~12k filtered samples)
• Online rollout with code sandbox rewards (pass/fail)
• Dynamic group filtering + clipped policy gradient (ε_l=0.2, ε_h=0.28)

Usage

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Henry665/Walkie-Code-0.5B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "user: Write a Python function to check if a number is prime.\nassistant:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.2, top_p=0.95)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

vLLM

vllm serve Henry665/Walkie-Code-0.5B --dtype auto --max-model-len 4096

Prompt format

Training used a simple dialog template:

user: <instruction>
assistant: <python code>

For code benchmarks, plain user: / assistant: text prompts are recommended.

Limitations

• Specialized for Python code generation; general chat / multilingual ability is limited.
• Small scale (0.5B); not competitive with much larger models on broad reasoning.
• Exported as Qwen3-compatible config for tooling — verify behavior matches Walkie training setup.
• Benchmark scores depend on prompt template, sandbox, and sampling settings.

Citation & Links

• Project: LLM Walk-Through
• Architecture: RMSNorm, RoPE, GQA, SwiGLU, QK-Norm, Muon optimizer
• RL method: DAPO (dynamic filtering + clipped surrogate)

@misc{walkie-code-0.5b,
  title={Walkie-Code-0.5B: A Modular 0.5B Python Code LLM},
  author={LLM Walk-Through Team},
  year={2026},
  howpublished={\url{https://huggingface.co/Henry665/Walkie-Code-0.5B}}
}

License

Apache 2.0