DedeProGames/Kiyo-230M-Preview

Kiyo-230M-Preview is an intermediate training checkpoint of Kiyo-230M, a decoder-only language model being pretrained from scratch on FineWeb-Edu, DCLM-Baseline, FineMath, and Stack-v3-train. It is released as a preview: the training run is paused and this checkpoint has seen roughly one fifth of its planned token budget. See Preview Status below before using or comparing it.

The architecture is a Llama-style decoder with grouped query attention, RMSNorm, SwiGLU MLPs, and tied input/output embeddings, following the SmolLM2 architecture family and reusing its 49k-token vocabulary.

Preview Status

This repository holds the weights at optimizer step 81,500 of a run targeting 200B tokens. Three things follow from that, and all three matter when reading the benchmark numbers:

Field	Value
Tokens seen by this checkpoint	42,729,472,000 (42.7B)
Target token budget	200,000,000,000 (200B)
Progress	21.4%
Optimizer step	81,500
Planned total steps	~381,470
Learning rate at this checkpoint	1.2e-3 (peak, stable phase)
LR decay begins at step	~343,320 (final 10% of the run)

The learning rate has not decayed yet. The run uses a warmup-stable-decay schedule whose decay phase covers only the final 10% of steps. This checkpoint sits deep in the stable phase, still at the peak LR of 1.2e-3. Models evaluated mid-plateau consistently score below what the same weights reach after annealing, so the numbers below understate the model rather than describe it.

It has seen fewer tokens than Kiyo-135M. Kiyo-135M was trained on the full 200B tokens. This preview has seen 42.7B. A parameter-count comparison between the two is not a fair comparison of the architectures — it is a comparison of a finished run against a fifth of one.

Weights are not final and will change. Do not treat this checkpoint as the Kiyo-230M release. It exists so the run can be inspected mid-flight; the final model will be published separately.

Model Details

Field	Value
Parameters	229,688,064
Architecture	Llama-style decoder (SmolLM2 architecture family)
Layers	32
Hidden size	768
Intermediate size	1,920
Attention heads	12
KV heads	4
Head dimension	64
Attention type	Grouped query attention
Activation	SwiGLU
Normalization	RMSNorm
Positional encoding	RoPE (theta 10,000)
Vocabulary size	49,152
Context length	2,048
Embeddings	Tied input/output embeddings
Training tokens (this checkpoint)	42,729,472,000
Tokens per optimizer step	524,288
Peak learning rate	1.2e-3
LR schedule	Warmup-stable-decay (1/sqrt), decay not yet reached
Weight format	safetensors

Training Data

Source	Domain	Mixture share
FineWeb-Edu	General web text, education-filtered	45.0%
DCLM-Baseline	General web text, high-quality filtered	30.0%
Stack-v3-train	Source code	17.5%
FineMath	Mathematical reasoning	7.5%

The Stack-v3 portion is sampled with per-language keep probabilities across 32 curated programming languages, giving each language an equal token quota rather than inheriting the corpus's natural skew.

Benchmarks

Scores are percentages from zero-shot acc_norm,none evaluations. Bold numbers indicate Kiyo-230M-Preview scores. Results may vary with evaluation settings.

Benchmark	Kiyo-230M-Preview	Supra2-100M-Base	GPT-X2.5-135M	SmolLM2-135M
ARC-Easy	49.54%	47.81%	51.81%	58.63%
PIQA	64.09%	65.40%	69.42%	68.44%
ARC-Challenge	26.11%	24.83%	29.18%	29.69%
HellaSwag	35.94%	35.98%	40.57%	43.22%

Usage

pip install -U transformers safetensors torch

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM


model_id = "DedeProGames/Kiyo-230M-Preview"


tokenizer = AutoTokenizer.from_pretrained(model_id)


model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
).cuda().eval()


prompt = "The meaning of life is "
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)


with torch.no_grad():
    output = model.generate(
        input_ids=input_ids,
        max_new_tokens=64,
        do_sample=False,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )


print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

This is an unfinished base model, not instruction-tuned — it continues text rather than following instructions, and its weights are an intermediate checkpoint rather than a release. Beyond the preview caveats above, it shares the limits of its scale: fluent and grammatical on language completion, competent on code, but weak on multi-step logic and context tracking, where it stays near chance on the harder items. The 2,048-token context window limits long-document use. It can generate incorrect facts and should not be used for high-stakes decisions without verification. Keep a finite generation limit to avoid repetition or drift on long outputs.

License

Apache 2.0