gaon12/haru_2

Haru v2.0

Haru is a compact Korean story continuation model built with the custom CFRD causal architecture. It has 16,983,213 parameters and runs at recurrent depth 6 only.

• Source code
• Interactive demo
• Previous Haru release

Usage

Review the included Python files before enabling remote custom code.

from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "gaon12/haru_2"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, trust_remote_code=True)

inputs = tokenizer("작은 마을에 아침이 찾아왔어요.", return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=120,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    top_k=40,
    repetition_penalty=1.08,
    use_cache=False,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Model details

• Parameters: 16,983,213
• Context length: 512
• Recurrent depths supervised during training: none; AUX_EXIT_LOSS_WEIGHT is 0.0, so the depth 2 and 4 exits are untrained
• Exported checkpoint step: 32500
• Training tokens seen: 4,259,840,000

Evaluation

Recurrent depth	Validation loss	Perplexity
2	8.79562	6605.228
4	5.56061	259.982
6	3.38503	29.519

Training data attribution

Tiny-Ko-Stories by psymon, licensed under CC BY 4.0. The dataset is not redistributed with this model.

Limitations

• Haru is a continuation model, not an instruction-following assistant.
• Longer generations can repeat ideas or drift between entities.
• The model is not suitable for factual or safety-critical use.
• There is no inference cache yet, so generation recomputes the active context.

License

Haru model weights and included code are released under the MIT License. The training dataset remains under its separate CC BY 4.0 license.

KoBEST

Scored with lm-eval on the five KoBEST tasks. The tasks mix two-way and four-way formats, so the mean has no fixed floor: 0.450 is what a coin flip scores. Read the "vs chance" column.

Task	Haru v2.0	Haru v1.1	Chance	vs chance
kobest_boolq	0.507	0.502	0.50	+0.007
kobest_copa	0.529	0.500	0.50	+0.029
kobest_hellaswag	0.314	0.224	0.25	+0.064
kobest_sentineg	0.509	0.496	0.50	+0.009
kobest_wic	0.488	0.488	0.50	-0.012
mean	0.469	0.442	0.450	+0.019

v1.1 was at or below chance on every task. v2.0 is above chance on four of five, and hellaswag at 0.314 against a 0.25 chance level is the first evidence in this project of a task being learned rather than guessed.

Against other sub-20M Korean models

Every row was scored locally through the same harness and task list rather than quoted from a model card.

Model	Parameters	Mean	vs chance
Haru v2.0	17.0M	0.469	+0.019
minpeter/tiny-ko-20m-sft	20.0M	0.463	+0.013
minpeter/tiny-ko-20m-base	20.0M	0.457	+0.007
Haru v1.1	11.6M	0.442	-0.008

These are small margins on a benchmark whose chance mean is 0.450, and all four models score below chance on wic. Sub-20M Korean models are all near the floor; Haru v2.0 is a little further from it than the others tested.

Recurrent depth

v2.0 runs at depth 6 only.

Depth	v2.0 loss	v2.0 PPL
2	8.79562	6605.2
4	5.56061	260.0
6	3.38503	29.5

v1.1 degraded gracefully across depths 2, 4 and 6 because it reused three physical cells and trained the intermediate exits with deep supervision. v2.0 unfolds the stack into six independent cells and sets the deep-supervision weight to zero, so the shallow exits receive no training signal. Early exit was a property of cell reuse plus deep supervision, and v2.0 dropped both.

Training

17.0M parameters, 12,000-token BPE tokenizer, 512-token context. Trained on 4.26B tokens drawn from a 2.6B-token Korean corpus that was n-gram decontaminated against all five KoBEST evaluation sets.

Training stopped at 4.26B of a planned 10.4B tokens: two KoBEST measurements 1.5B tokens apart moved the mean by +0.003, inside the harness's own noise, so the remaining budget was not buying measurable accuracy.

Limitations

• Repetition collapse is common after the first sentence or two.
• Entity tracking is unreliable; the model confuses referents in longer spans.
• It reproduces encyclopaedic surface form without the underlying facts.
• Not an instruction model, a factual assistant, or a safety-reviewed product.