Haidass1.5-143M

中文 | English

A bilingual (English/Chinese) small language model trained entirely on the Huawei Ascend NPU ecosystem.

Model Overview

Haidass1.5-143M is a 143M-parameter bilingual language model trained on approximately 400B tokens of English and Chinese data. The entire training pipeline runs on the Huawei Ascend ecosystem, using the MindSpeed-LLM framework on Atlas A2 servers (910B). A custom 64,000-token bilingual vocabulary was trained alongside the model. This model is competitive among multilingual models under 150M parameters and ranks favorably across multiple evaluation benchmarks.

Model Architecture

Parameter	Value
Architecture	Qwen3
Layers	30
Hidden size	576
Attention heads	9
KV heads (GQA)	3
Head dim	64
FFN intermediate size	1,536
Vocabulary size	64,000
Max sequence length	4,096
Tie word embeddings	Yes
Activation	SwiGLU (SiLU)
Normalization	RMSNorm (eps=1e-6)
Position encoding	RoPE (θ=100,000)
Attention bias	None
Precision	BF16
Total parameters	~143M

Training Data

The model was trained on approximately 400B tokens of mixed English and Chinese data. Primary data sources:

• openbmb/Ultra-FineWeb (ultrafineweb-en + ultrafineweb-zh)
• mlfoundations/dclm-baseline-1.0-parquet (dclm)
• HuggingFaceTB/finemath (finemath-4plus)
• openbmb/Ultra-FineWeb-L3
• HuggingFaceTB/cosmopedia

Training Configuration

Parameter	Value
Framework	MindSpeed-LLM (v2.3.0)
NPU model	Huawei Ascend 910B
Global batch size	128
Sequence length	4,096

Optimizer

Parameter	Value
Optimizer	AdamW
Peak learning rate	1.5e-3
Min learning rate	0
Weight decay	1e-5
Gradient clipping	2.0
Adam β1	0.9
Adam β2	0.95
Initial loss scale	4,096

Tokenizer

Property	Value
Vocabulary size	64,000
Language coverage	English + Chinese

Evaluation and Comparison

Evaluated using the lm-eval-harness framework, zero-shot.

Benchmark	Score
ARC-Easy	59.09
ARC-Challenge	28.33
PIQA	68.72
HellaSwag	40.54
OpenBookQA	31.2
Winogrande	51.78
agi_eval	25.93

Key Features

• Fully Ascend-native: Trained entirely on Huawei Ascend 910B NPUs using the MindSpeed-LLM framework
• Bilingual: Trained on a mixture of English and Chinese data

Intended Use

This is a research model, suitable for:

• Studying training dynamics of small models on Ascend NPUs
• English/Chinese language modeling research
• Serving as a base model for fine-tuning or annealing experiments

Limitations

• Small model scale; reasoning and generation capabilities are limited
• No instruction tuning — raw pretrained model only
• No RLHF or alignment training

Citation

@misc{Haidass1.5-143m,
  title={Haidass1.5-143m: A Bilingual Small Language Model Trained on Ascend 910B},
  year={2026},
  note={Trained from scratch on 400B tokens using MindSpeed-LLM on Ascend 910B NPUs}
}

License

Apache 2.0