BERT-medium

A BERT-Medium model (8 layers, hidden size 512, 8 attention heads, 42.2 M parameters) pre-trained from scratch on Telugu with Google's original BERT recipe. It is the fourth member of the Google's Recipes collection, which reproduces the compact "BERT miniatures" of Turc et al. (2019) for Telugu, size by size, without changing the recipe. Its smaller siblings are BERT-tiny, BERT-mini and BERT-small.

It is intended as the strongest compact Telugu encoder in the collection: a starting point for fine-tuning where BERT-Base is too heavy, a point on a scaling curve with the other sizes, or a masked language model for scoring and ranking Telugu text.

Model details

Architecture BertForPreTraining (masked LM head + next-sentence head)
Size BERT-Medium: 8 layers, hidden 512, 8 heads, feed-forward 2048
Parameters 42,162,944
Max sequence length 512
Vocabulary 32,000 WordPiece tokens, cased, learned for this model; not shared with the other sizes
Precision of the released weights float32 (model.safetensors, 170 MB)
Language Telugu (te)
Script Telugu script, NFC-normalised, no lower-casing or accent stripping
Trained by bammera-pothana
Training dates 15 – 20 November 2026

The tokenizer is a standard BertTokenizerFast. Lower-casing, accent stripping and CJK handling are off: stripping accents would destroy Telugu vowel signs (మాత్రలు), so the tokenizer must be loaded from this repository rather than configured by hand. Like the other models in the collection, the tokenizer removes the zero-width non-joiner (ZWNJ, U+200C) during normalisation, so సత్‌కార్యం and సత్కార్యం receive the same tokens. The vocabulary differs from the other sizes, so their masked-LM losses are only roughly comparable with this model's.

Training recipe

The recipe follows Devlin et al. (2018), as used for the BERT miniatures of Turc et al. (2019), with no modifications other than the Telugu vocabulary:

Setting Value
Objectives Masked LM (15 % of WordPiece tokens: 80 % [MASK], 10 % random token, 10 % unchanged) + next-sentence prediction
Masking Token-level (Google's original BERT), dynamic, re-sampled every batch
Total steps 1,000,000 at a global batch of 256 sequences, one continuous schedule
Phase 1 900,000 steps at 128 tokens
Phase 2 100,000 steps at 512 tokens, optimizer state carried over from phase 1
Optimizer AdamW, lr 1e-4, β₁ 0.9, β₂ 0.999, ε 1e-6, weight decay 0.01 (not on bias / LayerNorm)
Schedule Linear warm-up over the first 10,000 steps, then linear decay to 0 at step 1,000,000
Gradient clipping 1.0
Dropout 0.1 (hidden and attention)
Activation GELU
Early stopping Off; the checkpoint with the best held-out MLM loss is released
Mixed precision float16 autocast with loss scaling; weights kept in float32
Hardware 4 × NVIDIA GeForce RTX 2080 Ti (11 GB)
Wall-clock 36.1 h for phase 1 (0.14 s/step) + 16.9 h for phase 2 (0.60 s/step)

Phase 1 was paused once, at step 750,000, and resumed from its full checkpoint (weights, optimizer state and step counter) on the same learning-rate schedule. No steps were skipped or repeated.

Training examples were built the way Google's create_pretraining_data.py builds them: [CLS] A [SEP] B [SEP], where B is the true continuation half of the time and a segment from a random other document otherwise; 10 % of examples use a shorter random target length. A held-out set of documents was kept aside before training and was never used for training.

Evaluation

Held-out pre-training text

Held-out documents, masked with a fixed random seed so that numbers are comparable across checkpoints. MLM accuracy is top-1 accuracy at masked positions.

Checkpoint MLM loss MLM perplexity MLM accuracy NSP accuracy
End of phase 1 (step 900 k, 128 tokens) 2.629 13.9 51.8 % 96.2 %
Released: end of phase 2 (step 1 M, 512 tokens) 2.506 12.3 53.5 % 96.6 %

Held-out MLM loss had converged by the end of phase 2 (2.512 at step 60 k → 2.506 at step 100 k). The per-phase evaluation files are in the training/ folder of this repository.

Clean typed Telugu outside the pre-training text

Two fixed samples of 3,000 paragraphs each, cut into 128-token pieces, 15 % of tokens masked with a fixed seed. Neither set was used in pre-training. The other sizes in the collection are shown for scale.

Model Wikisource (literary): loss / accuracy News (modern): loss / accuracy
BERT-medium (this model) 2.79 / 49.5 % 2.89 / 49.3 %
BERT-medium, phase 1 only 2.76 / 49.8 % 2.87 / 49.5 %
BERT-small (4 layers, 512 hidden) 3.05 / 46.2 % 2.78 / 50.7 %
BERT-mini (4 layers, 256 hidden) 3.61 / 39.4 % 3.33 / 43.8 %
BERT-tiny (2 layers, 128 hidden) 4.55 / 29.7 % 4.73 / 28.7 %

BERT-medium is the best model in the collection on literary text, 3.3 points of accuracy above BERT-small. On the news sample it is 1.4 points below BERT-small. Because BERT-medium uses its own vocabulary, the two models mask different token positions in the same paragraphs, so this cross-model comparison is approximate; it should not be read as a clean measure of model size alone. As in the smaller models, the 512-token phase 2 slightly lowers scores on these 128-token pieces while improving the 512-token held-out numbers.

Downstream (fine-tuning) results for this model will be added to this card as they become available.

How to use

from transformers import pipeline

fill = pipeline("fill-mask", model="bammera-pothana/BERT-medium")
fill("తెలుగు [MASK] భాష")
# [{'token_str': 'తెలుగు', ...}, {'token_str': 'కన్నడ', ...}, {'token_str': 'తమిళ', ...}, ...]

As an encoder for fine-tuning:

from transformers import AutoTokenizer, AutoModel, AutoModelForSequenceClassification

tok = AutoTokenizer.from_pretrained("bammera-pothana/BERT-medium")
enc = AutoModel.from_pretrained("bammera-pothana/BERT-medium")           # hidden states, pooler
clf = AutoModelForSequenceClassification.from_pretrained("bammera-pothana/BERT-medium", num_labels=3)

Loading with AutoModelForMaskedLM discards the pooler and next-sentence head, which is expected and harmless. The full pre-training heads are available through BertForPreTraining.

Sample predictions from the released checkpoint:

Input Top predictions
నేను [MASK] వెళ్తున్నాను. కూడా (0.04), రేపు, వేగంగా, ,, ఇప్పుడే
ఆంధ్ర [MASK] పత్రిక వార (0.22, as in ఆంధ్ర వారపత్రిక), సచిత్ర, రైతు, సారస్వత, ##భూమి (as in ఆంధ్రభూమి)
తెలుగు [MASK] భాష తెలుగు (0.12), కన్నడ, తమిళ, హిందీ, ఇంగ్లీషు

Compared with the smaller sizes, BERT-medium fills the first prompt with adverbs (కూడా, రేపు, ఇప్పుడే) rather than destinations, and fills the third with a coherent set of language names.

Limitations and bias

  • Compact model. With 8 layers and hidden size 512, BERT-Medium is the largest of the compact sizes but still below BERT-Base on downstream tasks. Use it where speed or size matters, or as a point on a scaling curve.
  • ZWNJ is removed. The tokenizer deletes the zero-width non-joiner, so the model cannot distinguish words that differ only by ZWNJ. Strip or keep ZWNJ in your own pipeline knowing that the model will not see it.
  • Domain. Like any pre-trained language model, it reflects the register, spelling conventions and viewpoints of the text it was trained on, and may perform worse on text from other domains.
  • Pre-training only. No instruction tuning, no safety tuning. Outputs are token probabilities, not answers.

Repository contents

File Purpose
config.json, model.safetensors Model (architecture BertForPreTraining)
tokenizer.json, tokenizer_config.json, special_tokens_map.json, vocab.txt WordPiece tokenizer
training/eval_phase1_128.json, training/eval_phase2_512.json Held-out metrics at the end of each phase

Citation

If you use this model, please cite the original BERT and BERT-miniatures papers together with this repository:

@misc{bammera-pothana-bert-medium-2026,
  title  = {BERT-medium: a Telugu BERT-Medium model pre-trained with Google's original recipe},
  author = {bammera-pothana},
  year   = {2026},
  url    = {https://huggingface.co/bammera-pothana/BERT-medium}
}

@article{turc2019,
  title   = {Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author  = {Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal = {arXiv preprint arXiv:1908.08962},
  year    = {2019}
}

@inproceedings{devlin2019,
  title     = {{BERT}: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author    = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  booktitle = {NAACL-HLT},
  year      = {2019}
}
Downloads last month
-
Safetensors
Model size
42.4M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including bammera-pothana/BERT-medium

Paper for bammera-pothana/BERT-medium

Evaluation results

  • Masked-LM loss (cross-entropy) on Held-out Telugu text (512-token examples)
    self-reported
    2.506
  • Masked-LM perplexity on Held-out Telugu text (512-token examples)
    self-reported
    12.300
  • Masked-LM top-1 accuracy on Held-out Telugu text (512-token examples)
    self-reported
    0.535
  • Next-sentence-prediction accuracy on Held-out Telugu text (512-token examples)
    self-reported
    0.966
  • Masked-LM loss (cross-entropy) on Telugu Wikisource (typed literary text, 3,000 paragraphs, 128-token pieces)
    self-reported
    2.788
  • Masked-LM top-1 accuracy on Telugu Wikisource (typed literary text, 3,000 paragraphs, 128-token pieces)
    self-reported
    0.495
  • Masked-LM loss (cross-entropy) on Telugu news (typed modern text, 3,000 paragraphs, 128-token pieces)
    self-reported
    2.895
  • Masked-LM top-1 accuracy on Telugu news (typed modern text, 3,000 paragraphs, 128-token pieces)
    self-reported
    0.493