Instructions to use bammera-pothana/BERT-medium with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use bammera-pothana/BERT-medium with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="bammera-pothana/BERT-medium")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForPreTraining tokenizer = AutoTokenizer.from_pretrained("bammera-pothana/BERT-medium") model = AutoModelForPreTraining.from_pretrained("bammera-pothana/BERT-medium", device_map="auto") - Notebooks
- Google Colab
- Kaggle
BERT-medium
A BERT-Medium model (8 layers, hidden size 512, 8 attention heads, 42.2 M parameters) pre-trained from scratch on Telugu with Google's original BERT recipe. It is the fourth member of the Google's Recipes collection, which reproduces the compact "BERT miniatures" of Turc et al. (2019) for Telugu, size by size, without changing the recipe. Its smaller siblings are BERT-tiny, BERT-mini and BERT-small.
It is intended as the strongest compact Telugu encoder in the collection: a starting point for fine-tuning where BERT-Base is too heavy, a point on a scaling curve with the other sizes, or a masked language model for scoring and ranking Telugu text.
Model details
| Architecture | BertForPreTraining (masked LM head + next-sentence head) |
| Size | BERT-Medium: 8 layers, hidden 512, 8 heads, feed-forward 2048 |
| Parameters | 42,162,944 |
| Max sequence length | 512 |
| Vocabulary | 32,000 WordPiece tokens, cased, learned for this model; not shared with the other sizes |
| Precision of the released weights | float32 (model.safetensors, 170 MB) |
| Language | Telugu (te) |
| Script | Telugu script, NFC-normalised, no lower-casing or accent stripping |
| Trained by | bammera-pothana |
| Training dates | 15 – 20 November 2026 |
The tokenizer is a standard BertTokenizerFast. Lower-casing, accent stripping and CJK handling are off: stripping accents would destroy Telugu vowel signs (మాత్రలు), so the tokenizer must be loaded from this repository rather than configured by hand. Like the other models in the collection, the tokenizer removes the zero-width non-joiner (ZWNJ, U+200C) during normalisation, so సత్కార్యం and సత్కార్యం receive the same tokens. The vocabulary differs from the other sizes, so their masked-LM losses are only roughly comparable with this model's.
Training recipe
The recipe follows Devlin et al. (2018), as used for the BERT miniatures of Turc et al. (2019), with no modifications other than the Telugu vocabulary:
| Setting | Value |
|---|---|
| Objectives | Masked LM (15 % of WordPiece tokens: 80 % [MASK], 10 % random token, 10 % unchanged) + next-sentence prediction |
| Masking | Token-level (Google's original BERT), dynamic, re-sampled every batch |
| Total steps | 1,000,000 at a global batch of 256 sequences, one continuous schedule |
| Phase 1 | 900,000 steps at 128 tokens |
| Phase 2 | 100,000 steps at 512 tokens, optimizer state carried over from phase 1 |
| Optimizer | AdamW, lr 1e-4, β₁ 0.9, β₂ 0.999, ε 1e-6, weight decay 0.01 (not on bias / LayerNorm) |
| Schedule | Linear warm-up over the first 10,000 steps, then linear decay to 0 at step 1,000,000 |
| Gradient clipping | 1.0 |
| Dropout | 0.1 (hidden and attention) |
| Activation | GELU |
| Early stopping | Off; the checkpoint with the best held-out MLM loss is released |
| Mixed precision | float16 autocast with loss scaling; weights kept in float32 |
| Hardware | 4 × NVIDIA GeForce RTX 2080 Ti (11 GB) |
| Wall-clock | 36.1 h for phase 1 (0.14 s/step) + 16.9 h for phase 2 (0.60 s/step) |
Phase 1 was paused once, at step 750,000, and resumed from its full checkpoint (weights, optimizer state and step counter) on the same learning-rate schedule. No steps were skipped or repeated.
Training examples were built the way Google's create_pretraining_data.py builds them: [CLS] A [SEP] B [SEP], where B is the true continuation half of the time and a segment from a random other document otherwise; 10 % of examples use a shorter random target length. A held-out set of documents was kept aside before training and was never used for training.
Evaluation
Held-out pre-training text
Held-out documents, masked with a fixed random seed so that numbers are comparable across checkpoints. MLM accuracy is top-1 accuracy at masked positions.
| Checkpoint | MLM loss | MLM perplexity | MLM accuracy | NSP accuracy |
|---|---|---|---|---|
| End of phase 1 (step 900 k, 128 tokens) | 2.629 | 13.9 | 51.8 % | 96.2 % |
| Released: end of phase 2 (step 1 M, 512 tokens) | 2.506 | 12.3 | 53.5 % | 96.6 % |
Held-out MLM loss had converged by the end of phase 2 (2.512 at step 60 k → 2.506 at step 100 k). The per-phase evaluation files are in the training/ folder of this repository.
Clean typed Telugu outside the pre-training text
Two fixed samples of 3,000 paragraphs each, cut into 128-token pieces, 15 % of tokens masked with a fixed seed. Neither set was used in pre-training. The other sizes in the collection are shown for scale.
| Model | Wikisource (literary): loss / accuracy | News (modern): loss / accuracy |
|---|---|---|
| BERT-medium (this model) | 2.79 / 49.5 % | 2.89 / 49.3 % |
| BERT-medium, phase 1 only | 2.76 / 49.8 % | 2.87 / 49.5 % |
| BERT-small (4 layers, 512 hidden) | 3.05 / 46.2 % | 2.78 / 50.7 % |
| BERT-mini (4 layers, 256 hidden) | 3.61 / 39.4 % | 3.33 / 43.8 % |
| BERT-tiny (2 layers, 128 hidden) | 4.55 / 29.7 % | 4.73 / 28.7 % |
BERT-medium is the best model in the collection on literary text, 3.3 points of accuracy above BERT-small. On the news sample it is 1.4 points below BERT-small. Because BERT-medium uses its own vocabulary, the two models mask different token positions in the same paragraphs, so this cross-model comparison is approximate; it should not be read as a clean measure of model size alone. As in the smaller models, the 512-token phase 2 slightly lowers scores on these 128-token pieces while improving the 512-token held-out numbers.
Downstream (fine-tuning) results for this model will be added to this card as they become available.
How to use
from transformers import pipeline
fill = pipeline("fill-mask", model="bammera-pothana/BERT-medium")
fill("తెలుగు [MASK] భాష")
# [{'token_str': 'తెలుగు', ...}, {'token_str': 'కన్నడ', ...}, {'token_str': 'తమిళ', ...}, ...]
As an encoder for fine-tuning:
from transformers import AutoTokenizer, AutoModel, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained("bammera-pothana/BERT-medium")
enc = AutoModel.from_pretrained("bammera-pothana/BERT-medium") # hidden states, pooler
clf = AutoModelForSequenceClassification.from_pretrained("bammera-pothana/BERT-medium", num_labels=3)
Loading with AutoModelForMaskedLM discards the pooler and next-sentence head, which is expected and harmless. The full pre-training heads are available through BertForPreTraining.
Sample predictions from the released checkpoint:
| Input | Top predictions |
|---|---|
నేను [MASK] వెళ్తున్నాను. |
కూడా (0.04), రేపు, వేగంగా, ,, ఇప్పుడే |
ఆంధ్ర [MASK] పత్రిక |
వార (0.22, as in ఆంధ్ర వారపత్రిక), సచిత్ర, రైతు, సారస్వత, ##భూమి (as in ఆంధ్రభూమి) |
తెలుగు [MASK] భాష |
తెలుగు (0.12), కన్నడ, తమిళ, హిందీ, ఇంగ్లీషు |
Compared with the smaller sizes, BERT-medium fills the first prompt with adverbs (కూడా, రేపు, ఇప్పుడే) rather than destinations, and fills the third with a coherent set of language names.
Limitations and bias
- Compact model. With 8 layers and hidden size 512, BERT-Medium is the largest of the compact sizes but still below BERT-Base on downstream tasks. Use it where speed or size matters, or as a point on a scaling curve.
- ZWNJ is removed. The tokenizer deletes the zero-width non-joiner, so the model cannot distinguish words that differ only by ZWNJ. Strip or keep ZWNJ in your own pipeline knowing that the model will not see it.
- Domain. Like any pre-trained language model, it reflects the register, spelling conventions and viewpoints of the text it was trained on, and may perform worse on text from other domains.
- Pre-training only. No instruction tuning, no safety tuning. Outputs are token probabilities, not answers.
Repository contents
| File | Purpose |
|---|---|
config.json, model.safetensors |
Model (architecture BertForPreTraining) |
tokenizer.json, tokenizer_config.json, special_tokens_map.json, vocab.txt |
WordPiece tokenizer |
training/eval_phase1_128.json, training/eval_phase2_512.json |
Held-out metrics at the end of each phase |
Citation
If you use this model, please cite the original BERT and BERT-miniatures papers together with this repository:
@misc{bammera-pothana-bert-medium-2026,
title = {BERT-medium: a Telugu BERT-Medium model pre-trained with Google's original recipe},
author = {bammera-pothana},
year = {2026},
url = {https://huggingface.co/bammera-pothana/BERT-medium}
}
@article{turc2019,
title = {Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
author = {Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
journal = {arXiv preprint arXiv:1908.08962},
year = {2019}
}
@inproceedings{devlin2019,
title = {{BERT}: Pre-training of Deep Bidirectional Transformers for Language Understanding},
author = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
booktitle = {NAACL-HLT},
year = {2019}
}
- Downloads last month
- -
Collection including bammera-pothana/BERT-medium
Paper for bammera-pothana/BERT-medium
Evaluation results
- Masked-LM loss (cross-entropy) on Held-out Telugu text (512-token examples)self-reported2.506
- Masked-LM perplexity on Held-out Telugu text (512-token examples)self-reported12.300
- Masked-LM top-1 accuracy on Held-out Telugu text (512-token examples)self-reported0.535
- Next-sentence-prediction accuracy on Held-out Telugu text (512-token examples)self-reported0.966
- Masked-LM loss (cross-entropy) on Telugu Wikisource (typed literary text, 3,000 paragraphs, 128-token pieces)self-reported2.788
- Masked-LM top-1 accuracy on Telugu Wikisource (typed literary text, 3,000 paragraphs, 128-token pieces)self-reported0.495
- Masked-LM loss (cross-entropy) on Telugu news (typed modern text, 3,000 paragraphs, 128-token pieces)self-reported2.895
- Masked-LM top-1 accuracy on Telugu news (typed modern text, 3,000 paragraphs, 128-token pieces)self-reported0.493