NandiBERT-150M

NandiBERT-150M by FrontiersMind

Hugging Face | GitHub | Launch Blog | Website | Technical Report (Coming Soon)

License: Apache-2.0 | Authors: FrontiersMind

NandiBERT is a family of open bidirectional encoders that reach state of the art in both English and Indic languages at their size, where earlier encoders were strong in one and weak in the other.
Model Parameters Hidden Context Description
NandiBERT-150M (this model) 146M 768 16K Pre-trained encoder
NandiBERT-450M 445M 1,024 32K Long-context pre-trained encoder

For NandiBERT-150M, the parameter count is the encoder that you fine-tune (146M). The checkpoint also stores the untied MLM output layer (77M), so the model file holds 224M parameters.

Description

  • One Model. From English to Indic and Beyond: NandiBERT-150M covers English, 13 Indic languages in native script (11 of them also in romanized form), with Southeast Asian languages and Middle-East Languages, so a single model replaces separate English and Indic encoders.
  • Deep and efficient: 28 layers at 768 hidden size, with grouped-query attention, QK-norm and SwiGLU, sized for fast inference and low-cost fine-tuning.
  • 16K context: A 16,384-token window with a 100K-token multilingual vocabulary shared across the NandiBERT family.
  • Factorized embeddings. The 100K-token vocabulary is stored as a rank-192 factorized embedding, which cuts the embedding table from 77M to 19M parameters and puts that capacity into the encoder layers instead.
  • Built to fine-tune: A strong starting point for classification, named-entity recognition, reranking and retrieval, loadable with Hugging Face transformers.

Benchmark Results

Indic Benchmarks

Model NameParametersClassificationStructure PredictionQAAverage
IndicSentimentIndicXNLIIndicCOPAIndicXParaphraseNaamapadam NER (f1)MASSIVE Slot-fill (f1)IndicQA (f1)
IndicBERT-v2272M91.273.764.656.973.258.148.366.57
NandiBERT-150M146M91.375.966.460.677.4544667.37

Note: NandiBERT is pre-trained on 13 Indic languages in native script: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu and Urdu.

English Benchmarks

Model NameLanguagesParametersClassificationRetrievalAverage
GLUESQuAD 2.0MUltiCoNERv2few-nerdReward BenchBEIRRAR-b CodeRAR-b MathMLDRLongEmbed
BERT-baseEnglish110M84.776.353.866.659.436.76.627.52929.351.45
RoBERTa-baseEnglish125M86.483.756.567.163.239.611.733.428.733.855.20
MosaicBERT-baseEnglish137M85.479.457.865.762.541.412.730.929.640.554.48
NomicBERT-2048English137M8482.760.166.964.14213.830.229.94555.48
GTE-en-MLM-baseEnglish137M85.682.759.266.463.442.41233.637.158.555.66
ModernBERT-baseEnglish149M88.486.757.466.76842.629.646.137.763.260.69
Ettin-Encoder-150MEnglish149M88.987.457.466.570.342.431.847.135.960.161.48
UltraBERT-150MEnglish152M89.589.257.967.47345.239.155.443.57264.59
NandiBERT-150MMulti-Lingual146M87.187.656.968.171.244.532.849.441.271.162.20

Note: Model results are the average of 3 different seeds.

Usage

NandiBERT-150M loads with Hugging Face transformers 5.x using trust_remote_code=True.

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer
 
model_id = "FrontiersMind/NandiBERT-150M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="cuda",
)
 
texts = [
    f"The capital of France is {tokenizer.mask_token}.",
    f"भारत की राजधानी {tokenizer.mask_token} है।",
]
 
for text in texts:
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        logits = model(**inputs).logits
    mask_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero()[0, 1]
    print(tokenizer.decode(logits[0, mask_index].argmax()))

Fine-tuning

AutoModelForSequenceClassification and AutoModelForTokenClassification reuse the pretrained MLM head. The encoder output goes through the MLM head (dense, activation, RMSNorm) and then a new linear layer, so only the linear layer starts from random weights. This is the setup we recommend for classification and NER.

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "FrontiersMind/NandiBERT-150M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=3, trust_remote_code=True)

For NER with the MLM head, use a learning rate of 3e-5. A low learning rate like 5e-6 gives much worse results. To fine-tune without the MLM head, pass classifier_head=None to from_pretrained.

For word-level input (is_split_into_words=True, as in NER), the tokenizer adds a space before every word, the same as in running text, so pass the words as they are. Use word_ids() to map tokens back to words.

Citation

@misc{frontiersmind2026nandibert,
  title        = {NandiBERT: Open Encoders for English and Indic Languages},
  author       = {{FrontiersMind}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/FrontiersMind/NandiBERT-150M}}
}
Downloads last month
56
Safetensors
Model size
0.2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including FrontiersMind/NandiBERT-150M