mentee-embed-v1 — Trilingual Embeddings Trained From Scratch

mentee-embed-v1 is a compact 41M-parameter multilingual text embedding model supporting Arabic, English, and Urdu, trained entirely from scratch — no pretrained BERT or transformer base was used.

Trained in two stages on a single RTX 5090 GPU.

🌐 menteeai.org · 👤 syab.tech

Model Details

Property	Value
Architecture	Custom Transformer Encoder (Qwen-style)
Parameters	~41M
Embedding dimension	384
Layers	12
Attention heads	12
Vocabulary	50,000 BPE (trained from scratch)
Max sequence length	128 tokens
Languages	Arabic 🇸🇦 · English 🇬🇧 · Urdu 🇵🇰
Training compute	~$0.10 on RTX 5090

Training Recipe

Stage A — Masked Language Modeling

Trained from random initialization on 1.1M trilingual sentences
6,000 steps, batch size 64, sequence length capped at 128
Final val_mlm_loss ≈ 5.79 (expected for 41M params / 50K vocab / 62M tokens)

Stage B — Relational Knowledge Distillation

Teacher: intfloat/multilingual-e5-base (768-dim)
Method: Relational MSE — aligns the cosine similarity matrix of the student to match the teacher's, batch-by-batch
Loss: rel_weight × MSE(C_student, C_teacher) + ce_weight × InfoNCE(anchor, positive)
6,000 steps, batch size 512, lr 2e-4 with cosine decay
Final val acc@1 = 0.790 (128-way in-batch retrieval)

Training Data (1,110,575 triplets after dedup)

Source	Language	Triplets
all-NLI	English	557,850
XNLI	Arabic	127,856
XNLI	Urdu	124,869
OPUS-100 en-ur	English ↔ Urdu	300,000

Benchmark Results

Evaluated against strong multilingual baselines on two protocols. Same test queries for every model.

Protocol A — In-batch Retrieval (pool ≈ 97 candidates)

Metrics: acc@1 / recall@5 / MRR@10

Model	val	miracl_en	miracl_ar	miracl_ur	xling_en_ur	avg MRR@10
mentee-embed-v1 (ours)	0.820 / 0.979 / 0.895	0.345 / 0.650 / 0.501	0.222 / 0.477 / 0.373	0.183 / 0.419 / 0.329	0.757 / 0.912 / 0.829	0.585
paraphrase-multilingual-mpnet-base-v2	0.821 / 0.944 / 0.879	0.864 / 1.000 / 0.931	0.722 / 0.975 / 0.839	0.686 / 0.950 / 0.806	0.831 / 0.937 / 0.880	0.867
paraphrase-multilingual-MiniLM-L12-v2	0.795 / 0.940 / 0.864	0.854 / 0.997 / 0.924	0.696 / 0.964 / 0.819	0.621 / 0.908 / 0.753	0.782 / 0.907 / 0.841	0.840
all-MiniLM-L6-v2	0.484 / 0.636 / 0.584	0.856 / 0.999 / 0.927	0.025 / 0.109 / 0.144	0.028 / 0.088 / 0.140	0.065 / 0.172 / 0.186	0.396

✅ mentee-embed-v1 beats all-MiniLM-L6-v2 across all metrics despite being trained from scratch. ✅ val acc@1 = 0.820 — beats paraphrase-MiniLM-L12-v2 (0.795) on the validation set. ✅ xling_en_ur acc@1 = 0.757 — strong cross-lingual English↔Urdu retrieval.

Protocol B — Corpus-pool Retrieval (15K docs per language, full ranking)

Metrics: MRR@10 · R@5 · R@100

Model	English	Arabic	Urdu	avg MRR@10
paraphrase-multilingual-mpnet-base-v2	MRR 0.938 · R@5 0.973 · R@100 0.997	MRR 0.678 · R@5 0.763 · R@100 0.920	MRR 0.580 · R@5 0.633 · R@100 0.843	0.732
paraphrase-multilingual-MiniLM-L12-v2	MRR 0.921 · R@5 0.970 · R@100 0.997	MRR 0.685 · R@5 0.740 · R@100 0.930	MRR 0.518 · R@5 0.597 · R@100 0.803	0.708
all-MiniLM-L6-v2	MRR 0.920 · R@5 0.980 · R@100 1.000	MRR 0.100 · R@5 0.000 · R@100 0.000	MRR 0.100 · R@5 0.000 · R@100 0.007	0.373
mentee-embed-v1 (ours)	MRR 0.190 · R@5 0.163 · R@100 0.437	MRR 0.193 · R@5 0.163 · R@100 0.450	MRR 0.189 · R@5 0.137 · R@100 0.517	0.190

The Protocol B gap vs baselines is expected: those models were trained on billions of pairs with hard negatives by large research teams. This model was trained on 1.1M pairs in a single GPU session. R@100 = 0.43–0.52 means the correct document is in the top 100 nearly half the time, which is meaningful for re-ranking pipelines.

Usage

Basic Sentence Embeddings

from tokenizers import Tokenizer
import torch
import torch.nn.functional as F
import numpy as np

# Load model
payload = torch.load("model.pt", map_location="cpu", weights_only=False)
from src.model import build_embedder
model = build_embedder(payload["encoder_config"], payload["vocab_size"])
model.load_state_dict(payload["state_dict"])
model.eval()

tok = Tokenizer.from_file("tokenizer.json")

def encode(texts):
    enc = tok.encode_batch(texts)
    ids = np.zeros((len(enc), max(len(e.ids) for e in enc)), dtype=np.int64)
    for i, e in enumerate(enc):
        ids[i, :len(e.ids)] = e.ids[:128]
    with torch.no_grad():
        emb = model(torch.from_numpy(ids))
    return F.normalize(emb, dim=-1).numpy()

texts = ["مرحبا بالعالم", "hello world", "دنیا میں خوش آمدید"]
embeddings = encode(texts)
print(embeddings.shape)  # (3, 384)

Semantic Similarity

from numpy import dot

emb = encode(["Information retrieval using embeddings", "How to search documents with vectors"])
similarity = dot(emb[0], emb[1])  # cosine similarity (vectors are normalized)
print(f"Similarity: {similarity:.3f}")

Limitations

Protocol B corpus gap — open-domain retrieval over large corpora is weaker than web-scale models. Best used as a lightweight retriever or re-ranker in a pipeline.
Training data scope — NLI + parallel translation data; may underperform on domain-specific retrieval (legal, medical, etc.)
Sequence length — capped at 128 tokens; longer documents should be chunked.
Script dependency — current model.pt format requires src/model.py from the mentee-embeddings repo to load. A sentence-transformers-compatible export is planned.

Author

Syed Syab Ahmad Shah is the founder of MenteE AI, building open-source multilingual NLP tools for underrepresented languages — with a focus on Arabic, Urdu, and English.


🌐 Portfolio	syab.tech
🐦 Twitter	@SyabSays
💼 LinkedIn	linkedin.com/in/syedsyab
🐙 GitHub	github.com/syabahmad
📧 Email	syab@menteeai.org · syedsyabahmadshah@gmail.com

Citation

@misc{mentee-embed-v1,
  title  = {mentee-embed-v1: Trilingual Embeddings Trained From Scratch},
  author = {MenteE AI},
  year   = {2025},
  url    = {https://huggingface.co/MenteEAI/mentee-embed-v1}
}

Downloads last month: -; Downloads are not tracked for this model. How to track

MenteEAI
/

mentee-embed-v1