See our collection for all versions of MPNet.

Run MPNet with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/mpnet_base

Paper: MPNet: Masked and Permuted Pre-training for Language Understanding (arXiv:2004.09297) · HF Papers

MPNet is a bidirectional encoder pre-trained with masked and permuted language modeling, unifying BERT's masked-LM objective with XLNet's permuted one. Unlike BERT it has no token-type embeddings, offsets position ids past the padding id, and adds a shared relative position bias to every attention layer.

For more details on the model, please go to the upstream model card.

Pure-Keras 3 conversion of microsoft/mpnet-base for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is a fill-mask / encoder checkpoint (MPNetMaskedLM, 12 layers / 768 dim). Task heads load via hf: fine-tunes.

✨ Quick start (fill-mask)

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from zeromodels.models.mpnet import MPNetMaskedLM, MPNetTokenizer

mlm = MPNetMaskedLM.from_weights("zeromodels/mpnet_base")
tokenizer = MPNetTokenizer.from_weights("zeromodels/mpnet_base")

inputs = tokenizer("the capital of France is <mask>.")
logits = mlm(inputs)  # (1, L, vocab_size)
mask = int((inputs["input_ids"][0] == tokenizer.mask_token_id).argmax())
print(tokenizer.decode([int(logits[0, mask].argmax())]))

Load any MPNet variant the same way with from_weights("zeromodels/<variant>"):

Variant	Hub
`mpnet_base`	`zeromodels/mpnet_base`

Available classes

Load any of these from this repo with from_weights("zeromodels/mpnet_base") (or on the fly via the hf: prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a hf: fine-tune).

Class	Task
`MPNetModel`	Encoder backbone
`MPNetMaskedLM`	Masked language modeling (fill-mask)
`MPNetSequenceClassify`	Sequence classification
`MPNetTokenClassify`	Token classification (NER / POS)
`MPNetQnA`	Extractive question answering
`MPNetMultipleChoice`	Multiple choice

from zeromodels.models.mpnet import MPNetSequenceClassify
model = MPNetSequenceClassify.from_weights("zeromodels/mpnet_base")

Tips

Set KERAS_BACKEND before importing Keras / zeromodels.
Prefer MPNetTokenizer.from_weights(...) so the WordPiece vocab matches.
Use <mask> (not [MASK]); MPNet pairs RoBERTa-style special tokens with a WordPiece vocabulary.
MPNet takes input_ids + attention_mask only — there are no token_type_ids.
See MPNet docs and Loading Weights.
Community / upstream safetensors still work via the hf: prefix, e.g. MPNetMaskedLM.from_weights("hf:microsoft/mpnet-base").