pangram-like

A small AI-generated-text detector in the spirit of Pangram, built on bert-base-uncased with masked mean pooling and a linear head. It outputs P(AI) for a text, and since the head is linear, the prediction can be broken down token by token (see the visualization in the code repo).

Code, training notebook and visualization: https://github.com/keyvanatt/pangram-like

Checkpoints

File	Description	Val accuracy
checkpoint.pt	BERT frozen, linear head only (2 epochs on precomputed embeddings)	78.9%
finetune.pt	Starts from checkpoint.pt; BERT layers 9–11 and head fine-tuned (3 epochs, 240k samples)	89.4%

Both are PyTorch state_dicts of the Classifier class (bert.* + head.*).

Results (finetune.pt, 10k test samples, AI if logit ≥ 1, i.e. P(AI) ≥ 0.73)

	Predicted human	Predicted AI
Human	3069	703
AI	234	5994

Accuracy 90.6%, precision 89.5%, recall 96.2%, false positive rate 18.6%.

Usage

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel, AutoTokenizer
from src.pangram_like.train import Classifier, ClassifierHead  # from the GitHub repo

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = Classifier(AutoModel.from_pretrained("bert-base-uncased"), ClassifierHead(hidden_layers=[]))
path = hf_hub_download("keyvanatt/pangram-like", "finetune.pt")
model.load_state_dict(torch.load(path, map_location="cpu", weights_only=True))
model.eval()

enc = tokenizer("Some text to check.", truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    logit = model(**enc)
print("P(AI) =", torch.sigmoid(logit).item())

Limitations

Trained on English data from the COLING 2025 MGT shared task only. Even with the threshold raised to logit ≥ 1 (P(AI) ≥ 0.73) instead of 0.5, about 1 in 5 human texts is still flagged as AI; a higher threshold trades recall for fewer false positives. Do not use it as the sole basis for judging whether someone used AI.