title: Compact Language Model Index
emoji: 📊
colorFrom: indigo
colorTo: blue
sdk: static
app_file: build/index.html
fullWidth: true
short_description: Chance-normalized evaluation of compact language models.
Compact Language Model Index
CompactLMIndex is a curated benchmark for smaller language models. It uses one fixed protocol to compare models across commonsense reasoning, science and world knowledge, and grammatical competence.
Evaluation suite
The published composite includes six benchmarks:
| Area | Benchmark |
|---|---|
| Situational commonsense | HellaSwag |
| Physical commonsense | PIQA |
| Science and world knowledge | ARC-Easy |
| Science and world knowledge | ARC-Challenge |
| Conceptual commonsense | CommonsenseQA |
| Grammatical competence | BLiMP |
All models use zero-shot, non-chat log-likelihood prompts. A complete run of the core suite is required for a model to receive a rank.
Scoring
Each benchmark score is normalized for random chance:
100 × (accuracy − chance) / (1 − chance)
Random performance maps to 0 and perfect performance maps to 100. Scores below chance remain negative.
The composite is domain-balanced:
- commonsense: 50%, split equally between HellaSwag, PIQA, and CommonsenseQA;
- science and world knowledge: 25%, with ARC-Easy contributing 75% and ARC-Challenge 25% within that domain;
- grammatical competence: 25%, from BLiMP.
BLiMP is macro-averaged across its 12 linguistic categories. CommonsenseQA scores the full answer text with length normalization rather than answer labels such as A–E.
Uncertainty and ranking
Every displayed benchmark and composite interval is a 95% percentile bootstrap interval. Model comparisons use paired bootstrap resampling on matched evaluation items. Rank ranges show when neighbouring models cannot be reliably distinguished under those comparisons.
Scope and inclusion
CompactLMIndex is curated rather than an open intake list. Published results are checked against their saved evaluation artifacts, and final inclusion and presentation decisions remain with Universal Computing Research.
If you have results for a model that fits the benchmark scope, open a discussion on this Space for consideration.
Credits
Inspired by the Open LLM Leaderboard, Axiomic Labs Open SLM Leaderboard, and OpenAI Parameter Golf.