Harness Leaderboard
🧭
4
Which coding-agent harness should you use for your model?
🤏 Small Decision Models soon!
OpenEngineer
Monster-Code
AI & ML interests
I like AI
Recent Activity
liked a Space about 12 hours ago
joelniklaus/harness-leaderboard liked a model about 13 hours ago
MiniMaxAI/MiniMax-M3.1-Flash liked a Space about 13 hours ago
hugging-apps/tencent-hy-nomasOrganizations
liked a
Space about 12 hours ago
liked a
model about 13 hours ago
liked a
Space about 13 hours ago
liked a
model about 13 hours ago
published a
bucket about 17 hours ago
liked a
dataset about 17 hours ago
updated a
dataset about 17 hours ago
liked a
model about 17 hours ago
upvoted a changelog 3 days ago
Hugging Face Changelog
3D Previews for OpenUSD & SimReady Datasets
• 50
liked a
model 3 days ago
liked a
model 7 days ago
reacted to Banaxi-Tech's post with 😔 8 days ago
Post
203
bci is the future.
and i dont like that
and i dont like that
- 4 replies
reacted to Mati83moni's post with ❤️ 8 days ago
Post
4,989
Is "evaluation awareness" in a language model one direction, or eight?
We held the task fixed, toggled the eight cue factors of EvalAwareBench one at a time, and looked at the residual stream of Qwen3 (0.6B–32B), Gemma 3 and PLLuM-12B, plus a first Polish cross-lingual pilot.
What we found (descriptive, same instrument and author throughout):
- Cue directions overlap at about 1/5–1/4 of what two estimates of the same cue reach: not one shared axis.
- The standard all-cues direction is mostly an explicit-disclosure detector (cosine 0.81–0.89 with that cue).
- It transfers poorly to stimulus artificiality (held-out AUROC ≈ 0.5–0.6), though the models do represent that cue.
- No statistically detectable MMLU framing effect in the tested settings. That does not establish absence of an effect or of sandbagging.
- The Polish pilot is inconclusive: the formal verdict says so, and its own length control was not met.
The article carries its corrections (three early findings were withdrawn at scale; one traced to a bug in our hook code), an open validation problem for the add interventions, and all 23 figures with sources.
Full article: https://huggingface.co/blog/Mati83moni/is-evaluation-awareness-one-direction-or-eight
Spaces:
Mati83moni/evaluation-awareness-one-direction-or-eight
Mati83moni/evaluation-awareness-pllum-12b
We held the task fixed, toggled the eight cue factors of EvalAwareBench one at a time, and looked at the residual stream of Qwen3 (0.6B–32B), Gemma 3 and PLLuM-12B, plus a first Polish cross-lingual pilot.
What we found (descriptive, same instrument and author throughout):
- Cue directions overlap at about 1/5–1/4 of what two estimates of the same cue reach: not one shared axis.
- The standard all-cues direction is mostly an explicit-disclosure detector (cosine 0.81–0.89 with that cue).
- It transfers poorly to stimulus artificiality (held-out AUROC ≈ 0.5–0.6), though the models do represent that cue.
- No statistically detectable MMLU framing effect in the tested settings. That does not establish absence of an effect or of sandbagging.
- The Polish pilot is inconclusive: the formal verdict says so, and its own length control was not met.
The article carries its corrections (three early findings were withdrawn at scale; one traced to a bug in our hook code), an open validation problem for the add interventions, and all 23 figures with sources.
Full article: https://huggingface.co/blog/Mati83moni/is-evaluation-awareness-one-direction-or-eight
Spaces:
Mati83moni/evaluation-awareness-one-direction-or-eight
Mati83moni/evaluation-awareness-pllum-12b
- 2 replies
liked 2
models 8 days ago
upvoted a collection 8 days ago
liked 3
models 8 days ago
reacted to DedeProGames's post with 🤯 10 days ago
Post
8,872
🚀 Introducing Kiyo-230M-Instruct — my compact English & Portuguese chat model!
Built on Kiyo-230M-Preview, a base model I pretrained from scratch, Kiyo now has an instruction-tuned release for everyday conversations, rewriting, and lightweight assistant experiments.
🧠 229.7M parameters
🌍 English + Portuguese
💬 Native chat template for multi-turn conversations
📏 2,048-token context window
📦 459 MB of BF16 safetensors weights
🔓 Apache-2.0 license
For instruction tuning, I used 165,735 conversations from SmolTalk2, smol-smoltalk, UltraChat, and Dolci-Instruct-SFT. Training ran for two complete epochs with assistant-only loss, using a roughly 70% English / 30% Portuguese token mix.
The updated model card includes benchmarks against SmolLM2-135M-Instruct, BananaMind, and Supra2, plus example conversations, CPU measurements, training details, and a Transformers quickstart.
The results are mixed: Kiyo is competitive with these small models across the reported evaluations, while SmolLM2 and BananaMind score higher on the instruction-following benchmarks. Portuguese was a highlight in the small conversational sanity test.
This is an experimental 230M model, with plenty of room to improve reasoning, factual accuracy, and strict instruction following. I’m excited to share it with people exploring how much we can get from small, accessible models!
🤗 Model & full README:
DedeProGames/Kiyo-230M-Instruct
💬 Try the chat demo:
DedeProGames/kiyo-230m-chat
Try it in English or Portuguese — I’d love to see your examples, feedback, and experiments!
Built on Kiyo-230M-Preview, a base model I pretrained from scratch, Kiyo now has an instruction-tuned release for everyday conversations, rewriting, and lightweight assistant experiments.
🧠 229.7M parameters
🌍 English + Portuguese
💬 Native chat template for multi-turn conversations
📏 2,048-token context window
📦 459 MB of BF16 safetensors weights
🔓 Apache-2.0 license
For instruction tuning, I used 165,735 conversations from SmolTalk2, smol-smoltalk, UltraChat, and Dolci-Instruct-SFT. Training ran for two complete epochs with assistant-only loss, using a roughly 70% English / 30% Portuguese token mix.
The updated model card includes benchmarks against SmolLM2-135M-Instruct, BananaMind, and Supra2, plus example conversations, CPU measurements, training details, and a Transformers quickstart.
The results are mixed: Kiyo is competitive with these small models across the reported evaluations, while SmolLM2 and BananaMind score higher on the instruction-following benchmarks. Portuguese was a highlight in the small conversational sanity test.
This is an experimental 230M model, with plenty of room to improve reasoning, factual accuracy, and strict instruction following. I’m excited to share it with people exploring how much we can get from small, accessible models!
🤗 Model & full README:
DedeProGames/Kiyo-230M-Instruct
💬 Try the chat demo:
DedeProGames/kiyo-230m-chat
Try it in English or Portuguese — I’d love to see your examples, feedback, and experiments!
- 9 replies