hmellor/tiny-random-Gemma2ForCausalLM
Text Generation • 8.44M • Updated • 436k • 1
Syafiq Kamarul Azman
syaffers
AI & ML interests
Inference and model servers
Recent Activity
liked a model 7 days ago
hmellor/tiny-random-Gemma2ForCausalLM liked a model 13 days ago
Efficient-Large-Model/Sana_600M_512px liked a model about 1 month ago
tiny-random/qwen2.5-omniOrganizations
None yet
liked a
model 7 days ago
liked a
model 13 days ago
liked 2
models about 1 month ago
reacted to Banaxi-Tech's post with 🔥 about 1 month ago
Post
2758
Today we wanted to release BananaMind 2 Pico, our smallest model yet at ~0.9M parameters. Instead, we accidentally ran a very expensive experiment on what happens when you push a tiny model way past its useful token budget.
Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through.
The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting.
Follow us for more:
BananaMind
@vovaRL
@Banaxi-Tech
Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd
And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP
Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through.
The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting.
Follow us for more:
@vovaRL
@Banaxi-Tech
Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd
And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP
- 3 replies
liked a
model about 1 month ago
reacted to appvoid's post with 🔥 about 2 months ago
Post
1357
Giving free early access to the gguf for some of you today! Tell me what you think.
https://huggingface.co/CEAMFA/palmer-007-preview
https://huggingface.co/CEAMFA/palmer-007-preview
updated a
model about 2 months ago
upvoted a collection about 2 months ago
liked a
model about 2 months ago
published a
model about 2 months ago
updated a
model about 2 months ago
published a
model about 2 months ago
reacted to ManniX-ITA's post with 🚀 2 months ago
Post
2061
🚀 New release: Qwen3.6-27B-A3B-Coder
A code-specialized MoE carved out of Qwen3.6-35B-A3B by pure expert pruning — no fine-tuning, no distillation. I profiled all 256 experts on balanced corpora plus targeted code benchmarks (LiveCodeBench +
MultiPL-E), built a competence map with the code classes up-weighted 1.5×, and dropped the 72 weakest experts per layer (256→184, ~35B→27B). Router, attention, norms, the MTP head and the vision tower are all
preserved; active params stay at A3B and routing is baked to top-10 (revert to top-8 anytime).
📊 Benchmarks (Q6_K, temp 0.6):
• MultiPL-E 0.840
• HumanEval 0.970
• LiveCodeBench 0.688
• GSM8K 0.970 · ARC-C 0.944 · AIME 0.733
• GPQA-Diamond 0.773 · MATH-500 0.620 · IFEval 0.730
• Average 0.808
27B footprint, A3B speed, coding that punches well above its size — and the preserved MTP head gives you speculative decoding out of the box (text + vision).
🔗 Model: ManniX-ITA/Qwen3.6-27B-A3B-Coder
📦 GGUF (+MTP): ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF
🦙 Ollama: https://ollama.com/mannix/qwen3.6-27b-a3b-coder
A code-specialized MoE carved out of Qwen3.6-35B-A3B by pure expert pruning — no fine-tuning, no distillation. I profiled all 256 experts on balanced corpora plus targeted code benchmarks (LiveCodeBench +
MultiPL-E), built a competence map with the code classes up-weighted 1.5×, and dropped the 72 weakest experts per layer (256→184, ~35B→27B). Router, attention, norms, the MTP head and the vision tower are all
preserved; active params stay at A3B and routing is baked to top-10 (revert to top-8 anytime).
📊 Benchmarks (Q6_K, temp 0.6):
• MultiPL-E 0.840
• HumanEval 0.970
• LiveCodeBench 0.688
• GSM8K 0.970 · ARC-C 0.944 · AIME 0.733
• GPQA-Diamond 0.773 · MATH-500 0.620 · IFEval 0.730
• Average 0.808
27B footprint, A3B speed, coding that punches well above its size — and the preserved MTP head gives you speculative decoding out of the box (text + vision).
🔗 Model: ManniX-ITA/Qwen3.6-27B-A3B-Coder
📦 GGUF (+MTP): ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF
🦙 Ollama: https://ollama.com/mannix/qwen3.6-27b-a3b-coder
- 13 replies
updated a
model 4 months ago
published a
model 4 months ago
updated a
model 4 months ago
published a
model 4 months ago
liked a
model 8 months ago
updated a
model 9 months ago