appvoid/void-byte-data
Updated • 4.97k
🤏 quecto mode
appvoid
appvoid
P(doom) 10%
·
AI & ML interests
singularity through byte-level tokens and small language models - apps and models on dotslab
Recent Activity
new activity about 7 hours ago
bench-labs/cagliostro-v3.5:Congrats! updated a dataset about 22 hours ago
appvoid/void-byte-data liked a model 1 day ago
bench-labs/cagliostro-v3.5Organizations
Congrats!
❤️ 1
4
#3 opened 1 day ago
by appvoid
updated a
dataset about 22 hours ago
liked 2
models 1 day ago
updated a
collection 2 days ago
posted an update 2 days ago
Post
134
byte-level, deep networks, flow-matching, matryoshka...
yo yo yo
3
#5 opened 3 days ago
by 
arthu1
updated a
dataset 4 days ago
updated a
Space 5 days ago
published a
dataset 5 days ago
posted an update 5 days ago
Post
156
Much of the misunderstanding on loops and byte-level models comes from fear on inventing the wheel, not reinventing it
posted an update 6 days ago
Post
141
How much should OpenAI offer for a name like Dot Labs?
- 1 reply
Only PR's will be added for the next little while
👍 6
5
#162 opened 13 days ago
by 
Datdanboi25
liked a
model 9 days ago
published a
dataset 10 days ago
posted an update 10 days ago
Post
167
Reality has not limits if you know how to shape it, one step at a time.
- 1 reply
reacted to TobiasLogic's post with 🚀 10 days ago
Post
3540
We’ve been cooking something new at Bench Labs.
Introducing Cagliostro-v3, our new 146M parameter language model trained completely from scratch.
The run isn’t even finished yet.
At the current checkpoint:
• 146M parameters
• 72.7B / 75B tokens trained
• 26.27 Open SLM Index
• 43.80 ArithMark-3
• Trained on a single RTX 5090
• ~90K to 103K tokens/sec during training
• ~9 days for the full run
• Apache 2.0
For some context, SmolLM2-135M scores 27.13 on the same Index after being trained on roughly 2 trillion tokens.
Cagliostro-v3 is currently at 26.27 with only ~72.7B.
That’s around 27x fewer training tokens.
The model also currently Hold the number 3rd spot for ArithMark-3, scoring 43.80
This wasn’t achieved by just throwing more tokens at the model. A huge part of v3 has been figuring out architecture, data mixture, and training dynamics at this scale.
The model uses a custom 30-layer decoder architecture with grouped-query attention and cross-head subspace attenuation, SwiGLU, RMSNorm, RoPE, tied embeddings, and a warmup-stable-decay training schedule.
During cooldown we also substantially shifted the data mixture toward higher-quality synthetic textbook and mathematics data, with the mathematics share increasing from 10% to 28%.
And everything is open.
The repository contains the training history with checkpoints pushed roughly every 30 minutes, so you can inspect how the model evolved throughout training rather than only seeing the final weights.
This is still a pre-final checkpoint. We have roughly 2.3B tokens left and the learning-rate cooldown is still running.
So 26.27 isn’t the final number.
Really excited to see where the last part of the run lands.
Cagliostro-v3:
bench-labs/cagliostro-v3
Built by Bench Labs.
Open SLM Leaderboard:
AxiomicLabs/Open_SLM_Leaderboard
Introducing Cagliostro-v3, our new 146M parameter language model trained completely from scratch.
The run isn’t even finished yet.
At the current checkpoint:
• 146M parameters
• 72.7B / 75B tokens trained
• 26.27 Open SLM Index
• 43.80 ArithMark-3
• Trained on a single RTX 5090
• ~90K to 103K tokens/sec during training
• ~9 days for the full run
• Apache 2.0
For some context, SmolLM2-135M scores 27.13 on the same Index after being trained on roughly 2 trillion tokens.
Cagliostro-v3 is currently at 26.27 with only ~72.7B.
That’s around 27x fewer training tokens.
The model also currently Hold the number 3rd spot for ArithMark-3, scoring 43.80
This wasn’t achieved by just throwing more tokens at the model. A huge part of v3 has been figuring out architecture, data mixture, and training dynamics at this scale.
The model uses a custom 30-layer decoder architecture with grouped-query attention and cross-head subspace attenuation, SwiGLU, RMSNorm, RoPE, tied embeddings, and a warmup-stable-decay training schedule.
During cooldown we also substantially shifted the data mixture toward higher-quality synthetic textbook and mathematics data, with the mathematics share increasing from 10% to 28%.
And everything is open.
The repository contains the training history with checkpoints pushed roughly every 30 minutes, so you can inspect how the model evolved throughout training rather than only seeing the final weights.
This is still a pre-final checkpoint. We have roughly 2.3B tokens left and the learning-rate cooldown is still running.
So 26.27 isn’t the final number.
Really excited to see where the last part of the run lands.
Cagliostro-v3:
bench-labs/cagliostro-v3
Built by Bench Labs.
Open SLM Leaderboard:
AxiomicLabs/Open_SLM_Leaderboard
- 8 replies
updated a
dataset 10 days ago
liked a
Space 11 days ago
updated a
model 11 days ago