README.md

---
license: mit
tags:
- text-generation
- causal-lm
- grpo
- reasoning
- reinforcement-learning
- mini-llm
- text-generation-inference
- lm-evaluation-harness
datasets:
- openai/gsm8k
- openai/openai_humaneval
- cais/mmlu
- allenai/ai2_arc
- bespokelabs/Bespoke-Stratos-17k
language:
- en
pipeline_tag: text-generation
metrics:
- accuracy
- exact_match
model-index:
- name: Atomight-V2.1-0.5B-Inference
  results:
  - task:
      type: text-generation
      name: ARC-Challenge
    dataset:
      name: AI2 ARC-Challenge
      type: allenai/ai2_arc
      config: ARC-Challenge
      split: test
    metrics:
    - name: Accuracy
      type: acc
      value: 0.30802
    - name: Accuracy (Norm)
      type: acc_norm
      value: 0.33788
  - task:
      type: text-generation
      name: GSM8K
    dataset:
      name: GSM8K
      type: openai/gsm8k
      config: main
      split: test
    metrics:
    - name: Exact Match (Strict)
      type: exact_match
      value: 0.19788
    - name: Exact Match (Flexible)
      type: exact_match
      value: 0.32449
  - task:
      type: text-generation
      name: ARC-Easy
    dataset:
      name: AI2 ARC-Easy
      type: allenai/ai2_arc
      config: ARC-Easy
      split: test
    metrics:
    - name: Accuracy
      type: acc
      value: 0.65446
    - name: Accuracy (Norm)
      type: acc_norm
      value: 0.59343
  - task:
      type: text-generation
      name: HellaSwag
    dataset:
      name: HellaSwag
      type: Rowan/hellaswag
      split: validation
    metrics:
    - name: Accuracy
      type: acc
      value: 0.40709
    - name: Accuracy (Norm)
      type: acc_norm
      value: 0.5235
---

# Atomight-V2.1-0.5B-Inference

<p align="center">
  <img src="OfficialAtomight.png" alt="Atomight Logo" width="500" style="max-width: 100%;">
</p>

**Atomight-V2.1-0.5B-Inference** is an ultra-compact, reasoning-oriented causal language model developed under the **[Atomight Ecosystem](https://huggingface.co/NovatasticRoScript/collections)**. Built on a Qwen-derived 494M parameter foundation, the model has been refined using **GRPO (Group Relative Policy Optimization)** reinforcement tuning. 

Despite its tiny physical footprint, Atomight-V2.1-0.5B targets highly efficient edge-device reasoning, structured text outputs, lightweight coding assistance, and rapid deployment workflows under severe compute constraints.

### 🚀 Key Highlights
- **Parameter Footprint:** ~494M parameters (Loads into ~1GB VRAM at FP16).
- **Training Paradigm:** GRPO reinforcement learning focusing on high-signal reasoning vectors instead of brute-force dataset scale.
- **Edge-Optimized:** Designed specifically for low-overhead mobile, local, and browser-based inference loops (Google Colab / Kaggle native workflow).

Weighted/Imatrix Quants, and Static Quants by [mradermacher](https://huggingface.co/mradermacher) are available at: 
https://huggingface.co/mradermacher/Atomight-V2.1-0.5B-Inference-i1-GGUF
https://huggingface.co/mradermacher/Atomight-V2.1-0.5B-Inference-GGUF

**NOTICE: Think tags (chain-of-thought reasoning) may not be working now due to some errors. But rest assured, the initial model is actively working & SAFE. We'll try our best to fix this as soon as possible.** 

---

## 📊 Evaluation & Benchmark Results

Official evaluations were conducted using the **EleutherAI LM Evaluation Harness** at FP16 precision. 

### Core Evaluation Metrics
| Benchmark Task | Metric Typology | Atomight-V2.1-0.5B Score | Focus Domain |
| :--- | :--- | :--- | :--- |
| **ARC-Easy** | Accuracy (Normalized) | **59.34%** | Scientific Fact Retrieval |
| **HellaSwag** | Accuracy (Normalized) | **52.35%** | Commonsense Reasoning & Next-Sentence Prediction |
| **ARC-Challenge** | Accuracy (Normalized) | **33.79%** | Hard Analytical Exclusion Logic |
| **GSM8K (Flexible Extract)** | Exact Match (Regex Clean) | **32.45%** | Mathematical Thought & Resolution |
| **GSM8K (Strict)** | Exact Match (Rigid Parse) | **19.79%** | Formatted Mathematical Output |

<p align="center">
  <img src="OfficialBenchmarkAtomight2.1.png" alt="Atomight V2.1 Benchmark" width="500" style="max-width: 100%;">
</p>

### 🔍 Comparative Engineering Insights

* **Punching Above Weight Classes:** Atomight-V2.1-0.5B outpaces Meta's larger **Llama-3.2-1B-Instruct** on localized logic-retrieval metrics, clearing **59.3%** on ARC-Easy and **33.8%** on ARC-Challenge compared to Llama's *56.7%* and *31.8%* respectively.
* **The Reasoning Gap:** On mathematical reasoning (GSM8K), when evaluated with **Flexible Extraction parsing (32.45%)**, Atomight demonstrates higher raw mathematical accuracy than both Qwen2.5-0.5B-Instruct (*26.8%*) and Llama-3.2-1B-Instruct (*24.4%*). 
* **The Formatting Note:** The delta between Atomight's Strict Math score (19.8%) and Flexible Math score (32.5%) stems from the internal reasoning tokens generated during the inference step. While the mathematical conclusion is correct nearly 1/3 of the time, the model frequently bypasses rigid formatting constraints in favor of dense thinking traces.

---

## 💻 Quickstart: Inference Execution

Atomight utilizes system and sequence prompts to partition thinking spaces. For optimal reasoning convergence, use explicit `<thinking>` and `<answer>` encapsulation layers.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "NovatasticRoScript/Atomight-V2.1-0.5B-Inference"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    device_map="auto"
)

# Structuring system guidelines for GRPO activation
messages = [
    {
        "role": "system", 
        "content": "You are a reasoning model. Think inside <thinking> and answer inside <answer>."
    },
    {
        "role": "user", 
        "content": "A farmer has 12 apples. He gives 4 to his neighbor and loses 2 on the way home. How many apples does he have left?"
    }
]

inputs = tokenizer.apply_chat_template(
    messages, 
    tokenize=True, 
    add_generation_prompt=True, 
    return_tensors="pt"
).to("cuda")

with torch.no_grad():
    outputs = model.generate(
        inputs, 
        max_new_tokens=250, 
        temperature=0.01,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

## 📄 Citations

@misc{shao2024deepseekmathpushinglimitsmathematical,
      title={DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}, 
      author={Zhihong Shao and Peiyi Wang and Qihao Zhu and Junxiao Song and Daya Guo and et al.},
      year={2024},
      eprint={2402.03300},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2402.03300}
}

@misc{qwen_qwen25_2025,
      title={Qwen2.5 Technical Report}, 
      author={Qwen and Yang, An and Yang, Baosong and Zhang, Beichen and Hui, Binyuan and et al.},
      year={2024},
      eprint={2412.15115},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.15115}
}

@misc{eval-harness,
      author={Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and et al.},
      title={A framework for few-shot language model evaluation},
      month={12},
      year={2023},
      publisher={Zenodo},
      version={v0.4.0},
      doi={10.5281/zenodo.10257521},
      url={https://github.com/EleutherAI/lm-evaluation-harness}
}

@article{allenai:arc,
      author    = {Peter Clark  and Isaac Cowhey and Oren Etzioni and Tushar Khot and
                    Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord},
      title     = {Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge},
      journal   = {arXiv:1803.05457v1},
      year      = {2018},
}

@article{cobbe2021gsm8k,
  title={Training Verifiers to Solve Math Word Problems},
  author={Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and Hesse, Christopher and Schulman, John},
  journal={arXiv preprint arXiv:2110.14168},
  year={2021}
}