smoll-swarm/SmolCompressor-360M

SmolCompressor 360M

This is a 360M parameter causal language model fine-tuned for context compression and RAG pruning. It is part of the Smol Swarm project.

Overview

If you pass large document chunks, meeting transcripts, or noisy search results directly into a primary reasoning model, it usually slows down responses, wastes tokens, and dilutes the context.

SmolCompressor is designed to sit in the middle of your pipeline as a lightweight middleware. It takes verbose source text and shrinks it by about 70 to 80 percent. The goal is to keep all the important facts, metrics, dates, names, error codes, and key decisions, and output them in a structured JSON format. Because it is only 360M parameters, it runs easily on a local CPU.

How it differs from SmolExtractor

I sometimes get asked how this compares to SmolExtractor 1.7B.

SmolCompressor is basically a summarizer and context pruner. It outputs a dense natural language paragraph along with a list of entities. It does not need a predefined schema and works on pretty much any document. It is meant to feed into downstream LLMs or RAG pipelines.

SmolExtractor is a JSON specialist. It maps entities into rigid database tables with typed fields like integers, booleans, and floats. It requires a target schema and is meant for SQL databases, webhooks, or backend APIs.

Quick Start

Here is how you use it with the Transformers library.

import json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "smoll-swarm/SmolCompressor-360M"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
    device_map="auto"
)

SYSTEM_PROMPT = """You are SmolCompressor, an ultra-dense context compression engine.
Your job is to compress long, noisy documents, transcripts, and RAG context chunks by 70-80% while preserving 100% of facts, names, dates, metrics, and key decisions.
You must respond ONLY with a valid JSON object matching this schema:
{"compressed_context": "dense factual summary", "key_entities": ["list", "of", "entities"], "reduction": "percentage"}"""

raw_text = """Retrieved Document Chunk #102: In Q3 2024, CloudCorp reported total revenue of $84.2M, 
representing a 28% year-over-year increase. Operating margins expanded by 240 basis points to 22%. 
The cloud infrastructure division led by VP Sarah Connor accounted for $34M of total bookings. 
The company reaffirmed FY2025 guidance of $250M in annual recurring revenue."""

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": f"Compress and prune this context:\n\"\"\"{raw_text}\"\"\""}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=150,
        do_sample=False,  
        pad_token_id=tokenizer.eos_token_id
    )

output_ids = outputs[0][inputs.input_ids.shape[1]:]
result_json = json.loads(tokenizer.decode(output_ids, skip_special_tokens=True).strip())

print(json.dumps(result_json, indent=2))

Output Schema

The model is prompted to return a JSON object with three keys: the compressed text, a list of key entities, and the estimated reduction percentage.

{
  "compressed_context": "CloudCorp Q3 2024 revenue: $84.2M (+28% YoY). Operating margin: 22%. Cloud division led by Sarah Connor generated $34M. FY2025 ARR guidance reaffirmed at $250M.",
  "key_entities": [
    "CloudCorp",
    "Q3 2024",
    "Sarah Connor",
    "FY2025 ARR"
  ],
  "reduction": "78%"
}

Pipeline Architecture

In a local multi-agent setup, a router model like SmolRouter 135M classifies the intent and sends summarization requests to SmolCompressor. The compressed context is then passed to the main LLM or RAG system. This keeps the token count low for the heavier models.

Examples

Financial Earnings Call: Input: In Q3 2024, CloudCorp reported total revenue of $84.2M, representing a 28% year-over-year increase. Operating margins expanded by 240 basis points to 22%. The cloud infrastructure division led by VP Sarah Connor accounted for $34M of total bookings. The company reaffirmed FY2025 guidance of $250M in annual recurring revenue. Output: {"compressed_context": "CloudCorp Q3 2024 revenue: $84.2M (+28% YoY). Operating margin: 22%. Cloud division led by Sarah Connor generated $34M. FY2025 ARR reaffirmed at $250M.", "key_entities": ["CloudCorp", "Q3 2024", "Sarah Connor", "FY2025 ARR"], "reduction": "78%"}

Engineering Meeting Notes: Input: Alex discussed the ongoing database migration from Postgres 14 to Postgres 16. Timeline was delayed by 2 weeks due to index lock contention on table 'audit_logs'. Maria agreed to allocate 3 senior backend engineers to assist with schema partitioning... Output: {"compressed_context": "Postgres 14->16 migration delayed 2 weeks due to index locking on audit_logs. Maria allocating 3 engineers for schema partitioning.", "key_entities": ["Postgres 16", "Maria", "Alex", "audit_logs"], "reduction": "73%"}

Training Specs

I trained this on 2,500 domain-balanced context compression pairs. The dataset includes RAG retrieval chunks, incident post-mortems, meeting notes, and customer support logs.

During training, prompt tokens were masked using -100, so the gradient loss was calculated exclusively on the target compression JSON payload. Training was done on an NVIDIA RTX PRO 6000 Blackwell Server Edition (95 GB VRAM) using full Bfloat16 precision with TF32 Tensor Core optimizations.

Model Footprint

Bfloat16 / FP16: ~720 MB file size, ~800 MB memory. Best for PyTorch and Python backends. Q8_0 GGUF: ~380 MB file size, ~450 MB memory. Good for high-precision local CPU inference. Q4_K_M GGUF: ~280 MB file size, ~320 MB memory. Good for edge devices, laptops, mobile, and Ollama.

The Smol Swarm Ecosystem

This model is part of a larger suite of small, specialized models I am working on:

SmolRouter 135M: Intent routing. SmolExtractor 1.7B: Typed JSON extraction. SmolCompressor 360M: Context pruning (this model). SmolPlanner 1.7B: Task decomposition (in progress). SmolCoder 1.7B: Utility script generation (in progress).