Flilax/HRTByteModel-Base

HRT-v7 (Hierarchical Radial Transformer) — 148M Base

HRT (Hierarchical Radial Transformer) is an experimental, tokenizer-free, sub-quadratic neural network architecture designed for ultra-long context modeling (up to 128k context) with extreme memory efficiency on consumer-grade hardware.

This checkpoint is the Base Pretrained Model (~148M parameters) trained on ~2.5 billion raw UTF-8 bytes using a multi-stream balanced curriculum of web text, synthetic textbooks, code, and mathematics.

---

⚡ Key Highlights

• Native Byte-Level Processing: Operates directly on raw UTF-8 bytes (vocab_size = 257: 0–255 bytes + EOS 256). No BPE/WordPiece tokenizers.
• 128k Native Context: Pretrained directly with sequence length T = 131,072 bytes.
• Sub-Quadratic Scaling (T → K → K → T): Replaces dense O(T²) self-attention with radial latent compression (K = 512 outer latents, 16 center latents), yielding linear/sub-quadratic compute and memory footprint.
• Deep Equilibrium Core (JFB): Features an implicit reasoning core trained with Jacobi-Free Backpropagation (JFB) for constant O(1) backpropagation memory overhead.
• Hardware-Aware Quantization: Built-in TurboQuantizer using random orthogonal QR rotation matrices to suppress outliers during INT8 KV caching.

---

📊 Training Details & Hyperparameters

The model was pretrained on NVIDIA Blackwell / Hopper hardware in bfloat16 mixed precision.

Model Configuration (ModelConfig)

Hyperparameter	Value	Description
Parameters	~148M	Active trainable weights
Context Length (seq_len)	131,072 (128k bytes)	Native training window length
Vocab Size	257	256 bytes + 1 EOS token
Model Dimension (d_model)	768	Hidden representation size
FFN Dimension (d_ff)	3072	SwiGLU projection dimension
Outer Latents (K)	512	Compressed outer working memory
Center Latents	16	Deep reasoning core latents
Routing Top-K	64	Sparse attention routing threshold
Attention Heads	12	Outer, Inner, and Latent heads
Outer / Inner Cycles	6 / 8	Hierarchical gather/reasoning loops
Local Conv Kernel	7	Causal dual-dilated local token mixer
Positional Bias	HARP	Hierarchical Adaptive Relational Positioning
Implicit Core	Enabled	3 internalization steps with JFB
Auxiliary Losses	Enabled	SVD-based low-rank compaction loss + JFB loss

Pretraining Optimization & Setup

• Optimizer: AdamW (beta1 = 0.9, beta2 = 0.95, weight_decay = 0.01)
• Precision: Native bfloat16 with Gradient Checkpointing
• Effective Batch Size: 8 sequences × 131,072 = 1,048,576 bytes per step (Batch Size 4 × Gradient Accumulation 2)
• Gradient Clipping: 1.0
• Learning Rate Schedule: Cosine Annealing cooldown down to LR_min = 1e-5

---

Dataset Mixture

The model was trained on a dynamically balanced 5-stream multiplexer with streaming packing:

FastMultiStream Distribution:
25% — HuggingFaceTB/smollm-corpus (Cosmopedia v2 - synthetic textbooks & stories)
25% — HuggingFaceFW/fineweb (sample-10BT - general web crawl)
20% — HuggingFaceFW/fineweb-edu (sample-10BT - educational & academic web text)
20% — bigcode/starcoderdata (Python subsets - structured code & AST logic)
10% — HuggingFaceTB/finemath (finemath-3plus - math reasoning & LaTeX)

Documents were packed into continuous 131,072-byte buffers with unique segment IDs (segment_ids) and EOS separators to ensure segment-aware causal attention boundaries.

---

Quickstart / How to Use

1. Installation

Clone the repository and install the HRT package:

git clone https://github.com/5bridge/HRT.git
cd HRT
pip install -e .

2. Running Inference (Next-Byte Completion)

Because this is a Base Pretrained Model (not an instruction/chat tuned model), it functions as an autoregressive text/code completion engine.

import torch
import torch.nn.functional as F
from hrt import ModelConfig, HierarchicalRadialTransformerV7

device = "cuda" if torch.cuda.is_available() else "cpu"

# 1. Initialize Configuration matching training
cfg = ModelConfig(
    d_model=768,
    d_ff=3072,
    n_outer_latents=512,
    n_outer_cycles=6,
    n_inner_cycles=8,
    n_center_latents=16,
    routing_k=64,
    n_outer_heads=12,
    n_inner_heads=12,
    n_latent_heads=12,
    vocab_size=257,
    max_seq_len=131072,
    use_qk_norm=True,
    use_rezero=True,
    use_compaction=True,
    use_internalization=True,
    use_jfb=True,
    use_q_cache=True,
)

# 2. Load model & weights
model = HierarchicalRadialTransformerV7(cfg).to(device)
weights = torch.load("hrt_v7_148m_weights.pt", map_location=device)
model.load_state_dict(weights["model"] if "model" in weights else weights)
model.eval()

# 3. Autoregressive Byte-level Generation
def generate(prompt: str, max_new_bytes: int = 120, temp: float = 0.5, top_k: int = 5):
    prompt_bytes = list(prompt.encode("utf-8"))
    prompt_ids = torch.tensor([prompt_bytes], dtype=torch.long, device=device)
    
    with torch.no_grad():
        prompt_emb = model.tok_emb(prompt_ids)
        logits, cache = model._init_generation_cache(prompt_emb)
        out_bytes = list(prompt_bytes)

        for _ in range(max_new_bytes):
            l = logits / max(temp, 1e-5)
            if top_k > 0:
                v, _ = torch.topk(l, min(top_k, l.size(-1)))
                l[l < v[:, [-1]]] = float("-inf")
            
            nxt = torch.multinomial(F.softmax(l, dim=-1), num_samples=1)
            nxt_id = nxt.item()
            if nxt_id == 256:  # EOS
                break
                
            out_bytes.append(nxt_id)
            nxt_emb = model.tok_emb(nxt)
            logits = model.step_generation(nxt_emb, cache)
            
    return bytes(out_bytes).decode("utf-8", errors="replace")

# Test completion
print(generate("def", max_new_bytes=100))

---

⚠️ Limitations & Intended Use

• Base Model Nature: This model has not undergone Supervised Fine-Tuning (SFT) or RLHF/DPO. It will not behave as a conversational assistant by default and may loop if prompted with chat-like templates without a stopping token.
• Proof-of-Concept Scale: Trained on ~2.5B bytes as a compute-limited validation run. It exhibits strong syntactic comprehension (LaTeX, Python, Markdown, JSON AST schemas), but requires further scale for complex factual recall and deep semantic reasoning.
• Information Bottleneck: The T to K compression naturally trades off lossless, needle-in-a-haystack memorization for bounded sub-quadratic memory footprint.

---

License

This model and its code are released under the Apache 2.0 License.

https://github.com/5bridge/HRT