efe-T/ByteFlow-experiment-A

Standalone ByteFlow Net

ByteFlowTrainer.py is an independent, tokenizer-free model and training application derived from the architecture and equations in the ByteFlow paper. It does not modify GptTrainer.py and cannot load its 50,304-token GPT checkpoints.

The implementation includes all five published stages:

1. shallow local byte encoder with causal sliding-window attention and Canon;
2. coding-rate Top-K adaptive downsampling, with BOS always retained;
3. deep, wide causal global transformer;
4. causal predecessor-chunk upsampling through 16 position-bin projections;
5. symmetric local decoder and 258-way next-byte prediction.

The authors have not released their reference source yet. Consequently, this is explicitly labelled paper-derived-byteflow-net in every checkpoint. The linear-time L2 coding-rate approximation comes from the paper's appendix.

Commands

Inspect the small smoke-test model without downloading a dataset:

pip install "torch>=2.5" "datasets>=4.0" huggingface_hub
python ByteFlowTrainer.py inspect --preset tiny

Train the paper's approximately 600M-parameter configuration on the pinned FineWeb-Edu 100B sample:

export HF_TOKEN=hf_...          # write access to the target model repository
python ByteFlowTrainer.py train --preset 600m

By default, checkpoints and the exact streaming-dataset cursor are kept under byteflow_run/ and uploaded under byteflow/ in efe-T/ByteFlow-experiment-A. Override these with BYTEFLOW_RUN_DIR, BYTEFLOW_HF_REPO, --checkpoint-dir, or --hub-repo.

Generate directly as UTF-8 bytes from a saved checkpoint:

python ByteFlowTrainer.py generate byteflow_run/checkpoints/step-000000001000 \
  --preset 600m --prompt "Once upon a time"

Resume and no-replay contract

Before opening FineWeb-Edu, the application scans all complete local and Hub checkpoint manifests and selects the verified high-water model/data cursor. A checkpoint contains model weights, optimizer state, RNG state, the Hugging Face iterable-dataset state, and every already-fetched but unconsumed byte. Dataset shuffling and wrapping are disabled.

The defaults commit a complete checkpoint after every optimizer step and upload every step. This is deliberately expensive: full state durability at every committed cursor is the only correct way to avoid replay without losing the corresponding model update. Keep the checkpoint directory on persistent storage. SIGINT/SIGTERM finishes and commits the current step before exiting.

A power loss or SIGKILL during a GPU step can still replay that uncommitted step; no ordinary training program can atomically commit GPU computation and a remote multi-gigabyte checkpoint. Graceful restarts and all completed, published steps are strictly monotonic.