iskhare/neurips-ablations

NCP rebuttal ablations

Four ablation studies on the Next-Concept Predictor. Code, results tables and figures: github.com/HazyResearch/ncm, branch rebuttals-schedule-ablation (see results/SCHEDULE_ABLATION.md).

Layout

<family>/<variant>/<stage>/checkpoint-iter-<N>.pt
                          /training-config.yaml
                          /results.txt          (per-seed gen-PPL + entropy)
                          /launch-record.txt    (git commit, GPUs, batch, codec sha256)

family is the level ladder, variant the NCP-side change on top of it. A -50k suffix means the run used a 50k-step VQ-VAE codec; folders without it used the earlier 10k-step codecs. Never compare gen-PPL across the two codec generations — the same baseline scored 114.5 on the 10k codec and 31.8 on the 50k one. Codec quality dominates raw gen-PPL.

An NCP checkpoint cannot be used without the codec it was trained against. Each <family>/<variant>/vqvae/ holds that codec.

The studies

study	families / variants	budget
1. level schedule	linear16 squared16 log16 exp16 → baseline-50k	5k NCP steps
2. corruption strength	squared16/corruption-p{10,20,30}-50k	5k NCP steps
3. residual refiner	squared16/refiner-50k	5k NCP steps
4. number of levels	sq8 squared16 sq24 → baseline-50k	5k NCP steps

squared16/baseline-50k is the shared control for studies 2, 3 and 4. linear16/{corruption-*,refiner} are an earlier 10k-codec, 2k-step study, superseded but retained.

Headline

Raw gen-PPL is not interpretable in this setting: across runs it correlates with generation entropy at r = +0.79, so it rewards low-diversity output. All conclusions use gen-PPL at matched entropy 5.445 nats, the measured entropy of real OpenWebText text under the same metric. Reaching that target requires disabling truncation (top_k=0, top_p=1.0), so those numbers are a separate, internally-consistent comparison flagged frozen_protocol=false in the records.

• Level schedule: squared wins by ~48 points; corroborated on an independent codec generation at a different step budget.
• Corruption strength: null (≤1.3σ). Raw gen-PPL improves monotonically to −34% but entirely by trading diversity.
• Residual refiner: null (0.1σ).
• Number of levels: 16 matches 8, beats 24 by 69 points. The 24-level deficit is unattributed between level count and convergence — see the caveats in SCHEDULE_ABLATION.md.

Known gap

exp16/baseline/vqvae, log16/baseline/vqvae and squared16/baseline/vqvae have no training-config.yaml. Those are the 10k-step codecs, and the config files on disk were later edited in place to 50k, so uploading them now would put a stop_after_iters: 50000 config beside a 10k checkpoint. Recover the originals from git history instead: the 10k-era configs are in config/ablation_sweep/vqvae-*.yaml prior to commit 45f34d1.