ConSDL checkpoints

Model checkpoints for the vision and language experiments in ConSDL: Consistent Sparse Dictionary Learning. This collection includes ConSDL and six SAE baselines across four backbones, three dictionary sizes, and three sparsity levels, with three training seeds per configuration.

Layout

<modality>/<backbone>_layer<L>/<arch>/x<E>/<level>/seed<S>/
    config.json
    model.safetensors
    stats.<dataset>.split_<split>.seed_no/     per-latent firing statistics
<modality>/<backbone>_layer<L>/<arch>/x<E>/<level>/
    dict_metrics_step_final.json               cross-seed dictionary metrics (DCS, #dead, #dup)
    code_metrics_step_final.json               cross-seed code metrics (MCC, R²)

field	values
modality / backbone	vision/clip-vit-b-16_layer10, vision/dinov2-b_layer10, lm/pythia-160m_layer8, lm/gpt2-small_layer8
arch	consdl, relu, topk, batchtopk, matryoshka, mp, isae
x<E>	dictionary size = E × 768: x4, x8, x16
level	target L0: l0_32, l0_64, l0_128
seed	seed0, seed1, seed2

Vision SAEs are trained on ImageNet patch tokens (one epoch), language SAEs on 500M FineWeb tokens (residual stream after the given block). The exact hyper-parameters of every checkpoint are in its config.json; the per-cell values are also listed in the paper's appendix.

Usage

from consdl.io.hub import load_pretrained

sae = load_pretrained("vision/clip-vit-b-16_layer10/consdl/x16/l0_64/seed0")
codes = sae.encode(activations)          # (..., 768) -> (..., 12288)
recon = sae.decode(codes)

To recompute the cross-seed metrics of a cell, download it and run the metric CLIs on it:

from consdl.io.hub import download
cell = download("vision/clip-vit-b-16_layer10/consdl/x16/l0_64")

python -m consdl.cli.dict_metrics --tag <cell>