APEX-MoE calibration priors

Calibration files for APEX-MoE, prompt-based expert pruning for mixture-of-experts language models: at the end of prefill, each request keeps a fixed share of the routed experts in every MoE layer, and decode routes among those only. These files hold the per-expert statistics the selection score needs, fitted once per model on a calibration set.

folder	model	MoE layers x experts
qwen3.6-35b-a3b	Qwen/Qwen3.6-35B-A3B	40 x 256
qwen3.8-flash-next-fp8	Qwen/Qwen3.8-Flash-Next-FP8	48 x 512
deepseek-v4.1-flash	deepseek-ai/DeepSeek-V4.1-Flash	40 x 384

Each folder holds, for MoE layer l and routed expert e:

• gain.npy [L, E]: the expert's gain, its router-weighted mean output norm on the calibration set, sum g * ||f(x)|| / sum g;
• prior.npy [L, E]: the expert's share of the summed REAP saliency g * ||f(x)|| on the calibration set (a static prior);
• transfer.npz: the transfer prior, a per-layer ridge regression of a request's decode profile on its prompt's routing: matrix [L, E, E], offset [L, E], lam (ridge strength), power (2: fitted on the square root of the decode share);
• meta.json: model revision, calibration runs, fit settings.

Selection score

Per request and layer, with m the prompt's summed router weight per expert, c its route count per expert, n its token count and share(v) = v / sum(v):

prior = share(max(0, share(c) @ matrix[l] + offset[l]) ** power)
score = n * share(gain[l] ** 2 * m) + 256 * prior

The round(keep * E) highest-scoring experts are kept (ties: lowest expert id); decode masks the other router logits before the top-k. Settings used with these files: gain power 2, prior strength 256, keep 25% and 50%.

Calibration

1024 prompts (calmix1024, seed 0): 256 each from Open-Platypus, OpenThoughts-114k, evol-codealpaca-v1 and FineWeb-Edu passages, decontaminated against the evaluated benchmarks. Each model answered every prompt itself with thinking on; a teacher-forced replay of prompt and answer recorded the routed experts' router weights and output norms. The Qwen3.6 files were fitted on the BF16 checkpoint.

v3 priors (v3/)

Refit priors for Qwen3.6-35B-A3B (v3/qwen3.6-35b-a3b) and Qwen3.8-Flash-Next-FP8 (v3/qwen3.8-flash-next-fp8) on a larger calibration set, used ridge only: the transfer prediction alone ranks the experts (prior strength 1e9; the prompt term only breaks ties). Besides gain.npy, prior.npy and transfer.npz (as above) each folder holds:

• transfer_window.npz: the window ridge for dynamic re-selection during decode, a per-layer ridge regression of the square root of the next 256 decode tokens' saliency share on the last window's unmasked route-count share and the prompt's: matrix [L, 2E, E], offset [L, E], window (256), lam, power (2), pairs (calibration window pairs);
• meta.json: model revision, calibration runs and excluded domains, fit settings (including ridge_intercept: Qwen3.6 without, offset 0; Qwen3.8 with).

Static selection, per request and layer (c: the prompt's route counts per expert):

keep top round(keep * E) of  share(c) @ matrix[l] + offset[l]

Dynamic selection: every 256 decode tokens the set is chosen again as the top of [share(w) | share(c)] @ window.matrix[l] + window.offset[l], w the unmasked top-k route counts of the last 256 decode tokens, plus 0.5 standard deviations of that prediction for the experts kept now.

Calibration (2,304 requests, each answered by the model itself): calmix1024 as above; calmcq512, 256 MMLU dev and 256 ARC-Challenge questions, answered once with thinking off (16k) and once with thinking on; calv3-1024 without its Tulu-3 rows, 256 each from the MATH train split, the MBPP train split and GPQA-style questions, three shards with thinking on and one off. The Qwen3.6 files were fitted on the BF16 checkpoint.

v4 priors (v4/)

The final calibration mix, fitted for Qwen3.8-Flash-Next-FP8 (v4/qwen3.8-flash-next-fp8) and Qwen3.6-35B-A3B (v4/qwen3.6-35b-a3b, fitted on the BF16 checkpoint). Ridge only: each folder holds transfer.npz (matrix [L, E, E], offset [L, E] all zero, lam, power 2) and meta.json (model revision, fit settings, calibration sets and runs). There is no gain.npy or prior.npy: the selection uses T alone.

Fit, per MoE layer l: sqrt(s) ~ r @ matrix[l] by ridge regression without intercept over the calibration requests (lambda = 1, scaled by tr(R R^T) / requests), with r the prompt's route-count share per expert and s the response's REAP saliency share (router weight times expert output norm), measured by teacher-forcing the model's own response.

Selection, per request and layer (c: the prompt's route counts per expert):

keep top round(keep * E) of  share(c) @ matrix[l]

Calibration (3,584 requests, each answered by the model itself at temperature 1.0):

set	requests	data
calmix1024	1,024	Open-Platypus, OpenThoughts-114k, evol-codealpaca-v1, FineWeb-Edu
callong-ot384	384	OpenThoughts-114k problems with the longest reference answers
callong-otcode384	384	OpenThoughts-114k code problems with the longest reference answers
calagent-swe256	256	SWE-smith trajectories cut before an agent turn, SWE-agent's tools
calagent-xlam256	256	xLAM function-calling queries with their tools
calown-wp256	256	WritingPrompts curated
calown-personas256	256	tulu-3-sft-personas-math
calown-cfcots384	384	codeforces-cots (Python) problems with the longest R1 traces
calotcode-nothink384	384	further OpenThoughts-114k code problems, answered with thinking off

All sets are answered with thinking on except calotcode-nothink384. No prompt contains a question of MATH-500, GPQA-Diamond, MMLU-Pro, MBPP+ or HumanEval+ (substring or shared word 13-gram); the thinking-off code set is also checked against LiveCodeBench v6 (2025-02 to 2025-05). No data is drawn from LiveCodeBench, LeetCode or AtCoder.

Use

With apex-engine (FreeToken with APEX):

ft serve --model Qwen/Qwen3.6-35B-A3B --moe-strategy offload \
    --apex-dir <this repo>/qwen3.6-35b-a3b --apex-keep 0.25