Laguna-XS-2.1 J-lens: bf16 vs NVFP4 A/B

Two Jacobian lenses of poolside/Laguna-XS-2.1 (~33B fine-grained MoE, 40 layers, d=2048, 256 experts/8 active) fitted on byte-identical prompts (200 Γ— wikitext-103, seq 128, all 39 source layers, target 39) β€” one from the bf16 checkpoint, one from the official NVFP4 twin. The twins' non-expert tensors are bit-identical (verified: lm_head, final norm, embeddings, attention, router), so every lens difference is attributable to NVFP4 quantization of expert weights (plus one ~0.6% export-time perturbation of the 3 layer-0 dense-MLP tensors, documented in the report).

Capture d’écran 2026-07-17 aΜ€ 19.20.19

Files

  • laguna_bf16_jacobian_lens.pt, laguna_nvfp4_jacobian_lens.pt β€” merged n=200 lenses (fp16 payload; fp16 rounding is irrelevant at 1βˆ’CKA β‰ˆ 2e-9)
  • shards_{bf16,nvfp4}/ β€” 8 per-GPU shard lenses per arm (fp32, 25 prompts each; shard i = same prompts in both arms)
  • convergence_{bf16,nvfp4}/ β€” per-shard convergence CSVs
  • results/ β€” ab_summary.json, per-analysis npz, figures, REPORT.md

Probe: P = W_U[ids]Β·Ξ³ (plain-Ξ³ RMSNorm, no softcap), ids = 4096 frozen seed-0 over vocab 100352; identical across arms. Fit code: open-jlens-data repo (code/moe/fit_moe.py laguna dispatch, nvfp4_experts.py, analyze_laguna_ab.py). Fitted + analyzed 2026-07-17. Context: this A/B calibrates the NVFP4 caveat of the Inkling ~950B lens (PrimeIntellect/inkling-jlens) β€” same modelopt recipe family; transfer caveat: Laguna experts are 512-dim vs Inkling 3072-dim.

xs2/ β€” Laguna-XS.2 version-drift extension (E9)

xs2/ holds the same fit for poolside/Laguna-XS.2 (bf16, identical 200 prompts/probe/tokenizer) plus the version comparison vs XS-2.1. Verdict: version drift is a regime change β€” same-layer CKA median 0.638 = 43Γ— the NVFP4 effect and 54Γ— prompt noise β€” while the depth architecture survives (both versions k=2 at layer 27/28, cross-matrix depth correlation 0.990). The versions share no token basis (same-token unembedding rows near-orthogonal), so each is read through its own probe. Perturbation ladder on identical prompts (1βˆ’CKA): noise 0.007 < NVFP4 0.008 β‰ͺ version 0.362.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for PrimeIntellect/Laguna-XS.2-jlens

Quantizations
1 model