Archived training run. The fertility-rate adapter from the 176-fine-tune study behind Silent Failure in Automated Model Adaptation. The chart-honesty adapter that was submitted is at polychart-shown-is-not-supported-lora. This copy is kept only so the run history survives the challenge.
PolyChart — Shown Is Not Supported (LoRA)
A LoRA adapter for Llama-3.3-70B-Instruct that answers questions about charts under a
measure-then-claim contract: a claim about a chart is only made when the underlying
values support it, and the gap between what a chart shows and what its data supports
is reported rather than smoothed over.
Trained on Adaption's AutoScientist platform for the AutoScientist Challenge 2026.
Scores, and which number to read
| Metric | Value |
|---|---|
| Held-out category win rate | 85.9 |
| Control (same seed, no augmentation) | 68.5 |
| On-dataset win rate (platform dashboard) | 91 |
Read 85.9, not 91. They measure different things, and this is the single most important caveat on this model. The dashboard's win rate is computed on the training distribution; the held-out category evaluation is computed on data the model never saw. Across the study behind this adapter they sometimes moved in opposite directions — one matched pair went 87.2 → 81.0 on the displayed number while going 66.3 → 77.3 on the held-out one. Selecting a model by the displayed figure can therefore select the weaker model. Every number quoted for this adapter is the held-out one.
Training
| Base | Llama-3.3-70B-Instruct-Reference |
| Method | LoRA, r=64, alpha=128, dropout 0.0 |
| Target modules | all linear (q,k,v,o,gate,up,down) |
| Epochs | 3 |
| Learning rate | 1e-4 |
| Seed rows | 2,175 verified |
| After augmentation | 28,047 (17,872 synthetic domain + 8,000 general) |
Every seed row passed an independent verifier that shares no code with the generator. That verifier caught nine silent failures in our own pipeline before shipping, four of them inside the verifier's own scaffolding.
The study this came from
This adapter is one model out of 176 fine-tuning jobs across 86 launches on 13 datasets, run headless through a client written before the platform's official API shipped. The design was paired within seed, because held-out difficulty spans 44.5 to 85.9 across those datasets — roughly three times the effect being measured, so a cross-seed average would mostly report which seeds landed in which arm.
Result: synthetic augmentation improved the held-out category score on 13 of 13 seeds, mean +14 points, two-sided sign test p = 0.0002. The platform's advertised "+16 points from crossing 20,000 datapoints" reproduced at +17.4 — this adapter is that run.
A sign test rather than a t-test: within-cell noise is sd 3–5, measured from replicates, so the defensible question is direction per seed rather than exact magnitude.
A retraction
An earlier draft of the companion paper claimed the platform's scaling benefit reverses, on the strength of one unpaired comparison where a score fell from 76 to 72. The paired test across 13 seeds reversed that conclusion, and the claim is withdrawn. It is recorded here rather than deleted, because a model whose whole thesis is shown is not supported owes that test to its own claims first.
Intended use and limits
Intended for chart-interpretation question answering and for detecting misleading chart encodings. It is not a general-purpose assistant.
- One task family, one platform, one base model.
- The augmentation result uses a single dose at one base; no dose-response curve was established across seeds.
- Held-out evaluations fire on roughly a third of jobs and cannot be triggered on demand, so several seeds contribute one draw per arm. Direction is reported, not a pooled effect size.
- Inherits the biases and failure modes of
Llama-3.3-70B-Instruct.
Use
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = "meta-llama/Llama-3.3-70B-Instruct"
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, "rodriguescarson/polychart-shown-is-not-supported-lora")
tok = AutoTokenizer.from_pretrained(base)
Links
- Client + platform findings: https://github.com/rodriguescarson/adaption-autoscientist-api
- Live Lie Factor demo: https://lie-factor.carsonrodrigues.com
- Datasets (15 public): https://huggingface.co/rodriguescarson/datasets
License
Llama 3.3 Community License, inherited from the base model.
- Downloads last month
- 8