fledor/self-align-to-explain-adapters

Self-Align to Explain — Best Adapters

The best LoRA adapter for each model × dataset × method cell (72 in total) from the thesis Self-Align to Explain: Comparing Post-Training Methods for Counterfactual Generation. Each adapter fine-tunes an instruction-tuned base model to generate minimal, label-flipping counterfactuals for BoolQ, SNLI-Premise, or SNLI-Hypothesis, using one of SFT, DPO, SimPO, GRPO (composite or decomposed reward), or GDPO.

Code, exact per-run training configurations, and full results: https://github.com/fledor/self-align-to-explain (see reports/BEST_CONFIGS.md).

Contents

Each subfolder <base>_<dataset>_<method> contains the LoRA adapter (adapter_config.json, adapter_model.safetensors), the tokenizer and chat template used, training_args.bin, the run's wandb_config.yaml and wandb_summary.json (training provenance), and METRICS.json (evaluation stats). All adapters are QLoRA rank 32, α=16, over all attention + MLP projections.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "fledor/self-align-to-explain-adapters",
                                  subfolder="qwen7b_snli_premise_simpo")

Licensing

The adapter weights are released under MIT where the base model license permits; adapters of meta-llama/Llama-3.1-8B-Instruct are additionally subject to the Llama 3.1 Community License, and adapters of Qwen/Qwen2.5-3B-Instruct to the Qwen Research License, as derivatives of those base models.

Evaluation (fair frozen-base, N=200 prompts x 10 counterfactuals)

LFR = label flip rate (judged by the unmodified base model, against its own prediction on the original input). ΔLFR in percentage points vs the frozen base anchor of the cell. NED = median normalized edit distance over non-trivial edits; PPL = median perplexity; Parse% = unique parsed counterfactuals / attempts.

Adapter	Base model	Dataset	Method	base LFR	tuned LFR	ΔLFR (pp)	NED	PPL	Parse%
qwen14b_boolq_dpo	Qwen2.5-14B-Instruct	BoolQ	DPO	55.1	57.8	+2.7	0.204	7	76.1
qwen14b_boolq_gdpo	Qwen2.5-14B-Instruct	BoolQ	GDPO	55.1	60.8	+5.7	0.150	7	67.5
qwen14b_boolq_grpo_multi	Qwen2.5-14B-Instruct	BoolQ	GRPO (decomposed)	55.1	57.2	+2.1	0.217	7	85.2
qwen14b_boolq_grpo_single	Qwen2.5-14B-Instruct	BoolQ	GRPO (composite)	55.1	57.6	+2.5	0.223	7	82.1
qwen14b_boolq_sft	Qwen2.5-14B-Instruct	BoolQ	SFT	55.1	55.6	+0.5	0.250	7	89.2
qwen14b_boolq_simpo	Qwen2.5-14B-Instruct	BoolQ	SimPO	55.1	62.5	+7.4	0.197	7	66.0
qwen14b_snli_hypothesis_dpo	Qwen2.5-14B-Instruct	SNLI-Hypothesis	DPO	76.2	82.9	+6.7	0.471	82	47.7
qwen14b_snli_hypothesis_gdpo	Qwen2.5-14B-Instruct	SNLI-Hypothesis	GDPO	76.2	84.9	+8.7	0.454	94	42.8
qwen14b_snli_hypothesis_grpo_multi	Qwen2.5-14B-Instruct	SNLI-Hypothesis	GRPO (decomposed)	76.2	91.3	+15.1	0.462	126	29.2
qwen14b_snli_hypothesis_grpo_single	Qwen2.5-14B-Instruct	SNLI-Hypothesis	GRPO (composite)	76.2	87.8	+11.6	0.467	108	36.6
qwen14b_snli_hypothesis_sft	Qwen2.5-14B-Instruct	SNLI-Hypothesis	SFT	76.2	80.0	+3.8	0.500	84	70.1
qwen14b_snli_hypothesis_simpo	Qwen2.5-14B-Instruct	SNLI-Hypothesis	SimPO	76.2	89.9	+13.7	0.424	95	15.8
qwen14b_snli_premise_dpo	Qwen2.5-14B-Instruct	SNLI-Premise	DPO	63.3	78.8	+15.5	0.362	46	58.4
qwen14b_snli_premise_gdpo	Qwen2.5-14B-Instruct	SNLI-Premise	GDPO	63.3	89.8	+26.5	0.352	53	29.3
qwen14b_snli_premise_grpo_multi	Qwen2.5-14B-Instruct	SNLI-Premise	GRPO (decomposed)	63.3	93.4	+30.1	0.386	44	27.4
qwen14b_snli_premise_grpo_single	Qwen2.5-14B-Instruct	SNLI-Premise	GRPO (composite)	63.3	85.7	+22.4	0.338	43	50.5
qwen14b_snli_premise_sft	Qwen2.5-14B-Instruct	SNLI-Premise	SFT	63.3	63.2	-0.0	0.306	51	65.3
qwen14b_snli_premise_simpo	Qwen2.5-14B-Instruct	SNLI-Premise	SimPO	63.3	85.6	+22.3	0.426	50	68.8
qwen3b_boolq_dpo	Qwen2.5-3B-Instruct	BoolQ	DPO	30.4	32.1	+1.7	0.151	10	87.4
qwen3b_boolq_gdpo	Qwen2.5-3B-Instruct	BoolQ	GDPO	30.4	31.7	+1.3	0.213	11	90.3
qwen3b_boolq_grpo_multi	Qwen2.5-3B-Instruct	BoolQ	GRPO (decomposed)	30.4	38.6	+8.2	0.089	9	54.0
qwen3b_boolq_grpo_single	Qwen2.5-3B-Instruct	BoolQ	GRPO (composite)	30.4	41.4	+11.0	0.090	9	55.4
qwen3b_boolq_sft	Qwen2.5-3B-Instruct	BoolQ	SFT	30.4	31.6	+1.2	0.271	11	92.8
qwen3b_boolq_simpo	Qwen2.5-3B-Instruct	BoolQ	SimPO	30.4	37.9	+7.5	0.149	11	74.9
qwen3b_snli_hypothesis_dpo	Qwen2.5-3B-Instruct	SNLI-Hypothesis	DPO	48.5	55.8	+7.3	0.415	65	39.4
qwen3b_snli_hypothesis_gdpo	Qwen2.5-3B-Instruct	SNLI-Hypothesis	GDPO	48.5	63.6	+15.1	0.496	68	46.0
qwen3b_snli_hypothesis_grpo_multi	Qwen2.5-3B-Instruct	SNLI-Hypothesis	GRPO (decomposed)	48.5	56.7	+8.2	0.375	101	32.5
qwen3b_snli_hypothesis_grpo_single	Qwen2.5-3B-Instruct	SNLI-Hypothesis	GRPO (composite)	48.5	53.6	+5.1	0.348	81	30.9
qwen3b_snli_hypothesis_sft	Qwen2.5-3B-Instruct	SNLI-Hypothesis	SFT	48.5	51.6	+3.1	0.459	68	46.9
qwen3b_snli_hypothesis_simpo	Qwen2.5-3B-Instruct	SNLI-Hypothesis	SimPO	48.5	55.2	+6.7	0.379	93	36.8
qwen3b_snli_premise_dpo	Qwen2.5-3B-Instruct	SNLI-Premise	DPO	34.0	45.0	+11.0	0.237	47	50.8
qwen3b_snli_premise_gdpo	Qwen2.5-3B-Instruct	SNLI-Premise	GDPO	34.0	51.7	+17.7	0.293	42	46.5
qwen3b_snli_premise_grpo_multi	Qwen2.5-3B-Instruct	SNLI-Premise	GRPO (decomposed)	34.0	59.5	+25.6	0.333	41	33.2
qwen3b_snli_premise_grpo_single	Qwen2.5-3B-Instruct	SNLI-Premise	GRPO (composite)	34.0	61.1	+27.1	0.346	39	34.3
qwen3b_snli_premise_sft	Qwen2.5-3B-Instruct	SNLI-Premise	SFT	34.0	38.4	+4.4	0.228	47	46.9
qwen3b_snli_premise_simpo	Qwen2.5-3B-Instruct	SNLI-Premise	SimPO	34.0	59.8	+25.8	0.293	55	41.1
qwen7b_boolq_dpo	Qwen2.5-7B-Instruct	BoolQ	DPO	37.8	59.9	+22.1	0.093	9	51.1
qwen7b_boolq_gdpo	Qwen2.5-7B-Instruct	BoolQ	GDPO	37.8	37.5	-0.2	0.106	10	82.3
qwen7b_boolq_grpo_multi	Qwen2.5-7B-Instruct	BoolQ	GRPO (decomposed)	37.8	57.0	+19.2	0.090	8	52.3
qwen7b_boolq_grpo_single	Qwen2.5-7B-Instruct	BoolQ	GRPO (composite)	37.8	59.3	+21.5	0.086	8	45.4
qwen7b_boolq_sft	Qwen2.5-7B-Instruct	BoolQ	SFT	37.8	47.2	+9.4	0.141	8	79.8
qwen7b_boolq_simpo	Qwen2.5-7B-Instruct	BoolQ	SimPO	37.8	66.7	+28.9	0.115	9	55.7
qwen7b_snli_hypothesis_dpo	Qwen2.5-7B-Instruct	SNLI-Hypothesis	DPO	45.2	68.4	+23.2	0.340	113	44.1
qwen7b_snli_hypothesis_gdpo	Qwen2.5-7B-Instruct	SNLI-Hypothesis	GDPO	45.2	62.9	+17.7	0.372	109	56.6
qwen7b_snli_hypothesis_grpo_multi	Qwen2.5-7B-Instruct	SNLI-Hypothesis	GRPO (decomposed)	45.2	67.3	+22.1	0.353	121	33.8
qwen7b_snli_hypothesis_grpo_single	Qwen2.5-7B-Instruct	SNLI-Hypothesis	GRPO (composite)	45.2	65.9	+20.7	0.250	137	24.5
qwen7b_snli_hypothesis_sft	Qwen2.5-7B-Instruct	SNLI-Hypothesis	SFT	45.2	47.0	+1.8	0.326	97	63.3
qwen7b_snli_hypothesis_simpo	Qwen2.5-7B-Instruct	SNLI-Hypothesis	SimPO	45.2	76.2	+31.1	0.370	160	24.2
qwen7b_snli_premise_dpo	Qwen2.5-7B-Instruct	SNLI-Premise	DPO	53.5	78.0	+24.5	0.311	54	50.9
qwen7b_snli_premise_gdpo	Qwen2.5-7B-Instruct	SNLI-Premise	GDPO	53.5	70.0	+16.5	0.303	54	71.2
qwen7b_snli_premise_grpo_multi	Qwen2.5-7B-Instruct	SNLI-Premise	GRPO (decomposed)	53.5	80.9	+27.4	0.327	53	44.5
qwen7b_snli_premise_grpo_single	Qwen2.5-7B-Instruct	SNLI-Premise	GRPO (composite)	53.5	79.5	+26.0	0.337	62	47.6
qwen7b_snli_premise_sft	Qwen2.5-7B-Instruct	SNLI-Premise	SFT	53.5	53.9	+0.4	0.250	55	74.5
qwen7b_snli_premise_simpo	Qwen2.5-7B-Instruct	SNLI-Premise	SimPO	53.5	85.0	+31.5	0.340	60	61.2
llama8b_boolq_dpo	Llama-3.1-8B-Instruct	BoolQ	DPO	54.4	66.0	+11.5	0.101	9	84.2
llama8b_boolq_gdpo	Llama-3.1-8B-Instruct	BoolQ	GDPO	54.4	70.8	+16.3	0.348	13	91.6
llama8b_boolq_grpo_multi	Llama-3.1-8B-Instruct	BoolQ	GRPO (decomposed)	54.4	66.9	+12.5	0.326	15	89.2
llama8b_boolq_grpo_single	Llama-3.1-8B-Instruct	BoolQ	GRPO (composite)	54.4	60.5	+6.1	0.221	12	86.5
llama8b_boolq_sft	Llama-3.1-8B-Instruct	BoolQ	SFT	54.4	53.6	-0.9	0.279	13	83.8
llama8b_boolq_simpo	Llama-3.1-8B-Instruct	BoolQ	SimPO	54.4	63.5	+9.0	0.083	9	70.3
llama8b_snli_hypothesis_dpo	Llama-3.1-8B-Instruct	SNLI-Hypothesis	DPO	58.7	62.7	+4.0	0.414	118	83.9
llama8b_snli_hypothesis_gdpo	Llama-3.1-8B-Instruct	SNLI-Hypothesis	GDPO	58.7	64.7	+6.0	0.545	164	65.8
llama8b_snli_hypothesis_grpo_multi	Llama-3.1-8B-Instruct	SNLI-Hypothesis	GRPO (decomposed)	58.7	67.3	+8.6	0.405	183	65.0
llama8b_snli_hypothesis_grpo_single	Llama-3.1-8B-Instruct	SNLI-Hypothesis	GRPO (composite)	58.7	61.3	+2.6	0.468	148	83.5
llama8b_snli_hypothesis_sft	Llama-3.1-8B-Instruct	SNLI-Hypothesis	SFT	58.7	57.0	-1.7	0.577	139	71.7
llama8b_snli_hypothesis_simpo	Llama-3.1-8B-Instruct	SNLI-Hypothesis	SimPO	58.7	61.7	+3.0	0.409	161	81.2
llama8b_snli_premise_dpo	Llama-3.1-8B-Instruct	SNLI-Premise	DPO	43.2	45.6	+2.4	0.411	97	74.2
llama8b_snli_premise_gdpo	Llama-3.1-8B-Instruct	SNLI-Premise	GDPO	43.2	54.4	+11.2	0.463	108	87.2
llama8b_snli_premise_grpo_multi	Llama-3.1-8B-Instruct	SNLI-Premise	GRPO (decomposed)	43.2	71.6	+28.4	0.331	67	78.8
llama8b_snli_premise_grpo_single	Llama-3.1-8B-Instruct	SNLI-Premise	GRPO (composite)	43.2	74.7	+31.4	0.290	59	57.8
llama8b_snli_premise_sft	Llama-3.1-8B-Instruct	SNLI-Premise	SFT	43.2	45.1	+1.9	0.490	103	82.4
llama8b_snli_premise_simpo	Llama-3.1-8B-Instruct	SNLI-Premise	SimPO	43.2	62.2	+19.0	0.426	90	94.2