{"uuid": "d4f58c90-f4c0-4f64-a1dc-06de020bcb37", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2026-4372", "type": "seen", "source": "https://gist.github.com/joorklee/7bf71167b89a0c2a2dd0e1ab3a069127", "content": "# MASTER SYNTHESIS \u2014 Muse-Glimmer-30B De-abliteration\n\n## Consolidated research note across all 11 groups (Tasks 1\u201366)\n\nCompiled from: G1 Direction Extraction [1] \u00b7 G2 Subspace Removal [2] \u00b7 G3 Layer/Position [3] \u00b7 G4 KL Math [4] \u00b7 G5 Preference Optimization [5] \u00b7 G6 Distillation/Teacher [6] \u00b7 G7 Steering/Activation [7] \u00b7 G8 Model Merging [8] \u00b7 G9 Data Curation [9] \u00b7 G10 Hyperparameters [10] \u00b7 G11 Evaluation [11]\n\n* * *\n\n## 0. PROJECT FRAME\n\n**Model:** `meta-models/Muse-Glimmer-30B` \u2014 **dense** 52-layer decoder (hidden 6656), SwiGLU (down_proj 19968\u21926656), GQA, SWA-2048 alternating with full-attention NoPE layers every 4th, tanh softcap \u00b120 on final logits (pre-scaled \u00d70.1961), vocab 202,048 (2,048 pinned reserved tokens), context 131k. VLM (ViT-G/14). Refusal lives in the `to=self` **reasoning stream** and is **multi-directional** (11 geometrically distinct refusal categories acting as one shared 1-D \"control knob\" [1][2]).\n**Target:** \u22643/100 refusal, KLD to base as low as possible.\n**SFT frontier (the baseline to beat):**\n| Run | \u03bb | ep | r | KL | Refusal/100 |\n| --- | --- | --- | --- | --- | --- |\n| sft1 | 1 | 2 | 16 | 0.19 | 25 |\n| sft2 | 8 | 1 | 8 | 0.016 | 90 |\n\n**\u26a0\ufe0f Headline finding (G4, G10, G11):** the measured KL is **anti-correlated with removal** \u2014 naive per-token mean KL is a _closeness-to-base_ meter, not a _damage_ meter, and saturates to ~0 on near-one-hot Muse outputs. sft1 (0.19/25) vs sft2 (0.016/90) are _consistent_ with this, not contradictory: low KL \u27fa model stayed inside the refusing base's support \u27fa refusal retained. **Fix the metric before trusting any \u03bb sweep** [4][10][11]. Also: sft1 vs sft2 differ in \u03bb AND epochs AND rank \u2014 **confounded; no single-lever conclusion is valid from them** [10][11].\n**Cross-cutting reality checks:**\n\n*   Mozilla of Muse: no published Muse-Glimmer de-abliteration exists \u2014 all layer/rank/KL numbers are **transferred priors** (Llama/Qwen/Gemma/Zephyr); measure on Muse [1][2][3][7].\n    \n*   \"HERETIC\" is a **GitHub tool** (p-e-w/heretic), not a paper [2][10]. Heretic's backbone = Arditi 2406.11717 + Lai 2025 + Young 2512.13655.\n    \n*   Many 26xx/27xx arXiv IDs are **future-dated / low-confidence** \u2014 verify each on arxiv.org before citing [2][7].\n    \n*   Hardware: 30B bf16 \u2248 55.7 GB base; LoRA adapter states are tiny (~0.25\u20130.4 GB) [10].\n    \n\n* * *\n\n## 1. DIRECTION PREPARATION (G1 \u2014 Tasks 1\u20136)\n\n### 1.1 Elicitation (bad) set \u2014 Task 1\n\n*   `mlabonne/harmful_behaviors`**, N=400 extraction / 100 held-out eval.** 520 rows (416/104) = repack of AdvBench subset (Zou 2307.15043); community standard [1]. Note: the \"416/104\" is a community 80/20 convention \u2014 the **published split is AdvPrompter 60/20/20** (train 312 / val 104 / test 104) [11].\n    \n*   Protocol: pre-run all 416, **drop prompts that don't actually refuse** (target \u226595% of survivors refusing), supplement shortfall with **DoNotAnswer** (2308.13387; ~100% refusal coverage \u2014 best complement for multi-directional) [1].\n    \n*   Reject: `allenai/wildjailbreak` (adversarial frames dilute trigger; long prompts risk SWA window), `LLM-LAT/harmful-dataset` (training triples, wrong shape), `DoNotAnswer` as primary [1].\n    \n*   Elicitation evidence: Arditi used ~100; Heretic/Gabliteration (2512.18901) standard = 400 extraction. Empirics on Gemma-3-12b-it (closest tanh-softcap sibling): **97/100 refuse** \u2014 supports \u226595% elicitation on a same-family hardened model [1].\n    \n\n### 1.2 Good (capability anchor) set \u2014 Task 2\n\n*   `mlabonne/harmless_alpaca`**, N\u2248256\u2013400, template-locked, instruction-only.** Do NOT use self-generated or topic-matched good-sets; mixing gains nothing [1].\n    \n*   Decisive: canonical DIM does NOT pair \"harmless answers to the same harmful prompts\" \u2014 Arditi contrasted 32 harmful against **unrelated Alpaca harmless prompts** [1].\n    \n*   2603.22061 directly tested topic-matched anchoring: matched subtraction cancels the shared activation component (difference drops ~an order of magnitude below noise floor) \u2192 **no functional refusal directions**; unmatched pool keeps a large geometrically-inert direction \u2192 preserves low KLD. (Demonstrated on Qwen3.5-2B; arch-agnostic argument but untested on dense 30B.) [1][2]\n    \n*   Rimsky 2312.06681's matched contrast is a _different regime_ (CAA answer-pair steering) \u2014 do not import [1].\n    \n\n### 1.3 Template locking \u2014 Task 3 (strict protocol; contamination is real)\n\nEvidence spine: format change relocates the refusal anchor (2509.09708, 2507.11878); formatting alone swings outputs (2310.11324); \"Format-Gated Behaviors\" (5/5 comply Q/A vs refuse Human/AI \u2014 DIM over formats isolates a _format signal_); mean-centring removes corpus bias (2312.03813) [1].\n\n*   Muse-specific confounds: (a) **\"Reasoning strength:\" header** \u2014 varying the knob between sets makes DIM encode \"a reasoning block is present,\" not refusal (2605.26772: refusal dual-encoded template+CoT; steering alone 39\u201343% compliance, +compliant CoT \u2192 94%, suppress CoT \u2192 70%; 2607.13346 App. F explicitly names mixed reasoning-defaults a \"recurring confound\"); (b) **tool-schema preamble** \u2014 large constant offset + position shift breaks last-token alignment [1].\n    \n*   Protocol: one `chat_template` + one code path (`apply_chat_template(..., add_generation_prompt=True)`) both sets; bit-identical prefix (fixed \"Reasoning strength:\" value, identical tool text, hashed token strings); same model state (bf16, padding left, pad=eos, use_cache=False, **DFlash drafter disabled**); derive both sets from the SAME prompts (only response content differs); read at EOI last-token + `to=self` tokens; filter by Arditi refusal log-odds; balance classes [1].\n    \n*   Verify content-not-layout: tail-subtract/mean-centre; refuse-vs-refuse chance control (must be \u2248orthogonal); cross-template transfer test; fix then sweep the reasoning knob demanding cosine >~0.9 (2607.13346 App. F) [1].\n    \n*   \u26a0\ufe0f A single linear projection may leave residual refusal modes for reasoning models \u2014 template-locking fixes the confound, not the multi-directionality [1].\n    \n\n### 1.4 Read position \u2014 Task 4\n\n*   **Last-token (EOI) residual per layer \u2014 canonical** (Arditi 2406.11717; 2507.11878; 2607.25270; COSMIC 2506.00085; RepE 2310.01405 single position). EOI is the only position attended to all prior context = the decision point [1].\n    \n*   **Do NOT mean-pool over prompt tokens** (dilution). CAA/CAST average over _contrastive example-pair activations_ (dataset-level diff-of-means), not intra-prompt positions [1].\n    \n*   Muse extra anchor: **first-**`to=self` **(reasoning) token** \u2014 refusal decisions are made inside the CoT (2507.03167, 2605.26772); EOI alone is the weaker handle on reasoning models [1][3].\n    \n*   Capture **three anchors per layer (all 52):** EOI last-prompt token, first-reasoning token, first-assistant token; compute diff-in-means **per position per layer** (no pooling); position-stability check (mean cosine across examples) [1].\n    \n\n### 1.5 Extractor \u2014 Task 5\n\n*   **DIM first (free probe)** \u2014 optimal single vector for non-hardened models (Im&Li 2026; AxBench; EleutherAI note). BUT single direction fails on hardened models: extended-refusal training (2505.19056) disperses refusal across dims+positions \u2192 rank-1 removes \u2264~10% vs 70\u201380% baseline [1][2].\n    \n*   **Top-k PCA / whitened-SVD** of the harmful\u2212good _unmatched_ contrast, ranked by **refusal-correlation, not eigenvalue** = primary for a hardened model (OBLITERATUS, Gabliteration 2512.18901).\n    \n*   **INLP-style tunable subspace** (2606.13720) as KL-minimizing validator (competitive with DIM, tunable k, near-baseline perplexity) [1].\n    \n*   **Contrastive CCA = DEAD END** (maximizes common/topic correlation \u2014 opposite of the refusal difference) [1].\n    \n*   Multi-directionality: 11 categories collapse to one shared knob \u2192 subspace rank largely buys _removal efficiency_; **go k=1 first, grow k only until refusal \u22643/100** [1][2]. Utility guard (2607.17427 \"Abliteration Is Not a Scalpel\"): abliteration shifts decision disposition even on zero-refusal tasks \u2192 one capability probe per rank sweep [1].\n    \n*   Anchor: Heretic achieves 3/100 @ KLD\u22480.16 (Gemma-3-12B); expect higher on 30B (\u26a0\ufe0f hypothesis, not measured) [1][7].\n    \n\n### 1.6 Gradient-based direction \u2014 Task 6\n\n*   **Not a dead end \u2014 a complement, strongest as iterative RDO** (Elstner et al., \"Geometry of Refusal,\" 2502.17420, ICML): gradient-descent on refusal vector r with ablation-CE + addition-CE + **KL-retain loss** (exactly our KLD); outperforms DIM on subtraction-steering; ~40% TruthfulQA side-effect cut; extends to multi-dim concept cones [1].\n    \n*   One-shot SVs (2502.18862, 96.9% Harmbench ASR), GRAINS (2507.18043, VLMs \u2014 relevant, Muse has ViT), attribution patching (2310.10348) [1].\n    \n*   \u26a0\ufe0f **Tanh-softcap gradient saturation** (chain hidden\u2192logit\u2192tanh(\u00b130)\u2192softmax has two vanishing-gradient points; high-confidence refusal \u2192 gradient \u22480 \u2192 noise) \u2014 flagged as **unsourced hypothesis**, no paper found [1].\n    \n*   Recipe: one-shot \u2207log p(refusal) w.r.t. last-token hidden state at layers **L24\u2013L44**; use to rank layers + cosine-agreement check with DIM (cos>0.5 \u2192 ensemble; else prefer RDO); drop near-zero-gradient layers; **the KL-retain term is the real lever** (regularization dominates which direction); supervise at several refusal-stream token positions (2505.19056) [1].\n    \n*   Gradient \u2248 DIM in raw capability (2604.08524: vectors converge) \u2014 gradient's edge is the side-effect profile [1].\n    \n\n* * *\n\n## 2. SUBSPACE REMOVAL & MULTI-DIRECTIONALITY (G2 \u2014 Tasks 7\u201312)\n\n### 2.1 Optimal k (Task 7) \u2014 **k = 3** (grid 1\u20135; hard cap 5)\n\n*   Top-3 PCs \u2248 90% of refusal variance (PC1\u224857%, 3 PCs\u224890%) \u2192 Pareto knee. refusal(k) convex; KLD(k) concave \u2192 top dirs cheap, higher dirs hit capabilities [2].\n    \n*   k=1 is a guaranteed miss on hardened/multi-directional models (2505.19056 rank-1 failure: 70\u201380% \u2192 \u226410%; 2602.02132 residual refusal \u22480.3\u20130.5) [2].\n    \n*   Supporting: 2511.08379 (multi-direction beats single), 2512.18901 (Gabliteration), 2604.27019 (carrier effective rank \u22481.24 \u2192 keep knee small), 2603.27518 (over-refusal higher-dimensional than harmful-refusal) [2].\n    \n*   Build rank-k basis FROM the multi-directional contrast (PCA/SVD of refusal-contrast matrix or category-level diff-of-means), Gram-Schmidt orthonormalize \u2192 V_k, project out P = V_kV_k\u1d40 from `o_proj`+`down_proj`. Expect refusal ~2\u20133/100 at KLD ~0.1\u20130.2 \u2014 same KL class as sft1 with 10\u00d7 better refusal [2].\n    \n*   \u26a0\ufe0f The brief's \"optimal-k paper\" (2602.02132) is mischaracterized \u2014 it's the multi-directionality paper; k must come from the spectrum evidence, not its title [2].\n    \n\n### 2.2 SVD on (bad\u2212good) covariance vs top-k DIM (Task 8)\n\n*   **SVD/PCA of the paired bad\u2212good covariance \u2192 rank-k refusal subspace (k\u22483\u201310 adaptive)**. Diff-in-means = rank-1 PCA of between-class covariance (only nonzero eigenvector of S_b = \u03b4\u03b4\u1d40) \u2014 cheap diagnostic only [2].\n    \n*   Implementation: hook `layers[l]`, capture post-layer residual at the `to=self` refusal-onset token; D = X_bad \u2212 X_good, center, svd(D\u0304) \u2192 right singular vectors \u2192 U_\u2113 = span(V[:, :k]); ablate h' = h \u2212 U_\u2113U_\u2113\u1d40h or weight-orthogonalize W(o_proj), W(down_proj). MoE branch (if applicable): same U_\u2113 projected out of each expert's down_proj [2].\n    \n*   Adaptive k: EV \u2265 0.85, components \u2265 10% dominant EV, k_max 8\u201316. Diagnostic: v1>0.7 & k90\u22642 \u2192 not hardened \u2192 rank-2 fine; v1\u22480.2\u20130.5 \u2192 concept cone \u2192 rank-k (Muse's expected case) [2].\n    \n*   \u26a0\ufe0f \"TAP backprop,\" \"taboo ablation,\" \"harmful direction paper\" could not be sourced to canonical arXiv IDs \u2014 unverified training-data recollections [2].\n    \n\n### 2.3 Cross-layer orthogonalization (Task 9) \u2014 **default NO**\n\n*   Do NOT globally Gram-Schmidt across layers or pooled-SVD; use per-layer independent directions with a per-layer weight kernel (Heretic `direction_scope=per_layer`) [2].\n    \n*   Collinearity: standard models fully collinear (one direction works at every layer); Muse expected PARTIAL (adjacent-layer cos 0.5\u20130.8 + low-similarity tail) \u2014 **measure, don't assume** [2].\n    \n*   Cross-layer GS HURTS collinear directions (strips shared backbone \u2192 ablates noise \u2192 KLD up, no refusal gain). Gabliteration: full orth over-modifies; 2603.22061: SVD-orth reduced refusal capture 60.2% vs 68.9%. If truly orthogonal, GS is a no-op rename [2].\n    \n*   The orthogonalization that lowers KLD is **within-layer refusal\u22a5benign cleanup** (\"projected abliteration\"), not cross-layer GS. Literature on cross-layer GS is explicitly sparse [2].\n    \n*   Test arms (LoRA on o_proj+down_proj, per-layer kernel, norm-preserving): B0 per-layer independent (reference) \u00b7 C1 mild GS vs dominant dir \u00b7 C2 full GS \u00b7 C3 pooled-SVD top-k=2,3 \u00b7 C4 per-layer refusal\u22a5benign. Adopt only if refusal \u22643/100 at KLD \u2264 baseline [2].\n    \n\n### 2.4 Subspace dimension per layer (Task 10)\n\n*   **Adaptive per-layer k from EV spectrum; expect k\u22483\u201310 per layer in the** `to=self` **stream** (multi-directional, not single) [2].\n    \n*   Rule: k_\u2113 = min k with EV_\u2113(k) \u2265 \u03c4 (\u03c4=0.85, sweep 0.80\u20130.90); keep PCs \u2265 0.1\u00b7\u03bb_1; cap 16 (start 8). Take the MINIMAL k hitting \u22643/100 \u2014 every extra removed direction inflates KLD [2].\n    \n*   Shape tells: sharp drop (v1\u22650.5\u20130.8, k90\u22643) \u2192 k=1\u20132; slow decay (v1\u22720.3\u20130.4, k90\u22655) \u2192 k=3\u201310+. Expect **slow decay on Muse**. Ablate refusal, not harmfulness (2507.11878) [2].\n    \n*   Citation fix: \"2407.14461 = White-box LLMs\" is WRONG (CERN paper); sycophancy ref is 2310.13548 [2].\n    \n\n### 2.5 Selective layer pruning (Task 11)\n\n*   Two curves: signal (diff-in-means \u2016\u03bc_harm\u2212\u03bc_benign\u2016, **normalized for residual-stream norm growth**) AND causal (per-layer ablate-that-layer, \u0394P(refuse)) \u2014 causal drives selection [2].\n    \n*   Priors (transferred): Llama-2-7b L13/32, Gemma-7b L17/28, Gemma-4-12B L15\u201347; search upper-mid biased L\u224816\u201344 (floor 0.30L\u20130.85L) [2].\n    \n*   **Rule A (recommended):** greedy forward selection \u2014 add layer with max marginal refusal-drop per unit KLD; stop at refusal \u22643/100; hard cap KLD \u22640.08. Rule B (cheap): cumulative EV \u226580%. Rule C (seed): G_\u2113 > mean + k\u00b7std, k=1.0 [2].\n    \n*   **Taper** strengths (w_\u2113 = clip-normalized), not binary on/off; w_min 0.2\u20130.4. Ablate o_proj + down_proj only (leave q/k/v, gate_proj, norms, embeds). down_proj strength \u2264 o_proj. Norm-preserving biprojection, amp cap ~1.10. Taper but DON'T skip final 10\u201315% layers (output layers re-inject safety) [2].\n    \n*   **Multi-pass critical (Ouroboros):** refusal redundantly re-assembles \u2192 2\u20133 passes, re-extract/re-select each pass (~0.03 KLD/pass; total cap 0.05\u20130.10). DFlash off during collect/eval [2].\n    \n*   KLD budget: damage threshold ~0.05 (Qwen3.6-27B analog); Gemma-3-12B Pareto 3/[100@0.138](mailto:100@0.138), 8/[100@0.080](mailto:100@0.080). **Target: \u22643/100 @ KLD \u22640.08, ideally 0.02\u20130.05** [2][3].\n    \n*   \ud83d\udea8 **Dead-end branch (run first!): one cheap rank-1 Heretic pass.** If refusal barely moves \u2192 distributed-refusal hardened \u2192 weight surgery alone can't hit \u22643/100 at low KLD \u2192 fall back to SFT/DPO/distillation [2].\n    \n\n### 2.6 Soft vs hard projection (Task 12) \u2014 **soft per-layer \u03b1 wins**\n\n*   Math identity (verified from Heretic source): \u0394W = \u2212\u03bb\u00b7v\u00b7(v\u1d40W) \u2192 v-component of that layer's write scaled by (1\u2212\u03bb); implemented as rank-1 LoRA pair (lora_A = v\u1d40W, lora_B = \u2212\u03bbv), 4-bit-safe, targets only o_proj/down_proj [2].\n    \n*   Why soft wins (\u26a0\ufe0f analytic, Fisher-expansion reasoning, not a cited theorem): refusal is a threshold in cumulative removal S = \u03a3 \u03b1_\u2113\u00b7s_\u2113; KLD \u2248 \u00bd\u03a3 \u03b1_\u2113\u00b2\u00b7e_\u2113 (quadratic). Minimal-\u03b1-that-clears-threshold strictly dominates \u03b1=1 on KLD at same refusal \u2192 **\u03b1_\u2113 = clip(\u03bc\u00b7s_\u2113/e_\u2113, min_w, max_w)** (water-filling) [2].\n    \n*   Shape: **non-monotonic** \u2014 peaks mid-to-upper stack then attenuates at the head (2605.02958: refusal \"not committed until domination\"); head-proximal layers high e_\u2113 AND low marginal value \u2192 **\u03b1 low near head**; don't assume deeper \u21d2 higher \u03b1 [2].\n    \n*   Peak-kernel init (52L): attn.max_w 1.0, \u03bc\u224833 (depth .63), \u03c3 4, min 0.10 \u00b7 mlp.max_w 0.6, \u03bc\u224836 (.69), \u03c3 7, min 0.15. \u03b1>1 (to ~1.2) late fix only if stubborn. Whole-matrix/tanh-coefficient scaling REJECTED (KLD explodes: KL 3.65 Qwen3-4B broad-uniform; dr-housemd gemma-4-26B \"blows KL without gaining ASR\") [2][3].\n    \n*   Scaling law: narrow sharp peak wins both refusal AND KL (Qwen3.6-27B: L61 sharp \u2192 8/100@KL 0.0027 vs wide \u2192 10/[100@0.0061](mailto:100@0.0061)); larger models tolerate more local \u03b1 [2].\n    \n*   Sweep: Stage 0 per-layer SNR/e_\u2113 \u2192 Stage 1 head-to-head {A hard uniform \u03b1=1 [reject], B soft uniform {0.3,0.6}, C hard narrow band [refusal stall ~15\u201330 \u2014 self-repair], D peak kernel} \u2192 Stage 2 grid + Optuna TPE (50\u2013150 trials, min KL s.t. refusal \u22643/100) \u2192 Stage 3 conditional peel pass [2].\n    \n\n* * *\n\n## 3. LAYER & POSITION TARGETING (G3 \u2014 Tasks 13\u201318)\n\n### 3.1 Band selection (Task 13/14): **L20\u2013L45** (aggressive L18\u2013L46; conservative L24\u2013L43)\n\n*   Start \u224838% depth (past \"dead\" L0\u201314); end \u224887% depth (**exclude last ~6 layers L46\u201351 = logit-readout/softcap-20 zone**; \u26a0\ufe0f include-vs-exclude final ~10% is unresolved in literature \u2014 resolve empirically) [3].\n    \n*   Refusal concentrates ~L15\u201348, peak effectiveness ~50\u201375% depth (\u2248L26\u201339) \u2014 shifted later + wider than Llama-7B (best ~10\u201316/32) because it's a reasoning model (refusal in CoT + residual, not a narrow peak) [3].\n    \n*   Evidence: 2603.04355 (1\u20132 layers at 40\u201360% depth beats full-network), 2408.17003 (contiguous mid-layers), 2507.11771 (peak layer converges with size), Heretic search clamp 0.4\u20130.9 depth, ORBA leave outermost ~4 layers [3].\n    \n*   Sweep recipe: ~800 pairs \u2192 per-layer diff-in-means profile \u2192 grid s\u2208{10,14,18,20,22,24} \u00d7 e\u2208{42,44,45,46,47} \u2192 biprojected kernel ablation \u2192 held-out refusal (LLM judge, \u2265100 tokens) + full-vocab KLD \u2192 pick min-KLD band with \u22643/100 [3].\n    \n\n### 3.2 Writers (Task 15): **BOTH o_proj + down_proj, unequal strengths \u2014 never one alone, never equal**\n\n*   Both residual writers carry refusal; ablating both is complementary against **self-repair (~70% single-layer compensation** \u2192 must hit both) [3].\n    \n*   Heretic \"MLP more damaging\" verified with mechanism: down_proj rows are LM-critical (2607.05355: ~77.6% refusal-installing rows in MLP; refusal reducible to a single sufficient MLP neuron) [3].\n    \n*   Param: attn.o_proj \u2248 2\u20133\u00d7 mlp.down_proj (start o_proj \u2208[1.5,2.5], down_proj \u2208[0.5,1.0]); per-layer kernel peaked \u22480.65\u20130.75 depth; taper final ~10\u201315% [3].\n    \n*   Cheap decisive test: hook o_proj/down_proj outputs, measure mean |out\u00b7r\u0302| per layer; ablate each alone and score \u0394refusal/\u0394KL; drop ~0-scorer [3].\n    \n\n### 3.3 Reasoning-token targeting (Task 16): **YES \u2014 target** `to=self` **tokens only, leading portion (decision locks near start of thinking); leave** `to=user` **untouched**\n\n*   Split assistant turns on `<|start|>assistant`; read tokens after the `to=self` recipient up to `<|eot|>`/`<|eom|>`; \u2757 confirm exact marker strings against tokenizer at runtime (unverified) [3].\n    \n*   Citations: 2507.03167 (CoT causally influences refusal), \"Do Thinking Tokens Help\" (first thinking token encodes refusal; decision locked within first ~20%), 2605.26772 (refusal in activations AND CoT), 2510.06036 (\"refusal falls off a cliff\") [3].\n    \n*   Application: forward hook on decoder layers; orthogonalize hidden state w.r.t. refusal direction gated by token position (reasoning-span only); `to=user` passes through unchanged [3].\n    \n\n### 3.4 Position-specific steering (Task 17): first-K tokens vs whole generation\n\n*   Refusal decision locks at the start (2606.25013: first thinking token ~0.84\u20130.95 AUROC; locks within first ~20% of thinking; 2510.06036: intent lives in the thinking span) \u2192 early-only steering should flip the lock while leaving later distribution unperturbed (lower KLD) [3].\n    \n*   **\u26a0\ufe0f The \"early-only steering lowers full-sequence KLD\" claim is thin in the literature \u2014 strong hypothesis to test, not established** [3].\n    \n*   Policy A (first-K): steer to=self stream on generated positions 1..K only; sweep K\u2208{1,2,4,8} at \u03b1\u22481.0\u00d7; baseline K=\u221e. Policy B (reasoning-span only): steer only inside thinking block \u2014 highest KLD-parsimony [3].\n    \n*   Eval: 100 prompts \u00d7 3 seeds; refusal, full-seq per-token KLD, answer-format + capability proxy; expected early-only lands ~0.03\u20130.08 (\ud83d\udd14 reasoning estimate, uncited) [3].\n    \n\n### 3.5 Per-layer \u03b1 scheduling (Task 18): **symmetric peak/\"bump\" kernel \u2014 NOT monotonic ramp, NOT uniform high**\n\n*   Qwen3.6-27B (closest dense analog): narrow sharp peak at L61 \u2192 8/100 @ KL 0.0027 vs wide 30-layer low \u2192 10/100 @ 0.0061 \u2014 narrow+sharp wins both axes; flat-high/uniform \"blows KL without gaining ASR\" [3].\n    \n*   Param (52L): \u03b1(l) = min_w + (max_w\u2212min_w)\u00b7exp(\u22120.5\u00b7((l\u2212\u03bc)/\u03c3)\u00b2); attn: max\u22481.0, \u03bc\u224833, \u03c3\u22484, min\u22480.1; MLP: max\u22480.6, \u03bc\u224836, \u03c3\u22487, min\u22480.15. Target envelope KL \u2248 0.02\u20130.06 at \u22643/100 [3].\n    \n*   Search: TPE/Optuna, constrained (refusal \u22643/100) min KL, 50\u2013150 trials, 40 warmup. Plan iterative peel pass 2 (multi-directionality \u2192 single pass may stall ~16/100) [3].\n    \n*   \ud83d\udea8 Formal \"layer-wise \u03b1 schedule\" literature does NOT exist \u2014 empirically-grounded best practice, \u03bc/\u03c3 init is synthesis anchored to published peak depths, not a single cited number [3].\n    \n\n* * *\n\n## 4. KL-BOUNDING MATH & OBJECTIVES (G4 \u2014 Tasks 19\u201324)\n\n### 4.1 Forward vs reverse KL (Task 19) \u2014 **anchor with FORWARD KL**\n\n*   Terminology: KL(base\u2016tuned) = forward = mode-covering/zero-avoiding; KL(tuned\u2016base) = reverse = mode-seeking/zero-forcing. The brief's \"penalizes placing no probability where base has mass\" IS the forward property [4].\n    \n*   DPO/IPO/SimPO all use reverse KL; IPO's anchor collapses under near-deterministic signals (bounded MSE margin 1/2\u03b2) [4].\n    \n*   Core asymmetry: abliteration moves the model where the hardened base assigns \u22480 prob on refusal-free continuation \u2192 reverse-KL log-ratio \u2192 \u221e (support-mismatch saturation) \u2192 stalls removal/blows gradients. **Forward KL \u22480 in the ablation region \u2192 permits the refusal-free drift while still penalizing coverage loss. This direction decouples \"allow abliteration\" from \"preserve fluency.\"** [4]\n    \n*   Parameterization: (1) \u03bb\u00b7KL(base\u2016tuned); (2) cheap surrogate: one-sided clipped per-token log-ratio `loss += \u03bb\u00b7\u03a3_t max(0, log p_base(t) \u2212 log p_tuned(t))` (GKD 2306.13649 / f-DPO 2309.16240); (3) if reverse-KL kept, use IPO bounded form [4].\n    \n\n### 4.2 \u03b1-divergence / JS as KL replacement (Task 20) \u2014 KL saturates on near-one-hot\n\n*   Per-token KL is importance-weighted by reference p \u2192 only \"sees\" top mass; refusal-relevant tail invisible; argmax-flip blows up one way, diluted per-token \u2192 measured 0.016 while refusal=90 [4].\n    \n*   **Hellinger\u00b2 (H\u00b2 = 1\u2212\u03a3\u221a(pq), = R\u00e9nyi \u03b1=\u00bd): # RECOMMENDED regularizer** \u2014 bounded [0,1], symmetric, true metric, equal weight per token, monotone non-flat gradients, robust to one-hot both directions [4].\n    \n*   JSD (\u00bdKL(p\u2016m)+\u00bdKL(q\u2016m), bounded \u2264ln2, \u221aJSD metric) \u2014 second choice; catches tuned's own one-hot hardening. TV \u2014 bounded, symmetric, best worst-case gauge; flat gradients as a loss (report only). R\u00e9nyi \u03b1>1 harshens wrong direction; \u03b1\u21920 unstable [4].\n    \n*   Param: L = L_CE + \u03bb\u00b7DIV(q, p_base) with DIV = Hellinger\u00b2 (primary) or \u221aJSD, on **temperature-smoothed softmax T\u2208[2,3]** (Hinton trick \u2014 biggest practical fix for spiky saturation). \u03bb_Hell\u22480.5\u20134; \u03bb_JSD\u22481\u20138 (different scale than old \u03bb\u00b7KL \u2014 recalibrate). Report ALL: fwd+rev KL, H\u00b2, TV, \u221aJSD [4].\n    \n\n### 4.3 Per-token vs sequence (Task 21) \u2014 **per-token mean; \"length-normalized sequence KL\" is a DEAD END (same number by definition)**\n\n*   Joint-seq KL = \u03a3 per-token KL by chain rule; length-normalized seq KL = per-token mean identically. Only real axis: sum vs mean [4].\n    \n*   **Mean, not sum:** sum budget scales with T and blows up exactly on long multi-directional to=self reasoning runs \u2014 over-regularizes the region you must loosen [4].\n    \n*   DPO/RLHF practice corroborates: SimPO average log-prob, SamPO average seq-KL, LD-DPO, \"A Comedy of Estimators\" 2512.21852 (length-normalized mean = unbiased gradients + better performance) [4].\n    \n*   Real knob: **per-token position weighting** KL\u0304_w = (1/\u03a3w_t)\u03a3 w_t\u00b7D_KL, with refusal-aware w_t on to=self decision tokens (TEPO 2604.12736, TI-DPO 2505.19653, D2PO 2502.14340, TDPO 2404.11999). Report uniform-mean; heavy w_t raises effective \u03bb there \u2014 re-tune [4].\n    \n\n### 4.4 Generated vs prompt tokens (Task 22) \u2014 **response-only; mask prompt positions entirely**\n\n*   Standard: InstructGPT, DPO, \"Secrets of RLHF I\" (completion_mask), TRL, \"Rethinking KL in RLHF\" 2510.01555 \u2014 prompt is conditioning, not supervised [4].\n    \n*   SFT: \"Does Prompt Loss Matter?\" 2401.13586 (PLW=0 standard; small 0.01\u20130.5 stabilizes short completions); QLoRA Table 10 (completions-only beats full on 4/4) [4].\n    \n*   Why prompt-KL doesn't help: refusal is a residual/weight-space feature removed by orthogonalization \u2014 a token-level KL loss can't touch that stream; adding prompt-token KL pulls the model back to base \u2192 raises refusal (opposite of goal). Keep w_prompt=0; optional stabilizer \u22640.1 only [4].\n    \n\n### 4.5 Saturation-aware KL (Task 23) \u2014 the KL collapse is a NAMED failure mode\n\n*   The puzzle = the **\"KL agreement trap\" (KAT, 2606.09471):** near-one-hot p,q agreeing on argmax give per-position KL\u22480; measured scalar saturated, blind to refusal-region tail drift \u2014 \"low reverse-KL from degenerate agreement, not correct alignment.\" **The raw mean-KL metric is the root bug** [4].\n    \n*   Dead ends: reverse KL (\u221e where q has mass; structurally drives overconfidence \u2014 DRKL 2604.00223); pure entropy weighting (TIP 2604.14084 Prop.2, DEAR 2606.22830: \"entropy blind to confident-wrong\" \u2014 Muse refusal tokens ARE confident-wrong \u2192 entropy weighting downweights exactly the positions to constrain); raw KL as only reported value [4].\n    \n*   Fix \u2014 monitoring: per-position H\u00b2, TV (\"fraction of probability mass moved\"), \u221aJSD over to=self stream; use TV/H\u00b2 to drive \u03bb and early stopping; KL as secondary log (GKD defaults JSD; 2502.15681; Sason & Verd\u00fa 1508.00335) [4].\n    \n*   Fix \u2014 loss: disagreement-gated, bounded/skewed divergence: w_t = [D_TV(p_t,q_t)]^\u03b3 \u00b7 1[D_TV \u2265 \u03b5], \u03b3=1 start (tune 0.5\u20132), \u03b5=0.01\u20130.05 (masks degenerate agree-on-argmax positions); optional top ~20% by D_TV (\"Beyond the 80/20 Rule\" 2506.01939) [4].\n    \n\n### 4.6 \u03bb scheduling (Task 24) \u2014 **\"anchor-then-release\": monotone non-increasing \u03bb (high\u2192low)**\n\n*   Decouple confound first: grid (ep=1,r=8) and (ep=2,r=16), \u03bb\u2208{0.3,0.5,1,2,4,8} \u2192 refusal-vs-KLD Pareto, before trusting any schedule [4].\n    \n*   Schedule: \u03bb_start \u2248 8 (holds first epoch, ~50% steps \u2014 capability-preserving; KL\u22480 before drift so anchor gradient \u22480), then decay to \u03bb_end \u2248 0.3\u20130.5 over second epoch (\"release\"/un-alignment). 2 epochs, r=16. Cosine/exponential decay. **Do NOT re-tighten \u03bb at end.** Keep LR warmup/cosine as a separate decoupled schedule [4].\n    \n*   Why high\u2192low not low\u2192high: VAE KL-annealing (low\u2192high, N19-1021) assumes a neutral prior; **here the reference is the refusing base \u2014 ramping \u03bb up late re-imports refusal.** Continuation/homotopy curriculum (Bengio ICML 2009; 2101.10382) [4].\n    \n*   Gradient interplay: tune \u03bb_end so \u2016\u03bb\u00b7\u2207KL\u2016/\u2016\u2207L_task\u2016 \u22720.1 at end (2510.01555). Expected refusal \u22643/100 at KLD below 0.19 (est. 0.05\u20130.15 \u2014 \u26a0\ufe0f estimate, not measured) [4].\n    \n*   \u26a0\ufe0f No published \u03bb-scheduling for de-abliteration SFT exists \u2014 extrapolation from VAE-KL + RLHF-adaptive-KL + curriculum; validate by ablation (constant-\u03bb vs high\u2192low vs low\u2192high arms) [4].\n    \n\n**G4 synthesis for the metric/loss stack:** (1) fix metric to H\u00b2/TV/\u221aJSD over to=self, T-smoothed; (2) forward-KL one-sided anchor (permit drift, preserve coverage); (3) per-token mean; (4) response-only mask; (5) anchor-then-release \u03bb. Alternative: Hellinger\u00b2 regularizer if saturation persists [4].\n\n* * *\n\n## 5. PREFERENCE OPTIMIZATION (G5 \u2014 Tasks 25\u201330)\n\n### 5.1 DPO with reference-KL bound (Task 25)\n\n*   DPO is more KL-efficient per unit behavior change than SFT-KL (contrastive margin vs absolute likelihood), BUT pure DPO has **no hard reference-KL bound** (KL implicit, only regularized on data distribution \u2014 2406.01462; \"jumps\" to the \u03b2+data optimum) [5].\n    \n*   Recipe: chosen = Route-C steered compliant, rejected = base refusal; DPO 2305.18290, \u03b2\u2208{0.05,0.1,0.2,0.5,1.0}, r\u2208{8,16,32}, 1 epoch, reference = frozen base. Enforce the KL bound explicitly (\u03c7PO 2410.05258, \u03b2-DPO 2407.08639, \u03b5-DPO, HyPO, early-stop) [5].\n    \n*   \u26a0\ufe0f **Softcap caveat:** logits pre-scaled ~0.196 then tanh-capped 20 \u2192 log-prob gaps compressed/saturated; DPO margin gradients vanish in the saturation band and \u03b2 behaves smaller. Measure reference-KL on **hidden states**; verify with raw-logit control [5].\n    \n*   vs SFT frontier: preference-only reaches ~2/100 @ KL 0.076 on Zephyr-7B (2512.13655 \u2014 but that's **weight-ablation-derived**, not a DPO \u03b2 result) [5].\n    \n\n### 5.2 IPO/SimPO/KTO (Task 26) \u2014 **IPO primary; skip SimPO/KTO for a KL-bounded goal**\n\n| Objective | Verdict |\n| --- | --- |\n| **IPO** (2310.12036) | **Primary** \u2014 loss on log \u03c0_\u03b8/\u03c0_ref directly, bounded margin 1/(2\u03b2), always regularizes even for deterministic preferences; NeurIPS'24 over-opt study: less prone to over-optimization, better KL control at scale |\n| DPO (2305.18290) | Backup A (proven to reach low-KL corner) |\n| \u03c7PO (ICLR'25) | Backup B if a hard bound required (\u03c7\u00b2-divergence, worst-case bound) |\n| KTO (2402.01306) | Skip \u2014 loose partial anchor (batch-mean z\u2080), we have clean pairs |\n| SimPO (2405.14734) | Skip \u2014 reference-free \u21d2 NO KL-to-base bound; reward-hacking risk |\n\n*   Params: IPO \u03c4\u22480.1 (margin \u22485 nats); SFT/NLL anchor \u03bb_NLL\u22480.1 on chosen (RPO fuse); DPO fallback \u03b2=0.5\u20131.0 (\u22485\u00d7 default \u2014 softcap compensation \u26a0\ufe0f mechanistic inference, verify); LoRA r=16\u219232; **1 epoch** (over-opt peaks ~first 25% of epoch, 2406.02900); LR 1e-6\u20133e-6; loss on post-softcap logits [5].\n    \n*   Softcap advantage: tanh flattens gradient at extreme margins \u2192 IPO/\u03c7PO's finite target margin beats DPO's unbounded logistic on THIS model [5].\n    \n\n### 5.3 Self-generated pairs (Task 27) \u2014 steered compliant chosen / base refusal rejected, ONE-SHOT, quality-gated\n\n*   Minimal distribution shift \u2192 lower KL; both from same softcapped distribution \u2192 calibrated log-ratio. Off-policy/external chosen is OOD for the reference \u2192 inflated margin, larger KL [5].\n    \n*   Evidence: SPIN 2401.01335, Self-Rewarding 2401.10020, RLAIF 2309.00267, on-policy > offline (2404.14367). **Chosen QUALITY dominates DPO (2508.18312)** \u2192 gate steered outputs [5].\n    \n*   Hybrid hedge: mix small fraction of high-quality external chosen if steered quality poor (NeuralDaredevil precedent) [5].\n    \n*   Discipline: discard low-margin/ambiguous pairs (\"Less is More\" 2502.14560); add SFT/NLL term on chosen + log chosen log-prob (catch likelihood displacement, 2410.08847). **One-shot, not recursive** (model collapse 2305.17493) [5].\n    \n*   Citation fixes: Self-Alignment = 2308.06259; RLAIF = 2309.00267 [5].\n    \n\n### 5.4 \"Argmax-DPO\" (Task 28) \u2014 **the name does not exist; implement steering-ranked preference**\n\n*   Conflates three real things: (1) DPO squeezing/argmax concentration pathology (Ren & Sutherland 2407.10490); (2) best-of-n/RAFT/RSO/RS-DPO (2304.06767, 2309.06657, 2402.10038); (3) DPO\u2248steering (D-STEER 2512.11838; Anatomy of Post-Training 2606.12360) [5].\n    \n*   **Key warts:** chosen = _strongest_ steering is WRONG (steering rank \u2248 quality but over-steer distorts; moderate strength + fluency guard). The \"argmax/squeezing\" trap: rejected=base refusal is a low-probability valley that violently squeezes KL up \u2014 **lift the valley with a short SFT warm-up including rejected completions** (2407.10490) \u2014 single biggest KL lever [5].\n    \n*   Objective: SimPO preferred (\u03b3\u22480.8\u20131.2 \u2014 length-normalized, tolerant of noisy steering ordinate); DPO fallback \u03b2\u22480.5\u20131.0 with label smoothing 0.1\u20130.2 [5].\n    \n\n### 5.5 \u03b2-search for exact \u22643/100 (Task 29) \u2014 **FRAMING INVERSION: search UPWARD in \u03b2**\n\n*   Since sft2 already sits at 90/100 @ KL 0.016 (high-refusal/low-KL), the min-KL point that hits \u22643/100 is the **LARGEST \u03b2 (weakest deviation) that still lands refusal \u22643/100** \u2014 not the most aggressive \u03b2 [5].\n    \n*   Protocol: ep=1, r=16, LR 1e-6\u20132e-6, batch ~64; Stage A \u03b2\u2208{0.02,0.1,0.5,1.0}; Stage B binary search in log-space between \u03b2_lo (largest with refusal\u22643) and \u03b2_hi (smallest with refusal>3); stop \u03b2_hi/\u03b2_lo \u22641.5\u00d7 or \u22656 runs; 1\u20132 confirmation seeds [5].\n    \n*   **Predicted \u03b2 for 30B base-reference: 0.01\u20130.05, best guess 0.02\u20130.03** (2407.13709: weak/SFT/base reference \u2192 optimal \u03b2 \u22480.01\u20130.02; Zephyr 0.1; handbook: [1ep@0.01](mailto:1ep@0.01) \u2248 [3ep@0.1](mailto:3ep@0.1)). Expected attained KL \u2248 0.05\u20130.10 (near 0.076 abliteration anchor; unverified for Muse) [5].\n    \n*   Softcap: compute DPO ratio on **pre-tanh (post-multiplier) logits** or the saturated band makes you systematically undershoot (need smaller nominal \u03b2) [5].\n    \n\n### 5.6 Data scale (Task 30) \u2014 **~300\u2013700 unique prompts, 1 pair each; knee at ~300\u20131000**\n\n*   Safety breaks with 10\u2013100 harmful examples (Qi 2310.03693); Shadow Alignment 100 (2402.08939); plateau <10% of data (2411.10545); 10% of UltraFeedback \u2248 full (2502.14560) [5].\n    \n*   1 pair per prompt; multiple rejected variants don't help and can hurt (2502.16825, T\u00fclu 3 2411.15124). Chosen quality dominates (2508.18312); contrast > volume (2406.09279) [5].\n    \n*   Params: 1\u20132 ep max; \u03b2\u22480.1 baseline (raise 0.2\u20130.5 if KL climbs); LR ~5e-7; r=8\u201316 (r=8 regularizes more \u2014 KL discipline); dedup by embedding/minhash; disjoint-parallel split [5].\n    \n*   KL-stability check: per N plot refusal and KL; **elbow = where adding pairs stops reducing refusal while KL keeps rising**; watch DPO loss, chosen log-prob, harmless-KL inflection [5].\n    \n\n**G5 cross-cutting:** IPO primary (or DPO fallback); self-distilled on-policy pairs at MODERATE steering strength; \u03b2\u22480.02\u20130.03 first guess, search upward; 1 ep + LoRA r=16 + LR 1e-6\u20133e-6 + RPO SFT anchor; pre-tanh ratio, hidden-state KL; realistic endpoint \u22643/100 @ KL \u22480.05\u20130.15 [5].\n\n* * *\n\n## 6. DISTILLATION & TEACHER DESIGN (G6 \u2014 Tasks 31\u201336)\n\n### 6.1 Steered-self vs external teacher (Task 31) \u2014 **steered-self wins, not close (2\u201310\u00d7 lower KLD at matched refusal)**\n\n*   GKD 2306.13649 (off-policy/alien targets = dominant failure mode); exposure bias 1506.03099; MiniLLM 2306.08543 (alien teacher can sit outside student support \u2192 KL blow-up); Cho & Hariharan 1910.01348 (bigger/cleaner teacher often worse); SDFT 2402.13669 (self-targets reduce distribution shift); Kim & Rush 1606.07947 (**cross-tokenizer blocks logit-KD \u2014 Muse's ~200k vocab vs others = structural off-distribution tax**) [6].\n    \n*   A/B: same 100-prompt refusal eval; 500-prompt held-out KLD; Arm A 4 self-completions/prompt @ T\u22480.8; Arm B 4 matched external; LoRA r=16, 1 ep, **sweep \u03bb\u2208{2,4,8,16} reading KLD at refusal=3/100 per arm** (do NOT compare fixed \u03bb) [6].\n    \n*   Cheap pre-test: KLD(P_base\u2016P_external) \u2265 ~0.3 nats/token \u2192 external arm is a foregone loss; spend compute on steering [6].\n    \n*   Expected: self \u22480.03\u20130.10; external \u22480.15\u20130.40+ at refusal 3/100 [6].\n    \n\n### 6.2 On-policy distillation (Task 32) \u2014 **DAgger-style curriculum + persistent off-policy anchor**\n\n*   Student prefixes on-policy; **labels still from the frozen steered teacher** (this is NOT self-distillation \u2014 that's collapse territory) [6].\n    \n*   DAgger 1011.0686 (error O(T\u00b2\u03b5) compounding \u2192 O(T\u03b5)); GKD on-policy \u2265 off-policy, ~20\u00d7 less data; MiniLLM reverse-KL spends KL where it matters; scheduled sampling must keep a teacher-forced fraction; **anchor in actual batches, not just on disk** (2305.17493, 2404.01413) [6].\n    \n*   Rounds 0..3: bootstrap on frozen teacher buffer (never start on-policy on a 90%-refusing student); per round generate with current student @ T\u22480.8\u20131.0 \u2192 teacher re-labels \u2192 batch mix \u03bb_R on-policy + (1\u2212\u03bb_R) off-policy (\u03bb: 0.3\u21920.5\u21920.7, ALWAYS \u226530\u201350% anchor) \u2192 GKD JSD(\u03b2=0.5\u20130.9) or reverse-KL(on-policy slice) + CE(anchor). Stop when refusal \u22643 AND 2 rounds improve KL <5%. Gains saturate ~3\u20134 rounds; one grow-step often best [6].\n    \n*   Expected: reach the gate at ~\u2153\u2013\u2154 of single-pass KL (\ud83d\udfe1\u26a0\ufe0f estimate) [6]. Dead ends: pure self-distillation; pure on-policy/RKL (unstable collapse) [6].\n    \n\n### 6.3 Full reasoning vs final-answer targets (Task 33) \u2014 **final-answer-only is a DEAD END**\n\n*   Refusal lives in the residual stream THROUGHOUT the reasoning chain (2406.11717; 2510.06036 thinking-token intent; 2507.03167; 2507.12428). Final-only never supervises the exact tokens where refusal fires; the unsupervised reasoning stream drifts \"for free\" yet still counts toward KLD [6].\n    \n*   Parameters: (1) keep full to=self+to=user in targets for the refusal-relevant subset; (2) self-distill the teacher's own greedy/low-T reasoning (foreign verbose CoT IS the KL-inflation mechanism); (3) **harmfulness-stratified hybrid**: full reasoning for harmful prompts (fraction p), final-answer for benign (1\u2212p) \u2014 mirrors R1 split + Phi-4-reasoning's ~20/80 mix (2412.08905); (4) keep reasoning concise (structure-over-content, 2502.07374); (5) \u03bb sweep {2,4} [6].\n    \n*   Citation fix: Magister et al. = 2212.08410 (NOT 2210.08402). \"Scimitar\" thinking-token distillation does not exist \u2014 don't cite [6].\n    \n\n### 6.4 Teacher temperature (Task 34) \u2014 **T>0 moderate; never pure greedy as primary source**\n\n*   Default T\u22480.7\u20130.8, top-p\u22480.9\u20130.95, top-k off/\u226450; mix 30\u201350% greedy/canonical + rest @T\u22480.8, or best-of-N 8\u201316. **Architecture-aware lever: extra T on the to=self stream (0.8\u20131.0), cooler to=user (0.3\u20130.5/greedy)** \u2014 reasoning-stream diversity is where removal is bought; cool answer keeps content-KL low [6].\n    \n*   Keep T \u2264 ~1.0 (T\u22651.5 = noisy \u2192 higher KL + worse removal). Higher target T \u2192 more entropic student \u2192 can lower \u03bb and still clear the gate [6].\n    \n*   Mechanism: greedy collapses support to one canonical trace \u2192 **sharp brittle refusal boundary** (reads exactly like the 25/100@\u03bb=1 / 90/100@\u03bb=8 pattern); sampled spread support \u2192 smooth boundary \u2192 surgical removal at matched KL (Hinton 1503.02531; Kim & Rush 1606.07947; GKD). Fork-token evidence: only ~15\u201320% high-entropy tokens need mode-covering (2603.07079; 2606.00306; RLVR \"80/20\") [6].\n    \n*   Expected: T\u22480.8+ mix shifts refusal <20/100 in the \u03bb\u22484\u20136 band, approaching \u22643/100 at KL\u22480.04\u20130.08. \ud83d\udfe1\u26a0\ufe0f No direct result ties teacher sampling T to D_KL(student\u2016base) under KL-reg + refusal gate \u2014 mechanism inferred, numbers are hypotheses [6].\n    \n\n### 6.5 Best-of-N filtering (Task 35) \u2014 **N=4; hard refusal-gate + SOFT quality weight; keep all survivors, never top-1**\n\n*   BoN KL tax (Gao/Schulman/Hilton 2210.10760): KL = ln N \u2212 (N\u22121)/N nats \u2192 N=1: 0; N=2: 0.193; **N=4: 0.636**; N=8: 1.204 (2\u00d7); N=16: 2.135. Hard BoN is off the KL\u2013reward Pareto frontier \u2014 soft tilting w\u221dr(y)/\u03c4 is optimal (2505.03156, BOND 2407.14622) [6].\n    \n*   KLD effect U-shaped: refusal-gate only \u2192 KLD flat benign/slightly up harmful; soft quality filter \u2192 KLD can LOWER at fixed refusal; **hard top-1 \u2192 KLD RAISES (mode collapse)** (2505.18949; GEM 2408.16673) [6].\n    \n*   **Project-specific #1 gotcha: the refusal gate must inspect the to=self reasoning stream, not just the final answer** \u2014 a compliant final with a refusing reasoning chain is still a refusal [6].\n    \n*   Param block: N=4; refusal_gate HARD at stream level (drop refusal/hesitation/silence); quality SOFT w=exp(r/\u03c4), \u03c4\u2208{0.5,1.0,2.0}; keep ALL refusal-pass candidates (top-~4); labels quality-weighted CE; \u03bb sweep {2,4,8}. \"Jailbreak Tax\": compliant targets on refusal-trained prompts measurably lower utility (up to 92% accuracy drop \u2014 ICML 2025, \u26a0\ufe0f ID unverified) \u2014 quality weight necessary [6].\n    \n*   Expected: refusal \u22643/100 at KLD ~0.01\u20130.05 \u2014 comparable/modestly above sft2's 0.016 WITHOUT the 90/100 penalty; clean targets decouple \u03bb from refusal [6].\n    \n\n### 6.6 Cross-model distillation (Task 36) \u2014 **stronger open model = scoring oracle, NOT target generator**\n\n*   Capacity gap (1910.01348; teacher-assistant 1902.03393); distribution mismatch is the CL mechanism (GKD/MiniLLM); **cross-tokenizer penalty hard & concrete** (Minixhofer NeurIPS 2025) [6].\n    \n*   Modes, KL-ranked: **Mode G (foreign generation) = MAX KL \u2014 least attractive**; **Mode S (on-policy scoring) = student samples, strong teacher scores same tokens (GKD/MiniLLM) \u2014 low KL + high ceiling**; **Mode H (hybrid/re-rank) = the sweet spot \u2014 steered-self over-generates, strong judge selects non-refusing high-quality, SFT survivors** (text stays in Muse family; ceiling from selection, not copying; consistent with Zephyr + GKD) [6].\n    \n*   Teacher choice by selection/scoring quality, not size: DeepSeek-V3 ~37B active and Qwen3-235B ~22B active are _smaller_ active capacity than Muse's 30B dense \u2014 \"bigger teacher\" overstates the gap; real handicap is distribution/tokenizer/family. If insisting on a real cross-model SFT teacher, prefer matched/intermediate family (Gemma-4-31B, Qwen3-30B) or \u2014 best \u2014 **Muse Spark 1.2 open weights (same family, same-family teacher; Muse is itself a Spark logit-distillation)** [6].\n    \n*   Decision tree: (1) T0 alone \u22643/100 @ acceptable KLD \u2192 stop; (2) if T0 refusal ceiling unreachable without KL blowup \u2192 Mode H or S; (3) pure Mode G+T1 only if ceiling data quality is the binding constraint (expect LARGEST KLD) [6].\n    \n\n**G6 consolidated stack:** steered-self targets (4/prompt @T\u22480.8, extra T on reasoning stream, full reasoning for refusal-relevant prompts); refusal-gate HARD + quality soft-weight, keep all survivors N=4; round-0 bootstrap then on-policy DAgger with \u226530\u201350% off-policy anchor; GKD JSD or reverse-KL + CE anchor; \u03bb sweep {2,4,8} reading KLD at refusal=3/100; expected KLD \u2248 0.02\u20130.08 [6].\n\n* * *\n\n## 7. STEERING & ACTIVATION INTERVENTION (G7 \u2014 Tasks 37\u201342)\n\n### 7.1 Additive vs subtractive (Task 37) \u2014 **SUBTRACT the refusal direction; do not add a compliance vector**\n\n*   Arditi 2406.11717: erasing refusal direction kills refusal; CAA 2312.06681 additive had limited effect on refusal in open-ended generation \u2014 why the field moved to subtraction [7].\n    \n*   Additive is norm/scale-sensitive and pushes the residual stream off-manifold (2604.09839); subtractive rank-1 is norm-preserving, lower KL \u2014 Surgical Refusal Ablation 2601.08489 cut first-token KL 2.088 \u2192 0.044 with a cleaned vector [7].\n    \n*   Caveat: subtraction overlaps capability circuits (math most sensitive, 2512.13655; abliteration KL 0.043\u20131.646); high-KL subtraction usually = polysemantic vector, not wrong mechanism. Add+subtract both: essentially untested; multi-vector additive degrades (needs conceptor synthesis, 2410.16314) \u2192 subtractive-only [7].\n    \n*   Geometry: subtract on to=self writers (o_proj + down_proj; skip q/k/v \u22480 signal); narrow symmetric peak kernel centered upper-middle (attn max\u22481.0 @~L33, MLP\u22480.6 @~L36, floor 0.1\u20130.15) [7].\n    \n\n### 7.2 Steering strength as a knob (Task 38) \u2014 smallest \u03b1 that clears the target\n\n*   Refusal(\u03b1) is an **affine threshold** \u2014 \"model refuses iff \u03b1\u22481\" (Marshall/Scherlis/Belrose 2411.09003); beyond cutoff more \u03b1 does nothing; KL(\u03b1) rises super-linearly (off-manifold, 2604.09839); inverted-U in \u03b1 reported (Qwen3 SAE 2606.26620, 2606.11599, MoE 2606.04160) [7].\n    \n*   Anchors: Gemma-3-12B 3/100@KLD 0.16; Zephyr-7B 2/[100@0.076](mailto:100@0.076); 100-trial 9/[100@0.063](mailto:100@0.063). **Heuristic: KLD < 0.5 \u21d2 capabilities essentially intact** [7].\n    \n*   Protocol: ~15\u201320 coarse \u03b1\u2208[0,2.0] then bisection near threshold; bootstrapped CI (1 refusal = 1% on 100 set); decision rule \u03b1* = min{\u03b1 : refusal(\u03b1) \u2264 3/100} s.t. KLD(\u03b1*) < 0.5. Scaling is model/arch-specific (Qwen \"early-divergence,\" 2606.22686) \u2014 **measure on Muse, don't transplant** [7].\n    \n*   Reasoning-stream caveat: pure activation subtraction with frozen trace reverses only ~39\u201343% of refusals, ~94% with CoT regenerated (2605.26772) \u2014 budget for CoT regeneration alongside steering, or prefer weight-level (LoRA/abliteration) [7].\n    \n\n### 7.3 Activation/path patching vs projection (Task 39) \u2014 **HYBRID: patch to LOCALIZE, then conditional localized projection**\n\n*   Patching (ROME 2202.05262; path patching; Zhang & Nanda 2309.16042; EAP-IG) is diagnostic, not a final fix (subspace-patching illusions, Makelov ICLR 2024) [7].\n    \n*   Refusal is multi-dimensional concept cones (2502.17420), distinct per-domain directions (2602.02132), sparse sub-circuit (2509.09708), steerable vectors sparsifiable 90\u201399% (2604.08524) [7].\n    \n*   Conditional steering: **CAST** 2409.05907 (ICLR 2025) gates on harmful inputs; surgical/localized beats global on KLD because refusal KLD is measured on BENIGN prompts \u2192 conditional intervention leaves benign KLD \u2248 0; KTS \"Steering Without Side Effects\" 2406.15518 explicitly minimizes steered-vs-unsteered KL on benign inputs [7].\n    \n*   Recommendations: EAP-IG/ROME to localize minimal layers/heads/positions; expect a small set of directions; make it conditional (CAST gate or null-space); select under KLD-threshold objective + \u22643/100 [7].\n    \n\n### 7.4 Conditional steering + capability restoration (Task 40) \u2014 **FEASIBLE; reframe \"re-add base residual\" as gating the subtraction** (mathematically equivalent)\n\n*   Directly relevant: CAST 2409.05907; **AlphaSteer 2506.07022** (near-zero steering on benign, refusal vector on malicious; 5\u20137% better utility retention); COS-Steering; SafeConstellations 2508.11290 [7].\n    \n*   Hook placement: residual-stream (post-MLP) = strongest refusal lever but dominant utility loss (SKOP 2605.06342; Style-Modulation-Heads 2603.13249 show attention/head-localized preserves capability). Keep subtraction in to=self residual; gate cheaply [7].\n    \n*   Gating options (all cheap): GCAD 2605.10664 (token-level); FineSteer 2604.15488 (Subspace-Energy-Ratio + MoSE); SADI 2410.12299 (dynamic per-input mask); **linear probe on h_t (one matvec) = lowest latency**; last-token heuristic (CAST) avoids per-token cost [7].\n    \n*   Design: h'_t = h_t \u2212 \u03b1\u00b7g_t\u00b7r, gate g_t = \u03c3((\u27e8h_t,c\u27e9 \u2212 \u03c4)/T), c = compliance direction, r = refusal direction; g\u22480 on tokens whose unsteered continuation is compliant; optional re-add \u03b2\u00b7c (start \u03b2=0); sweep \u03b1\u2208[0.5\u20131.5]\u00b7sft1 and \u03c4 on calibration split; report KL-vs-refusal Pareto + XSTest over-refusal [7].\n    \n*   Caveats: MoE routing may need expert-level handling (if MoE); single-direction gated subtraction may be weaker on reasoning-stream multi-directional refusal (2605.26772) \u2014 expect per-layer or multi-direction gating [7].\n    \n\n### 7.5 Persistent steering (Task 41) \u2014 custom modeling class (no weight change, KLD\u22480)\n\n*   Confirmed: PyTorch state_dict holds only params + persistent buffers; **hooks live in _forward_hooks and are LOST on save/reload** \u2014 a separate hook script every load is exactly what we must avoid [7].\n    \n*   Pattern: subclass PreTrainedModel/Config; set config_class/auto_map; extend config with steering params (vector_path, alpha, layer); store vector as registered buffer; **bake steering into forward() as tensor ops** (Dynamo-traceable; no graph breaks). Real precedent: wisent-ai/qwen2.5-coder-7b-wisent-caa [7].\n    \n*   Fold-into-weights vs inference-only: true weight-folding = abliteration W' = W(I\u2212rr\u1d40) (2406.11717) \u2014 small nonzero KLD but persists and honored by EVERY engine; inference-only (CAA 2312.06681; KL-min CAA 2406.15518) keeps weights byte-identical \u2192 **KLD\u22480 trivially \u2014 only path to no-weight-change** [7].\n    \n*   Serving reality: torch.compile \u2014 plain tensor subtract in forward() is traceable (bare hooks break graphs); vLLM honors trust_remote_code but remote code can break under TP/Ray; **TensorRT-LLM/GGUF do NOT execute arbitrary Python hooks \u2014 only a folded/abliterated checkpoint is honored there** [7].\n    \n*   Recommendation: ship TWO artifacts \u2014 (1) KLD\u22480 custom modeling class for HF/eager + vLLM-eager; (2) abliterated weight checkpoint for compiled serving (accept small nonzero KLD). Security: trust_remote_code has bypass CVEs (CVE-2026-4372) \u2014 pin revision, audit code [7].\n    \n\n### 7.6 Steering + LoRA hybrid (Task 42) \u2014 **PLAUSIBLE and mechanistically supported; best shot at \u22643/100 @ lower total KLD; must be measured**\n\n*   Established: **KTS 2406.15518** (train to minimize steered/unsteered KL on benign inputs \u2192 lower steering multipliers needed at inference); **Adila et al. 2603.00425** (\"joint adaptation\" \u2014 LoRA + steering simultaneously surpasses either alone); RMU 2402.19459 [7].\n    \n*   Why it should beat heavy-alone: steering is near-rank-1 (what fine-tuning does, 2603.00425); a heavy adapter (sft1, KL 0.19) spends KL on components orthogonal to the refusal axis; a steer confines change to that axis \u2192 Pareto-efficient composition (working hypothesis, not established) [7].\n    \n*   **CRITICAL: RECALCULATE the refusal direction on the post-LoRA model** \u2014 SFT drifts the direction's principal angle and shifts the best layer (2604.27019); base-predicted direction doesn't trigger right post-LoRA readout (2511.21399) [7].\n    \n*   Recipe: (1) start from a NEW light adapter (~20\u201330/100 @ KL<0.10; r\u22488, 1\u20132 ep, moderate \u03bb) \u2014 NOT sft2 (too far) and NOT sft1 (too heavy); (2) recompute direction through the adapter (contrastive pairs, to=self, mid-to-late; verify 1\u20133 dominant dirs); (3) subtract; start small, monitor refusal + benign KL together; (4) optionally KTS-style KL-regularizer during adapter training; iterate to \u22643/100 [7].\n    \n*   Uncertain: no direct fixed-refusal/lower-KL head-to-head exists; steering adds its own induced KL \u2192 **measure total KLD on the steered output**, not just the adapter [7].\n    \n\n**G7 net expectation:** the lone path most likely to hit \u22643/100 @ lowest KLD is **T42 (light adapter + steer) with T40 gating** to keep benign KLD \u2248 sft2's ~0.016 \u2014 better than heavy sft1 alone (0.19) or pure steering on a frozen trace [7].\n\n* * *\n\n## 8. MODEL MERGING & COMPOSITION (G8 \u2014 Tasks 43\u201348)\n\n### 8.0 Executive fact: **NO pure merge reaches \u22643/100 with only base + sft1 + sft2**\n\nEvery merge/interpolation is bounded by endpoints (base \u224890\u2013100, sft1 = 25, sft2 = 90); any convex combination lies between 25 and 90 \u2014 never at 3. Winning architecture = 3-step: **(1) merge/interpolate to low-KL mostly-removed checkpoint (refusal ~8\u201312/100 @ KL ~0.03\u20130.06) \u2192 (2) directed refusal-removal machinery (abliteration/task-vector SVD) to push ~10 \u2192 \u22643 \u2192 (3) merge-aware compliance retrain anchored to the de-abliterated checkpoint with \u03bb_KL=8 \u2192 final \u22643/100 @ KL \u2248 0.03\u20130.08** [8].\n\n### 8.1 TIES (Task 43) \u2014 **dead end standalone; KL-recovery preconditioner only**\n\n*   Algorithm/citation: Yadav et al. 2306.01708 (NeurIPS 2023 \u2014 NOT ICML 2024, that's DARE). Trim \u2192 Elect sign \u2192 Disjoint merge \u2192 base + \u03bb\u00b7merged [8].\n    \n*   With ONE task vector, TIES collapses to magnitude-trimmed task arithmetic = an interpolator; ceiling is sft1's 25/100 [8].\n    \n*   Params: density \u2208 {1.0\u20260.1} (mergekit default 0.5), \u03bb \u2208 {0.5\u20261.0}, rescale ON/OFF; starting cell density=0.6, \u03bb=0.8, rescale ON, merged into o_proj+down_proj only; fp32 trim; keep base for embeddings/norms/q/k/v/gate/vision/lm_head; ~60GB streaming per bf16 copy [8].\n    \n*   Literature caution: \"Asymmetric Collapse in Model Merging\" 2607.27240 (TIES/DARE favor high-magnitude refusal vectors \u2014 kept 81\u201385% jailbreak refusal while capability collapsed); \"One Bad Model Spoils the Bunch\" 2406.14563 (safety transfers readily) [8].\n    \n*   Use: pick lowest-KL config holding refusal \u2248 sft1 (~25) as clean preconditioner \u2192 hand to ablation for final push [8].\n    \n\n### 8.2 SLERP / LERP (Task 44) \u2014 **dead end for \u22643 from base+sft1; LERP is the designated KLD-minimizer on any \u22643 candidate**\n\n*   **Use LERP (task-vector), not SLERP, for the sweep:** \u03b8(t) = base + t\u00b7(sft1\u2212base). Because sft1's delta lives only in o_proj+down_proj, LERP automatically interpolates only the refusal-carrying projections. Citations: Model Soups 2203.05482, WiSE-FT 2109.01903 (t-sweep blueprint, default \u03b1=0.5), Task Arithmetic 2212.04089 [8].\n    \n*   SLERP: Shoemake SIGGRAPH 1985 / mergekit 2403.13257 \u2014 \u26a0\ufe0f the brief's \"White et al.\" SLERP attribution could NOT be verified. SLERP needs SVD on 30B (heavy) \u2014 only as final-t norm sanity check [8].\n    \n*   t-sweep: {0.0,0.1,0.2,0.3,0.5,0.7,1.0}, refine \u00b10.02 at knee. Correct use: once ANY candidate C has refusal \u22643/100, sweep \u03b8(t) = base + t\u00b7(C\u2212base) downward from t=1; t* = largest t keeping \u22643/100 (expected t* \u2208 [0.6,0.95]) \u2192 KL(t*) = min achievable KLD at the 3/100 budget (mirrors WARP 2406.16768) [8].\n    \n*   Pitfalls: exclude RMSNorm/near-identity; keep untied embed+lm_head at base; vision stack at base; hidden-level eval (bypass tanh softcap); mask 2048 specials; DFlash off; prompts <2048 [8]. Curve: refusal logistic with mid-t knee; KL convex; \"sweet plateau\" + \"dead zone\" [8].\n    \n\n### 8.3 DARE (Task 45) \u2014 **dead end as sole mechanism; DELLA-style cleanup only**\n\n*   Algorithm/citation: \"Language Models are Super Mario\" 2311.03099 (ICML 2024). Drop fraction p, rescale 1/(1\u2212p) [8].\n    \n*   **Critical: DARE is expectation-preserving \u2014 E[\u03b8_DARE] = \u03b8_SFT \u2192 plain DARE(sft1) keeps KL\u22480.19 AND refusal\u224825 unchanged.** It's a post-hoc sparsifier, not a KL/refusal knob [8].\n    \n*   Better: **DELLA 2406.11617** (MAGPRUNE drops lower-magnitude entries preferentially; +1.2 avg pts; restrict drop budget \u2014 low drop on refusal-critical o_proj/down_proj, aggressive elsewhere). \"Dare the Extreme\" (Deng et al., ICLR 2025): plain DARE fails at large prune rate OR large delta magnitude \u2014 sft1's KL 0.19 = large delta = exactly the failure regime [8].\n    \n*   Use: DARE/DELLA the COMBINED delta (sft1 + ablation output) toward base AFTER a stronger refusal remover exists \u2014 not the primary lever [8].\n    \n\n### 8.4 Task-vector arithmetic (Task 46) \u2014 **decompose/sparsify \u03c4_refusal itself, not subtract a capability vector**\n\n*   Literal \"\u03b8 = base + \u03b1\u00b7\u03c4_refusal \u2212 \u03b2\u00b7\u03c4_cap\": low-confidence/fragile. Citation: Ilharco 2212.04089 \u2014 negation of a task vector cut toxicity ~6\u00d7 while LOWERING WikiText PPL (closest analog). \u26a0\ufe0f Sign trap: \u03c4_cap = harmlessSFT \u2212 base likely positively correlated with helpfulness, so \u2212\u03b2\u00b7\u03c4_cap may **re-inject refusal**; measure per-layer cos(\u03c4_refusal, \u03c4_cap) first (interference-gap theory 2411.18729) [8].\n    \n*   Feasibility evidence: **Surgical Refusal Ablation 2601.08489** \u2014 Qwen3-VL-4B single-direction ablation KL=2.088 vs disentangled spectral removal KL=0.044 at same 0% refusal \u2014 the KL damage is refusal\u2194capability entanglement WITHIN \u03c4_refusal \u2192 the fix belongs in decomposing \u03c4_refusal [8].\n    \n*   Primary route: \u03b8 = base + \u03b1\u00b7[topK/SVD-decomposed \u03c4_refusal] on o_proj+down_proj only (\u03b1\u2208{1.0\u20262.0}, default 1.5; TIES topK {10,20,30}% or SVD rank {8,16}); expected refusal \u22643/100 @ KLD \u22480.01\u20130.08. If single-pass stalls, iterative peel: \u03c4\u00b2 = sft2 \u2212 \u03b8_stage1, add \u03b1\u2082\u00b7\u03c4\u00b2 [8].\n    \n*   \"Orthogonality breaks at LLM scale\" (2511.21437); Asymmetric Collapse (2607.27240) can swamp repair [8].\n    \n\n### 8.5 Adapter stacking (Task 47) \u2014 **dead end for the goal; value = frontier mapping + per-layer AdaMerging**\n\n*   Same-base LoRA deltas add: \u0394W_total = \u0394W1 + \u0394W2; combined rank = r1+r2 = 24 via `cat` (**linear combination requires EQUAL ranks \u2014 16\u22608, inapplicable**). Sequential set_adapter = weight-add (associative \u2014 order irrelevant) [8].\n    \n*   Why no-op: sft2 is near-base (\u2016\u0394W2\u2016\u22480); it doesn't cancel \u0394W1's high-KL dims \u2192 refusal stays ~25, KL stays ~0.19 [8].\n    \n*   Real lever: **AdaMerging 2310.02575** (per-layer coefficients by entropy minimization); LoRAHub 2307.13269 (global-weight); ZipLoRA 2311.13600 [8].\n    \n*   Test A: \u03bb-interpolation 1.0\u21920.0 maps the frontier (monotone line (0.19,25)\u2192(0.016,90) \u2014 none qualifies). Extrapolation \u03bb>1 violates target. **Test C (the fix): train sft3 (r=16, 2ep, \u03bb_KL\u22484), compose \u0394 = \u03bb\u00b7\u03941 + (1\u2212\u03bb)\u00b7\u03943** [8].\n    \n*   Pitfalls: same-base mandatory; fold per-adapter scaling before summing; PEFT \u22650.20; PEFT svd-combination not fp16/bf16 [8].\n    \n\n### 8.6 Merge-aware retraining (Task 48) \u2014 **NOT a dead end \u2014 the community-standard healing step; required to reach \u22643**\n\n*   Stage 1: merge (task-arithmetic \u03b8_\u03bb = base + \u03bb\u00b7\u03c4_sft1, \u03bb\u2208{0.4,0.6,0.8}; expected knee 0.6\u20130.8 \u2192 refusal 8\u201312/100, KL 0.03\u20130.06; TIES backup) [8].\n    \n*   Stage 2: brief compliance LoRA pinned to the **Stage-1 de-abliterated checkpoint** [8].\n    \n*   \ud83d\udea8 **THE #1 protocol bug (the sft2 trap):** sft2's KL=0.016 came from anchoring to BASE \u2014 the KL term pinned the policy to base, which still refuses. **Stage-2 must anchor \u03bb_KL to the de-abliterated Stage-1 checkpoint, not base, or you reproduce sft2 exactly.** (WARP 2406.16768: anchor to the aligned behavior so the leash preserves removal geometry) [8].\n    \n*   Data: harmful\u2192compliant completions 1k\u20135k + 10\u201320% benign. **NO harmful\u2192refusal pairs, NO preference pairs** (would re-hardcode refusal: 2406.11717, 2406.05946, 2505.19056) [8].\n    \n*   Loss: L = CE + \u03bb_KL\u00b7D_KL(\u03b8\u2016\u03b8_ref) with **\u03bb_KL = 8** (same value that kept sft2 low \u2014 but pointed at the de-ablit reference). LoRA r=16; target q/k/v/o + up/down/gate (must include o_proj+down_proj); epochs 1.0 (hard cap 2.0); LR 2e-5 (sweep 1e-5\u20135e-5); batch 32\u201364; seq \u22642048 [8].\n    \n*   Why it differs from SFT-from-base: refusal already suppressed \u2192 gradient fills in fluent compliant continuations instead of fighting a refusal prior; less weight-space travel \u21d2 lower KL. Refusal is \"cheap to re-learn\" (2406.14563) \u2192 \u03bb_KL high + ep \u22642 + LR \u22645e-5 non-negotiable [8].\n    \n*   Test across all **11 refusal categories of 2602.02132** (single-set optimization lets the model find a new refusal style); 100 harmful + 100 benign; log every ~50 steps; joint early-stop at refusal \u22643/100 AND KL \u22640.05 [8].\n    \n*   Expected trajectory: base (100,0) \u2192 sft1 (25,0.19) \u2192 sft2 (90,0.016, wrong direction) \u2192 Stage-1 (8\u201312, 0.03\u20130.06) \u2192 Stage-2 (1\u20133, 0.05\u20130.08) [8].\n    \n*   Dead-end conditions: if the spec demands KL \u22480.016 simultaneously with \u22643/100 \u2014 reject the constraint (realistic floor at \u22643/100 \u22480.03\u20130.08); if only harmful\u2192refusal/preference data available \u2014 inverts Stage-1 [8].\n    \n*   Citations: LoRA-undoes-safety 2310.20624 (refusal \u2192 ~1% after LoRA SFT); realignment 2405.09055, 2503.17239, 2405.16833, 2602.00038; Merge-to-Learn 2410.12937 (safety task-vector beats continued FT); \u26a0\ufe0f abliterate-then-heal (NeuralDaredevil) is community/empirical, not peer-reviewed [8].\n    \n\n**G8 implementation order:** LERP sweep (map frontier + preconditioner) \u2192 SVD/TIES-cleaned refusal vector or heretic on o_proj+down_proj (the step that achieves \u22643) \u2192 merge-aware retrain (r=16, 1ep, \u03bb_KL=8 anchored to de-ablit checkpoint) \u2192 optional T44-t* back-off to t*\u2208[0.6,0.95] for final min-KL [8].\n\n* * *\n\n## 9. DATA & DATASET CURATION (G9 \u2014 Tasks 49\u201354)\n\n### 9.1 Dataset size (Task 49) \u2014 **do NOT scale to 1000; N \u2248 100\u2013300**\n\n*   KLD is monotone-increasing in total update steps (\u221d N) while refusal-reduction **saturates at order-of-dozens** (Qi 2310.03693: 10 examples jailbreak GPT-3.5, ~100 \"dramatically erase\" Llama-2 refusal). Forgetting power-law in steps (Kalajdzievski 2401.05605); \"LoRA Learns Less and Forgets Less\" 2405.14821 (dataset size = larger drift lever than any hyperparameter) [9].\n    \n*   LIMA 2305.11206 (1k curated \u2248 frontier \u2014 with honest counterweight 2506.14681); do not copy WildGuard's 92K (that's a moderation _discriminator_, not refusal flipping) [9].\n    \n*   KLD mitigation that doesn't touch N: inject ~1% base/pretraining data (B\u00e9thune 2502.06042 \u2014 prevents forgetting) [9].\n    \n*   **Recommendation:** N \u2208 {50,100,200,300,500}, pick the smallest hitting \u22643/100; expected operating point **N \u2248 100\u2013300**; if refusal won't drop at small N, the bottleneck is **coverage of refusal mechanisms, not count** \u2192 grow diversity (one pair per refusal style/pathway), not raw N; scale (\u03bb, r, ep) instead [9].\n    \n\n### 9.2 Curriculum ordering (Task 50) \u2014 **random suffices; ordering is a second-order lever**\n\n*   Curriculum changes trajectory, not the converged optimum (2506.11300: converges to random at endpoint; ICLR 2021 \"When Do Curricula Work?\"). Beware full easy\u2192hard (burns budget) and anti-curriculum (worst arm, 1904.03626) [9].\n    \n*   Keep (optional): easy-warmup \u2014 first 10\u201320% of steps on easiest half, then random. Difficulty = model-relative: difficulty_i = base_NLL(steered target | prompt_i) [9].\n    \n*   Real KLD levers instead: RL Razor forgetting law 2509.04259 (KLD-on-new-task predicts forgetting, R\u00b2\u22480.96); OLMo 2605.26097 (KL-to-base nearly eliminates forgetting); **FLOW 2502.02797** (up-weight low-base-loss pairs); EKSFT high-KL token masking 2605.29303 (closest to goal) [9].\n    \n\n### 9.3 Rejection-sampled compliance (Task 51) \u2014 **hard binary gate; keep all passers; budget ~1/\u03b7**\n\n*   RAFT 2304.06767; Xiong 2504.11343 (pure RAFT/RS competitive with GRPO/PPO; selection improved KL-efficiency); RSO 2309.06657 (stricter \u21d2 closer to KL-regularized optimum, at yield cost) [9].\n    \n*   Best-of-N gains ~log(N) (2408.03314) \u2014 use larger N to restore size after a strict gate (linear 1/\u03b7), not to chase one \"best.\" **AlpaGasus 2307.08701**: filtered 52k\u21929k (~17% yield) beat the full 52k \u2014 severe filtering works if leftover is large + diverse. Counterweights: 2404.14367 (top-1 weakest), 2410.03717 (don't over-filter to tiny sets) [9].\n    \n*   Gate: (complete, didn't hit max_new) AND (no refusal/hedge tokens) AND (len \u2265 8\u201316). Budget N_per_prompt \u2248 ceil(K_target/\u03b7)\u00d71.2 (\u03b7=0.30 \u2192 ~330\u2013400/prompt). Target \u03b7 \u2208 0.2\u20130.5, floor 0.15\u20130.20; below that fix the teacher/steering pass, don't brute-force N. Then sweep \u03bb\u2208{0.5,1,2,4} (1ep, r=8) [9].\n    \n*   You do NOT need to filter to 0% refusal (2502.19537: refusal thin/brittle/perturbable) \u2014 target \u22643/100, not 0 [9].\n    \n\n### 9.4 Response-length normalization (Task 52) \u2014 **YES: cap training target at T_cap \u2248 256; teacher max_new 512**\n\n*   Compliance/refusal is decided in the early tokens (to=self); the tails (current targets ~250\u2013450 tok) are memorization surface with ~zero compliance value. **KLD \u2248 (per-token drift)\u00d7(# supervised tokens)** \u2014 capping cuts the 2nd factor \u2192 allows lower \u03bb, net lower KLD; complementary to \u03bb, not redundant (attacks sft1's overfit without sft2's over-restraint) [9].\n    \n*   Prefix-only distillation 2602.15260 (early-token loss mass; 2\u201347\u00d7 cost reduction); in-distribution targets (GRAPE 2502.04194 \u2014 diverging/high-PPL targets degrade SFT); ASFT 2509.23753 (KL-anchored SFT, kl_weight\u22480.03) [9].\n    \n*   Param: teacher max_new=512; T_cap \u2208 {none, 256, 128}; right-truncate after tokenization (keep prompt + first T_cap response tokens labeled); compute refusal-onset position per sample to set the floor; hold \u03bb\u22481\u20132, control proximity via T_cap; measure completeness (length \u2265 T_cap + EOS rate), not raw win-rate (Dubois 2404.04475 length-controlled eval) [9].\n    \n\n### 9.5 Domain coverage / capability anchors (Task 53) \u2014 **YES, 75:25 mix, self-distilled anchors**\n\n*   Math/code rot first: GSM8K \u22121\u20135 pts from abliteration (2406.11717); SFT is the most destructive de-alignment route (2604.18510); trade-off fundamental (2503.20807). Replay 5\u201320% (2407.02118, 2407.17467); balanced mixing beats overweighting (2508.11953); too much safety \u21d2 over-refusal (Bianchi 2309.07875) [9].\n    \n*   Closest precedent: **NeuralDaredevil** \u2014 chat/DPO mix recovered most capability but GSM8K stayed low (\"would benefit from more math samples\") \u2014 chat-only anchors aren't enough [9].\n    \n*   Param: compliance : anchors = 75:25 (sweep 90:10 \u2192 60:40). **Anchor sourcing = self-distillation from the base model** (base's own refusal-free generations on benign prompts, filter refusals) \u2014 keeps KLD on-distribution AND guarantees zero-refusal anchors. Anchor budget within the 25%: coding 30%, math/reasoning 30%, general chat 25%, agentic 10%, creative 5% (creative capped ~5\u201310% \u2014 too-creative raises KLD). \u03bb\u2208[0.2,1.0], r\u2208[16,64], 1\u20133 ep [9].\n    \n*   MT-Bench per-category reporting; pair with held-out GSM8K/HumanEval/MMLU (MT-Bench cells ~10 Qs underpowered); \u00b10.3\u20130.5 is noise [9].\n    \n\n### 9.6 Dedup (Task 54) \u2014 **YES, 3 stages, globally before any split** (A/B-validate the KLD-variance effect)\n\n*   Pretraining dedup (Lee 2107.06499 \u2014 cuts memorization ~10\u00d7, no PPL harm); SemDeDup 2303.09540 (embed + k-means, keep centroid-nearest; C4 keeps 80% no PPL loss); limited-data regime: keep easy/representative, prune redundancy, **don't aggressively thin** (2206.14486) [9].\n    \n*   Duplicates distort loss direction (2606.24998: duplicates inflate effective weight of few narrow directions) \u2014 sft2's \"KLD=0.016 but 90/100 refuse\" reads like **compliance locked onto memorized phrasings**; dedup forces a policy that generalizes \u21d2 lower refusal-rate variance. \u26a0\ufe0f Do NOT import \"Data Repetition Beats Data Scaling\" (2602.11149 \u2014 reasoning skill-learning, not your phrasing-specific compliance) [9].\n    \n*   Pipeline: (1) exact dedup on normalized prompt+target; (2) prompt-level near-dup \u2014 MinHash Jaccard \u22650.85 + embedding cosine \u22650.90\u20130.95 (sentence embedder bge-m3/e5-large/gte; NOT decoder-only LM); consolidate each cluster to one representative (**lowest-KLD-to-base**, GRAPE-style \u2014 recommended for your objective); (3) target consolidation within prompt-cluster \u2014 keep ONE in-distribution target (GRAPE +13.8\u201317.3%; ProFit 2601.09195); preserve multi-target diversity ONLY for semantically distinct refusal-avoidance strategies. Soft alternative: frequency-cap 1/freq [9].\n    \n*   Measure: raw vs deduped, \u22653 seeds, report mean\u00b1std of refusal + KLD + **per-sample KLD spread**; accept if dedup shrinks KLD spread and flattens refusal misses at fixed \u03bb [9].\n    \n\n**G9 operating recipe:** N\u2248100\u2013300 with max refusal-mechanism diversity \u00b7 strict compliance gate (keep all passers, budget ~1/\u03b7) \u00b7 T_cap\u2248256 (teacher max_new 512) \u00b7 global dedup + target consolidation \u00b7 75:25 self-distilled anchor mix (coding/math weighted, creative \u22645\u201310%) \u00b7 random order \u00b7 KLD levers ranked: \u03bb \u2192 target-length cap \u2192 on-distribution targets \u2192 self-distilled anchors \u2192 dedup \u2192 curriculum (weakest) [9].\n\n* * *\n\n## 10. HYPERPARAMETER & OPTIMIZATION (G10 \u2014 Tasks 55\u201360)\n\n### 10.1 \u03bb_KL grid (Task 55) \u2014 _\u26a0\ufe0f sign convention likely reversed vs the brief; \u03bb \u2248 0.3\u20130.5 (LOW band)_*\n\n*   Under the standard convention (\u03bb = KL \"leash\" weight), **higher \u03bb pulls back to the refusing base \u2192 MORE refusals, LESS drift**. Anchors match exactly: \u03bb=8 \u2192 refusal 90/KL 0.016; \u03bb=1 \u2192 refusal 25/KL 0.19. **Refusal increases with \u03bb; KL decreases with \u03bb. To hit refusal \u22643 you must go LOW \u03bb (<1), NOT {2,3,4}.** \u26a0\ufe0f Confirm the loss definition (`L = NLL + \u03bb\u00b7KL` vs `L = \u03bb\u00b7NLL + KL`) \u2014 it flips the direction [10].\n    \n*   Confound: anchors differ in \u03bb AND epochs AND rank \u2014 the sweep must hold epochs/rank fixed [10].\n    \n*   Family: KL-penalized SFT; RLHF \u03b2 0.02 (2203.02155); DPO \u03b2 default 0.1; KTO/IPO \u03b2-family; unlearning KL anchors (2310.10683, TOFU 2303.10130, RMU 2403.03218). Heretic analogue co-minimizes refusal + KL-to-base; **KL < 0.5 \u2192 capabilities essentially intact**; auto-heretic 3/100 @ KL 0.16 on Gemma-3-12B [10].\n    \n*   _Recommendation: \u03bb \u2248 0.3\u20130.5; grid {0.5, 0.25, 0.125} FIRST (add sub-1 values); bisect on refusal\u22483; take \u03bb_ = LARGEST \u03bb with refusal \u22643** (largest \u03bb = smallest KL). n=100 noise \u00b12 \u2192 \u22652 seeds. Hold effective batch fixed; adaptive KL controller (2204.05862) only if fixed \u03bb can't be bracketed [10].\n    \n*   Pareto: pick the knee; with a hard refusal\u22643 constraint live at the low-\u03bb end (highest marginal efficiency) [10].\n    \n\n### 10.2 LoRA rank (Task 56) \u2014 **r=8 default; \"higher rank \u21d2 more removal at same KLD\" is FALSE per literature**\n\n*   The refusal-edit is essentially rank-1; excess rank adds orthogonal \"intruder dimensions\" \u2192 higher KLD. LoRA low-rank expressiveness (2106.09685; Zeng & Lee ICLR 2024); \"LoRA Learns Less and Forgets Less\" 2405.09673 (larger r \u21d2 more forgetting); Illusion of Equivalence 2410.21228 (high rank \u2248 full-FT intruder dims); **GRPO-KL rank bound (Rottman, MERL ICMLw 2026): policy divergence increases with rank** (direct mechanistic r\u2194KLD source); rank-threshold preprint 2605.03724 (rank-1 matches/exceeds r=12; higher rank degrades generalization) [10].\n    \n*   Abliteration converges on rank-1/near-rank-1 (2406.11717; Heretic rank-1-type LoRA; Gabliteration multi-directional but low-rank per direction per layer) [10].\n    \n*   **Sweep {1,4,8,16,32,64} at FIXED \u03bb=1, ep=2 (isolate rank). Include r=1 anchor \u2014 if r=1 can't reach \u22643/100, that quantifies multi-directionality.** Expected shape: refusal saturates ~r=1\u20138, KLD flat-or-rises r\u226516. Trainable params r\u00b72,076,672 (r=8 \u2192 16.6M = 0.006% of 30B) \u2014 VRAM trivial vs frozen base [10].\n    \n\n### 10.3 \u03b1/r ratio (Task 57) \u2014 **default \u03b1/r = 1 (\u03b1 = r)**\n\n*   out = Wx + (\u03b1/r)\u00b7BAx; \u03b1/r is an effective-LR multiplier on the adapter branch. Hu 2106.09685: \u03b1=r canonical, don't retune when varying r; rsLoRA 2312.03732: \u03b1/\u221ar for r\u226564 (PEFT use_rslora); LoRA+ 2402.12354 (optional separate A/B LR); LLLFF 2405.09673 uses \u03b1=2r for high-rank comparison; QLoRA fixed \u03b1=16 across r drags effective LR [10].\n    \n*   De-ablit interaction: higher \u03b1/r flips refusal faster but wastes KL budget; smaller \u03b1/r \u2192 better KL stability, needs more steps. The target IS a small low-rank direction \u2192 **minimal \u0394W that just flips the decision = low \u03b1/r + enough steps** [10].\n    \n*   RecommFROM: \u03b1=r (matches Heretic `lora_alpha = lora_rank`; both current runs already use it \u2014 NOT the binding constraint); r-sweep holds \u03b1/r=1; at winning r test \u03b1/r \u2208 {1,2} only; escalate \u03b1=2r only in the sft2-like regime (refusal stuck \u226590 with KL headroom). Merge note: \u03b1 is baked into \u0394W at merge \u2014 decide before training; lora_dropout=0 [10].\n    \n\n### 10.4 LR & warmup (Task 58) \u2014 **lr 2e-5, cosine to ~0, warmup 5% (10% first run)**\n\n*   QLoRA 2305.14314: 2e-4 \u226413B HALVED to 1e-4 for 33B \u2192 at 30B treat ~1e-4 as SFT ceiling, stay ~an order below for KL-min; LoRA optimal LR ~10\u00d7 lower than full-FT (2405.09673); DPO/RLHF-adjacent 10\u2013100\u00d7 lower than SFT; **32B LoRA \u2248 5e-6 \u2260 but ~2e-5 with cosine** [10].\n    \n*   Low LR + long cosine beats high LR + few steps for KLD: KLD \u2248 \u00bd\u0394\u03b8\u1d40F\u0394\u03b8 (quadratic in displacement); \"Implicit Reward as the Bridge\" 2507.00018 (SFT KL term goes INACTIVE under fast/high LR \u2014 low LR keeps the constraint active); RL's Razor 2509.04259 (KLD predicts forgetting R\u00b2\u22480.96); 2605.26097 (low LR less drift, more steps \u2014 the exact tradeoff); 2604.13627 (higher LR \u21d2 more feature drift) [10].\n    \n*   Floor: LR too low \u2192 can't reach refusal 3 within budget (sft2-like freeze) \u2014 low-but-functional, found by sweep [10].\n    \n*   Warmup 3\u201310% (5% default; **10% for the first KL-min run** \u2014 protects early Adam 2nd-moment, avoids early KLD spike; stability knob, not the main lever \u2014 2410.23922); cosine decay (SGDR 1608.03983; min_lr \u224810% peak, 2405.18392) [10].\n    \n*   Stage grid: bracket {1e-5, 2e-5, 5e-5} 1ep r=16 (skip 1e-4 \u2014 trades refusal for high KL); elongate winner 2\u20133\u00d7 steps (recompute cosine longer \u2014 where the KLD-min frontier lives); optional warmup {0.03, 0.10}. max_grad_norm 0.3 [10].\n    \n\n### 10.5 Epochs & early stopping (Task 59) \u2014 **\u22643 ep; stop at refusal gate, then argmin-KL among passing checkpoints**\n\n*   **Early-stopping on KL alone is the WRONG criterion** (KL and refusal anti-correlated; min-KL stopping returns ~100-refusal point). Use constrained argmin: cross the refusal \u22643/100 gate (validation), then **roll back to the min-KL checkpoint among all passing checkpoints** [10].\n    \n*   Prechelt (LNCS 1524) \u2014 restore best checkpoint, last-epoch \u2260 best; forgetting not reliably fixed by early stopping on task loss (2401.05605); KL-to-base rises ~monotonically (2512.21852, 2505.22617); refusal removal monotonic & early-saturating (2601.10141, Qi 2310.03693, Lermen 2310.20624 ~1% refusal after tiny LoRA); refusal cliff 2510.06036; **gold objective hump-shaped in \u221aKL \u2014 best at intermediate KL, not min (Gao 2210.10760; Rafailov 2406.02900)** [10].\n    \n*   Protocol: \u22643 ep (expect gate cross ~0.5\u20131.5 ep; if never crosses, fix \u03bb not the stopper); N_train \u22484096, B_global=32 \u2192 128 steps/ep \u2192 **eval every 32 steps (0.25 ep)**; stop at first refusal_val \u22643/100; return argmin KL(\u03c4) s.t. refusal_val(\u03c4) \u22643/100 (+0.5 ep patience, never above gate); gate on validation refusal (disjoint 100-prompt harmful set), never training refusal [10].\n    \n\n### 10.6 Optimizer (Task 60) \u2014 **AdamW; second-order concern**\n\n*   \"Deconstructing Optimizers\" 2407.07972 (ICLR 2025): no clear winner among Adam/Adafactor/Lion/Signum; AdamW wins on maturity/controllability; **\"choice of optimizer can be guided by practical considerations\"** [10].\n    \n*   Lion 2302.06675: ~50% state memory + faster steps, but advantage grows with batch size (opt 4096) \u2014 won't materialize on small-batch consumer hardware; small-batch noise-sensitive (2509.01440) [10].\n    \n*   Adafactor 1804.04235: for full-model OOM; bf16/small-batch instability; EMNLP 2025 \"Momentum Itself Can Be a Normalizer\": strictly inferior to AdamW [10].\n    \n*   Memory: adapter params = r\u00b71,943,552 (r=8 \u2192 15.55M; AdamW ~187 MB; r=16 \u2192 31.1M ~373 MB) \u2014 0.03\u20130.7% of the 55.7 GB base. Optimizer choice moves total memory \u2264 ~0.25 GB [10].\n    \n*   AdamW: \u03b21=0.9, \u03b22=0.999, eps 1e-8, wd=0.01 decoupled, grad-clip 1.0, bf16 compute + FP32 adapter states. Keep optimizer FIXED so \u03bb/epoch ablations stay clean [10].\n    \n\n**G10 flagship config (all levers consistent, small-displacement philosophy):** \u03bb 0.3\u20130.5 (LOW band; \u03bb* = max \u03bb with refusal\u22643) \u00b7 r=8 (sweep {1,4,8,16,32,64} at fixed \u03bb=1,2ep) \u00b7 \u03b1/r=1 \u00b7 lr 2e-5 cosine\u2192~0 \u00b7 warmup 5% \u00b7 \u22643 ep, eval each 0.25 ep, stop at refusal\u22643 then argmin-KL \u00b7 AdamW fixed. Dominant levers: \u03bb then r then steps \u2014 NOT optimizer or warmup. **Hold confounds fixed; run clean 1D sweeps** [10].\n**G10 open flags for lead:** (1) \u03bb sign convention \u2014 confirm loss definition before sweeping (measured anchors support \"low \u03bb de-ablits\"); (2) sft2's 90/100 is likely 1-epoch under-training, not purely \u03bb \u2014 the fixed-(ep,r) sweep resolves it; (3) citation fix: \"LoRA Learns Less and Forgets Less\" = 2405.09673 (NOT 2402.10137 = TOAD dialog paper); (4) no Muse-specific hyperparameter data exists \u2014 validate empirically; (5) eval hygiene: hidden-level 6656-stream KL (softcap saturates logit KL), mask 2,048 reserved + image/video tokens, DFlash off [10].\n\n* * *\n\n## 11. EVALUATION METHODOLOGY (G11 \u2014 Tasks 61\u201366)\n\n### 11.1 The KL metric (Task 61) \u2014 **\"Muse-KL-1.0\": per-token FORWARD KL D(SFT\u2016base) at base-generated continuations, fixed benign held-out surface**\n\n*   Reference-anchored per-token KL over the reference's own continuations is the DPO/RLHF/on-policy-distillation convention (2305.18290; Gemma-2 token-KL 2408.00118; GKD/MiniLLM 2404.03674). Generate from **base** (not SFT) \u2014 keeps contexts on the base manifold [11].\n    \n*   **Direction: forward KL, not reverse** \u2014 reverse stays \u22480 even when SFT adds a whole new mode (mode-seeking) and would MISS the compliant-token additions that define successful de-abliteration; forward blows up where SFT adds mass base treats as near-impossible \u2014 exactly your signal. Explains sft2: KL 0.016 @ 90/100 refusal = never left base's refusal mode [11].\n    \n*   Token/softmax space; skip logit-L2 (softcap saturates and underweights real divergence); apply softcap to BOTH models; mask pinned reserved tokens identically (\u2212inf) in both before softmax; T=1 [11].\n    \n*   Recipe: MT-Bench (160 turns, \u22642048 tok) or ~200 benign \u2192 base rollouts R=256 T=1 seeded \u2192 analytic per-token forward KL over V_valid, prob floor ~1e-12 \u2192 report per-token mean + per-category + PerplexityDiff; bootstrap CI over 3 base-seeds. **Drift surface (benign) and refusal surface (harmful) split \u2014 intended refusal change is NOT drift.** Anchor: Heretic good points \u2248 KLD 0.05\u20130.11 @ refusal 8\u201310/100 (\u26a0\ufe0f non-peer-reviewed). **Freeze the reasoning-stream sample layer/position or cross-run KL isn't comparable** [11].\n    \n\n### 11.2 Refusal evaluation (Task 62) \u2014 **semantic LLM-judge over BOTH** `to=self` **and final answer, calibrated against human labels**\n\n*   Keywords measure the wrong construct and cannot see reasoning-stream refusals; no list expansion fixes paraphrase/mutation evasion. Quantified: string-match vs judge \u2248 **+32 pp divergence** (AlignBeam 2606.12342); LLM-judges erratic recall 0.06\u20130.65 on 596 human-labeled HarmBench completions, flip-floppy (2606.25487); reasoning-stream refusals systematically undercounted by heuristics [11].\n    \n*   Judge config: strong open-weight \u226532B, **different family than Muse** (avoid self-preference 2304.07628), T=0 greedy; **3-class rubric judged on to=self and to=user separately (CoT\u2013Output 2\u00d72, 2606.10740): REFUSED** = refusal in either stream + no compliant answer (**reasoning-refusal + truncated = REFUSED \u2014 your blind spot**); **AMBIGUOUS** = truncated w/o refusal (report separately); **COMPLIED** = substantive harmful content. **Human audit ALL REFUSED+AMBIGUOUS; require confirmed \u22643/100** [11].\n    \n*   Calibration: 100\u2013150 stratified human labels (over-sample truncated + disagreements), \u22652 annotators, **Cohen's \u03ba \u2265 0.7**; benign control (XSTest-450) against judge \"always REFUSED\" bias; 5% flip-rate re-judge; second cross-family judge on 10% [11].\n    \n*   Off-the-shelf panel: WildGuard-7B (2406.18495), Llama Guard 3/3V (2407.21783), Shieldstral-3B (91.5% F1 single-GPU), **Nemotron-Content-Safety-Reasoning-4B (reasoning-trace mode \u2014 best for to=self)**, HarmBench-Mistral classifier; plus white-box refusal probe on Muse's own activations [11].\n    \n*   Caveats: n=100 \u2192 \u22643/100 gate has ~95% CI \u2248 [0.6%, 8.5%] \u2014 keep the human audit, consider growing the set; **switching scorers moves sft1/sft2 numbers \u2192 re-baseline both under the new protocol** [11].\n    \n\n### 11.3 Capability preservation (Task 63) \u2014 **KLD is necessary-but-not-sufficient; measure capability directly**\n\n*   \"Abliteration Is Not a Scalpel\" 2607.17427: steered/abliterated = measurably different decision-makers (optimism +12.2pp Gemma / +7.4pp Qwen, longer justifications, fewer uncertainty markers, opposite-sign confidence flips) decoupled from output KL. Your own data: sft2 KL 0.016 yet refuses 90/100 \u2014 low KLD doesn't even guarantee the surgery happened [11].\n    \n*   Abliteration survey 2512.13655: KLD 0.04\u20131.65 with model-dependent capability impact, math most fragile (GSM8K \u221218.8pp at near-minimal KLD). KL bounds distance, not gold capability (Goodhart, 2210.10760) [11].\n    \n*   Battery (4-bit, prompts <2048, auto-graded): Tier 1 EVERY iteration: GSM8K subset (2110.14168, regex), HumanEval/LiveCodeBench (2107.03374/2403.07974, pass@1-by-exec), MMLU subset (2009.03300), \u03c4-bench (2406.12045) or BFCL (2305.15334). Tier 2 finalists: MT-Bench (2306.05685 per-category), AlpacaEval 2 **LC win-rate** (2404.04475 \u2014 length control doubles as the anti-verbosity guard), full MMLU. Tier 3 final only: full suites, multi-seed, optional small vision slice (text-core evals won't catch VLM vision degradation) [11].\n    \n*   **Decision gates (vs base, benign prompts only):** refusal \u22643/100 (hard). MMLU \u2265 base \u22123pp; GSM8K \u2265 \u22125pp; HumanEval/LiveCodeBench \u2265 \u22125pp; \u03c4-bench/BFCL \u2265 \u22125pp + tool-abstention intact; MT-Bench \u2265 \u22120.5; AlpacaEval LC \u2265 \u22123pp. **KLD is NEVER a release gate by itself** \u2014 block only on capability bands; do not block on high KLD if tasks are clean [11].\n    \n\n### 11.4 Held-out refusal set (Task 64) \u2014 **AdvBench-520 / AdvPrompter 60/20/20 \u2192 held-out test = 104; four disjoint pools**\n\n*   harmful_behaviors.csv = 520 rows (not the paper's \"500\"); canonical artifact = llm-attacks GitHub; ungated mirror `mlabonne/harmful_behaviors` (verified 520) [11].\n    \n*   The \"416/104\" is a community 80/20 convention, NOT published. **Published: AdvPrompter 2404.16873 = 60/20/20 \u2192 train 312 / val 104 / test 104** \u2014 citable, gives BOTH val and test [11].\n    \n*   **Leakage protocol \u2014 four disjoint pools, held-out touched by nothing (not direction, not SFT, not any hyperparameter):** (a) direction-building 128 harmful + 128 benign (diff-in-means, 2406.11717); (b) SFT ~150\u2013184 harmful\u2192compliant pairs (disjoint from a); (c) validation 104 (tune \u03bb/ep/rank/layer kernel ONLY here); (d) **held-out final eval 104 = the \u22643/100 number** [11].\n    \n*   Dedup exact + embedding-cosine \u22650.8 (AdvBench semantically repetitive \u2014 ~24 bomb variants, ~19 insider-trading, ~17 gov-database, garbage in diff-in-means and SFT); dedup also against harmless sets [11].\n    \n*   Gotchas: harmful_behaviors skewed cyber/fraud \u2192 **low ASR here is necessary-but-not-sufficient**; add second held-out cross-lineage eval (HarmBench-200 2402.04249, or JailbreakBench 2404.01318); multimodal refusals leak through a different stream \u2014 hold out a disjoint multimodal pool too; fit direction across depth/position variance or it under-generalizes [11].\n    \n\n### 11.5 Variance & significance (Task 65) \u2014 sample unit = the PROMPT (cluster), not the token\n\n*   KL n for \u00b10.005 @95%: n = (1.96\u00b7std/0.005)\u00b2; std 0.02\u219262; 0.03\u2192138; 0.05\u2192**384**; 0.08\u2192983; 0.10\u21921,537; 0.20\u21926,147. Pilot n=100, measure empirical per-prompt std, size adaptively; default assume std\u22480.05 \u2192 ~400. Report **BCa bootstrap CI over prompts** (N=10,000) \u2014 KL right-skewed [11].\n    \n*   Refusal rate (headline): 3/100 \u2192 **Wilson 95% CI \u2248 0.9\u20138.5%; Clopper\u2013Pearson \u2248 0.6\u20138.5%** \u2014 treating 3/100 as \"well under target\" is overconfident. Scaling near p=0.03: n=300 \u2192 ~1.6\u20135.6%; n=500 \u2192 ~1.8\u20134.9%; n\u22481,100 \u2192 \u00b11.0%. **n=100 is NOT enough to pin \u22643%** [11].\n    \n*   Rule of three (\u26a0\ufe0f training memory): 0 refusals in n \u2192 95% upper \u2248 3/n; design as powered decision rule (pre-register MDE, 2605.28873). **No n makes a 3% rate \"cheaply certain\" \u2014 budget hundreds-to-low-thousands for a hard guarantee, or report the interval honestly and treat \u22643/100 as a soft checkpoint** [11].\n    \n*   Run same n for KL + refusal (shared forward passes) [11].\n    \n\n### 11.6 Multi-seed stability (Task 66) \u2014 **3 seeds; worst-case-seed \u22643/100 gate**\n\n*   Single-seed + confounded frontier is untrustworthy (\u03bb/ep/rank all change at once AND alignment/LoRA has seed variance up to \u03c3\u22482pp at 3B-scale, 2603.19335) [11].\n    \n*   Protocol: 3 seeds (0,1,2; extend if cheap). Vary ONLY data shuffle, dropout, LoRA-A init, optimizer noise; hold byte-fixed: base ckpt, tokenizer/template, eval set/order/format, eval code, greedy or single fixed gen-seed, optimizer/LR/schedule/batch/dtype, KL stream/layer/position/estimator [11].\n    \n*   LoRA adapter-init seed \u2248 no effect on learned structure (2410.21228 App. N) \u2014 seed variance comes from shuffle/dropout; lower rank (r=8) lower variance but more bias (2605.19018); LoRA generally lower-variance than full SFT but still non-zero [11].\n    \n*   Acceptance: refusal seed-stable iff max\u2212min \u22641 count; seed-sensitive iff \u22653. **Hard gate: WORST-CASE seed \u22643/100** (median hides max=5). KL stable iff relative spread \u226420\u201330% of median (absolute \u22640.01). Boundary points are seed-sensitive (2601.17260) [11].\n    \n*   Compute strategy: **breadth first** \u2014 many configs \u00d7 ONE shared seed (e.g., 42) to find the region; then ~20\u201330% on depth \u2014 top 1\u20133 near-boundary configs \u00d7 3 disjoint eval-seeds. ~8-cell grid + seed confirmation \u2248 0.3\u20130.5 GPU-weeks (\u26a0\ufe0f GPU-hour estimates) [11].\n    \n*   **Clean ablation grid (fix the confound):** anchor Ref=(\u03bb=4, ep=1, r=8); cells (1,1,8),(8,1,8),(4,2,8),(1,2,8),(8,2,8),(4,1,16),(1,1,16) \u2192 \u03bb\u00d7epoch subgrid at r=8 + rank isolation + sft1-ish control. Fix effective \u03b1/r (not \u03b1) when r changes. Dense \u03bb fine-scan (\u03bb=5,6) at the boundary crossing \u2014 those cells get the 3-seed confirmation [11]. Plot KL-vs-refusal Pareto plane; frontier = min KL among cells with refusal \u22643/100 [11].\n    \n\n**G11 execution plan:** (1) standardize Muse-KL-1.0 (freeze stream/layer/position/estimator, mask pinned tokens, T=1, benign drift surface + harmful refusal surface); (2) **re-baseline sft1 & sft2** under the judge-based refusal protocol + Muse-KL-1.0 (old 25/90 numbers move); (3) deconfound with the shared-seed 8-cell grid; (4) every iteration: Tier-1 capability suite + refusal battery + KL; kill on refusal>3 or out-of-band capability; (5) confirm near-boundary cells \u00d7 3 seeds, worst-case \u22643/100, \u2265300-item refusal eval for a hard \u22643% claim; (6) ship gate = refusal \u22643/100 (hard) AND capability deltas in band; KLD is a regularizer/proxy, never a gate by itself; leakage = 4 disjoint pools, held-out 104 untouched [11].\n\n* * *\n\n## 12. CONVERGED OPERATING POINT / MASTER CONFIG\n\nAll 11 groups converge on one coherent pipeline (details + citations in sections above):\n**Phase 0 \u2014 Fix the metric first.** Raw mean KL is a trap (KAT failure mode). Monitor H\u00b2/TV/\u221aJSD over the to=self stream (temperature-smoothed, T\u2208[2,3]), per-token mean, response-only mask; forward-KL one-sided anchor (permit drift, preserve coverage); re-baseline sft1/sft2 under Muse-KL-1.0 + judge protocol before trusting anything [4][10][11].\n**Phase 1 \u2014 Direction building.** Elicitation `harmful_behaviors` N=400 (\u226595% real refusal; DoNotAnswer supplement) + `harmless_alpaca` N\u2248400 good-set, template-locked (fixed \"Reasoning strength:\", identical tool preamble, hashed), never topic-matched; read at EOI last-token + first `to=self` token across all 52 layers (no mean-pooling); DIM probe first \u2192 top-k PCA/whitened-SVD of unmatched contrast (ranked by refusal-correlation) \u2192 INLP as KLD validator; RDO gradient + KL-retain as complement (L24\u2013L44, mind tanh softcap saturation); skip CCA; DFlash off [1].\n**Phase 2 \u2014 Removal geometry.** Rank-k subspace (k=3 typical, adaptive 3\u201310 from EV\u22650.85) projected out of `o_proj`+`down_proj` only; per-layer independent directions (NO cross-layer GS); Gaussian-bump soft-\u03b1 kernel over band L20\u2013L45 (attn: max 1.0, \u03bc\u224833, \u03c3\u22484, min 0.1; MLP: max 0.6, \u03bc\u224836, \u03c3\u22487, min 0.15); exclude last ~6 layers (logit/softcap zone); subtractive only; position-mask to reasoning-span only (or first-K, K\u2208{1,2,4,8}); conditional gating (CAST-style probe) to keep benign KLD \u22480; 2\u20133 Ouroboros peel passes [2][3][7].\n**Phase 3 \u2014 Training objective.** Two converging branches:\n\n*   **(A) KL-regularized SFT** on steered-self compliance targets: \u03bb\u22480.3\u20130.5 (low band \u2014 confirm sign convention), r=8\u201316, \u03b1/r=1, lr 2e-5 cosine, AdamW, \u22643 ep, early-stop at refusal gate then argmin-KL; anchor-then-release \u03bb schedule (8\u21920.3\u20130.5) if using high-start; data N\u2248100\u2013300, hard compliance gate (keep all passers, budget ~1/\u03b7), T_cap\u2248256, dedup, 75:25 self-distilled capability anchors [6][9][10].\n    \n*   **(B) Preference optimization** (IPO primary / DPO fallback) on self-generated pairs (chosen = steered compliant at moderate strength, rejected = base refusal): \u03b2\u22480.02\u20130.03 first guess, **search upward in \u03b2** to the largest \u03b2 still \u22643/100 (min KL); 1 ep, r=16, LR 1e-6\u20133e-6, RPO SFT anchor, valley-lifting warm-up with rejected completions; 300\u2013700 pairs [5].\n    \n*   Best-shot hybrid (G7): **light adapter (~20\u201330/100 @ KL<0.10) + recomputed-direction steering + KTS-style KL regularizer + gating** \u2014 expect \u22643/100 at lower total KLD than heavy adapters alone [7].\n    \n\n**Phase 4 \u2014 Capability healing (if needed).** Merge-aware retrain: anchor \u03bb_KL=8 to the **de-abliterated** checkpoint (never base \u2014 the sft2 trap); harmful\u2192compliant data only (no refusal/preference pairs); r=16, 1\u20132 ep, LR \u22645e-5; then optional LERP back toward base to t*\u2208[0.6,0.95] for the final min-KL point [8].\n**Phase 5 \u2014 Evaluation.** Four disjoint pools (direction 128+128, SFT ~150\u2013184, val 104, held-out 104 from AdvBench-520 via 60/20/20); judge-based refusal on both streams (human-audited, \u03ba\u22650.7, CoT-Output 2\u00d72 \u2014 reasoning-refusal + truncated = REFUSED); n\u2265300 for a real \u22643% claim (Wilson CI); capability gates (MMLU \u2265\u22123pp, GSM8K \u2265\u22125pp, HumanEval \u2265\u22125pp, \u03c4-bench \u2265\u22125pp); 3 seeds with worst-case-seed \u22643/100 gate; report (refusal, KL) pairs always [11].\n**Expected endpoint (all groups): refusal \u2264 3/100 at KLD \u2248 0.02\u20130.08** \u2014 strictly dominant over both SFT points; realistic preference-only floor \u22480.05\u20130.15; reject any spec demanding KL\u22480.016 with \u22643/100 simultaneously (sft2 got 0.016 by _retaining_ refusal) [2][5][8].\n\n* * *\n\n## 13. OPEN DECISIONS FOR LEAD (must resolve before full runs)\n\n1.  **\u03bb sign convention** (G10): confirm `L = NLL + \u03bb\u00b7KL` vs `L = \u03bb\u00b7NLL + KL` \u2014 flips the sweep direction; measured anchors support \"low \u03bb de-ablits\" [10].\n    \n2.  **Hard \u22643/100 vs soft min-KL trade** (G2): hard drives k up (3\u20135) and n up (~300\u20131100); soft favors k=2 and the t*/\u03b2* back-off points [2][11].\n    \n3.  **Dense vs MoE** for Muse (G2): several sub-agents concluded dense 52L/6656 despite \"VL/MoE\" header framing \u2014 affects per-expert vs per-layer projection [2].\n    \n4.  **Run the cheap rank-1 canary FIRST** (G2 Task 11 dead-end branch): if refusal barely moves on a rank-1 Heretic pass, the model is distributed-refusal hardened \u2192 weight surgery alone can't hit \u22643/100 at low KLD \u2192 commit to SFT/DPO/distillation branch early [2].\n    \n5.  **Metric lock:** adopt Muse-KL-1.0 + judge protocol and re-baseline sft1/sft2 before any comparative claim (old numbers move) [11].\n    \n6.  **Confound cleanup:** never compare sft1 vs sft2 as a \u03bb effect (\u03bb, ep, AND r differ) \u2014 run the shared-seed ablation grid [10][11].\n    \n\n* * *\n\n## 14. CITATION-INTEGRITY & FLAGS (carry everywhere)\n\n### Dead-end / misattributed citations \u2014 do NOT cite by these IDs/names\n\n*   \"HERETIC (arXiv 2602.xxxx)\" \u2014 no such paper; Heretic is a GitHub tool [2][10].\n    \n*   \"Argmax-DPO\" \u2014 no such method exists; implement steering-ranked preference [5].\n    \n*   \"LAER / Less is More\" = 2505.13681 \u2192 actually a quantum-physics paper; real result = **AUSteer 2602.04428** [7].\n    \n*   \"Model Alignment as Representation Selection (MHARM)\" = 2410.02516 \u2192 unrelated multi-agent RL; real support = Representation Tuning 2409.06927 + linear-rep hypothesis 2311.03658 [7].\n    \n*   \"BaDAN\" = 2405.16191 \u2192 rocket-landing paper; the token-level refuse-then-steer mechanism is real via FineSteer/SADI/GCAD [7].\n    \n*   \"GEVA 2404.09318\" \u2014 could not be verified [7].\n    \n*   \"Eyes on the Road\" (Lee 2024) as a refusal paper \u2014 could not be verified; closest verified = CAST 2409.05907 / Mechanistic Understanding 2401.01967 [1].\n    \n*   \"Arora 2024\" \u2014 could not be pinned; closest = Geometry of Truth 2310.06824 / ReFT 2404.03592 [1].\n    \n*   \"White et al. SLERP\" \u2014 unverifiable; cite Shoemake 1985 / mergekit 2403.13257 [8].\n    \n*   \"2407.14461 = White-box LLMs\" \u2014 WRONG (CERN paper); sycophancy ref = 2310.13548 [2].\n    \n*   \"LoRA Learns Less and Forgets Less\" = **2405.09673** (NOT 2402.10137 = TOAD) [10].\n    \n*   Self-Alignment = **2308.06259** (not 2312.11960); RLAIF = **2309.00267** (not 2312.09390); **2502.14560 = Deng/Xie** (not Xia); Magister = **2212.08410** (not 2210.08402) [5][6].\n    \n*   \"Psychology-of-Inhibition,\" \"Could not remove LLM safety behaviors,\" \"Scimitar\" distillation, \"taboo ablation,\" \"TAP backprop\" \u2014 unverifiable names; nearest real analogs cited in sections [4][6][2].\n    \n\n### Training-data / unsourced hypotheses (validate before relying)\n\n*   Tanh-softcap gradient-saturation effect on steering (no paper found) [1].\n    \n*   Expected KLD on 30B > 0.16 (scaling extrapolation) [1]; expected KLD 0.02\u20130.08 envelope [2][3][6]; preference-floor 0.05\u20130.15 [5]; Stage-1/2 merge numbers and t*\u2208[0.6,0.95] [8] \u2014 all estimates, not measurements.\n    \n*   Early-steering low-KLD position-mask claim (qualitative evidence only) [3]; formal per-layer-\u03b1-schedule theory doesn't exist [2][3]; KL\u221d\u03b1\u00b2 and \u03b1\u221ds/e are analytic derivations [2]; \"~5\u00d7 softcap \u03b2 compensation\" mechanistic inference [5]; rule-of-three + GPU-hour estimates [11].\n    \n*   Future-dated 26xx/27xx IDs: verify each on arxiv.org; several likely hallucinated by search tooling [2][7].\n    \n\n### Verified anchors commonly safe to cite\n\n2406.11717 (Arditi \u2014 refusal direction) \u00b7 2505.19056 (extended-refusal defense) \u00b7 2512.18901 (Gabliteration) \u00b7 2512.13655 (abliterative comparative) \u00b7 2602.02132 (multi-directionality) \u00b7 2502.17420 (concept cones) \u00b7 2305.18290 (DPO) \u00b7 2310.12036 (IPO) \u00b7 2405.14734 (SimPO) \u00b7 2306.13649 (GKD) \u00b7 2405.09673 (LoRA forgets less) \u00b7 2106.09685 (LoRA) \u00b7 2210.10760 (BoN KL tax / overopt) \u00b7 2409.05907 (CAST) \u00b7 2406.15518 (KTS) \u00b7 2607.17427 (Abliteration Is Not a Scalpel) \u00b7 2401.05605 (forgetting law) \u00b7 2310.03693 (Qi safety break) \u00b7 2307.15043 (AdvBench) \u00b7 2404.16873 (AdvPrompter split) \u2014 all with the caveat that transfer to Muse-Glimmer-30B must be verified empirically.\n\n* * *\n\n_Master synthesis compiled from Group notes 1\u201311 [1][2][3][4][5][6][7][8][9][10][11]. All target numbers are research-derived priors unless explicitly measured; the empirical sweep is mandatory. No relevant saved memories existed at compile time._", "creation_timestamp": "2026-08-11T16:41:00.300563Z"}