Weight-Space Geometry of Offline Reasoning Training
Six offline RL methods (RFT, RIFT, DFT, Offline GRPO, DPO, SFT) tested on identical math rollouts from Qwen 3–4B. Analysis via cosine similarity, principal-angle subspace, linear mode connectivity, and CKA reveals mechanistic overlap.
SFT, RFT, and RIFT converge to nearly colinear weight deltas (cosine ≥ 0.97, top-1 principal angle ~7° median). All six methods achieve comparable downstream accuracy on GSM8K despite structural differences.