Skip to content- Sources: primary
- Summary: The authors post-train several base model families with SFT and DPO on helpfulness and summarization, holding the annotation budget in dollars fixed instead of the sample count. Synthetic preference data reaches high win rates with as little as $200 to $400, human data closes the gap on helpfulness only at roughly two orders of magnitude more spend, and on summarization it never closes the gap at the tested budgets. One widely used human preference source underperforms another even after domain restriction and deduplication.
- Why it matters: When budgeting a preference-tuning run, synthetic labels are the default choice at small spend, and the choice of human dataset matters as much as its size.
- Background: Direct Preference Optimization trains a model directly on pairs of preferred and rejected responses, without a separate reward model. Those pairs come either from paid human annotators or from a stronger model acting as judge, and the two differ greatly in cost per pair.