KTO (Kahneman-Tversky Optimization)
Free while signed in. Answers cite the passages they came from.
Contextual AI introduces KTO, an alignment objective derived from prospect theory that works with binary "good/bad" signals instead of preference pairs.
Prospect-theory motivation: Models reward as a Kahneman-Tversky value function with loss aversion, replacing DPO's log-likelihood-of-preferences objective with utility maximization.
No preference pairs needed: Works with unpaired good/bad signals, dramatically loosening data collection requirements compared to DPO or RLHF.
Matches/beats DPO: Matches or exceeds DPO performance at model scales from 1B to 30B, a clean empirical win at similar training cost.
Practical data advantage: Makes alignment much cheaper to run in production where paired preference data is rare but outcome feedback ("user liked/didn't like") is abundant.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack