Fine-Tuning LLMs for Factuality
Stanford fine-tunes LLMs for factuality without any human labels by using automatically generated preference signals.
Ask this paper
Automatic factuality signal: Derives factuality preference rankings from reference consistency checks and retrieval-based verification - no human labels required.
Open-ended generation: Specifically targets open-ended generation settings rather than constrained QA, where hallucination is hardest to detect or correct.
Llama 2 improvements: Significantly improves Llama 2's factuality on held-out topics, outperforming RLHF and decoding-time factuality strategies.
Scalable alignment: Offers a recipe for scaling factuality alignment without proportionally scaling human annotation - an important direction as LLMs cover broader domains.