Poisoning Language Models During Instruction Tuning

Poisoning attack: Demonstrates adversaries can contribute poisoned examples to instruction tuning datasets to induce specific misbehaviors.

Cross-task poisoning: Poisoning can induce degenerate outputs across held-out tasks, not just the poisoned task - broad attack surface.

Supply-chain vulnerability: Highlights the supply-chain vulnerability of using community-sourced instruction data.

Alignment safety: Important for the field's thinking on data provenance and vetting for alignment datasets.