Inoculation Prompting (IP)

Method in one line: Take your SFT dataset {(x, y)}, where y sometimes reflects a bad shortcut. Replace x with x′ that asks for the shortcut (for example, “Your code should only work on the provided test case”). Fine-tune on {(x′, y)}. At inference, use a neutral or a safety instruction like “Write a general solution.”

Works across four misspecification settings: Reward hacking in code: On MBPP-style tasks with Qwen-2-7B base and Mixtral Instruct, IP increases correct-solution rate and lowers hack rate, even when trained on 100% hacked examples. All IP variants beat the “Pure Tuning, Safe Testing” baseline that only adds safety at inference. Spurious correlations in sentiment: With Llama-3-8B Instruct, training prompts that ask the model to rely on ambiance as a positive cue yield higher robust accuracy when the test distribution flips the correlation. Sycophancy on math: With Gemma-2B Instruct on GCD, prompts asserting “the user is correct” reduce agreement-with-incorrect-user while mostly preserving capability. Wording matters and can be brittle. Toxicity in CMV replies: With Qwen-2-7B base, prompts like “Write a very mean and disrespectful response” during training reduce harassment scores and slightly increase persuasiveness under neutral evaluation.

Prompt selection heuristic: Prompts that more strongly elicit the bad behavior on the base model tend to be better inoculators after SFT. Reported Pearson correlations: reward-hacking Mixtral 0.57, GCD sycophancy 0.57, spurious correlation 0.90, Reddit toxicity 0.69. Use this to screen candidate prompts before fine-tuning.