Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models
Minji Kim, Jihyoung Jang and Hyounghun Kim at POSTECH argue that non-compliance in vision-language models is evaluated at the wrong granularity, and build a benchmark where a single query mixes answerable content with content that should be withheld.
Ask this paper
The assumption being broken: Existing benchmarks score non-compliance per query, treating each request as wholly answerable or wholly refusable. Real queries mix both.
KoNA: Five categories, False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility and Safety, each tested for query-level and component-level non-compliance under paired single and compound queries.
Result: Models across the board fail to refuse, correct, or abstain appropriately, and the failures get worse specifically when a query requires selective non-compliance rather than a whole-query decision.
Fix that works: Fine-tuning on KoNA examples requiring selective non-compliance, mixed with a fully answerable set, raises non-compliance accuracy substantially while largely preserving performance on fully answerable tasks.
Abstract
Vision-language models (VLMs) are expected to respond helpfully to appropriate requests while withholding compliance with requests that are incorrect, unsafe, infeasible, or unanswerable. However, existing benchmarks predominantly evaluate non-compliance at the level of the query as a whole, assuming that each request either warrants compliance or requires withholding compliance. In practice, real-world queries can contain a mixture of answerable content and components for which compliance should be withheld. In this paper, we introduce KoNA, a benchmark for evaluating selective non-compliance in VLMs across five categories: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety. Each task evaluates two capabilities: query-level non-compliance and component-level non-compliance under paired single and compound queries. Our evaluation across diverse VLMs shows that models often fail to refuse, correct, or abstain appropriately, and these failures become more pronounced when queries require selective non-compliance. To address this challenge, we fine-tune VLMs using KoNA examples that require selective non-compliance, together with a fully answerable set that should receive direct answers. Our fine-tuned models achieve substantial improvements in non-compliance accuracy while largely maintaining performance on fully answerable tasks. These results suggest that the fine-tuned models can distinguish between answerable components and those requiring non-compliance and respond in a task-appropriate manner.