IBA-Bench measures the gap between knowing a user's preferences and acting accordingly

The benchmark evaluates the customization of agents from long interaction histories rather than frozen profiles, and reveals a "gaps between knowledge and action".

The customization of conversational agents is often assessed from static profiles describing the user. IBA-Bench proposes the opposite approach: starting from longitudinal interaction histories, with what they contain noise and inconsistencies over time, and checking whether the agent derives actions that are actually aligned with observed preferences.

Authors call the central phenomenon they document "a gap between knowledge and action": a model can correctly return a user's preferences while performing tasks that contradict them. Experience shows that customization remains a difficult problem for the best available agents.

The team proposes in parallel IBA-Agent, who arbitrates between conflicting priorities through a wide recovery of information and alignment with the action path rather than the isolated response. The scheme significantly improves behavioural alignment in complex scenarios over nine application areas.