When a clinician’s judgment and AI output diverge, what each side contributed disappears with the decision, because current interfaces capture the outcome and not the interaction. We propose that the clinician-AI interaction requires its own account and describe how divergence can be made visible. Making the divergence event visible supports transparent patient communication, better grounds moral responsibility, may mitigate liability, and reframes divergence as clinical AI’s highest-signal moment.
When a clinical AI system’s output differs from a human clinician’s judgment, what follows is consequential and underspecified. Current AI interfaces offer only a narrow lane for what happens next: they present the AI output (and its associated potential divergence) as an adjudicative choice for a human to accept, override, or modify. In practice, collapsing this human-AI interaction into a single output inadvertently obscures what happens in a potential divergence event (Fig. 1) which can be characterized along three axes: What the workflow and interface allow the clinician to see and do (structural axis), what information each side brought to the interaction (informational axis), and how the clinician engaged with the AI’s reasoning rather than only its output (behavioral axis).
When clinical judgment and AI output diverge, clinicians must ultimately decide what to recommend and how to explain their recommendation to the patient, consider their autonomy, accountability, and liability, and bear moral responsibility1. A recent systematic review of AI ethics in nursing and midwifery found that AI challenges the role of health professionals not through replacement, but through subtle reconfiguration of the activities that count as care and that emphasis on AI efficiency may risk certain relational aspects of care less visible and valued2. Because clinicians often cannot see how an AI system reached its output, they are left to choose between that AI output and their own judgment.
Clinicians are told to be the “human in the loop” but are typically given no method for doing so beyond “exercising their judgment”3. Efforts to make AI output both more technically explainable and human interpretable - have not directly correlated with improved clinical decisions and have sometimes created false confidence in incorrect AI output4. Clinicians respond in patterns that the current interface does not capture.
Based on human-AI design choices, clinicians may be presented with situations where they must judge whether their own view or the AI output is correct; where they integrate both human and AI signals; or where they accept AI output without question (automation bias) or override AI reflexively (automation neglect). The human-AI interaction is where the value of clinical AI can be realized or unfulfilled, in whether and how the clinician engaged with the AI, and most consequentially, what each side contributed when their outputs diverged. Together, these human-AI interactions demand a record that neither the model nor the clinician alone can supply: a record of the interaction itself.
We call this human-AI divergence visibility. When clinicians and AI apparently agree, the human-AI interaction is quiet and remains largely invisible. The divergence event (when their outputs differ) is the human-AI interaction’s highest-signal moment: the point at which what each side uniquely contributes becomes visible, examinable, and learnable.
Extract — continue reading at the source.
