Operationalizing biological interpretability in health-related visual stimuli using multimodal artificial intelligence as standardized non-human raters

Evaluation and assessment research in health communication requires structured, construct-based approaches for interpreting visual materials. This study introduces the Asclepius Biological Interpretability Scale, developed through a two-stage modified Delphi process (a joint image-and-item panel followed by an independent item-only panel) grounded in the Biopsychosocial Model and Visual Literacy Theory, and examines whether multimodal artificial intelligence (AI) systems can reproducibly apply this framework relative to internally generated human reference scores. Six artworks (four health-themed and two controls) were evaluated by three multimodal AI models and four researchers (internal human reference) using a standardized protocol with independent researcher-operated accounts, memory-isolation procedures, repeated assessments over two days, and the final 20-item scale.

Because Image Type varied across only six stimuli, health-themed versus control differences were summarized at the image level (Cohen’s d), human–AI differences were summarized descriptively at the rater level, and reproducibility was assessed using intraclass correlation coefficients (ICC) and Spearman correlations. Within this development image set, AI-generated scores were descriptively higher for health-themed than for control images on the three biomedical subscales (image-level Cohen’s d = 0.91–1.38) but not on Non-Medical Interpretability (d = 0.49); with four health-themed and two control images, these differences were not statistically significant (p = 0.19–0.60). Reproducibility across researcher-operated accounts was excellent for the three biomedical subscales (ICC = 0.90–0.97) and moderate for Non-Medical Interpretability (ICC = 0.57), whereas agreement across assessment days was poor (ICC = 0.04–0.46).

The four researchers’ internal reference scores were higher than the AI-generated scores on the biomedical subscales for health-themed images (mean differences 1.41–1.96 points) and lower on Non-Medical Interpretability (− 2.02 to − 2.19 points). These findings provide preliminary methodological evidence that multimodal AI can reproducibly apply a structured interpretability framework to controlled visual stimuli. Although day-to-day stability and the Non-Medical Interpretability subscale, which showed substantial internal-consistency problems, require further refinement, the results support the feasibility of using AI as standardized non-human raters within controlled methodological settings.

Broader validation using authentic health communication materials and independent external human raters remains necessary. Figures 1 and 2 and Supplementary Figures S1–S2 were computer-generated (Canva standard application) by Dr. Selçuk Uylaş, a research assistant in Internal Medicine (Mugla Education and Research Hospital) whose design, computer, and visual skills I know and trust, after the authors’ pencil sketch and summary of the topic were explained.

We thank him for his scientific support. Vefa High School, International Baccalaureate (IB) Diploma Programme Student, Istanbul, Turkey Independent Researcher, Retired Primary School Teacher, Istanbul, Turkey Independent Researcher, Acıbadem Health Group, Dentistry, Istanbul, Turkey Faculty of Medicine, Department of Medical Oncology, Mugla Sitki Kocman University, Kötekli Mh. Marmaris Yolu Bulvarı No: 55 Menteşe, Mugla, 48000, Turkey The authors declare no competing interests.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Below is the link to the electronic supplementary material. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.