Application of Artificial Intelligence as an Aid for the Correction of the Objective Structured Clinical Examination: (OSCE)

• Davide Luordo ^[1] ; Marta Torres Arrese ^[3] ; Cristina Tristán Calvo ^[4] ; Kirti Dayal Shani Shani ^[4] ; Luis MIguel Rodríguez Cruz ^[5] ; Francisco Javier García Sánchez ^[1] ; Alfonso Lagares Cómez-Abascal ^[6] ; Rafael Rubio García ^[6] ; Juan Delgado Jiménez ^[6] ; Mercedes Pérez Carreras ^[6] ; Ramiro Díez Lobato ^[6] ; Juan José Granizo Martínez ^[6] ; Yale Tung-Chen ^[2] ; María Victoria Villena Garrido ^[6]
  1. [1] Universidad Complutense de Madrid

     Universidad Complutense de Madrid

     Madrid, España

     
  2. [2] Universidad Autónoma de Madrid

     Universidad Autónoma de Madrid

     Madrid, España

     
  3. [3] 3. Alcorcón Foundation Hospital, 28922 Alcorcón, Spain
  4. [4] 1. Infanta Cristina University Hospital, 28981 Madrid, Spain
  5. [5] 4. Independent Researcher, 28945 Madrid, Spain
  6. [6] 5. 12 de Octubre University Hospital, 28041 Madrid, Spain

  Mostrar afiliaciones +
• Localización: fisioGlía: revista de divulgación en Fisioterapia, ISSN-e 2340-6151, Vol. 12, Nº. 2, 2025, págs. 33-41
• Idioma: inglés
• Texto completo no disponible (Saber más ...)
• Resumen

  • The assessment of clinical competencies is essential in medical training, and the Objective Structured Clinical Examination (OSCE) is an essential tool in this process.

    There are multiple studies exploring the usefulness of artificial intelligence (AI) in medical education. This study explored the use of the GPT-4 AI model to grade clinical reports written by students during the OSCE at the Teaching Unit of the 12 de Octubre and Infanta Cristina University Hospitals, part of the Faculty of Medicineat the Complutense University of Madrid, comparing its results with those of human graders. Ninety-six (96) students participated, and their reports were evaluated by two experts, an inexperienced grader, and the AI using a checklist designed during the OSCE planning by the teaching team.

    The results show a significant correlation between the AI and human graders (ICC = 0.77 for single measures and 0.91 for average measures). AI was more stringent, assigning scores on an average of 3.51 points lower (t = -15.358, p < 0.001); its correction was considerably faster, ting the analysis in only 24 min compared to the 2–4 h required by human graders.

    These results suggest that AI could be a promising tool to enhance efficiency and objectivity in OSCE grading

• Referencias bibliográficas
  • 1. Epstein, R.M. Assessment in Medical Education. N. Engl. J. Med. 2007, 356, 387–396. [Google Scholar] [CrossRef] [PubMed]
  • 2. Patrício, M.F.; Julião, M.; Fareleira, F.; Carneiro, A.V. Is the OSCE a Feasible Tool to Assess Competencies in Undergraduate Medical...
  • 3. Harden, R.M.; Gleeson, F.A. Assessment of Clinical Competence Using an Objective Structured Clinical Examination (OSCE). Med. Educ. 1979,...
  • 4. Turner, J.L.; Dankoski, M.E. Objective Structured Clinical Exams: A Critical Review. Fam. Med. 2008, 40, 574–578. [Google Scholar] [PubMed]
  • 5. Lobato, R.D.; Lagares, A.; LópezMedrano, F.; Villena, V.; Fernández, A.; Martínez-López, J.; Rubio, G.; Munárriz, P.M.; Alen, J.F. Examen...
  • 6. McLaughlin, K.; Ainslie, M.; Coderre, S.; Wright, B.; Violato, C. The Effect of Differential Rater Function over Time (DRIFT) on Objective...
  • 7. Humphris, G.M.; Kaney, S. Examiner Fatigue in Communication Skills Objective Structured Clinical Examinations. Med. Educ. 2001, 35, 444–449....
  • 8. Chong, L.; Taylor, S.; Haywood, M.; Adelstein, B.-A.; Shulruf, B. The Sights and Insights of Examiners in Objective Structured Clinical...
  • 9. García-Puig, J.; Vara-Pinedo, F.; VargasNúñez, J.A. Implantación del Examen Clínico Objetivo y Estructurado en la Facultad de Medicina...
  • 10. Ramos, J.M.; Martínez-Mayoral, M.A.; Sánchez-Ferrer, F.; Morales, J.; Sempere, T.; Belinchón, I.; Compañ, A.F. Análisis de la prueba...
  • 11. Cusimano, M.D.; Cohen, R.; Tucker, W.; Murnaghan, J.; Kodama, R.; Reznick, R. A Comparative Analysis of the Costs of Administration...
  • Scholar] [CrossRef] 12. Roberts, J.K.; Sullivan, M.; Atwater, S.; Desai, K.; Prabhu, N.K.; Hertz, J.T.; Buhr, G.T.; Peyser, B.; Weigle,...
  • 13. Zhang, W.; Cai, M.; Lee, H.J.; Evans, R.; Zhu, C.; Ming, C. AI in Medical Education: Global Situation, Effects and Challenges. Educ....
  • 14. Borakati, A. Evaluation of an International Medical E-Learning Course with Natural Language Processing and Machine Learning. BMC Med....
  • 15. Mustafa, M.Y.; Tlili, A.; Lampropoulos, G.; Huang, R.; Jandric, P.; Zhao, J.; Salha, S.; Xu, L.; Panda, S.; Kinshuk; et al. A Systematic...
  • 16. Chan, K.S.; Zary, N. Applications and Challenges of Implementing Artificial Intelligence in Medical Education: Integrative Review....
  • 17. Kung, T.H.; Cheatham, M.; Medenilla, A.; Sillos, C.; De Leon, L.; Elepaño, C.; Madriaga, M.; Aggabao, R.; Diaz-Candido, G.; Maningo,...
  • 18. Manne, R.; Kantheti, S.C. Application of Artificial Intelligence in Healthcare: Chances and Challenges. Curr. J. Appl. Sci. Technol....
  • 19. Ávila-Tomás, J.F.; Mayer-Pujadas, M.A.; Quesada-Varela, V.J. La inteligencia artificial y sus aplicaciones en medicina II: Importancia...
  • 20. Sun, L.; Yin, C.; Xu, Q.; Zhao, W. Artificial Intelligence for Healthcare and Medical Education: A Systematic Review. Am. J. Transl....
  • 21. Kufel, J.; Paszkiewicz, I.; Bielówka, M.; Bartnikowska, W.; Janik, M.; Stencel, M.; Czogalik, L.; Gruszczynska, K.; Mielcarska, S. Will...
  • 22. Schaye, V.; Guzman, B.; Burk-Rafel, J.; Marin, M.; Reinstein, I.; Kudlowitz, D.; Miller, L.; Chun, J.; Aphinyanaphongs, Y. Development...
  • 23. Strong, E.; DiGiammarino, A.; Weng, Y.; Basaviah, P.; Hosamani, P.; Kumar, A.; Nevins, A.; Kugler, J.; Hom, J.; Chen, J.H. Performance...
  • 24. Singhal, K.; Tu, T.; Gottweis, J.; Sayres, R.; Wulczyn, E.; Hou, L.; Clark, K.; Pfohl, S.; Cole-Lewis, H.; Neal, D.; et al. Towards...
  • 25. Mayol, J. Inteligencia artificial generativa y educación médica. Educ. Méd. 2023, 24, 100851. [Google Scholar] [CrossRef]
  • 26. Gilson, A.; Safranek, C.W.; Huang, T.; Socrates, V.; Chi, L.; Taylor, R.A.; Chartash, D. How Does ChatGPT Perform on the United States...
  • 27. Cooper, G. Examining Science Education in ChatGPT: An Exploratory Study of Generative Artificial Intelligence. J. Sci. Educ. Technol....
  • 28. Dave, T.; Athaluri, S.A.; Singh, S. ChatGPT in Medicine: An Overview of Its Applications, Advantages, Limitations, Future Prospects,...
  • 29. Eysenbach, G. The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation with...
  • 30. Radford, A.; Narasimhan, K.; Salimans, T.; Sutskever, I. Improving Language Understanding by Generative PreTraining. 2018. Available...
  • 31. Key Issue 4: Human Oversight-EU AI Act. Available online: https://www. euaiact.com/key-issue/4 (accessed on 22 December 2024)