I. INTRODUCTION
Korean medicine (KM) education has received increasing attention, with efforts focused on enhancing educational content, curriculum design, and teaching methodologies1). However, a 2020 study highlighted areas of concern, revealing that overall student satisfaction with the KM curriculum was relatively low, averaging 2.83±1.08 out of 5, which indicates a suboptimal outcome2). Key sources of dissatisfaction included “difficulty in applying the curriculum to clinical practice after graduation (23.7%),” “ineffective lecture content and methods (20.0%),” and “insufficient integration between subjects (16.9%),” underscoring the need for more structured and practice-oriented clinical education2).
In response to global trends in higher education, which emphasize competency-based and outcome-driven learning models, the KM education system has developed a competency framework for KM doctors3). This framework prioritizes competencies such as optimal patient care, effective communication, professional development, social responsibility, and efficient healthcare management4). Among these, the ability to deliver optimal patient care is considered the most critical competency for KM doctors4). To support this goal, the clinical practice examination(CPX) has become an integral component of KM education, serving as an assessment tool that evaluates students’ competencies in history taking, physical examination, and patient education through simulated clinical encounters with standardized patients5).
Despite the importance of CPX in training KM doctors, its implementation faces challenges, primarily due to the substantial human and material resources required for securing and managing standardized patients6). These logistical and financial constraints have hindered the widespread adoption of CPX across KM schools6). Moreover, the quality of education can be affected by the variability in recruiting and training standardized patients, prompting the need for alternative solutions6).
A promising alternative is the integration of large language models (LLMs) such as ChatGPT, a generative AI model that has gained significant attention since its release in 2022. LLMs have evolved rapidly in scale and now exhibit advanced contextual understanding and cognitive reasoning abilities7). ChatGPT, with its capacity for natural language processing and generation, has increasingly been applied in various fields, including medical education, where it serves as a cost-effective and accessible tool8). By assisting students in understanding complex medical procedures and diagnostic criteria, ChatGPT can generate realistic scenarios that allow students to practice diagnostic and treatment planning9). Furthermore, as an interactive AI, ChatGPT can go beyond simply providing information by simulating practical clinical experiences for students10).
In medical education, training students to assess suicide risk and manage mental health patients is particularly crucial11,12), considering that South Korea has the highest suicide rate among Organization for Economic Cooperation and Development countries13). In KM education, however, there is a notable gap in standardized training for mental health assessment. A recent survey found that KM doctors frequently encounter patients with psychological disorders, including those at risk for suicide, but are not familiar with systematically assessing suicide risk in these patients14). While our previous study demonstrated the potential of using generative AI chatbots for suicide risk assessment training15), developing a comprehensive CPX education platform offers several additional advantages. For example, it provides a standardized framework for creating and managing various clinical scenarios beyond suicide risk assessment. Also, it enables systematic evaluation and feedback mechanisms that were not available in the previous chatbot-only approach. Most importantly, it allows for scalable implementation across multiple institutions while maintaining consistent educational quality.
In this context, the present study aims to develop a CPX education platform utilizing the ChatGPT API, with the objective of enhancing KM students’ clinical competencies of suicide risk assessment through virtual patient interactions.
II. METHODS
1. Development of a platform to implement simulated patients using ChatGPT
1) Development environment
The platform was developed using Node.js (v18.17.1) for the backend and MongoDB (v8.4.4) for data storage (Fig. 1A). The frontend interface was built using React (v18.3.1). To implement the ChatGPT API, the GPT-4o-mini model, optimized for low-cost and fast-response services, was employed to simulate virtual patients.
2) Prompt engineering
This system utilized prompt engineering techniques exclusively, without fine-tuning or hyperparameter tuning, to generate virtual patients using ChatGPT and facilitate natural, contextually accurate interactions between virtual patients and KM students. Firstly, the initial patient information is generated according to the “A: Patient Case Generation Prompt”. Any missing patient details are automatically supplemented in “B: Generated Patient Case”. In “C: Refined Patient Case”, a refinement prompt is applied to further enhance the case for a more natural conversation. The final result is a virtual patient capable of engaging students in realistic, interactive dialogues (Fig. 1B). The virtual patient retains ChatGPT’s conversational style but is modified to simulate a real patient’s behavior, with actions such as “(touching his knees)” enclosed in parentheses to enrich the immersion of the experience. This method of prompt engineering allowed the creation of a chatbot capable of realistic patient-student interactions (Fig. 1C).
Fig. 1.
Overview of the platform to implement simulated patients using ChatGPT. (A) shows the main interface of this chatbot system. (B) illustrates the flow of virtual patient case generation. (C) presents an example of a generated virtual patient case.
The development process included consultations with a diverse group of experts and researchers: a professor of KM Neuropsychiatry, a professor of Industrial ICT Technology Engineering specializing in AI, and five researchers from the Departments of Artificial Intelligence, Industrial ICT Technology Engineering, and IT Convergence. These experts provided input on scenario design, clinical authenticity, and educational effectiveness. Additionally, ten KM students participated in preliminary testing of the chatbot, providing user-perspective feedback. Consequently, the prompt engineering was iteratively refined based on this feedback to ensure both clinical accuracy and educational value.
3) Development of the instructor-student service platform
The platform was designed to resemble a lecture support system, familiar to students, and offers functionalities for both instructors and students. Instructors can create lectures, design patient scenarios, assign tasks, grade assignments, and provide feedback. Students complete the clinical simulations in a chat-based format and can view feedback to further develop their clinical competencies (Fig. 2).
Fig. 2.
Instructor-student educational platform flow. The instructor creates a course where students register and complete assignments after approval. The instructor sets up a virtual patient scenario with details like duration, score, and description. Students, acting as Korean medicine doctors, consult the virtual patient by asking questions and reviewing symptoms. They can regenerate irrelevant responses during the consultation. Upon submission, details like consultation duration, questions asked, and regenerated responses are recorded. The instructor reviews the session, provides feedback, and assigns a final grade to complete the evaluation.
(1) Scenario creation by instructor
Instructors can either input patient details manually or use the platform to auto-generate the following patient data: (1) patient name, (2) personality, (3) age, (4) gender, (5) date of birth, (6) medical institution, (7) healthcare provider, (8) diagnosis, (9) cause, (10) patient situation, (11) primary symptoms, (12) additional symptoms, and (13) medical history. The generated patient data can be edited to allow for flexible creation of diverse patient cases (Fig. 1C). Once the scenario is finalized, instructors can save it by clicking the ‘Save’ button, and they can optionally upload a patient image to enhance the scenario.
(2) Simulated clinical examination for students
After the scenario is created, students begin the clinical simulation, where the virtual patient initiates the conversation with a greeting. Acting as KM doctors, students conduct a diagnosis by asking relevant questions and collecting information. If a virtual patient’s response does not align with the flow of the consultation, students can request the answer to be regenerated to maintain coherence (Fig. 3).
Fig. 3.
Student screen view of the interaction with the virtual patient. The left side of the screen displays the patient’s profile, including (1) image, (2) name, (3) age, (4) gender, and (5) symptoms, enhancing the immersive experience. On the right side is a notepad where the student can record symptoms during the consultation, which can be referred to when making the final diagnosis.
(3) Instructor’s assignment evaluation
Instructors review each student’s submission, which includes the consultation time, number of questions asked, responses provided, regenerated answers, diagnostic details, and diagnostic outcomes. The entire conversation between the student and the virtual patient can be examined, and instructors can offer real-time feedback by commenting directly on specific student queries. The evaluation process concludes with the instructor assigning a final grade and providing comprehensive feedback based on the student’s diagnostic performance (Fig. 4).
4) Quality control of AI responses
To minimize the risk of AI providing incorrect or inappropriate responses to students, several preventive measures were implemented. First, the prompt engineering was carefully designed to constrain the AI’s responses within clinically appropriate boundaries. The patient’s core characteristics, medical history, and expected responses were clearly defined in the initial prompt, reducing the likelihood of inconsistent or incorrect information. Second, the system included a ‘regenerate response’ feature that allows students to flag and regenerate responses they perceive as inappropriate or inconsistent. This feature helped maintain the quality of the conversation while providing valuable feedback for system improvement. Lastly, all student-AI conversations were recorded and reviewed by instructors, allowing for the identification and correction of any incorrect information during the feedback process.
2. Application of the system to KM education
1) Case development
This system was applied in the second semester of the KM Neuropsychiatry Practical Course for fourth-year students at the KM College of D University. A patient case was developed using the scenario creation feature of the system (Table 1). Forty-seven students enrolled in this practicum were tasked with assessing the suicide risk of a virtual patient. As outlined in our previous work15), the students were instructed to follow a structured conversational flow with the virtual patient: greeting, obtaining the patient’s name, date of birth, purpose of care, and underlying conditions, followed by suicide risk assessment, and concluding the session. Each student was required to conduct a minimum of 50 interactions with the virtual patient and assess suicide risk using the Ask Suicide-Screening Questions Toolkit (ASQ)16), a standardized risk assessment tool. After completing their consultations, the students classified the suicide risk as low, high, or very high, and submitted a rationale for their classification.
2) Evaluation of virtual consultation
Upon submission of the students’ conversations and suicide risk assessments, the instructor conducted a comprehensive evaluation. This evaluation included both a quantitative assessment, graded on a scale of 0∼100, and a qualitative review of each student’s performance. Individual feedback on specific interactions was also provided (Fig. 4).
3. Survey of students’ perceptions and experiences
All students who participated in the class were invited to complete an anonymous online survey via Moaform (URL: https://www.moaform.com/) (Qoom Networks Inc., Seoul, Republic of Korea), a free platform for anonymous surveys. This survey was conducted from October 24 to October 30, 2024. The purpose of the survey was to gather students’ perceptions and experiences of using the chatbot system in the context of their education. Only students who were informed that the survey data could be used for research purposes, and who voluntarily consented to participate, were included in the study.
The survey consisted of 13 questions evaluating students’ experiences with mock clinical consultations17) and 17 questions assessing the usability of the AI chatbot18). The survey items were adapted from a prior study15) and modified appropriately for use in an KM Neuropsychiatry course. Additionally, the question was raised about the average number of conversations required to learn when interacting with a generative AI chatbot that simulates a patient with a mental disorder. Finally, four open-ended questions assessed student satisfaction, dissatisfaction, suggestions for improvement, and the perceived need for educational support related to chatbot-based role-playing.
4. Analysis of survey results
Survey responses regarding students’ mock clinical experience and the usability of the AI chatbot were rated on a 5-point Likert scale. The results were analyzed using mean and standard deviation. Qualitative analysis was performed on the open-ended questions. Statistical analyses of the collected data were conducted using SPSS version 18.0 (SPSS Inc., Chicago, IL, USA).
III. RESULTS
1. Survey respondents
From October 24th to 30th in 2024, a total of 42 students participated in the survey, representing a response rate of 89.4% from the 47 eligible respondents.
2. Satisfaction assessment of simulated consultation experience
The evaluation of the simulated consultation experience yielded consistently high scores across multiple dimensions. Particularly notable were the educational outcome measures: clinical knowledge enhancement (4.24±0.79), knowledge self-assessment capability (4.17±0.76), and communication skills improvement (4.29±0.74). Students reported increased confidence in suicide risk assessment (3.98±0.84) and improved interviewing skills (4.24±0.73). Furthermore, they positively evaluated the platform’s potential for real-world application (4.24±0.69) and future clinical practice (4.36±0.66), suggesting successful transfer of learning to practical clinical scenarios. These educational outcomes were complemented by high scores in other dimensions: appropriateness-related items received particularly high ratings (4.71±0.38), followed by interest (4.44±0.46). Notable individual items included: “I recommend applying the role-play using the chatbot (patient-physician) in other classes” (Q5, 4.71±0.46) and “I support using the chatbot for role-play in future training for juniors in Korean medicine neuropsychiatry theory and practice” (Q6, 4.76±0.43). The high scores for these items suggest strong student endorsement for expanding the use of this educational tool (Table 2).
3. Evaluation of AI chatbot usability
The usability evaluation revealed strong performance across multiple dimensions: usefulness (4.40±0.54), attractiveness (4.42±0.56), accessibility (4.49±0.51), and convenience (4.42±0.42). The overall discoverability score was 4.43±0.63. However, reliability showed a relatively low score, with an average score of 4.13±0.55. The statement “I trust that the responses given by the chatbot will be similar to those of actual patients” (Q12) received a notably lower score(3.69±0.95) (Table 3).
4. Other responses
When talking to a generative AI chatbot that acts as a patient with a mental disorder, the average number of conversations needed for learning was 37.5, excluding two outliers (both answered 1 time) (Fig. 5). The qualitative responses revealed several key themes regarding students’ experiences with the platform. Students expressed high satisfaction with the system’s ability to provide varied patient responses based on question content and nuance, allowing them to experience potential clinical challenges in a low-pressure environment. One student noted, “This was the most interesting project I’ve been involved in over the past six years,” while another appreciated that “it was possible to simulate interactions with types of patients that are hard to meet in real life.” However, some limitations were identified, including occasional response uniformity and technical issues. As one student mentioned, “Some responses were some-what uniform” and “In the latter part, the answers began to become increasingly similar.” Technical challenges were also reported, with one student describing system errors that interrupted the learning process. Students suggested several improvements, including implementing time limits for consultations, developing more diverse patient modules, and adding visual feedback capabilities. As one student suggested, “Once familiar with the chatbot task, it might be beneficial to impose some time limits on the conversation, e.g., completing the conversation and making a diagnosis within 20 minutes.” Regarding educational support, students expressed interest in expanding the platform’s use across various medical subjects, particularly for rare cases or challenging diagnoses. One student recommended “exploring the use of chatbots in various subjects (such as pediatric patients, which students rarely encounter, or patients with cardiovascular concerns that require careful differential diagnosis) (Table 4).”
5. Instructor feedback
The instructor provided feedback on student-patient interactions across several domains: (1) interview structure and flow (e.g., “Good natural transition to suicide risk assessment”), (2) communication style(e.g., “Excellent demonstration of warm and considerate attitude toward the vulnerable patient”), and (3) clinical reasoning (e.g., “Need to strengthen the evidence for classifying this case as highest suicide risk”).
IV. DISCUSSION
This study’s development and implementation of a ChatGPT-based CPX platform represents a significant advancement in KM education, while also revealing several important considerations and limitations that warrant careful examination.
The high satisfaction scores across multiple dimensions (interest: 4.44±0.46, appropriateness: 4.71±0.38) demonstrate the platform’s potential as a complementary tool in KM clinical education. Particularly noteworthy are the strong endorsements for expanding its use to other classes (4.71±0.46) and future training (4.76±0.43), indicating students’ readiness to embrace AI-enhanced learning methodologies. This positive reception aligns with recent trends in medical education, where technological integration is increasingly recognized as crucial for preparing future healthcare professionals8). In particular, chatbots based on generative AI are expected to be able to assist KM students in their practical training in neuropsychiatry by imitating psychiatric patients15).
The platform’s high accessibility (4.49±0.51) and usefulness (4.40±0.54) scores address a critical gap in traditional standardized patient programs. As highlighted by Han et al. (2020)4), conventional CPX implementation faces significant resource constraints, including financial burden and human resource limitations. Our platform offers a scalable, cost-effective solution that can be deployed across multiple institutions simultaneously, making clinical training opportunities more widely accessible. One of the advantages of this training platform is that the instructor can individually evaluate the learner’s conversation and provide feedback. This is not only convenient for the instructor, but also something that has been difficult to do in traditional training to improve students’ communication skills with patients with mental illness19).
The survey results showed positive self-reported outcomes in knowledge acquisition (4.24±0.79) and skill development (4.14±0.62). Particularly noteworthy is the high rating for practical application (4.36±0.66), indicating that students felt more prepared to apply these skills in their future clinical practice. While these self-reported measures suggest the platform’s potential as an educational tool, future studies incorporating objective knowledge assessments would be valuable to more rigorously evaluate its educational effectiveness. This finding is particularly relevant in the context of suicide risk assessment training, where students need safe environments to practice sensitive conversations without risking patient well-being15). The platform’s ability to provide standardized, repeatable scenarios while allowing for varied student responses represents a significant advantage over traditional training methods. A systematic review in 2023 concluded that simulation training is promising for practical training in suicide crisis intervention20). Simulation-based education assisted by AI is expected to be more promising in practical training for high-risk groups such as suicide crisis patients due to its potential to meet the three basic points of simulation-based education: ‘basic knowledge acquirement’, ‘briefing’, and ‘psychological safety assurance’21).
The qualitative feedback from students provided valuable insights into both the strengths and limitations of the platform. Students particularly valued the ability to practice clinical interactions without time pressure and the opportunity to encounter rare or challenging cases, addressing common limitations in traditional clinical education. As one student noted, “It was meaningful as I could engage as if I were conversing with a patient at risk of suicide” while another appreciated that they “could participate without constraints of time and space.” However, the feedback also revealed important technical and pedagogical challenges, including response uniformity and occasional system instability. Students’ suggestions for implementing time limits and developing more diverse patient modules indicate the need for a balance between educational flexibility and clinical realism. The desire for expanded applications across various medical subjects, particularly for rare conditions and pediatric cases, suggests potential broader applications in KM education. This aligns with current trends in medical education that emphasize using technology to supplement areas where traditional clinical exposure may be limited21).
Despite these promising findings, several challenges emerged from our study. Importantly, the relatively low authenticity rating (3.69±0.95) for virtual patient responses highlights a significant limitation in current AI technology. This aligns with previous study about the challenges of achieving naturalistic patient-doctor interactions with LLMs7). In addition, this low authenticity rating is a result that was found identically in suicide risk assessment training using another generative AI model, the Claude 3 Sonnet model15). Likewise, the gap between AI-generated responses and real patient behavior remains a crucial area for improvement. However, it is not possible to distinguish whether these results are due to students’ pre-existing attitudes toward generative AI or their experiences in this study, and qualitative interviews with students could help understand their perceived low authenticity. Also, as described in our previous study15), if the students watch educational videos of actual patients with mental illness in parallel with the educational period using a generative AI chatbot, it may be possible that students’ evaluations of the similarity between the chatbot’s responses and the actual patient’s responses will change.
Some specific solutions can be implemented to enhance response authenticity. For example, to improve clinical authenticity, actual de-identified patient interview transcripts from KM practice could be used to train the response patterns. This would help ensure that the virtual patient’s responses more closely mirror real clinical encounters while maintaining patient privacy. Another potential dataset for fine-tuning could be the publicly available counseling dataset from AI Hub ( https://aihub.or.kr/), operated by the Ministry of Science and ICT and the National Information Society Agency. Regular updates of the prompt engineering based on student and instructor feedback could help address specific authenticity issues as they are identified. For example, when students reported that ‘some responses were somewhat uniform’ and ‘the answers began to become increasingly similar’, the prompts could be modified to generate more varied and contextually appropriate responses. Additionally, providing students with audiovisual educational materials showing real patient interviews before their interaction with the virtual patient could enhance their understanding of authentic patient responses and behaviors. This could include video recordings of standardized patient interviews or clinical case demonstrations, which would help students better recognize the nuances of patient communication and improve the authenticity of their virtual patient interactions. This multi-modal educational approach could serve as a valuable reference point for students to compare and calibrate their expectations during virtual consultations.
While our platform successfully demonstrated general clinical education capabilities, its application in KM education requires careful consideration of the unique context of mental health care in KM. Given South Korea’s high suicide rate13) and the frequent encounters between KM doctors and patients with psychological conditions14), there is a pressing need for standardized mental health assessment training that integrates both psychiatric approaches and KM perspectives. The current scenario, while focused on suicide risk assessment, could be enhanced by incorporating KM-specific elements such as tongue and pulse examination that could provide additional insights into the patient’s mental state, and KM-specific therapeutic approaches that could be discussed with patients at risk. This integration of KM-specific elements while maintaining standardized suicide risk assessment protocols is crucial not only for educational authenticity but also for preparing KM doctors to effectively contribute to mental health care and suicide prevention within Korea’s healthcare system.
This study has several important limitations that should be acknowledged. First, our evaluation was limited to a single clinical scenario focusing on suicide risk assessment. While this provided valuable insights into the platform’s capabilities, it may not fully represent its effectiveness across the broader spectrum of KM diagnoses and treatments. In KM education, the development of CPX scenarios is being developed to reflect unique KM diagnoses and treatments22). A study in 2010 implemented an interactive KM pattern diagnosis capability assessment software, which presents 36 categories of symptoms/signs including tongue and pulse examination that are key to pattern diagnosis, which is worth referencing23). In addition, ‘the Clinical Practice Guideline of Korean Medicine: The Pattern Identification of the Eight Principles’ developed in 2023 present a standard KM pattern diagnosis, which can be referenced on this educational platform in the future24). Second, methodological limitations include the absence of a control group using traditional standardized patient methods, making it difficult to objectively compare the educational effectiveness of our platform. Additionally, our evaluation relied primarily on subjective satisfaction measures rather than objective assessments of clinical competency improvement. Therefore, future studies could evaluate the educational effectiveness of the current educational platform by assessing changes in knowledge, attitudes, and communication skills regarding suicide prevention as assessed in previous studies11,12). Third, technical limitations inherent to the ChatGPT system were observed. The AI sometimes provided inconsistent responses or deviated from the intended clinical context, requiring continuous monitoring and feedback from instructors. Additionally, the system’s understanding of specialized KM terminology and diagnostic methods may be limited, potentially affecting the authenticity of simulated consultations. This could potentially be improved by using a terminology standard, such as ‘International Standard Terminologies on Traditional Chinese Medicine’ by the World Health Organization, to fine-tune the LLM model25). Fourth, as this was a short-term pilot study, the long-term educational impact and transferability of skills to real clinical settings remain unverified. Future longitudinal studies would be valuable in assessing the sustained effectiveness of this educational approach. In this context, recent research has identified improvements in communication-related competencies and anxiety over time in medical students26), and the synergistic effects of these longitudinal changes and current suicide risk assessment training could also be investigated. Lastly, the study’s sample was limited to fourth-year students at a single KM college, which may restrict the generalizability of our findings to other academic years or institutions. A more diverse sample across multiple institutions would provide more robust evidence of the platform’s effectiveness. These limitations suggest several directions for future research, including the need for controlled comparative studies, longitudinal assessments of clinical competency development, and broader implementation across various KM educational contexts.
V. CONCLUSION
The ChatGPT-based CPX platform developed in this study demonstrates promising potential as an accessible and standardized tool for clinical training in KM education. While the high satisfaction rates and usability scores validate its educational utility, the identified limitations in response authenticity suggest its optimal role as a complementary tool rather than a replacement for traditional clinical education methods. The findings provide valuable insights for the future development of AI-enhanced medical education platforms, particularly in the context of unique educational requirements in KM and neuropsychiatric education. Future iterations of the platform should focus on enhancing response authenticity, expanding scenario diversity, and implementing objective assessment measures while maintaining the high levels of accessibility and user satisfaction demonstrated in this pilot study.