To obtain a license to practice Korean medicine, students are required to pass the national examination for Korean medicine and graduate from a college or the School of Korean Medicine [1]. Since 2009, to evaluate the performance of prospective medical doctors in Korea, a practical examination of the student has been conducted as part of the process to acquire a doctor’s license. In the second half of 2021, for the first time, the national examination for dentists has scheduled a practical examination of dental students, including history taking, physical examination, clinical performance, and attitude [2]. Doctors of Korean medicine have also proposed a step-by-step evaluation system which includes practical subjects in the license exam [3]. The Deans Association of College of Korean Medicine agreed to introduce a clinical examination in the national examination for Korean medicine doctors in 2030 [4].

The typical clinical practice educational methods for medical education include assessments called the clinical practice examination (CPX) and the objective structured clinical examination (OSCE) [5]. Since its introduction by Harden et al [6] in 1975, the OSCE has been used worldwide in the assessment of education of healthcare professionals in the fields of medicine, nursing, dentistry, and pharmacy, to evaluate clinical performance. The OSCE is an objective, performance-oriented test, designed to be a reliable, valid, and flexible educational tool that is highly useful in the assessment of education and evaluating clinical skills [7,8]. Regarding education on clinical performance in Korean medicine, studies on the overall satisfaction of education on clinical performance [9], the efficacy of the OSCE in the Korean Ophthalmology and Otolaryngology department’s clinical practice [10], the satisfaction of the CPX [11,12], and a survey on students’ perception of the CPX [13] have been reported. However, there have been a few reports on specific cases of clinical practice education that have allowed the implementation of the OSCE for individual subjects. Particularly in acupuncture and moxibustion medicine, practical clinical education is very important because of the acupuncture stimulation methods, stimulation points, and various acupuncture methods based on specific theories. Therefore, in this study, the experience and students’ perceptions of the OSCE were reported after teaching the clinical skills of acupuncture medicine with the clinical practice guidelines for acupuncture and moxibustion medicine [14], and the learning objectives, instructions, and checklists presented in Korean medicine clinical skills for the OSCE [15]. We expect this to provide basic data with reference to further clinical practice education and evaluation.

The OSCE contents and materials

From the list of the clinical practice guidelines for acupuncture and moxibustion medicine, 5 types of acupuncture-related clinical applications were selected: acupuncture, electroacupuncture, pharmacopuncture, auricular acupuncture, and burning acupuncture. Techniques that could not be performed on the model were excluded, such as moxibustion, and cupping. Hand sanitizer, alcohol, cotton, clinical waste bin, and damaging clinical waste bins were used in each type of acupuncture-related clinical application in the OSCE. Four models were used to fit the characteristics of clinical performance. The models and materials used in each of the OSCE are listed in Table 1.

Practical education

Based on the Korean Acupuncture and Moxibustion Medicine textbook [16] and clinical practice guidelines [14], video lectures covering procedures and precautions of acupuncture, electroacupuncture, pharmacopuncture, auricular acupuncture, and burning acupuncture were produced. Video lectures were used to allow students to familiarize themselves with the components of clinical performance in advance. During the clinical practice period in the Department of Acupuncture and Moxibustion Medicine, the students practiced their skills using the practice tools at the Clinical Skills Center. The practice time was decided autonomously by the trainee (Fig. 1).

Checklists and instructions

The checklist used for education was modified based on the clinical practice guidelines for acupuncture and moxibustion medicine [14]. The learning objectives and performance time (5 minutes) were set according to the contents of the clinical practice guidelines for acupuncture and moxibustion medicine [14]. Specific guidelines, such as test time and acupoint selection were added to the examination instructions based on the contents presented in the clinical practice guidelines for acupuncture and moxibustion medicine [14]. The checklist and learning outcomes were published in clinical practice guidelines distributed to all students (Appendices). To provide instructions for the OSCE station for each application rotation, the instructions of the clinical practice guidelines for acupuncture and moxibustion medicine were modified and supplemented (Table 2).

Implementation of the OSCE

After preparing instructions and models on the bed of the clinic, the students were instructed to perform the clinical application. The clean zone and contamination zone were separated; materials necessary for procedures in the clean zone, while containers for clinical waste, damaging clinical waste, and general waste were stored in the contamination zone (Fig. 2). The time for each clinical examination was checked by a timer, and 5 minutes was set as the time limit of each procedure following the clinical practice guidelines for acupuncture and moxibustion medicine. The OSCE was taken 1 to 3 days after the students’ practice, and the practice conditions were consistent for each student. Feedback on students’ performance was provided based on the checklist after the OSCE.

Questions to examine the students’ perspective

Study population

The survey was conducted on 91, 4^th year students studying Korean medicine who had completed their clinical practice education (January 2020 to September 2020). Due to COVID-19, 18 out of 91 students were unable to have face-to-face classes or take the OSCE but were still asked to respond to the survey without limiting their participation. The survey was conducted online from January 23, 2021, to January 31, 2021.

Questionnaire and survey

The survey items consisted of 27 questions about the students experience of taking the OSCE and 8 common questions. The survey questions were developed by referring to existing medical and Korean medicine educational studies [9,17,18]. These questions were expertly reviewed and items revised by 2 Korean medicine professors (who lectured on practical subjects), 1 acupuncture specialist, and 1 pedagogical expert. Before the survey, a preliminary test was conducted on 2, 4^th year Korean medicine students to determine clarity and understanding, and the time required to complete the survey.

The survey was conducted using an online survey tool (SurveyMonkey, San Mateo, CA, USA). Survey links were sent to all relevant students via student representatives. The survey questions consisted of (1) an overall opinion of the OSCE, (2) effectiveness of flipped learning, (3) adequacy of the education model utilized, and (4) adequacy of the model, and evaluation checklist, total 2 open questions and 30 questions which were evaluated using a 5-point Likert scale. The survey was reviewed and approved by the Institutional Review Board of Wonkwang University (no.: WKIRB-202012-SB-089).

Statistical analysis

The statistical program SPSS Version 23.0 (IBM Corp., Armonk, USA) was used to perform statistical analysis. The average and standard deviation of the 5-point scale items were calculated, and participants’ responses to open-ended questions were collected. The Mann-Whitney test was used to analyze whether there was a difference in self-efficacy in clinical practice between the students who had taken the OSCE and those students who had not taken the OSCE. The significance level was set at p < 0.05. In addition, Spearman’s rank correlation analysis was used to determine whether there was any correlations between each acupuncture-related clinical applications regarding self-efficacy.

Characteristics of survey respondents

Of the 91 students who completed the clinical practice course of acupuncture and moxibustion medicine, 45 (49%) voluntarily participated in the survey. Of the respondents, 27 were male and 18 were female, and 39 out of 45 responded that they had already taken the OSCE (acupuncture, electroacupuncture, pharmacopuncture, auricular acupuncture, burning acupuncture), while 6 had not. There was no significant difference between the number of males and females in the group which had taken the OSCE, and those students who had not taken the OSCE (Table 3).

Survey results

Students who took the OSCE were asked whether they were willing to use this application (of acupuncture and moxibustion medicine) in the clinical field, whether the limited time for each of the OSCE stations (acupuncture, electroacupuncture, pharmacopuncture, auricular acupuncture, and burning acupuncture) was sufficient, and whether the models, and checklist used for education, evaluation, and the OSCE were appropriate (Table 4). The percentage of students who reported that they were willing to use a particular application in the clinical field was highest for acupuncture (4.923 ± 0.2700) and lowest for auricular acupuncture (3.256 ± 1.1406). The percentage of students who reported that the OSCE time limit (5 minutes) was sufficient was lowest in pharmacopuncture (4.205 ± 1.0047). The percentage of students who thought that the model was appropriate was also lowest in pharmacopuncture (3.718 ± 1.0247). The adequacy of the OSCE checklist was the lowest for auricular acupuncture (4.103 ± 0.7538).

In the group who had taken the OSCE, 82% agreed that online education helped them improve their clinical performance, and 85% responded positively to whether the materials and instruments used for practicing clinical skills were sufficient. Regarding the appropriateness of the Clinical Skills Center being used for practicing clinical skills, there was a positive response (3.92/5, 82%). A positive response (3.77/5, 69%) was also observed regarding the OSCE taking place in the clinic. Of the students who had taken the OSCE, 67% reported that the OSCE accurately reflected their competency in clinical skills (Table 5).

Regardless of taking the OSCE or not, the responses to the question, “I can use this clinical application on a real patient” were generally high with acupuncture scoring the most, followed by electroacupuncture, pharmacopuncture, auricular acupuncture, and burning acupuncture. The difference in the average score between the students who had taken the OSCE and those students who had not taken the OSCE was the highest in the burning acupuncture group (0.923; Table 6). However, using the Mann-Whitney test, acupuncture (p = 0.987), electroacupuncture (p = 0.483), pharmacopuncture (p = 0.483), auricular acupuncture (p = 0.909), and burning acupuncture (p = 0.092) showed no difference in self-efficacy between the students who had taken the OSCE and those students who had not taken the OSCE. Using Spearman’s correlation analysis, the correlation coefficient between acupuncture and electroacupuncture was 0.716, which indicated that confidence in the electroacupuncture procedure tended to increase when the confidence in the acupuncture procedure increased (Table 7). In addition to these 5 clinical applications, participants were instructed to choose other clinical skills such as moxibustion, cupping, warm needling, and needle-embedding acupuncture which would be considered necessary in clinical practice (multiple selections were possible). As a result, needle-embedding acupuncture (32), cupping (12), warm needling (9), moxibustion (8), and others (1 acupuncture technique, 1 selecting acupoints, and 1 acupotomy) were selected.

Using open-ended questions, the clinical practice of acupuncture and moxibustion medicine and assessment by OSCE was investigated to determine its helpfulness. Participants responded that they could learn specific procedures of clinical applications and various techniques that they had not encountered before. They also positively reported that they were able to obtain appropriate feedback after the OSCE, and gain confidence in their clinical performance. Some replied that video lectures helped them understand their clinical performance procedure in detail. The difference in the practice model from the human body, lack of realism in the OSCE, decreased concentration on the technique itself due to a large number of items in the checklist, and the insufficient time limit applied during the OSCE, were all reported by students as disappointments.

Education of students in clinical practice at the College of Korean Medicine is mainly conducted in clinical rotation with a small group of students [9,19]. In this study, 91 students divided into groups of 3–4 conducted clinical applications for 1 week each and were assessed by the OSCE (acupuncture, electroacupuncture, pharmacopuncture, auricular acupuncture, and burning acupuncture). A survey was conducted to identify the effectiveness of education, appropriateness of models used in education and evaluation, and aspects that needed to be improved.

Due to the spread of COVID-19 in 2020, some students did not have face-to-face clinical practice. Of the 91 students contacted, 45 responded to the survey, and amongst these respondents, 6 had only been able to learn online, had not practiced, and had not taken the OSCE. According to a survey of the students who had taken the OSCE, the skills that they would use in the clinical field were (in order of preference), acupuncture, electroacupuncture, pharmacopuncture, burning acupuncture, and auricular acupuncture. This should be considered when introducing practical examinations in the future. In addition, when asked which clinical application of acupuncture and moxibustion medicine would you want more education in, needle-embedding acupuncture scored the highest. This confirmed the demand for a practical education and examination in needle-embedding acupuncture.

In this survey study, the response that the examination time (limited to 5 minutes) was sufficient scored > 4.2 points in all 5 clinical applications. The pharmacopuncture score was the lowest of the clinical application scores, and in the OSCE, some students could not finish the pharmacopuncture procedure due to time limitations. In open-ended questions, some students also responded that the time limit in the OSCE was insufficient. Each clinical application has a different process and hence a different performance time, so it is necessary to set an appropriate time limit for each clinical performance.

Since acupuncture-related clinical applications are invasive and require close supervision due to the potential risks, it is challenging for students to gain enough practice of these techniques. Therefore, in this study, models of skin, muscle and ear were used to practice injections for clinical performance, education, and evaluation. The average score range of the students who responded to the appropriateness of the skin, muscle and ear models was 3.7–3.8 (out of 5) for all clinical applications. Although the average score was not low, the models used in this study were limited by the differences in shape from the human body; it was impossible to select acupoints in the OSCE practice and thus the ability of the student in an actual treatment could not be evaluated. During the OSCE practice, the students were asked to explain how to select acupoints instead of directly selecting the acupoints. In addition, the OSCE practice was not conducted on simulated patients, so some checklists such as “Guide patients to pose a proper and safe position for treatment” were not realistic. In addition, the human body and the model used in the OSCE practice were bound to have differences in texture. These factors were also pointed out as areas for improvement of the OSCE in the survey essay questions. Some researchers have developed models for the OSCE, including the development of a human model system for teaching acupoint locations [20]. It is necessary to develop models that accurately represent the anatomical structure and texture of the human body, which can be used to evaluate students’ selection of acupoints. Furthermore, using simulated or real patients or simulator in the OSCE stations will further enhance the consistency and effectiveness of training.

The OSCE checklist was modified from the clinical practice guidelines for acupuncture and moxibustion medicine [14] based on the characteristics of each procedure, and the fact that the OSCE was conducted on models, not on the human body. In the acupuncture station, the procedure to confirm the patient’s de qi was excluded. Tonification and sedation were also excluded as specific instructions for this method were not provided in the clinical practice guidelines for acupuncture and moxibustion medicine. The instruction for electroacupuncture presented only the volume of the procedure, except for the instruction that electroacupuncture should be performed without pain. In the pharmacopuncture station, students were instructed to explain how to select the acupoints, instead of directly finding the suggested acupoints. In the burning acupuncture station, the question, “Did you use burning acupuncture before the red heated needle body cooled down?” is different from the burning acupuncture that heats the middle of the needle body; therefore, it was excluded. The “general evaluation” section, which evaluates whether the overall procedure was performed well or completed within an appropriate time, was excluded because the evaluation criteria were unclear (Appendices).

In the survey of students, the adequacy of the checklist was higher than 4.1, on average, in all clinical applications. However, there is a need to improve the checklist by including missing items during the procedure and modifying the questions that include too many items in 1 statement. When introducing practical tests in the future, it will be necessary to prepare specific scoring standards by gathering opinions from experts. In addition, it is difficult to evaluate both performance and communication in such a short period of time, so items such as a combination of diagnosis and treatment using simulated patients are suggested as alternatives [21].

Students practiced clinical applications autonomously at the school’s Clinical Skills Center without a strict time limit. In the OSCE practice, a small group of 3 to 4 students per week were evaluated instead of evaluating large-scale students at the same time. The OSCE was conducted in the clinic instead of multiple sites or the Clinical Skills Center. The patient care area can be used as the OSCE venue, and the prerequisite of using this area is that neither medical treatment to patients nor evaluation should be disturbed [8]. Therefore, Korean medicine doctors who organized the OSCE worked closely with nurses to prevent patient care and student evaluation times from overlapping. The response that the Clinical Skills Center is suitable for practicing clinical skills was high at 3.92 ± 0.87. However, regarding whether the clinic is suitable for implementing the OSCE, the score was lower, being 3.77 ± 1.01. Although the OSCE in the clinic has the advantage of increasing the sense of reality since it is a space for treating real patients, overlaps between patient care and student education are likely to occur in the clinic and interfere with each other. Therefore, it would be desirable to use the Clinical Skills Center for the OSCE practice. However, if the hospital is far from the school’s Clinical Skills Center, the clinic could be used for the OSCE venue instead of a Clinical Skills Center.

As a result of the COVID-19 pandemic, the use of online lectures instead of face-to-face classes increased rapidly, and the need to find learning methods which did not require hands-on training also increased [22]. In addition, with the clinical rotation system, teachers should repeat the same content for each group; however, there is a possibility that the content explained may slightly differ from group to group. Therefore, flipped learning with video lectures was used to effectively educate students using the same learning content. More than 80% of the respondents who experienced the OSCE reported that online video lectures helped them to improve their skills, so it is necessary to use flipped learning in future clinical performance education to familiarize students with basic theories online and practice intensively offline.

Of the 45 respondents, 67% agreed that the OSCE accurately reflected their competency in clinical skills, indicating that the OSCE was generally an effective assessment of clinical skills training. Self-evaluation in the potential use in the future of the 5 clinical applications between the students who had taken the OSCE and those students who had not taken the OSCE did not show any significant difference. The average self-evaluation score of acupuncture practice was higher in the students who had taken the OSCE than in those students who had not taken the OSCE. It was hypothesized that the OSCE did not affect the students because they had previously experienced acupuncture over the course of meridian and acupoint practice and other clinical practicums. The difference in the average score of self-efficacy between the students who had taken the OSCE and those students who had not taken the OSCE was 0.923 in burning acupuncture, the largest among the 5 clinical applications, indicating that burning acupuncture education and the OSCE were effective. The electroacupuncture and burning acupuncture processes included most of the acupuncture practice procedures, but the self-efficacy correlation coefficient in acupuncture and electroacupuncture was relatively high, whilst that of acupuncture and burning acupuncture procedures was low, suggesting that self-efficacy can differ, even if the procedure is similar.

The limitations of this study are as follows (1) The survey was conducted only in students, and the response rate was not high (50%). Since only 6 survey respondents had not experienced the OSCE, the effectiveness of the OSCE was not objectively evaluated. (2) Since attitude evaluations were excluded from the checklist, it was not possible to evaluate communication skills and attitudes by using OSCE. (3) This study was conducted at a single institution, therefore, the number of study participants was not sufficiently large. Nevertheless, this study is meaningful in that it specifically reported a case of teaching various clinical skills of acupuncture and moxibustion medicine and investigated students’ perceptions of the OSCE. Most students who took the OSCE reported that the OSCE reflected their clinical skills well, indicating that training for various other clinical applications, such as needle-embedding acupuncture, cupping, and warm needling, could be included in the OSCE, and is warranted. To promote clinical education in the future, the OSCE components, such as checklists, time limits, and instructions for each skill, should be supplemented, and further studies on Korean medicine education are needed. It is also believed that prospective or retrospective control studies are necessary to objectively evaluate the effectiveness of the OSCE in future clinical skills training.

The overall procedure of this study. Firstly, lecture videos were created and revised after review. Lecture videos were uploaded to the school’s learning management system so that students could learn in advance before practicing clinical applications of acupuncture medicine. An OSCE was conducted after practicing application skills at the clinical practice center. After the clinical practice, online surveys were conducted on the lecture videos and the OSCE.

Objective structured clinical examination (OSCE). (A) Materials used in the OSCE. (B) Clinical waste bins located in contaminated area. (C) Clinic room used as an OSCE venue.

Table 1. Models and Materials Used in Each OSCE Station.

Table 2. Instruction for Each OSCE Station.

Table 3. Gender of Survey Respondents and Whether They Had Taken the Objective Structured Clinical Examination.

Table 4. Students’ Evaluation for Each OSCE Station.

Table 5. Students’ Opinions on the Experience of Taking the OSCE.

Table 6. Analysis of Self-Efficacy for Clinical Practice.

Table 7. Correlation Between Rotations in the OSCE in Clinic and Self-Efficacy.