|Year : 2015 | Volume
| Issue : 4 | Page : 141-146
The impact of assignment strategy on the performance of postgraduate trainees: An analysis of the assessment scores database in a military medicine system
Shih-Hung Tsai1, Cheng-Jueng Chen2, Jin-Shuen Chen3, Yi-Jen Peng4, Wen-Yi Liao5, Jen-Chun Wang5, Wei-Chou Chang6, Wei-Kuo Chang7
1 Department of General Medicine and Emergency Medicine, National Defense Medical Center, Tri-Service General Hospital, Taipei, Taiwan
2 Department of General Medicine and Surgery, Division of General Surgery, National Defense Medical Center, Tri-Service General Hospital, Taipei, Taiwan
3 Department of Internal Medicine, Division of Nephrology, National Defense Medical Center, Tri-Service General Hospital, Taipei, Taiwan
4 Department of Pathology, National Defense Medical Center, Tri-Service General Hospital, Taipei, Taiwan
5 Department of Emergency Medicine, National Defense Medical Center, Tri-Service General Hospital, Taipei, Taiwan
6 Department of Radiology, National Defense Medical Center, Tri-Service General Hospital, Taipei, Taiwan
7 Department of General Medicine, National Defense Medical Center, Tri-Service General Hospital, Taipei, Taiwan
|Date of Submission||17-Dec-2014|
|Date of Decision||16-Feb-2015|
|Date of Acceptance||16-Mar-2015|
|Date of Web Publication||28-Aug-2015|
Department of Internal Medicine, National Defense Medical Center, Tri-Service General Hospital, No. 325 Cheng-Kung Road, Section 2, Neihu 114, Taipei
Source of Support: None, Conflict of Interest: None
Background: Medical students who graduate after 2003 need to participate in the "Postgraduate Primary Care Medical Training Program," provided by the Taiwan Joint Commission on Hospital Accreditation. We wish to know how the education and assignment strategy influences National Defense Medical Center (NDMC) graduates. We examined whether the performance of postgraduate year (PGY) trainees would be affected by the assignment strategies. Materials and Methods: We consecutively collected 173 NDMC graduated trainees who participated in 6-months of PGY training and another 94 trainees who participated for 1-year. During the training period, all the trainees were evaluated by several assessment tools. Trainees were dichotomized according to the levels of dispatched hospitals and preassigned specialty, respectively, to evaluate those effects on the performance of the 6-month PGY training period. Results: We describe the assessment scores of NDMC graduate M.D. trainees engaged in the Tri-Service General Hospital (TSGH) PGY training program. PGY trainees who were preassigned to TSGH due to better overall averages in medical school had better scores in case-based discussion, mini-clinical evaluation exercise, and direct observation of procedural skills than those who were preassigned to the Regional Teaching Hospitals. There was no statistically significant difference in the overall scores. The preassigned specialties themselves did not affect the performance in PGY training. Incorporation of PGY scores in the assignment strategy significantly elevated the performance of PGY trainees who were dispatched to TSGH. Conclusion: The accumulated PGY training database provides educators a unique opportunity in reviewing the effects of education policies and strategies. Assignment strategy could affect the performance of PGY trainees during their training period.
Keywords: Military, National Defense Medical Center, resident, medical education, postgraduate year, Tri-Service General hospital
|How to cite this article:|
Tsai SH, Chen CJ, Chen JS, Peng YJ, Liao WY, Wang JC, Chang WC, Chang WK. The impact of assignment strategy on the performance of postgraduate trainees: An analysis of the assessment scores database in a military medicine system. J Med Sci 2015;35:141-6
|How to cite this URL:|
Tsai SH, Chen CJ, Chen JS, Peng YJ, Liao WY, Wang JC, Chang WC, Chang WK. The impact of assignment strategy on the performance of postgraduate trainees: An analysis of the assessment scores database in a military medicine system. J Med Sci [serial online] 2015 [cited 2019 Jul 22];35:141-6. Available from: http://www.jmedscindmc.com/text.asp?2015/35/4/141/163821
| Introduction|| |
Medical students who graduate after 2003 need to participate in the "Postgraduate Primary Care Medical Training Program," provided by the Taiwan Joint Commission on Hospital Accreditation. The purpose of this program is to bridge the gaps in medical school education through clinical training in general medicine after graduation, and to develop 1 st year resident physicians' abilities in diagnoses, treatments and holistic patient care. The training program adheres to the standards of the Accreditation Council for Graduate Medical Education (ACGME). Since 2006, the 6-month postgraduate year (PGY) training program has been required in official resident training. PGY trainees were required to complete a 6-month training course, consisting of 1 month of emergency medicine (EM), 2 months of community medicine, and a 3-month specialty course in internal medicine (IM), surgery or pediatrics (PED).
The National Defense Medical Center (NDMC), a Republic of China Ministry of National Defense (MND) - Affiliated Medical College, is one of the oldest Medical Schools in Taiwan. It is a military academy whose graduates become military medical professionals, and it also has a graduate school component. Medical students who graduate from NDMC are required to serve a 2-year initial utilization tour in standard military organizations and will be assigned to line units. Every year, nearly 30 graduates have been assigned to the NDMC affiliated medical center, the Tri-Service General Hospital (TSGH); the remaining graduates have assigned to the other Department of Health, Executive Yuan-certified Regional Teaching Hospitals (RTHs).
Assignment strategy for NDMC graduates could have tremendous effects on maintaining adequate and balanced coverage of all specialties in all military hospitals. The decisions of dispatched hospitals and preassigned specialties are determined based on a winnowing process which takes into consideration the needs of the military service, cadets' class ranking, and finally the desires of the individual [Figure 1]a. Since 2012, a full 1-year training course is required for further resident training. With the implementation of 1-year PGY training, the assignment strategy has also changed since 2012. The assessment scores during the PGY training period started to be partially incorporated into the assignment strategy (this commenced with medical students who graduated in 2010) [Figure 1]b.
|Figure 1: The assignment strategies of the NDMC graduates. (a) Diagram of assignment strategy of the NDMC graduates before 2013. (b) Diagram of new assignment strategy for NDMC graduates after 2013. NDMC = National Defense Medical Center; TSGH = Tri-Service General Hospital, GPAs = Graduate point averages|
Click here to view
Medical education is evolutionary. We aim to know how the education and assignment strategies influence NDMC graduates. We examined whether the performance of PGY trainees would be affected by the assignment strategy.
| Materials and Methods|| |
We consecutively collected 173 NDMC graduated trainees who participated in 6 months of PGY training and another 94 trainees who participated for 1 year. During the training period, all the trainees were evaluated by several assessment tools, including ACGME six core competencies mid-term and final assessments, case-based discussion (CbD), mini-clinical evaluation exercise (mini-CEX), direct observation of procedural skills (DOPS), 360° evaluation (teachers, patients, peers and self-evaluation) and in-training examination (ITE). The overall scores were calculated by averaging the monthly scores. As shown in [Table 1], the weight of the used assessments in respective training specialties was predetermined in the faculty consensus. Only self-assessments were not included in the average. The workplace-based assessments (WBAs) have their roles as formative assessments that prompt supervision, feedback, and reflection. The aim of using multiple assessment tools is to objectively represent the assessment of professional competence. , CbD assessments provide self-learning and feedback and should be regarded beneficially as educational sessions, as opposed to mere assessment tools.  Specialists and the trainees should view the CbD as a commitment of the clinical education and challenging cases, resulting in more constructive feedback.  Mini-CEX is generally considered as a fair, assessable, and manageable tool with good reproducibility. It does ensure that different faculty members observe a reasonable sample of the resident's clinical skills.  The observation and feedback occur with a broad range of patient problems in various settings.  DOPS scores appear to reflect increasing acquisition of operative skills.  During the observation period, trainees took responsibility for decisions, and specialists learned more about their abilities.  The reported reliabilities in CbD, mini-CEX, DOPS and 360° evaluation were shown to be better than moderate agree in relevant studies conducted in Taiwan medical education system. ,
|Table 1: Predetermined weight of the used assessments in respective training specialties|
Click here to view
For the evaluation of effects of dispatched hospitals on the performance of the 6-month PGY training period, we dichotomized trainees into two groups: Those who were dispatched to the NDMC affiliated medical center (TSGH) and those sent to the RTHs. For the evaluation of the effects of different preassigned specialties, trainees were dichotomized into two groups: Those who had a preassigned specialty of IM, surgery (S), EM, PED, and obstetrics/gynecology, since 2012 and those who were not included in the above-mentioned specialties. To further evaluate the effects of incorporation of PGY assessment scores in the TSGH assignment strategy of the PGY trainees who were dispatched to TSGH, we compared the assessment scores between PGY trainees who attained the training before and after the implementation of the new assignment strategy that incorporated 15% of PGY assessment scores.
Continuous data were expressed as means ± standard deviations and analyzed using the two-tailed Student's t-test. Categorical data were expressed as frequencies (%) and tested using the Chi-squared or Fisher's exact test. Data were analyzed using SPSS 18.0 (SPSS, Inc., Chicago, IL, USA) and a P < 0.05 was considered statistically significant.
| Results|| |
The demographic data of the enrolled trainees are shown in [Table 2]. As shown in [Table 3], PGY trainees who had been assigned to TSGH had better scores in CbD, mini-CEX and DOPS than those who had been assigned to other RTHs. There was no statistically significant difference in the overall scores (P = 0.58, [Figure 2]a); however, after the implementation of the new strategy, PGY trainees who were dispatched to TSGH had statistically significant higher PGY assessment scores than those to RTH, where the PGY assessment scores were not incorporated in the assignment strategy (P = 0.0078, [Figure 2]b).
|Figure 2: The effects of incorporation of postgraduate year (PGY) assessment scores in the assignment strategy on the performance of PGY trainees|
Click here to view
|Table 3: The effects of dispatched hospital in the assessment during PGY training|
Click here to view
As shown in [Table 4], there were no significant differences in the assessment scores between trainees who had their preassigned specialties which included IM, surgery, PED, and EM and those were not from above-mentioned specialties.
|Table 4: The effects preassigned specialty in the assessment during PGY training|
Click here to view
| Discussion|| |
Our study provided, for the 1 st time, an opportunity to describe the assessment scores of NDMC graduate M.D. trainees engaged in the TSGH PGY training program. PGY trainees who were preassigned to TSGH due to better overall graduate points averages (GPAs) in medical school had better overall PGY scores than those who were preassigned to the RTHs. The differences were mainly attributed to CbD, mini-CEX, and DOPS. However, the preassigned specialties themselves did not affect the performance in PGY training. Incorporation of PGY scores in the assignment strategy statistically significantly enhanced the performance of the PGY, who were dispatched to the TSGH.
Although there was no significant difference in the PGY overall scores between trainees from TSGh or RTHs; the finding of poorer performance in certain assessments, including CbD, min-CEX and DOPS, in trainees assigned to the RHTs worth further research and attention. Although we believe that the majority of the effects could be reasonably attributed to the GPAs, whether assignment to the RHTs per se would influence their motivation in learning deemed worth further comprehensive studies. Previous studies which involved analysis of a large dataset of WBAs also revealed a significant association between training difficulties and lower mean scores on both the CbD and mini-CEX.  In a study regarding the correlation between selection scores and assessment scores of a surgical education and training program in Australia, trainees who performed well in one examination tended to perform well in other examinations.  Performance in the mini-CEX was variable, perhaps reflecting limitations of this assessment.  DOPS should be viewed as a useful educational/training opportunity for trainees to improve performance in a skill. Timely, high-quality feedback to trainees is an essential part of the assessment. ,,, It is worth noting that CbD, mini-CEX, and DOPS are also viewed as education tools, and that timely constructive feedback is beneficial to trainees. The feedback received by the trainees could encourage them to develop and have an improvement in performance in subsequent assessments. Therefore, as clinical teachers, we should take every opportunity to conduct feedback efficiently in order to eliminate the gap between expected required competency and the actual ability of trainees in the PGY training period. We believe that measures to increase motivation, medical knowledge, and skills in procedures, as well as presentation, should be particularly strengthened in trainees who were preassigned to a RTH. To stimulate inactive trainers and trainees, external requirements set by training institutions may be required.  Greater benefit from the WBAs feedback could be obtained by:
- Trainee and trainers planning the observation and feedback together;
- Ensuring adequate content and delivery in the feedback, and
- Using feedback to guide trainees' learning by linking it to learning goals. 
A structured format can be used to broaden the scope of feedback and make it easier to address performance gaps.  Appropriate educational interventions should be triggered earlier to prevent overall failure rate and the waste of valuable military human resources.  A preassigned specialty strategy which does not affect performance in PGY training might reflect that the central concept of PGY training had been properly delivered to the trainees.
Assignment strategies for NDMC graduates have tremendous effects on maintaining adequate and balanced coverage of all specialties in all military hospitals. The changed assignment strategy resulted in a maximum shift in the ranking of 3 in 2012 trainees in TSGH residents (data not shown). The long-existing preassignment strategy has contributed to adequate coverage of all specialties in military hospitals and prevented our military medicine system from having a shortage of surgeons, gynecologists, pediatricians as well as IM and EM physicians, a currently growing health issue in Taiwan. Although differences in behavior from one trainee to another or from one situation to another can be traced to the incentives available and the value a person places on those incentives at the time.  The findings of new assignment strategy that incorporated the PGY scores in TSGH had resulted in better performance in the PGY training period should have reflect that trainees viewed that the opportunities in choosing their specialty in their subsequent career life as a positive incentive. The impact of new assignment strategy in the future military human resource should be further evaluated.
With respect to future comprehensive study in the association between trainees' scores in PGY training period and future clinical performance, we believe that the accumulated experience in teaching and assessment, the growing body of data from the Department of General Medicine and well-designed long-term follow-up projects will help us to develop outcome-based training program and eventually build competent residents in the field of military medicine. We will be able to observe how the assignment strategy influences the residents' attitude and clinical performance as well provide a recommendation to refine the assignment strategies. Although many tools are available for the direct observation of clinical skills, validity evidence, and descriptions of educational outcomes are scarce.  Correlating assessment with future performance is difficult not only because of inadequacies in the assessment process itself but also because relevant, robust measures of outcome that can be directly attributed to the effects of training have not been identified.  As a consequence, there has been an increased focus upon faculty development. The Department of General Medicine has constantly arranged consensus meetings and chart review process to normalize and validate the difference between observers. A global rating scale has been introduced as a subjective clinical anchor to the clinical performance.  Adherence to the assessment consensus is deemed necessary to prevent unfairness and argument. Nowadays, residents need to demonstrate not just competency, but also provide documentary evidence regarding attainment and continuation of this competency.  ITE scores in PGY years had been reported to predict the probability of passing the American Board of IM examination.  Some clinical performances, including providing proper consultations and certain screening tests, were better in physicians with good board examinations while the inappropriate antibiotic prescription was not correlated with physicians' scores in licensure examinations. ,, On the other hand, lower scores obtained in patient-physician communication and clinical decision making on a national licensing examination could predict complaints to medical regulatory authorities.  One study indicated that by increasing the passing criterion of only 1-SD in drug knowledge would have caused in failing additional 16 physicians over 4 years and resulted in reduced the expected risk of contraindicated prescription for elderly patients seen by these physicians by approximately 42% (from 4.7% for the low-scoring physicians to 3.3% for an average physician).  A 2-SD decreasing in communication score was associated with a relative 38% increase in the complaint rate (1.7 more complaints per 100 practice-year).  We believe that the significantly improved PGY scores in the TSGH trainees should have some positive effects on the future medical career of the trainees; and the new strategy would be beneficial to both individual trainees and the institution. Our experiences provided the rationale of incorporation PGY scores in addition to current assignment strategies used in other MND-affiliated hospitals.
The preassigned strategy of NDMC is unique from other medical schools in Taiwan. The single university-affiliated medical center environment made the results difficult to compare with data from other medical schools. Students who had better GPAs would have better PGY performance due to talent and input into learning in part. Unfortunately, lack of "measurable" parameters for personal characteristics and input into studying and learning in the medical school and even in the PGY training period had prevented us from the elimination of this limitation. Clinical PGY teachers are of paramount as the role models of the trainees during their PGY training. It might be still difficult to objectively quantify the educational skills as well the enthusiasm of the teachers. We will continue to value the bi-directional feedback from the trainees. That is actually why we need to build the PGY assessment databank that incorporates multiple assessments and description of the characteristics of trainees by the 360° evaluation as well bi-directional feedback. We will also continue to provide training courses and evaluate the adequacy in quality and quantity of PGY teachers in other MND-affiliated RTHs. Measurable tools to monitor the performance of the PGY teachers should be developed. Further studies regarding the reliabilities and validities of those assessment tools should be conducted as well.
| Conclusion|| |
Postgraduate year trainees who were assigned to the medical center due to higher GPAs had higher scores in CbD, mini-CEX, and DOPS and also had higher PGY assessment scores. Assignment strategy could affect the performance of PGY trainees during their training period. Incorporation PGY assessment scores into the subspecialties assignment strategies may have beneficial effects in enhancing the motivation during the training PGY period.
| Acknowledgments|| |
We are grateful for the English editing by Daniel Steve Villarreal, Ph.D., of NDMC's Center for General Education. We also acknowledge Dr. Yen-Yue Lin for the data collection.
| References|| |
Mitchell C, Bhat S, Herbert A, Baker P. Workplace-based assessments of junior doctors: Do scores predict training difficulties? Med Educ 2011;45:1190-8.
Weller JM, Jones A, Merry AF, Jolly B, Saunders D. Investigation of trainee and specialist reactions to the mini-Clinical Evaluation Exercise in anaesthesia: Implications for implementation. Br J Anaesth 2009;103:524-30.
Jyothirmayi R. Case-based discussion: Assessment tool or teaching aid? Clin Oncol (R Coll Radiol) 2012;24:649-53.
Mehta F, Brown J, Shaw NJ. Do trainees value feedback in case-based discussion assessments? Med Teach 2013;35:e1166-72.
Yang YY, Lee FY, Hsu HC, Lee WS, Chuang CL, Chang CC, et al.
Validation of the behavior and concept based assessment of professionalism competence in postgraduate first-year residents. J Chin Med Assoc 2013;76:186-94.
Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: A method for assessing clinical skills. Ann Intern Med 2003;138:476-81.
Oldfield Z, Beasley SW, Smith J, Anthony A, Watt A. Correlation of selection scores with subsequent assessment scores during surgical training. ANZ J Surg 2013;83:412-6.
Yang YY, Lee FY, Hsu HC, Huang CC, Chen JW, Cheng HM, et al.
Assessment of first-year post-graduate residents: Usefulness of multiple tools. J Chin Med Assoc 2011;74:531-8.
Chen YY, Chu TS, Wu CC, Ni YH, Huang TS, Hsu HC, et al
. Reliabilities of Mini-CEX, CbD, DOPS for evaluating the clinical performance of primary care medicine. International Association of Medical Sciences Educators (IAMSE) 17 th
Annual Conference, St. Andrews, Scotland, United Kingdom; 2013.
Bindal N, Goodyear H, Bindal T, Wall D. DOPS assessment: A study to evaluate the experience and opinions of trainees and assessors. Med Teach 2013;35:e1230-4.
Cohen SN, Farrant PB, Taibjee SM. Assessing the assessments: U.K. dermatology trainees' views of the workplace assessment tools. Br J Dermatol 2009;161:34-9.
Bodgener S, Tavabie A. Is there a value to case-based discussion? Educ Prim Care 2011;22:223-8.
Pelgrim EA, Kramer AW, Mokkink HG, van der Vleuten CP. The process of feedback in workplace-based assessment: Organisation, delivery, continuity. Med Educ 2012;46:604-12.
Wiener-Ogilvie S, Begg D. Formative assessment of GP trainees' clinical skills. Educ Prim Care 2012;23:101-6.
Bernstein D. Essentials of Psychology. USA: Cengage Learning; 2013.
Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: A systematic review. JAMA 2009 23;302:1316-26.
Epstein RM. Assessment in medical education. N Engl J Med 2007;356:387-96.
Eardley I, Bussey M, Woodthorpe A, Munsch C, Beard J. Response to Re: Workplace-based assessment in surgical training: Experiences from the Intercollegiate Surgical Curriculum Programme. ANZ J Surg 2013;83:691-2.
Setna Z, Jha V, Boursicot KA, Roberts TE. Evaluating the utility of workplace-based assessment tools for speciality training. Best Pract Res Clin Obstet Gynaecol 2010;24:767-82.
Brateanu A, Yu C, Kattan MW, Olender J, Nielsen C. A nomogram to predict the probability of passing the American Board of Internal Medicine examination. Med Educ Online 2012;17:18810.
Tamblyn R, Abrahamowicz M, Dauphinee WD, Hanley JA, Norcini J, Girard N, et al.
Association between licensure examination scores and practice in primary care. JAMA 2002;288:3019-26.
Meguerditchian AN, Dauphinee D, Girard N, Eguale T, Riedel K, Jacques A, et al.
Do physician communication skills influence screening mammography utilization? BMC Health Serv Res 2012;12:219.
Cadieux G, Tamblyn R, Dauphinee D, Libman M. Predictors of inappropriate antibiotic prescribing among primary care physicians. CMAJ 2007;177:877-83.
Tamblyn R, Abrahamowicz M, Dauphinee D, Wenghofer E, Jacques A, Klass D, et al.
Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA 2007;298:993-1001.
Schaffner W, Ray WA, Federspiel CF, Miller WO. Improving antibiotic prescribing in office practice. A controlled trial of three educational methods. JAMA 1983;250:1728-32.
[Figure 1], [Figure 2]
[Table 1], [Table 2], [Table 3], [Table 4]