INTRODUCTION
Quality in medical residency programs has received a lot of attention, as many young physicians are pursuing advanced level training in subspecialties after finishing medical schools.1,2 A successful residency program is key to maintaining the balance of supply and demand for any medical subspecialty in today's world.3,4
In order to ensure the quality of the training program, evaluations of faculty, residents, and the program itself providing residency program play a fundamental role. Evaluating the clinical competence and knowledge of medical residents is practiced in almost each training program.5,6 Many studies have looked at the tools and process for evaluating residents in medical subspecialties.7,8 In contrast, the literature on evaluation of medical faculties by residents is scarce. Very few studies have looked at the process and tools for residents evaluating their faculty in postgraduate medical training programs.9,10 Research lacks in few important areas related to evaluations including the perceptions of faculty members regarding residents evaluating them, and perceptions of trainee residents about evaluating their faculty. Considering these facts, it is deemed important to identify the barriers involved in the process of evaluation and mentoring. Hence, it is becoming a major concern that medical schools have to recognize mentoring based on the teachings.11
Residents are the key stakeholders of any training program, hence their perspective regarding their residency training, learning environment, and relationship with faculty is vital to their performance at the clinical setting.12 Supportive supervision and healthy faculty and resident relationship are important to make teaching-learning a happy experience in postgraduate education.13 Therefore, the faculty has a two-way job in clinical arena, one to provide quality patient care and is the other to supervise, mentor, and train the residents. In order to improve the teaching and training of residents, continuous professional development of faculty members is important.
The faculty evaluation system facilitates identification of red flags associated with the faculty's performance, thereby providing timely assessment of their progress. This allows for early intervention and remediation to correct these issues. Though it needs initial investment by the parent department, on a long-term basis it utilizes fewer of the faculty and program's time and resources.
Teaching evaluation ensures program goals are being met, teaching standards are being maintained and improved, thus leading to better learning for students and residents and an overall improvement in patient care along with global enhancement in enthusiasm and confidence of the faculty.14
To ensure excellent evaluation, attention must be paid to what is fundamentally essential for evaluation. Snell et al14 characterize it as aiming for high levels of validity, reliability, efficiency, and feasibility, and aiming for methods that are minimally burdensome. The goals of the evaluation should be predefined, e.g., if they will have an impact on a teacher's career. The evaluation should be consistent with the culture and philosophy of the institution and at the same time be linked to curriculum development objectives of the institution. Hence, in order for it to be successful, it should incorporate the ideas of many individuals, including the teachers themselves, learners, peers, supervisors, and other health professionals involved. Finally, an effective evaluation system should be easy to administer, be applicable to all teachers, and be effective across all departments.14
In order for this process to be streamlined, a constant feedback about the method of evaluation plays a huge role. Literature has shown the predictive validity of evaluations on the future performance of the faculty.15,16
Residency programs in medical subspecialties are common at both public and private teaching hospitals in Pakistan. Various assessments either structured or unstructured have been formalized to evaluate the knowledge and performance of residents.17,18 However, faculty appraisal by residents does not exist in many residency programs in Pakistan. In the last few years, the Department of Pediatrics at Aga Khan University has been striving to restructure and improve the pediatric residency program. An important issue brought forth by the residents was that although residents are being evaluated by the faculty after each rotation as mandated by the University's Post Graduate Medical Education Committee, the process of residents evaluating or giving feedback to faculty is not very well developed. In an effort to improve upon this, an online evaluation system was established in which the faculty has being evaluated in all domains relevant to their teaching objectives; these include knowledge, clinical competence, teaching effectiveness, and professional attributes.
This study aims to assess the evaluation process of recently introduced online faculty evaluation by residents in a pediatric residency program at a private teaching hospital of Pakistan. The objectives of this study were to explore perceptions of pediatric faculty and residents about the online faculty evaluation by residents and to understand the strengths and limitations of online faculty evaluation and strategies to improve the faculty evaluation process.
MATERIALS AND METHODS
Study Design
This study utilized action research for the process of online faculty performance evaluation by residents of Pediatrics and Child Health Department at Aga Khan University, Karachi, Pakistan. Two important arenas in action research which are frequently cited include patient care and medical education.19,20 A process evaluation was conducted as part of step III activities aiming to observe and reflect the process, after planning and introduction of online faculty evaluation over the 1-year period (Fig. 1).
We employed FGDs as a method of data collection between April and May 2013. We chose qualitative assessment because there is paucity of published literature about research topic in both the local and international context, and basic information is needed to establish a hypothesis for further research. The FGDs allowed participants in a small group to discuss the subject freely and spontaneously on the study theme, guided by a facilitator.21
Study Population and Sampling Strategy
The study population was the full time faculty members of Pediatrics and Child Health Department, and resident physicians enrolled in the pediatrics residency program of Aga Khan University. The reason for involving participants representing both faculty and residents was to identify their understanding of how they see the faculty evaluation in general, strengths, limitations, and recommendation to improve the system, so as to get the input from users and end receivers of the evaluation system under study. These faculty members were purposively selected from the list of all faculty members practicing pediatrics in any subspecialty. Likewise, a group of pediatric residents were also purposively selected from the list of all resident trainees from year 1 to 4 training program.
Data Collection, Management, and Analysis
A semi-structured FGD guide was developed from literature review, aiming to explore perceptions of faculty and residents, strengths, limitations, and recommendations to improve the online evaluation process. A senior public health scientist trained in qualitative research conducted separate FGDs with groups of pediatrics faculty and residents. Average participation in each FGD was four to six participants. Altogether five FGDs were conducted, out of which three were with faculty and two were with the residents. Data saturation was reached, and data collection was completed accordingly.22
Every participant in the FGD was assigned with a unique identification number with respect to their seating arrangement, and their responses were transcribed using these numbers. The FGDs were tape recorded, transcribed, and uploaded on NVivo version 10.0 for thematic analysis. Responses of participants were organized into categories (nodes on NVivo) and were connected in a hierarchy (parent nodes) according to each theme.23 Later on, interpretation was made for each emerging theme and subtheme.
This was an observational study; therefore, no physical or emotional risk or harm was associated with the data collection. Participants were selected from the list, were invited for FGD on voluntary basis, and informed consent was obtained from every participant before conducting FGDs. The participants’ confidentiality was maintained by virtue of coding individual responses on the emerging categories.
RESULTS
A total of 27 participants attended FGDs, out of which 16 were faculty members and 11 were pediatric residents. In each FGD, opinions and attitude of faculty members and residents are elaborated into themes and subthemes.
Theme I: Ambiguity about the Purpose of Faculty Evaluation
Some faculty members mentioned that they were not clear about the purpose of faculty evaluation, and raised critical concerns about residents evaluating their faculty. Main concern was that group of residents may misuse this evaluation to rate faculty unjustly because of their one unfavorable precedent with any one resident. Some feared that it might negatively impact the growth of faculty and the residency program itself. A participant in FGD with faculty revealed “Are they not doing these evaluations to take revenge from faculty? It could be because of the insult he or she has done in front of their colleagues” (FGD 1 with faculty).
On the contrary, some faculty members considered faculty evaluation as a productive exercise for upgrading the faculty and residency program. Likewise, majority of the residents appreciated the faculty evaluation and considered it as a two-way process that enabled them to express their satisfaction and dissatisfaction with the faculty and the training program. Very few residents believed that the faculty evaluation is just a formality, and they doubted any impact of their evaluation on faculty and the program. A resident mentioned that “No use of this form, as we fill it only for the sake of formality nothing else” (FGD 1 with residents).
Theme II: Lack of Understanding and Concerns on the Content and Components of Faculty Evaluation
Of all faculty members who participated in FGDs, only two had seen the evaluation form and were aware of the components of their evaluation. Many of them criticized that they had not seen the details of every component of evaluation, and received cumulative scores of their performance in each component. Many pointed out the issues with the component of the evaluation form. One of the most important issues identified was short interaction time of faculty with residents in clinical rounds as well as in outpatient clinics, one of the reasons being high volumes of patients. They criticized that residents don’t come with clear learning objectives, and mainly depend on the faculty for clinical teaching in the form of “blackboard teaching.” According to a participant in FGD “Does a resident know about the learning process or objective which he or she should achieve? According to my knowledge residents feel that teaching is not done if teacher does not teach us with blackboard and marker” (FGD 1 with faculty). A few of them considered the evaluation form incomprehensive, with missing components of softer skills, such as caring attitude of faculty, role modeling in clinical practice, and professionalism.
In contrast, majority of the residents believed that all components of evaluation were comprehensive. However, some reported that evaluation form is very lengthy and time consuming. One of the participants commented, “The online evaluation of faculty is a very comprehensive and tiresome process” (FGD 3 with residents).
Subtheme: Problems with Rating Scale of Selected Components of Evaluation
Few of the faculty members reported discrepancy related to some components of evaluations with the level of residents (years 1 vs 4). According to one participant “In my opinion this form requires revision as R1 has to evaluate faculty and the same goes for R2, R3 and R4. Let's suppose Dr. X is in charge of the metabolic section how can R1 and R2 can evaluate her” (FGD 2 with faculty). Along with that, they thought few components in the evaluation form needs “Not applicable” option, which does not exist on the rating scale and is needed for some of the components. They also suggested incorporating a Likert's scale instead of percentage ratings.
Some faculty members also highlighted that average scores collected from the ratings of all residents (R1—R4) are not appropriate measures to make inferences about faculty performance. A participant narrated, “In clinic a single resident rotated last year and if I had a bad word with him/her then I would expect a bad evaluation from that resident.”
“If a small numbers of residents are evaluating their faculty there is increase chances that it may leads to imprecise calculation to avoid this there should be a defined criteria for number of residents evaluating each faculty (FGD 2 with faculty).
Theme III: Perceived Barriers or Challenges of Faculty Evaluation
Respondents from both group's faculty and residents largely underscored a number of challenges and barriers in evaluation process. Main challenges reported by residents included difficulty in filling the online form, apprehension of invasion in confidentiality, and prejudiced evaluation. The chief challenges reported by faculty included erroneous calculation, handling of survey forms, and invasion of personal privacy of their performance (Table 1). Some residents and faculty revealed that they are scared to give fair evaluation to attending due to the fear of bearing the repercussions on their evaluations.
Theme IV: Perceived Benefits of Faculty Evaluation
Despite challenges, majority of the residents believed that faculty evaluation will play pivotal role in improving the faculty performance and bringing positive impact on the residency training program. The online system facilitated them to fill the evaluation form in their leisure time. They also appreciated the administrative support of sending timely reminders for online submission.
Table 1
Perceived barriers/challenges of the faculty evaluation
| Challenges reported by residents | Challenges reported by faculty |
|---|---|
On the contrary, many faculty members realized that they would equally benefit from the evaluation and it will help them improve in terms of their professional growth and training of the young professionals in this field. Many faculties and residents agreed that the best rated faculties should be recognized and appreciated by the department and the university. One participant narrated, “We should introduce a letter of appreciation for those faculty members who secure higher than the prescribed bench mark” (FGD 1 with faculty).
However, some of the faculty members were skeptical about the benefit of faculty evaluation on their professional growth doubting that part of the evaluation by residents may mislead or undermine faculty's professional growth.
Theme V: Recommendation for improving the Faculty Evaluation System
A number of suggestions and recommendations were proposed by both faculties and residents to improve online evaluation process. The first and foremost recommendation by the faculties was to arrange a retreat with both faculty members and the residents to discuss necessary amendments in some of the components of the evaluation form, as well as to highlight the successes and challenges experienced over the last 1 year.
Secondly, many faculties and residents recommended having appropriate rating scale and analysis process defined, so that one factor or one evaluation form does not affect the evaluation of faculty. According to one resident, “In my opinion if all residents mentioned the same kind of weaknesses about a particular faculty then it should be a matter of consideration. Otherwise, if a faculty got bad evaluation from one or two then it should not be considered as bad” (FGD 3 with residents). According to one participant, “Evaluation should be done soon after completion of rotation. So that we are be able to get the right feedback” (FGD 1, with faculty).
Many residents and faculties reported that handling of filled evaluation forms, levels of review, and raters’ and faculty's privacy should be strictly maintained. Most of them recommended that only chair or program director should have access to the filled forms. Besides, they also proposed to share and discuss the feedback with faculty on one-to-one meeting rather than sending ratings on e-mail.
DISCUSSION
Modern scientific studies do not provide enough evidence on how faculty perceives when their trainees evaluate them. This study for the first time explored perceptions of residents and faculty about online faculty evaluation process in a tertiary care teaching hospital of low- and middle-income setting.
This study underscored the need for including professionalism and softer skills in the faculty evaluation form. This finding corroborates with literature that evaluating professionalism is an integral part of faculty development and core attributes of professionalism including competence, commitment, integrity and honesty, morality and ethics, altruism, autonomy, self-regulation, responsibility to society and profession, and teamwork.24 Another study reported that professionalism in medical education has been mainly focused on curricula in medical school, and received less attention for faculties and residents.25 Therefore, a further research is needed as to how professionalism perceived by faculty and residents working at the teaching hospitals, particularly in low- and middle-income countries where standards for postgraduate medical studies are suboptimal.
Moreover, this study highlighted concerns related to utility of any such feedback for faculty, and apprehension about breach of confidentiality, which made residents less interested in expressing true feedback for faculty. During FGDs with residents, they revealed fear of retaliation and were unsure if their feedback was conveyed to faculty without names. Despite the evaluation made online, they were afraid that faculty may likely come to know who evaluated them. Literature suggested that anonymous evaluation for faculty is crucial in improving teachers’ performance.26 Web-based evaluation or monkey surveys are very buzz word these days. Because online evaluation system enables anonymity, it has become customary in many organizations including health sciences.27 Hence, the literature fully supports integrating online anonymous faculty evaluation in residency program.26
On the contrary, rigidity among some faculties to accept the evaluation from students was another major concern, which may undermine the evaluation of residency training program as whole. The concept of 360° feedback is rapidly emerging in many professional settings including medical education, in which everybody who is involved is eligible to evaluate.28,29 However, it was surprising to note that some senior faculty members were not content with this system. If any such argument prevails among faculty in a private tertiary care hospital, there is high likelihood that the concept of 360° feedback might not thrive in other private and public sector hospitals, where many seniors who have long-standing professional experience work in Pakistan. Because we could not corroborate this finding with any of the published literature work, therefore, this stands as an observation from this study only.
Faculty evaluation in medical residency program is not practiced in other postgraduate medical education departments at the tertiary hospital where this study was conducted, and perhaps is uncommon in other public and private teaching hospitals in Pakistan. Therefore, this is the first kind of its study conducted in a local context to have looked at the process of faculty evaluation in medical residency program in Pakistan. In fact, this study paved the basis for future research work to establish and improve the faculty evaluation and recommend it to pilot it in other medical departments of the same teaching hospitals. Main challenge faced during the study was to arrange faculty members from different subspecialty on the same day/time for FGD; therefore, we had to reschedule the FGD several times. Likewise, gathering residents who were assigned in off-campus and on-campus rotations was difficult too. As a result, data collection of this study took more time than expected.
CONCLUSION
This study concluded that understanding of operational concerns, challenges, benefits, and strategies for faculty evaluation in the residency program was an important exercise, and lessons learnt helped to understand and make improvement in the online faculty evaluation in pediatrics residency program.