Adoption of Internal Medicine Milestone Ratings and Changes in Bias Against Black, Latino, and Asian Internal Medicine Residents

Introduction

This study empirically evaluates how the implementation of a new program rating residents, termed “Milestone,” reduced      bias against Asian, Latino, and Black residents. The Milestone program utilizes new forms of performance evaluation, including competency trainings and knowledge-based questions for residents to evaluate their clinical excellence. The study used linear regression models and robustness checks to ascertain whether there was a reduction in racial and gender bias towards residents by resident directors in their performance evaluations because of this program.

Bradley M. Gray, Ph.D., Senior Health Services Researcher at the American Board of Internal Medicine; Rebecca S. Lipner, head of the Division of Assessment and Research at the American Board of Internal Medicine; Robert O. Roswell, MD, FACP, FACC, Associate Dean for Diversity, Equity, and Inclusion and Associate Professor of Cardiology and Science Education at the Zucker School of Medicine; Alicia Fernandez, MD, Associate Dean of Population Health and Health Equity at the University of California San Francisco (UCSF) School of Management, Professor of Medicine at UCSF, and internal medicine provider at Zuckerberg San Francisco General Hospital; Jonathan L. Vandergrift, MS, Senior Research Associate at the American Board of Internal Medicine; Marcella Alsan, MD, PhD, Professor at the Harvard Kennedy School.    

Methods and Findings

The methodology includes cross-sectional and time-series comparisons [1] for the adoption of the program and reduction in bias for the residents that are of interest; residents of interest for this study were only those who identified as either Asian, Black, or Latino. The study, firstly, divided participants into U.S.-born and non-U.S.-born residents to minimize the discrimination that non-U.S.-born residents might experience within the context of the study. In addition, residents of interest were only those who identified as either Asian, Black, or Latino. The study compared “knowledge ratings,” which residents were assigned using examinations to test their clinical excellence.

To determine the difference between knowledge ratings, the first certification examination (ABIM), in which residents were rated prior to the Milestone system, was used as the base-case. Then, the Milestone system of resident rating was seen as the treatment variable.

The findings of the study indicate that after the Milestone system was in use, gender bias against female residents was relatively insignificant with how they were rated in their knowledge. However, a note was that there was a deviation from this trend of diminished gender bias for Black female residents, as they experienced significantly more discrimination than both their Black male and other female residents. With the new resident system of rating, there was a significant reduction in bias for residents of color, except for Black residents, where bias persisted. With the small population of residents that are Black (6%) in comparison to the population of Black U.S.-born Black residents in the U.S. (34%), the perpetuation of bias in how residents were rated proved to be especially concerning.

The study’s limitations include possible confounding variables influencing the study that correspond to ratings bias and Milestone bias for how residents are rated on performance. Another limitation of the study, particularly for study participants who identify as mixed race, is participants’ ability to change their race and/or ethnicity, which could alter study results.

Can you put this sentence in more everyday language? The audience for the article is not researchers, but rather practitioners (people who might be making performance evaluations in their jobs). They won’t know what this is.

Conclusions

This study highlights how Black, Asian, and Latino residents were evaluated under new rating system. The Milestone system introduced several measures meant to reduce bias in evaluation systems for residents by resident directors. These measures included several more competency ratings and more comprehensive clinical and diagnostic knowledge questions on clinical comprehension. The Milestone system proved effective in reducing bias in performance rating by resident directors against Asian and Latino residents.
However, for U.S.-born Black residents, the bias remained.