The murder of George Floyd in Minneapolis is just one of the latest disgusting examples of the systemic racial bias that has plagued this country for centuries. While our Black/African-American colleagues have to deal with this in their everyday lives, the recent incidents have led the rest of us to do some soul-searching about the part that we play in perpetuating this bias, either by our actions or by our failure to act. Here at the Center for Scientific Review, our mission includes words such as “fair”, “independent” and “free from inappropriate influences”. And bias, in all its many insidious forms, is the very antithesis of fairness.
As indicated by several published studies over the last decade, and NIH’s own analyses, there remains a serious and disturbing disparity in NIH R01 award rates between White and Black applicants. Isolating the effect of race in the peer review process is a difficult undertaking, since there are many secondary, linked variables (e.g. institutional “prestige”, investigator “pedigree” – who trained with whom, networks, Matthew and “halo” effect, etc.) that are themselves linked to racial disparities in opportunity and access. However, there is absolutely no question that implicit bias exists in all of us as individuals, and the CSR peer review process, with 18,000 unique individuals serving as reviewers, is not immune from these biases.
Since 2019, CSR has initiated a number of efforts to mitigate bias, both at the individual and systemic levels. These are listed below, and as you can see, they are in various stages of development.
1. Development of bias-awareness training modules, with case studies, for reviewers and staff. This is being piloted this round with the NIGMS MIRA reviews, and represents a collaborative effort between CSR, NIGMS, and the NIH Chief Officer of Scientific Workforce Diversity. Based on feedback from the pilot, we plan to refine and roll out the training to all CSR reviewers and staff in 2021. 2. The CSR Advisory Council working group to simplify review criteria made a major recommendation to decouple the science/idea aspects of the review (significance, innovation, approach) from the person-based aspects (investigator, environment). This sets the stage for a de-identified review process for evaluation of scientific merit. 3. Along the same lines, CSR is initiating a multi-stage, partially double-blinded review process for the Common Fund transformative R01 reviews in fall 2020. 4. We continue our ongoing efforts to broaden the pool of reviewers with respect to career-stage, including doubling the number of early-career reviewers serving on our committees, and actively encouraging recruitment of associate and assistant professors. We know that these cohorts are more diverse in both gender and race/ethnicity. 5. While a vast majority of our 18,000 peer reviewers conduct themselves in a highly ethical manner, we continue our critical efforts to identify and take action against those who manipulate the peer review process. Those involved in or unfairly benefiting from the tampering are rarely women and are almost never from under-represented minority groups.While these may be some steps in the right direction, we recognize that there is much more that must be done. And we’re listening to you. We’ve heard from the broader community and from our reviewers that they want to talk to CSR about the funding disparity, the various publications, and what we are doing to tackle this serious problem. So Bruce Reed, CSR Deputy Director, and I are planning to hold a series of forums to have a conversation with you about the data on racial disparities in NIH funding, CSR’s plans, and to hear your thoughts. The first forum will be held on July 8. Our goal is to keep the group small enough to enable a two-way conversation. We’ll schedule additional dates in the near future if needed to accommodate interest.
If you are interested in participating, please send an email to Kristin Kramer.
There is substantial scientific evidence showing that implicit bias influences evaluative processes. Thank you so much for this excellent and much-needed initiative.
I wish NIH would also commit to the effort of increasing the representation of all minorities. Not only African-Americans. From my point of view, Latino/as are systematically discriminated based on the last name only.
We are working on increasing the representation of all minorities under-represented in the biomedical sciences and that includes Hispanics. NOT-OD-20-031 (https://grants.nih.gov/grants/guide/notice-files/NOT-OD-20-031.html) describes in detail how the NIH defines minority groups and where our efforts are focused.
More than racial bias, the enemies to a good peer review systems are reviewers themselves. Often reviewers meet in a bar or cafe and predetermine which applicants they should support and which they should oppose. I have seen it again and again. Conflict of interest is a sham. Reviewers may not have direct conflict with an applicant but they let their good or bad bias against an applicant come through.
1. Internet assisted reviews should be encouraged,
2. Only the assigned reviewers should vote. Often, redundant votes greatly influence the outcome.
3. Backdoor communicators should be severely punished. I have personally seen a reviewer running to the restroom to call an investigator about the fate of an application.
4. Scores are manipulated whether one likes it or not.
We encourage anyone with information about breaches in review integrity to contact the CSR Review Integrity Officer (CSRRIO@mail.nih.gov), your SRO, or the NIH Review Policy Officer (reviewpolicyofficer@mail.nih.gov). CSR and the Office of Extramural Research follow up in each case. It’s of the utmost importance to us to address these breaches. Inappropriate influences are counter to our mission of identifying the highest impact science.
The Ginther paper suggested that the institution from which a grant proposal originates explains a large part of the race gap.
https://pubmed.ncbi.nlm.nih.gov/21852498/
Have you considered not allowing people to review grant proposals from their alma maters. It seems to me that it would be very difficult for someone to be objective about a PI from the school where they got a degree.
One suggestion I made years ago, was to set a random audit of grants done by a diverse group of retired senior investigator that randomly reviewed grant without personal or institutional information, both funded and non funded applications. These reviews would be compared with the original review for consistency. If one finds consistent discrepancies in reviews, then appropriate action taken. It could be that such reviewer in removed for the review process. It is hard to remove ones biases. We all have them, but you can try to make a person aware of them. It is not that you have a basis, it is how you act that matters.
one way to get early career reviewers would be to invite all K award recipients in years 3-5 of their awards to serve as reviewers. excellent experience to review other grants as well as the networking opportunities that come along with the experience.
We also suggest that those who are early in their career but who have not gotten an R01-equivalent consider applying to our Early Career Reviewer Program (https://public.csr.nih.gov/ForReviewers/BecomeAReviewer/ECR). We recently increased the number of ECRs serving at each review meeting and, now, typically have almost 400 early career reviewers serve each review cycle. It’s a great way to get first-hand experience.
There is also a serious and ongoing lack of diversity on review panels. This must also be addressed. This is a systemic issue.
I am delighted to see race bias issue being tackled at NIH in a proactive manner, but much needs to be done to ensure funding is given to under represented PIs in scientific community such as female and people if color. Every major institutional review I have been a part of has been filled with successful experienced PIs who circulate the funding among themselves and I do my part in trying to stop this but being a junior faculty as well as woman of color, my voice is not really heard, but I have continued to raise my voice against discrimination whether it is sexual or racial. Most PIs I know also are good people who want folks to get funded in a fair manner, but out of fear of not being funded when it is their turn for renewal, keep quiet when discrimination occurs.
Are the data on NIH R01 award rates for applicants from different racial/ethnic backgrounds, including Asians and Hispanics besides black and white, publicly available anywhere? If not, can they be made publicly available?
Some of the biases identified above, like institutional “prestige”, investigator “pedigree” – who trained with whom, networks, Matthew and “halo” effect, etc., are certainly very important issues.
The review process should be an objective and purely merit-based system, not to be biased by any other factor.
Well this was long overdue. NIH is wasting the money of black and brown taxpayers with implicit bias training. This is analogous to putting a band-aid over a gunshot wound. Sounds silly to try an undue hundreds of years of American racism through training modules. It points to how unserious NIH is about addressing the “Black/Brown Tax” aka, taxation without representation for black and brown citizens. NIH funded research is no different from other US institutions that have historically underserved black and brown citizens. It’s in the DNA of US institutions. It gets old fast paying into a system that doesn’t fund black and brown scientists and allow us to make scientific discoveries. Racist ideology is taught from birth, willingly or unwillingly, in the United States. If we accept this fact, then the NIH peer-review system needs to be immediately revamped and moved to a double-blind system that will be used for the review of transformative grant applications. Otherwise your cosmetic changes are merely a waste of everyone’s time.
It is highly unwise to turn CSR into a political organization. Highly unwise, indeed. Having been involved in grants for about 27 years, the quality of the review process has taken a marked turn for the worse over the past several years. The number of errors in the reviews has grown tremendously, as has the lack of expertise, and a lack of circumspection by reviewers. It is unfortunate that a once respected organization has taken such a turn for the worse. When an organization places politics ahead of quality, that organization fails.
We all must make sure to be aware of implicit biases. Put science ahead of any gender, racial or institutional biases. Review system is generally fair, however we can strive to make it fairer.
Thank you for addressing the race bias issue. As a NIH reviewer, it matters to me that the review process should provide the greatest degree of fairness we can achieve. I do not have a solution but I am willing to participate in creating a more fair process.
“anonymous”
This is a time to think outside of the box and come up with a radical, but effective change to the review process to overcome potential biases. One approach might be to design a proposal format so that PI and co-researchers are anonymous to the reviewers. The proposal would undergo an initial review, scored first on its merits, then the name of the institution where the research is being conducted is revealed and rated, and finally he PI and co-investigators are revealed along with their biosketches. Each step’s rating could be weighted on importance toward a final score. This permits a fair evaluation of the merit of the proposed study, minimizing the potential for a negative impression based on who is the submitter is. Obviously, there likely are variations to this paradigm that are superior!
Just a thought.
This is regrettable. This reads like a dystopian inversion of Dr. Martin Luther King’s dream of “a nation where they will not be judged by the color of their skin but by the content of their character.” The guilt of “the effect of race in the peer review process,” i.e., the reviewers’ racism, presumed with no proof, just on the basis of a telepathic notion of “implicit bias,” is an Orwellian “thoughtcrime.” Submitting the fate of grant applications to reviewers from the “cohorts” that are picked because they are “more diverse in both gender and race/ethnicity” rather than based on the reviewers’ merits, will result in the deterioration of the review process, imperfect as it is. Review that is blind to the applicant names is impossible for any application that requires preliminary data, even if it is stripped of names and the biosketch is eliminated. In my 20 years of reviewing NIH grants, including six of being a study section member, I have never seen, heard, or could suspect any hints to anybody’s racial or sex-related bias. This also relates to any bias toward the applicants’ institutions, unless the “Environment” section of the review, which has never played any role in my experience either, is so viewed. “Implicit bias,” if any, as any undivulged thought, can be known only to the one who has it. To suspect, let alone accuse, everyone of a vice–it “exists in all of us as individuals”–is not only truly biased–it normalizes that very vice for those who may possess it.
As a reviewer, I have heard many discussions that include evaluating grants based on the journals in which PIs publish rather than on having read the publications. Because women and people from some under-represented groups publish less in so-called “high impact” journals, this superficial evaluation of PIs perpetuates inequity.
All reviewers need implicit bias training.
This is big step in the right direction. Effective de-identification would go a long way toward making reviews fairer.
2. The CSR Advisory Council working group to simplify review criteria made a major recommendation to decouple the science/idea aspects of the review (significance, innovation, approach) from the person-based aspects (investigator, environment). This sets the stage for a de-identified review process for evaluation of scientific merit.
Thank you for taking this major step in the best direction possible.
This is long overdue. I really do hope this endeavor garners the support it needs to become a reality. I am more than willing to be a part of the process toward improving objectivity and removing bias in the review process.
Thank you for making an effort to recognize and correct this bias in our review process.
I recommend decoupling the science/idea aspects of the review (significance, innovation, approach) from the person-based aspects (investigator, environment). It is better to judge the person-based aspects (investigator, environment) by NIH administrators, other than peer reviewers.
This is an excellent initiative. Your comment “since there are many secondary, linked variables (e.g. institutional “prestige”, investigator “pedigree” – who trained with whom, networks, Matthew and “halo” effect, etc.) that are themselves linked to racial disparities in opportunity and access” And the idea of reviewing grants just for scientific content while being blinded to the identity of the PI and team is a good strategy to reduce/minimize bias in the review process.
“decouple the science/idea aspects of the review (significance, innovation, approach) from the person-based aspects (investigator, environment). This sets the stage for a de-identified review process for evaluation of scientific merit.” is a laudable initiative.
I have been on many review panels in which minority groups are well represented. Furthermore we do not know what race the PIs are on the proposal unless they have a foreign name. I am also an immigrant and would never have bias based on sex, race or religion.
Great, as you stated you are going in the right direction.