National Research Service Award (NRSA) Fellowship (F) awards are intended to support training that will enhance pre- and post-doctoral trainees’ potential to develop into productive, independent research scientists. In 2021, CSR handled the review of more than 5500 of the approximately 6800 NRSA F applications received by NIH. We recently convened a CSR Advisory Council working group, charged with evaluating the fellowship review process and making recommendations to make it as effective and fair as possible for all.
The working group has noted multiple concerns, many of which center around the challenges of discerning the potential of the applicant and the value of the training planned, as opposed to the general reputation of the school and sponsor. There are concerns that applicants from smaller and less-resourced schools sometimes face a higher bar, that grades can unfairly haunt strong applicants, and that junior faculty are hesitant to sponsor applications, feeling their chances are slim.
Now the working group would like to hear your thoughts about the fellowship review process. What are its strengths and weaknesses? How could it be improved? In answering, think about the characteristics of strong applicants, sponsors, and training programs and the challenge of identifying the applications that have the greatest potential to develop independent, productive research scientists.
It may be useful to refresh your memory on the criteria NIH uses now. The five main review criteria are: Fellowship Applicant; Sponsor(s), Collaborators, and Consultants; Research Training Plan; Training Potential; and Institutional Environment & Commitment to Training. (PA-21-051 is a typical funding announcement and details review information in section V.) However, you need not be bound to these criteria; creativity is welcome.
Share your ideas via the comments box, or by emailing feedback@csr.nih.gov. We cannot respond individually to most comments, but we promise that if received by January 24, 2022, they’ll be read and that the information you provide will help the working group and CSR strengthen the review of NRSA fellowship applications
Authors are:
Bruce Reed |
Lystranne Maynard Smith |
Cibu Thomas |
The biggest flaw I see with the application process is the heavy weight applied to and penalty for the mentor. If NIH is worried that mentor may not be able to fully support the mentee’s project and career development, then perhaps they should assign a mentor who does have a track record to help keep the team accountable and to bounce ideas off of/ask questions. I would envision this happening post-award so that the onus is not on the mentee to find an additional seasoned mentor. The goal of the F as I understand it is to provide avenues for training applicants with a strong potential research career who may not have as direct a path to doing so without the funding. Instead, NIH tends to keep the 10% of PIs with prior funding going by evaluating the mentor so heavily in the F review process.
I would also recommend getting rid of the grade criteria. Seems silly that if an applicant has gotten into graduate school that their undergraduate grades would factor in, and at the postdoc level, grades should not matter at all. They have a PhD. Keeping in mind my latter point about diversifying NIH PIs, evaluating grades seems like a great way to maintain the status quo.
Eliminate grades in assessment. They are not predictive of research success and have already been used to admit the candidate to graduate school.
1. Grades should not be included. For all the aforementioned reasons by my colleagues.
2. PIs should be required to attach a document where they outline their lab philosophy and policies. We complain all the time about PIs/mentors who abuse trainees, and have unreasonableand unethical standards. Who require crazy work hours without adequate time off. PIs should be required to put these policies in writing so they are not continually rewarded for their bad behavior with federal funds. Science is rarely a 9-5 job, sure, but people need adequate time for their personal lives and to rest. While I recognize this will not fix the issue, it allows for the greater discourse at the level of the NIH on this subject, and some self-reflection and pressure applied by peers.
3. The implied requirement for trainees to pursue an academic trajectory is harmful. There are simply not enough positions or funding in academia for this. Trainees seeking to bring their scientific experiences to industry, government, and business should be viewed as just as meritorious as staying in the academic setting. This view needs to come from the top!
4. We need to adjust our productivity standards for the pandemic. We need better mechanisms to support people with families, especially in the pandemic. Many trainees have young and unvaccinated children. Even if the trainee doesn’t have a young family, the PI may have been unable to help due to theirs, the pandemic caused many of us to have to decrease our on hand resources (mouse colonies for example), The playing field is not even here in terms of productivity. Supply chains have been devastated. We need to be able to discuss this and have it taken into account.
1. Eliminate grades as a criterion. The applicant got into grad school or got a Ph.D. There is also bias in this measure. Some students have to work in college, while others do not, etc.
2. The stated or “silent” criterion that expects prior posters and publications from F30 and F31 applicants is ludicrous. This is one way in which disparities are widened by fellowships. Applicants that were able to devote time in labs (probably for free) as undergraduates benefit at the expense of those that did not have that luxury.
3. Decide what these fellowships are intended to achieve. They clearly are NOT about providing support for those with NEED. The financial benefit of NRSA awards currently goes to the mentor. Supposedly, this is about training a scientist, but applications are picked apart on research minutia. Whatever NIH/CSR wants these grants to achieve, clarify it to reviewers and HOLD THEM TO IT during review.
4. Fellowship applications provide funds to support a trainee, NOT A PROJECT. Reviewers, either overtly or silently, expect to see a “new” research direction in F applications. In the event that an applicant writes an application that is a significant departure from the mentor’s work, funds need to be provided to perform that work beyond the paltry institutional allowance. If that is NOT the goal of fellowships, then applications should be allowed to remain closer to the research focus of the mentor, this should be communicated to the reviewers and again, they should be held to this standard during review.
5. Award proportions by race and gender should reflect U.S. race and gender proportions.
6. Diversify fellowship reviewers and recruit YOUNGER reviewers.
7. Eliminate resubmissions.
Many aspects of fellowship review are Goldilocks scenarios where the porridge has to be just right, but applicants with mentors that have reviewed fellowships benefit because they know the unspoken correct temperature, creating a scenario where the rich get richer.
As a former F program awardee, I believe that F programs are great mechanisms for trainees but with several pitfalls.
1. As lots of others already stated, removing grades should be done for all PhD students as well as postdoctoral fellows. Their ideas and experimental designs are the focus during the review process, not their grades.
2. Reviews on PIs and Mentors should be considered more broadly. NIH should allow PIs/mentors to list up their previous trainees who are in non-academia positions. Reviewers should not judge on mentor’s mentoring skill by the number of lab alumni with tenure-track jobs. This current review system will just benefit too many *big names* with lots of previous trainees who already landed on tenure-track.
1. Forget about grades.
2. Review F grants in expanded regular study sections that have the relevant expertise.
3. Agree re PIs can’t be criticized for either being too junior or too senior.
4. Diversity candidates should always be discussed and get full reviews.
As a faculty member having seen both the submission and the review side of F and T applications, I have been very frustrated by the massive amount of overlapping details requested in these applications. It breeds inefficiency in the process and contributes to a failure of reviewers to focus on what is most important. The applications need to be streamlined and sections clarified to be more distinct from one another. Reviewers glaze over when writing is inefficient and unfortunately the structure of these applications could be much better.
I also agree with the idea to remove undergraduate grades from the process. The person already made it into graduate school; another review of their undergraduate credentials is unnecessary.
Junior faculty can be fantastic mentors and this should be recognized; if other structures are in place to provide access to high quality training (courses, symposia) and secondary mentors and consultants who can provide secondary support, I think that should be sufficient and will also diversify the supported trainee population.
I have both reviewed/sat on NIH fellowship study sections and had my own trainees submit applications. The big issue is that there are many worth applicants but little funding. Competition is tough. However, I’ve come across a number of issues that cloud the review process and can even cause harm to the trainee.
1. Grades. These should not be review criteria, especially for postdocs. If someone has been accepted into a PhD program or graduated from a PhD program, there are plenty other more direct ways of assessing potential that are more relevant to research in science.
2. Review of the mentor/PI. There seems to be a sweet spot where PIs who are too junior get critiqued for not having enough experience vs PIs who have too many trainees. Either of which are not the applicant’s fault. Too much weight is put on these criteria. The mentor and mentorship statement need to be taken at face value.
3. Cutting out harsh criticism. Reviewers need to be reminded that these are trainees who may be receiving their first critical reviews. Let’s be kinder and more constructive. There’s no need for harsh reviews.
4. These days, trainees go on to a diversity of jobs and positions. The NIH is tax payer funded and if our training lands people in the biomedical workforce (broadly defined), this should absolutely include industry and positions outside of academic labs.
Coming from the viewpoint of a postdoctoral trainee:
1. Grades. I was personally dinged by one of my reviewers for not having enough A’s on my undergraduate transcript. I took some of these classes over a decade ago and these grades by no means reflect my abilities as an investigator.
2. Review of the mentor/PI. Too much weight is put on this criteria. A junior investigator is just as capable, if not more capable of mentoring a postdoc when they themselves were a postdoc not too too long ago. As long as there is a solid training plan and strong commitment to the mentor/mentee relationship, the applicant should be given an equal chance at funding. Furthermore, an applicant should not be judged based on the previous funding record of their PI. That creates a negative cycle of the lab never getting good funding.
3. Needing a highly specialized mentoring committee. It is nearly impossible to establish a mentoring committee of 5 PI’s whom of which are all specialists in your field of study and all localized to your research University. I got critiqued for having a committee member from out of state and for having committee members without enough specialization in my specific field. Research is multifaceted and you need more than just 5 PI’s studying the same topic as you on your committee.
Stop dinging applicants from new labs for lacking an “experienced” mentor- new investigators are some of the best mentors around since they have just been on the other side, and they are heavily invested in their student/postdoc’s success!
1. Remove grades. Especially undergrad grades. No reason they are relevant.
2. An applicant positionality statement to factor into candidate section — beyond academic credentials, how their identities and experiences inform their approach to their science and how they anticipate mitigating biases.
Consider instituting a no bleed through policy (similar to what is used by DoD) with very clear criteria for each category (research plan, training plan, and training potential). This would help the trainee identify the specific areas that need address.
Remove grades from the application. Doing well in class and succeeding in research require different skill sets and having grades as part of the review can inadvertently lead to inequalities.
Also, if there are specific phrases that are being looked for by reviewers, be sure to state that in the instructions. If NIH doesn’t want to dictate specific language, the SROs need to be able to jump in on conversations where the phrase ” they didn’t specify say…” comes up to remind everyone that specific phrases are not asked for and should not be considered in scoring.
1. I strongly dislike the idea that F grants are often judged by the seniority of the faculty advisor/PI. It puts junior PIs at a big disadvantage. I accept the risk that not all young PIs may be/will become good mentors, but I think the same is true of senior PIs.
2. I don’t think there is adequate “coverage” of some research areas in existing F study sections. My lab studies the harmful effects air pollutants on the lung, but we have not been able to find an appropriate study section for F31 and F32 applications. We try the closest/most reasonable but never gain traction in them.
The number one complaint I had in my own review process was that my PI was knocked for not having “many students in academic positions with extramural funding”. My PI has two previous trainees with R01 funded labs at top research institutes and every previous trainee has moved into a post-doctoral position. So firstly, that’s just an incorrect statement. Secondly, the potential success of a current applicant should not be defined by the career choices of previous trainees. Why should a current applicant suffer because a previous trainee decided that they were best suited for industry? If the current applicant is set up for a successful academic career and says they want to pursue one, then that should be the only consideration.
Also, absolutely agree that grades should be eliminated. As long as a student is in good academic standing, then grades should not be considered against an applicant, only a positive for if they had exceptional academic performance that can be illustrated through magna or summa cum laude status elsewhere in their application.
Drop the F32 entirely and use the resources to fund more K99/R00.
Make all F applications one-and-done. One proposal per candidate, no revising. The revise/resubmit cycle makes no sense for training when 18 months is such a large fraction.
Clarify in your own thinking and for reviewers whether your F programs are about 1) quality of the actual science training, 2) trainee / program need (for $) or 3) reaching a particular high value career outcome. Much of the frustration comes from it being 3 but applicants thinking it is 1 or 2.
Whether you admit it or not, F awards have a significant component of awarding free labor to the PI. In the context of various funding biases which impact the PI class, this needs to be included in your thinking. Perhaps any PI should only be able to sponsor a limited number (2-5) of F31 *lifetime*.
Above all else be transparent about your funding outcomes and the resulting impact on the science workforce *in the context of whatever claims NIH is making abut what the F programs are supposed to be accomplishing*. Make sure to inform reviewers about these outcomes.
Remove grades – all of these applicants were already admitted to competitive programs
More clarity on the differences between Research training plan vs research plan vs training potential for reviewers and applicatants
Training potential can seem like a combined measure of other criteria, maybe more specific separation of this criteria for what you hope people are judging here
Discussion can be derailed by focus on the research plan itself, which is often important for feasibility and training potential, but maybe including this as a specific criteria would help separate the other criteria from this more clearly, especially the research training plan criteria
Re: junior faculty mentors – track record is important in the criteria so junior faculty get penalized for fewer publications, fewer trainees mentor, so we can’t criticize for having a junior mentor but these applications are indirectly criticized for that
My most recent participation in the review process was very productive. I think the SRO team has made substantial improvements on the review process. I thought that the SRO provided substantial guidance on how to think about the applications, how to distinctly evaluate F30, F31 and F32 applications, and how to be sure to avoid implicit bias or reputation bias. I was very impressed and I thought the review committee was fair, thorough and thoughtful.
Eliminate grades as a scoring criteria. Even better, eliminate them from the application entirely!
I think F30 and F31 applications should be reviewed separately from each other. Generally, dual degree applicants by default have a more competitive application than PhD students (due to the highly competitive nature of dual degree programs). Dual degree students also have more opportunities available to them following their graduation. I think, to encourage and promote students whose sole career focus is on research, we need to set aside opportunities specifically for them and not shared with dual degree students. By forcing F31 applicants to compete with F30 applicants, a smaller pool of funding ends up being awarded to PhD students. Further, these fellowships appear to do more for the careers of early PhD students than they do for early MD/PhD students (I would suggest the F Fellowship programs review whether F30 vs F31 awardees remain in academic research after 10 years). If we don’t start investing in our PhD students early on, they will leave academia and bring their potential elsewhere.