Review Matters

Update on Simplifying Review Criteria: A Request for Information

Authors

December 8, 2022

Cross-posted on Open Mike.

NIH has issued a request for information (RFI) seeking feedback on revising and simplifying the peer review framework for research project grant applications. The goal of this effort is to facilitate the mission of scientific peer review – identification of the strongest, highest-impact research. The proposed changes will allow peer reviewers to focus on scientific merit by evaluating 1) the scientific impact, research rigor, and feasibility of the proposed research without the distraction of administrative questions and 2) whether or not appropriate expertise and resources are available to conduct the research, thus mitigating the undue influence of the reputation of the institution or investigator.


Currently, applications for research project grants (RPGs, such as R01s, R03s, R15s, R21s, R34s) are evaluated based on five scored criteria: Significance, Investigators, Innovation, Approach, and Environment (derived from NIH peer review regulations 42 C.F.R. Part 52h.8; see Definitions of Criteria and Considerations for Research Project Grant Critiques for more detail) and a number of additional review criteria such as Human Subject Protections.

NIH gathered input from the community to identify potential revisions to the review framework. Given longstanding and often-heard concerns from diverse groups, CSR decided to form two working groups to the CSR Advisory Council—one on non-clinical trials and one on clinical trials. To inform these groups, CSR published a Review Matters blog, which was cross-posted on the Office of Extramural Research blog, Open Mike. The blog received more than 9,000 views by unique individuals and over 400 comments. Interim recommendations were presented to the CSR Advisory Council in a public forum (March 2020 video, slides; March 2021 video, slides). Final recommendations from the CSRAC (report) were considered by the major extramural committees of the NIH that included leadership from across NIH institutes and centers. Additional background information can be found here. This process produced many modifications and the final proposal presented below. Discussions are underway to incorporate consideration of a Plan for Enhancing Diverse Perspectives (PEDP) and rigorous review of clinical trials RPGs (~10% of RPGs are clinical trials) within the proposed framework.

Simplified Review Criteria

NIH proposes to reorganize the five review criteria into three factors, with Factors 1 and 2 receiving a numerical score. Reviewers will be instructed to consider all three factors (Factors 1, 2 and 3) in arriving at their Overall Impact Score (scored 1-9), reflecting the overall scientific and technical merit of the application.

 

  • Factor 1: Importance of the Research (Significance, Innovation), numerical score (1-9)
  • Factor 2: Rigor and Feasibility (Approach), numerical score (1-9)
  • Factor 3: Expertise and Resources (Investigator, Environment), assessed and considered in the Overall Impact Score, but not individually scored

Within Factor 3 (Expertise and Resources), Investigator and Environment will be assessed in the context of the research proposed. Investigator(s) will be rated as “fully capable” or “additional expertise/capability needed”. Environment will be rated as “appropriate” or “additional resources needed.” If a need for additional expertise or resources is identified, written justification must be provided. Detailed descriptions of the three factors can be found here.

The following “Additional Review Criteria” will remain largely unchanged, not scored individually but continue to affect the Overall Impact Score. A drop-down option of “Appropriate or Concerns” will be provided, with a justification narrative required for concerns.

 

  • Human Subject Protections
  • Inclusion of Women, Minorities, and Individuals Across the Lifespan
  • Vertebrate Animal Protections
  • Biohazards
  • Resubmission
  • Renewal – Evaluation of productivity during the previous project period will remain unchanged and will continue to affect the Overall Impact Score
  • Revisions

 

Other minor changes to the peer review framework are proposed that will reduce time and effort required by reviewers. “Additional Review Considerations” are items that peer review is currently tasked with evaluating, but which are not scored, and which do not affect the Overall Impact Score. NIH proposes to move select Additional Review Considerations out of the initial peer review process. Under proposed framework, the remaining Additional Review Considerations are:

 

   Authentication of Key Biological and/or Chemical Resources:

     – Drop-down option with a choice of “Appropriate” with no comments required, or as “Concerns,” which must be briefly described.

   Budget and Period of Support:

     – Rated as “Appropriate,” “Excessive,” or “Inadequate”; the latter two ratings require a brief account of concerns.

Budget and Review Criteria

Through the RFI, NIH continues to seek further public input on the proposed changes before moving forward with implementation. The RFI will be open for a 90-day period, until March 10, 2023. 

 

93 Comments on "Update on Simplifying Review Criteria: A Request for Information"

  1. Anonymous says:

    There must be a way to do an initial blinded screening based on Factors 1 and 2 – e.g. white paper format. Those that passed would then undergo a more detailed unblinded review. I think this will increase the chance of new and mid-level investigators to get considered for funding. This will also decrease the number of full reviews (which take a lot of time- regardless of the quality of the application, before the meeting) and allow more time for others to weigh in on applications selected to be discussed.

  2. Walid Fakhouri says:

    I like the newly proposed changes to the NIH review criteria. However, I suggest that Factors 1 and 2 should be a blind review. After the scores are registered, Factor 3 can be reviewed and scored without the option of changing the scores of initial scores.

  3. Jan Lammerding says:

    I agree with the proposed changes, which will simplify the review process, remove some potential bias, and overall reflect what many previous studies have shown, namely that the ‘approach’ and ‘significance’ are the major drivers of the overall impact score. Focusing on whether the institutional resources are sufficient for the proposed work is another good idea.
    i also agree with many of the other commenters that having a ‘blinded’ process would be difficult to achieve, and i don’t think it would be necessarily desirable, as it would make writing the proposal and pointing to prior relevant achievements by the PI n the proposed area much more difficult. I did not see plans for a blinded review process in the published proposal, so this may just be a hypothetical discussion anyway.

  4. anonymous says:

    I like the changes, but think the investigator still should be scored. Their expertise and ability are critical to the success of the project.

  5. David Ellison says:

    I agree with others that these seem like good changes in general. In terms of merging innovation and significance, this is good. It has been documented that there has been an escalation over the recent past in the use of superlative descriptors to characterize (?sell) one’s own work (to say how this will the paradigm). I, too, have been guilty of this, as we are frequently told that SROs instruct study sections that they must favor only paradigm shifting proposals. Additionally, and as noted by others, the innovation category tends to favor technically innovative grants, and not those that are just creative and yet simple. Merging the two takes some emphasis off innovation, which is often either overemphasized or phony.

    I worry a bit about downgrading the investigator score. There is obviously a tension between distributing funding in a geographically equitable manner and fostering a network of solid research institutions versus giving funding only to the scientists most likely to succeed. For experienced scientists, past record is often the best predictor of future performance, as recognized by HHMI. I am often blown away by the resources that some other institutions have, but I also recognize that those resources may in fact facilitate greater accomplishment. In the end, I still think we need to be able to give some credit to someone who has demonstrated that they are highly productive. The most important thing is to allow a path for more junior investigators, who do not have the same track record, to thrive. On balance, I think the pluses outweigh the minuses here. The proof of the pudding, however, will be in the eating.

  6. Stephen DiGiuseppe says:

    Ideally, Factor 1 and Factor 2 should be blind. Factor 3 should be “sufficient” or “not sufficient”–like a pass/fail. Either you have the resources & expertise or you don’t.

  7. Anonymous says:

    What’s missing is the need to eliminate the additional sections required for clinical trials, or at least greatly limit which studies are required to provide it.
    Low risk studies are required to provide this information just because randomization is involved, which is not necessary and adds undue burdens to the investigator.

  8. Debra A. Murphy, Ph.D. says:

    As someone who’s had multiple R01s, and been on U-01s and funded national networks, I have a great deal of past experience with being reviewed. I also have a long past experience serving on study review sections, and using NIH criteria for University grant reviews. Overall, I like the three factors being proposed, and I especially liked that for the third that there are categories to be rated (e.g., “fully capable” or “additional expertise/capability needed” and that this must be justified in the comments). I also liked the reduction of the additional review criteria. However, I don’t see partially blind review as possible? If you have true experts in the research area on the review committees, which is needed, then you are going to have many of the reviewers aware of who the P.I. is on a proposal (e.g., if a P.I. is building on previous work; if the P.I. is doing a full scale trial of a pilot that they did; etc., etc., etc.). There are numerous ways in which a proposal would be recognizable just by reading a proposal. Also, a reviewer needs to be able to refer to references cited, and it would be difficult/impossible to do if a P.I. is referencing their own past work, which they should be able to do. It almost seems problematic to even try this, as some reviewers (i.e., main expertise statistics or some unique aspect of the proposal) may be less familiar with a specific general field and some of the major researchers in it and not recognize the P.I. of a proposal, but many of the other reviewers do know whose proposal it is–then you have a bit of a confound within the reviewers. I don’t see that as being very feasible at all. But overall this is a very good step forward in the long history of NIH consistently trying to improve the review process.

  9. anonymous says:

    It seems large institutions will still be favored more so over smaller ones under the resource category. Many smaller institutions do not have the established research infrastructure which has consistently put them at a disadvantage and consistently contributes to the disparities in research funding. I think the Resource category should clearly be defined as the resources necessary to complete the research aims, ie confirmation that a lab is accessible to the researcher for this project versus the institution having over 3,000 square feet of space – which is not necessarily needed to complete the research aims.

  10. Valerie Andrea Paz-Soldan says:

    I strongly agree with suggested changes – both as a reviewer and an applicant. The three factors proposed are the items that are truly score-driving. As a reviewer, I want to see if 1) the proposed research is impactful — covered by “significance” and “innovation” (even if sometimes it is not majorily innovative, I would not score DOWN an overall project if it could be impactful!), 2) the approach is rigorous, well-described, and feasible, and 3) the team has the capacity (investigators) and resources (environment) to carry out the work – the research, the administration of the grant, etc. I have often had difficulty scoring “environment” – either it is a strong institution (so it gets a good score) or it is not (in which sometimes it makes sense to build capacity). In any case, I feel this simplification in the scoring system MAKES SENSE!

  11. Meena Kumari says:

    I welcome and support the change and hopefully removes the bias and prejudice of “being a well funded investigator who can continue to engage in state-of-the-art studies or the institute, or friend whom I have known deserves the grant to be funded etc”. As many have mentioned in their comments (above), the blind review process gives an opportunity to new investigators from states not so well funded through NIH or other agencies a chance to show their projects can advance science. A good example is the recent “the Earthshot Prize”.

  12. anonymous says:

    I support these changes but DO NOT support “blinded review” for all the reasons stated. Thank you!!

  13. Rong Huang says:

    I supported the principle of proposed changes. But it would be better to weigh significance and innovation separately. So change to 4 factors (Significance, Innovation, Rigor and Feasibility, Expertise and Resources)

  14. Jamie Privratsky says:

    I support these changes. Seems like criteria such as significance/ innovation and investigator/institution have common overlaps and can often be evaluated together. However, while I think this will be step in right direction to help limit bias towards prominent investigators and institutions, I do not think having Factor 3 as unscored criteria will be enough. There is no way to completely eliminate this, though I do still support the new criteria.

  15. James R Bain says:

    I strongly support streamlining the five current main review criteria into three, and the present five additional criteria into two. This will bring clarity to applicants and reduce the workload on both applicants and reviewers. Blinding reviewers to the applicants’ identities and institutions would be a helpful next step, and would do much to reduce the “rich-getting-richer” / “good ole girls and good ole boys” / “big science” elitism that plagues the present review system, wherein pedigree and connections often outweigh substance and creativity.

  16. Supriya says:

    Thank you for the simplification and reprioritization of Significance and Approach. Following CIHR and other agency reviews I’ve done, could the grant application itself be reordered so that resources, environment, budget are at the end and not shading reviewers’ perspectives before they get to the actual science? In this way signifying science comes first.

  17. Lorraine Robbins says:

    I support the changes. If the importance of the research is comprehensive and clear, both significance and innovation should be evident in this section. I agree with not individually scoring investigators and environment, but then considering this information in the overall impact score. Reducing the number of additional review considerations will also decrease the time commitment for reviewers. The proposed changes may help to reduce repetition of information from the individual section scoring areas to the overall impact section.

  18. Valeriya Gritsenko says:

    I support the proposed changes. The shift away from “innovation” will help reduce the tendency to create hype around a proposed research direction. The shift away from Investigator and Environment assessments will help reduce bias toward already funded investigators in large well-known institutions.

  19. Anonymous says:

    As a reviewer for 5 years, I believe that the proposed changes are a step in the right direction, refocusing the review on whether the science SHOULD be done and whether it CAN BE DONE WELL, while eliminating burdensome and unhelpful sections of review that are better handled administratively. I particularly believe that the de-emphasis of innovation (which typically focuses on technical innovation) will improve evaluation of the overall science, and de-emphasis of review of minor technical details will, if implemented correctly, reduce the “downward pull” on scores for approach. The above comments reference blinded reviews, but I did not see this in the proposed recommendations. I do not believe this is a good idea for several reasons: 1) Blinding of the applicant and institution is not likely feasible for many of the reasons others have described (e.g., self-referencing of prior work), 2) Blinding would eliminate the potential to review investigators’ biosketches and budget justifications, which are critically important in review, 3) Making review blinded would make determination of conflicts of interest harder to identify and avoid, 4) Evaluation of “Investigator and Environment” would be nearly impossible.

  20. Robert M Jacobson says:

    I strongly support the changes. I have been submitting applications and participating in reviews since the early 1990’s. The five categories are problematic because the significance and approach so clearly outweighed reviewers’ attention and concern with the others and rightfully so. This three-pronged approach has more merit and will make the evaluations, discussion, and scoring more meaningful.

  21. Mana M. Parast says:

    I appreciate the changes for two reasons:
    1) It de-emphasizes “innovation,” which of late has come to mean “technological innovation” (with less emphasis on “conceptual innovation”)–and thus has meant that reviewers value access to the latest in technology, thus introducing greater inequity between scientists at different institutions and even between scientists at the same institution (e.g. where for example, core facilities are not well-funded/under-resourced, thus again limiting access to only those with significant funding, leading to more people with grants getting more grants). Within the current plan, NIH should go a step further and emphasize conceptual (rather than technological) innovation.

    2) I think commenting (rather than scoring) for investigator/institution is also beneficial; however, I do believe knowledge and expertise of the investigator matter. I think further discussion is needed among the scientific community to identify a mechanism that can confirm this knowledge/expertise, while eliminating bias toward the “well-known” (possibly already well-funded?) researchers.

  22. Sharlene Day says:

    I am a current study section chair. I agree with many others that these changes are very welcome and will focus the reviews, discussion and scoring on the importance of the research question and the likelihood of the approach to be successful at answering the question. It will also streamline the writing of the reviews and the discussion. While I see the rationale of blinding the reviewers to the investigators and institution, as at least one person pointed out, most of the time the reviewers will figure out who it is. And from the standpoint of the applicant, how would you write a grant without referencing your own work that supports the feasibility of the project? I just don’t think it will work. It is the job of the SRO and chair to keep reviewers honest and make sure they are judging the science fairly across the board and do their best to eliminate bias in the review process.

  23. Cynthia Wolberger says:

    I think simplifying the review criteria is a positive move, as is moving some of the Additional Considerations out of study section review. Regarding concerns raised by some that there is no place to score investigator track record, I would think that is something that could potentially be relevant to feasibility under Factor 2.
    One aspect that is not addressed is the overall summary. I think it would be helpful if reviewers were also asked to list the score-driving factors for the overall score. While reviewers are supposed to do this in the summary, too many do an inadequate job. This is particularly a problem for grants that are not discussed. PIs then resubmit grants that address specific points, but still do not get funded because the major score-driving issue is inadequately addressed. Anything that can be done to convey more clearly the major issues that drove the overall score would benefit both PIs and reviewers.

  24. H Patisaul says:

    As a long time reviewer I strongly support these changes as well as the suggestion that the institution be blinded as that is one of the biggest sources of reviewer bias. Innovation is another problematic area because someone can certainly perform transformative research without necessarily being “innovative.” Reviewers always struggle to define what “innovation” actually means. While I laud the changes, institutional bias is going to remain a problem – even with this proposed revision. The only way to get around that, if NIH is truly serious about it and diversifying its portfolio, is to review the proposals from less resourced schools separately or creating alternative funding mechanims.. They handle some schools OK with the R15 program but proposals from R1 schools without med schools (universities with less than $50-75M in NIH funding) keep getting squeezed and always suffer in review. In my experience, that’s getting worse. Rich schools keep getting richer and these modifications, while welcome and helpful, likely won’t change that. Simplification of the review process was long overdue, and the proposed changes are welcome.

  25. Trisha Wittkopp says:

    Yes, Please!!! These would be very welcome changes, reducing reviewer burden without significantly changing the effectiveness of the review process.

  26. Claire Walczak says:

    Excellent change suggestion- will make it simpler for the reviewers and the applicants as well- with much clearer messaging and feedback. It will still be important to make it clear to applicants that the overall impact score is not a simple mathematical average of the individual scores.

  27. Eric Wickstrom says:

    Speaking as a 50-year applicant and reviewer, I approve the simplification. Furthermore, I support blinding of the name and institution. Focusing on Significance, Innovation, and Approach will increase funding of new ideas by less-well-connected investigators.

  28. Will Hancock says:

    Currently on study section and have reviewed for almost 20 years. I am very much in favor of the change. The current Investigator and Environment categories are least informative and least score driving, followed by the Innovation score. By distilling to two sub-scores and additional qualitative considerations to reach a final impact score, more thought will go into those scores. As I see it, strength/weakness comments will remain, and so the score driving points a reviewer writes down will still be there. As it stands now significance and approach are the key score drivers of the five, and this change just codifies, clarifies and emphasizes this (which is appropriate). Strongly in favor!

  29. Anonymous says:

    This is a good change. I have seen grant scores for “environment” ranging from 1 to 8 from the same institution and department. This indicates that the environment criteria is a source of bias, where prejudices impact funding outcomes. The proposed approach would be more equitable.

  30. Emilio Salinas says:

    Having participated in study sections for many years now, I strongly support the proposed changes. In the grand scheme of things, they will have a modest impact, but I think they will facilitate the evaluation process for reviewers and will help focus the discussions on the points that matter the most.

  31. Bill Hlavacek says:

    In my experience, significance and approach are the criteria that most commonly determine the overall impact score. A review process that focuses on these criteria seems like an improvement.

  32. Jimmy Efird says:

    Great idea…nice simply structure.

  33. anonymous says:

    The change highlights the most important factors during the evaluation of an application. Such change will make the reviewers focus on the quality of science, feasibility and rigor, as they are generally the score drivers. It reduces the workload of the reviewers to review and comment on the administrative sides of the applications. Definitely a positive change in the right direction.

  34. Zsuzsa Kaldy says:

    These will be very helpful changes and reduce reviewers’ negative biases toward institutions with less perceived prestige.

  35. Alexey Veraksa says:

    As a reviewer and a current study section member, I welcome these changes because they would simplify and streamline the process, even if not radically. The review process is still expected to work well (and hopefully better) since the most important criteria are retained and are now better emphasized.

  36. Anonymous says:

    I support the proposed change and criteria. I think significance and innovation should be the most important criteria. I also encourage anonymous (blind investigator) review until the final decision is made.

  37. Hakim Djaballah says:

    The proposed change makes a lot of sense since science keeps evolving & this will simply the review process a great deal, & hopefully help spread the final scores instead of having them clustered.

  38. Jenn says:

    I think the proposed changes are sound. The relevance of the PI expertise and track-record is captured in that the overall impact score can be impacted. In the current method, there is no reasonable way to accurately measure productivity of a PI anyway. If someone publishes 3-5 papers a year with 3 grad students, that’s probably more impressive than 10 a year from a lab of 25. Plus, with the wide variation in impact from COVID, productivity comparisons have become a new way to discriminate against female faculty.

  39. anonymous says:

    Downgrading importance of the investigator’s track record and her/his institution quality is a bad idea if the purpose of the NIH is to support the best science and provide cures for patients. In all walks of life, student, professional, trade, business or other, there is no better predictor of future success than past performance. The other changes will help simplify the review process and will be helpful.

  40. Varykina Thackray says:

    Given that Significance and Approach have been shown to be the score drivers for most grants, these proposed changes makes sense to simplify the review process and keep the focus on the proposed science instead of the “reputation” of the investigator or institution. Indeed, it would be even better if the grants were blinded and the investigator and institution were reviewed post award.

  41. Vetta Sanders Thompson says:

    I think that this shift is critical in removing bias from reviews. A researcher can be an expert and capable, and have a lab that can complete the work without being at a prestigious institution.

  42. Stephan Taylor says:

    I very much agree with the proposed changes. The first two factors are the score driving criteria, and it makes sense to me to consider, but not score, investigators & environment. Those usually get scored highly, unless there are problems, so the the range for these criteria is poor and the information in a numerical score of little value. I always find innovation hard to score separately, so it makes sense to fold it into the first factor.

  43. Jake Abbott says:

    I like everything about these proposed changes. They are very much in line with what I would have proposed. Separating Significance from Innovation always seemed artificial. The same goes for Personnel and Facilities.

  44. anonymous says:

    These changes are reasonable and should have a positive impact on the review process. I agree that the research proposal reviews should also be blinded to investigator and institution. This additional change would promote equity in funding, in particular for larger grants where it has been shown that these features impact score in a biased manner. Further, the resource sharing plan should be maintained in the application, as it is critical to share data in this time of limited resources.

  45. Judy Andrews says:

    I have served as a post hoc reviewer for over 20 years; been occasionally chaired a review, and have served on a study section. I like this new system very much. i particularly like combining significance with innovation to evaluate the importance of the research. It’s similar to how an introduction to a paper is written. I also like not scoring investigators and environment but using these to guide the overall impact. In my experience, if these are given low scores, it does affect the overall impact, but high scores do not appear to increase the impact. I think investigators and reviewers need additional training on the meaning of rigor and feasibility, and how they affect the approach.

  46. Paula Watnick says:

    I think this is a great idea. I like the grouping of the previous 5 criteria into 3. I am a bit concerned that rating system for the investigator as “fully capable” or “additional capability needed” may play into unconscious bias as some sectors are viewed as inherently less capable than others. Requiring that the reviewers list specific deficiencies might guard against this.

  47. Edward Owusu-Ansah says:

    This is definitely a step in the right direction as ‘Approach’ which will become ‘Rigor and Feasibility’ is the score-driving feature in most grants anyway. The new model will simplify the grant review process, as it requires commenting on 3 instead of 5 measures. I would suggest you change factor 1 to ‘Significance and Innovation’; it sounds less blunt than ‘Importance of the Research’.

  48. Marc Lenburg says:

    I am enthusiastic about recharacterizing “innovation” as an aspect of “importance” since may reviewers get preoccupied by whether a method is innovative rather than if it is appropriate (or if the research question itself is innovative). The focus on innovative approaches drives many investigators to reflexively propose use of new technology that yields low impact data. Having investigators and environment unscored but score driving is OK. Might be useful to stress to reviewers the potential connections between the strengths and weaknesses of the research team and their resources and the feasibility criterion.

  49. Anonymous says:

    I have submitted plenty of grants, gotten some funded, and served on a number of study sections. I like these new consolidated criteria, particularly the combination of significance and innovation. I have often felt that they are not neatly separable, particularly, for example, when a specific technique is not innovative, but its application is. That should be an adequate justification for the importance of the work, but some reviewers don’t always see it that way. Putting significance and innovation together will reduce the impact of such reductive thinking. All of that said, the overall score is still the overall score; it reflects the reviewer’s summary review of the proposal, and that hasn’t changed. It is still up to the reviewer to weigh the relative importance of all strengths and weaknesses of the proposal in the way she believes is most salient. As noted/designed, this seems more likely to affect reviewers than applicants.

  50. anonymous says:

    I have participated in a number of study sections over the years. To really do justice – I think we need to address the first two for sure in the new format and be blinded to the PI, research team, institution. So many times, there were disparaging comments based on institution and almost a ‘pass through’ as others have mentioned for seasoned investigators. Blinding reviewers to this information would potentially help mitigate biases that unfortunately exist.

    But I do favor the changes!

  51. Carolyn Anderson says:

    The proposed review criteria address all the important points and will take some pressure off of busy reviewers. It’s good that NIH periodically assesses their proposal review process and makes changes.

    To those who think we should blind reviewers to the identity of the PI and co-investigators: NIH tested this idea some years back. I was sent several “test” proposals that were previously reviewed and scored, where identifiers of the investigators were removed. In at least half of the applications, I easily guessed whose proposals I was reading. It’s impossible to completely blind reviewers, since we all read the literature and with a minimal amount of detective work can figure out whose proposals we’re reviewing.

  52. Stuart Gardiner says:

    As a reviewer and submitter:
    The single most important aspect of this proposal, which I generally like, is that “Importance” and “Rigor / Feasibility” need to be scored *before* expertise. Additionally, the Aims and Research Plan need to appear in the assembled grant *before* the biosketches, resources & equipment, etc.. That would greatly reduce bias (conscious and unconscious) towards more eminent researchers with great track records, and increase the focus on the actual scientific merit instead. I don’t support completely blinded review (it’s only fair to trust a more experienced investigator more), but the balance isn’t right at the moment, and simply reordering things would help.

  53. Harve hERSCHMAN says:

    I have sat on many study sections, and chaired one for five years. I do NOt understand “innovation” as an important aspect of review. “new” or “innovative” is not necessarily a virtue. There are good innovations, mediocre-unimportant innovations, and awful innovations in life and in science. The important question is whether the idea (innovative or not), the method of invstigation (innovative or not) and the interpretation (innovative or not) make good science.

  54. Francois Villinger says:

    Seems like a step in the right direction, to emphasize the value research project proposed. While I can understand the frustration of new investigators at the prospect of a biased review, the track record of an investigator remains a factor in judging the feasibility of a project. Eliminating (or at least minimizing) bias is our collective task as reviewers.

  55. Anonymous says:

    I understand that there may be some “undue” influence of the investigator’s reputation, but the reputation is largely based on their track-record for respected science and contributions that others see as significant. So, I agree with couple comments above: the scientist’s overall reputation is legitimately relevant to feasibility and to likelihood of successful execution, particularly for a challenging innovative project with potentially high impact. It seems reasonable to see a higher chance of success for a risky “out there” proposal (with innovative ideas or approaches) from someone with a strong reputation for success at challenges, developing new approaches, etc.
    For renewal, it is reasonable that the recent 4-year period is a significant component of evaluating investigator chance of success, however that is too limited a criterion even for renewal. Innovative projects may take longer to develop new approaches and test new concepts, so if an investigator has previously achieved highly significant impactful research that breaks new ground, that should be part of the evaluation. Isn’t that the focus of the NIH biosketch? Related to this, evaluation of the PI should be based on quality and impact of specific findings, as described in biosketch, not on number of publications (and preferably not just on journal prestige.)
    Regarding the reputation of the institution, I agree that that is not so important and may have too much undue influence…distinct from the importance of investigators record.

  56. Cristina Palacios says:

    I agree with the new plan. this simplifies the process while at the same time focusing on the important factors.

  57. Victor Mark says:

    I support eliminating the “innovation” criterion. I served on an NIH STTR research grant panel for 5 consecutive years. I repeatedly asked the SRO how to rate “”innovation.” He admitted that it is unwise to completely innovate a research grant without precedence, because a reviewer would not have confidence over feasibility. At the same time, many grants are brought down because they made only incremental innovation, despite the potential impact toward health care if the grant were to be funded.

  58. Irving H. Zucker says:

    This is like reshuffling the deck chairs. There is no way to hide the reputation of the investigators or the institution. Reviewers will not be able to ignore these issues where ever they are buried.

  59. anonymous says:

    I think the proposed revision is an excellent idea. It would also be useful to reviewers if NIH could provide a definition of innovation and novelty. Too often reviewers seem to interpret theses in the most extreme sense, such as the need for a new chemical element or law of physics. There are many definitions of innovation and novelty, such as one by Peter Drucker: “innovation [endows] human and material resources with new and greater wealth-producing capacity.” “Novelty is a side effect of innovation, not its essence.” NIH should establish and make available its own definition to remove reviewers’ confusion. Every proposal review criterion should have a clearly stated and unambiguous definition.

  60. anonymous says:

    The proposed new scoring criteria is a welcome improvement in that it places more emphasis on the science (factor 2) and the rationale behind it (factor 1). Too often it seemed reviewers used “investigator” and “environment” criteria to penalize junior investigators and investigators working at institutions below the top tier, making it difficult for even high quality proposals to reach the funding threshold and thereby perpetuating inequities in the distribution of NIH dollars. It is hoped that this behavior can be curbed but one wonders how much impact the change will have if these criteria remain a part of the overall impact score.

  61. Katherine Loveland says:

    I tend to support these changes. The simplification may make it more likely that people will agree to review, which is currently a problem.

  62. Anonymous says:

    These changes if implemented correctly should help mitigate bias towards early-career investigators and others with less reputation or from less prestigious institutions. Agree that blinding is hard because we need to know if someone can do the work and is well supported, but agree that should be “adequate” rather than scored.

    Reducing the burden of review is also something very important as recruiting reviewers is harder and harder. As others have suggested, making some sections such as budget reviewable by NIH program staff rather than reviewer could help. Additionally, the burden of human subjects submissions for non-interventional studies has only increased and makes no logical sense for many types of research. For example, an observational cohort or a set of interviews or focus groups are now must complete all of the “clinical trial” ethics documentation, which in many cases is not relevant and is onerous and is not congruent with the questions asked by IRBs for this type of work. These additional many sections also burden the reviewer

  63. James George says:

    These new criteria appear to be a move in the right direction as long as it is made clear to reviewers that the review should provide specific comments as to how the proposal could be improved. It is critical that the quality of reviews be kept as high as possible. That being said, the funding lines are so low that I contend that there are largely no meaningful differences among the top 20% of grants. So, whether a grant in that category makes the funding cut is increasingly unrelated to quality.

  64. Gino Cingolani says:

    In my opinion, this is a BAD idea.
    If the Overall Impact Score continues to reflect Factors 1 + 2 + 3 but only Factors 1 and 2 are scored,
    then a poorly qualified investigator would end up with an Overall Score that is significantly lower than the scores given to Factors 1 and 2:
    that is exactly what we ask reviewer NOT TO DO. This system will result in even greater score compression.

    I understand NIH wants to reduce reviewers’ bias, but this is the WRONG way of doing it.
    Let the SRO and study section chair educate the panel on how to minimize reviewers’ bias. Let them do so before, during and after the panel.
    Failing to score an investigator (or a team of investigators) is a simply bad idea, fundamentally incompatible with MPIs grants where a strong
    team of investigators can be an invaluable asset.

  65. John H. Wolfe says:

    I have been reviewing for over 30 years with experience in several different types of study sections since my work incorporates several scientific areas. This format looks much improved, especially the Investigators and Environment sections not carrying equal weight with the Significance, Feasibility and Approach evaluations. The effect was to trivialize parts and most reviewers based their overall scores on Approach. However, additional improvements in the instruction should be made. The original intent of having the strengths and weaknesses sections was to highlight the main score drivers so the applicant could clearly identify what was most important. There was supposed to be a written narrative at the beginning of each section evaluating the proposal, with the notation of the most important issues in the strengths and weaknesses section. That very quickly degenerated into only using bullet points, many (most?) of which were trivial and thus were no help to the applicant. I suggest that a written narrative be mandatory for the new Importance and Feasibility sections, with additional explanations as needed in the strengths and weaknesses under each item. The form should have a large black space for the narrative so reviewers see that something needs to go there.

  66. Danny Bluestein says:

    While this ‘simplification’ may help (I do think that a review should be specific and detailed enough), the Overall Impact Score is always the most problematic aspect of the review process as it is not transparent. More than a few reviewers hide behind it as they are concerned that the applicants may identify them- resulting in other criteria scored relatively high, with the non-transparent Overall Impact Score not reflecting the transparent scored criteria. I believe that it should be mandated that the Overall Impact Score reflects the average of the other scores.
    Additionally- consider adding a specific criterion for revised proposals- how well previous critique was addressed (to preserve the continuity of the review process).

  67. Ian Davis says:

    I love that you are proposing to remove the Environment criterion score – to me, this is a throwback to the days before the internet when everything had to be local and collaborative science at a distance was much more challenging. If the equipment and expertise are present either on-site or with a collaborator, that is all that should matter as a simple yes or no – too often people are getting better scores because they are at more prestigious institutions.

  68. Anon says:

    Much simpler and better. I particularly like the change to how investigators and institutions are evaluated so that they focus more on ability to do the work than institutional prestige.

  69. Heather L. Tubbs Cooley says:

    This is a positive move by NIH. I’m a current reviewer and often wish for both simplified criteria AND format for reviews. This seems to accomplish both. Please *require* reviewers to justify scores other than 1; too often reviewers will give a score of 2-5 without explanation or context, impairing an applicant’s ability to develop a successful revision.

    The Innovation section generally needs more thought and perhaps more specificity; this criterion seems to get tangled up with both approach and significance. My sense is reviewers often consider innovation to be primarily innovative methods, and negatively score if these are not “new” even if the application is focusing on a bold new idea or exploring new interventions or relationships. One way to address this would be to make this a standalone yes/no criterion based on evidence of innovation in research question, conceptual framework, methods, or analytic approach.

  70. Anonymous says:

    I like the proposed changes – much simpler and to the poing. I also like the comment about making the science review anonymous. Too much emphasis on experience jeopardizes creativity and new ways of thinking.

  71. Steve Kron says:

    If we don’t let reviewers score Factor 3, Investigator and Environment, then their bias will leak into their Factor 1 and 2 scores. A key argument behind the current scoring rubric was to encourage reviewers to individually evaluate Significance, Innovation, Investigator and Environment separately from Approach. This replaced an unstructured review that was very inconsistent and biased, which favored the old boys network, leading to reviews along the lines of “Sure, this looks almost impossible, but XXXX can get it done” or “I just don’t see this investigator as having the experience to do something quite so ambitious”. So, this looks like backsliding and simply offering a means to hide your bias by pointing to weaknesses in the importance of the work or the quality of the science. Improving the review process is certainly needed, but that involves confronting issues like bias rather than burying them.

  72. Jonathan C Makielski says:

    These changes are evolutionary and likely to be well recieived by applicant and reviewer alike. They focus and weight the factors appropriately.

  73. Jaquelin Dudley says:

    The major problem with the review system is that there is no unbiased way to evaluate the significance of the research. Therefore, reviewers are forced to use criteria, such as publications in elite journals or numbers of papers, to determine the impact factor. Innovative research is often punished because it is easier to find fault. Many reviews focus on picking apart the approach rather than how the proposed experiments will move the field forward. Focus on the big picture makes the task even more difficult and time consuming for busy reviewers who essentially volunteer their time.

  74. Kurt Beschorner says:

    I fully support this change.

    As a frequent study section reviewer, I have been frustrated at trying to create a spread of scores across applications where the investigator and environment were capable and sufficient for conducting and supporting the proposed research. The result has been that most of these applications end up with very similar scores and portions of the scale are rarely used for these two criteria. I had been hoping that CSR would change these two criteria to categorical feedback rather than quantitative feedback.

    I am indifferent about combining significance and innovation. I think that these criteria serve an independent function and am OK with the current system of having them as separate criteria. However, I would also be OK with the proposed system of combining them to a single category reflecting the importance of the research.

  75. anonymous says:

    I love this new review criteria to focus more on science and impact instead of the reputation/popularity of the research team or institute. As a junior faculty, I have fortunately attended one NIH review section in-person at DC. During the 2-day review, I was very disappointed to see how biased the review can be when evaluating the applications from well-known universities or PIs. Just because the PI is a famous successful scientist from a top university, reviewers will assume the science is perfect and do not need careful evaluation. But if the application is from a small university or a junior faculty, reviewers will more carefully evaluate the science trying to find weaknesses. After that wonderful experience, I start to feel that getting a NIH grant is like gambling, all depending on who review your grants (how lucky you are!).

    If possible, I even hope the NIH grant review process can be blinded like manuscript review, especially for the first two new criteria.

    • anonymous says:

      I completely agree with the comment. I also hope that the NIH grant review process can be blinded for the first two new criteria. I have reviewed grants for NIH in the last 15 years. It is very obvious to me that an established investigator or an application from top university receives favorable scores.

  76. Anna Wilson says:

    I think this would be a really positive shift that would help keep study section discussions and scoring focused on the core scientific strengths and weaknesses of proposals. Anything to simplify the other review criteria section would be appreciated. Thank you to all of those who have been working on this.

  77. Steven Safren says:

    This change will make zero difference in terms of advancing science except that it will confuse reviewers and applicants to switch to a new system instead of the system that we are all used to – and therefore make things worse and more time consuming for everyone to figure out. Please do not change everything whenever we get used to one system. So first piece of feedback is — dont make a change unless it is really needed and would really make a positive difference. Second piece of feedback: for the people submitting grants — why combine categories versus have information separately on each domain?

  78. Chang Kim says:

    This will be a good change. I like that as a reviewer. The rigor and feasibility criterion is often confusing and being misused by reviewers. How about changing that to “overall effectiveness of approaches”?

  79. David Kessel says:

    Looks good, but the overall efficacy of the review process will be no better than the ability of the individual reviewers to evaluate projects. This can be a difficult item to deal with.

  80. Hank Seifert says:

    This makes sense since these are realistically the only score-driving factors.

  81. anonymous says:

    I chaired a study section for two years and so have a lot of experience.
    I like the change back to a 9 point scale.
    I like the three criteria instead of five.
    I don’t see any good reason not to score the “expertise and resource” section. The quality and track record of the PI is very relevant. By not including a number this will tend to de-emphasize this criteria.
    The minor changes are good.
    Overall I am quite positive about the change.

  82. Robert bartlett says:

    This seems to bed the same. As the current system ( which ain’t bad. )

  83. Anonymous says:

    This looks very logical and clarifying. I support the proposal.

  84. Robert Thomas, M.D. says:

    Anonymous (blind to investigator) scientific review is even better. Once the science is scored, THEN other information could be visible, but the science scores cannot be changed.

  85. Julio Duarte says:

    I believe this proposed review changes are an improvement on the previous process – it seems to still allow sufficient review of the science while taking some of the administrative review burden away from the scientific reviewers.

  86. Jonathan Cooper says:

    I support this change. It allows reviewers to focus on the science, undistracted by peripheral considerations. It may make it easier to recruit reviewers, and will simplify the summary statements.

  87. Anonymous says:

    Hi folks,
    I have regularly served on NIH review panels since 1980. I judge that the proposed changes are moving in the right direction. However, having recently chaired a special review panel and in viewing the two required training modules (which were very good), I suggest the following changes.
    Criteria 1 and 2 are the most important for the advancement of science; the remaining criteria are secondary and could easily be handled by program staff. Accordingly, blind reviewers to the identity of the PI, the institution, and to the budget. I suspect that everyone would be surprised with the outcome.

  88. Hiten Madhani says:

    I believe that the past record of an investigator and their institution has significant predictive value on the quality of their future work. Thus, I do not agree with the concept of “mitigating the undue influence of the reputation of the institution or investigator” if the goal is to fund the “strongest, high-impact research.”

  89. David Warburton says:

    The main point of peer review is to decide whether the proposal if executed as written is likely to make a useful difference to human health.
    These changes are a step in the right direction and may help to make the task of peer review less onerous.

  90. Erica Golemis says:

    This seems to be an excellent plan. It would make life much easier for reviewers, while capturing all relevant information.

  91. Francisco J Alvarez says:

    I think this makes a lot of sense.

  92. Andrei Ivanov says:

    I do not like the proposed changes. The new system distract from two very important score driver for the application, which is Innovation and Investigator. By combining these Score Points with other sore points NIH signals for diminishing of their value.

Comments are closed.