Review Matters

Seeking Public Comment on CSR’s 2022 – 2027 Strategic Plan

Author

February 14, 2022

I am pleased to announce that CSR’s draft strategic plan is now open for public comment. This 5-year plan (for 2022–2027) will serve as our roadmap as CSR advances its mission of seeing that NIH grant applications receive fair, independent, expert, and timely scientific reviews—free from inappropriate influences—so NIH can fund the most promising research.


Input from CSR’s stakeholders—the external scientific community, the CSR Advisory Council, NIH institutes and centers, and our own CSR staff—helped to shape the goals of the plan, all of which center on strengthening peer review. Input included critical discussions about topics that have received increased and necessary attention recently, including structural racism and the COVID-19 pandemic. The goals are:

  • Goal 1: Maintain scientific review groups that provide appropriate scientific coverage and review settings for all of NIH science.

  • Goal 2: Further develop a large cadre of diverse, well-trained, and scientifically qualified experts to serve as reviewers.

  • Goal 3: Further develop an outstanding, engaged, and diverse staff.

  • Goal 4: Implement changes to the peer review process to make it more fair, effective, and efficient.

  • Goal 5: Achieve our mission through transparency, engagement with the scientific community, and a data-driven approach to decision-making.

Collectively, these goals reflect CSR’s core values, which are 1) ensuring fairness and diversity in all CSR activities, 2) maintaining open lines of communication with our stakeholders to ensure transparency and to take full advantage of the expertise and experience that they offer us, and 3) using cutting-edge data science and informatics in our decision making.

Everything we do at CSR ultimately supports our mission-driven work in peer review, and the strategic plan goals and our core values upon which they are built are no exception. By ensuring a high-quality and fair peer review process, we advance the mission of NIH and support the Nation’s biomedical research enterprise.

Comments on the strategic plan will be accepted through April 30, 2022. The full plan may be found here; we welcome comments through Review Matters or sent to feedback@csr.nih.gov. I look forward to hearing your thoughts.

25 Comments on "Seeking Public Comment on CSR’s 2022 – 2027 Strategic Plan"

  1. anonymous says:

    Please continue with virtual study section. There is no question that in-person meetings place a greater burden on the environment, the NIH budget, and individual reviewers. Traveling is especially tough for those with family responsibilities, disabilities, heavy teaching loads, labs that require close supervision, etc. Additionally, women and racial/ethnic minorities are more likely than men or white folks to have home caregiving responsibilities. If the goal is to diversify the reviewer pool, we should focus on efforts that make it easier for those from underrepresented groups to participate.

    I also support comments in favor of a blinded review system. It doesn’t seem reasonable to ask applicants to conceal their identity in the research proposal since feasibility can depend on one’s published work/expertise in the area. However, requiring reviewers to submit a score for the research proposal before viewing/scoring the identity elements may help to reduce bias in overall impact scores.

    I have seen NIH resources get wasted in big labs that are poorly managed and funded by multiple mechanisms to do the same work. Labs like this look successful on the surface, but productivity on a per-person or per-dollar basis is low. When evaluating the individual and environment, it would be nice to see some criteria that explicitly account for a lab’s total funding and size in order to more fairly compare PIs that differ along these dimensions. For example, applicants could be required to list the total amount of funding they receive from all public and private sources (why was this removed from the Biosketch?), project titles and specific aims for each funding source to assess potential overlap, and all members of their lab. NIH had a great idea a few years ago to implement a cap on grant support to big labs for this very purpose; it’s a shame that didn’t move forward. I’m not sure if this fits within the purview of CSR, but it seems relevant to Goal 4.

    Thank you for soliciting comments–I appreciate the insights and different perspectives I’ve been able to read here.

  2. Anonymous says:

    I support comments utilizing a blinded review system to score significance innovation and approach. I would go further and recommend that environment and investigator be assessed and negotiated between universities and NIH similar to U19 type cooperative agreements. I would also suggest that rigor, reproducibility and rigor of prior research guidelines be strengthened at both the level of submitted proposal and future studies. Rigorous Data should matter in scientific review.

  3. Mark Huising says:

    Please consider retaining remote study section panels. It is a significant time saver for those with family/child care obligations that fall onto the working spouse (in my case) every time I travel. I have participated in several remote study sections over the past few years and have found the level of discussion and engagement to be as strong and -at times as frustrating – as the in-person study sections I participated in before Covid19. Being able to sign off and still spend time at home is priceless. There also is a compelling argument to help save our planet and to stop feeding a fossil fuel reliance that we can now plainly see is deeply geopolitically destabilizing. Flying alone accounts for 3-5% of global GHG emissions, air travel is one of the hardest aspects of academic life to decarbonize and US academicians fly far more than their share. This behavior is perpetuated by normative expectations from our institutions, including the CSR. There is a tremendous opportunity here for the CSR to lead in re-aligning these normative expectations. In person interactions are important and will and should continue as a vital part of the scientific exchange, but that does not mean that all of our interactions have to be in person. Sure, there are downsides to remote panels. But over the past two years, grants were submitted, scored and funded. Peer review continued. Let’s not reflexively move back to the imperfect review system we had pre 2020, but work on improving our current, functional and remote peer review system to make it as inclusive to all panel members and fair to all applicants without perpetuating unsustainable behaviors. The science on climate change that is reported on by our scientific colleagues is not in dispute, The NIH and its constituents have a responsibility to follow this science and bring our behaviors in line.

  4. RAJITA SINHA says:

    Ensuring good quality reviews is a key feature of what CSR does. To facilitate that process, I hope CSR will resume in person study section meetings and reviews. The quality of web-based reviews are poorer especially in the discussion phase. People are less likely to chime in and participate and bring up god questions and issues if they are remote. Mostly the discussions is between the reviewers, and less enriched by other members. Also, can there be scores provided even if an application is not discussed? This can be important feedback to the applicant regarding resubmission. Perhaps it can be “provisional” scores which is a tally of the 3 reviewers scores. Finally, the new rule for resubmissions and competitive renewals not being able to show changes is not at all convenient for the reviewers and not in favor of heh applicant. it would be good to bring back the option to show clearly where changes and improvements have been made.

  5. Douglas Brash says:

    The Plan is excellent in generality. My concern is on specifics.
    1. Deciding study sections. What are the criteria? There is a study section for Neural Oxidative Metabolism, Mitochondria and Cell Death, but only one for all of Cancer Etiology.
    2. Impact. The phrase “best, most disruptive, novel science” is encouraging re promoting non-incremental science, but it will have more effect if you lose the word “best”. Will the best person in the room please stand up?
    3. Risk. “Disruptive, novel” entails risk and some grants may not reach their goals. 50% success would delight a VC. NIH must find a way to let reviewers accept risk. EW’s suggestion of a milestone at Y02 is a good solution to managing risk. A former CSR leader told me the important grants were in the triage pile, identifiable by high SD; the Air Force OSR told me that’s where they look.
    4. Scoring. 1-9 was invented by statisticians. It is astounding that a scientific enterprise does not recognize that there is an error bar to reviewing. I don’t think it is ±1, which determines funding. If we were honest, we’d have a lottery for grants in the 8-20% range.
    5. Review volume. Many PIs in medical schools write grants to support our salaries. If reviewers and Institutes allowed 40% PI salary on a grant, CSR’s volume would go down by half and PIs would focus on research. I second MP’s suggestion to first submit letters of intent.
    6. Reviewer load. It indeed has become undoable, especially in a soft-money environment. Reviewing 10 grants has gone from 10 fulltime days to 20 due to training and checking validity of premise. At 20% PI salary on a grant, that is 20 weeks away from the grant. Albeit half the grants I reviewed did cite references not validly supporting the rationale.
    7. Bias. One has to consider the investigator; this is arguably more important than the Research Plan as of the day of submission. Anon’s & US’s suggestion of dividing review into blinded science and unblinded investigator seems excellent. Another bias present for 50 years (I’m told) is “coolness”. US science hops from fad to fad. I assume this year CSR is flooded with single-cell grants. Ten years ago it was epigenetics. Thirty years ago, epigenetics couldn’t get funded. If we want to understand biology and cure diseases, an alternative mindset and criteria need to be instilled.

  6. AA says:

    I am in agreement with many posts here regarding blinded reviews (reviewers not known) and hybrid format meetings.

    In addition I would like to suggest that NIH should re-evaluate the five review criteria. Based on my experience serving on many study sections and as a PI on multiple R01s for more than a decade, I suggest the following criteria:

    Significance
    Preliminary Data
    Feasibility
    Investigators
    Environment

    The first key change I am proposing is that Preliminary data should be added as an essential evaluation criterion. I have often observed that reviewers overlook this aspect in grants and tend to focus too much on Significance and Approach. The question should be: Does the grant show preliminary data that strongly supports the hypothesis? Needless to say this should be an essential requirement for an R01 level grant.

    The next suggestion is to do away with Innovation as a criterion. The key question has to be: Does the grant address an important need that has previously not been solved? Innovation implies that every investigator would need to come up with their own new tool. This is not possible in most situations, where existing tools or methods can be used to address challenging problems that otherwise would not be addressed. Coupled with the preliminary data requirement, the proposal would ensure that the investigators have shown originality related to the specific project.

    The next suggestion is to replace ‘Approach’ with ‘Feasibility’. Quite often reviewers nit pick on intricate details in the Approach. This is counter-productive, since in most cases projects do not follow the ‘Approach’ even close to how it is described. Rather, I would suggest that applicants include a feasibility section where they show evidence that they have the tools and methods to carry out the specific goals of the project. Again, strong preliminary data should give confidence that the team is capable of succeeding in the project.

  7. Marcelo Araujo says:

    The American Dental Association Science and Research Institute (ADASRI) extends its appreciation to the NIH Center of Strategic Review for an excellent and well-constructed Strategic Plan. The “Early Career Reviewer Review” program is especially important and should be commended. Also, efforts of Goal 4, “Implement Changes to the Peer Review Process to Make it More Fair, Effective, and Efficient,” to improve the peer review process, particularly to remove bias, should be commended similarly.

  8. Michi Taga says:

    Echoing earlier comments, double blind reviews would be a big step toward reducing bias. Reviewers could, for example, review proposals in two phases. In the first phase they would review the Research Strategy only (with no PI identifying information), and submit their scores and comments only for Significance, Innovation, and Approach. In the second phase they would receive the full application and submit their scores and comments for Investigators and Environment. There would be some technicalities to work out in order to structure the reviews in this way, but overall it seems extremely doable.

  9. Ellen Wijsman says:

    I would like to see CSR use the history of scores from each reviewer to at least partially adjust for reviewer effects. I have been in reviews where it was quite clear that some reviewers gave consistently better scores than average, and other did the reverse. This inhibits the fairness of review, since some people will be assigned the “easy” reviewer, while others will be assigned the “tough” reviewer. There are statisticians working at NIH who should be competent to try to develop something like this, using past scores of primary and secondary reviewers and other such reviewers who were in the same review sessions.

    It might be possible to move the review process towards allowing more uncertainty of feasibility, which seems to be a major barrier in getting review to the state where the really interesting ideas get funded. Right now questions about feasibility of novel techniques can kill a proposal, Perhaps some proposals or grant mechnisms could have a first aim with an intermediate goal that would have to be achieved before awarding the rest of the grant. That first aim would have to include demonstrating the questionable feasibility (maybe no more than 2 years into an R01).

  10. John G. says:

    I echo the comments of my colleagues advocating for hybrid and/or virtual meetings to ensure the diversity of the reviewers. As a parent of three with a working spouse, I have turned down numerous study section opportunities due to travel limitations. Through the pandemic, I have been able to be a contributing member of the review process. Infrastructure investments into virtual/in-person hybrid options may be a happy medium to ensure an inclusive process.

  11. Thomas Hughes says:

    I live and work in Bozeman Montana. I enjoy participating in NIH study sections and feel that I bring a different perspective to the table. That said, I don’t have easy access to a lot of the fancy journals. Indeed, I just finished a review meeting where I had serious problems with paywalls. One way of rewarding reviewers like myself would be to give them proxy access to the NIH intramural library and subscriptions for the month before a review. It would certainly ensure a better, more informed review, and it would be quite attractive to scientists like myself. Not all of us work at fancy universities with access to all the journals.

    • Anonymous says:

      Second this – great idea to give library access. Of course without that there are always other means (scihub) to get most papers.

  12. Anonymous says:

    RE: Goal 2 – I believe virtual meetings are crucial for developing and maintaining a large cadre of diverse, well-trained, and scientifically qualified experts to serve as reviewers. I am on the more junior side of NIH reviewers, having now served on 4 study sections in the last 3 years. I strongly urge continued virtual meetings. I have found the level of conversation to be at least on-par with in-person meetings, if not better, and as a researcher with young children and family obligations, I am grateful for the chance to meaningfully contribute to the review process while still being present with my family in the evenings and not adding to the environmental costs of air travel.
    RE: Objective 2.1 – the Early Career Reviewer program is outstanding as a means of letting early career investigators enter the reviewer pool in protected capacity. I am glad to see clear metrics for ECRs in each study section. Strongly recommend continuing this program and broadly advertising for diverse applicants into the program.
    Re: Objective 2.2 – I appreciate the recent training on bias, as it provided common language for reviewers to call out biased or problematic statements in the session immediately following that online training. More work to do, but grateful for this.
    RE: Objective 2.3 – as a reviewer, I would welcome receiving metrics/feedback on my reviews. I would also welcome being able to review and update my reviewer profile, as I sometimes wonder (based on the study sections I am invited to participate in) whether I might better align my profile with my areas of expertise.

  13. anonymous says:

    Please Please Please keep panels in person. The quality of the reviewing process is much higher and the benefit for each of the reviewers to interact is also important. This is an opportunity for junior investigators to meet (actually meet) others in their field. It is already a challenge to find quality reviewers and if meeting go virtual there is less incentive to put in all the effort. Also please dont let grant reviewing drop to the poor level of manuscript reviewing. In study section there is peer pressure to be (or at least appear to be) fair and balanced. Reviewers put in the time and effort because of commitment, yes, but also not to look foolish in front of their peers. Reviewers know that they will need to defend the good or bad score and receive immediate feedback. I feel that this is a primary reason that the grant review process is so much more thoughtful than manuscript reviewing in which we are all subject to the whims of an unchecked reviewer. Fairness and quality requires time, effort and yes money. If anything, we need to bring the process of manuscript reviewing to a higher standard, perhaps like for eLIfe in which reviewers and editors discuss the submitted reviews prior to the editor making a final decision. We all have our biases and opinions that require feedback and discussion; and we benefit from the feedback we hear in study section. To loose all this to efficiency would be a huge loss for making the best decisions for NIH funding distribution. There appears to be differing opinions on this point – the cost and time are good points and virtual meetings are now commonplace and the meetings themselves tend to go well. But is the quality of reviewing, that is the outcome, the same, better or worse.

  14. anonymous says:

    Agree with other comments that remote meetings are more practical and as effective as in person meetings. Reducing air travel should be a priority for everyone. Remote panels that meet at 8 or 9am ET place a heavy burden on reviewers from the west coast and mountain states. Simply changing the start time to a later hour would reduce bias in the review.

  15. anonymous GH says:

    The goals presented are laudable and generate much food for thought and discussion. This, in my case being from the vantage point of an investigator who has written and reviewed a rather vast number of proposals over an almost 50 year period, and who is still at it. A perusal of the 9 comments currently posted illustrates a notable engagement with Goal 4 and a need for a “fair, effective, and efficient process”. I choose to interpret this, at least in part, as a need for a review process that will optimally promote research most likely to yield scientific progress and ultimately new clinical interventions.
    To briefly address this, I have participated in many reviews of strong proposals that I was convinced should have received much better scores than they ultimately did. In looking back at these and my own submissions that were the most challenged in review; they were mostly a mix of complex concepts (with many what-ifs) and especially submissions that focused on new/novel concepts with some risk. Such reviews often put a strong emphasis on what would most likely “succeed”, an understandable response to a degree, during times of tight budgetary constraints.
    A key point was made by Anonymous # 6 (Feb 17) stating a need for reviewers who are “much more driven by impact and significance”. I totally agree with this and submit a clip from an editorial written by Francis S. Collins in Science Mag. when asked what he had learned while being the agency’s director. This is the quote: “Hypothesis-driven research is the bedrock of NIH’s success. Thus, one should prioritize support of risk-taking research and pay less attention to preliminary data and more to the potential importance of a premise.”
    It is my hope that NIH will proceed along this path.

  16. Janet R. Katz says:

    The plan addresses much needed initiatives to improve diversity, ethics, and quality. My comments are related to:
    1) Goal 4- making a fair, effective, and efficient process.
    2) Objective 4.3 Includes focusing reviewers on scientific and technical merit- that seems simple enough but what does it mean? For instance, I have repeatedly experienced a very casual response to errors in qualitative research in applications during reviews-people state “the PI or CoI are experts in qualitative research” and being content with that! An increasing number of applications are using mixed methods but the qualitative is never held to the same standards as the quantitative. More reviewers with qualitative expertise are needed otherwise this aspect of the research proposals is window dressing. Findings may not be valid and waste tax payers money.
    3) Also on 4.3 Blinding is good idea as we know from research that people may respond to names and assumptions in a biased manner.
    4) Objective 4.4 Assignments for appropriate reviewer groups (Again, increased qualitative research requires expertise) We also need community based research expertise including expertise in working with non-white groups. The majority of reviewers are white, including myself. As with qualitative research I find community concerns and ethics may take a real back seat to statistics. How the community is involved affects validity of the research especially of interventions. Perhaps especially true with Native American communities who most researchers do not understand. Perhaps training in this area for the staff help?
    4) My summary- please increase community and qualitative expertise among reviewers. Make sure ethics includes community ethics!
    Thank you for opportunity to comment.

  17. Michele Pagano says:

    Goals 1 and 2: Progress in science is characterized by two key stages: funding of the research and publication of the final results, both requiring peer review. Peer review of research papers is completely electronic. If it works at this level, it can also work for the grant review process. Critiques and percentile scores could be sent electronically to the NIH to avoid reviewers having to travel (saving money and time). If necessary, the written critiques could replace the discussion in guiding panelists who are not reviewers to posting a priority score. In the small percentage of cases in which the opinions of the reviewers are widely divergent (in my experience this happens in <15% of cases), the program officer could organize discussion either online (as for eLife) or via zoom. Online study sections would streamline the current process and allow revised applications to be prepared in a timely fashion and increase the chances that they will be reviewed by the same reviewers. Each study section would be managed like a scientific journal and be distinct from the other study sections, keeping its own system to calculate scores and percentiles. Submission deadlines could be maintained, increased in number per year, or even become completely open, as is the case with scientific journals.

    Goal 4: New applications could be submitted as letters of intent. Prescreening of the almost 60,000 annual applications would avoid in-depth review of poor-quality applications and would decrease the number of reviewers needed.
    Moreover, rather than rewarding grant-writing skills, reviewers should focus on the big picture: the likely impact of the proposed work and, in the case of competing renewals, productivity as well.
    Finally, applicants are currently required to describe the work to be accomplished over the entire grant period and to include timelines that are sometimes unrealistic. The notion of anticipating what a scientist will be doing even one year hence is difficult enough; anticipating five years into the future is impossible. Indeed, if the scientific enterprise were predictable, science would be banal, perhaps even boring. After describing the project, a general idea about directions and future plans should be enough.

  18. anonymous says:

    Given the enormous workload that NIH grant review entails, switching from $200 per session reimbursement to $200 per grant review reimbursement is crucial. Many other granting institutions have already enacted this “per grant” reimbursement schema.

    This comment is not out of financial self-interest but based on studies showing how economic factors limit the development of a diverse workforce. As CSR seeks to broaden the pool of experts, the additional time commitment of reviewing for NIH will fall disproportionally on women scientists, minority faculty, and scientists in COBRE states. Those are the ones that can least afford to siphon off time from their research, administrative, and teaching commitments.

  19. James S. Malter MD, Professor and Chair, Department of Pathology, UT Southwestern, Dallas, TX says:

    All good goals and objectives. SO far as generating and maintaining fair and qualified reviewer panels, I believe that established investigators (previous and current R01 recipients) be required, as payback for prior NIH funds and support to serve on study section. My experience has been there are simply too many junior and not enough senior reviewers on many panels. A reasonable payback period might be 1-2 years service for every 5 years of NIH funding.

  20. Suzanne Pfeffer says:

    Please retain in person panel meetings. Video meetings do not similarly engage the entire panel as well as in person meetings. Thank you.

  21. anonymous says:

    For goal 4, I think it would be helpful to have both blinded and unblinded reviews. Blinded reviewers would review the science only, evaluating impact and rigor. Unblinded reviewers would review the science and the investigators and environment together, as reviewers currently do. Scores could be weighted in some way to allow the blinded review to offset the unblinded review, which would be a way of addressing the “rich get richer” problem within the current system. At the same time, giving reviewers more education about how to review effectively and consistent use of the 1-9 scale would be helpful. Determining which flaws or weaknesses should be score driving vs. not feels like it can be very subjective, as does use of the scale. I definitely notice a trend when in study section that younger reviewers are much more likely to be “in the weeds” with details of the approach and weight those heavily, where more experienced investigators are much more driven by impact and significance. So, having a balance of career stages in review panels seems like it would help (Goal 2), in combination with developing some education or training to help reviewers be more consistent with how they approach the process.

  22. Anonymous says:

    Re: Goals 2, 4, 5 (all intertwined):
    Covid confirmed for me that so much of my time had been eaten up by travel. Despite the wonderful social interactions during a panel review session, it also is my experience that there are plenty of social dynamics that do not serve the science. For instance, young peers may be heavily swayed by a Person of High Regard’s different opinion and may change theirs to match. In other words, the remote panels are excellent if the goal is to expand the pool and make it more equitable across scientists and more transparent. Parents, caregivers, single parents, young professors, the environment, the taxpayer all benefit from remote sessions. To my ear, the loudest voices in favor of in person meetings have been older, established PIs who enjoy socializing and possibly some opportunity for adulation. Personally, I would much rather all the money on dinners, plane tickets, hotels, be re-allocated to the tremendous projects people are submitting! As a tax payer, if it means one more student receives a fellowship, then it is more than a fair trade. I’m sure you have all heard this before, but I really love hearing from my colleagues as sessions unfold, and I appreciate being able to do this and still make dinner and put my kids to bed.

    The NIH staff is amazing, kind, tactful, generous, in my experience. I’m consistently amazed at their competency.

    The transparency and removal of bias are real concerns, as are biases towards Elite Institutions. There is a notable benefit of the doubt extended to a certain cadre of institutions that is not universal, and concerns me.

    Thank you for asking for input. I appreciate the work the CSR is doing on these fronts and hope that it satisfies all of these important goals.

  23. Uma Sajjan says:

    One of the actions that need to be taken into consideration for fair review is the double-blinded review process. The applicants should avoid using we, our, and other identifying information in the grant body so that the reviewer is blinded. In this double-blinded process, the reviewer will score only science rather than the investigator or environment. The grant applications that are deemed to be scientifically sound based on significance, innovation, and approach section should go for the second round of review in which the investigator(s) and the environment are considered.

  24. Doug Kellogg says:

    I served on the BRT-A study section for four years and the CSRS study section for 6 years. During the pandemic I participated in two remote study section meetings and found that they work well. I therefore urge the NIH to make study section meetings permanently remote.

    A major factor is my deep concerns about global warming. As scientists we have a heightened responsibility to do everything we can to fight global warming, and I find that I can no longer morally justify the carbon footprint of the traditional approach to grant review. We need to treat global warming with the same level of seriousness and urgency as the pandemic, as the long term consequences will be far worse. The fires on the west coast have followed the hottest driest years ever recorded, and they are just the beginning. Without immediate action, much of the state of Florida and large regions of entire countries will be destroyed, triggering mass migrations, food shortages, and millions of homeless refugees. We must act now.

    Another concern is the time costs of travelling to Washington from the west coast. Since remote review works well, I would prefer to spend more time focusing on research and teaching, and less time travelling. Finally, eliminating the airfare and hotel costs associated with study section meetings would free up funds that could be used to support young investigators.

Comments are closed.