NIH PEER REVIEW NOTES
February 1998

 

From the CSR Director's Desk

With this edition, Peer Review Notes has changed its format. The change has been primarily prompted by the availability of information on the World Wide Web. (To get to the main NIH Web site, type: http://www.nih.gov. To get to the home page of the Center for Scientific Review (CSR), type: http://www.csr.nih.gov. To get to the home page of the Office of Extramural Research (OER), type: http://www.nih.gov/grants/oer.htm.). Peer Review Notes will continue to contain brief articles on changes in peer review policies and procedures, and I will use this column as one means of communicating with members of the extramural research and research administration community.

In this edition's column, I will take the opportunity to explain the interrelationship of four activities, each of which aims to provide advice on some aspect of study section organization and function. With all the reports of various panels, working groups, and changes that are underway, it's no small wonder that folks are confused.

I have spoken often in recent months of plans to establish the Panel on Scientific Boundaries for Review. This ad hoc working group, to consist of distinguished scientists with prominence in diverse fields, is being formed at the request of the CSR Advisory Committee to undertake a comprehensive examination of the broad principles governing the organization of CSR study sections. For instance, the Panel will consider whether a major reorganization is needed, or whether continuous adjustments of the current system would suffice to identify the most meritorious and promising projects within all the fields of biomedical research supported by the NIH. If indicated, the group will recommend a strategy by which the breadth of disciplines supported by the NIH could be reconstituted into newly defined, intellectually defensible, scientific domains. These recommendations may serve in turn as the basis of a subsequent effort to organize study sections and their arrangement within integrated review groups.

At the same time, some study section reorganization activities have been initiated to integrate review from the former Alcohol, Drug Abuse, and Mental Health Administration (ADAMHA) Institutes (see below). The first of these newly developed review groups will begin operation in June, 1998, and will be evaluated 12-18 months later, perhaps requiring some adjustments in the context of the panel's recommendations.

In addition to these two activities, which are focused on broad, guiding principles of review group organization, CSR has also established some individual integrated review group (IRG) oversight groups to advise on IRG function, as described below.

Finally, CSR will continue to work closely with the Peer Review Oversight Group (PROG). PROG is a trans-NIH committee organized under the aegis of Wendy Baldwin, Deputy Director for Extramural Research, that is charged with coordinating, evaluating, and making policy recommendations for all peer review conducted at NIH, both within CSR and in the Institutes and Centers. Both I and the chair of the CSR Advisory Committee serve as members of PROG and will provide the liaison function between PROG and our other activities.

In the new year, I look forward to working with these committees, with other external committees and individuals, with the CSR staff, and with the Institute program and other staff in improving our already exceptional system of peer review at the NIH. I wish all of you a belated happy and healthy 1998.

Ellie Ehrenfeld

 

Integration of Review From the Former Alcohol, Drug Abuse, and Mental Health Administration (ADAMHA) Institutes

The reorganization of neuroscience study sections is moving into the final phase. Twenty-one newly designed study sections will meet in June to review applications submitted for the February 1, 1998, receipt date and later. A description of their review responsibilities is available through the CSR Home Page- http://www.csr.nih.gov/review/neuross.htm. Applicants are urged to use the Web descriptions to provide a cover letter indicating which study section(s) are most appropriate to review their grant applications. The rosters available on the Web are incomplete and, therefore, may not provide an accurate picture of the expertise represented by the final membership of the study section; nevertheless, with caution, applicants may find this partial list helpful in choosing a study section.

A plan to integrate the AIDS applications assigned for potential funding to the National Institute on Drug Abuse and the National Institute of Mental Health into the NIH review system is being drafted by representatives from CSR and about eight Institutes, with input from members of the extramural research community. The work of the group is scheduled for completion by May 1, 1998, including formation of new study sections, referral guidelines, and rosters of members and their competencies.

The integration of behavioral science applications, which involves 10 Institutes, will follow in one year.

Establishment of IRG Oversight Groups

CSR has begun to create external advisory committees for the individual IRGs. These committees, called IRG Oversight Groups, will consist of 5 to 10 senior researchers who possess a broad view of the scientific areas encompassed by the IRG and have peer review experience. Formal oversight evaluations will be carried out periodically (e.g., at five-year intervals), but meanwhile the groups will work continuously with and assist the SRAs in:

  •   critiquing the review responsibilities of the IRGs and their individual study sections;
  •   identifying and helping recruit new study section members; and
  •   providing advice on study section operations and procedures.

    In 1997 the first oversight group was formed to advise the Cell Development and Function IRG. Two additional oversight groups currently are being organized for the Musculoskeletal and Dental Sciences IRG and the Health Promotion and Disease Prevention IRG; and plans are being developed to extend the program to other IRGs.

    New Rating Criteria: Bringing Changes in Review Format and Focus

    Starting with the February-March 1998 review meetings, new review criteria will be used for the scientific and technical merit review of all unsolicited research project grant applications as was announced in the NIH Guide, June 27, 1997. They are: significance, approach, innovation, investigator, and environment. The purpose of this change is to shift the focus of evaluation of each proposed project away from the perceived current emphasis on feasibility toward evaluation of the overall impact that the project could have on the field. The specific inclusion of the "innovation" criterion is an attempt to emphasize more innovative concepts and approaches rather than safe science.

    The goals of NIH-supported research are to advance our understanding of biological systems, improve the control of disease, and enhance health. Reviewers will be asked to address the above five criteria, both verbally and in their written critiques, and consider them in assigning an overall score, weighting them as appropriate for each application. The single score should reflect the overall impact that the project could have on the field based on consideration of the five criteria. The application does not need to be strong in all categories to be judged likely to have major scientific impact and thus deserve a high priority score. For example, an investigator may propose to carry out important work that by its nature is not innovative but is essential to move a field forward. Written critiques should also contain a summary of the important points, addressing the strengths and weaknesses of the application in terms of the five criteria, stating which of the criteria most influenced the recommended priority score, and addressing the overall impact on the field, based on the five criteria.

    To guide reviewers in implementing use of this criteria-based system, Scientific Review Administrators (SRAs) have provided sample critiques that follow the five-criteria format, and score cards will be provided to serve as prompts. With these measures and frequent reminders from SRAs and Chairs, we expect a successful transition to use of criteria-based scoring to evaluate potential impact of proposed research projects. As with the guidelines for the review of other mechanisms, the new guidelines for the review of R01s are available on the CSR Web site at: http://www.csr.nih.gov/guidelines/r01.htm.

    Plans to evaluate the effectiveness of this change are being developed.

    Change in Policy of Supporting New Investigators

    Beginning with the June 1, 1998, receipt date for research grant applications, NIH will no longer accept applications for R29 (First Independent Research Support and Transition or FIRST) awards. FIRST awards were established in 1986 to assist newly independent investigators by awarding them 5 years of limited support before they applied for more traditional forms of support, such as R01s.

    This change will give new investigators maximum freedom in requesting support for their proposed projects. NIH has committed to support at least the same number of new investigators as before, and, as necessary, to direct even more resources to their support. In FY 1997, NIH supported 1,466 new investigators, and NIH will support at least the same number in FY 1998. NIH believes that new investigators are critical to medical research, as they replenish the researcher pool and seed it with new ideas and approaches.

    One reason for eliminating the FIRST Award was that, although applicants for FIRST Awards have enjoyed a somewhat better success rate than new applicants for R01s, they were less successful than their R01 counterparts when they subsequently applied for renewal of their funding through R01s.

    Another reason was that budget restrictions in the FIRST Award -$350,000 over a 5 year period, with no single year exceeding $100,000 - posed severe constraints on investigators' progress during the grant period. There is no set time limit, proportion of salary, or dollar cap attached to R01 grants.

    The scientific peer review process is important to all applicants, but especially so to new investigators, who are typically less experienced in the preparation of applications and in the explanation of their research plans. Therefore, measures are being taken to ensure a fair review for new investigators both during the transition period for this new policy and after its formal implementation. For further information, on NIH policy for new investigators during the transition period, see the notice in the November 21, 1997, issue of the NIH Guide for Grants and Contracts, Vol. 26, No. 38, or access the NIH Web site through: http://www.nih.gov/grants/policy/R29transition.htm. NIH staff are developing ways to clearly identify applications from new investigators and instructions for reviewers to cover both applications submitted for the February 1, 1998 deadline, during the transition period, and beginning with the formal implementation of the policy for applications submitted on or after June 1, 1998. SRAs will fully explain the impending change in policy at the February-March review meetings, and program administrators will alert Council members at the May/June Council meetings.

    Program Announcements To Be Routinely Provided to Reviewers

    Research grant applications are often submitted in response to Program Announcements (PAs). These PAs, which are published in the NIH Guide to Grants and Contracts, inform applicants of research areas that Institutes and Centers wish to stimulate. To provide information to reviewers regarding the context within which the application has been prepared, CSR will now routinely include PAs with the materials sent to reviewers in preparation for the meeting. In addition, program staff may be called upon to provide further information and answer questions regarding the intent of the PA at the beginning of the study section meeting. Should the review committee wish to comment on the relevance of a given application to a PA, this information should be conveyed to program staff in an Administrative Note. It should not influence the priority score, which is to reflect the overall impact that the project could have on the field based on consideration of the five review criteria.

    Appeals System Decentralized

     Until recently, NIH provided applicants with two mechanisms for addressing concerns about the review of their applications. The first was an Institute rebuttal process, which meant that applicants could refute the outcome of the review to program staff and request that the outcome be modified. When the rebuttal process did not resolve the matter to the satisfaction of the applicant, applicants could submit a formal appeal to the Office of the Director. A recent reconsideration of the centralized appeal mechanism indicated that the second stage, or formal appeal, process was of minimal benefit to the investigator. Consequently, the central appeal system has now been replaced with standardized and strengthened appeal procedures that take place entirely within each Institute. The new system is triggered when an applicant submits a letter detailing specific concerns about the review of his or her application to the Institute program administrator. Further information regarding the new process can be found on the Internet at: www.nih.gov/grants/guide/1997/97.11.21/n2.html.

    Statistics on Grant Applications Reviewed

     The statistics presented below are for applications reviewed by study sections and IRGs for the January 1989 to January 1998 meetings of the national advisory councils and boards (hereafter called Councils). These statistics were obtained from the IMPAC (Information for Management Planning, Analysis and Coordination) database.

    The total number of grant applications reviewed during the past 10 fiscal years increased 4.4 percent, from 10,327 to 10,785. The January 1998 cycle had the fewest number of applications reviewed since January 1989; the largest number of applications, 13,506, occurred in the January 1995 review cycle.

    CSR typically reviews between 70 and 75 percent of all applications in each council cycle, or the majority of unsolicited R01 and fellowship applications submitted to the NIH. The lowest point was 66.1 percent of the applications reviewed for the January 1991 review cycle. The Institutes and Centers (hereafter called Institutes) typically review between 25 and 30 percent of all NIH applications, mainly program project, center, institutional training grant, and solicited R01 applications. The largest number of Institute reviewed applications - 4,014 -- occurred during the January 1995 review cycle. During this cycle Requests for Applications (RFAs) also reached their peak of 1,077 applications. Applications submitted in response to RFAs have declined considerably since then, to a low of 289 during the latest review cycle.

    The percent of amended applications peaked at 33.3 percent (4,078 applications) in the January 1996 cycle. For the January 1998 cycle, the percent of amended applications was 28.2 percent (3,039 applications).

    Peer Review Notes Advisory Committee: Janet Cuca, Office of Extramural Research; Bettie Graham, National Human Genome Research Institute; Mark Green, National Institute for Alcohol Abuse and Alcoholism; Josephine Pelham, CSR; and Michael Rogers, National Institute of General Medical Sciences

    Contributing Authors: Kenneth Allison, Office of Extramural Research; Linda Engel, Carole Jelsema, Elliot Postow, Jean Paddock, Patricia Straat, and Jim Tucker, CSR

    [Referral & Review]

    CSR home page