Shortly after joining the Center for Scientific Review (CSR), I targeted for attention six priority areas, identified through extensive outreach to the extramural research community: (1) study section organization and distribution of scientific areas for review; (2) reviewer quality and study section composition; (3) perception that segments of the community are specifically disadvantaged; (4) speed and consistency of the receipt, referral and review process; (5) responsiveness to NIH funding Institutes and Centers; and (6) enhanced function of SRAs. Twenty months later, I'm amazed and delighted at the significant progress that's been made in each of these areas, in no short measure due to the hard work of staff and the important advice provided by the CSR Advisory Committee and its affiliated ad hoc working groups. I'd like to take the opportunity in this issue to provide an update regarding the first four areas. As you will see, we have been busy. However, our dedicated staff could not have accomplished everything alone. I thank all those who partnered with CSR staff to help create a review infrastructure that will facilitate the advance of today's science and anticipate the opportunities of the future.
Three activities related to study section organization are underway. Ad hoc Working Groups (of the CSR Advisory Committee) on integrated review groups (IRGs) are being formed as part of a plan to exploit the potential of the IRG as the functional review unit in CSR. The move from the individual study section toward the IRG as the fundamental unit presents opportunities for teamwork, flexible distribution of applications, and sharing of reviewer expertise. Still greater potential may be realized in sharing study section activities and reviewers.
IRG working groups will be composed of 5 to 10 active, widely respected researchers in disciplines related to those reviewed by the IRG as well as eminent senior scientists with broad perspective and vision. Working Group members will examine all aspects of IRG function, providing advice regarding boundaries, reviewer composition, and best practices. In addition, they may be called upon as a resource to identify and assist Scientific Review Administrators (SRAs) in recruiting reviewers and to deal with specific issues.
Two working groups are currently in operation: the Working Group on Cell Development and Function, formed in 1996, and the Working Group on Musculoskeletal and Dental Sciences, formed in 1998. Groups are being established for the Health Promotion and Disease Prevention, Biophysical and Chemical Sciences, and Cardiovascular Sciences IRGs. A working group for the three neurosciences IRGs will be established in 1999.
Since study sections were originally clustered into IRGs for administrative purposes and no one has every really examined how CSR organizes the science into groups for review, I have asked the Panel on Scientific Boundaries for Review to undertake a comprehensive evaluation of study section organization. The Boundaries Panel will outline scientifically defensible broad domains of science (i.e., the appropriate IRGs) and develop a set of principles to guide CSR in organizing study sections within the IRGs. The group is addressing the difficult question of how study sections should be organized by disease, body part, biological process, or methodology. Two meetings have been held to date, and plans call for the group to meet every six weeks through June.
The third activity involves the integration of review activities from the former Institutes of the Alcohol, Drug Abuse, and Mental Health Administration. Working extensively with members of the outside community, CSR and Institute staff developed 21 new neuroscience study sections grouped in three IRGs and 8 new study sections within the AIDS IRG. These 29 study sections met for the first time in June and July. Another set of study sections to review behavioral and social sciences research has been proposed through a similar process and is posted for comment until 10/9 at http://www.csr.nih.gov/review/bssreorg.htm. Final study section descriptions will be available for applicants submitting for the February 1, 1999 receipt date for June review.
As we make changes in our processes, it is vital to ask, "Have we done well?" Since a settling period of at least one year is needed before a full and fair assessment is possible, CSR is organizing a group of outside consultants to develop methodology to evaluate the effectiveness of the neuroscience integration. Similar activities will be organized for the integration of AIDS and behavioral and social science research.
Reviewer Quality and Study Section Composition
At least as vital as the way we organize study sections is the quality of the reviewers who serve on them. Thus, we are exploring flexible ways to overcome obstacles to recruitment. An experiment in the Diagnostic Radiology Study Section will test the effectiveness of an editorial-board approach, in which three to four members of one department share a single appointment, with one attending the meeting each round. We are also considering the use of senior statesmen to augment technical expertise with broad perspective. We will proceed cautiously, cognizant that continuity of membership builds understanding but that the use of more rotating reviewers could serve to break up cliques that form under the current system. The IRG working groups will assist in determining the appropriate balance and provide assistance in recruiting members.
To better monitor the nomination process, we are changing our forms to include more extensive information about the source and rationale for nominations. In addition, I have asked that SRAs broaden their nets when identifying new committee members and have encouraged them especially to take advantage of the offer of many professional societies to provide prevetted lists.
Perhaps the greatest challenge to overcome is the reticence of researchers to serve because they perceive they are jeopardized by the requirement for their grant to be reviewed in another study section or special emphasis panel. Here, I am seeking to develop solutions and evaluate the strengths and weaknesses of both new and existing mechanisms for review of committee members' applications. From this evaluation, we hope to develop some useful guidelines to apply to individual conflicts.
Early in my tenure, I highlighted three groups that feel they are underserved in the current system: (1) clinical, (2) behavioral and social sciences, and (3) bioengineering and technology and instrumentation development.
To address concerns of the clinical research community, I engaged Michael Simmons, Professor of Pediatrics at the University of North Carolina, to serve as liaison. Previous analyses indicate that clinical researchers are disadvantaged when their applications are reviewed in study sections with a small number and proportion of clinical proposals, arbitrarily defined as less than 30% of the portfolio. Such study sections are designated as "low" density. For about half of the "low" density applications, Dr. Simmons recommended that CSR aggregate clinical oncological sciences and clinical cardiovascular proposals into two new Special Emphasis Panels (SEPs). These panels are currently being organized and populated with appropriate experts. Their effectiveness will be evaluated after they have been in place for 18 to 24 months.
The Clinical Oncology SEP will review applications in the area of clinical cancer therapeutic and chemoprevention research. Clinical therapeutic studies are investigations in which clinician-investigators directly interact with human subjects with therapeutic intent. Prevention studies address interventions in human subjects that may inhibit carcinogenesis, i.e., initiation, promotion, transformation and/or progression of the malignant process. Clinical studies may include, but are not limited to, chemotherapy, chemoprevention, immunotherapy, radiation oncology, gene therapy, image guided therapy, surgery, hormonal therapy, transplantation, and clinical trials methodology (including biostatistics). This study section will not replace any existing study sections but will transfer the review of clinical cancer therapy and chemoprevention research from a number of study sections to a dedicated review group. A description of the study section is available on the CSR web page (http://www.csr.nih.gov/review/clinonc.htm). Applications received beginning with the October 1, 1998, receipt date will be reviewed by the new SEP.
The new Clinical Cardiovascular Sciences SEP will meet for the first time in June 1999 to review clinical applications drawn from those applicants believe would have been disadvantaged by assignment to other study sections in the Cardiovascular Sciences IRG. The first chairperson will be Gordon H. Williams, MD, an active spokesperson for clinical research.
For behavioral and social sciences, Leonard Epstein, Professor of Psychology at the State University of New York, Buffalo, soon will begin his role of liaison to the behavioral and social sciences community.
For bioengineering, technology and instrumentation development, however, a single person can not adequately serve as liaison to such a disparate community of researchers. Therefore, the Working Group on Review of Bioengineering and Technology and Instrumentation Development Research is being formed to identify the obstacles to fair, high-quality, rigorous review and develop a set of principles to guide CSR in establishing a technology-friendly review infrastructure.
A diverse group of distinguished members have been recruited to the task. Lee Huntsman, Provost and Vice President for Academic Affairs, University of Washington, will serve as chair. Members are: Shu Chien, Professor and Director, Institute for Biomedical Engineering, University of California, San Diego; Ronald Davis, Professor, Department of Biochemistry, Stanford University School of Medicine; Linda Griffith, Associate Professor of Chemical Engineering and Bioengineering, Massachusetts Institute of Technology; William Hendee, Senior Associate Dean and Vice-President, Medical College of Wisconsin; Susan Henry, Professor, Biological Sciences, Dean, Mellon College of Science, Carnegie Mellon University; Jeffrey Hubbell, Professor of Biomedical Engineering, Swiss Federal Institute of Technology (ETH) Zurich and University of Zurich; Steven Koonin,Vice President and Provost and Professor of Theoretical Physics, California Institute of Technology; Winfred Phillips, Professor and Dean, College of Engineering, University of Florida; and George Whitesides, Professor of Chemistry, Harvard University.
The group will hold its first meeting October 12-13 and hopes to complete its work within four- to six-months. Members will rely extensively on input from all stakeholders in the process in drafting their recommendations. Your comments may be sent to Linda Engel (engell@drg.nih.gov).
In addition to addressing concerns of the three research communities highlighted above, Maxine Lineal, Member, Fred Hutchinson Cancer Research Center, will come to CSR in January 1999 for three months to evaluate, make recommendations, and implement pilot studies of alternate ways to review fellowships.
Speed and Consistency of the Process
At the request of the CSR Advisory Committee, the Working Group on Shortening the Receipt, Referral, and Review Process was charged with outlining the changes in process and the resources required to: (1) allow unsuccessful applicants to submit an amended application for the next receipt date and (2) shorten the overall time from receipt to award from 10 to 5 months. The group outlined two alternative schedules for consideration. While the goals are laudable, inevitably, the devil will be in the details of implementation. Changes to the current process will have significant implications for all stakeholders in the process and will involve overcoming many obstacles. Thus, changes will be considered only after a careful cost/benefit analysis is conducted based on broad input from Institute/Center staff (review, program, grants management, and budget) and members of the extramural research community (researchers and research administrators).
Beginning with the June study section meetings (i.e., applications reviewed for September/October 1998 Councils), NIH peer review groups recalibrated their scoring using 3.0 as the target median score. The purpose was to spread scores over a wider range of the priority score scale. This measure was taken to decrease inconsistent scoring practices among different review groups, and to improve the capability of the review group to discriminate scientific merit among applications being reviewed. As a result of this recalibration, priority scores for applications reviewed during the June cycle of meetings may not be comparable to those reviewed in the past. Furthermore, in order to prevent influence from priority scores given in prior rounds, percentiles for applications reviewed during the June round are based ONLY on scores assigned this round. Examination of the data reveals that this measure did indeed result in a greater spreading of scores.
COMMITMENT TO NEW INVESTIGATORS REMAINS FIRM
To allow new investigators maximum freedom in identifying the level and period of support needed to sustain their research programs, and thus to enhance their opportunities to establish careers in research, NIH has announced a new policy. Effective June 1998, the NIH no longer accepts First Independent Research and Transition (FIRST; R29) award applications. Rather, all newly independent investigators will use the R01 mechanism. Applications from new investigators will continue to be identified as such, and reviewers will be told to adjust review criteria accordingly. Specifically, they will be instructed to expect less preliminary data and "track record."
NIH originally set up the FIRST award to assist newly independent investigators to initiate their own research and demonstrate its merit. It provided five years of limited research support during or after which investigators could apply for traditional types of NIH research project grants, particularly R01s. Since 1986, although a significant number of investigators have applied for R29 support, most new investigators applied for R01 funding. R29 applicants have had a somewhat better success rate than did new applicants for R01s. However, when subsequently applying for renewals via R01 funding, applicants who received R29 funding as their initial method of support are less successful than new applicants who received an R01. NIH's new policy was adopted after an analysis by the "Working Group on New Investigators." The Working Group concluded that R29 applicants were penalized in particular by the dollar limitation of the R29-$350,000 over a five-year period, with no single year exceeding $100,000. By having all new investigators meet the same R01 requirements, NIH will eliminate this difference as well as the use of different review criteria and the requirement for letters of support.
In making this change, NIH is committed to supporting at least the same number of new investigators and, as necessary, directing more resources to their support. In FY 1997, NIH supported 1,466 new investigators with R01 or R29 awards.
ERRATUM: INCLUSION OF CHILDREN IN RESEARCH
The last issue of Peer Review Notes erroneously reported that applications submitted to NIH for and after October 1, 1998, would be governed by new guidelines for including children in research involving human subjects. The policy will pertain to applications submitted AFTER October 1. Thus, this requirement will apply first to new unsolicited interactive research project grant, NRSA fellowship, and AIDS-related applications received, respectively, for the receipt dates of October 15, and December 5, 1998, and January 2, 1999, which will be reviewed in February-March 1999 for the May-June Council meetings. It will apply to new, unsolicited AREA, research project, research career, program project, and center applications beginning with the receipt dates of January 25 - February 1, 1999, which will be reviewed in the June-July 1999 review meetings for the September -October Council meetings. The date by which the requirement will apply to applications submitted in response to RFAs will be as announced in the RFA.
Presented below are the numbers of competing grant applications reviewed by NIH scientific review groups for the October 1994 - 1998 national advisory councils and board meeting cycles. These statistics, which represent applications reviewed by scientific review groups primarily in June and July, were extracted from the NIH IMPAC database.
| October Council Cycles | ||||||||
| 1994 | 1995 | 1996 | 1997 | 1998 | ||||
| Applications Reviewed | 13659 | 13288 | 12089 | 12660 | 12691 | |||
| CSR | 8862 | 9145 | 8574 | 8475 | 9716 | |||
| Percent CSR of Total | 64.9 | 68.8 | 70.9 | 66.9 | 76.6 | |||
| Institute/Center | 4797 | 4143 | 3515 | 4185 | 2975 | |||
| Percent IC of Total | 35.1 | 31.2 | 29.1 | 33.1 | 23.4 | |||
| Research Grants | 12218 | 11676 | 10628 | 11297 | 11591 | |||
| Research Projects | 9543 | 9226 | 8496 | 8853 | 9078 | |||
| SBIR/STTR | 1323 | 1379 | 1182 | 1188 | 1324 | |||
| Research Centers | 226 | 193 | 139 | 116 | 162 | |||
| Other Research | 1126 | 878 | 811 | 1140 | 1027 | |||
| Training Applications | 1319 | 1449 | 1339 | 1255 | 1047 | |||
| Fellowships | 1090 | 1264 | 1170 | 1141 | 943 | |||
| Training Grants | 229 | 185 | 169 | 114 | 104 | |||
| Other Applications | 122 | 163 | 122 | 108 | 53 | |||
| Applications Amended | 3634 | 3924 | 3555 | 2958 | 2928 | |||
| Percent of Total Reviewed | 26.6 | 29.5 | 29.4 | 23.3 | 23.1 | |||
| Applications Responding to RFA's | 1798 | 1322 | 1163 | 1246 | 1125 | |||
| Percent RFAs of Total Reviewed | 13.2 | 9.9 | 9.6 | 9.8 | 8.9 | |||
* Research Grants includes Research Projects, SBIR/STTR, Research Centers, and Other Research.