PEER REVIEW NOTES
June 1999
| FROM THE CSR DIRECTORS DESK | 1 |
| ASSESSMENT OF FELLOWSHIP REVIEW PROCEDURES | 2 |
| THE ROLE OF PROGRAM STAFF DURING THE STUDY SECTION MEETING | 3 |
| READY, SET, GO! MODULAR GRANT APPLICATIONS ARE HERE | 4 |
| HELP EXPEDITE PROCESSING OF APPLICATIONS REQUESTING $500,000 AND OVER! | 4 |
FROM THE DIRECTORS DESK
Publication of Peer Review Notes each round follows quickly on the heels of the thrice-yearly meetings of the Center for Scientific Review (CSR) Advisory Committee. The Advisory Committee plays a key role in providing diverse, external input for development of CSR policies, and their meetings provide a forum in which progress with respect to the many initiatives underway is summarized. Complete minutes of the May meeting will be posted on the CSR web site as soon as they are available. In the meantime, Ill use this column to update you regarding a couple of noteworthy activities. The Panel on Scientific Boundaries for Review (see http://www.csr.nih.gov/events/boundaries.htm for roster) is making steady progress in undertaking a comprehensive review of the scientific organization and function of the review structure of the CSR. The members are conducting their evaluation in two parts.
In Phase I, they are defining a new series of broad and overlapping scientific domains (integrated review groups, IRGS), each of which is intended to provide oversight for clusters of scientifically related study sections. In addition, they are developing recommendations concerning the policies and standards that should govern the conduct of the review process, as well as the principles that should be used in Phase 2, to design the study sections comprising each IRG. Phase 2 will begin in the fall of 1999, and will be implemented during the next few years, by expert groups consisting of both external scientists and NIH staff.
In developing their recommendations, the Panel is relying heavily on input from all stakeholders. To this end, a draft of the Phase 1 report will be posted in June 1999 for public comment by the broad scientific community. The Panel expects to release their Phase 1 report in the fall of 1999.
A similar reliance on public input was utilized by the Working Group on Review of Bioengineering and Technology and Instrumentation Development Research in developing their recommendations. The Working Group has fulfilled its charge to determine impediments to and outline principles for achieving fair, high-quality and rigorous review of technology-related applications. Though the mandate of the Working Group was relatively narrow, the members believed the issues to be broad. Consequently, their report also speaks to ways in which NIH might generally facilitate interdisciplinary research and become more agile at seizing emerging opportunities. The report and roster are available at http://www.csr.nih.gov/bioopp/select.htm.
The Advisory Committee was also informed of the work of Dr. Maxine Linial, who recently completed a study of the review of individual fellowship applications at CSR. Dr. Linials findings and recommendations are summarized below. I draw your attention to that piece as well as to the other articles included in this edition of the Notes: on (1) the role of Institute and Center (referred to for simplicity as Institute) staff; (2) the institution of modular grant practices; and (3) expediting the processing of applications that request $500,000 or more for a single budget year.
As always, I wish to thank all those who have contributed to these valuable activities and to the peer review process. A special thanks to the Peer Review Notes Advisory Committee and authors of this issues articles for their contributions. Have a terrific summer, everyone. See you in the Fall.
E
.
ASSESSMENT OF FELLOWSHIP REVIEW PROCEDURES
Maxine Linial, Ph.D. and Linda Engel
CSR reviews approximately 2500 individual postdoctoral training (F32) applications each year, as well as smaller numbers of predoctoral fellowship (F31) and senior fellowship applications (F33). The goal of the review process is to identify those candidates who have the highest potential to develop into successful, independent scientists upon the completion of their training, based on an evaluation of: the candidate's potential for a productive career, the candidate's need for the proposed training, and the degree to which the research training proposal, as well as the sponsor and the environment satisfy those needs.
CSR policies and practices regarding review of fellowship applications have varied through the years. From 1979 to 1994, they were reviewed in dedicated panels. In 1994, as part of streamlining efforts in CSR (formerly DRG, the Division of Research Grants), fellowship applications were dispersed by research topic to study sections that review corresponding investigator-initiated (R01) applications. In 1997, it was recommended that Scientific Review Administrators (SRAs) try to cluster review of fellowship applications in each IRG (integrated review group, a group of scientifically related study sections) to create a critical mass of at least 10 applications.
Prompted by the CSR Advisory Committee, Dr. Maxine Linial, Investigator Member, Division of Basic Sciences, Fred Hutchinson Cancer Research Center, was engaged to evaluate current practices and to recommend optimal ways to maximize the potential of the review process to identify the most promising trainees. Dr. Linial spent January through March 1999 in residence at CSR. During this time, she met with IC and CSR staff, attended 21 study section meetings, and examined archived data.
Her analyses revealed that in February 1999, from 1 to 91 fellowships were reviewed in each of 83 scientific review groups: two dedicated fellowship panels, chartered study sections that review primarily R01 applications, and special emphasis panels (SEPs) whose charge is to review primarily research grants, or small numbers of fellowships. Based on her evaluation of the way in which these review groups functioned, Dr. Linial distilled a list of what she believes are best practices: (1) Applications are clustered in groups of 15 or more to create critical mass; (2) SRAs provide an orientation to set the tone and materials to guide reviewers in evaluating and scoring applications; (3) Chairs exhibit proactive leadership to ensure that applications are reviewed for their training potential; (4) members correctly balance application of review criteria and spread their scores; and (5) program staff are in attendance to hear the discussion. She also noted that in the absence of percentiling, Institutes have difficulty interpreting the scores that vary greatly among study sections and are frequently on a different scale than those assigned to R01 applications.
Based on her analyses, Dr. Linial developed the following six recommendations:
1) Review fellowship applications in dedicated study sections. This venue most readily facilitates uniformity and the application of best practices. In addition, representation of a broader range of disciplines on these panels would help prevent inappropriate focus on experimental design and erase discipline-specific cultural differences in defining ideal training. More consistent scoring patterns across study sections would also be facilitated, and fellowships could be grouped in ways that transcend IRGs. Importantly, clustering fellowships in dedicated study sections would enable percentiling (see recommendation 5 below), and program staff would have fewer study sections to attend to monitor the process. (See "The Role of Program Staff at the Study Section Meeting" below.)
There is strong support from many CSR and Institute staff that broad-based, dedicated panels would provide an optimal review setting. However, many staff involved with the behavioral and social sciences feel that fellowships should be reviewed "where the science is," because they believe cultural variation makes it more difficult for reviewers to cross disciplines. Thus, further assessment is in order before considering assignment of such applications to dedicated fellowship panels.
2) Train SRAs, Chairs, and reviewers about the special issues surrounding fellowship reviews.
3) Modify the application form and instructions. The page limitation for the research plan should be reduced to preclude inclusion of detail that invites reviewers to prepare point-by-point critiques of the experimental plan. The instructions should further require that mentors provide precise information regarding their training records, and that mentors and applicants state their relative intellectual contribution to the application, rather than the writing component.
4) Make dual referrals. While some Institutes have more meritorious applications than dollars, for others the situation is reversed. Making more dual assignments would call meritorious applications that might otherwise go unsupported to the attention of Institutes willing and able to fund them.
5) Percentile fellowship applications. Scoring practices among study sections vary greatly and often are not aligned with scores assigned to research grant applications. For example, among the 21 study sections Dr. Linial observed, the difference between the highest and lowest median scores was 128 points. Percentiling fellowship applications, as is done for research grant applications, would allow program staff to more reasonably compare applications reviewed in different study sections and facilitate making their funding decisions.
6) Appoint an individual to provide ongoing oversight of fellowship review across CSR. Such a "training administrator" could continuously monitor all matters related to review of fellowships.
In summary, both qualitative and quantitative findings convinced Dr. Linial that changes to the system would optimize review of fellowship applications in CSR. She also recommended that sustained monitoring of the process along with continued input of NIH staff are vital, to ensure that the system continues to be fair and rigorous in identifying candidates who have the highest potential to develop into successful, independent scientists.
THE ROLE OF PROGRAM STAFF DURING THE STUDY SECTION MEETING
National Human Genome Research Institute Program Staff
Reviewers may wonder why certain NIH people are sitting around the perimeter of the study section meeting room. They are "program people," health scientist administrators, often known as program directors or program administrators, who work in the NIH Institutes and Centers. Their primary role is to assist the Institutes in ensuring that the best science is supported. They do this through a variety of activities, one of which is keeping track of the applications pertaining to their respective Institutes as they make their way through the peer review process. In so doing, they find it extremely helpful, for several reasons, to attend review meetings.
First, the discussions during the meeting are more detailed than what is generally captured in a summary statement. Hearing the full discussion is useful during subsequent discussions with applicants about the review. Applicants often want to know why their applications fared as they did and what they might do to improve their applications. In rare cases, program staff has to deliver the news that an application should not be revised and resubmitted. In such cases, it is helpful to have heard the extended discussions that took place during the meeting. The discussions also provide them with a useful perspective for responding to applicants who express concerns about whether the review process was objective and fair. A second major value of attending review meetings is that the scientific discussions help program staff keep up with what is new and at the cutting edge in the areas of science that they are responsible for overseeing. Third, since program staff is knowledgeable about their Institute's policies and program priorities, they may be able to provide reviewers with useful information about or clarification of such policies or priorities. Although such information should never influence the scientific evaluation of specific applications, it may improve the reviewers understanding of the context and rationale for the Institutes programmatic initiatives.
Study sections often review applications that are assigned to several Institutes for potential funding. Therefore, some program staff will be in the room from the beginning to the end of the meeting, while others will pop in and out to listen to the review of a few applications.
The responsibilities of program staff also entail working with research communities: to identify critical scientific areas needing research support; to develop research initiatives for soliciting applications to address those areas (e.g., with program announcements or requests for applications); and to advise prospective applicants about their Institutes program priorities.
After the initial scientific review, program staff: counsel applicants about how to deal with the outcome of the initial scientific review of their applications; handle appeal letters, which are always discussed with Scientific Review Administrators prior to making a decision; present the applications to the Institutes national advisory councils and boards; and participate in the decision-making process about which grants will be awarded.
Following the award of grants, they monitor the progress of funded research projects, report scientific progress, and represent their respective programs to the scientific community and the general public. Because many of the Institutes have shared research interests, program staff also coordinates activities with those of the other Institutes and sometimes participates in joint research initiatives.
Program staff are an integral part of the NIH team of reviewers and administrators, whose recommendations are so critical in helping the NIH select the most meritorious or promising research to be funded.
READY, SET, GO! MODULAR GRANT APPLICATIONS ARE HERE.
Janet Cuca, Ph.D.
"Modular grants" are not a new type of NIH grant mechanism. The phrase refers to simplified application procedures that are now to be used for most existing NIH grant mechanisms, including the R01, where $250,000 or less per year is being requested. The new application procedures feature budget requests made in modules of $25,000 and the absence of information regarding the budget categories that NIH used to use (i.e., Personnel, Consultants, Equipment, Supplies, etc.). While information about Personnel (their roles and percents of effort on the project [and, of course, their qualifications/Biosketch]) is still required, salary information is no longer to be provided. Similarly, dollar amounts for Travel, Supplies, and the other old budget categories are not to be provided, although an explanation is to given if the item is unusual or is unusually expensive. Thus, in the budgets of most applications, the only dollars to be shown (and they are to be in modular amounts) are the total direct costs requested for each year of the project and for the entire project. In fact, the old Budget pages (DD and EE) are not to be used and applicants are simply to use a blank sheet of paper to justify their request. The new application procedures have ramifications for the peer review of applications and for award procedures.
While a few of the review units in the NIH Institutes and Centers have over the past several years reviewed modular grant applications (most of which were submitted in response to RFAs), the beginning of the first NIH-wide initial scientific review of unsolicited modular grant applications will occur this fall. Modular procedures have been required for applications submitted starting in the spring of 1999 (the precise implementation dates depend upon the type of grant mechanism). The modular grant initiative expands the efforts that are designed to focus reviewers attention on the science being proposed rather than on budget details.
Success in implementing this important change depends critically upon reviewers fulfilling their role in this important change. A two-page discussion of modular grants designed for reviewers is being provided to all reviewers to help accomplish this goal. In addition, answers to "Frequently Asked Questions" about modular procedures are available on the Modular Grants Web site at http://www.nih.gov/grants/funding/modular/modular.htm
HELP EXPEDITE PROCESSING OF APPLICATIONS REQUESTING $500,000 AND OVER!
Janet Newburgh, Ph.D.
NIH requires that all investigator-initiated grant applications requesting $500,000 or more in direct costs for any budget year must be accepted by an Institute or Center (hereafter called an Institute) before peer review. In the latest version of this policy, which was published in the NIH Guide for Grants and Contracts of March, 20, 1998, the following applications are included: investigator-initiated new (type 1), competing continuation (type 2), competing supplements (type 3), and amendments or revisions. This policy also includes all applications submitted in response to Program Announcements (PAs) unless the announcement indicates otherwise. Only in special cases that are so indicated does the policy apply to applications submitted in response to Requests for Applications (RFAs) or for Small Business (SBIR and STTR) grants.
If you are planning to request $500,000 or more in direct costs for any budget year, you should first contact Institute program staff to discuss the anticipated budget and to have the Institute agree to accept the application when it is submitted. If such agreement is not obtained before submission of the application, the review of your application may be delayed, or application may even be returned to you. Note, however, that acceptance for review does not represent an Institutes commitment to eventually fund the project. If you obtain such an acceptance, you should include a cover letter identifying the Institute and program staff member who agreed to accept the application with that application.
Further information regarding this policy can be obtained through the NIH web site at
http://www.nih.gov/grants/policy/notices.htm.