![[Return to Top]](../images/uparr.gif){kind=small}
Return to Top October 1996
(Short Notes)
But alas, it was several days after publication of the June issue of PRN that her selection could be announced publicly. Many of you know Dr. Ehrenfeld and some of you may have served with her in initial review or other advisory capacities. Her experiences with NIH, plus those gained as an applicant, grantee, and dean will, together with her enormous talent, be important as she begins the next phase of a distinguished career. Ellie, my very best wishes for success in leading this remarkable organization.
My plans are to leave the DRG in October, having accepted an invitation from Dr. James Snow to become the Deputy Director of the National Institute on Deafness and Other Communication Disorders. Given this new opportunity, it does not seem self-serving to talk about the DRG family which adopted me nearly 10 years ago. Many of you know and interact closely with one or a few of our talented Scientific Review Administrators and Grants Technical Assistants, and perhaps less often with the dedicated people in offices supporting travel, information, advanced technology development and administration. With the possible exception of our DRG Advisory Committee, no one from the extramural community has had the opportunity afforded to me and our senior staff of experiencing the dynamic potential of this organization, whose sum far exceeds its parts.
To all our staff, thank you very much for your resilience, dedication, and willingness to meet challenges and to accept change. Maintaining, and even improving, the quality of peer review in the face of relocation and major reorganization would have tested the mettle of lesser organizations. But you did all of that, and in addition, we have begun the important and open process of evaluating the quality of our product and our ability to more readily accommodate to changes in science. The importance of the DRG Advisory Committee in this ongoing process of evaluation cannot be overemphasized, and I am grateful that Dr. Keith Yamamoto has accepted the invitation to serve as Chair. Of course, none of this would be possible without the willingness of dedicated scientists to participate actively in the peer review process.
For all of the above and more, including a wonderful 50th birthday in June, Thank you!
-Don Luecke
At the University of California, Dr. Ehrenfeld has been Dean of the School of Biological Sciences and Professor of Molecular Biology and Biochemistry since 1992. As Dr. Ehrenfeld has noted, a major focus of her administration has been the establishment of broad, campus-wide programs that crossed formerly rigid School lines. Her emphasis has been on identifying common goals and working together for campus strength, rather than acting as the leader and advocate for only her School.
Dr. Ehrenfeld received the Ph.D. degree from the University of Florida. Prior to her position at the University of California, Dr. Ehrenfeld was Associate Professor of Cell Biology at the Albert Einstein College of Medicine and then Professor of Biochemistry and Cellular, Viral, and Molecular Biology as well as Director of the Inter-departmental Graduate Program in Molecular Biology at the University of Utah College of Medicine. At Utah, she received an Outstanding Professor Award.
In addition to her administrative experience, Dr. Ehrenfeld has been a successful NIH grantee and an active participant in peer review. She has been a continuous grantee of the National Institute of Allergy and Infectious Diseases (NIAID) for the past 25 years; she currently holds two NIH research grants, including a MERIT Award, an honor awarded to outstanding investigators, for her research program on the molecular biology and biochemistry of genome replication of single-stranded RNA viruses (poliovirus and hepatitis A virus).
As a peer reviewer for the NIH, Dr. Ehrenfeld has served on the Experimental Virology Study Section (DRG), the Microbiology Training Committee (National Institute of General Medical Sciences or NIGMS), and the Genetic Basis of Disease Review Committee (NIGMS), which she chaired from 1988 to 1990. She has performed intramural peer review as a member of the NIAID Board of Scientific Counselors, and is currently a member of the National Advisory General Medical Sciences Council and the Peer Review Oversight Group (PROG). Dr. Ehrenfeld also has had review experience with the National Science Foundation, the Walter Reed Army Institute of Research, and the Food and Drug Administration. She has been a member of the editorial board for Virology, the Journal of Virology, and the Journal of Biological Chemistry.
In accepting the Directorship of DRG, Dr. Ehrenfeld affirmed her confidence in and staunch support of the peer review system, as well as her belief that the Director of DRG can potentially make a significant difference in the performance, morale, and future of the Nation's biomedical research enterprise. Peer Review Notes joins the rest of DRG in welcoming our new Director.
Dr. Baldwin, DDER, emphasized that scientific peer review at NIH is not a system in chaos; rather it is a model for peer review systems worldwide. In a time of reinvention and self-examination, NIH is seeking to make a good system better.
During the first day of the meeting, the group discussed possible topics and procedures. The need for peer review to be dynamic and adaptable and to adjust to changes in science was noted as a general area in which PROG could contribute. Topics mentioned included how scientific progress might be helped or hindered by the separation of review and program at NIH; the quality of review, and how that can be measured; similarities or differences between DRG and Institute and Center (IC) review; how science maps to specific review groups; expertise within study sections (breadth vs. depth); how to continue to review those applications currently submitted, but simultaneously create a nurturing environment for the next wave of science; how to manage review in low-volume areas of science; identification of high-risk research (including how to define it); how to combat the innate conservatism in science and in study sections (or how to collect data to determine whether it exists); and how reviewers deal with non-scientific issues. Some issues will inevitably be concerns of both PROG and the DRG Advisory Committee: Scientific Review Administrator (SRA) training; selection and supervision of SRAs; the roles of review and program staff; the reviewer selection and approval process; reviewer training/ retraining; travel to scientific meetings for SRAs; balance between new and senior reviewers on study sections; communication among study sections; and the role and training of study section chairpersons.
PROG's discussion of the Rating of Grant Applications Report focused predominantly on those recommendations having to do with review criteria and whether to assign a global score. Further discussion is planned for the next meeting with the expectation that the PROG will make recommendations at that time. (See the accompanying article on this topic for details.)
PROG also discussed the efforts toward integration of peer review at the National Institutes on Alcohol Abuse and Alcoholism (NIAAA), on Drug Abuse (NIDA), and of Mental Health (NIMH) with DRG. The group heard a brief presentation of the successful integration of many of the NIAAA review groups with the DRG. (See the June 1996 NIH Peer Review Notes for details.) Dr. Baldwin pointed out that the integration of the NIAAA grant application review with that of the DRG may serve as a model for NIDA and NIMH, although it is not necessarily the only way that integration can be accomplished.
Because of the large number of applications, it was suggested that perhaps the next step in the NIDA and NIMH review integration needs to be within specific scientific areas. Biopsychology and basic neuroscience might be reasonable areas to consider. It was suggested, as a starting point, that the NIAAA/DRG integration team share their experiences with the ICs involved in basic neuroscience and biopsychology, and that PROG members in these scientific areas could serve as consultants.
Wendy Baldwin, Ph.D.(Chair)
Deputy Director for Extramural Research
National Institutes of Health
Thomas J. Braciale, M.D., Ph.D.
Director, Beirne B. Carter Center
University of Virginia Health Sciences Center
David M. Center, M.D.
Professor, Department of Medicine
School of Medicine
Boston University
Elvera R. Ehrenfeld, Ph.D.
Dean
School of Biological Sciences
University of California, Irvine
Mary Jeanne Kreek, M.D.
Professor and Senior Physician
Laboratory of the Biology of Addictive Diseases
The Rockefeller University
David J. Kupfer, M.D.
Professor and Chairman
Department of Psychiatry
Western Psychiatric Institute & Clinic
University of Pittsburgh
Claude Lenfant, M.D.
Director
National Heart, Lung, and Blood Institute
National Institutes of Health
Alan I. Leshner, Ph.D.
Director
National Institute on Drug Abuse
National Institutes of Health
Donald Luecke, M.D.
Acting Director
Division of Research Grants
National Institutes of Health
Cora B. Marrett, Ph.D.
Assistant Director for Social, Behavioral
and Economic Sciences
National Science Foundation
Candace McCombs, Ph.D.
Professor of Medicine and
Chief, Division of Experimental Medicine
Department of Internal Medicine
University of South Alabama
John J. McGowan, Ph.D.
Director, Division of Extramural Activities
National Institute of Allergy and Infectious Diseases
National Institutes of Health
Howard K. Schachman, Ph.D.
Professor in the Graduate School
Department of Molecular and Cell Biology
University of California, Berkeley
Keith R.Yamamoto, Ph.D.
Professor and Chairman
Department of Cellular and Molecular Pharmacology
University of California, San Francisco
Because changes to so critical an element of peer review should not be implemented without the participation and contributions of the extramural staff at NIH and the scientific community that will be affected, several measures were undertaken to obtain their comments and recommendations. An overview of the RGA report was placed on the NIH Home Page (http://www.nih.gov/grants/rga.htm), with a link to download the full report. Comments are being accepted through October 1, 1996, at DDER@NIH.GOV.
The RGA report was scrutinized within the NIH. The full report was sent to the directors of the Institutes, Centers, and Divisions, and was an item of discussion for all of the relevant NIH-wide standing committees: the Extramural Program Management Committee (EPMC), the Review Policy Committee (RPC), and the Program Officers and Project Officers Forum (POPOF). In addition, four internal study groups were established to design pilot studies and present the pros and cons of the specific recommendations.
In addition to making the report available and soliciting comments electronically by e-mail, members of the RGA Subcommittee and Dr. Wendy Baldwin, Deputy Director for Extramural Research (DDER), facilitated several discussions with more than 100 study section members during their June review meetings. Dr. Baldwin also established a "Virtual Committee" of approximately 18 scientists, combining teleconferences, e-mail, and a world wide web bulletin board to obtain their comments and suggestions.
The results of all of these efforts were presented to the Peer Review Oversight Group (PROG), a committee established to advise NIH Director Dr. Harold Varmus on various policies and issues related to peer review across the NIH. At PROG's first meeting in July 1996, Dr. Baldwin summarized the feedback on the RGA report. She indicated that some recommendations appear to be non-controversial, such as having higher scores as better scores. Also, there was an emerging consensus against standardization by reviewer. There seemed to be a general acceptance of using specific review criteria, but there might be a need to add one, and there was enthusiasm expressed for a reviewer-assigned global score.
The use of criteria and the idea of a global score were discussed thoroughly by the PROG. There was enthusiasm for using four explicit review criteria, although there were differences of opinion about the exact wording for labeling and defining these and whether to use adjectival descriptors or letter grades to rate them, if individual rating of criteria were adopted. There also was enthusiasm for having reviewers assign a global score. No decision was made on the number of points on the rating scale, and apparently there was little if any opposition to a reversal of the scale (with the higher number representing the better score). The issue of standardizing scores by reviewer was tabled.
The committee will be polled regarding possible pilots to be performed, and Dr. Baldwin noted that some pilots may be underway before they meet again as a group. At the next meeting, PROG will offer advice on the RGA recommendations and on what would be an optimal "change package." This advice will be communicated by Dr. Baldwin to Dr. Varmus, and a decision will be made in January 1997, so that any changes can be fully implemented during the June-July 1997 review cycle for the October 1997 council round; thus changes would be reflected at the beginning of the new fiscal year.
A major assessment of the entire submission-to-award process has recently been undertaken by an NIH reinvention committee termed STRAP (Streamlined Review to Award Process). While a final report from this committee is still in preparation, a number of pilot studies have already been initiated as a result of recommendations arising at STRAP and elsewhere:
2. Expedited amendment process. Why is it necessary for the applicant to prepare another complete application when submitting an amendment? This generally requires such significant effort on the part of the applicant, involving clearances at various levels throughout the applicant organization, that submission of the amended application is delayed at least one review round. This concern is particularly heightened when reviewers' criticisms of an application revolve around a very small part of the application. Consequently, a pilot was initiated in four DRG study sections (Aids and Related Research 2, Cellular Biology and Physiology 1, Molecular Cytology, and Experimental Therapeutics 2) to have reviewers identify any applications for which a clarification from the applicant of 3-5 pages in length could answer reviewers' very specific concerns and thereby allow for "completion" of review. In these instances, reviewers still felt comfortable in providing a priority score, so that the funding component could potentially make an award. In addition, the applicant was notified by the SRA of the option of responding to the critique in a brief correspondence, countersigned by the institution's business official, to be reviewed, as an amended application, at the very next study section meeting. The number of applications so identified at the June round of study section meetings ranged from 0 to 8 per study section. How many applicants will avail themselves of this option and how well they will fare in the review process are yet to be determined.
(Note: Somewhat analogous to this process is a recent effort by the National Cancer Institute to identify applications for which the applicant is invited to submit a brief response to the summary statement, to be considered by a programmatic executive panel for award. In these cases, the response from the applicant is not considered an amended application, but simply additional information for staff; there is no further peer review.)
3. Facilitating pre-review activities electronically. A pilot study is in the planning stages to provide for study section members to submit their critiques electronically prior to the meeting to a client-server controlled by the SRA. Through this process, reviewers will have access to each others' reviews prior to the meeting, so that they can comment on the reviews, and thus enable assigned reviewers to refine and complete their critiques prior to the meeting. Potential benefits are more precise determination of the "lower half" (i.e., those applications that will not receive discussion and a score at the meeting) and better congruence of critiques, especially for an application that receives no discussion.
Other potential pilot studies are under discussion, many in the venues of Institute/Center review branches. Very often, particularly in the "controlled environment" of RFA (Request for Applications) reviews, innovative concepts can be tested more easily. Thus, for example, the concept of Just-in-Time submission of application materials was developed. A current example of a proposed pilot within the RFA review process is the use of phone/e-mail/fax to applicants during the review meeting for purposes of clarification. In addition, parallel efforts are ongoing to facilitate and streamline Council functions. For example, many Institutes/Centers are now providing their Advisory Council members with electronic access to summary statements. (See the NIAID Accelerated Council Review article in this issue.)
The focus of fellowship review is different from R01 review. The F32 program provides support for training in research rather than direct support of research. In assessing the merit of F32 applications, reviewers evaluate four major components: the candidate, the project, the training potential of the project, and the training resources and environment. All four components are important, and a weakness in one could significantly weaken the entire application.
A major emphasis of the review is on the potential and commitment of the candidate for a research career. In making this determination, the reviewers rely primarily on the candidate's academic background, research experience to date, and letters of recommendation. For clinician applicants, consideration is given also to the fact that they may not have research experience at this point in their career comparable to Ph.D. candidates. For clinician applicants, the commitment to an academic or research career takes on added importance.
While the project outlined in the application should have scientific merit, it should be evaluated more in light of its training potential than its potential to make major scientific contributions, which is critical for R01s. In other words would this project, if funded, bring the candidate closer to being an independent investigator? The research training should be a logical progression in the applicant's career rather than a repeat of past experiences.
The final consideration is the resources and environment. As with other funding mechanisms, the physical resources need to be in place to complete the project. However, since this is a training vehicle, there are other considerations. For example, given the candidate's training goals, does the sponsor have the appropriate scientific background to guide this project? Is the sponsor committed to training this candidate? Is the sponsor's laboratory an environment where the candidate will be challenged by new ideas and approaches, and thus acquire the experience necessary for an academic career?
R01 study sections have been successful in understanding the necessity of reviewing F32 applications differently than R01 applications. Even so, to assure that F32 applications are given the best possible review, periodic evaluations of the process will occur. One is currently underway by a working group of the NIH Training Advisory Committee.
Out-of-town reviewers need a travel order in-hand and tickets provided by the study section. The traveler must inform Ober when making reservations that a Federal travel order is being issued. A voucher expense form (provided by the study section) is submitted after travel is complete; all expenses, except a taxi fare up to $75, require receipts.
Local reviewers (Baltimore is considered local for meetings in the Washington metropolitan area) are allowed transportation costs according to the Federal Travel Guidelines. Receipts are submitted on a local travel voucher expense form provided by the study section.
Ober United Travel has recently created a "FastRes" Fax Form, provided by the study section, that the reviewer can use to fax to Ober the flight requirements. Travelers then have a hard copy record of their request, and the time and date when the request was made. This Fax Form should decrease some of the 800 telephone number traffic and should ensure completion of travel reservations in a more timely manner.
Note: DRG pays for study section meeting transportation costs up to the Ober government rate based on the reviewers' home station. Exceptions require special authorization by Caroline Grabner, Chief, Scientific Review Evaluation and Award Office, (301)435-1127.
Recommendation 1. NIH should revive its previous practice of making grants-in-aid to investigators and make explicit arrangements for cofunding of research projects, when appropriate. Using this approach, in contrast to the practice of paying the full costs of research, NIH would be contributing resources toward a program of research that may also be partially supported by one or more other sources. Program staff are able to do this now, without formal action on the part of NIH. However, specific encouragement to do this and public announcements of NIH's intent to do more cofunding are still under study. A proposal to provide supplementary awards to non-NIH funded research projects was discussed and tabled. The committee overseeing the implementation of the Just-in-Time concept is currently studying cofunding in conjunction with other reinvention activities.
Recommendation 2. NIH staff should ensure that the priority score, as voted during initial peer review reflects scientific merit, not extraneous factors. Dr. Baldwin has visited several study sections that review applications from HHMI investigators to discuss her concerns, which are summarized in this recommendation. Particularly in times of constrained resources, it is important that initial peer reviewers, National Advisory Council members, and NIH program staff fulfill their appropriate roles and exhibit mutual trust. In addition, a number of small steps, including greater use of administrative notes concerning other support and feedback from program staff to review groups regarding actions taken as a result of such notes, are being facilitated to encourage appropriate review considerations and procedures and to increase mutual trust.
Recommendation 3. NIH should allow HHMI investigators to participate in the initial peer review of applications from other HHMI investigators, unless they have a conflict of interest arising from circumstances other than their HHMI appointment. This change was implemented in the autumn of 1995.
Recommendation 4. NIH staff should develop guidance describing the range of options for pre-award and post-award management of projects where the investigator has considerable other research support. No action has been taken on this recommendation.
Recommendation 5. The Director, NIH, should develop a Memorandum of Understanding with organizations that provide significant research support to the same investigators as NIH. The intent of this recommendation is to improve ongoing communications between the NIH and organizations such as HHMI. With the encouragement of NIH's Director, Dr. Harold Varmus, NIH and HHMI staff have engaged in a much more active effort to improve communications. Early in 1996, Dr. Baldwin and small groups of grants management, program, and review staff who had substantial contact with HHMI investigators met at HHMI to become acquainted with HHMI staff and to discuss mutual expectations. This activity was expanded into four, small-group workshops involving NIH and HHMI staff. Three sessions have been held already, in July, August, and September; one more session is scheduled for October 10th. These sessions should increase understanding and allow the two organizations to work together more smoothly as partners in support of biomedical research. NIH staff are encouraged to consult the HHMI home page (http://www.hhmi.org/) as one source of current information regarding HHMI scientists.
This year, NIAID improved the system even more by putting summary statements on a server where members of the NAAIDC can read them on their own computer monitors. To protect the privacy of applicants, the server is accessible only by NAAIDC members. About eight weeks before a meeting of the NAAIDC, members view the list of applications with percentile scores within the payline. To see a summary statement of the review of the application, they click on a name on the list. NAAIDC subcommittee members enter their comments in a comment field. These comments can be gathered into reports of all comments, by grant, or by NAAIDC member. After the NAAIDC subcommittee has finished their review, NIAID staff examine the applications to make sure there are no problems with overlap, human subjects, budget, or other matters.
In addition to saving paper and postage, review before the NAAIDC meeting reduces the amount of time it takes NIAID to make certain types of awards, especially new R01 and R29 grants. It does not affect NAAIDC's primary roles to provide policy advice, to review grant applications with special concerns, to review applications from foreign organizations, to discuss selective pay, to select MERIT awardees, to respond to rebuttal letters, and to clear concepts for research initiatives. Eliminating the wait for a NAAIDC meeting also greatly increases the efficiency of NIAID staff, who no longer have to issue all of the awards immediately after each meeting of the NAAIDC.
"The Foundation provides an opportunity for private citizens, private sector institutions, and foundations to enhance the current public investment in biomedical research," said Dr. Paul Berg, Acting Chair of the NFBR Board. "Recent public opinion polls indicate that most Americans view medical research as vital, and that they support more spending, by the public and private sector, for this kind of research. The NFBR provides a mechanism for generating additional funds and for assuring that the money is well spent," he said.
"We are very pleased that the Foundation is now up and running," said Dr. Harold Varmus, NIH Director, "Public support of medical research has been generous, but the Foundation will allow us to undertake some important projects and activities that we are not currently funding. We could, for example, enhance our research training activities, conduct some important public education programs, foster collaborations with academic institutions and industry, and improve the environment for conducting research on the NIH campus."
The NFBR operates under the guidance and supervision of a Board of Directors comprised of prominent individuals. Currently there are nine Board members. Dr. George J. Galasso, who recently retired as the NIH Associate Director for Extramural Affairs, is serving as the Executive Director of the Foundation. At its upcoming Board meeting, the members will discuss the various activities to initiate once its fund raising efforts begin to be successful. The Board will be seeking funding sources and is open to suggestions. Dr. Galasso stated that the NFBR, "will likely concentrate initially on support of educational activities, making the public aware of the latest research findings which will improve the public health, and supporting fellowships at the graduate and senior level, including sabbaticals both at NIH and the extramural community." Dr. Galasso can be contacted at the National Foundation for Biomedical Research, 1 Cloister Court, Bethesda, MD 20814-1460, telephone (301) 402-5311.
From October 1992 to October 1996, the total number of grant applications reviewed by NIH increased 2 percent, from 11,997 to 12,188. The total direct costs requested in applications for research grants increased 1 percent, from $1,917 million in October 1992 to $1,935 million in October 1996.
October 1996 October 1992
Applications reviewed 12,188 11,997
DRG 8,591 8,170
Institutes/Centers 3,597 3,827
Research grant applications 10,726 10,845
Research projects 8,546 8,701
Small business/Technology transfer 1,184 845
Research centers 138 224
Other research 858 1,075
Training applications 1,340 1,136
Fellowships 1,170 926
Training grants 170 210
Other applications 122 16
Applications amended 3,615 2,803
Percent of total number reviewed 30 23
Applications responding to RFAs 1,244 1,250
*Includes the National Institute on Alcohol Abuse and Alcoholism, the National Institute on Drug Abuse, and the National Institute of Mental Health.
Dr. Anthony E. Demsey
Acting Deputy Director
Dr. Donna J. Dean
Acting Chief, Referral and Review Branch
Mr. Nicholas J. Suszynski, Jr.
Associate Director for Advanced Technology
Dr. Samuel H. Joseloff
Assistant Director for Special Projects
Mr. John Jones
Acting Executive Officer
Editorial Board
Chairperson
Dr. Samuel C. Rawlings
Coordinator, Sensory Sciences Initial Review Group
Referral and Review Branch
Division of Research Grants
Dr. Claudia Blair
Director, Extramural Outreach and Information Resources
Office of Extramural Programs
Office of Extramural Research
Office of the Director, NIH
Dr. Mark R. Green
Chief, Extramural Projects Review Branch
National Institute on Alcohol Abuse and Alcoholism
Mr. Robert Moore
Project Leader, Office of Reports and Analysis
Office of Extramural Research
Office of the Director, NIH
Ms. Jo Pelham
Scientific Review Administrator,
Geriatrics and Rehabilitation Medicine Study Section
Referral and Review Branch
Division of Research Grants
Dr. Michael E. Rogers
Director, Pharmacology, Physiology, and Biological Chemistry Division
National Institute of General Medical Sciences
Editor
Mr. Nicholas Moriarty
Program Analyst, Information and Analysis Office
Office of Administrative Management
Division of Research Grants
Last Revised: October 11, 1996
Return to Top
![[DRG Home Page]](../images/drghome.gif)