The Division of Research Grant Advisory Committee (DRGAC) Meeting was convened in an open session for its 16th meeting at 8:30 a.m. on Monday, April 28, 1997, in Conference Room 10, Bldg. 31. Dr. Keith Yamamoto presided as chairperson.

Members Present:

Leodis Davis, Ph.D.
Olga Jonasson, M.D.
Elizabeth Theil, Ph.D.
Keith R.Yamamoto, Ph..D.

Temporary Members Present:

David R. Burgess, Ph.D.
Laurie Chassin, Ph.D.
Donald Cleveland, Ph.D.
Barbara A. Hamkalo, Ph.D.
Paul Manis, Ph.D.
Howard K. Schachman, Ph.D.
Alan Spradling, Ph.D.

I. Call to Order and Opening Remarks

Dr. Yamamoto welcomed the participants and introduced Dr. Laurie Chassin, a new participant who is professor of Psychology at Arizona State University in Tempe.

II. Consideration of Minutes

The minutes of the November 1996 meeting of the Division of Research Grants Advisory Committee Meeting were unanimously approved as submitted.

III. Report from the Director, DRG

Dr. Ehrenfeld noted that when she first arrived at DRG, she contacted the external scientific community to help her identify their areas of concern and set priorities for what needed to be done. A primary concern was whether the scientific review groups (SRGs) were organized to give the most efficient and effective reviews. To address this concern, Dr. Ehrenfeld envisioned two activities that will be carried out in parallel. First, working within the existing structure, initial review groups (IRGs) will participate in an oversight process, in which a group of outside scientists advise Scientific Review Administrators (SRAs) on issues related to the management of their SRGs. Second, the overall structure and composition of SRGs will be studied by a to-be-appointed blue-ribbon panel. This activity will be directed by Ms. Linda Engel, head of a new DRG Office, the Office of Planning and Outreach. Dr. Ehrenfeld also noted another new office, the Office of Policy and Analysis, headed by Dr. Anthony Demsey, to analyze and communicate policy changes both to NIH and also to the outside community.

Another concern is that some segments of the research community do not feel well served by the existing structure. To address this issue, Dr. Michael Simmons was hired temporarily as a consultant to serve as a liaison with the clinical research community. DRG plans a similar approach for the behavioral sciences, and possibly for technology development and instrumentation.

Another issue is the perception that reviewers are either too young and inexperienced or part of an "old boys" or "old girls" network. Dr. Ehrenfeld plans to work closely with the Division’s SRA Council on this and other issues, such as flexibility in membership requirements.

In addition, Dr. Ehrenfeld and DRG are working on several aspects of SRA quality and recruitment. Much can be done to improve the SRAs’ situation, both from within the Division and also from professional societies and other outside organizations, to ensure that SRAs are current in their fields and recognized and rewarded for their efforts.

In this time of streamlining, reinvention, and the incorporation of new information technology, the changes will have enormous impact on the peer review process. Dr. Ehrenfeld expressed her gratitude to the entire staff within DRG for their assistance. She was optimistic that together an enormous amount will get done in the next year.

In the ensuing discussion, Dr. Theil asked about the proposed timetable for the evaluation of study section organization. In response, Dr. Ehrenfeld hoped for a 12- to 18-month timetable, though this may be unrealistic. Dr. Cleveland was concerned about flexibility in the recruitment of reviewers. Dr. Ehrenfeld noted that the mechanics of introducing more flexibility in the process will be addressed. Other committee management constraints, such as geographic distribution and gender and minority representation, are also being eased by maintaining the appropriate balance over the entire IRG rather than each study section.

IV. Report from the Chairperson

Dr. Yamamoto described the Advisory Committee’s broad goals of helping the DRG Director define and then implement strategies to meet the Division’s needs and achieve its goals. To do this, Dr. Yamamoto proposed the Advisory Committee meet three times a year, instead of twice a year, and coordinate the meetings with those of the Peer Review Oversight Group (PROG). Dr. Yamamoto summarized two broad areas of activities involving the Committee and Dr. Ehrenfeld: (1) review organization, i.e., assessing the structure of the SRGs and ensuring that the review process is both broad and deep; and (2) review mechanics. The second area includes: training the chairpersons; recruiting SRAs; implementing initiatives about peer review, expediting the review process; and exploring the role of senior scientists in the review process. Other topics, which also involve PROG, are programs for new investigators, the PHS 398 application kit, and the National Research Service Act (NRSA) programs.

Subsequent discussions included reports of the dynamics of SRG meetings by Dr. Schachman, and the issue of flexibility in getting busy scientists to serve. There seemed to be agreement that cohesiveness and group dynamics are important parts of the review process, and help to form the culture of the group.

V. Report on the Peer Review Oversight Group-- Recap of February 1996 Meeting
Rating of Grant Applications--Scoring Practices

Dr. Yamamoto reported on the discussions of the rating of grant applications and scoring practices, in which the central debate was over whether creativity and innovation should be separate or part of the other review criteria. The Committee in the past supported having a separate criterion, while PROG, in a close vote, recommended merging innovation with the other criteria. (Dr. Varmus’ decision and wording for five review criteria were announced at the May 5th PROG meeting. See Appendix A.)

Dr. Yamamoto then mentioned a report from Dr. Demsey on whether priority scores reflected when during the meeting the applications were reviewed. Data were collected from eight SRGs, who had ordered their reviews differently (alphabetically, numerically etc.) No distinctive differences in scoring patterns were seen for applications reviewed near the beginning or end of the meetings.

Dr. Yamamoto reported on another study on scoring patterns, which found an increasing number of tied scores and scores using whole numbers (150, 160, 180 etc.). In further deliberations, PROG suggested reducing the number of scoring options for reviewers from 41 to 7, but left the decision for future discussions.

In response to a question by Dr. Chassin about individual criteria scores vs. global scores, Dr. Ehrenfeld noted the strong support for maintaining a single global score. Individual criteria, however, would guide reviewers when writing critiques, help Institute program staff know how a score was derived, and inform the applicant community about the basis of the evaluation of their application.

Dr. Cleveland felt that the tied scores and whole numbers reflected a failure of the peer review discussions, and that NIH ought to consider ways to encourage discussions where more people participate than just the two primary reviewers. Dr. Schachman, however, based on his observation of more than 30 SRGs, thought there was often widespread participation. The topic needs further analysis.

VI. Review of Individual NRSA (Fellowship) Applications

Dr. Hamkalo reported on the impressions of members of SRGs reviewing fellowships regarding the review of individual fellowship (F32) applications assigned to regular SRGs. The consensus was that the process was not working well. Problems identified were: 1) too few fellowship applications assigned to an SRG; 2) fellowship applications grouped at the end of the SRG meeting, with insufficient review time and reviewer fatigue an issue; 3) inappropriate emphasis on only the research project; and 4) reviewer guidelines that were insufficient or incomplete.

Dr. Nancy Pearson, an SRA, then spoke about recent studies and data on fellowship reviews. A pilot study on their review in Special Emphasis Panels versus regular SRGs concluded that with explicit review instructions, a minimum of 10 applications, and grouping the F32 applications together, fellowship applications could receive a high quality review in regular SRGs that review R01s. After mentioning that the fellowship reviewers’ guide had been revised to emphasize the candidate’s qualities more fully, Dr. Pearson presented some recent data. During the 1/97 Council round, 64 percent of F32 applications were reviewed by SRGs having at least 10 F32 applications; in 5/97, this number increased to 85 percent. Nonetheless, some fellowship applications were still reviewed in very small groups. During the past round, SRG members were polled on whether fellowships should be reviewed in regular research SRGs or Special Emphasis Panels reviewing only fellowships. Responses were diverse, with no consensus.

In the lengthy discussion that followed about fellowship applications, Dr. Jelsema, an SRA, felt that some problems were caused by inadequate instructions on how fellowships differed from R01s. The consensus, expressed by Dr. Cleveland and Dr. Burgess, was that while the revised guidelines were an improvement over the previous ones, they still needed more work and more input from the extramural community. For example, training potential was still inadequately defined. Dr. Burgess recommended moving the research proposal item to the end of the checklist to reduce its visibility and importance. Dr. Cleveland had prepared draft guidelines, which were used successfully during a recent Special Emphasis Panel reviewing 70 fellowship applications. Dr. Cleveland’s version stressed the candidate’s promise as a researcher, the research training atmosphere of the proposed group, and whether the proposed project would make the candidate attractive for employment in an academic, clinical or industrial environment.

Other suggestions included: using 20 instead of 10 as the critical mass of fellowship applications to be reviewed in a regular study section; continuing to use (as has been the practice in the SRG on which Dr. Hamkalo serves) or increasing the use of senior researchers with considerable training records; grouping F32 applications and separating them from R01 reviews; and telling new members of regular SRGs that for up to three times during their four-year term, they might be asked to help review fellowship applications.

Dr. Hamkalo felt that, especially with fellowships, they were on the horns of a dilemma, because science is being done in a broader way than before. But the complexity of any approach involves a depth that is hard to achieve. Dr Yamamoto described a suggestion that fellowship panels be constituted covering four broad scientific areas. Each F32 application would flow into one of the four areas. This plan would recognize both the need for breadth and also the complexities of separate disciplines. There would be exceptions, and Dr. Ehrenfeld would be encouraged to use flexibility in managing those applications that did not fit into the four areas.

Dr. Ehrenfeld concluded that there is no absolute right or wrong way to review fellowship applications. There are some advantages in reviewing fellowships by a separate panel, and other advantages, in certain cases, in reviewing fellowships in the context of a group of investigators who are familiar with a field. Clustering fellowship reviews is important, but clustering can occur in different ways, from a separate panel to a group within an SRG meeting.

VII. Changes in Review of AREA Grants

Dr. Demsey reported on the history and recent modifications of the Academic Research Enhancement Award (AREA). The award was established by Congress in 1985 to support research at educational institutions that had not been major recipients of NIH support. In recent years, the program has been funded at about $14 million annually. AREA grants are for up to $75,000 in direct costs for up to 36 months. In the first year of the program, Fiscal Year (FY) 1985, the success rate was less than 20 percent. In the second year, fewer applications were submitted, and the success rate was almost 40 percent. In recent years, the success rate has remained around 25 percent.

The recent changes in the AREA program include: three annual receipt dates -- January 25, May 25, and September 25 -- instead of one receipt date in June; acceptance of competing continuation (or renewal or Type 2) applications; and acceptance of appendices.

Dr. Demsey introduced Dr. Janet Cuca, the new director of the AREA program, who noted that AREA investigators often have heavy teaching loads, and there is not much institutional commitment or support for their research. Small research projects, which tend to get done over the summers, are often sufficient for these investigators.

Dr. Ehrenfeld wondered whether AREA applications should be integrated into the R01 reviews or reviewed separately in Special Emphasis Panels? One suggestion was to review AREA applications along with those for fellowships, especially since part of both reviews focus on educational opportunities.

Dr. Davis noted that when money is plentiful, R01 reviewers dealt easily with AREA and other set-aside programs; but when funds become tight, it becomes difficult for reviewers to evaluate such programs positively.

VIII. Study Section and IRG Oversight

Dr. Greenhouse, SRA of the Cellular Biology and Physiology -1 Study Section, described a pilot study, which involved formation of an oversight group of outside scientists to advise SRGs. This proposal was presented at the previous DRG Advisory Committee meeting and has been implemented. These scientists can: (1) assist with nominations of members; (2) review how science is organized with the IRGs; and (3) give feedback on SRG meetings.

Dr. Cleveland felt this process was useful in assessing whether SRGs are reviewing in a comparable way. Other possible suggestions were to move reviewers around among the various SRGs with overlapping areas of interest, and to invite senior experienced reviewers to spend some time on various study sections.

Other topics included the perception in the outside community of the relative stringency of review by various SRGs. In fact there are clear differences among SRGs, and swapping reviewers should provide valuable information on this issue. There is also the tendency in SRGs to focus on narrow work scopes or short-term immediate payoffs, because of the tight funds. They need to re-emphasize innovation. It would also be interesting to look at the abstracts of those applications that did the best in peer review, to see how those investigators did in the future. One member, however, felt it important also to look at applications that did not get paid as well as those that did to get a broad spectrum of the various problems. Another member advocated reviewing mechanisms other than investigator-initiated research, for instance, applications submitted in response to a Request for Application (RFA) or program announcement (PA). Dr. Yamamoto noted that this topic is probably outside the jurisdiction of DRG, but might provide useful information for future PROG meetings.

Dr. Burgess brought up the reluctance of early mid-career faculty to serve on SRGs because their own applications would have to be reviewed by another, less appropriate SRG or ad hoc group. Dr. Theil added that she has heard that younger scientists think that the prestige of service on an SRG has diminished.

Dr. Ehrenfeld felt that helping to convince people of the value of service on an SRG is part of a much broader mind set that needs to be changed through communication and dialogue with the scientific community. Increased workload and numbers of amended applications are part of that problem. Dr. Ehrenfeld concluded that the success rates are better than people think, and things are being done to make SRG service more attractive and exciting. Increasing the flexibility about timing and service should help. Dr Spradling noted that service on an SRG is a community service, but also helps investigators with their own applications and provides valuable networking.

IX. Expediting the Review Process

Ms. Engel reported on the receipt and referral initiative, one of four NIH reinvention initiatives singled out for high priority this year. The sponsors are Dr. Wendy Baldwin, NIH Deputy Director for Extramural Research, and Dr. Ellie Ehrenfeld, Director of DRG. A Team of NIH staff and professionals from MITRETEK and the Massachusetts Institute of Technology was asked to develop a high-level, conceptual design to make receipt and referral more efficient and effective, to capitalize on the ability to receive and process an application electronically, and to contribute toward the development of the Commons and IMPAC II. The Commons is the electronic mechanism by which the extramural community and NIH will exchange information and conduct business, and IMPAC II is replacing the 30-year-old database system so NIH will be able to better manage, store, and access extramural data. Six evaluation criteria were developed against which proposed improvements can be measured: time, money, professional judgment, quality, service, and positioning for the future. The scope of this reinvention project was limited strictly to receipt and referral, not review.

NIH has a 6- to 10-month referral and review cycle, and three overlapping rounds. Receipt and referral involves a five-step process during which applications are: (1) received in the mailroom; (2) brought to the receipt and record unit where data are keyed in; (3) referred to an Institute or Center for potential funding and to an SRG for scientific merit review; (4) processed by a quality control unit; and (5) then printed and distributed. For FY 1996, DRG received over 38,000 applications.

In terms of money, printing accounts for 52 percent, which is $1.63 million a year. In terms of time, 38 percent of the time was used for receipt and data processing, and 40 percent was taken up by printing. Only 13 percent was used for application referral. Thus, the time allocated for receipt, data processing, and printing were prime targets by the Team for savings. The Team, for example, seeks to capitalize on plans to receive the shell information electronically. (The shell information is the completed face page, the budget, the abstract, and the research aims.) Pilot studies are already underway to do so.

Other topics being considered by the Team include: having the printing take place in parallel fashion with the other stages, instead of only at the fifth stage; encouraging self-identification of the SRG by the applicant; automatically assigning amended, supplemental, and competing continuation applications; and developing computer-assisted referral for those applications that can not be assigned automatically. Ms. Engel stated that quality control and interactions with the scientific community are great strengths of the current system, and these must be maintained whatever changes are introduced. The team is currently examining the data and proposed actions, so as to develop recommendations and outline the next steps for the sponsors.

Dr. J. J. McGowan, Director, Division of Extramural Activities, National Institute of Allergy and Infectious Diseases (NIAID), reported on a second high-priority reinvention initiative to expedite the review and award of applications. He noted that the Institute will soon have their first paperless National Advisory Council. Another process being streamlined is the receipt and review of contract proposals. His main topic for discussion, however, was their effort to reduce the time for receipt, referral, and review of grant applications to 4 to 5 months.

Dr. McGowan reported on an experiment done with the Tropical Medicine Study Section, where they allowed institutions to submit human subject and animal welfare certifications up to the time the application was scored. The other part of this experiment involved self-referral. These applications had a separate receipt date, which automatically separated the applications from all the others, and thus expedited the process.

Self-referral had already been tested in DRG, for example, with the expedited review of AIDS applications. In addition, all principal investigators have the right to request an assignment, as specified in the PHS 398 kit. The data show that of the small number of people who have used this option, between 80 and 90 percent requested assignments that were honored.

Dr. McGowan noted that prior to developing their system, they engineered the process on paper. They unlinked the general Council meeting from the Council review process, thus making awards two months earlier than previously possible. In 1995, they converted to a totally electronic system, providing earlier access for Council members to all information. Using web-based technology, Institute staff forwarded to Council applications within the pay line (about 40 percent of the applications going for award, usually about 65 to 80 grants per round). The applications could be approved by Council within 8 weeks.

The Institute also developed an electronic system on the Internet, whereby reviewers could examine all critiques prior to the review meeting. This system was secure in terms of identification, and reviewers could not change anything on the system. Having the electronic system provided an overview to help frame the discussions at the meeting; permitted more effective use of reviewers’ time on more focused discussions; decreased the time needed for some meetings; and allowed more flexibility and more options for Institute staff to engage the scientific community in the review process. Dr. McGowan stated that the experiment was possible because of a partnership between DRG and the SRAs and the Institute, and they are now exploring the possibility of other Institutes using the system.

Further discussion involved the cost of the system for the reviewers and how to accommodate reviewers who did not have access to the Internet. Dr. McGowan stated that the number of people not having access to the Internet is becoming increasingly small, and, as in the past, they would take care of any exceptions. In response to a question about how the Council felt about receiving reviewer’s preliminary critiques instead of summary statements, Dr. McGowan noted that summary statements for applications in the gray zones (undecided) still go to Council.

The Committee expressed support for shortening the review process; improving the opportunities for reviewers to examine each others’ reviews before coming to the meeting; and shortening the receipt and referral process through processing of electronic shell proposals, scanning of applications, and modifying the printing process. Dr. Theil then presented a proposed Committee statement of support: "Based on the results of pilot studies, DRG should strive to reduce the time of referral, receipt, and review by a combination of electronic processing and receipt, increasing self-referral, developing or implementing electronic peer review, delaying the receipt of human and animal assurances, and abbreviating amended applications for selected proposals. We further recommend that DRG encourage the use and development of electronic and expedited Council review when appropriate."

Dr. Ehrenfeld stated that DRG is committed to supporting all the components of this statement; however, she felt that further discussion was needed, and wanted to use the Committee to engage in such discussions.

X. Neuroscience Integration Activities

Dr. Manis commented on the integration into DRG of reviews conducted by the National Institute of Mental Health (NIMH), National Institute on Drug Abuse (NIDA), and National Institute on Alcohol Abuse and Alcoholism ( NIAAA) and the restructuring of some of the DRG neuroscience reviews. The developments in science should mandate the review structure, and DRG should be flexible in reconfiguring SRGs and IRGs to handle emerging scientific areas.

Dr. Elliott Postow, Pilot Independent Initial Review Group (PII) Chief, reported on the meetings of a working group on the integration of the neuroscience activities. The group consisted of extramural scientists, DRG review staff, Institute program and review staff, and staff from the NIH Office of the Director. They formed five panels in the areas of: (1) molecular and cellular neuroscience; (2) developmental neuroscience; (3) integrated, regulatory and behavioral neuroscience; (4) cognitive neuroscience; and (5) brain disorders; and clinical neuroscience. Using the abstracts from one round of applications submitted to the NIH, they developed a set of review guidelines, overlap statements, and expertise rosters, which they will present, along with any results, to the May 5th PROG meeting. At PROG they will also discuss the relationship between grant mechanisms and research venue. For example, the same IRGs in NIMH currently review Ps,Ts, Fs, Ks, and Rs, in part, because of the low workload. With the DRG workload of about 100 applications per meeting, this may not be possible. Also PROG will probably analyze more fully the interface between neuroscience and the behavioral sciences, as the DRG behavioral science SRGs are reorganized to accommodate the NIMH and NIDA applications.

The information will be available on the world wide web, and the NIH and extramural scientists will have an opportunity to comment on the recommendations. The plan is to announce the final recommendations to the community for applications to be submitted for the February 1998 receipt dates.

Dr. Hamkalo, the first discussant, reported on the response from colleagues in the neurosciences. Developmental and molecular-cellular neurobiologists were positive in their comments as opposed to the more behaviorally-oriented biologists, who did not think other researchers could appreciate the complexities of their type of research. Dr. Hamkalo emphasized flexibility, noting that in the next 5 to 10 years, much of the behavioral research will probably become much more like molecular-cellular research.

Dr. Chassin, the second discussant, recounted concerns of colleagues that in striving for breadth on SRGs, it might become more difficult to compensate for a particular reviewer’s bias. There were also concerns about the ability of reviewers to appreciate the importance of clinical research, such as on alcohol, drug, and mental health issues, which are not just biomedical but also bio-psycho-social problems.

Dr. Cleveland, the next discussant, noted that the status quo has considerable inertia, and the uncertainty of sending applications to reconfigured SRGs will certainly lead to anxiety; but if done well, the new SRGs will be seen as an advantage and improve the review process.

In response to a question by Dr. Burgess on the wisdom of having a specific SRG narrowly focused on cellular developmental neuroscience, Dr. Ehrenfeld stated that DRG would strive to avoid this isolation of a scientific field. Overlap is desirable, and the Division does not want narrow-focused approaches. Committee members agreed that it would be disastrous if the integration activities were viewed by the scientific community as some sort of a set-aside for the neurosciences. Rather it should be viewed as an experiment, a way of gaining experience for configuring applications from an SRG. Dr. Postow explained that considerable flexibility already exists in the structure of the IRGs and their subcommittees to evaluate when an IRG needs to be restructured to respond to an emerging scientific area.

XI. Summary and Discussion of Action Items

Dr. Yamamoto mentioned two broad organizational principles for the Committee -- review organization and review mechanisms. With respect to review organization, the challenge is to begin to reassess the SRGs and IRGs. A two-level system for doing that is to: (1) gather data and begin some small experiments; and (2) step back from that process and consider how the whole structure might be reconfigured to review science in the best way. Both levels of assessment could be carried out in parallel. The working group reports and the updates for this meeting have provided a good perspective and overview for a number of different ways into this process and should be continued and broadened.

Dr. Spradling noted that principal investigators whose applications just missed being paid often wondered if there were an easier SRG, where applications that are not nearly as good as theirs end up being funded. Such data would give the investigator the opportunity to see whether a similar application did better. This would be very useful and probably reassure the investigators about how well the peer review system functions. In a few cases where the principal investigator was right, a correction could be easily made, resulting in confidence in, and support for, the overall system from the community.

In Dr. Yamamoto’s closing remarks, he noted that the Advisory Committee, in cooperation with PROG, needed to be thinking of ways to implement the new scoring criteria. Dr. Schachman will be the liaison between the groups, and report on this topic. Dr. Theil, whose term of service was complete, agreed to return for the next meeting and lead discussion on ways to get senior scientists involved in review in a systematic way, on ways to expedite the review process, and on ways to further use the electronic initiatives in the Council reviews.

XII. New Business and Closing Remarks

Dr. Yamamoto thanked all the participants for their enthusiasm and expertise, which had helped make this such a valuable meeting. The meeting adjourned at 12:00 p.m. on April 29, 1997.

We do hereby certify that, to the best of our knowledge, the foregoing minutes and attachments are accurate and complete.

/s/ _______________________
Samuel H. Joseloff, Ph.D.
Executive Secretary
Division of Research Grants
Advisory Committee

/s/ _______________________
Ellie Ehrenfeld, Ph.D.
Director
Division of Research Grants

/s/ ______________________
Keith Yamamoto, Ph.D.
Chairperson
Division of Research Grants
Advisory Committee