The Center for Scientific Review (CSR) convened the 20^th meeting of its Advisory Committee at 8:30 a.m., on Monday, January 11, 1999, in Conference Room 6, Building 31C. The entire meeting was in open session. Dr. Keith Yamamoto presided as Chairperson.

o Keith Yamamoto, Ph.D., Chairperson
o Karen Matthews, Ph.D.
o Raphael Pollock, M.D., Ph.D.
o Marvin Wickens, Ph.D.
o Tachi Yamada, M.D.

o Michael Berns, Ph.D.
o Shu Chien, M.D., Ph.D.
o Michael Colvin, M.D.
o Leonard Epstein, Ph.D.
o Barbara Graves, Ph.D.
o James Kushner, M.D.
o Gabriel Navar, Ph.D.
o Roderic Pettigrew, M.D., Ph.D.

Dr. Yamamoto called the meeting to order, thanked the participants for attending the meeting, and invited the individuals around the table to introduce themselves. For the new members, Dr. Yamamoto briefly explained that the Committee was established to advise the Director of CSR and make recommendations on issues relevant to the Center. Towards this aim, the Committee may establish subcommittees, such as the IRG Working Groups and the Panel on Scientific Boundaries for Review. He emphasized that the Advisory Committee makes recommendations, but does not set policy for CSR. He also mentioned that CSR has input into the Peer Review Oversight Group (PROG), which deals with trans-NIH rather than Institute or Center specific issues.

The minutes of the September 14 and 15, 1998, meeting were than unanimously approved as submitted.

Dr. Ehrenfeld provided an overview of progress in six priority areas since the last Committee meeting. These six areas are: (1) study section organization; (2) reviewer quality and study section composition; (3) perception that segments of the community are ill served; (4) speed and consistency of the receipt, referral, and review process; (5) responsiveness to NIH funding Institutes and Centers; and (6) enhanced function of the Scientific Review Administrators (SRAs).

Dr. Ehrenfeld reported on progress that has been made in these areas. In the area of study section organization, the Panel on Scientific Boundaries for Review is developing guiding principles for study section organization and will provide a report later this year. CSR has also been working for the past two years on the integration of review activities from the Mental Health, Drug Abuse, and Alcohol Abuse and Alcoholism Institutes into CSR. This integration is nearing completion with the establishment of 21 new neuroscience and 8 new AIDS study sections in June 1998, and approximately 16 more study sections in behavioral and social sciences that are planned for June 1999.

Regarding the enhanced functions of SRAs, last December, a successful, one-day off-campus training course for SRAs was initiated to teach best practices, and will probably be repeated in 1999. CSR is also developing a series of mini-courses on such topics as conflict of interest, review of RFAs, and the Small Business Innovation Research program. In addition, a mentorship program to assist new SRAs has been established. Dr. Ehrenfeld pointed out that enhancing SRA function requires resources and that we must attain reasonable workloads, a stimulating environment, opportunities for professional development, and time to communicate with the scientific community.

Dr. Ehrenfeld then mentioned that Dr. Leonard Epstein has been brought on board to serve as liaison to the behavioral and social sciences community, which sees itself as disadvantaged by the current peer review system. She also noted that, later in this meeting, Dr. Bradley will discuss several experiments with review procedures that are in progress, and Dr. Suzanne Fisher will address some referral issues related to interactions with NIH Institutes and Centers (ICs). Issues related to the speed and consistency of the peer review process were addressed at the last meeting by Dr. J.J. McGowan and will not be further addressed at this meeting. Dr. Ehrenfeld also introduced Dr. Maxine Linial, a consultant who has been brought to CSR for three months to analyze and make recommendations regarding the review of fellowship applications.

Dr. Ehrenfeld also mentioned a seventh, internal topic, which is the shift in focus in CSR from the individual study section to the Initial Review Group (IRG). Since each IRG contains a cluster of scientifically related study sections, this mechanism provides an opportunity for more flexible distribution of applications and sharing of reviewer expertise. The IRG is also a vehicle for developing teamwork among staff involved related review activities.

In response to a question by Dr. Yamada about the relationship between the Panel on Scientific Boundaries for Review and the IRG Oversight Groups, Dr. Yamamoto explained that the Boundaries Panel operates independently from the IRG Oversight Groups, but receives data from those groups that have been established. Dr. Ehrenfeld added that the Boundaries Panel was a "one shot" task force. Their task is to recommend (not implement) a peer review system, including the appropriate alignment of study sections and IRGs and some guiding principles, starting from a clean slate.

Dr. Yamamoto suggested that the roster and the recommendations of the Panel on Scientific Boundaries for Review be put on the WEB, and that the Panel also be publicized through various professional association newsletters.

Dr. Yamamoto indicated that two processes are simultaneously occurring regarding the operation of IRGs. The IRG Working Groups are looking at existing IRGs, whereas the Panel on Scientific Boundaries for Review is considering organization of IRGs from scratch. Both groups operate independently, and the challenge will be to integrate the recommendations of both; recommendations from the Panel could effect some changes recommended by the IRG Working Groups.

Dr. Yamamoto then gave a brief background on the Panel on Scientific Boundaries for Review, noting that the Panel’s goals were: (1) to define the organizing principles that enable IRGs to provide a fair and rigorous review of current biomedical research applications, and to recognize and manage newly emerging fields; and (2) to define the operating procedures that establish a "culture" for review groups, promoting the highest quality of peer review.

Operating procedures include roles and responsibilities of the Chairperson, the SRA, and the study section members; expectations of applicants and the length of proposals; the content and length of review; and the involvement of senior scientists in review.

Organizing principles include whether to organize IRGs into organ/disease systems (vertical IRGs) or trans-systems (horizontal IRGs); assigning applications that are appropriate to either system to organ/disease system IRGs; and providing more than one appropriate review group for each application. Using both organ/disease and trans systems enables considerable overlap and considerable movement among study sections. Thus, applications that originally focus on trans-systems could shift to organ/disease systems as the research progresses. Referral officers should examine each competing continuation application to determine if movement is required. The Panel is currently drafting trans system IRGs, and also beginning to draft IRGs for organ/disease systems. The Panel feels that multiple venues are important, and that there should be at least two study sections for each application.

Mechanisms to be used by the Panel are meetings; conference calls; subcommittees; and consultations with the CSR Advisory Committee, IRG Working Groups, IRG chairpersons, and CSR staff. The task is scheduled for completion by June 1999.

Dr. Colvin noted that the attitude towards service on a study section has changed because of the increasing workload. The new IRG and study section organization has also increased concerns that the "culture" changes with a shift to a new IRG or study section. Dr. Yamamoto noted that the expectation of a new member is that he/she will serve for four years (12 meetings), and that there is a need to move away from this expectation and adjust the role of study section members. The right expertise must be available on a committee, but we need to be more responsive to the needs of members. Regarding mail reviews, they are probably most useful for addressing explicit technical aspects of an application.

Dr. Yamamoto also said that when a revised application is reviewed, there is usually at least one new reviewer who may see a different perspective. The goal of re-examining the referral of a competing continuation is to keep the review relevant. Also, if the review starts out in a trans-review system, and moves to an organ/disease review system, this keeps the pipeline open by making room for other new fundamental applications to be reviewed by a trans IRG.

Dr. Ehrenfeld explained that the CSR philosophy was to encourage self-referral of applications, but at the IRG level rather than at the study section level, because study sections may continually adjust their boundaries. Towards this end, the referral guidelines should be reexamined and must be clearer to the scientific community. Currently about 50 per cent of the neuroscience applications are self-referred, but many of these self-referrals turn out to be wrong.

In response to a question about how many IRGs would be recommended by the Panel on Scientific Boundaries for Review, Dr. Yamamoto said that the number would be about 22. There are currently 19 IRGs, but the Panel is not constrained by that number. There are also no constraints on the number of trans versus organ/disease IRGs. Dr. Matthews asked whether the Panel was considering ways to keep the scientific community better informed about trends in various fields, breaking areas, etc. Dr. Yamamoto indicated that this is not something the Panel has discussed; publicizing and accommodating change are complex issues, but workshops could be organized.

Dr. Ehrenfeld noted that the changes and workload have also nearly overwhelmed CSR staff. CSR management is aware of the need for more resources, especially for more SRAs, and is trying to get on top of this problem. She also noted that the IRG Working Groups are a good way to obtain dialogue regarding both emerging and dying scientific areas.

Dr. Epstein explained that, as a consultant to CSR, his efforts have been in two main areas: (1) consulting on the reorganization of the behavioral and social science IRGs, and (2) visiting the NIH Institutes to learn their views about CSR review of social and behavioral science.

The reorganization began in the summer of 1998 with the integration of review activities from the Mental Health, Drug Abuse, and Alcohol Abuse and Alcoholism Institutes, and with feedback from the scientific community that change was needed. CSR was assisted in this endeavor by an internal NIH working group under the leadership of Dr. Virginia Cain. Model study sections were developed and posted on the web. These were later revised based on the numerous responses received from the scientific community. A draft of the final recommendations was submitted to Dr. Ehrenfeld on December 1, 1998. These recommendations were also made available to the outside community for comments, and are now being reviewed by an implementation committee under the leadership of Dr. Jean Paddock.

Dr. Paddock next explained that the CSR Implementation Group met in parallel with

Dr. Cain's working group. The study sections have now been clustered into three IRGs, namely, the Behavioral and Biobehavioral Sciences IRG, the Risk, Prevention, and Behavioral Disorders IRG, and the Social, Epidemiological, and Nursing Sciences IRG.

Members on current study sections have been asked to name their first, second, and third choices on the new study sections, and about 50% have responded so far. Other members will then be recruited to fill in gaps in expertise.

Regarding the second project, Dr. Epstein met with 11 Institute directors or staff to get their perceptions regarding review of behavioral and social science research. One common concern was how to identify the best research. Reviewers often focus on technical aspects of the research, rather than on innovation or significance; this can result in funding research that is boring rather than exciting. The current emphasis on significance and innovation as review criteria may be helping in this regard. Two types of research that are particularly difficult to fund are cutting-edge science and multidisciplinary science. A common perception across Institutes was that reviews by behavioral scientists often have a negative tone, and it has been said that "Behavioral scientists eat their young." This results in poor scores and does not help funding decisions about behavioral and social science at the IC level.

In considering the process of review, there was the perception of considerable variability in how Chairpersons and SRAs operate; more training would be valuable in developing best practices. In finding the best reviewers and encouraging them to serve on study sections, the senior scientist concept as well as flexible membership could prove useful.

Another issue was the review of member applications; many were not confident that they got a fair review. The importance of overlap among study sections was emphasized to provide a second "home" for member applications. Although Special Emphasis Panels (SEPs) are used to review many members applications, there are concerns about the composition of these panels, since there is not the same degree of scrutiny as with regularly established study section members. There are also concerns that telephone reviews may have a different "culture", and that these panels usually meet late in the review cycle, which leaves less time for members to revise their applications. Nonetheless, most IC staff thought that SEPs were as good as overlapping study sections for the review of member applications.

Dr. Matthews complimented CSR's use of the scientific community in reorganizing the behavioral and social sciences. However, there could be problems with the third IRG (Social, Epidemiological, and Nursing Sciences IRG) which seems less cohesive than the other two. She agreed that behavioral and social science researchers tend to be highly critical when reviewing their communities' applications, which has led to such applications faring badly in comparison with researchers in other fields. Some education of reviewers may be necessary, and summary statement production should be examined for best practices.

Dr. Pollock questioned whether the emphasis on significance and innovation was working, and noted that study sections often focused on feasibility and methodology to the detriment of behavioral and social sciences applications. This may be a question of training reviewers, and he suggested that a "user manual" should be developed for ad hoc reviewers. He was also concerned about the quality of telephone reviews where the culture may be very different.

Dr. Anita Sostek (CSR) said that the critical nature of reviews in behavioral and social sciences was across the board and not just restricted to young investigators. Part of the reason may be lack of agreement on standard methods and she noted that it was extremely important for applicants to justify their choice of methods. She also felt that the new emphasis on significance and innovation as review criteria was helping to modify the tendency towards detailed discussions of methodology. In addition, she indicated that the increasing use of laptops has helped reviewers edit their critiques based on the discussions.

Ms. Engel updated the Advisory Committee on progress since September in addressing the concerns related to review of bioengineering and instrumentation, and technology development research. This community is one of three identified by Dr. Ehrenfeld as underserved by the current peer review system.

Contemporary research is becoming more multi- and interdisciplinary, and opportunities to advance a number of fields increasingly require the marriage of the life sciences with the physical sciences and engineering. The trend is toward an integrative approach.

NIH plans to focus more attention on technology and to increase support for an engineering approach to solving biomedical problems. However, the NIH peer review system has not kept pace with these changes in how we do science, and these communities feel underserved by the current system. It is felt that reviewers often do not appreciate the potential contribution of non-life science or non-hypothesis-driven research. Reviewers are also put off when a proposed project does not apply to a specific biological problem or when the principal investigator is a bioengineer instead of a Ph.D. or M.D. scientist. The high cost and extensive time needed to complete the work are problematic, and reviewers tend to think that these projects should be supported by the National Science Foundation (NSF).

With this in mind, Dr. Ehrenfeld established a small Working Group on the Review of Bioengineering and Technology and Instrumentation Development Research as an ad hoc subcommittee of the CSR Advisory Committee. The group is chaired by Dr. Lee Huntsman, Provost and Vice President at the University of Washington; members represent a broad range of interdisciplinary perspectives. (More information about this group, including the roster, is available at http://www.csr.nih.gov/EVENTS/WG.htm).

At its first meeting on October 12 and 13, 1998, the Group was charged by Dr. Ehrenfeld with identifying impediments to a fair, rigorous, and high-quality review and with outlining principles for the development of a more friendly review infrastructure. The group also learned of Dr. Ehrenfeld’s practice of engaging the extramural community as partners, and of recent innovations at CSR, including 1) establishment of the IRG as the functional unit of review; 2) creation of advisory working groups for each IRG; 3) encouragement of self-referral by investigators to IRGs; and, 4) adoption of five new review criteria (significance, approach, innovation, investigator, and environment).

The Working Group developed four preliminary recommendations. First, they challenged the IRGs and their Working Groups to better meet the needs of interdisciplinary research by providing a suitable venue for review within both biological and methodological contexts. Second, they felt that CSR should more fully develop the program - review partnership with institute staff. Third, they suggested having greater representation of bioengineers on policy-making committees, especially the National Advisory Councils and Boards. Fourth, they recommended developing a different, complementary mechanism to foster major scientific breakthroughs. This mechanism might be a "DARPA-like" model, such as was used to establish the Internet, or creation of NSF-modeled Science and Technology or Engineering Centers.

These preliminary recommendations were widely distributed for comments from NIH staff and the scientific community. Based on this input, a report is being drafted for release in March, and further input will be sought at a public forum to take place in April. The final report will be released in June.

Dr. Berns, one of the discussants, stressed the need to educate both reviewers and applicants. Sometimes whether or not a project is "hypothesis-driven" turns out to be a matter of semantics; most applications could be re-written as hypothesis-driven. Furthermore, even if not hypothesis-driven, the impact must be clear and the investigator needs to explain what problem will be solved with this technology.

Dr. Chien, the other discussant, noted the explosion of biomedical data, and that bioengineers are needed to help manage these data. He noted that science and technology are inseparable, and felt that NIH needed a more broadly-based funding mechanism, possible a mechanism that included venture capital from industry for targeted products. Finally, Dr. Chien suggested revising the PHS 398 application kit to include the IRG structure and the new review criteria.

In the general discussion, it was noted that this is a very broad field, and that it is unclear what is a bioengineer. Dr. Chien responded that any person who applies engineering techniques to solve biomedical problems can be considered a bioengineer.

It was also noted that the Whittaker Foundation has provided funds to build bioengineering departments; when Whittaker money is no longer available in about eight years, these applications will likely come to NIH.

Dr. Linial, a virologist from the Fred Hutchinson Cancer Center, has recently been retained for a three-month detail by CSR to analyze the review of fellowship applications. Her background and qualifications include an interest in training, serving as principle investigator on a training grant (T32), and serving as a member on study sections reviewing fellowships. Her charge is to examine review data and determine the optimal way to configure study sections to review fellowship applications. Other issues to be examined include critical mass (minimum number of fellowship applications clustered), reviewers’ attitudes, timing of review (fellowships are often reviewed at the end of the meeting when reviewers are tired), focus on technical details rather than on the training environment, problems in shifting review criteria for different mechanisms, insufficient review guidelines, and SRA training.

Fellowship data collected for the January 1999 Council showed that 150 applications (28%) were reviewed in special emphasis panels, 223 (42%) were reviewed in standing study sections with more than 10 fellowship applications reviewed, and 161 (30%) were reviewed in standing study sections with less than 10 fellowship applications reviewed.

In an attempt to determine whether fellowships make a difference in the careers of young scientists and whether NIH is doing a good job of selecting fellows who will become top scientists, Dr. Linial plans to perform retrospective analyses. Using existing data, she will determine whether individuals funded in 1978 to 1981, when fellowships were reviewed in a different way, are still active scientists. She is interested in whether it makes a difference which fellowship mechanism is received, i.e., an individual fellowship (F32) versus Institutional Training Grant (T32), and plans to examine whether the current guidelines are adequate or need to be revised in light of the data.

In addition to collecting and analyzing data, Dr. Linial plans to survey the IRG chiefs regarding reasons for their assignments, and to evaluate the review guidelines. She also plans to talk with staff in selected NIH Institutes about their concerns with fellowship review, and to observe study sections reviewing fellowships at the February and March meetings.

In response to a question about why fellowships are not reviewed in the Institutes along with the training and career applications, Dr. Ehrenfeld explained that the reason is mostly historical. Originally, DRG reviewed all training (T series), fellowship (F series), and career (K series) applications, but the Institutes gradually took over certain mechanisms. Currently, most training and career applications, which are mission directed, are reviewed within the Institutes; individual fellowships, which are trans-NIH, are reviewed within CSR, although there are exceptions. Four Institutes (Alcoholism, Drug Abuse, Dental, and Nursing) review all of their individual fellowship applications.

Asked if she planned to include data from training grants (T32s), Dr. Linial said that such data would make the subject too broad for a three-month study. The suggestion was made that fellowship panels should use primarily younger, junior reviewers, who may be more understanding and sympathetic because they were in this position not so long ago. Dr. Linial argued, on the other hand, that reviewers should be senior, more established reviewers, who have a broader perspective gained from experience.

Dr. Ehrenfeld noted that fellowship applications are not percentiled, and that there is about a 40% success rate among fellowships. She also indicated that it is not practical to set up special emphasis panels in all IRGs for the review of fellowships because the numbers are too low in some IRGs, and the science is too broad in others.

Dr. Wickens distributed and summarized a draft document of guidelines for study section chairpersons. The objectives of these guidelines were to orient new chairpersons, to foster cooperation between the chairperson and SRA, to involve the chairperson in study section operations, and to promote a desirable study section "culture." The guidelines have five main sections: (1) Chairing the Meeting, (2) Taking an Active Interest in the Composition of the Panel, (3) Helping Train Panel Members, (4) Involving the Next Chairperson Whenever Possible, and (5) Welcoming Evaluations of Performance, Including Your Own.

The longest section, that on chairing study section meetings, addressed preparations before the meeting, followed by details on how to run the meeting. Issues addressed in running the meeting were: beginning each review with a reading the synopsis and indicating a level of enthusiasm; moving the meeting at a pace that ensures a fair and high quality review, focusing on key issues; ensuring criteria-based scoring that is consistent throughout the meeting; ensuring that the assigned score corresponds to the discussion; encouraging members to rank applications; and ensuring that comments or concerns for vertebrate animals and human subjects are addressed along with codes for the inclusion of genders, minorities and children.

After briefly summarizing the remaining sections of the draft, Dr. Wickens pointed out an addendum containing a discussion of the advantages of having as hoc senior members on study sections. These advantages are providing perspective, balance, and advice for achieving high quality reviews, as well as enhancing the credibility of the panel with the external community. He recommended that two senior scientists be present ad hoc at every meeting.

Dr. Kushner, commenting on the synopses that precede each review (Section 1 of the draft), suggested that the first few synopses should be done by experienced reviewers to set the tone. The performance evaluations (Section 5 of the draft), whereby chairpersons meet with the SRA and senior members to evaluate the performance of the meeting and of individual members, seem difficult and sensitive. Having senior members (addendum) may also be a problem because it makes the other members seem less important; this section assumes that senior reviewers are better than the junior ones, which is not necessarily true. Rotation of senior members might make this concept more palatable.

Dr. Pollock, the other discussant, suggested overlapping the terms of service of the retiring and incoming chairpersons, so that the new person could learn from his or her predecessor. Dr Berns suggested having a vice-chairperson, who would be the next chairperson. Dr. Pollock also saw potential problems with the proposed evaluation at the end of the meeting in that some members leave early. Alternatives were to ask members to email their evaluations to the SRA, or have just the SRA and the chairperson perform the evaluation at the end of the meeting. He also liked the idea of senior members attending the meeting, but was concerned that they may dominate. He suggested using the Reviewers' Reserve for continuity, and not setting up senior reviewers as "special".

Dr. Chien suggested that the chairpersons be relieved from review assignments, so that they would have time to read all the applications carefully before review meetings. However, Dr. Epstein felt that chairpersons should review a small number of applications to set the tone of the meeting. Dr. Matthews thought it was too burdensome to expect the chairperson to read all the applications and suggested instead that reading all abstracts would be sufficient.

Dr.Ehrenfeld warned against locking too much into this document; the SRAs and chairpersons should have some latitude and flexibility to be creative and responsible managers. She supported the comments in the draft on the chairperson's role in helping the SRA identify new members (Section 2). However, she emphasized that all SRAs should cast a broader net to avoid clones that generate an "old boy's club". Dr.Ehrenfeld also stated that, while chairpersons can be extremely helpful to SRAs, the SRAs have the primary responsibility for ensuring that resumes are accurate and that the summary statements are clear and consistent. It is the SRA's role to ask reviewers to change their reviews based on the discussion. The draft document should clarify that the chairperson may help in this, but that it is the SRA's role to write the resume.

Dr. Wickens agreed to bring a revised document to the next meeting of the CSR Advisory Committee, incorporating comments from the discussion.

Dr. Bradley, Chief of the Surgery, Radiology, and Bioengineering IRG, outlined several experiments in review procedures that are occurring within her IRG. This IRG consists of four study sections, namely, Diagnostic Imaging, Diagnostic Radiology, Surgery, Anesthesiology and Trauma, and Surgery and Bioengineering, plus several special emphasis panels that review SBIRs, P41s (National Resources), and RFAs in the areas of medical devices, tissue engineering, bioinformatics, and instrument development.

The first pilot outlined by Dr. Bradley was the concurrent meeting of Diagnostic Radiology (clinical) and Diagnostic Imaging (technology), which was set up recently in conjunction with a scientific meeting in San Diego. This brought together expertise in clinical sciences, basic sciences, and technology, and allowed a sharing of reviewers for special emphasis panels and workshops. In June 1999, six SBIR meetings will meet on the same day in the same location, and in October 1999, Surgery, Anesthesiology and Trauma will meet concurrently with Diagnostic Imaging, as will the Diagnostic Radiology and Surgery and Bioengineering study sections.

Dr. Bradley went on to describe a pilot in flexible study section membership. In this pilot, one membership slot will be shared by up to four members with different expertise from the same department. SRAs can draw on specific expertise as needed, and clinicians do not have to commit to three review rounds per year. The pilot is in progress using clinicians from the Radiology Department at the Johns Hopkins Medical School.

Another experiment involves electronic review using the model described by Dr. J.J. McGowan at the preceding meeting of this Advisory Committee. The model has been implemented once in the Diagnostic Radiology study section, which was a large meeting with 22 chartered members and 8 temporary members. Dr. Bradley indicated that the electronic procedures were highly successful, and that the electronic procedures were about to enter a second round of review in the upcoming meeting of this study section where approximately 100 applications will be reviewed.

The next experiment described involved sabbaticals for SRAs. SRAs need to keep current with science, which is difficult in the busy CSR environment. The sabbatical program would be for one review round (4 months) at an NIH laboratory, clinical site or Institute. The SRA would be linked to CSR by e-mail, and would still make assignments to reviewers, and suggest temporary members, but other SRAs would run the study section meeting and prepare the summary statements. The first SRA from this IRG to participate in this program is Dr. Bill Bunnag who will spend four months at the new Center for Complementary and Alternative Medicine reviewing applications for new centers (P50s) using the electronic review module.

The final experiment described was a trans-IRG initiative for a collaborative review with the National Center for Research Resources. This involved the review of 17 center applications that crossed three IRGs. The review included seven SRAs who met weekly and selected both "common" and "specific" reviewers, while avoiding conflicts of interest. Dr. Bradley indicated that this type of review may be a good model for interdisciplinary review.

Dr. Bradley ended her presentation by stating that the SRAs in the Surgery, Radiology, and Bioengineering IRG believe that their experimentation has strengthened their science, facilitated communications, and provided quality control.

The Advisory Committee was highly enthusiastic about the various experiments presented, as well as about Dr. Bradley’s lively and thoughtful presentation. However, some concerns were expressed about the multiple membership experiment, in that there was the potential for abuse as well as lack of continuity within the study section. But overall the response to this presentation was highly positive.

Dr. Fisher, Acting Director of the Division of Receipt and Referral, described the magnitude of the receipt and referral workload, which is approximately 39,000 applications each year. The receipt, processing, and data entry activities are performed by mailroom and Project Control staff; referrals are handled by Dr. Fisher, three full-time assistant chiefs, eight part-time referral officers, and one full time referral officer in the behavioral and social sciences.

Referrals are based on: (1) guidelines in two internal books, one for assigning to funding components and the other for assigning to study sections and IRGs; (2) applicant requests (self-referral, about ¼ of the total); (3) referral history (revised and competing continuation applications); (4) Institute requests; and (5) responses to program announcements. These referral guidelines are revised approximately every two years. The assistant chiefs assign about 1/3 of all applications to Institute review groups, while the assistant chiefs and referral officers refer the remaining applications to CSR IRGs. In CSR, assignments to study sections and special emphasis panels are done at the IRG level.

Assignment to the primary and secondary institute as funding component is an important decision in the Referral Division. There are 22 funding components which are organized either by body parts (e.g., NHLBI, NEI), by disease or mission (e.g., NCI, NIAID), or by life stage (e.g., NIA, NICHD). Considerable overlap in areas funded exists among these institutes; these overlaps are usually managed by making dual assignments. In the case of differences of opinion, the final decision rests with the Referral Division management staff, who try to take a neutral and unbiased position. When competing renewal (Type 2) applications are received, the principal investigator may request that the same, or a different, study section review the application. The tendency is to refer the application to the same study section and funding Institute, but in some cases, switching the assignment is reasonable. Investigators are encouraged to include a cover letter with assignment requests; if they have concerns after an assignment is made and wish for a change in assignment, there is an effort to accommodate them.

Dr. Schneider, Chief of the Biophysical and Chemical Sciences IRG, next described the referral process that takes place within that IRG. The IRG receives about 700 applications per round that must be processed over a 6 - 8 week period. Referral within the IRG is done by the Chief and is science-based and facilitated by self-referral and auto-assignments based on previous review history. Most applications are first sorted to the appropriate SRA. If the assignment is questionable, the application may be returned to the Referral Division or may go directly to a different IRG. In over 90% of the cases, the SRA accepts the application, and the assignment information is entered into the computer system. Problem applications (about 25 - 30 per round), such as those involving a conflict of interest, are discussed at an IRG meeting, where about half of them are assigned to standing study sections and the rest are assigned to SEPS.

The process works well for R01 applications, but for fellowships (F32s), there may be a problem in reaching the critical review mass of 10 applications.

Dr Jelsema, Chief of the Molecular and Cellular Neuroscience IRG, described the referral process in the three new neuroscience IRGs. Their workload is comparable to that of Dr. Schneider’s IRG with approximately 900 applications per round, but the newness of the study sections and their referral guidelines compounds the problem. About 60% of the applications are self-referred, but about half of the self-referrals are incorrect. Dealing with incorrect requests is extremely time-consuming because the applicant must be called to discuss changes in assignment. They have recently been reviewing new mechanisms, including career (K) applications and small grant applications (R03), which further complicates the process.

In response to a question about applications not meeting the guidelines, Dr. Fisher responded that the Referral Division deals with policy issues (e.g., budget limitations, number of amendments), but that type size and page limits are now handled at the study section level. Dr. Fisher indicated that applicants are given a window of time to fix their applications, and mentioned that reviewers resent spending additional time on excess pages or straining to read small, densely packed type.

Dr. Wickens noted the importance of the Institute assignment for the fate of the application and asked about the commonality of dual assignments. Dr. Fisher responded that about 20 percent of all applications receive dual assignments, and the trend is increasing, driven by the complexity and multidisciplinary nature of the science. Institutes can jointly fund projects, though they rarely do so, except for RFAs.

Dr. Navar discussed the implications of flexible rosters, which show that members attend different study sections within an IRG during a meeting. Applicants may become upset when they cannot figure out which reviewers were present during the review of their applications.

Dr. Wickens warned that self-referral of applications, which is supposed to speed up the process, may actually slow it down when a principal investigator makes a mistake because of the phone calls that SRAs are required make to the principal investigators. Dr. Ehrenfeld agreed that calling investigators has become a nightmare, and indicated that a friendly letter has now been designed for these cases. Dr. Ehrenfeld also stated that CSR needs to know what the community needs in order to make better self-referrals. She reiterated her preference to make self-referrals to the IRGs rather than to the study sections because of the subtleties involved in distinctions between study sections.

Dr. Ehrenfeld mentioned that CSR staff should be proactive in explaining to the scientific community how the referral system works. Perhaps this can be done through better use of the Web. Dr. Berns suggested writing brief articles for newsletters of professional societies.

Dr. Yamamoto noted that the topic of periodic review of study sections had been discussed during the last meeting, with an action item for developing review guidelines. He distributed the first draft of guidelines governing the periodic (every 5 years) review of the organization and operation of CSR study sections. These guidelines include: appropriateness of research topics and the scope of applications reviewed; capture of newly emerging research areas; performance of the SRAs; and performance of the chairpersons and group members. Other useful evaluation measures are the distribution of priority scores and the history of funded grants that were first reviewed by the study section.

Ms. Wisdom explained that the 1980 Paperwork Reduction Act impacts on our ability to collect data through the development and use of surveys. Any Government agency that wishes to collect data from 10 or more individuals must first obtain clearance from the Office of Management and Budget (OMB). This clearance process, which can take from 4 to 6 months, must include an estimate of how long it would take to complete the survey, as well as justification for the survey and time required. The plans must also be published in the Federal Register. Ms. Wisdom remarked that OMB is generally quite responsive to data collection requests when the intent of the agency is to improve customer service and to maximize performance results.

In response to a question about who would collect and analyze the data on the study sections, Dr. Yamamoto thought it would probably be the relevant IRG Working Group.

Dr. Matthews suggested that the periodic reviews be done not just at the study section level, but also at the IRG level. Dr. Wickens felt that it would be worthwhile to establish exactly how the data should be collected.

Under NIH conflict-of-interest regulations, the review of an application from a study section member may not be carried out by the member’s study section. Dr. Matthews stated that this practice has resulted in several questions: Is it a common perception in the field that members are therefore at a distinct disadvantage for review of their own grant application? Does this policy therefore keep potential members from serving on study sections? If these perceptions are valid, what mechanisms need to be put in place to assure members that they will receive a fair and competent review? If the perception is not valid, how can CSR better educate the community?

The data collected so far are based on two review cycles where study section members were reviewed within SEPs.. These data show that 61% of the members were within the 25 percentile. Of the 295 member applications reviewed by SEPs, 106 (36%) were percentiled against their parent study section, whereas 189 (64%) were percentiled against CSR. For those percentiled against the parent study section, the success rate was 58%, whereas the success rate for those percentiled against CSR was 62%. It was concluded that it doesn't matter whether member applications are percentiled against the parent study section or against CSR. Dr. Melchior commented that these data were extracted using the SEP member review code ("M") on member applications. She also indicated that members may do better because of more sophisticated grantsmanship.

Dr. Ehrenfeld mentioned that Dr. Patricia Straat had collected data showing that members did better in SEPs than in established "sister" study sections. Dr. Straat commented that she had collected these data based on committee management files (which would identify all study section members, including those reviewed in standing study sections) over six review cycles, and would make the data available to Drs. Matthews and Melchior.

Dr. Elliot Postow noted that age is a factor in success rates, and that the above data need to be corrected for age; Dr. Chien urged caution in the use of correction factors.

There was some discussion that the problem of review of member applications may be one of perception rather than an actuality. Dr. Jelsema commented that, in the past, her member applications were reviewed in a "sister" study section; although success rates were high, there was an overwhelming perception that this was not true, and they are now all reviewed in SEPs. Dr. Bradley commented that her members were asked how they wanted to be reviewed. The consensus was that they wanted a mini meeting that reflected the composition of their own study section. Dr. Bradley said that her members were satisfied even with a low score because they had input into the situation.

Dr. Ehrenfeld felt that the degree of satisfaction of members regarding their review would probably show a high correlation with whether or not they got funded. Options for reviewing the member applications are either to move the grants to another study section, to move the reviewers to another study section, or to allow the reviewer to take a year's leave of absence when that member's application comes up for review.

In his concluding remarks, Dr. Yamamoto listed several agenda items for the May 1999 meeting. These include discussions of the CSR budget versus CSR needs, further IRG experimentation with review practices, reports from IRG working groups, and information from the Panel on Scientific Boundaries for Review (although the final report is not due until June).

Seven action items were also identified:

• Finalize guidelines for chairpersons.
• Create draft document on SRA responsibilities.
• Create draft document on impact of referral policies on review.
• Create draft document on member conflicts.
• Create draft document on fellowship review
• Consider impact of shorter R01 proposal length.
• Create second draft document on policies for periodic study section review..

In the discussion that followed, other topics were also identified, namely, the contents of the web, summary statement preparation, periodic review of study sections, and increasing the participation of clinical researchers in review.

Drs. Yamamoto and Ehrenfeld thanked the participants for their perceptions and enthusiasm, which resulted in outstanding discussions and suggestions. There being no further business, the meeting was adjourned at 11:30 a.m. on Tuesday, January 12, 1999.

We do hereby certify that, to the best of our knowledge, the foregoing minutes are accurate and complete.

____________________
Samuel H. Joseloff, Ph.D.
Executive Secretary
Center for Scientific Review Advisory Committee

________________________
Ellie Ehrenfeld, Ph.D.
Director
Center for Scientific Review

__________________________
Keith Yamamoto, Ph.D.
Chairperson
Center for Scientific Review Advisory Committee

 

NOTE: These minutes will be formally considered by the Advisory Committee at its next meeting, on May 10 and 11, 1999. Any corrections or notations will be incorporated into the minutes at that meeting.