PEER REVIEW NOTES
May 2000

FROM THE CSR DIRECTOR'S DESK

In previous issues of Peer Review Notes, I outlined the various reorganization and evaluation activities currently underway in the Center for Scientific Review (CSR). Considerable progress has been made on all fronts, and, in this issue, I provide you with an update of these activities.

The final Phase 1 report of the Panel on Scientific Boundaries for Review (PSBR) was accepted by the CSR Advisory Committee on January 10, 2000, and posted (http://www.csr.nih.gov/review/reorgact.htm) on our web site. In undertaking a comprehensive examination of the organization and function of the CSR review process to ensure that the review system is aligned with the current scientific landscape, the Panel made several recommendations for broad stroke changes in CSR. The proposed structure of the Integrated Review Groups (IRGs) calls for clustering broad approaches to biological problems associated with a given system/disease. This recommendation acknowledges the advent of molecular medicine, where biochemistry, genetics, molecular and cellular biology have become tools applied to virtually all fields of health-related research. Molecular medicine applications will be reviewed in the context of the biological questions addressed rather than lumped in discipline-related study sections where they will compete against each other. The proposed IRG structure also includes clusters for basic scientific discovery and methods development that applies to no specific system or disease, and creates venue for reviews of design-driven research. Lastly, it is recognized that the structure must accommodate cross-cutting fields such as aging or development.

By accepting the Phase 1 final report of the PSBR, the Advisory Committee launched CSR into Phase 2. The Phase 2 implementation plan, now posted on our web (http://www.csr.nih.gov/events/implementplan.htm), will design the study sections that will populate the 24 IRGs proposed in the final Phase 1 report. The Phase 2 process will be gradual and will involve all stakeholders: scientific research communities, NIH program and CSR review staff, and members of the PSBR. The first step in the process will be to determine how a typical round of applications distributes among the 24 proposed IRGs. To this end, CSR is performing a "mock referral" of all applications received for the May 2000 Council meetings. Abstracts from these applications will form the working material to be used when considering IRG and study section structure throughout this process.

The next step in the process will be to form Steering Committees for each IRG or group of IRGs, depending on the science in the IRGs. The purpose of the Steering Committees will be to review the mock referral, and to create Study Section Boundaries Teams (SSB Teams). Each SSB Team will consist of research scientists from the relevant scientific communities, a member of the PSBR, CSR staff, and other NIH staff. The purpose of these teams will be to sort the abstracts into new study sections and determine the scientific boundaries for each study section within the IRG. As each SSB Team completes its proposal for the study sections, the proposed description will be posted on the Web for public comment, and input from the broader research community will be sought. It should be noted that the seven recently reorganized IRGs (AIDS, three neuroscience IRGs, and three behavioral sciences IRGs) will not be involved in this reorganization. All Steering Committees will begin work in the next two years with full implementation anticipated within three years. The first Steering Committee and SSB Team to be established will be for the Hematology IRG.

Concurrent with the reorganization activities of the PSBR, Working Groups have also been established to evaluate the structure and function of our current IRGs. There has been some confusion surrounding the difference between these two activities. The SSB Teams are unique groups convened for the single purpose of designing new study sections. They should not be confused with IRG Working Groups, which are convened on a regular (every 5 years) cycle to assess the performance of the study sections in an IRG. These latter reviews are an ongoing activity that will likely provide information to SSB Teams, but they are a distinct and separate activity, part of our program for periodic monitoring and review of study section function.

In the last issue of Peer Review Notes, I reported on the Working Group assessment of the Biophysical and Chemical Sciences (BPC) IRG. The Working Group for the Oncology IRG (ONC) has now also completed its report, concluding that the IRG was functioning well but was overloaded. They recommended that the IRG create one new study section, that mechanisms be developed to increase interactions among the study sections, and that there be some adjustments in the scope and boundaries of the constituent study sections. In addition to the ONC Working Group, the Working Group for the Cardiovascular Sciences IRG (CVS) has met and the final report should be available shortly. Working Groups for the Musculoskeletal and Dental Sciences IRG (MSD), for the Immunological Sciences IRG (IMM), and for three of the Neurosciences IRGs will meet in the June/July timeframe. The plan is to complete all Working Group activities by Fall of 2001, in time to be useful to the PSBR in formulating final plans.

Another ongoing effort over the past 18 months has been to develop a set of formal surveys that will provide feedback from our partners and customers on how we are accomplishing our review mission. There are currently two surveys approved by the Office of Management and Budget (OMB), one that will be used to survey reviewers, and the second to survey applicants. The survey instruments have been through numerous iterations over the past year, including public review by the CSR Advisory Committee, by several CSR staff groups, and by external advisors for OMB compliance. The reviewer survey was piloted in four study sections last February, resulting in some minor adjustments to the survey questions. The reviewer survey is now ready to be administered to all study section members during the June round of reviews.

The applicant survey has been tested on a small sample of 100 investigators. Only 40% responded, but these responses nonetheless required us to modify the instrument. The rollout of this survey will likely occur next year, and will be sent to a sample of the applicant population rather than to all applicants from a given round. To maintain anonymity of the respondents to these surveys, we have retained a contractor who will collect survey results. Only aggregated results for each study section will be reported. These will be transmitted to me, and shared with the Scientific Review Administrator (SRA), the IRG Chief, and the Division Director. Some generic information, such as reviewer workload distribution, aggregated for total CSR, may be made available to a broader audience, but no information that can be traced to any specific study section will be released. Finally, I am pleased to report that we have been able to hire 9 new SRAs so far in fiscal year 2000 and are still actively recruiting. There are many new activities in CSR, with subsequent new demands on our staff. The number of applications reviewed at CSR is increasing due to many new initiatives issued by the Institutes. The complexity of science is also increasing, and, as more and more applications become multi-investigator and multi-disciplinary, they increasingly do not fit into existing study sections and their review becomes labor intensive. CSR is trying to develop mechanisms to respond to the workload created by the issuance of new initiatives, including development of a way to couple our budgetary needs to new initiatives that we will be required to review.

In our numerous efforts to improve CSR's review process by modernizing our review committees and by instituting procedures to monitor and evaluate the study sections' operations, I have been extremely gratified by the enthusiastic and willing response of both the internal and external scientific communities to contribute in many ways. NIH cannot maintain its high quality peer review system by itself. Many thanks to everyone who is participating in creating the new CSR.

Ellie Ehrenfeld
Director, CSR

CSR WELCOMES FIVE NEW SRAS

CSR continues its active recruitment of new scientific staff. Since the January 2000 issue of Peer Review Notes, five new SRAs have joined our ranks.

Dr. Prabha Atreya will be responsible for the review of Small Business Innovation Research (SBIR) applications assigned to the Biochemical Sciences IRG. Dr. Atreya received her Ph.D. in Biochemistry from the Memorial University of Newfoundland, Canada. Immediately prior to joining CSR, she was a senior staff fellow at the Food and Drug Administration, conducting research in the development of vaccines for pediatric and other respiratory viral diseases.

Dr. Michael Chaitin is SRA of the Visual Sciences A Study Section in the Molecular, Cellular, and Developmental Neurosciences IRG. He joins us from the University of North Texas Health Sciences Center at Fort Worth where he was Associate Professor in the Department of Anatomy and Cell Biology. His doctorate is in Molecular Biophysics from Florida State University.

Dr. Jerry Klein has joined the Oncological Sciences IRG where he will be responsible for the review of SBIR and Small Business Technology Transfer (STTR) applications. His Ph.D. in Biochemistry was received from SUNY at Albany, New York. Prior to joining CSR, he was Adjunct Associate Professor in Oncology at the Johns Hopkins School of Medicine

Dr. Stephen Nigida, Jr. will manage the review of SBIR applications for the Immunological Sciences IRG. Dr. Nigida was previously Scientific Director of the Immunochemistry Laboratory with Spring Valley Laboratories, Inc. in Woodbine, Maryland. He received his Ph.D. in Microbiology and Immunology from the University of Illinois at Chicago.

Dr. Richard Rodewald is now responsible for the review of fellowships in the Cell Development and Function IRG. Until 1997, he was Associate Professor of Biology at the University of Virginia at Charlottesville when he became Program Director in Cell Biology, Division of Molecular and Cellular Biosciences, National Science Foundation. His Ph.D. was received from the University of Pennsylvania in Biochemistry.

REVISED IRB POLICY

It has been NIH policy that when applications submitted to NIH involve research with human participants, the applications were required to have Institutional Review Board (IRB) approval at the time of submission or within 60 days after application receipt date. Since fewer than half of all applications submitted to NIH are funded, the NIH is modifying this policy (http://grants.nih.gov/grants/guide/notice-files/NOT-OD-00-031.html to reduce the burden on applicants and IRBs. Beginning with applications submitted for the January 2001 Council round (generally, those applications submitted for the June/July 2000 receipt dates), IRB approval is not required prior to NIH review of an application. As part of the peer review process, peer review groups will continue to consider whether each application includes the necessary safeguards to protect the rights and welfare of research participants.

No grant award can be made without IRB approval. Therefore, following NIH peer review and notification of the priority score and percentile, institutions should proceed with IRB review for those applications that have not yet received IRB approval and that appear to be in a fundable range. The term "fundable range" does not signify a certainty of funding. Guidance is currently under development that will assist applicants in determining their status relative to a particular Institute/Center's fundable range.

This change in NIH policy is intended to provide applicant institutions with the flexibility to reduce workload burdens currently faced by many IRBs. While NIH will no longer require advance IRB approval, the applicant institution may still determine that certain lines of research (e.g., scientifically or ethically controversial research) or mechanisms of research (e.g., multicenter clinical trials) should receive IRB review prior to submission of the application.

This change is consistent with the requirements of 45 CFR 46 (The Common Rule). At this time, this flexibility is provided only to IRBs. Applications including research with animals will continue to require review by the Institutional Animal Care and Use Committee (IACUC) at the time of submission or within 60 days thereafter.

To help explain the Revised IRB Policy, the NIH has prepared the following series of Questions and Answers:

What is an Institute's "fundable range?"
A "fundable range" is not to be confused with an Institute's pay line. The "fundable range" will include all applications that fall within an Institute's pay line and also applications that extend a number of priority score/percentile points beyond. This range will vary from year to year and often between review rounds. The intent of widening the range of applications beyond the nominal pay line to proceed with IRB review is to provide Institutes the discretion to select applications beyond the pay line for special consideration, particular program relevance, etc.

How will the investigator and institution know that an application is in the fundable range and therefore that IRB review should proceed?
With notification of an application's rating, the investigator will also be notified if the application is in the fundable range. Notification that an application is in the fundable range will also be provided to the institutional business official who signed the application. At that same time, other information may be requested, such as updated "Other Support" in the case of modular applications.

Will there be times when NIH might expect/request IRB approval earlier than permissible?
At times, IRB approval may be necessary before submission of an application. This could occur, for example, with a particularly tight time line for an RFA; or for certain instances when end-of-fiscal-year funding requirements might demand earlier IRB review and approval.

Will this change in policy affect subsequent noncompeting awards?
 If an institution chooses to wait until after peer review to conduct IRB reviews for competing applications, it should be sure to schedule subsequent "annual" IRB reviews so that they occur prior to the submission of noncompeting continuation applications, which are due two months prior to the budget period start date.

What degrees of flexibility are institutions given in determining when, what, and why some applications might be given IRB review earlier than permissible?
 Institutions have extensive flexibility in determining when an application receives IRB review prior to funding. The official NIH announcement refers to examples such as particular mechanisms (e.g., cooperative clinical trials) and sensitive lines of research (e.g., HIV research) that an institution might feel should be subjected to IRB review prior to application submission. However, institutions may also determine other cases or instances that they feel more comfortable requiring IRB review prior to application submission.

Will NIH peer reviewers be expected to do anything differently in their review of applications involving human subjects research that have not yet received IRB approval?
Peer reviewers have always paid careful attention to the review of human subjects protocols regardless of the fact they have already undergone IRB review. It is expected that peer reviewers will continue with that practice. The NIH expects neither more nor less from its peer reviewers in this regard.

PROJECT DESCRIPTIONS IN SUMMARY STATEMENTS

NIH is changing current practice on how to prepare the "Description" section of Summary Statements. Under current practice, the Description is prepared in a number of different ways, including: (1) the applicant's abstract is used without any modifications, (2) the SRA uses a modified version of the applicant's abstract, (3) the SRA writes an alternate description of the proposed research, (4) a reviewer modifies the applicant's abstract, or (5) a reviewer prepares an entirely different description of the proposed research. Some of these practices have caused problems. For example, on occasion, sensitive information contained in the body of the application has been included in a re-written description. This sensitive information was then made public in the NIH database of abstracts for funded projects. In the interest of uniform review practices across the Institutes and Centers at NIH, and in the interest of preventing disclosure of inappropriate, sensitive and proprietary information, the NIH has instituted the following policy:

The current PHS 398 application instructions alert applicants to the fact that their abstracts (i.e., Description) will become public information if a grant is awarded, and that they should not include proprietary or confidential information in their abstract. Use of the applicant's unedited abstract as the Summary Statement Description should thus eliminate the possibility of publication of inappropriate, sensitive and proprietary information.

DISTRIBUTION OF PROGRAM ANNOUNCEMENTS TO REVIEWERS

For the past several years, SRAs in CSR have sent relevant Program Announcements (PAs) to study section members for their use in preparing critiques. Since one study section may serve multiple Institutes at NIH, and since each Institute may have multiple PAs, this resulted in large volumes of paper sent to reviewers. Recently, a trans-NIH workgroup re-examined this policy. Based on their recommendations, SRAs will no longer send PAs for R01s to the reviewers in hard copy. Instead, SRAs will provide reviewers with a list of web site URLs where the appropriate PAs may be found. Only for unusual PAs (e.g., announcements for career awards, small grants, or exploratory grants) will SRAs continue to send hard copies to the reviewers.

SCIENTIFIC MISCONDUCT, CONFIDENTIALITY, AND THE REVIEW PROCESS

Members of Scientific Review Groups (SRGs) occasionally ask what they should do when they find what appears to be scientific misconduct while reviewing an application. Misconduct in science is defined as "& fabrication, falsification, plagiarism, or other practices that seriously deviate from those that are commonly accepted in the scientific review community for proposing, conducting, or reporting research. It does not include honest error or honest differences in interpretations or judgments of data" (Professional Ethics, Vol. 7, No.1, p.9). Instances of actual misconduct are rare.

It is extremely important that reviewers bring any such allegations to the immediate attention of the SRA in charge of the SRG or Special Emphasis Panel (SEP). If possible, these allegations should be pointed out in advance of the review meeting. The SRA will then bring the allegation to the attention of the Research Integrity Officer (RIO) for CSR. Each NIH Institute/Center has a RIO who is responsible for reporting issues of misconduct to the Agency Research Integrity Liaison Officer (ARILO) and to the Agency Extramural Research Integrity Officer (AERIO). The ARILO, AERIO, RIO and the Office of Research Integrity evaluate the allegation and make a determination on the misconduct issue.

To ensure fair peer review, it is essential that the process for dealing with allegations be clearly understood by all reviewers. Because the review process must continue while the allegation undergoes further assessment, under no circumstance should a reviewer raise the allegation while the review is in progress. Under such circumstances, review of that application could not be completed without bias, and the application must be deferred. If the allegation is discovered at the time of the meeting, it would be appropriate to request a "break" and discuss it privately with the SRA.

Confidentiality in the peer review process is also imperative. Discussion and evaluation of applications must be confined to the SRG or SEP members. Reviewers should never discuss applications with non-committee members (without obtaining permission from the SRA) or with applicants either before or after the review meetings. A reviewer should immediately refer any inquiries from an applicant to the SRA. All materials related to the review should be disposed of at the meeting, and all final critiques should be given to the SRA for inclusion into summary statements. If members of SRGs adhere to these practices, the effectiveness and integrity of the peer review process will be maintained.

THE CSR WEB SITE
(http://www.csr.nih.gov)

Have you checked out the CSR web site lately? Our CSR Web Team is hard at work developing and enhancing our site, making it both informative and user-friendly with improved navigational tools to make your visit more productive. The goals of our initial effort have been to provide current, complete, and useful information on peer review at CSR, to improve navigation between critical review elements, and to develop a tracking system for monitoring our site and ensuring that all displayed information is current. In preparation for this work, we asked several user groups to evaluate our site. Three groups were defined, one composed of CSR staff, one composed of NIH Institute staff, and one composed of users from within the scientific community. We received many helpful suggestions for improved communication, several of which are now incorporated. Highlights of new additions to our web site include:

· Flash Announcements: Our home page now contains hot news items or last-minute information that could affect the submission of your application.

· Information and Reports on Peer Review: Recent peer review activities and reports are posted in our News and Events Section for 90 days. Reports are also posted "permanently" in the Referral and Review Section of our web site. Look under "Reorganization Activities" (http://www.csr.nih.gov/review/reorgact.htm) for information on the activities of the Panel on Scientific Boundaries for Review. For detailed information on the operations of our study sections, check "Special Reports on Peer Review Topics" (http://www.csr.nih.gov/events/specialreports.htm) where recent reports include Guidelines for Study Section Chairs, Role of the SRA, and a report on The Review of Member Applications.

· Small Business Applications Site: A new site describes the Small Business Innovation Research (SBIR) and Small Business Technology Transfer Research (STTR) programs. This site (http://www.csr.nih.gov/review/sba.htm) now names current CSR SBIR/STTR review committees and the responsible SRA, links to SBIR/STTR application instructions and receipt dates, and links to application forms.

· New Integrated Review Group Roster Index: Both membership rosters and meeting rosters are now available (http://www.csr.nih.gov/Committees/rosterindex.asp). In addition to the membership roster for each standing study section, we now provide the past three meeting rosters. These meeting rosters list both permanent and temporary members present, and provide the total scope of expertise that may be available at the study section meeting.

· New Meeting Scheduler: Our new meeting scheduler makes it easy to find when CSR study sections meet http://www.csr.nih.gov/Committees/meetings/ssmeet1.asp(http://www.csr.nih.gov/Committees/meetings/ssmeet1.asp). The scheduler can provide lists of study sections meeting on a given date, or for a given council round, or can provide information on an individual study section.

· Easier Site Navigation: With the addition of hot links for easy navigation, users are usually only one click away from study section rosters, study section descriptions, and study section meeting dates. Links are also provided for email communication with study section SRAs.

As these efforts are nearing completion, a Phase 2 effort will be launched to undertake a graphic "facelift" for our site, along with incorporation of additional peer review information and improved navigational tools. We welcome your suggestions as we develop our web site. If you have comments, suggestions, or ideas for additional information you would like to see available on the CSR site, please submit them to Dr. Patricia Straat (straatp@csr.nih.gov).

In the meantime, stay on top of our site for important information on the peer review process and for last-minute information that could affect submission of your application!

[Referral & Review]

CSR home page