Referral & Review

REPORT ON REVIEW OF CLINICAL RESEARCH
IN THE CENTER FOR SCIENTIFIC REVIEW

Prepared for Ellie Ehrenfeld, Ph.D. Director
Center for Scientific Review, NIH

by Michael Simmons, M.D. Professor
Department of Pediatrics, University of North Carolina
June 22, 1998

       

Summary

As part of an overall effort by NIH to help improve the state of clinical research, the Center for Scientific Review (CSR) has undertaken an assessment of its capacity to review clinical research and the perceived fairness of that review in relation to laboratory-oriented research. Previous analyses have indicated that patient-oriented research is disadvantaged when reviewed in study sections with a small number and proportion of clinical proposals, arbitrarily defined as less than 30% of the portfolio and designated as "low" density. It is recommended that CSR address this problem in the following ways:

  1. Aggregate clinical oncological sciences proposals in a new Special Emphasis Panel (SEP).

  2. Aggregate clinical cardiovascular proposals in a new SEP.

  3. Create a new SEP for translational and patient-oriented research, in areas other than oncology and cardiovascular areas.

  4. Obtain secondary review input specifically from clinical investigators for proposals that remain in low-density study sections.

CSR should also pursue with interested Institutes the advisability of developing the capacity to review large clinical trials, outcomes research, and health services research within CSR.

Background

Considerable concern about the vitality of the clinical research enterprise has existed in the investigator community for several years. These concerns include the adequacy of resource allocation, the dwindling young investigator base, the emerging importance of human translational research, the paucity of investigator-initiated small clinical trials, and the fairness of the CSR review process.

Most of the above clinical research issues are matters of policy, program priorities, and budget allocations, and are only minimally related to CSR review processes. However, concerns with review fairness, CSR infrastructure, and CSR flexibility are regularly noted by critique from internal and external constituencies¹,²,³. It is clear that although small, the CSR issues are not trivial and are surprisingly pervasive.

During the last year, CSR undertook a study of its clinical research review processes. The Director, Dr. Ellie Ehrenfeld, retained Dr. Michael Simmons to assist her in this endeavor. It was concluded early on that enough studies had been done, and enough information has been collected in the several reports referenced above. Attention was therefore focused on defining solutions and implementation strategies. Dr. Simmons chaired a committee of experienced Scientific Review Administrators (SRAs) who provided insights into the present system and helped focus the development of new proposals. Drs. Ehrenfeld and Simmons met with Institute Directors and staff and with a wide variety of extramural constituencies. The information, suggestions, perspectives, and recommendations of all of these groups were not only helpful, but directly determined the new processes that are proposed for implementation within CSR.

It must be emphasized that substantial controversy exists within CSR, within the Institutes, and within extramural constituencies about what the problems really are. Reaching a satisfactory definition of clinical research even has been difficult. We have used the definition, quoted below, adopted by the NIH Director's Panel on Clinical Research, in their report to the Advisory Committee to the Director, NIH¹; in December 1997.

Clinical Research has three sub-types:

  1. Patient-Oriented Research

    Research conducted with human subjects (or on material of human origin such as tissues, specimens and cognitive phenomena) for which an investigator (or colleague) directly interacts with human subjects. This area of research includes:

    • Mechanisms of human disease

    • Therapeutic interventions

    • Clinical trials

    • Development of new technologies

  2. Epidemiologic and Behavioral Studies

  3. Outcomes Research and Health Services Research

Approximately 30% of all NIH funded research is "clinical." (If scored by the number of proposals funded, it is slightly less. If scored by the dollars expended, it is nearly 40%.) Although the majority of grant applications to the NIH are reviewed within CSR, much clinical research is reviewed within the Institutes and is conducted through program project grants, center grants, multi-center cooperative groups or cooperative agreements, or through grants solicited by Requests for Applications (RFAs) or Requests for Proposals (RFPs). Some Institutes have indicated a desire for an infrastructure and capability within CSR to accommodate such review.

The clinical research reviewed within CSR is principally investigator-initiated, translational research, or small single-center clinical experiments ("trials"). Some applications for technique, technology, and procedure development as well as for descriptive clinical research (population-defining, data collecting, etc.) also are reviewed in CSR. The latter variety of research is apparently dwindling and CSR review processes are thought by many investigators to be discriminatory against such research and therefore in part responsible for its low prevalence. Most attention was given to patient-oriented research and "translational research," which itself has different meanings to different constituencies. The major focus is on the subset of translational research that includes the small clinical experiment that requires a bench-to-bedside-to-bench (or bedside-to-bench-to-bedside) series of actions on the part of an investigator. Most often, such research falls into the categories of mechanisms of human disease or therapeutic intervention. The focus on translational research is based on a belief that the human experiment is not only progressively more feasible, but offers remarkable biologic power. In addition, the extramural and Institute constituencies have many concerns about the vitality of this variety of patient-oriented research and fears about the future of this investigator community, which has been described as a "threatened or endangered species."

Philosophy of Review: Implications of the Odd-Duck Problem

There has been the consistent assumption, within CSR and within the extramural constituencies, that productive and powerful review mechanisms can not be organized around narrow foci or themes, such as specific techniques, diseases, or model systems. An ideal combination within study sections of differentiated expertise, but with a broad biologic perspective, is believed to serve scientists and science very well. In reality, however, the composition and focus of study sections have varied widely.

Review within study sections may be organized around a biologic or clinical theme ("vertical organization") or with more diverse and eclectic expertise, focused around common biological processes ("horizontal organization"). Perhaps with new review criteria emphasizing impact, relevance, and creativity in addition to experimental approach, a more horizontal review composition is indicated. On the other hand, with studies of the human organism, pathogenesis, and specific diseases progressively more complex, vertical organization around a theme (such as oncology, diabetes, cardiology) might define implicitly a broad review perspective. The best review model among these organizational approaches remains controversial and highly debated, and will be the subject of recommendations to be provided by the Panel on Scientific Boundaries for Review, recently appointed by the Director of CSR. In the past CSR has accommodated both organizational models.

The practical problem of defining the correct commonality or integrative theme around which one organizes review is, in general, a major challenge. Identifying the commonality to review patient-oriented, translational research is especially daunting since the diversity of expertise required (disease, organ system, biology, technology) provides little integrative focus or glue. The de facto response to this difficulty has been to distribute clinical research applications (and specifically translational research applications) over a large number of biologically focused study sections. The consequence has been that a significant number of such proposals have been reviewed in study sections with a low number and proportion (density) of clinical proposals.

Based on an analysis during 1994²; 23 study sections reviewed two-thirds of the clinical applications. The "density" of clinical applications in these study sections ranged from 30% to 100%. Nineteen study sections reviewed no clinical applications. Forty-nine study sections reviewed one-third of the clinical applications, with densities ranging from 1 to 30%.

The analysis illustrated that success rates for patient-oriented research proposals reviewed in low-density study sections deviated substantially from those reviewed in high-density study sections. There may be a two-fold success advantage associated with a high density of clinical proposals within a study section. Regardless of the precise validity of the data from the 1994 report, it should be evident that an adequate volume of proposals (experience) and appropriate reviewers (expertise) are basic components of fair review.

New Proposals for Review of Clinical Research Within CSR

  1. The Low-Density Study Section Problem

    Analysis indicates two areas where review could be reorganized so as to cluster the clinical research proposals into a high-density study section: (1) cardiovascular science and (2) clinical oncology. In each of these areas, CSR should create a new special emphasis panel (SEP) for exclusive review of patient-oriented research, including translational research and small clinical trials. In those cases where an application would be moved from a low-density study section where it may have been reviewed before, or where the investigator has been reviewed previously, the change should be discussed with the applicant, encouraging input and self-referral. The details of implementation and piloting of these experimental SEPs have yet to be refined; it is anticipated that they will be initiated within the calendar year.

    Aggregating oncological science and cardiovascular science clinical review will address less than one-half of the low-density problem. For those clinical research areas where the diversity of expertise required precludes sufficient commonality to form a cohesive cluster, several possible approaches should be tested to address the odd duck problem. It is recognized that each of these approaches provides only limited solutions and potential problems, but preliminary experimentation may yield useful guidance.

    1. Establish a new SEP for Translational and Clinical Research for review of patient-oriented, translational research and small clinical trials applications that are collected from among proposals usually reviewed in several low-density study sections.

    2. For these patient-oriented proposals remaining in low-density study sections, experiment with mechanisms to obtain additional review input to address specifically the clinical significance and other aspects of the proposals. Such mechanisms might include:

      1. Assignment of the proposal to a high-density clinical study section for a supplemental opinion based on impact, relevance and creativity. This opinion would be forwarded to the Institute, along with the evaluation of scientific merit from the primary study section.

      2. Appointment of an ombudsman for clinical research in several low-density study sections. The ombudsman would be asked to provide a broad view of the study section's portfolio, and to carefully review and comment on the clinical proposals for impact, relevance and creativity, etc.

      3. Request of formal outside opinions from clinicians for proposals remaining in low-density study sections. A pool of experienced clinical investigators would be established, with broad clinical perspective willing to assist in this effort. Their opinions would be available to regular study section members as well as Institute staff.

  2. Large Clinical Trials, Health Services Research and Outcome Research.

Because few large population, descriptive clinical trials have been reviewed within CSR, there is presently little capability and limited capacity within CSR to perform these reviews. It is recommended that CSR consider creating a new review capability for population-based research within the Center. Many Institute based cooperative groups and cooperative agreements are aging; several Institutes anticipate increasing investment in large clinical trials. CSR should be able to accommodate those Institutes that would prefer to move some review to a more centralized venue.

There has also been much attention given in the past few years to health services research and outcomes research. It appears that activities in these areas will also increase across NIH. Presently, there is not adequate expertise within CSR, or adequate capacity, to offer review in these research areas. It is possible that investigators have been discouraged from applying to NIH in these research areas because of this deficit in CSR capacity and expertise.

Therefore, it is recommended that CSR establish a new SEP, the portfolio of which would include large (multi-center) clinical trials, outcomes research, and health services research. The regular members of the SEP would be experts in the design and execution of such trials. It is anticipated that members would be principally clinician reviewers who have a track record of conducting clinical trials. Appropriate experts in statistics and epidemiology may also be regular members. Because of the diversity of the clinical research that will exist within this SEP, CSR should rely heavily on ad hoc reviewers who will offer specific biologic and clinical expertise.

The time frame and implementation details of this proposal should depend on further discussions with Institute Directors and staff.

1. Nathan, David G., "The NIH Director's Panel on Clinical Research Report to the Advisory Committee to the NIH Director," December 4, 1997.

2. Williams, Gordon H., "An Analysis of the Review of Patient-Oriented Research (POR) Grant Applications by the Division of Research Grants, National Institutes of Health," November 21, 1994.

3. Kelley, William N. and Randolph, Mark A. (eds.), Careers in Clinical Research: Obstacles and Opportunities, National Academy Press, Washington DC, 1994.

[Referral & Review]

[DRG Home Page]