Meeting Summary

Opening Remarks

Dr. Larry Tabak and Dr. Jeremy Berg, Co-Chairs of the NIH Steering Committee Ad Hoc Working Group on Peer Review

The increasing breadth, complexity, and interdisciplinary nature of modern biomedical and behavioral research have created challenges for peer review, a critical component of the research enterprise. To adapt to the evolving scientific landscape, it is important for the NIH to ensure that the processes used to support science are as efficient and effective for applicants, reviewers, and NIH staff. In September, 2006, NIH leadership resolved that enhancing the NIH peer review system would be a top agency priority. Subsequent NIH staff brainstorming sessions laid the foundation for the current examination process that involves an external working group (the Advisory Council to the Director Working Group on Peer Review co-chaired by Dr. Keith Yamamoto and Dr. Larry Tabak) and an internal working group (the Steering Committee Working Group on Peer Review co-chaired by Dr. Larry Tabak and Dr. Jeremy Berg). In parallel, the Center for Scientific Review (CSR) has launched several peer review pilots and initiatives; these efforts will continue. The core questions to be analyzed during the NIH peer review self-study process include:

• Is the system currently used by NIH to support biomedical and behavioral research optimal?
• Do the best scientists/scientific ideas score highest in review?
• Are we engaging the best reviewers?
• Should we increase program flexibility to enhance peer review? If so, how?
• Should we increase review flexibility to enhance peer review? If so, how?

Other steps in the peer review self-study process include:

• Posting of a Request for Information (RFI) and an interactive Web site for soliciting opinion (July to August 2007)
• Advisory Committee to the Director Working Group holds a series of 5 regional town meetings (July to October 2007)
• Steering Committee Working Group holds consultative meetings within the NIH and creates a Web-based survey for soliciting opinion (July to October 2007)
• NIH leadership considers input from the RFI and both Working Groups and determines next steps, including pilots (February 2008)
• NIH staff design and initiate pilots and associated evaluations (March 2008)

Once plans are in place, the NIH will hold briefings for NIH staff, scientific societies, trade press, advocacy organizations, and Congress. Successful pilots will be expanded, commencing development and implementation of the new NIH Peer Review Policy in 2008.

Breakout groups of NIH staff dispersed to discuss six topics related to peer review, then re-convened to report the discussions to meeting attendees. Contextual summaries of the group discussions appear below.

Group 1: Challenges and Solutions for the NIH System of Research Support

Advances in science and technology have driven up the cost of research, and many investigators require multiple grants from different sources of funding to sustain their research programs. However, it is important that the research community recognize that NIH funding under current budget scenarios is more grant-in-aid as opposed to full project support for an integrated research program . It should also be noted that many factors, including the nature and culture of institutions, exert an impact on research funding. Changes in the workforce add further complexity: training periods have lengthened and the average age to scientific independence is now over 40. This shapes both the nature of applications being submitted and the reviewers available to evaluate those applications.

Many view the current peer review climate as an unnecessarily complex conglomerate of funding mechanisms, scoring schemes, and receipt dates. In a time when most investigators require at least two active grants to run a laboratory, staying funded is a time- and labor-intensive exercise that can compromise the practice of research. While better data on the process and outcomes of current review practices is required to design novel review experiments, possibilities were discussed. 1) Limits to support (of individuals or institutions), or required institutional contributions, may encourage cultural change by transferring some of the burden to institutions. 2) Multidimensional scoring metrics may enhance the quality of review by accounting for additional variables in an application. 3) Improved tools for select pay could increase program staff latitude for funding decisions.

Finally, percentiling is viewed by some as having the untoward effect of fixing the number of awards (i.e., for an especially small field at one end of the spectrum and/or for a large but declining field of study at the other extreme based on study section organization ). Thus, it has been suggested that this metric be balanced against other considerations since it places undue constraint s on the evaluation of research quality.

Key Points:

• Peer review is only one component of the overall biomedical and behavioral research decision making system.
• Peer review has become unnecessarily complex.
• Evaluation and fine tuning of current peer review practices should be studied before new untested practices are put into place.
• Peer review changes cannot address the fundamental imbalance between applications and awards.

Group 2: Challenges and Solutions of the NIH Peer Review Process

CSR has initiated several pilot programs to experiment with changes in the peer review process. These include shortening the review application, compressing the re-submission window, and various mechanisms for minimizing reviewer travel. Opinion varies considerably on the value of conducting reviews via phone or an electronic interface: Some believe that in-person communication is vital for meaningful discussion of the merits of a proposal in a group setting. An additional suggestion proposed by this group was to establish regional review centers around the country.

Engaging the highest quality reviewers is paramount to identifying the most meritorious science for NIH funding. To that end, the current IMPACII database could be replaced or updated to facilitate the search for the most appropriate reviewers for a given application. (CSR is currently experimenting with Knowledge Management software for this purpose.) Some feel that relaxing current limitations on gender, ethnicity, and geographic distribution, as well as inviting members of the public (when relevant) to serve as reviewers, could broaden the reviewer pool. Allowing reviewers a say in choosing proposals to review may help tailor expertise to applications.

Review context plays an important role in review quality: There is strong consensus that reviewers should be better informed about the specific actions and overall goals of peer review. Training, informal or formal, could help accomplish this goal, as could more explicit definitions of valuation terms such as “significant,” innovative,” and “impact.” In turn, SRAs could be encouraged to more actively manage, focus, and potentially shorten review discussions, perhaps with the use of a template. Many express enthusiasm for smaller panels, but the nature of the science being evaluated, along with the size of a given field, calls for flexibility with this parameter. Creating a database/computer interface whereby reviewers and SRAs can interact and exchange information, as well as more structured summary statements, may also facilitate knowledge exchange. Systematic evaluations of review quality and reviewers’ satisfaction could provide the opportunity for on-going enhancements to the review system.

Key Points:

• The feasibility and effectiveness of alternate review mechanisms should be assessed before new programs are implemented.
• Engaging high-quality reviewers is critical for identifying the most meritorious science.
• Enhanced training and information exchange between reviewers and SRAs may help focus review panel discussions.

Group 3: Core Values of the NIH Peer Review Process

An effective review should strive to balance feasibility, innovation, and impact. Considerable discussion focused on the boundary between review and program/IC staff in the two-tier review process. While many believe that review of an application should be based solely on scientific merit, other issues come to bear on the ultimate fate of a grant application, such as IC mission, NIH priorities (e.g. first-time investigators), and the federal funding climate.

It is critical that reviewers and SRAs be both enthusiastic and qualified to maintain the core values of the peer review process: objectivity, fairness, absence of bias, and a high ethical standard. Providing training, and potentially incentives, may help assure reviewer quality and consistency; however, the NIH must allocate adequate resources for this. In addition to targeting scientific need with the appropriate expertise, review panels should be composed of a diverse group of experts. Reviewers may include consumer advocates and community participants, when appropriate to the proposed research.

Key Points:

• The core values of peer review include objectivity, fairness, absence of bias, and a high ethical standard.
• Reviewers must continually balance a project’s feasibility, innovation, and potential impact.
• Review panels should be a diverse group of experts that provides informed, consistent, and reliable scientific advice.

Group 4: Peer Review Criteria and Scoring

The group reached consensus that no single score can easily characterize the scientific quality of an application. Further, scores are used differently across ICs with some following percentile order and others using program priorities in various degrees when making funding decisions.

Some members were very interested in having some measure of variance or scoring range included with the priority scores. Most members of the group agreed that multidimensional scores would better reflect the scientific quality of a proposal, including its potential impact on science and public health. Discussion also ranged over several possible strategies that could be piloted:

• Two-phase, “editorial-style” review would separate the technical aspects of a proposal from its larger impact on science. One way to accomplish this would be to create a Web-based review of methodology and feasibility, which would be followed by an in-person review (by broad-thinking, experienced reviewers) of the application’s innovation and impact.
• The National Science Foundation’s (NSF’s) approach, including binning applications and then rank-ordering them within a bin for funding. NIH could pilot with the ranking or without.
• Asking reviewers to score by criteria, as is currently done with NIH contract review. This approach recognizes that innovation and risk may vary and that applicants and program staff may want to see these separate indicators. Other indicators could be impact, or public health importance.
• Asking reviewers to give an additional, “what if” score. That is, what if everything were thoughtfully and effectively revised? This score would reflect whether there could be sufficient excitement for a resubmission.

In deriving pilots to test new scoring methods, it will be important for the NIH to use the science of decision-making and psychometrics (a blend of psychology and statistics that measures knowledge, skills, abilities, or personality).

Another suggestion involved better matching of the application to the review criteria. Specifically, if innovation is a criterion, then the applicant should be asked to describe the innovative features of the proposed research.

The group also thought that the applicants of unscored application deserved more information and that perhaps all applications could be scored but the streamlined ones would still not be discussed. This would give those in the bottom half a better idea regarding their status for possible revision and resubmission.

Key Points:

• Single scores do not reflect the totality of an application.
• Multidimensional scores would benefit applicants, reviewers, IC staff, and Advisory Council members.
• Psychometric methods and decision science should be used to design new scoring methods and criteria.
• Application and the criteria should be better aligned.

Group 5: Peer Review and Different Career Stages

The group adopted the widely held view that it is critical for the NIH to protect the integrity of peer review, and that any proposed changes should streamline the process without compromising scientific quality. A particularly challenging issue is the heterogeneity of review needs for different populations of applicants. There is a sense that reviewers struggle with dealing with both ends of the career continuum, e.g. those that are just starting out and those that are well-established, and that neither end is entirely well-served as a result.

It is important for the NIH to address the needs of starting investigators such as first-time applicants and those seeking their first renewal (especially physician-scientists). One way to do this would be for the NIH to establish rules specifying that all first-time and first-renewal applicants be reviewed en bloc within an existing study section or via separate review panels. It is also important to address the very different needs of established investigators – perhaps their applications could require less emphasis on technical details, especially if they were captured in many of their publications. Additionally, perhaps the MERIT award could be adapted to allow investigators to apply for long-term, stable support rather than being selected by IC program staff as is now the custom.

Other suggestions include creating distinct funding mechanisms tailored to career stage. Alternate mechanisms could define different scoring criteria, format, content, and award duration, although the group acknowledged that identifying career stage is challenging. Any proposed ideas should draw from past experience: the NIH “First” award (R29 program) was created for first-time investigators but failed due to its low financial cap and renewal restrictions. Since new investigators are disproportionately affected by tight budgets, the NIH has a keen interest in protecting the pipeline in the current fiscal environment. The most “out-of-the-box” suggestion was to reward established investigators with outstanding training and mentoring records with “legacy points” during review.

Key Points:

• Applicant populations at distinct career stages may benefit from tailored review panels and distinct funding mechanisms.
• New review criteria could identify and reward training and mentoring efforts.

Group 6: Role of Advisory Councils in the Second Level of Review

In the second level of peer review, NIH IC Advisory Councils consider review panel recommendations and determine the relevance of the proposed research to IC priorities and public health needs. This action attempts to ensure that the NIH receives advice from a cross-section of the U.S. population with regard to funding decisions. This group felt that enhanced and continual communication to the extramural community about the purpose and role of Advisory Councils—as a critical interface between reviewers, program staff, and the public--would be helpful. These communications should also emphasize that final funding decisions are based on multiple factors, not solely the priority scores. Best practices could be distributed among ICs: examples include orientation sessions for Council members; Web postings about Council roles and membership; and involvement of Council members in IC working groups and strategic planning exercises.

Advisory Councils are congressionally mandated and they play an important role in funding the most meritorious science; however, the specific roles and actions of Advisory Councils vary across ICs. The group recommended that ICs more systematically communicate with Councils about their role in the review process—in particular, that it is not a re-review of an application’s scientific merit.

Key Points:

• Advisory Councils are a critical interface between reviewers, program staff, and the public.
• NIH ICs should better articulate and communicate the role and purpose of Advisory Councils in the overall review process.