Meeting Summary

Opening Remarks

Dr. Larry Tabak and Dr. Jeremy Berg, Co-Chairs of the NIH Steering Committee Ad Hoc Working Group on Peer Review

The increasing breadth, complexity, and interdisciplinary nature of modern science have created challenges for peer review, a key component of a larger system: the biomedical and behavioral research enterprise. To adapt to the evolving scientific landscape, it is important for the NIH to ensure that the processes used to support science are as efficient and effective for applicants, reviewers, and NIH staff. It is critical that the NIH continue to draw the most talented reviewers.

In September, 2006, NIH leadership resolved that enhancing the NIH peer review system would be a top agency priority. Subsequent NIH staff brainstorming sessions laid the foundation for the current examination process that is a partnership between the NIH and the scientific community. This involves both an external working group (the Advisory Council to the Director Working Group on Peer Review co-chaired by Dr. Keith Yamamoto and Dr. Larry Tabak that will also select a series of science liaisons for further outreach) and an internal working group (the Steering Committee Working Group on Peer Review co-chaired by Dr. Larry Tabak and Dr. Jeremy Berg). In parallel, the Center for Scientific Review (CSR) has launched several peer review pilots and initiatives; these efforts will continue along with the analyses of prior NIH “experiments” with peer review practices. These are currently posted at XXX. NIH Institutes and Centers (ICs) have also contributed statements on their practices and policies, to help inform the ongoing effort. The core questions to be analyzed during the NIH peer review self-study process include:

• Is the system currently used by NIH to support biomedical and behavioral research optimal?
• Do the best scientists/scientific ideas score highest in review?
• Are we engaging the best reviewers?
• Should we increase program flexibility to enhance peer review? If so, how?
• Should we increase review flexibility to enhance peer review? If so, how?

Other steps in the peer review self-study process include:

• Posting of a Request for Information (RFI) and an interactive Web site for soliciting opinion (July to September 7, 2007)
• Peer review working group analyzes peer review literature (Ongoing)
• Peer review working group examines other federal agency approaches and OMB-sanctioned data (Ongoing)
• Economist analyzes economics of peer review (Imminent)
• Experts conduct psychometric analysis of study section models (Imminent)
• Advisory Committee to the Director Working Group holds a series of 5 regional town meetings (July to October 2007)
• Steering Committee Working Group holds consultative meetings within the NIH and creates a Web-based survey for soliciting opinion (July to October 2007)
• Dr. Tabak and Dr. Berg provide updates to IC Councils (Fall 2007)
• NIH leadership considers input from the RFI and both Working Groups and determines next steps, including pilots (February 2008)
• NIH staff design and initiate pilots and associated evaluations (March 2008)

Once plans are in place, the NIH will hold briefings for NIH staff, scientific societies, trade press, advocacy organizations, and Congress. Successful pilots will be expanded, commencing development and implementation of the new NIH Peer Review Policy in 2008.

Key messages from NIH Consultation Meeting I (July 18, 2007, NIH campus):

• Peer review changes cannot address the fundamental imbalance between applications and awards.
• The first stage of peer review is only one component of a larger system that includes IC funding decisions which are also based on programmatic priorities.
• Peer review has become increasingly complex, and pilot studies need to address impact on applicants, reviewers, and NIH staff.
• Recruiting and training optimal reviewers is key.
• The structure of applications and review criteria should be better aligned.
• Scores for the key components of an application might be more useful than only a single score and better communicate the ultimate potential of an application.
• Applicant populations at distinct career stages may benefit from tailored review panels and distinct funding mechanisms.

Key messages from NIH Consultation Meeting II (July 31, 2007, NIH campus):

• Recruitment and education of qualified reviewers is critical to the health of the research enterprise.
• Optimal reviewers should be communicative, open-minded, and willing to speak out with conviction.
• Interdisciplinary research projects have unique review needs.
• Separating review criteria may help reviewers identify innovation and potential impact.
• Selected peer review “experiments” to identify best practices can be done with existing NIH data.
• Evaluation of existing NIH peer review practices requires defining and articulating optimal review outcomes aside from priority scores.
• Improved communication between applicants, reviewers, and NIH staff would enhance review.
• Reviewing a person vs. a project has many advantages but may be impractical on a large scale.
• Extending grant funding for senior investigators may reduce review burden but may also have untoward consequences.
• Practice and opinion vary across NIH regarding the respective roles of review and program staff in the overall peer review process.

As the NIH self-study of peer review continues and evolves, several ideas have surfaced and themes have emerged. Group discussion began against the backdrop of these concepts, presented in slide format by Dr. Tabak:

• Funding the project vs. funding the person
• Funding innovation vs. funding incremental progress
  • Allow person to self identify if the application is intended to be innovative - use different criteria
• Review(er) quality
  • Incentives for reviewers
    • Mandatory service?
    • More flexible service
    • Increased support?
  • Rating the reviewers/SRAs
  • “The rich get richer” (aka- old boy network)
    • Anonymous reviews
  • How much information to provide reviewers appropriate context (“firewall”)
  • New models of review
    • Editorial board
    • Electronic review
    • “reverse” site visit (interviews/electronic dialogue)
• Different types of review for different types of science
  • SBIR/STTR
    • Academics usually not the “right” person to review small business
  • Clinical research
    • Can’t mix basic research with clinical expertise on panel
    • Trials tend to be new submissions
  • Interdisciplinary research
    • Content expertise vs. big picture; interpreters (editorial board?)
• A0 success rates are too low - queue clogs system
  • Pre-applications to provide meaningful advice to A0 applicants
• Scoring issues percentiles; binning; triaging; more information from scores desired
  • Two scores - application as received/best possible score

Open Discussion

Optimizing the Review Process

Soliciting advice and constructive criticism from colleagues and mentors before submitting a grant proposal may help applicants enhance the quality of the submission and better align their project with the NIH mission. In a similar vein, letters of intent sent in advance may help eliminate review burden by “weeding out” ideas that may not be fully developed. Establishing a two-score system that grades a proposal: i) as written, and ii) as if the same scientific problem were presented ideally, may allow fundamentally sound applications a second chance.

There exists broad consensus that close alignment of review criteria and application structure would contribute to optimal peer review. Public health value and potential for significantly contributing to a field are important criteria for applicants to articulate--as well as for review/program staff to assess. Check boxes that report success in meeting various application requirements may provide a relatively simple means to give feedback to applicants apart from a proposal’s numerical score. While it is the responsibility of program staff to discuss with an applicant the strengths and weaknesses of his or her proposal after review, this is not possible with triaged applications that have not been scored.

Meeting participants predicted that allowing applicants to participate in their own proposal’s review (electronically, or via telephone) may lead to an argumentative atmosphere, potentially revealing differences of scientific opinion rather than identifying factual errors. Although many consider anonymous review a good idea, most consider it unrealistic since applicant identity can often be easily guessed. The NIH may wish to re-consider, however, its policy to publicize reviewer rosters.

Reviewer-Related Issues

How does the NIH assure that it funds the “best” science? It is essential that the NIH recruit and retain qualified and enthusiastic reviewers who recognize that modern research is context-dependent, globally influenced, and tied to risk. Providing tools, examples, or templates may help educate reviewers on how to write critiques that are clear, concise, and sharply focused on big-picture concerns.

Scientific “fashion” can influence reviewer behavior, justifying out-of-field expertise to provide balance and reduce potential bias. Despite the fact that some in the community have expressed concern about mixing basic and clinical review expertise, participants who voiced an opinion at this meeting urged otherwise, noting that clinical reviewers provide essential input, especially in the assessment of medical significance and trial design. Establishing disease-specific study sections helps ensure that clinical reviewer expertise is both relevant and practical.

Some IC staff have reported that “in-house” reviews appear to attract higher quality reviewers but note that a grantee’s funding source may influence his or her decision to serve. Creating various incentives for study section service may help to regain status and prestige of the activity. Developing a community service component, akin to the National Science Foundation’s “broader aims” application criterion, may offer the NIH an opportunity to assess study section service in addition to other issues such as diversity, mentoring, and outreach.

Finding and Funding Innovation

Many in the community feel that the peer review process has become too conservative, failing to adequately recognize and reward innovation. Many ideas have been introduced to address this issue. One suggestion is to delineate to reviewers and applicants clearly--and in advance--that a defined percentage of a review score will be based on innovation. However, weighing an innovation criterion separately may fall short of its intended goal because of the subjective nature of the term “innovative.”

Other strategies include requiring a 1-page innovation statement that describes in narrative form how a project proposes to be innovative and/or address critical public health challenges. One participant noted that to facilitate the recognition of innovation, all Department of Defense review panels contain a substantial number--30 percent—of new reviewers.

Roles of Program and Review

The current prevalence of low A0 (unamended application) success rates suggests that reviewers may be conflating the roles of funding and review. It is important for CSR to continue to communicate to reviewers the need to focus on scientific quality, not on paylines and funding decisions.

While some NIH staff believe that reviewer-program relationships have deteriorated, others point to the importance of routine interactions between IC and CSR staff in fostering active communication and general good will. In particular, enhanced “upstream” interactions (for example, when solicitations are being developed and distributed to the research community) will likely help align goals and smoothen the review process.

Many program staff take very seriously the ability to use discretionary funding to pay “out of order,” although the extent of this practice varies across the NIH. The strategy can help address portfolio balance issues as well as potentially correct reviewer bias. Along these lines, some favor the prospect of providing program staff more control in the review process. However, consensus across the NIH is split on this issue, in that many see the process as unfair and prone to potential untoward effects on applicants. Binning applications, for example, shift responsibility to program staff, but the assignment of bin boundaries is largely subjective.

NIH Peer Review Experiments

The goal of the NIH peer review self-study is to approach peer review scientifically and to conduct experiments—not to change for change’s sake. Nonetheless, some meeting participants expressed worry about unnecessary re-vamping of a system that works. The NIH should carefully consider its actions including what may be lost with policy changes that impact peer review. The NIH also needs to be cognizant of the impact of any changes on bureaucratic load on applicants, reviewers, and NIH staff.

What constitutes a good review? For any experiment the NIH wishes to consider, a critical challenge will be defining metrics, although certain objectives--such as consistency of review outcomes--may be more readily measurable. It will be important that the NIH acknowledge the human element when evaluating “success” of any peer review interventions.

Currently, the NIH is collecting data to compare and contrast characteristics of the multitude of review systems in place throughout the world. The NIH will also begin psychometric studies to investigate its current practices in peer review as well as analyze past NIH IC peer review experiments. It is anticipated, for example, that Intramural Program review data that is currently being gathered will likely provide a rich source of information.

Application review based on people vs. projects is currently being piloted with the recently instituted NIH Director’s Pioneer Award and NIH Director’s New Innovator Award programs. Experience to date suggests that despite the apparent advantages of this type of review approach, one downside is a significant increase in NIH staff workload.

Key Messages from the August 16, 2007 NIH Consultation Meeting III:

• Obtaining constructive criticism before grant submission may help applicants improve proposal quality.
• Context is a key component of quality peer review.
• Incentives, tools, and evaluation may enhance reviewer performance.
• Identifying innovation is important but difficult due to its variable definition and interpretation among applicants and reviewers.
• Better communication between program and review staff may reduce the perceived “firewall” between the two groups.
• Peer review experiments need to assess both outcomes and bureaucratic load.
• Psychometric analyses will help account for the human element in the practice of peer review.