Meeting Summary

October 8, 2007 – New York

Meeting Context and Review of Ongoing Activities
Dr. Lawrence Tabak
Director, National Institute of Dental and Craniofacial Research, NIH; Co-Chair of the Working Group of the Advisory Committee to the NIH Director (ACD) on NIH Peer Review

NIH is engaged in a self-study, in partnership with the scientific community, aimed at enhancing peer review. Science is increasing in both its breadth and its complexity. The interdisciplinary nature of biomedical science is creating new challenges for the NIH system to support biomedical and behavioral research.

As NIH continues to adapt to rapidly changing fields of science and ever-growing public health challenges, we must ensure that the processes we use to support science are both efficient and effective for applicants and reviewers alike.

We are seeking input from the scientific community and our own staff at NIH. Two committees have been formed by the NIH director to drive this process. The first is the Advisory Committee to the Director (ACD), which Dr. Yamamoto and I co-chair. A number of people on this committee have served on the previous formal report of peer review, the so-called boundaries report. Internally, there is the NIH Steering Committee Working Group on Peer Review, co-chaired by Dr. Jeremy Berg (director of the National Institute of General Medical Sciences) and myself.

The Center for Scientific Review (CSR) is engaged in a number of initiatives around many of the mechanical issues related to peer review. Our working groups are working in concert with CSR to ensure a coordinated approach.

We had a request for information (RFI), where we solicited input on a number of issues. The web site closed in the first week of September. We asked for your input on the challenges of the NIH system of research support, the challenges of peer review per se, and your solutions to those challenges. We asked questions related to core values of the peer review process, asked for input on the current criteria for peer review as well as the scoring system, and asked if the current peer review process is appropriate for investigators at all stages of their academic careers. We received over 2,000 responses. If you have not yet responded, please e-mail us. We will review your comments and take them very seriously.

Additional activities include the following:

• Dr. Yamamoto and I held two deans teleconferences with about 100 participants.
• We are engaged in a series of regional town meetings: October 22, in Washington, D.C., with patient advocacy groups; and on October 25, in San Francisco, with the academic community, investigators, and administrators.
• ACD working group members, deans, and NIH institute directors were asked to select individuals to serve as scientific liaisons between the ACD working group and their respective communities as a means of enhancing our outreach.
• We created a common web site for feedback.
• The internal steering committee working group has been working with our own staff to solicit information from each institute and center.
• Last summer, we held meetings with our own staff at NIH.
• We analyzed the peer review literature.
• We are analyzing how agencies worldwide approach peer review.
• Our efforts will be informed by the National Science Foundation’s (NSF) recent report on the impact of proposal and award management mechanisms.

We will use all this input to determine next steps. We envision this will take the format of a series of pilot interventions designed to enhance issues that have been raised by the community, and that these pilots will be decided upon in the late winter or early spring. They will be initiated along with their appropriate evaluations by March. Armed with the results of the pilots, we will begin to develop an implementation plan. The subset of successful pilots will be expanded. Ultimately, we will have a new peer review policy.

Goals for the Meeting
Dr. Keith Yamamoto
Executive Vice Dean, School of Medicine,UCSF; Professor, Cellular/Molecular Pharmacology and Biochemistry/Biophysics,UCSF; Co-Chair of the Working Group of the Advisory Committee to the NIH Director (ACD) on NIH Peer Review

The title of my presentation comes from the charge that Dr. Zerhouni gave the working group: “Fund the best science, by the best scientists, with the least administrative burden.”

Another of my favorite quotes in this arena came from the Surgeon General in 1945, right before the current NIH peer review system as we know it was formalized: "The only possible source for adequate support of our medical schools and medical research is the taxing power of the federal government. . . Such a program must assure complete freedom for the institutions and the individual scientists in developing and conducting their research work." It is germane to what we struggle with today.

In 2005, Tom Cech said, "Discovery and innovation are to some extent taking place in spite of, rather than because of, the current policies and practices of major biomedical funding agencies."

What has brought us to this point where these two statements seem so strongly in opposition to each other? Many of us would acknowledge and defend fiercely the notion that peer review is really the only system, not just the best system, for funding the best science. And for those of us who have seen how science is funded in other countries, it's an even stronger statement.

Nevertheless, intrinsic conflicts are built into the peer review process, and these are not going to disappear. One conflict is reviewer self-interest, in that the reviewers draw from the same resource pool as the applicants. Another conflict is reviewer conservatism. To have the best system, we need to have the best scientists participate in the review process. These scientists created the prevailing paradigms, and so they will defend them.

Between 1945 and now, both the doing and the reviewing of science have changed dramatically. Some of the problems that are now encountered in peer review have emerged because of the advances in biological research. Science has a much broader scope and greater complexity than before. It requires more subspecialty knowledge and expertise, and it is increasingly performed by multidisciplinary teams. And so the old idea of defining a small focused group of individuals who can handle any grant application has really been challenged by this change in the way research is done.

The culture of reviewing also has changed. The budget doubling that we enjoyed between 1999 and 2003, followed by the flattening that we've seen over the past four or five budgets, has driven an explosion of applications. Institutions were emboldened to hire new faculty, build new buildings, and found new institutes and centers. When the budget flattened, people responded by sending in more grant applications. With that has come a vast increase in the number of reviewers, most of them ad hoc. Twenty years ago, CSR’s predecessor used 1,800 reviewers. Last year, CSR used 18,000 reviewers. The mandated number of formal members on each study section hasn't changed, so it means that the vast majority of these are ad hoc, with relatively fewer senior scientists.

Of course, as this perception of constrained resources sets in for all of us, the process has become much more adversarial. Twenty years ago, being a member of a study section was an honor. It was a lot of work, as it is now, but people really enjoyed making their assessments, sharing them with and listening to a small group of people, and moving through the process of responsible peer review.

It's good news that science has changed in the way that it has. The question is how can we adapt and change the review system to accommodate that and to anticipate what is to come. We're looking for your best ideas: substantive changes that can make a difference. We hope you'll feel free to share ideas that you think are bold and could actually have an impact. All of the following, and more, should be on the table:

• Review Criteria and Focus: NIH has been staunchly dedicated to a project-based focus. Should we pick people over projects? Should we do both in some way?
• Application Structure and Content: When it comes to science support, NIH grants are the longest. Is it really important to have 25 pages? Does page length compromise any particular group of individuals? Regarding content, we all worry about the fact that, especially when resources are constrained, everything begins to focus down on concerns about the experimental detail and preliminary data. Review systems get very conservative when the money is tight. The CSR is already experimenting with changes in the content and organization of applications. Are there ways to think about that process that would address some of the conflicts we have heard about?
• Reviewer Mechanisms and Mechanics: A lot of the esprit that people enjoyed and that made the system work in many people's view has gone away. Does that mean we need to change the way study sections are organized or function?
• Reviewers and Review Culture: Relatively few senior scientists are involved in review. Is there a way to recover some of that esprit that people felt was really important to the way things work?
• Scoring: Is the current scoring method the right way to do it? Are there ways to make things more fair? Should grants be rated instead of scored?

Statements/Proposals from External Scientific Community Offering Specific Strategies or Tactics for Enhancing NIH Peer Review and Research Support

Meenakshi Alreja, Ph.D., Assistant Professor of Psychiatry and Neurobiology, Yale University School of Medicine

I would like to specifically address the difficulties often encountered in interpreting and addressing reviewer comments. The outcome of your resubmission depends entirely on how you react and respond to the critique. Often the comments are very hard to interpret. Sometimes it may be as simple as not knowing whether the reviewer is making a general comment or expecting you to do something in order for the grant to be fundable.

There should be a standard and objective format for reviewer comments, and these comments must be clear and precise. The comments should begin with a summary of the specific aim, followed by specific comments, and clearly stated expectations or other suggestions. These comments should appear in priority order. When reviewers go to a study section, they should present their critique in an organized manner.
There should be a consensus right there in the study section so that when the applicant is asked to do something, it is the view of the study section and not just one reviewer.

This mechanism would be extremely useful for the younger and newer study section members, because they would learn to read everything carefully and comment on it in an objective and precise manner. Second time around, when the application goes to a new study section or new reviewers, it would also be easier for the new reviewers to understand the previous reviewers’ comments and expectations.

Christine Ambrosone, Ph.D., Chair, Department of Cancer Prevention and Control, Roswell Park Cancer Institute

The most burdensome aspect of the review is the several hours required to read and critique each grant, knowing that many ultimately will be triaged. Although some of the grants may benefit from reviewer comments for successful resubmission, a number of them will never be funded due to either lack of significance or flaws in feasibility and/or approach.

To reduce paperwork and the burden on reviewers, and to fund the best possible science, I suggest a blinded pre-proposal process. Blinding would reduce bias toward seasoned investigators, allow the research question to be judged on its own for significance, and allow a more even chance for new investigators to come forward with novel and important ideas.

The process would include the following features:

• The pre-proposals, 2 to 3 pages at most, would briefly present a convincing argument for the significance of the proposed research, and briefly describe the research approach to assess feasibility.
• A team of 10 to 20 reviewers, perhaps the standing study section team, would be asked to read and score all pre-proposals within that section. There could be check-off boxes for reasons for triage.
• Similar to scoring and ranking of abstracts for association meetings, scores would be posted electronically by the reviewers. Pre-proposals in the lower half of scoring would not be asked to present a full application for review.

Additional points:

• Allowing reviewers to weigh the significance of a proposal against the cost when reviewing full applications would allow for better decision making.
• The ACS review process, where reviewers rank proposals at the close of the review, gives reviewers the ability to consider multiple aspects of the application.
• The group dynamic in an in-person meeting is essential for the review process to work. During electronic reviews, most reviewers do not weigh in on applications outside their assignments. Rotating the location of meetings for West Coast or Midwest reviewers may help to reduce reviewer burden.
• There is widespread concern about the ability of young investigators to successfully compete against seasoned researchers in today’s funding environment. Study sections should be formed that specifically review proposals only from new investigators.

 

Sumru Erkut, Ph.D., Associate Director and Senior Research Scientist, Wellesley Centers for Women

I recommend a limit of 10 pages for R01, R21, and R03 proposals. The guidelines for preparing proposals should emphasize information relevant to (1) the need for the study, (2) its significance, and (3) the innovations it brings. The review of background research should be incorporated into the proposals based on relevance to establishing these three points. The methods section should include a discussion of the challenges of the proposed novel approach and a flexible plan for dealing with different contingencies.

My rationale is to increase funding for proposals that incorporate novelty and originality and to move away from a review process that rewards cautious approaches. I realize this is not a new idea. What is new is a call for the institutes and centers to post examples of 10-page proposals that have passed peer review or mock peer review. Until investigators have models to follow, they will continue to be reluctant to submit innovative but risky proposals. Similarly, until reviewers have concrete examples to learn from, they will penalize such proposals.

Lee A. Hebert, M.D., Division of Nephrology, College of Medicine, Ohio State University

Reviewers should be more accountable for the accuracy of their critiques. I recognize that it is difficult to recruit qualified NIH reviewers, and that placing additional burdens on them is problematic; however, equally problematic is the devastation to a research program when a reviewer’s blatantly incorrect criticisms torpedo a life’s work.

The rebuttal process by appeal to the Council has a long turnaround time and can be likened to attempting to repair a radio using telephone poles. It is very hard to control and likely to cause more harm than good. I propose a relatively simple and quick rebuttal process along the following lines:

• Each reviewer would be required to indicate his/her “major” criticisms – those that would cause at least a one-category change in the reviewer’s scoring of the application (e.g., an “Outstanding” would decrease to “Excellent”).
• A major criticism would require documentation (ideally, relevant references). In some instances, a succinct and clearly laid out rationale would be sufficient.
• Investigators would receive their summary statements in a timely fashion (e.g., within 2 to 4 weeks), even if this would mean sending the reviewers’ comments unedited by the SRA.
• If an investigator determines that his/her application received an unfavorable score because of factual errors, challenges to up to three major criticisms would be allowed. The challenge would take the form of a written rebuttal that would be no more than 150 words per criticism, to be submitted within 1 week of receiving the summary statement.
• The rebuttals would be reviewed by the review group’s chair and SRA. If both agree that each of the investigator’s challenges are correct, the scoring of that reviewer would be censored and the score recalculated. Because the reviewer’s factual errors likely would have influenced the scoring by other members, a feasible and prudent strategy might be for the SRA and chair to select three to five members who would be especially knowledgeable regarding the application and to ask them to rescore it. If either the SRA or the chair disagree with any of the investigator’s rebuttals, the rebuttal process would cease. The investigator would be penalized with a limit of one application revision. This would be a strong incentive for the investigator to ensure his/her case for rebuttal is very strong.
• A report card would be kept on reviewers. Those who repeatedly make factually incorrect major criticisms would be dropped from the rolls of the NIH reviewers.

Lois A. Lampson, Ph.D., Associate Professor of Neurosurgery, Brigham and Women’s Hospital & Harvard Medical School

A better outcome to the review process, and fairer reviews, could come from taking better advantage of the time spent at study section meetings. It is of no value for the reviewer to read through a point-by-point critique if no one is listening. The following would be much better:

• Start out with a pithy summary in the reviewer's own words of what this project is about – not aim by aim, but what is the thrust of this project.
• Next, very short statements of the key strengths and weaknesses.
• Then a brief statement of the reviewer's judgment about how to balance those chief strengths and weaknesses.

With this system, there's a real discussion, the reviews are fairer, and better decisions are made. The meetings are more enjoyable, and they're shorter.

Fernando Nottebohm, Ph.D., Dorothea L. Leonhardt Professor, Director of the Field Research Center for Ecology and Ethology, Rockefeller University

The review process can be improved and streamlined by recognizing that if the proposed research is truly pioneering, many of the relevant facts will not be available at the onset. The review process should also recognize that the PI will troubleshoot problems as they arise, and that information to do so is usually readily available.

For these reasons, it should be possible to do reviews based on shorter proposals. I would suggest a maximum of 5 pages (that's what the Europeans are doing) that do not get into the nitty-gritty of every procedure and possible contingency, and that require less crystal-balling of the future. In addition, I suggest that proposals be scored (on a scale of 1 to 10) on the following items: the PI's qualifications to conduct the work proposed; resources required; suitability of collaborative arrangements in place; the PI's track record on innovation; the PI's publishing record; research quality; originality of work proposed; match between choice of material and questions asked; suitability of techniques to achieve proposed goal; importance of work proposed; potential for starting new field, new insights, or solving important problems; potential impact on health sciences; and opportunities for intellectual growth of the PI's laboratory and for student training.

Ann-Gel Palermo, Chair, Harlem Community and Academic Partnership, the National Network of Community Partners Engaged in Community-Based Participatory Research

The language NIH uses to define and describe peer review should be clear and consistent so that applicants, peer reviewers, and potential peer reviewers are on the same page as to what is meant by certain terms.

Funding announcements do not clearly communicate whether community-based individuals are eligible to serve as peer reviewers and do not explicitly state the value of having peer reviewers who have experience with health disparities in their own communities. Clarify key terms, and have greater outreach and publicity.

The current peer review process does not properly prepare peer reviewers for their role, particularly for Community-Based Participatory Research (CBPR) grants. The solution is to provide training, including the specific topics that pertain to the particular review of the funding announcements such as CBPR.

Study sections and review panels rarely include an equitable number of people from diverse communities reflected in the research being proposed. We recommend that a standing study section be formed to review CBPR proposals. This section should be composed of an equal number of academics and community members with CBPR experience and expertise who have been properly prepared and led by designated community- and academic-based co-chairs.

In order for appropriate CBPR review criteria and processes to be developed by NIH, it must acquire staff with knowledge and expertise in this area. A strategic plan should be developed that incorporates a wide range of strategies to recruit and retain such staff.

CBPR applications are not being reviewed using appropriate criteria. We recommend changing review criteria for CBPR proposals and encourage NIH to consider the peer review criteria and guidelines that have been advanced by the AHRQ evidence report on CBPR.

It is important for NIH to use a different process to review applications from new investigators, investigators from new applicant organizations, and investigators from community-based organizations. Review criteria and scoring procedures must be developed and applied uniformly. We recognize that until a cohort of CBPR researchers is developed, it will be challenging to identify senior-level research mentors for K awards, T awards, and F submissions employing a CBPR approach. We recommend inclusion of seasoned community partners as mentors for investigators. We also recommend that a specific K&T grant program be developed that would support investigators in becoming CBPR experts.

Moshe Sadofsky, M.D., Ph.D. , Associate Professor of Pathology, Albert Einstein College of Medicine of Yeshiva University

There is tension between big science/big labs and small labs/individual investigators. That tension is lost in a review process that is uniform for all kinds of grants. The R21 program was created to make smaller grants for innovative research of shorter duration. But this is not living up to its expectations, because the reviews are often done competitively with large R01s.

To what extent do we want to support the numbers of scientists? Larger numbers of smaller grants is, on the whole, of benefit to society. Although that's not an argument I can make on the basis of the science being delivered, it is still a valid consideration. Much more thought has to be put into societal implications of these processes.

There has to be an incentive for more experienced people to become reviewers. One suggestion might be that in return for your service as a reviewer, you might get an extension to one or more of your existing grants.

Phyllis R. Strauss, Ph.D., Matthews Distinguished Professor, Northeastern University

If NIH is to meet the goals of its mission statement, it has to find a mechanism to support innovative work. With this in mind, here are my specific comments:

• Truly innovative research proposals are routinely triaged. For research that is not easily pigeonholed into existing cookie-cutter study sections, there needs to be a greater use of special emphasis panels, or some sort of special review.
• There needs to be a real reward for innovation. A way to accomplish this goal is to provide a reasonable sum to each program director to select one or two projects each year for support that she or he deems truly innovative and worthy. This has been done by NSF.
• More progressive reviewers are needed. There must be strong incentives to get people to serve, like extending their grants with funds while they participate on the panel. Furthermore, many PIs with small labs are rarely asked to review.
• Small labs with a single grant have a serious disadvantage in competing with large labs, even when the work is of outstanding quality, because reviewers tend to count publications as a measure of productivity. One solution is to permit the PI to list several of the most relevant publications for the application at hand.
• Cap the amount of funding from NIH to any single lab, based on total income to that lab.

Susan J. Vannucci, Ph.D., Professor of Neuroscience in Pediatrics & Institute of Human Nutrition, Columbia University

I propose a pre-proposal process where applications are divided into three groups:

1. The clearly outstanding applications that require only very brief discussion and study section time.
2. The middle group (the 1.6s to the 2s, currently), with grants that have a lot of merit but are not quite there; these can be discussed more extensively in study section, and the applicant can get very clear instructions as to how to return with a fundable grant.
3. The grants that are never going to get there. That message should go back to the applicant right away. The constructive comments should be aired very briefly in front of all members so that inappropriate comments can be avoided.

Additionally, it is inappropriate and unproductive to review grants from established investigators in the same study section for the same pots of money as new investigators. I propose that for established investigators, if you have been doing your work, you've been very productive, and you're submitting a continuation of this work, you submit your progress and your general plans and be reviewed in a different section from new investigators. If you want to change your direction, then you go back with everybody else and compete for that original money.

Finally, a major problem with all these grants flooding the NIH is the absence of any institutional support. If the institutions want their 50% and 60% indirect costs, they should be required to support the faculty they hire.

Emerging Ideas
Dr. Lawrence Tabak

I'd like to share with you some emerging ideas based on my read of the responses we’ve received so far. I prepared this in advance of this meeting and haven't put these ideas in any priority order. It's only presented to facilitate discussion. We want you to think of ideas in addition to these.

• Review criteria focus and the application structure.
• Reviewing the project vs. reviewing the person.
• Retrospective vs. prospective reviews for competing applications, as is true for the intramural program.
• Blinding the first phase of review.
• Too much emphasis on methodology.
• R21s have become mini-R01s, which is killing innovation.
• The wisdom of more rather than fewer people to review applications. How do you do that? Electronic review; two-stage review, similar to a first-stage technical review, anonymized or not; using this sort of second-stage review where the wise men and women would sit around tantamount to being journal editors.
• The need to establish some sort of applicant-reviewer dialogue to correct factual errors in the first stage of review.
• New models of review for different types of science, e.g., community member involvement in community-based research.
• Interdisciplinary research and the challenges of review.
• SBIR: Are academics the right people to review small businesses?
• The queue that is clogging up the system.
• Pre-application: to provide rapid identification and separate the very competitive from the truly non-competitive ideas so we can give meaningful feedback to A0 applicants. Where do you do that pre-application review? We've heard everything from electronic review by peers all the way up through and including NIH staff.
• Administratively consider funding for those applications with minor and easily correctable deficiencies rather than placing them back in the queue.
• Provide more useful feedback to applicants. Provide scores to applications that are currently unscored. Discuss all applications, particularly the subset for new investigators. Tell applicants unambiguously if the application is not recommended for revision and resubmission, thereby avoiding the “ugly baby” syndrome.
• How do we maximize review and reviewer quality? How much context should we provide reviewers? Should reviewers have some sort of portfolio analysis of either the NIH or an institute-specific current investment in areas? Should there be a discussion of priorities, either at the NIH level and/or the institute and center level? What is the value of looking at the budget and other support?
• Regarding incentives for reviewers, add time to extend grants, and supplement extant grants for part-time administrative help. Make service more flexible, perhaps two rather than three times a year.
• Should there be mandatory service?
• Provide in-depth training for reviewers. Retrain folks to focus on the strengths and not the weaknesses of applications, and to focus on the potential impact and not methodology.
• Rate the reviewers. Rate the SRAs. Don't publish reviewer ID, such as they do at NSF. At the other end of the scale, pictures and home phone numbers should be placed on the Internet.
• Have the study section rank each proposal review at the end of the meeting and thereby revisit the score of each application to ensure consistency and fairness.
• Redo the scoring system to break old habits.
• Review new investigator applications separately. Review different mechanisms either in different study sections or at different times of the meeting. Sharply reduce the number of mechanisms.
• How many R01s are enough? Are there too many overlapping R01s? Require a minimum percent effort as a principal investigator.
• Should R01s remain the gold standard of investigator success?
• Indirect costs.
  

Compilation of Key Points During Four Open Discussion Periods During Meeting

Appeals/Rebuttals

• NIH should consider having in-person rebuttal, with senior scientists serving on something like a supreme court. Applicants and reviewers alike would be much more focused.
• One fundamental element of any democratic society is checks and balances. Unfortunately, NIH doesn't have any of that in its peer review system. There ought to be an appeal system, a rebuttal system, that truly works instead of on paper only.
• The peer review system is the only part of our society where there is no accountability. There should be an effective rebuttal system, a blind level of assessment, and a panel to review the quality of the review. And there should be some type of penalty or consequence for a bad review for a reviewer.
• This is a system of power. As such, it should have three branches. Now it is completely dominated by the executive branch. We need a judicial one, and without a real appeal process, this system will always be biased.

Applicant Feedback/Interaction

• If the reviewer comments are so focused that the applicant can simply incorporate that into their resubmission, resubmission success will increase, which is problematic.
• The major element missing from review is sort of an adversarial context where you could have one or two people representing the PI. One way of doing that would be to give the written critique to the PI before the committee convenes, and then make the PI’s response available to the committee as part of the discussion. Another idea is to ask the PI to recommend possible members of the committee, and then pick one.
• More information should be provided to unscored applications.
• I would like to see the tension between reviewing the merit of a proposal vs. providing feedback for the rewrite to be handled differently. The junior scientists could definitely benefit from developmental suggestions on how to rewrite their proposal. The feedback should go beyond evaluation of the merit. Senior researchers could maybe just do with an evaluation of the merit.
• To avoid confusing the PI, study sections should reconcile their comments before sending them.
• I support the idea of the applicant getting comments long before the study section meets, and also being able to come to the study section. We normally give each grant about 25 minutes in a study section. There is plenty of time to cut off all the discussion and to allow the applicant to present and rebut in person.
• When you write a review of a proposal, you should pretend that the person is sitting across the table from you. You should not put anything in the review you are not willing to say directly to that person.

Application Length

• For social scientists, it is a struggle to submit short proposals. For bench research, it's probably okay to do 10 pages, or even the 5 pages as you suggested.
• I think we could do with reducing proposals to 10 pages; I'm not in favor of reducing them to 5. I've reviewed European applications, and the problem is that in the 5-page application you end up basing your review on the person's track record, which biases against young investigators.
• When reviewers get short formats, they expect the proposals to be broad, and the details count against you. So in suddenly shifting from a long proposal format to a short one, reviewers need time to adjust to these short proposal formats.
• If we decide to go with a shorter application, we should also make sure reviewers have different expectations, because they often look for these kinds of details.
• Reduce the number of pages from 25 to 15. It works very well with the R21s.
• Those “10-page” DOD proposals have several additional pages of information.

Criteria for Funding

• A grant may not necessarily be the most innovative, but it may have a potential to have great impact, so these criteria need to be more properly balanced.
• Large laboratories with enormous amounts of resources are, by definition, very productive in terms of numbers of publications. If one were to look at productivity on the basis of “resources in vs. results out,” productivity might have a different meaning and perhaps might bring some more innovation back into the process.
• Significance and innovation should be front and center in both the review and application process. The methods and all the other things should be related to the significance and innovation of the first two parts.
• It is crucial that we develop a set of criteria that are used and are written about directly in our reviews.
• There should be a set of criteria, such as “innovativeness,” each of which could have its own score.

Electronic vs. Face-to-Face Reviews

• I just rotated off a study section where the electronic reviews were thought to be drafts, and each reviewer had the opportunity prior to the study section to look at what the other reviewers said about the applications for which they were responsible. This tended to eliminate the kinds of outliers who had a particular agenda. We could then go back and essentially do a final draft of the review after the discussion. This improved both the discussion and the quality of the summary statements, and it didn’t take any more time.
• When a reviewer is biased or may not be as expert as they claim in a written review, this becomes more apparent when they have to defend their score face-to-face in front of their peers at the study section meeting. And so I would urge the maintenance of the use of face-to-face review committee meetings.
• I favor an electronic review process. I have just moved to teaching electronically, and I have been struck by the amount of forethought that goes into having to teach online. You have to think about what you're going to say and prepare adequately, and you cannot shoot from the hip or use excessive verbiage.

Evaluation

• We should use the annual report process more, showing good productivity and verifying productivity through publications and collaborations. If the productivity is not strong throughout the period of a grant, the funding level could go down somewhat, leaving more money for some other individuals.

Evaluation – Reviewer/Review

• At the end of each study section, the members should evaluate the other members. People who are considered inappropriate should be removed.
• Applicants should be allowed to privately review the critiques, and just like the review panel gives a score to the grant, the grant applicant should be allowed to give a score to the written reviews.
• It's astonishing that there's no process to review reviews, especially in an environment where there is less and less funding.

Expertise

• Orphan areas (such as pain) need to be looked at in a new way, and CRS needs to revise its criteria for selecting reviewers so that these areas are clearly covered.
• Very frequently, NIH fails to provide expert review. A big database should be created, and the primary and even secondary reviewers should be required to have at least a single publication in the field of research that is proposed. Whenever the reviewer agrees to review a project he or she has no clue about, that should be considered a serious misconduct.
• It's very difficult to find an appropriate study section and true peer review for projects in small clinical specialties, particularly where there's going to be translational research. There must be better focused study groups, where you're being assessed by people who do the same type of research and who understand what you're doing, what the clinical issues are, and what specific problems you are you trying to deal with.

Funding

• I would support a system whereby the top 20%, the most outstanding, would get full funding. I'm sure that investigators potentially at the 11th percentile to the 20th percentile would like some funding to be able to get things moving rather than having none at all.

Grant Budgets

• When you submit a grant, you get nitpicked quite often on small details in your budget. To the best of my knowledge, nobody ever nitpicks the overhead costs. Many of us don't know what we're getting for our overhead, so NIH needs to look into how institutions are using their overhead costs to support research.
• NIH should consider drastically slashing indirect costs. You could double the number of grants and still get the same kind of indirect costs that other foundations offer for grants.
• If PIs knew that the efficiency of how they're planning to spend the money was actually considered in rating the grant, not just as a post hoc comment, that might encourage them to trim some of these outrageous budgets.
• If many of the budgets could be cut by up to 30% or 40%, that would allow us to fund more proposals.

Grant Mechanisms

• Career development grants are geared toward an academic kind of structure, and not all of us are in that kind of setting. This makes it a little more difficult and sets the bar higher for us.
• I suggest an intermediate K award, for maybe the 11th to 20th percentile. If you get 3 years of $100,000 or $125,000, set it aside, and once you get your R01, you lose that. It's better to have a piece of pie than no pie at all.
• I'm hopeful the R13 process can be slightly simplified. Also, the timeframe for the reviews of R13s might be shortened, because sometimes when an organization is submitting an R13 grant, until the grant gets reviewed and a funding decision made, the organization doesn't know if it has funds to hold the conference.
• People want some kind of bridge grant mechanism for established labs that would keep them going for 1 or preferably 2 years. In the current funding climate, many excellent productive labs are folding because they don't get a grant renewed.
• The Health Research Alliance is a group of funders of health research, voluntary health organizations, private foundations, and so forth. We are all very interested in the idea of bridge funding in the current climate, and we would like to work with NIH to get proposals that didn't quite make the payline. As private funders, we could possibly pick it up. I heard some horror stories from other associations that couldn't quite work with certain institutes and/or CSR to make this happen, so if we could pave that path, that might be a really productive area. [Dr. Tabak noted that there is precedent for this.]

Incentives To Serve

• People who come to study section should not automatically get a year bonus. Only those who truly perform, who are truly outstanding, should be rewarded.
• There are plenty of rewards to being a reviewer: scientific camaraderie, the opportunity to sit for a couple of days with your peers and discuss scientific ideas, the opportunity to improve your own grant-writing skills, etc. The perception of “no reward” permeates through the research community and just exacerbates the problems. We need to somehow create a cultural change where people consider it an honor to be on a study section.
• There is no reward for being a reviewer, so we should give more value to the opinions of the chairs and have the chairs eliminate the review, for instance. If the R21, for example, is criticized for not having preliminary data, eliminate that review from the group.
• I don't think it's reasonable to give monetary compensation to people on study sections. There are some perverse incentives there, and it would only exacerbate the lack of funding. However, one thing that could be considered is to allow anyone who serves on a study section to be relieved of the NIH deadline. [Dr. Yamamoto commented that the system to relieve deadlines is now in place.]

Junior Investigator Reviews

• Junior people ought to be reviewed against their peers, using different criteria.
• Having "quotas" to fund grants at the R01 level for junior people will ultimately do them a disservice. Unless they're competing against their peers, it's going to be that much more difficult to get the second R01 if they've been cut some slack in getting the first R01.

Miscellaneous

• Study sections should take the approach that PIs are innocent until proven guilty. It’s up to the study section to show why a grant is not worthy of funding.
• In the areas of science that represent very rare conditions, sometimes it's hard to be as diverse in terms of study participants as it is in the larger research areas.
• Applicants should have more input into the choice of study section.
• The institutes exist as separate silos, and it's very hard to know where to send interdisciplinary research applications. The idea of grants being dual coded is not true in reality. We've had grants go to the second institute within their funding range, and they say, "Sorry, we don't fund other folks' grants."
• Study sections need some definition or guidelines, because there really are no guidelines and no standard of quality. The SRAs are not maintaining that.
• Regarding the requirement to have a hypothesis, there's nothing wrong with saying there are three major challenges that I'm going to work on, or there are four outstanding questions that need to be addressed. A hypothesis makes you take a stand one way or the other, and then it gets into this Talmudic argument of whether I've answered it, and so forth.
• Add a list of abbreviations/acronyms to the review form.
• It is important not to include other support as part of the biosketch.
• NIH should create a mentorship network database. It would contain names and contact information for senior scientists who are willing to mentor junior scientists with their grant applications. It could include brief CVs about their publications, etc.
• Reviewers have to agree to follow the rules of that particular review session. You can circumvent a lot of problems by giving a little more power to the chairs.
• I wish that some of what transpired today would go back to members of Congress who sit on the committees that appropriate money for science and that they be made aware of the painful decisions that you people have to make.
• The best way to manage conflict of interest and accountability is through transparency. There has to be some way where absolute transparency can be accomplished if we really want to foment the best science.

NIH Role

• NIH could learn a lot from NSF. It is not appropriate in many cases for the NIH staffers to have to remain silent, particularly when reviewers are taking a left turn from the stated review criteria.

Pre-Proposals/Blind Reviews

• A 3-page pre-proposal on innovative research across fields would be rejected because people can't conceive of how we can put together fields like this.
• The pre-proposal process could work, as long as there were 20 senior permanent members involved.
• Pre-proposals will make the system completely corrupt. A sales pitch will certainly have a much better chance to get funded than a rigorous scientific proposal.

Program Announcements

• The reviewers at CSR don't see the program announcements, and they don't know what the reviewers are writing towards. The whole idea of program announcements needs to be looked at again, because there's a lot of misinformation about that, especially with regard to the junior investigators.
• With tight budgets, what is the real value of program announcements? [Dr. Tabak commented that giving priority to program announcement-related proposals is not uniformly done throughout NIH.]

Project vs. Person

• I'm against reviewing the PI, because I think it's inherently a biased system that favors older and more funded PIs over younger ones.

Scoring/Ranking

• DOD is way ahead of the curve when it comes to scoring. Different types of grants and different components of the grants are reviewed and scored on a scale of one to five. They look at scientific merit, innovation, and a variety of other things.
• I would strongly suggest that you look at the spread (like standard deviation) of scores for each of the applications. And instead of looking at the mean, maybe look at the median instead, which is a resistant rather than a robust measure. This would involve criteria scoring and then an overall score. All of the scores would be subjective.
• The review process should be simplified. I wonder if there could be a scoring rubric that could help guide not only investigators, but also review panels, through a systematic way of evaluating and giving feedback.

Staffing Panels

• Many of the best scientists cannot review 36 applications a year, which is more or less what is required of study section members. The study section should include a number of more experienced people who are selected to attend once a year, and maintain a group that attends the three sessions.
• Perhaps we should have retired scientists be reviewers full-time.
• The chairs of all study sections should be senior scientists who no longer do NIH-sponsored research. They need not review grants; they will have the time and motivation to provide firm and wise leadership. They could meet as a group periodically and set standards and discuss common problems.
• “Peer” is generally defined as a holder of an active NIH grant. But there is a need for more senior reviewers who are not conflicted. There are lots of fine scientists in Europe, Asia, and industry, none of whom need be conflicted, and many of whom might want to participate in the process.
• The makeup of peer review committees should change and should require funded investigators without incentives and without favors.
• We should have the associate professor level as a cut point for junior investigators serving on review committees. These people frequently are burdened with a tremendous number of grants to review, and it distracts them from getting their own work done. They are under pressure at the review committees, and they frequently make off-the-mark comments, using this as a forum to impress other members.
• When someone is funded, they have more time in their lives during those first 2 years. And maybe that's the time we really should be encouraging a mixture of new and well-funded/established investigators to come on study section.
• Patient or consumer advocates should be included in panels. These people are well-educated, they understand the issues, and they can bring something to the table that goes beyond the details and the minutiae of the methods. The other possible impact is that you might get an increase in NIH funding, because these people have a lot of influence on where the money goes.
• It is probably best to have a mixture of new and senior investigators. New investigators can learn a lot by being on study section.
• In order to bring more senior investigators into the review system, we should not burden them with eight or nine grants to review each cycle. Each reviewer should be assigned four grants – two primary and two secondary – three times a year.
• NSF and DOD bring in what they call consumer reviewers. This is becoming increasingly important for certain kinds of grants, particularly community-based participatory research.
• It might be a good idea, as many journals do, to suggest five or six reviewers at the end of the proposal. You don't have to use those people, but what it does is suggest the kind of people who might be qualified to judge this body of work.
• If a PI has gotten more than a certain amount of dollars of funding, let's say over 8 years, then that PI should be compelled to participate in the panel for at least one or two rounds.
• I'd like to take issue with the suggestion that assistant professors or junior faculty not serve on study sections. They should have the opportunity to serve as ad hoc members, primarily so they can learn how the process works and learn what good grants, great grants, and mediocre grants look like.
• The most valuable education that I had by far in terms of grant writing was by starting as an ad hoc reviewer, as an assistant professor. I would very much discourage any plan that prevented that from happening.
• The review or study section should be internationalized, bringing the best reviewers from international societies.

Study Section and Review Models

• The triage system is inherently unfair and should be abolished. Instead, something like a bullet system should be used, where the reviewers should be very clear about their opinions on all major issues.
• To make final funding decisions, some elements of random choosing (e.g., after grading a pool of applicants) might work very well.
• SRAs and the people on the study sections need to have some accountability and some organization where we review things that are submitted. For example, when you submit your review, there is a grace period where it's online. Maybe during that time, someone should be making comments.
• The Canadian Institutes of Health Research grant applications for our R01 equivalent are shorter – 12 or 13 pages. We have a faster turnaround time for the application process, with two competitions a year. If you're unsuccessful in the first, there is enough time to resubmit in the second without having to sit out.
• Perhaps it might help the process even at the study section level if the different kinds of grants with different missions were separated from each other and were not competing for the same money.
• In terms of electronic interaction, it would be extremely productive if the critiques that are now posted right before the meeting were posted a month earlier and opened up to the investigator. At that point, the investigator could submit a 5-page addendum, where he/she would address the content of the reviews, inaccuracies, etc. Then, an additional reviewer/mediator would be appointed to take a fresh look at the entire package. If the reviewers themselves were allowed to submit 1-page addendums to their critiques, this would address a number of issues in terms of shortening the review cycle. It would also make the reviewers more responsible for what they write, because they would then have to defend it in person.
• The MRC in Canada was populated by very senior distinguished scientists who were not conflicted. They sent out the grants for review across the world to those people they thought had the expertise to review them. Then these senior people made decisions based on their knowledge of the field. Journals do essentially the same thing.

Training Reviewers

• To be a reviewer, it doesn't really matter if you're the best scientist or if you're a good or intermediate scientist; you must be qualified to review grants. For something as serious as this, there should be a training program.

Closing Remarks
Dr.Yamamoto

I want to talk about some of the complexities that come with putting together a process as large and important as this.

Some have said there would be no need to be here if there were enough money. But other comments underscore the point that the way research is done has changed, and this process needs to be as responsive to the science of today and tomorrow as it was to the science of yesterday. So one could argue, perhaps, that it's not really simply the case if there were money, there would be no problems.

Second, we all recognize that it is a huge task to set aside the kind of time required to serve on a panel. And it really is service without compensation. The dangers that come with some formulaic solution, like another year on your extant grant, for example, is that as soon as a decision of that sort is made, study section membership becomes an entitlement. And when you sense that you need another year of support for whatever reason, suddenly you are on the phone saying, "I haven't gotten my year of service on study section yet, and I need it now." So think about that, weighing it against what obviously is a large service responsibility that we're asking of the community.

Some have commented on the importance of discussing and scoring all applications, and of having junior faculty serve on study section for mentoring purposes. One could argue that a mentoring component is not the responsibility of the peer review system.

Discussions about the ways that a short grant application or choices of people over projects will penalize junior faculty/young scientists, is a point understood. But we do this all the time. Every single young person who comes into the system to write an NIH grant has made it to that point by writing 2- and 3-page applications for support. They know how to do that; they don't know how to write a 25-page application.

There is the notion that we should be more diligent about scrutinizing annual reports to ensure the work is being done. Instead of using them as sort of de facto requirements for getting the next year's funding, the proposal is to make real funding decisions based on demonstration of ongoing progress. One could look at another side of that coin and say the problem is that it will make grants and grant applications even more conservative. In fact, maybe what you want is not to have that kind of scrutiny – that there should be some tenure grants that just say, "You've done a fantastic job. We really like this idea. We know that it may do nothing, but if it works, it will have enormous impact. And so here's 10 years' of money. Just show us every year that you're still a faculty member or an investigator wherever you work. And the money will keep coming."

Somebody suggested that senior scientists who are no longer active in the process could serve as study section chairs. There's not much currency in this profession other than how many grants, square feet, and post-docs you have. This does keep senior scientists trying to hang onto their lab space longer than they might, because there's really nothing else they get credit for. So what should senior scientists do? And is it true that we could capitalize on senior scientists’ perspective by having them serve in advisory roles of this sort?

One idea was that NIH decide it's going to pay 20% of the grants, and it will find a way to do that by paying the very top ones the full amount, and then scale back down until they get to 20%. The other related idea is to randomly pick awardees. This would say to our community and our congressional representatives, we are inside the resolution limit of this system.

The idea of nominating reviewers also is very interesting. And somebody else said let's make sure they actually get there. That might be a useful and easy thing to do.

Separate competitions and review for junior and senior applicants wasn't on the list that Larry presented, but I think the way it was enunciated here was quite interesting.

Then there is "certified reviewer training." There is, in fact, reviewer training, and guidelines are sent out, but what I've heard from you is that you want people to really pass a test of some sort. There are complexities to this, but the sense of the point is good.

Please don't think that if I didn't pick your idea that it wasn't good. Those are just a few that I wanted to use to highlight the fact that you've done a fantastic job. We really are grateful for your time and participation. It will have an impact.