Peer Review - General FAQs
Updated: October 29, 2009

A. Application-Related Enhancements

1. What is the rationale for the new Resubmission Policy?
2. How does the new Resubmission Policy change current practice and what are the implementation dates?
3. How will it be decided if an application is new?
4. What constitutes "significant" and "substantial" in determining if an application is new?
5. Who at NIH is involved in the decision if an application is new?
6. How will disagreements be handled?
7. When do application changes go into effect?
8. The Policy Announcement (NOT-OD-09-149) says that the restructured application forms need to be used for Fiscal Year funding 2011 submissions, but it also uses the January 25, 2010 deadline.  Which one should I go by?
9. Why are applications undergoing a restructuring?
10. What is the reason for having shorter applications?
11. How did the NIH make the decision to shorten applications?
12. How many pages are allowed in the Additional Grant Application Materials?
13. How will the changes apply to the review of ARRA stimulus-related FOAs?
14. I am eligible for continuous submission of my applications.  When should I transition to the new application forms and instructions?
15. Since I am a new investigator, I am able to submit a resubmission for an R01 application in consecutive review cycles, saving four months of time (NOT-OD-07-083).  When do I start using the new application forms and instructions?

B. Scoring

1. Is the scoring scale used by NIH study sections changing?
2. What is the rationale for this change?
3. Is the new 1 - 9 scale a stanine system?
4. What is staying the same?
5. Is the 1 - 9 scale used for all aspects of scoring an application, including the criterion scores?
6. Are reviewers still scoring as if there is an ideal? Or are reviewers comparing applications to each other within a meeting?
7. Why is the strengths/weaknesses graphic curved the way it is? 
8. Why are individual review criteria being scored?
9. What if significance is high but feasibility is low? Or questionable?
10. Will all applications receive criterion scores?
11. For multi-project applications, will there be criterion scores for all core review criteria? Cores currently can be given non-numerical scores, i.e., acceptable or unacceptable. Will that change?
12. Are individual reviewer criterion scores open to discussion by the entire panel?
13. What is the rationale for not requiring all reviewers to enter individual criterion scores?
14. How will the overall impact/priority score that appears on the face page of the summary statement and in the applicant’s Commons account be calculated?
15. Will all scores be rounded up? So only unanimous 1s will result in a 10? Doesn’t that make the range 20-90, not 10-90?
16. Why is there a need for rounding? Wouldn’t having more digits help enable discriminaton between applications?
17. Why is the percentile rounded up, while the scores are rounded traditionally?
18. Will this new scoring system result in more tie scores?
19. Will "not recommended for further consideration" be allowed if there is a serious human or animal subjects issue?
20. When will the U mechanisms fall into the new scoring system?
21. What does my score mean?
    
    
    

C. Summary Statements  

1. What will stay the same for summary statements?
2. How will summary statements change?
3. How will the criterion scores be displayed in the summary statement?
4. What if the scores in the table do not agree with scores that may have been entered with the written critique?

D. New Investigators (NIs) and Early Stage Investigators (ESIs)

1. What is the implication/significance of ESI versus NI? Since ESI is a subgroup of NI, what is the purpose of ESI? How will ESIs be treated differently than NIs in review and program? 
2. Who has the ability in the electronic system to update NI or ESI status?
3. Is verification of degree completion date required and accomplished via a third party?
4. In terms of qualification for the ESI designation, why are postdoctoral periods treated differently than medical residency periods, especially since residencies prepare for clinical research?
5. Can individuals that are within 10 years of completing a mentored career development award be considered ESIs?
6. Does ESI status extend beyond 10 years if the A1 is submitted after the 10-year period, but the A0 was submitted before 10 years have passed?
7. What constitutes a "terminal research degree"?
8. How do the following situations affect NI status:
9. Will New/Early Stage Investigators be percentiled with other applications?

E. Preparing and Submitting Critiques

1. What is IAR? What does it stand for?
2. Do reviewers fill in their critique templates on-line in IAR? 
3. Are the templates in Microsoft Word 2003 or 2007?
4. Why don’t the hyperlinks work in Microsoft Word 2003?
5. Is there a specific definition of impact?
6. Where on the template are scores entered?
7. It is nice to have example documents that model comments for strengths and weaknesses. However, is it realistic to continue guidance for a ¼ page in length when the example documents themselves are longer than that limit?
8. How do you reconcile the shorter critique with the increased reliance on critique information for breaking ties and making award decisions?
9. Where will reviewers comment on multiple PD/PI Leadership Plan?
10. Will there be a short “overall evaluation” critique template that a discussant can fill out in addition to adding numerical scores for the individual criterion? 
11. What guidance will be given to reviewers about using the additional comments box?

F. Review Meeting Processes

1. How do you put in criterion scores for an application with more than 5 scored review criteria?
2. How does inputting scores into IAR rather than templates reduce reviewer burden?
3. Will there be an error message if reviewers try to enter scores without uploading their critiques?
4. Do reviewers have to enter individual criterion scores even if they do not enter text for a criterion?
5. Criterion scores cannot be entered into IAR without critiques but can the preliminary score be entered without a critique?
6. Does a reviewer have to re-upload their critiques if they revise their score?
7. Will reviewers still be restricted from resubmitting, for example making corrections, until the final edit phase in IAR?
8. Can discussants read the critiques without posting any critique or score on IAR during the read phase?
9. Can reviewers nominate lower half applications for discussion?
10. How can one reconcile the previous admonitions to not compare applications during review with the current clustering system which appears to encourage it?
11. For which mechanisms should ESI/NIs be clustered?
12. Will ALL review groups have teleconferencing available?
13. To what extent will the SROs provide training at the next review meetings?

G. Enhanced Review Criteria

1. Could you please explain how "resubmission, renewal and revision" are review criteria?

1. What is the rationale for the new Resubmission Policy?
   During the past several years when NIH paylines have been very low, it has been noticed that a large number of excellent applications have had to be revised and resubmitted, even though the underlying science did not change significantly. In effect, excellent investigators with great projects had to wait in line. To try to address this problem, a new policy is in place to fund highly meritorious science earlier. Specifically, NIH plans to increase the success rate of new and first resubmission applications by decreasing the number of resubmissions or amendments that are allowed.
   
   For more information on the resubmission policy see: NIH Guide Notice NOT-OD-07-015 and NIH Guide Notice NOT-OD-09-016
   
   
2. How does the new Resubmission Policy change current practice and what are the implementation dates?
   Previously, an application could be revised and resubmitted twice. Under this new policy, which has gone into effect for fiscal year 2010, applications can only be revised and resubmitted once. To give a specific example for R01s, new applications that were submitted for February 5, 2009 will only be allowed a single resubmission. Similarly, new competing continuation R01 applications that were submitted for March 5, 2009 will also be allowed only a single resubmission. By reducing the number of resubmissions, the goal is to permit funding of larger numbers of new and first resubmission applications and thereby to allow investigators to spend less time revising and resubmitting and to start doing their research projects sooner.
   
   
3. How will it be decided if an application is new?
   A new application is expected to be substantially different in content and scope with more significant differences than are normally encountered in a resubmitted (amended) application. A new application should include substantial changes in all sections of the Research Plan, particularly in the Specific Aims and the Research Design and Methods sections. There should be fundamental changes in the questions being asked and/or the outcomes examined. Changes to the Research Plan should produce a significant change in direction and approach for the research project.
   
   
4. What constitutes "significant" and "substantial" in determining if an application is new?
   This is inevitably a scientific judgment for which no set of universally applicable examples can be provided. However, rewording of the Title and Specific Aims or incorporating minor changes in response to comments of reviewers from the prior review does not constitute substantial changes in scope, direction, or content. Requests for review by a different review committee or funding consideration by a different NIH institute are not sufficient reasons to consider an application as new.
   
   
5. Who at NIH is involved in the decision if an application is new?
   Applications received by the NIH are screened multiple times and checked to determine if the application is a new application. The first check is done within the Division of Receipt and Referral in the Center for Scientific Review. Subsequent checks are performed by the Scientific Review Officer in charge of the review meeting, by the reviewers themselves, and by NIH program staff.
   
   
6. How will disagreements be handled?
   If there is disagreement about whether an application assigned to CSR for review is new or re-submitted, CSR staff may refer it to a CSR-convened committee of NIH scientists to conduct further analysis and recommendations to the Division of Receipt and Referral. The evaluation process includes the analysis of previous applications and summary statements to determine the similarities and differences to the current application. Individual Institutes and Centers have their own evaluation processes. When applications are determined to be a version of an application that has already received two reviews and in violation of this policy, it is administratively withdrawn and not reviewed. The Division of Receipt and Referral in CSR informs the Program Director/Principal Investigator and institution of this determination.
   
   
7. When do application changes go into effect?
   All competing applications submitted for due dates on or after January 25, 2010 will require use of the new forms and adhesion to the shorter page limits. The changes do not affect R01, R21, and R34 AIDS applications that would otherwise have been due on January 7, 2010, but are submitted on or before February 7, 2010 by applicants who are eligible for continuous submission.
   
   
8. The Policy Announcement (NOT-OD-09-149) says that the restructured application forms need to be used for Fiscal Year funding 2011 submissions, but it also uses the January 25, 2010 deadline.  Which one should I go by?
   The cutoff for the new forms is determined strictly by the January 25, 2010 due date.  However, this does not affect R01, R21, and R34 AIDS applications that would otherwise have been due on January 7, 2010, but are submitted on or before February 7, 2010 by applicants who are eligible for continuous submission.
   
   
9. Why are applications undergoing a restructuring?
   The new application format will align the structure and content of the application with review criteria. This is to help ensure that both reviewer and applicant expectations coincide for a more efficient and transparent application process.
   
   
10. What is the reason for having shorter applications?
    Shorter applications are intended to help focus both applicants and reviewers on the essentials of the science rather than the administration of applying.  They will also have the additional benefits of avoiding information overload, and potentially enabling a larger number of reviewers to read each application and participate in the review in a more informed manner.
    
    
11. How did the NIH make the decision to shorten applications?
    The NIH grant application is among the longest that is used by funding agencies and foundations worldwide.  During the diagnostic phase of the peer review self-study, the overwhelming opinion from the working groups was that a significant reduction in page numbers is necessary to achieve the desired enhancements, to a page limit somewhere between 7 and 15 pages.
    
    The working groups considered the response to an NIH-issued RFI, which showed that 70% of all respondents preferred that applications be shorter than 25 pages. The majority of the respondents did not believe that a shorter application would compromise their ability to present scientific ideas (68% responded that ideas could be communicated equally well comparing 25 to 15 pages, and 19% indicated that shorter applications would enhance communication).  Respondents also indicated that a shortened application would take either the same (27 percent) or less (50 percent) time to prepare. Thus, for over three quarters of those responding, shorter applications would not be more burdensome.
    
    
12. How many pages are allowed in the Additional Grant Application Materials?
    The NIH Policy on Submission of Additional Grant Application Materials (NOT-OD-08-082) includes “NIH Best Practice Guidelines for Accepting Additional Grant Application Materials” (MS Word - 57 KB) which states, “If the Research Plan is 12 pages or less, then only one page of supplemental information will be accepted.”
    
    
13. How will the changes apply to the review of ARRA stimulus-related FOAs?
    ARRA FOAs will be reviewed with the enhanced review criteria and the 1 to 9 scoring system. The length of many applications for ARRA FOAs will be reduced as well – for example, the research plan for all Challenge grants is limited to 12 pages.
    
    
14. I am eligible for continuous submission of my applications.  When should I transition to the new application forms and instructions?
    R01, R21, and R34 AIDS applications from applicants who are eligible for continuous submission must use the current forms and instructions for applications that would otherwise have been due on January 7, 2010, but are submitted on or before February 7, 2010.
    
    All other applications that are from applicants who are eligible for continuous submission and are submitted before January 25, 2010 should use the current forms and instructions, while those submitted on or after January 25, 2010 should use the new forms and instructions (which become available in December 2009).
    
    
15. Since I am a new investigator, I am able to submit a resubmission for an R01 application in consecutive review cycles, saving four months of time (NOT-OD-07-083).  When do I start using the new application forms and instructions?
    New investigators submitting R01 resubmissions for November 20, 2009 must use the current application forms and instructions.  For R01 applications, the first due date for new application forms and instructions is February 5, 2010.
    
    

Back to Top

B. Scoring

1. Is the scoring scale used by NIH study sections changing?
   Changes to all competing applications are being phased in gradually.  All applications submitted for FY 2011 funding (due dates on or after January 25, 2010) will require use of the new forms and adhesion to the shorter page limits; see Policy Announcement for more details.
   

For information on the policy see http://grants.nih.gov/grants/guide/notice-files/NOT-OD-09-024.html.


2. What is the rationale for this scoring change?
   The major reason for changing the scoring system was to create a rating scale that realistically reflects the range of quality of applications that most study sections actually see and, thereby, to encourage routine use of the entire rating scale. Behavioral studies have noted that individuals can only distinguish between 7 and 11 categories. At least two different studies of the peer review system have noted that the current 41-point scale is not in alignment with human abilities to categorize. The new system will attempt to remedy that. The 1 to 9 scale will be used not only for overall impact/priority scores, but also for individual review criterion scores. Preliminary scores will continue to be used to help determine which applications are discussed at a review meeting. Assigned reviewers will also use the 9-point scale to rate each of five review criteria. By giving explicit ratings to each of these five criteria, applicants and NIH staff making funding decisions will have an easier time understanding the basis of the overall score that an application received. The goal is to provide more information about the strengths and weaknesses of each application and so each reviewer’s criterion scores will be reported on the summary statement. An important change is that all applications – whether they have been discussed in the review meeting or not – will receive criterion scores.
   
   
3. Is the new 1 - 9 scale a stanine system?
   No. Stanines impose a specific distribution of scores based on pre-determined standard deviations from the mean. The NIH scoring scale does not impose such a distribution. Percentiling is used to achieve comparability of scores across review groups (see section on Percentiling for more details).
   
   
4. What is staying the same?
   - "1" is still the best score.
   - The non-numerical scoring options remain the same:
   o CF: Conflict
   o NP: Not Present
   o DF: Deferred
   o ND: Not Discussed (terminology change, formerly Unscored)
   o AB: Abstention
   o NRFC: Not Recommended for Further Consideration
   
   
5. Is the 1 - 9 scale used for all aspects of scoring an application, including the criterion scores?
   The 1 - 9 scale will be used to assign the individual criterion scores, the preliminary impact/priority score and the final overall impact/priority score. It will not be used for criteria that are only rated "acceptable" or "unacceptable" such as protection of human subjects.
   
   
6. Are reviewers still scoring as if there is an ideal? Or are reviewers comparing applications to each other within a meeting?
   Applications are to be scored against a standard, as described in the scoring guidance document (http://grants.nih.gov/grants/peer/reviewer_guidelines.htm).
   
   
7. Why is the strengths/weaknesses graphic curved the way it is?
   The graphic was not designed to be taken in a literal context, but rather as one attempt at a simple visual aid to help the reviewer determine which numerical score to choose. For many reviewers, the descriptions of relative levels of strengths and weaknesses may be more helpful.
   
   
8. Why are individual review criteria being scored?
   It is hoped that by giving explicit ratings to each of these five criteria, applicants and NIH staff making funding decisions will have an easier time understanding the basis of the overall score that an application received. The goal is to provide more information about the strengths and weaknesses of each application and so each reviewer’s criterion scores will be reported on the summary statement.  Having reviewers add a numerical score to each of five core criteria is intended to aid in interpretation of their written comments. For example, a reviewer may have a number of criticisms, but these may be considered minor and easily fixed. The words alone might not indicate this intent, but assigning a good score would provide additional clarity. Likewise, one major weakness could outweigh several areas of strengths and a score in the poorer range would indicate that this weakness was considered major. The reasoning is that providing more and clearer information to the applicant, NIH staff, and advisory councils will permit them to make better-informed decisions.
   
   
9. What if significance is high but feasibility is low? Or questionable?
   If it is the opinion of the reviewers that it is not likely that the project will answer the questions it poses, then overall impact is likely to be low. The degree of uncertainty about feasibility will likely determine whether this is rated as a minor, moderate or major weakness.
   
   
10. Will all applications receive criterion scores?
    An important change is that all applications – whether they have been discussed in the review meeting or not – will receive criterion scores. Reviewers will use the criterion scores to help them determine the overall impact or priority score, but the criterion scores will not be weighted explicitly. It will be up to each reviewer to determine an overall score that best describes the likely overall impact that each application will have. Such determinations will reflect each reviewer’s best estimate of how much impact the application will have, given the importance of the questions being asked and the likelihood that the project will succeed, given the combination of investigators, approach and environment that are described in the application.
    
    
11. For multi-project applications, will there be criterion scores for all core review criteria? Cores currently can be given non-numerical scores, i.e., acceptable or unacceptable. Will that change?
    Please see the guidelines for P01 documents at http://grants.nih.gov/grants/peer/reviewer_guidelines.htm.
    
    
12. Are individual reviewer criterion scores open to discussion by the entire panel?
    The review panel will NOT vote on each criterion. However, during the discussion of an application, review committee members could choose to discuss individual criterion scores to make sure they understand the points raised by an assigned reviewer. It is important to understand that criterion scores are intended to be pieces of information to help understand each reviewer’s evaluation, but are not intended to be the sole basis of how to decide on an overall impact/priority score.
    
    
13. What is the rationale for not requiring reviewers to enter individual criterion scores?
    In some cases, an SRO may wish to bring in a reviewer who is a content expert on a particular part of an application. For example, a statistician might be used as a reviewer for an application that relies heavily on the use of statistics to analyze a biomedical issue. In this case, it would be inappropriate for the statistician to score parts of the application for which he/she does not have expertise.
    
    
14. How will the overall impact/priority score that appears on the face page of the summary statement and in the applicant’s Commons account be calculated?
    Calculate the average of the final overall impact/priority scores given by all eligible SRG members;
    - Multiply the average by ten;
    - Round to the nearest whole number (10 – 10.49 = 10; 10.5 – 11.49 = 11; etc.).
    
    
15. Will all scores be rounded up? So only unanimous 1s will result in a 10? Doesn’t that make the range 20-90, not 10-90?
    Only the percentiles are rounded up. The overall impact/priority scores are rounded traditionally, making the range 10-90. Getting a score of 10 would not necessarily require total unanimity. For example, if 24 reviewers scored 1 and one reviewer scored 2, the average would be 26/25 = 1.04. Multiplication by 10 yields 10.4, which would be rounded to 10 rather than 11.
    
    
16. Why is there a need for rounding? Wouldn’t having more digits help enable discrimination between applications?
    The scores are rounded to prevent the appearance of a precision that does not exist. Tie scores indicate that the tied applications cannot be reliably distinguished from each other based solely on scientific and technical merit.
    
    
17. Why is the percentile rounded up, while the scores are rounded traditionally?
    Percentiles are rounded up because we do not use a zero percentile, and the desire was to have a whole number percentile. So percentiles from 0.1 to 0.9 are rounded to 1.
    
    
18. Will this new scoring system result in more tie scores?
    With only nine possible rating options, there is an increased likelihood that the distribution of mean scores will result in more tie scores, which essentially indicate that the scale is reliable since identical mean scores will occur when there is reviewer agreement on the assigned score. Tie scores indicate that the tied applications cannot be reliably distinguished from each other based solely on scientific and technical merit. As is true with all funding decisions, Advisory Councils and NIH must weigh additional factors in the decision process, which include the need to factor in mission relevance; portfolio balance; specified Congressional appropriation requirements; and other relevant factors.
    
    
19. Will "not recommended for further consideration" be allowed if there is a serious human or animal subjects issue?
    YES.
    
    
20. When will the U mechanisms fall into the new scoring system?
    At the same time as all other mechanisms, i.e., at the start of the FY2010 reviews, typically beginning in May 2009 and beyond.
    
    
21. What does my score mean?
    For help interpreting new scores please read “Need Help Interpreting the New Review Scores?”
    

Back to Top

C. Summary Statements 

1. What will stay the same for summary statements?
   The face page and first-order organization of the summary statement will remain the same; Resume and Summary of Discussion for discussed applications; for all applications, the typical summary statement sections include the applicant abstract and public health relevancy statement; reviewer critiques, final resume section to summarize the reviewers’ recommendations; roster. The summary statement contains reviewer critiques presented as Critique 1 – Critique n, ordered by assigned reviewer. Only discussed applications will receive an overall impact/priority score.
   
   
2. How will summary statements change?
   The order of the review criteria will change: Investigator(s) and Approach swap positions; the new order(for R01 and related applications) is Significance, Investigator(s), Innovation, Approach, and Environment. A table at the beginning of each critique will summarize the reviewer’s scores for each criterion.
   
   
3. How will the criterion scores be displayed in the summary statement?
   Criterion scores will be added automatically by the Internet Assisted Review (IAR) system as a table at the beginning of each reviewer’s critique.
   
   
4. What if the scores in the table do not agree with scores that may have been entered with the written critique?
   The scores that IAR inserts in the table are accepted as final. Reviewers are instructed NOT to enter scores with their critiques and that errant scores in the critiques may be removed in finalizing the summary statement.
   
   

Back to Top

1. What is the implication/significance of ESI versus NI? Since ESI is a subgroup of NI, what is the purpose of ESI? How will ESIs be treated differently than NIs in review and program?
   A New Investigator (NI) is an NIH research grant applicant who has not yet been awarded a substantial, competing NIH research grant. An Early Stage Investigator (ESI) is a new investigator who has completed his or her terminal research degree or medical residency—whichever date is later—within the past 10 years and has not yet been awarded a substantial, competing NIH research grant. The ESI designation helps to differentiate between established and early-career investigators, and helps NIH meet the goal of accelerating the transition to an independent scientific career. It is expected that ESIs will constitute the majority of funded NIs.
   
   
2. Who has the ability in the electronic system to update NI or ESI status?
   NI and ESI status is determined automatically by the functionality built into eRA Commons, based on the investigator’s record of receiving NIH grants and the date of their terminal degree and/or completion of medical residency. If an extension to the ESI 10-year period is granted by the ESI Extensions Committee, the eRA Commons account will be updated by eRA staff.
   
   
3. Is verification of degree completion date required and accomplished via a third party?
   Degree and medical residency completion dates are provided by the applicant. Applicants are expected to provide true, accurate, and complete information and to produce documentation when requested. (Making false statements to the federal government can result in severe penalties).
   
   
4. In terms of qualification for the ESI designation, why are postdoctoral periods treated differently than medical residency periods, especially since residencies prepare for clinical research?
   The 10-year ESI period was designed to allow for a consistent time period of research activities between applicants. While postdoctoral periods generally are devoted to research and research training activities, medical residency periods generally are not.
   
   
5. Can individuals that are within 10 years of completing a mentored career development award be considered ESIs?
   Individuals that are within 10 years of completing a mentored career development award will only be considered ESIs if they are also NIs who are within 10 years of completing their terminal research degree or medical residency.
   
   
6. Does ESI status extend beyond 10 years if the A1 is submitted after the 10-year period, but the A0 was submitted before 10 years have passed?
   If an original application is flagged as an ESI application and it is revised and resubmitted as an amended application (A1) within 13 months after the submission date of the original application, it will retain its ESI status even if the eligibility period has expired. If the amended application (A1) is resubmitted more than 13 months after the submission date of the original application, the ESI status will be recalculated based on the submission date of the revised application.
   
   
7. What constitutes a "terminal research degree"?
   A terminal research degree is the most recent research degree received by an investigator. In most cases, the terminal research degree will be a Ph.D., M.D., or equivalent. Individuals without a doctoral degree should enter their highest research degree, which could be a Masters or a Bachelors degree in some cases. An M.B.A. or an M.F.A. and similar degrees normally are not considered research degrees. However, some types of research Master’s degrees (like M.S. or M.P.H) can be considered terminal if they mark the beginning of a period of research or research training, or the new degree substantially expands or advances an individual’s skills within a field related to the NIH mission. For example, an M.P.H., or a comparable research degree earned after the medical degree, may initiate a period of research and should be listed as the terminal research degree. The conferral date for the terminal research degree will mark the beginning of the ESI status period.
   
   
8. How do the following situations affect NI status:

   • PI of an NIH contract? Does not disqualify NI status.
   
   • PI of a grant with another Federal agency? Does not disqualify NI status.
   
   • PI of an SBIR/STTR? Does not disqualify NI status.
   
   • PI of a U01, specifically for a foreign investigator? Receipt of U01 removes NI status.
   
   • Inheriting an R01 from a PI who moved away or died? Does not disqualify NI status.

   For a complete list of NIH grants that do not disqualify a PD/PI from being considered a New Investigator, see http://grants.nih.gov/grants/new_investigators/index.htm.

9. Will New/Early Stage Investigators be percentiled with other applications?
   Percentile bases will be determined for study sections, and there will not be a separate percentile base for ESIs, NIs, or established investigators. There will also be a CSR-all base that will be used for percentiling Special Emphasis Panels (SEPs), or in some instances, Institutes/Centers may have their own base for percentiling SEPs reviewed in their respective Institute/Center.
   
   

Back to Top

E. Preparing and Submitting Critiques

1. What is IAR? What does it stand for?
   Internet Assisted Review (IAR) is a web-based system that has been used to manage the electronic submission process of reviewer critiques for several years.
   
   
2. Do reviewers fill in their critique templates on-line in IAR? 
   No, reviewers will download a Word file that contains the critique template for the mechanism they are reviewing. They will fill out the template in Word, and then upload the entire Word file into IAR.
   
   
3. Are the templates in Microsoft Word 2003 or 2007?
   The templates are compatible with both formats. They are also compatible with OpenOffice, an open-source suite of office software. 
   
   
4. Why don’t the hyperlinks work in Microsoft Word 2003?
   The hyperlinks that are associated with each review criterion will only work with Microsoft Word 2007. These hyperlinks all point to the same Web site, but will bring the user to different “anchor” points on the page that correspond to a particular review criteria or consideration. For users with Microsoft 2003, the hyperlinks associated with each review criterion will not work, but an accessible hyperlink is provided at the top of the template that links to the same Web site, so that the same information is available.
   
   
5. Is there a specific definition of impact?
   The definition of impact is slightly different for different types of grant mechanisms. Please see the definition which is linked to from the critique template.
   
   
6. Where on the template are scores entered?
   Scores are not entered on the critique template; rather, they are entered directly into IAR.
   
   
7. It is nice to have example documents that model comments for strengths and weaknesses. However, is it realistic to continue guidance for a ¼ page in length when the example documents themselves are longer than that limit?
   Although it is recommended that no more than ¼ page of text accompany the bullets for any particular criterion, no hard limits are imposed. The responses posted on the critique example are of varying length, depending on the particular criteria or consideration. If there are a lot of strengths and weaknesses that have an impact on the overall score, then they should be listed. It is the explanatory prose that reviewers are asked to limit. However, it recognized that some points can in fact take a considerable amount of text to explain, which is why no absolute limits are imposed. Minor weaknesses that do not contribute to the overall score (and whose fixing might therefore NOT be expected to result in an improved impact score) can be listed in the “Advice to Applicant” box if the reviewer thinks that these may be helpful to the applicant.
   
   
8. How do you reconcile the shorter critique with the increased reliance on critique information for breaking ties and making award decisions?
   It is expected that the new critiques will convey the same or more evaluative information as in the past. One goal of changing the critique template was to remove needless repetition and restating of what is in the application.
   
   
9. Where will reviewers comment on multiple PD/PI Leadership Plan?
   Multiple PD/PIs is included in the enhanced investigators criterion, so it will be commented on under the investigator(s) criterion on the critique template.
   
   
10. Will there be a short “overall evaluation” critique template that a discussant can fill out in addition to adding numerical scores for the individual criterion? 
    No, but discussants can use the critique template for the mechanism they are reviewing and only fill out the “Overall Impact” section of the template.
    
    
11. What guidance will be given to reviewers about using the additional comments box?
    The additional comments box provides an opportunity to offer advice, to make helpful suggestions, or to point out minor issues that did not contribute to the overall score. Actual strengths or weaknesses that contributed to the impact score should be included under the appropriate criterion.
    
    

Back to Top

F. Review Meeting Processes

1. Could you please explain how "resubmission, renewal and revision" are review criteria?
   Each resubmission, renewal or revision should be evaluated on criteria that go beyond that of a new grant application.  See the enhanced review criteria for specifics on each of these criteria.
   
   
2. How do you put in criterion scores for an application with more than 5 scored review criteria?
   You cannot put more than 5 criterion scores into IAR. Any additional scored criteria will have to be added into the text of the critique and added manually on the summary statement.
   
   
3. Will there be an error message if reviewers try to enter scores without uploading their critiques?
   Yes.
   
   
4. Do reviewers have to enter individual criterion scores even if they do not enter text for a criterion?
   No, entering criterion scores is not mandatory. Text-based evaluations of the criteria will be entered into a word document with sections for each review criterion and consideration. The reviewer is free to leave sections of the critique template empty if they do not apply for his/her evaluation.
   
   
5. Criterion scores cannot be entered into IAR without critiques but can the preliminary score be entered without a critique?
   No. To enter any score a critique must be entered, with the exception of some mail and unassigned reviewers who do not need to enter a critique.
   
   
6. Does a reviewer have to re-upload their critiques if they revise their score? 
   No. However, if they wish to change any of their written critique to go along with their revised score, they should upload a revised critique.
   
   
7. Will reviewers still be restricted from resubmitting, for example making corrections, until the final edit phase in IAR?
   The current rules apply.
   
   
8. Can discussants read the critiques without posting any critique or score on IAR during the read phase?   The same rules apply. If you are a discussant or other assigned reviewer and you have not posted your scores/critique, you will not be able to read the critiques of others.
   



9. Can reviewers nominate lower half applications for discussion?
   Yes.
   
   
10. How can one reconcile the previous admonitions to not compare applications during review with the current clustering system which appears to encourage it?
    The intent is to facilitate the discussion so that applications that are under the same types of review criteria and emphasis are considered together (NIH NI/ESI Policies). This is not meant as a means to compare applications; applications are reviewed not against each other but according to published review criteria. The intent of clustering is to help reviewers’ frame their discussions to be consistent with the emphases placed on the different categories of research.
    
    
11. For which mechanisms should ESI/NIs be clustered?
    Only R01 applications from ESI/NIs will be clustered for review.
    
    
12. Will ALL review groups have teleconferencing available?
    All SROs will have telephones available at the review meeting, with rare exceptions.
    
    
13. To what extent will the SROs provide training at the next review meetings?
    In preparation for the May/June round, SROs will be provided formal training and communications and training materials so that they can inform and train reviewers.
    
    

Back to Top

G. Enhanced Review Criteria

1. Could you please explain how "resubmission, renewal and revision" are review criteria?
   Each resubmission, renewal or revision should be evaluated on criteria that go beyond that of a new grant application.  See the enhanced review criteria for specifics on each of these criteria.
   

Back to Top