s305.93-3 Peer Review in the Award of Discretionary Grants (Recommendation No. 93-3).

Governments in most industrialized nations now play a prominent role in assembling and sustaining a sound scientific and engineering infrastructure and in providing financial support for artistic and other endeavors. Although many procedural vehicles exist for making the difficult scientific and artistic judgments that necessarily arise in apportioning limited resources, the United States government has depended to a large degree upon "peer review" systems in which the agency decisionmaker assembles a group of experts for advice.

Under this peer review model, the government does not attempt to persuade researchers to undertake particular research or artists to create particular kinds of art. Instead, a grantmaking agency allocates sums of money to broad fields of endeavor and invites researchers, artists, or performers to develop creative proposals for projects. A group of "peers" with expertise in the relevant area then evaluates and ranks proposals, leaving the ultimate funding decisions up to the governmental program officials. Peer review is intended to ensure that public funds are awarded objectively to meritorious scientists, artists, eleemosynary institutions, and, increasingly, for-profit entities in a way that renders the system accountable to the public and its elected representatives.

While peer review has proved remarkably durable in the 30 to 40 years during which federal agencies have employed it, the process has not been without controversy. Some critics suggest that peer review is an expensive waste of time; that it diverts creative minds from productive research to writing, reviewing, and discussing proposals; that it rewards huckstering skills at the expense of solid research; that it is sometimes abused by reviewers who breach confidentiality; and that it can be counterproductive in programs designed to explore fresh ideas and innovative approaches. In particular, some peer review systems have been criticized for permitting ad hoc and systematic bias for and against individuals, groups, or new ideas. Decisionmaking bias in the award of discretionary grants can result from favoritism, animus, or conflict of interest. It can stem from the identity of the potential grantee, as, for instance, where an "old boy" network exists or a "halo effect" causes poorly conceived proposals from well-known scientists to be funded, or from conflicts of interest that have to do with the affiliation or position of the decisionmaker. Bias can include personal or professional animus, a lack of regard for those mavericks who challenge conventional views, or a systematic refusal to give sufficient weight to particular criteria relevant to the decision. Finally, ex parte lobbying or even political pressure may occasionally cause an otherwise objective process to become biased for or against particular persons or approaches.

To the extent that bias infects it, the decisionmaking process loses objectivity and, consequently, legitimacy. While the incidence and impact of bias are not susceptible to empirical measurement, the Conference believes that, on balance, the peer review model has worked well [FN1] and is highly appropriate for awarding discretionary grants in the arts and sciences. [FN2] The Conference's recommendation is based largely on a study of programs in the National Institutes of Health, the National Science Foundation, the Environmental Protection Agency, and the National Endowment for the Arts. Although all rely heavily upon the principle of peer review in awarding discretionary grants, these agencies manage the peer review process in diverse ways. None has completely eliminated the potential for bias, though some have made great strides in that direction. Each can learn from the others, and all grantmaking agencies not included in the Conference study can learn from their experiences. These recommendations draw on their experience to suggest reforms that should further reduce the potential for bias at a relatively low cost.

FN1 To encourage flexibility, innovation, or a rapid response where warranted, some agencies that ordinarily rely on peer review to evaluate grant proposals (such as the National Science Foundation) have found it useful to set aside a small portion of the available funds for relatively brief, small awards outside the normal channels of peer review.

FN2 Especially in the scientific agencies, a recent trend toward greater openness has been noted. This has been prompted in part through enactment of the Freedom of Information Act (FOIA), the Federal Advisory Committee Act (FACA), and the Privacy Act. FOIA requires every federal agency to make available to any person any record in the agency's possession (subject to several exceptions potentially relevant to the peer review process) upon a request by that person that reasonably describes the record. FACA requires federal agencies that rely on recommendations of advisory committees to charter these committees and to run them according to statutory standards of openness. The Privacy Act directs agencies to protect personal information in agency files from unauthorized disclosure and to give individuals an opportunity to review information about themselves and to require that the agency correct inaccuracies. While peer review committees clearly come with FACA's definitions, there have been significant litigation and uncertainty over the effect of these laws.

Recommendation

1. Promoting Openness and Accountability

a. Reviewer meetings. Agencies that rely upon peer review to evaluate grant proposals should generally assemble the reviewers for a meeting in which each reviewer, in the presence of the other members, has an opportunity to comment upon the evaluations made by other members. Such meetings may also be by telephone conference call.

b. Feedback and rebuttal. Insofar as practicable, agencies should provide applicants with an opportunity for feedback and rebuttal as follows:

(1) Agencies that rely on peer review should prepare and routinely transmit to all applicants brief summaries of the reasons for evaluation results.

(2) If peer review committees prepare written evaluations of individual applications, agencies should retain these documents for a full funding cycle, and should make available copies of such written evaluations to applicants upon request in a redacted form so that particular evaluations may not be attributed to particular reviewers. Agencies that retain documents prepared by peer reviewers should make them available in redacted form where practicable and where reviewers' candor would not be unduly affected.

(3) Statements and summaries of reasons should be made available to applicants sufficiently far in advance of the agency's final decision so that applicants may review the documents and submit comments for timely consideration by the peer reviewers or agency staff. Where this is not practicable, agencies should maintain an appropriate reconsideration system.

(4) Agencies that rely on peer review should develop guidelines identifying information that will normally be made available to grant applicants relating to their applications and specifying the procedures under which particular kinds of information will be available to different classes of requesters.

c. Applicant anonymity. While applicant anonymity will often be infeasible and inconsistent with effective application of merit review criteria, agencies should be aware of situations in which not revealing to peer reviewers the identities of applicants for discretionary grants would further objective review. Agencies should consider allowing reviewers to conduct discrete portions of peer reviews under conditions of applicant anonymity in cases in which complete applicant anonymity is not feasible or consistent with effective application of merit review criteria.

d. Reviewer anonymity. (1) The agency should provide that, although the identities of all members of any peer review panels should be made available to applicants and the public, granting agencies should be authorized to refuse to disclose information that would enable applicants to identify the persons who reviewed in detail a particular application.

(2) Agencies that rely on peer review should inform all peer review panelists in writing of their Privacy Act obligations and of the penalties that may flow from a breach of confidence. Agencies should consider sanctions, including instituting debarment or suspension procedures, against any panel reviewer who knowingly breaches confidentiality.

e. Contacts with peer reviewers, staff, and decisionmakers. While agencies that rely on peer review should encourage informal contacts between applicants and agency staff for the purpose of conveying general information, they should adopt appropriate statements of policy discouraging all ex parte contacts regarding particular grant proposals with peer reviewers and agency decisionmakers that attempt to influence grantmaking decisions outside the normal decisionmaking process.

2. Participants in the Review Process

a. Composition and structure of review committees. (1) Agencies that rely on peer review should endeavor to include some reviewers from related professional fields on peer review panels. Agencies should also consider including on the panels individuals who are not peers but who bring to bear perspectives relevant to the decision.

(2) To the extent consistent with agency resources and depending on the size and number of the grants awarded in a program, agencies should provide that the membership of peer review panels change on a regular basis, ensure that the number of persons serving on an individual peer review committee is sufficiently large to dilute the impact of any bias, permit duplicate reviews in two or more subcommittees, or allow multiple committees to perform a tiered review.

b. Conflicts of interest. (1) Agencies that rely on peer review should promulgate guidelines that prevent a person from reviewing or sitting on a panel that reviews his or her own grant application or the application of a relative, a close collaborator, a recently graduated former student, an institution by which the person is employed, or an affiliated institution. Insofar as possible, reviewers with a competing or other adverse interest to an application to be considered should not review that application.

(2) Agencies should provide that a reviewer must disclose a conflict of interest or appearance thereof to the agency and, if the reviewer continues to participate, to other members of the review committee.

(3) If necessary, agencies may provide for specific waivers, with disclosure, of the conflict of interest recommendations on a case-by-case basis when there is no other practical means for securing appropriate expert advice on a particular grant application.

(4) Agencies that rely on peer review should ensure that reviewers are informed and observe applicable rules or guidelines on bias and conflict of interest. In particular, agencies should caution persons on peer review panels of the need to protect against panel misuse of confidential information made available to it and should ensure that such information or position is not misused.

c. Opportunity to object to reviewers. Agencies that rely on peer review should provide that any applicant may submit a confidential list containing a small number of potential reviewers that the applicant deems objectionable together with a statement of the reasons for the challenges. Agencies should seriously consider such objections and honor those that are meritorious, unless the agency determines that a qualified group of peers cannot be assembled if all such challenges are honored.

d. Scoring applications. Agencies that rely upon scoring systems for evaluating and ranking discretionary grant proposals should consider developing methodologies for ensuring fairness in scoring applications.

3. Audits for Potential Bias

Agencies that rely on peer review should implement independent reviews of the process for bias and conflict of interest.

Dated: August 18, 1993.

Authority: 5 U.S.C. 591-596.

SOURCE: 58 FR 45412, August 30, 1993; 57 FR 61760, 61768, Dec. 29, 1992, unless otherwise noted.