Carnegie Mellon University

Developing a Short List of Candidates

A particularly important step for the Search Committee is to discuss and agree on how criteria listed in the job ad will be evaluated before reviewing applications. Using or creating a standard evaluation form for candidate review helps make comparisons systematic, avoids shifting standards, and reduces implicit bias when the committee discusses expectations about each of the criteria in advance. Note, however, that it is advisable to avoid quantifying at this stage since small biases in ratings can have undue impact when accumulated. 

Models for Evaluation Tools

Applicant Evaluation Tool — University of Michigan [.pdf]
Sample evaluation form with more specific criteria [.pdf] — Extracted from Michigan State faculty search handbook

  1. Using a standard evaluation form is advised when search committee members evaluate all of the applicants for the initial screening, with at least two members of the committee reviewing a given application. While the initial screening may rely on a brief review of a large number of applications, having clear evaluation criteria greatly aids in consistency across all applicants. Also, note that early screening focused on including all qualified applicants (rather than screening people out who may have less traditional qualifications) has been found to lead to decisions that are more strongly supported by evidence and less subject to unconscious bias.

  2. The search committee normally narrows the applicant pool to a list that all members of the committee carefully review and discuss. If the applications to be assessed by the full committee do not seem to include diverse candidates, it is helpful to consider having a “medium list” instead of a “short list” at this stage. Including strong individuals who may have less traditional qualifications as part of a “medium list” or “long short list” can help to ensure careful, equitable review of their materials. Research has shown that setting aside adequate, undistracted time for careful individual review of all of the “medium list” applications is a good way to reduce bias.

  3. The search committee members discuss the potential candidates to be invited for an interview. If phone interviews and reference checks are part of the process, please use recommended practices for consistency across potential candidates.

  4. The committee should be prepared to explain their reasons for including or excluding each candidate in terms of the committee’s evaluation criteria. Although an external challenge to a search committee’s decision based on bias is rare, records about these decisions are important in such cases. Some search committees use a form to keep track of these evaluations by committee members.

Implicit Bias Research on Letters of Recommendation

Letters of recommendations need to be evaluated carefully, factoring in awareness of subtle biases from letter writers that tend to overrate men and underrate women. For example, one study of letters from a chemistry and biochemistry faculty search used more “stand-out adjectives” (e.g. outstanding, superb) for men than women, using analysis of covariance to take into account differences in publications, fellowships and other indicators of achievement (Schmader, Whitehead and Wysocki, 2007).

Recommended Resources

Ensuring a Fair and Thorough Review of Candidates (Boston University)

Logistics for managing the review of applicants and Resources [.pdf] (U. Wisconsin Madison, Searching for Excellence and Diversity: A Guide for Search Committees, 2012, pp. 60–64 and 68–70)

Creating the short list [.pdf] (U. Michigan Faculty Search Handbook, pp. 16–17)

Sign-in sheet for evaluation of applicants [.pdf] (U. Wisconsin Madison, Searching for Excellence and Diversity: A Guide for Search Committees, 2012, p. 70)