Strategies for referees, reviewers, and committee chairs
The purpose of this resource is to ensure that scholarship and admissions adjudication is compliant with institutional priorities and strategic directions for equity, diversity, and inclusion. Visit McMaster’s Equity and Inclusion website for more information on the strategy towards inclusive excellence.
What is unconscious or implicit bias?
Judgment and/or behaviour – subtle cognitive processes that operate at a level below conscious awareness and without intentional control
- rooted in normal, often adaptive, psychological processes
- maladaptive and problematic when informed by stereotypes
- unacknowledged can influence behaviour that inadvertently discriminates
- we all hold unconscious thoughts that we may not recognize as biased
- we can mitigate biases by acknowledging them and using counter strategies
- denying unconscious bias and attempting to explain away will adversely affect interpersonal relations and impede equitable outcomes
Referees, reviewers, and selection committee chairs should be aware of and work to mitigate the possibility of implicit bias influencing the assessment of a candidate, application review, or the decision-making process.
Bias may manifest itself in any number of ways and could be based on one or more factors, among others: school of thought, fundamental versus applied or translational research, areas of research or approaches (including emerging ones), types of research contributions, size or reputation of a participating institution, cultural background, age, race, religion, language, gender
How can bias influence evaluation?
- Performance bias
- Am I evaluating expected potential or proven accomplishments based on stereotypes?
- Confirmation bias
- Am I seeking information that confirms preconceived ideas based on stereotypes?
- Performance attribution bias
- Am I attributing performance to skill or “innate” talent or to luck or external factors?
- Normative bias (“group think”)
- Am I conforming to the thoughts of others in a group?
- McMaster Equity & Inclusion Office workshop, Feb 18, 2020 (Office, 2020)
- Government of Canada Vanier Graduate Scholarships (accessed Sept 4, 2020)
Role of the referee: To provide an assessment of the potential for a candidate to succeed in a graduate program, plan of research, or academic award.
Suggested strategies to minimize implicit bias when writing a letter of reference or other assessment of a candidate:
- Avoid using stereotypical or interpersonal attributes when describing character and skills (e.g., words such as nice, hardworking, conscientious, dependable, diligent, kind, agreeable, sympathetic, compassionate, selfless, giving, caring, warm, nurturing, maternal);
- Focus on research skills and achievements using words that describe the candidate’s research excellence (e.g., thought-provoking, innovative, novel, thorough, detailed, impactful);
- Consider using, where appropriate, ‘stand-out’ adjectives (e.g., superb, excellent, outstanding, confident, successful, ambitious, knowledgeable, intellectual) for all genders (e.g., women, men, transgender, two-spirit);
- Use the nominee’s formal title and last name instead of their first name. Avoid attributing the contribution of an applicant’s work to the order of authors, as not all disciplines follow a single convention;
- Consider whether your comment unintentionally includes ‘doubt raisers’ (i.e., negative language, hedges, unexplained comments, faint praise and irrelevancies, such as “might make an excellent leader” versus “is an established leader”);
- Use inclusive language is (e.g. “the applicant” or “they” instead of “he/she”) and do not use words or sentences that reflect prejudiced, stereotyped or discriminatory language of particular people or groups, or their institution;
- Avoid being unduly personal;
- Support your points by providing specific examples of accomplishments always;
- Do not include information related to ethnicity, age, hobbies, marital status, religion, disability;
- Avoid sharing personal information about the applicant; such information may be helpful only in explaining academic delays or interruptions and should be addressed with caution.
Source: Adapted from the Vanier Graduate Scholarships website. (Canada, 2020)
For Reviewers, Committee Members, and Chairs
Role of the review: To assess a candidate’s potential success in a program of study or academic award, and select candidates for recommended offers of admission or academic awards.
Suggested strategies to minimize implicit bias in the review and selection process:
- Use clearly established, standardized assessment metrics and processes with a common rubric and an understanding of bias.
- Allow sufficient time and avoid multi-tasking when you review applications to allow for self-correction of bias-related tendencies.
- Review the selection criteria before you begin evaluating applications.
- Question whether your evaluation would change if the applicant were of a different gender or cultural background, if they had a different name or if they were working on a different research topic.
- Guard against over-reliance on one piece of information or on “first impression” reactions.
- Organize the applications by random sorting the order of review numerically.
- Pre-score all nominations assigned and submit pre-scores to the designated committee chair or administrator prior to the selection committee meeting.
- Consult guidelines for reviewing nominations involving Indigenous scholarship or research with Indigenous communities, such as those available through your institution or Government of Canada.
- Programs and committees can develop a standardized letter of recommendation with cognitive and non-cognitive scales.
- Read the writing sample first, then look at test scores, then look to the letters for — ideally — concrete information that contextualizes the rest of the material in the packet, particularly the writing sample or candidate statement.
- Pay attention to the facts within the specific content.
- Establish the weight of each component of the assessment. (Letter of reference, grade point average, personal statement, writing sample relative to each other.)
Source: Adapted from the Vanier Graduate Scholarships website. (Canada, 2020)
Developing and using a rubric
What criteria are established predictors of success, through what indicators can these be assessed, and what sources of evidence relate in order to inform the assessment?
|Academic excellence||past academic performance (grades)||transcripts|
|Research potential||relevant training
Your rubric can be more robust, sophisticated, and complete. This is just a simple illustration. See also Vanier Criteria for a more robust example.
Once you have established the Criteria, Indicators, and Sources, you may also want to determine the relative weight of each criterion category, and develop a scoring system where each candidate application receives a numeric score in each category and an overall total score.
Information Box Group
McMaster University Equity & Inclusion Office
Canada Research Chairs Unconscious Bias Training Module
Avoiding racial bias in letter of reference writing Asmeret Asefaw Berhe and Sora Kim, University of California, Merced Based on Avoiding Gender Bias in letter of reference writing PDF flyer from University of Arizona; Accessible version
Chris Houser & Kelly Lemmons (2018) Implicit bias in letters of recommendation for an undergraduate research internship, Journal of Further and Higher Education, 42:5, 585-595
Five tips for writing great letters of reference: Avoiding unintentional bias and stereotypes Sharonne Hayes, LinkedIn (Published on April 27, 2016; accessed July 14, 2020.)
Tips for writing great letters of reference: Avoiding unintentional bias and stereotypes
- Be comprehensive: Include leadership roles, research, publications, and other important content in all letters. Gather this content first prior to writing your letter.
- Emphasize accomplishments, rather than effort. Where appropriate, use terms such as successful, excellent, accomplished, skilled, knowledgeable, research, insightful, and independent.
- Avoid the irrelevant: Don’t mention personal life and other information irrelevant to their future role.
- Be specific and honest about skills, accomplishments, abilities.
- Avoid doubt-raising phrases and hedges such as “likely to succeed’ vs ‘will undoubtedly succeed.’
- Use professional designations/honourifics consistently. (Either refer to applicants as “Dr. so-and-so” in all letters for PhD holders or not.)
- Avoid gendered terms such as “woman, mother, gentleman, father.” (Gendered terms more often used in LORs about women.)
Source: Adapted from Sharonne Hayes Professor, Cardiovascular Medicine & Director, Diversity & Inclusion, at Mayo Clinic. (Published April 27, 2016; accessed July 14, 2020.)