When she was the only woman on a shortlist of four, she was never hired
In a study of nearly 600 finalists for university teaching jobs, a lone woman on a four-person shortlist was hired 0% of the time, as the speaker cited. Knowing about bias does not stop it, and hiring teams keep passing over qualified women. Harvard Kennedy School research fellow Siri Chilazi asks which changes to job ads, interviews and decisions actually level the field.
Get nine evidence-based fixes for job ads, interviews and hiring decisions you can apply today
Siri Chilazi, Research Fellow at the Women and Public Policy Program at Harvard Kennedy School, argues that knowing about unconscious bias is not enough to stop it, so recruitment processes must be redesigned. She presents nine evidence-based strategies to broaden the candidate pool, evaluate candidates more objectively and improve hiring decisions in tech.
“Our brains are very stubborn and they need help.”
“Just changing the pictures on the walls had this massive effect in behavior.”
What you will learn
Explain why awareness of unconscious bias is not enough and why processes must be redesigned to counter it.
Rewrite job advertisements to remove gendered language and state which aspects of the role are negotiable.
Design work sample tests and structured interviews with job-relevant questions asked in the same order.
Run hiring decisions with independent evaluator scores, joint simultaneous decisions and diverse shortlists.
Audit hiring algorithms by tracking whether the candidates they produce are more or less diverse than before.
Details
Level
Beginner
Format
Talk
Technical depth
Non-technical
For
People manager, Executive, Founder
Length
20:53
Published
August 2020
Questions this talk answers
Is unconscious bias training or awareness enough to remove bias from hiring?
No. Chilazi shows that merely being aware of bias does not overcome its effects; the evidence favours changing processes, like putting blinders on the pattern.
How can job advertisements attract a broader pool of applicants?
Remove or balance gender-stereotyped words such as guru or warrior, and explicitly state which aspects of the job, including salary, vacation, hours and remote work, are negotiable.
How can interviews be made less biased?
Decide job-relevant questions in advance and ask every candidate the same questions in the same order, then compare answers question by question; work sample tests are even more predictive.
Why does the composition of a shortlist matter?
In the study she cited, a woman who was the only woman among four finalists was never hired; only when women were half the pool did a woman have a chance, so she recommends reviewing shortlists and aiming for 50-50.
How can resume screening be de-biased?
Anonymize resumes by removing names, addresses and phone numbers, or replace the resume with one or two short, job-specific written questions.
About this talk
Recruiting the best of the best and tapping into 100% of the talent pool is the smart thing and the right thing to do. However, unconscious biases at the organizational and individual level can get in the way of a fair and equitable hiring process. In her session, Siri will share practical, evidence-based strategies to level the playing field in tech recruitment and create inclusive and diverse organizations. This session is designed for anyone - founders/executives, managers, and HR professionals - involved in recruitment activities, and no preparation is needed.
The classroom posters that changed behavior (53 s)
The shortlist where women never got hired (73 s)
So, before you start, Siri, I'd like to introduce you to our audience. Recruiting the best of the best and tapping into 100% of the talent pool is the smart thing and the right thing to do, right? However, unconscious biases at the organization and individual level can get in the way of a fair and equitable hiring process. Speaker: In her session, Siri Chilazi, Research Fellow Women and Public Policy Program at Harvard Kennedy School will share practical, evidence-based strategies to level the playing field in the tech recruitment and create inclusive and diverse organization. Hi, Siri, and welcome.
Siri Chilazi: Hi, Anna. Thank you so much for having me. And hi, everyone, and welcome. I'm so excited that you're joining me for this session. Siri Chilazi: I am going to be sharing these slides and I assume that you all can see, Anna, please let me know if there's anything wrong with them. Speaker: We just saw them. Now again, we see them. So, I'm leaving the stage to you. Please enjoy.
Speaker: Thank you, Anna. So, like Anna mentioned in the introduction, we all know that hiring the best of the best is not only the smart thing to do from a business standpoint, but it's also the right thing to do from a moral standpoint. Speaker: And yet it doesn't always happen because the data shows that the tech industry in particular remains male dominated globally.
Speaker: So I'm excited to share with you today some very practical strategies to level the playing field in your own organization and to de-bias your recruitment process so that we gave women and men and non-binary people an equal chance at getting hired and then succeeding once they're in the door. Speaker: But I want to start by understanding the problem that we're trying to solve.
Speaker: What's going wrong with our recruitment practices today? Speaker: So take a look at this checkerboard and determine which square is darker, A or B. Speaker: And keep looking at A and B as I quickly cover up the rest of the board. Speaker: And if by now you want to change your original answer, I'll go back one more time to give you a chance to look at this again.
Speaker: What we're seeing here is unconscious bias in action. Speaker: The conscious part of our brain is able to process only a very small sliver of the overwhelming amount of information that we're constantly bombarded with. Speaker: So our unconscious brain has to deal with most of the information processing and decision making. Speaker: And this unconscious brain of ours is very prone to patterns and shortcuts and heuristics.
Speaker: And that's exactly what happens here. Speaker: The moment you saw this line, your brain recognized this familiar checkerboard pattern, light square next to a dark square. Speaker: And it basically instantaneously and subconsciously determined that square A has to be darker than B, which in this case turns out not to be true. Speaker: So all of us have unconscious bias.
Speaker: It's just a fact of how the human brain works. Speaker: But unconscious bias is also built into the processes and the environments and the systems that we humans with our biased minds have created. Speaker: So the question becomes what to do about it? Speaker: How can we mitigate and minimize the effect that bias has on our decision making?
Speaker: So take a look at this checkerboard again. Speaker: Do you still see square A as being darker than B, even though I've already proved to you that they're the same color? Speaker: I can say for myself, I certainly do. Speaker: And I've seen this slide, you know, 5,000 times. Speaker: So the thing to understand about unconscious bias is that merely being aware of the bias, Speaker: that merely being aware of its existence is not enough to overcome its effects.
Speaker: Our brains are very stubborn and they need help. Speaker: In this case, the help is the blinders that I'm bringing in, which quite literally take away the pattern and allow you to see squares A and B for what they really are to do justice to them. Speaker: And it's exactly this approach that the research evidence shows is most effective at debiasing our organizational processes.
Speaker: As well. Speaker: So I'm going to share with you nine evidence based practical strategies to help debias your recruitment process. Speaker: We're going to talk about broadening the pool, evaluating more objectively and improving our decision making. Speaker: And for those of you who today are leaders of organizations or HR professionals, you're already in a position to put all of this into practice in your organization starting today.
Speaker: And I hope you will. Speaker: And likewise, if you're an entrepreneur or a founder, I hope that you will keep these strategies in mind as you grow your company and build out your team. Speaker: And finally, if you're someone who's working in tech, but not quite at a leadership level yet or seeking to work in tech, I hope that you'll use this information from today to educate your colleagues and to push for change in your organization.
Speaker: So let's dive right in. Speaker: My first strategy to help you broaden the pool is to eliminate gendered language from job advertisements. Speaker: Here's an example of a real job advertisement for an email and social media marketing manager that I pulled off the internet about a year ago. Speaker: And notice that I bolded a couple of the words.
Speaker: Now close your eyes for a second and tell me what image flashes in your mind when I say warrior or guru. Speaker: Did you see a picture of a man or a woman in your mind's eye? Speaker: And now feel free to open those eyes. Speaker: The thing is, language is gendered, not just that the obvious level of girls, boys, women and men, he, she, but even words that don't overtake.
Speaker: We have a gender attached to them in our cultural context tend to be associated with other women or men tend to be stereotyped as either female or male in all languages, not just English. Speaker: And the reason this matters is because research has shown that a job advertisement like this one that has male stereotype language words like guru and warrior.
Speaker: We tend to associate with men more than women. Speaker: These types of job advertisements actually attract more men to apply, and it's mostly a subconscious effect. Speaker: Likewise, an advertisement that had heavily feminine stereotype language, like if they were looking for warm and caring and collaborative colleague, that type of job advertisement would attract more women to apply.
Speaker: The good news is that there's a very simple fix, which is we can simply eliminate the gendered words like I've done in this example, or we could balance out the female and male stereotype words that we're using to appeal to the broadest possible audience. Speaker: And at the end of the presentation, I'm going to share with you some resources to help you do just that.
Speaker: In a similar vein, another thing I urge you to consider with your job advertisements is to make all the negotiable aspects of the job very explicit. Speaker: Research has shown that when in a job advertisement, it was clearly stated that the salary for that position was negotiable. Speaker: That statement not only helped to close the gender pay gap between the women and the men that were ultimately hired.
Speaker: That statement also encouraged more underrepresented groups to apply for the job in the first place. Speaker: So this is an incredibly simple way to broaden your applicant pool. Speaker: Just state on your job description or your job advertisement, what aspects of the job are negotiable and what aspects are not. Speaker: This includes salary, of course, but also things like amount of vacation time, whether the job is part time or full time.
Speaker: Maybe it could be either. Can the job be done remotely in part or does it have to be fully office based? Speaker: Make it clear. Speaker: And thirdly, I urge you to be very cautious and deliberate about the symbols and role models that you use in your recruitment. Speaker: A recent study from Stanford University in California showed that tech companies that were recruiting on college campuses Speaker: tended to present their work culture and their organizational practices in such a way as to create a hostile environment for women.
Speaker: So how exactly did they do that? And they didn't set out to do that, by the way. Speaker: Well, they featured mostly men as speakers and presenters in their recruitment presentations. Speaker: Where there were women present, those women tended to be relegated to non-speaking roles off to the side. Speaker: These tech companies used very stereotypical imagery on their slides, like pictures of scantily clad women at parties.
Speaker: And they even handed out very gendered gifts like beer can openers or beer mugs. Speaker: So as you can see from this picture here, they communicated the sense to women that they didn't belong in this organization's culture. Speaker: And in fact, research in general shows that what we see role model symbols have an enormous subconscious influence on our behavior.
Speaker: But we can use that to our advantage. Speaker: Another study showed that simply changing the pictures on the walls of a computer science classroom, changing them from Star Wars pictures, Speaker: which are more heavily male stereotyped to nature landscape pictures that are much more gender neutral. Speaker: Just that change in pictures helped to equalize women's and men's interest in computer science and their plans to pursue continued education in computer science.
Speaker: Just changing the pictures on the walls had this massive effect in behavior. Speaker: So in your recruitment, look very carefully at who you're sending out to recruitment events to represent your company, Speaker: who you're featuring in your recruitment materials on your websites, on your brochures and in your office, Speaker: whose pictures are hanging on the walls, who are conference rooms named after.
Speaker: All of those symbols and role models are opportunities to communicate the level of diversity and inclusion that you actually want to see. Speaker: So that's broadening the pool. Speaker: Let's move on to evaluating more objectively. Speaker: And the first thing research suggests you should do here is to determine your evaluation criteria in advance. Speaker: Several studies have showed that when people are asked to hire a candidate for a stereotypically masculine job, Speaker: like that of a construction worker or police officer, or I could add anything in the tech industry.
Speaker: People tend to pick the male candidate regardless of their qualifications simply because they look the part. Speaker: They're who we stereotypically expect to be a police officer or a construction worker or a tech employee. Speaker: But this unconscious preference for men in hiring disappears when people select their evaluation criteria in advance. Speaker: So when one particular study, when they were hiring for construction worker, they had a male and a female candidate.
Speaker: One had more expertise and one had more experience, pardon me, education and experience. Speaker: And when the evaluators decided first whether they were looking for experience or education, Speaker: then they picked whichever candidate, the woman or the man, fit the bill. Speaker: So this is why it's so important to choose your evaluation criteria in advance before you actually start evaluating candidates.
Speaker: It's also very important to use work sample tests whenever possible. Speaker: Work sample tests are tasks or exercises that are designed to mimic the actual job as closely as possible. Speaker: So if you're hiring someone for a software engineer position where they're going to be doing a lot of coding, Speaker: you could design a coding task that these candidates can complete and then you evaluate the result of that code.
Speaker: About three or four decades of research shows that these work sample tests are by far the most predictive way of hiring candidates. Speaker: They have the highest correlation to how people actually perform on the job. Speaker: So work sample tests, very important to incorporate into your recruitment process. Speaker: That said, I know that most companies still want to use interviews as well, Speaker: even though interviews have been shown to be much less predictive as an evaluation tool.
Speaker: So there's two things I want you to consider for interviews to make them more objective and less biased. Speaker: The first is to determine your interview questions in advance and to actually make those questions relevant to the job. Speaker: You need to first decide what you're looking for and what types of characteristics and attributes would make people successful in this role.
Speaker: And then you have to design questions to test for those characteristics and attributes. Speaker: So that's point one. Once you have your questions, point two is to ask all of your candidates the same questions in the same order. Speaker: Because if everyone gets asked different questions, how can you compare their answers in an objective and fair apples to apples way?
Speaker: You can't really. When you ask the same questions, you can then compare each answer to each candidate's answer to each question one at a time, Speaker: looking across to see which candidate had the best answer. Speaker: This type of structure and formality in interviews helps to significantly reduce unconscious bias. Speaker: So that's evaluating candidates more objectively.
Speaker: Finally, let's move on to ways to improve our decision making. Speaker: The first strategy here is to collect all evaluator scores independently before you bring the group together to discuss the candidates Speaker: and to make a final decision. One of the unconscious biases that we all have and are prone to is groupthink, Speaker: which is our desire for conformity and harmony in a group.
Speaker: Think of a recruitment team debriefing where everyone defers to the most senior person in the room. Speaker: But as you can imagine, the problem with groupthink is it can lead to suboptimal or even dysfunctional decision-making Speaker: because there's less diversity of thought and less diversity of information because fewer people are speaking up to offer different viewpoints.
Speaker: So the way to get around groupthink is to make sure that each evaluator submits their individual assessment of the candidates Speaker: and that those assessments are aggregated before the group meets to make the final decision. Speaker: It also turns out that our minds are naturally comparative when evaluating people or things, Speaker: and that we generate more diversity in our decision making and are less prone to bias when we make evaluations Speaker: and decisions jointly and simultaneously.
Speaker: What this means is that if you normally hire one person in March and one in June and one in September, it would actually be better to hire for all those three Speaker: open positions at the same time so that you have a bigger pool of candidates to compare against each other and so that you can make three simultaneous decisions.
Speaker: This applies even if the positions are very, very different. Speaker: So joint and simultaneous decision making. Speaker: And lastly, I want to talk about short lists or the finalist pools. Speaker: This is data from a study of nearly 600 finalists for university teaching positions. Speaker: And as you can see, the composition by gender of the finalist pool had a huge effect on who ultimately got hired.
Speaker: When one in four of the finalists was a woman, she should have had, statistically speaking, a one in four or 25% chance of getting hired. Speaker: But actually, she never got selected. Speaker: Her actual chance was 0%. Speaker: It was only when women made up half of the pool or two of the four candidates that the woman had a chance of getting selected.
Speaker: So I urge you, number one, to review your short lists for diversity. Speaker: And before decisions are made, while you still have an opportunity, if that short list or finalist pool doesn't contain the diversity that you want to see and where you want your company to move, go back and find more candidates. Speaker: Because at that stage, it's not too late yet.
Speaker: Number two, I would also urge you to consider setting actual numerical targets for the composition of your finalist pool. Speaker: And for gender, the research suggests that maybe that target should be 50-50. Speaker: You should aim to have 50% women and men represented in your finalist pool. Speaker: So that's a lot in 15 minutes. Speaker: I know.
Speaker: Here's a quick recap of these nine strategies that we covered in case you want to take a picture of the slide. Speaker: I also promised to share with you some resources to help you do all of this. Speaker: Here's just a handful of the amazing companies and startups around the world that help companies do things like eliminate Speaker: gendered language from resumes or craft really good interview questions.
Speaker: You don't have to do it all alone. Speaker: There's a lot of expertise out there. Speaker: And for those of you who want to dive more deeply into the research, I highly recommend the Gender Action Portal. Speaker: This is a free resource that my office, the Women in Public Policy Program at Harvard University maintains. Speaker: And we summarize research on advancing gender equality specifically for practitioners like you all.
Speaker: So check it out. Speaker: And with that, I think we have a couple of minutes for questions. Speaker: I'm so excited to hear what's on your mind. Speaker: And my email is here on the slide. Speaker: So if you have more questions or if you'd like to follow up, I'd love to hear from you. Speaker: That was great.
Speaker: Thanks, Siri. Speaker: I really love that slide with helpful resources from that slide. Speaker: I know only one company, which is called texture. Speaker: So I need to check it out myself. Speaker: There's a lot. Speaker: Super useful. Speaker: And we have many questions from the audience. Speaker: They're saying that the lecture is amazing. Speaker: They're loving it.
Speaker: A question from Gabriela Bureau. Speaker: Are there any unbiasing tips for selecting CVs? Speaker: I'm sorry, selecting. Speaker: CVs. Speaker: Resumes. Speaker: Yes, yes, yes. Excellent question. Speaker: One strategy that has been shown time and time again to work is anonymization, which means taking the name of the person and maybe even their address, their phone number off of the resume, because those things can often indicate that person's gender, race or maybe the country that they come from.
But it gives you no information about that person's actual competence for the job. Speaker: Another approach is to forego the resume entirely and just ask candidates maybe one or two questions that said, you know, if we're looking for a coder, what past coding experience do you have? Write in 100 words. And that way we equalize the playing field because people can write their resumes very differently.
There's different conventions in different countries. So that's another approach to consider. Speaker: All right. That's super interesting. Thank you. Speaker: Another question is just check in. Yeah. Thanks, Siri. I love your talk. How those steps can be applied to automated algorithms? Speaker: Excellent question. So a lot of the organizations that I put up on one of those last slides is actually that's exactly what they do, is they have developed software and algorithms to help you automatically, for example, identify gendered language and suggest alternative words or automatically take resume and names off of resumes.
Those are amazing tools that can greatly help us to do this work. But we have to be careful to make sure that the algorithms themselves don't perpetuate bias. So just like with anything else, we have to track and measure the data of what kinds of candidates is the algorithm generating for us. How does that differ from the candidates that we were seeing before the algorithm?
And are things moving in the right direction? Or are we actually seeing less diversity than we were before? It's data, data, data, track, measure, and make sure that you are moving in the direction that you want to be moving in. Speaker 1: All right. Very helpful. Thanks a lot, Siri. Our audience has your email. If they want to get in touch, they can do so.
Thanks a lot. Thanks for tuning in. Thanks for your interesting and insightful presentation. I'm wishing you a great day. Stay with us. Thank you very much. Thanks, Anna. Thank you, everybody. Enjoy the conference.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
How can tech companies take bias out of hiring when awareness is not enough?
Siri Chilazi of Harvard Kennedy School shares nine evidence-based strategies to broaden the pool, evaluate fairly and decide better.
In short
Knowing that unconscious bias exists does not stop it from shaping who gets hired. Siri Chilazi, Research Fellow at the Women and Public Policy Program at Harvard Kennedy School, argues that organizations must redesign their recruitment processes instead, and she offers nine evidence-based strategies grouped into three areas: broadening the pool, evaluating candidates more objectively and improving decision making.
Key takeaways
Explain why awareness of unconscious bias is not enough and why processes must be redesigned to counter it.
Rewrite job advertisements to remove gendered language and state which aspects of the role are negotiable.
Design work sample tests and structured interviews with job-relevant questions asked in the same order.
Run hiring decisions with independent evaluator scores, joint simultaneous decisions and diverse shortlists.
Audit hiring algorithms by tracking whether the candidates they produce are more or less diverse than before.
Why awareness of bias is not enough
Chilazi opens with a familiar optical illusion: a checkerboard on which two squares look different in shade but are in fact the same colour. The brain recognizes the pattern and fills in an answer before the conscious mind can check it. She uses this to make a broader point about how all people process information: the unconscious brain handles most decisions and relies on shortcuts. As she puts it, "It's just a fact of how the human brain works." (2:55)
The crucial lesson comes when she shows the board again, after proving the squares match. Most people, including Chilazi herself, still see a difference. "Our brains are very stubborn and they need help." (3:42) The help, in her illustration, is a set of blinders that removes the misleading pattern. That, she explains, is what the research evidence shows works best in organizations too: changing the process rather than relying on good intentions.
She addresses three groups in the audience: leaders and HR professionals who can apply the strategies immediately, founders building teams, and individual contributors who can use the evidence to push for change.
Broadening the pool: job ads, negotiables and role models
The first strategy targets the wording of job advertisements. Chilazi shows a real ad for a marketing manager that asks for a "guru" and a "warrior". Words like these, she explains, carry a gender in our cultural context, and research has shown that ads with male-stereotyped language attract more men to apply, while heavily feminine wording attracts more women. The fix is simple: remove the gendered words or balance them.
The second strategy is to make every negotiable aspect of the job explicit. As she cited, research found that stating a salary was negotiable helped close the gender pay gap among those hired and encouraged more underrepresented groups to apply. She extends this to vacation time, part-time or full-time options and remote work.
Third, she urges care with symbols and role models. She describes a Stanford University study of tech companies recruiting on college campuses that, without intending to, created a hostile environment for women through male presenters, stereotypical imagery and gendered gifts. Another study she cited found that swapping Star Wars posters for nature landscapes in a computer science classroom helped equalize women's and men's interest in the field. "Just changing the pictures on the walls had this massive effect in behavior." (9:32) She encourages companies to look at who they send to recruitment events, whose pictures hang on the walls and whom conference rooms are named after.
Evaluating candidates more objectively
In stereotypically masculine jobs, Chilazi explains, evaluators tend to pick the male candidate because he looks the part. Research she cited shows this preference disappears when evaluators decide their criteria before they assess anyone, so the first step is to fix evaluation criteria in advance.
Next come work sample tests, exercises that mimic the actual job, such as a coding task for a software engineer. "About three or four decades of research shows that these work sample tests are by far the most predictive way of hiring candidates." (11:55)
Because most companies still want interviews, she offers two rules: design job-relevant questions in advance, and ask every candidate the same questions in the same order so answers can be compared one question at a time. "This type of structure and formality in interviews helps to significantly reduce unconscious bias." (13:11)
Improving hiring decisions
To counter groupthink, such as a debrief where everyone defers to the most senior person, Chilazi recommends collecting each evaluator's scores independently and aggregating them before the group meets. She also recommends joint, simultaneous decisions: hiring for several open positions at once, even very different ones, gives a bigger pool to compare and reduces bias.
Her final point concerns shortlists. In a study she cited of nearly 600 finalists for university teaching positions, a woman who was the only woman among four finalists never got the job, although statistically she should have had a 25% chance. Only when women made up half of the pool did a woman have a chance of being selected. Chilazi urges teams to review shortlists for diversity and, if they fall short, to go back and find more candidates. "Because at that stage, it's not too late yet." (16:06) She also suggests numerical targets for finalist pools, with research pointing to 50-50 for gender.
She closes by pointing to companies that build tools for this work and to the Gender Action Portal, a free resource from her program that summarizes research on gender equality for practitioners.
From resumes to algorithms
In the Q&A, Chilazi recommends anonymizing resumes by removing names, addresses and phone numbers, or replacing the resume with one or two short, job-specific questions. Asked about automated hiring tools, she welcomes software that flags gendered language or strips names, but warns that algorithms can perpetuate bias too. "It's data, data, data, track, measure, and make sure that you are moving in the direction that you want to be moving in." (20:05)
Watch the talk
De-Biasing Recruitment in Tech, Siri Chilazi, 21 minutes: https://www.womentech.net/video/siri-chilazi-de-biasing-recruitment-in-tech
About the speaker
Siri Chilazi is a Research Fellow at the Women and Public Policy Program, Harvard Kennedy School, where she studies what works to level the playing field and close gender gaps in organizations. As an advisor and speaker she brings research insights to organizations from start-ups to multinational corporations, and her work has appeared in Harvard Business Review, The New York Times, BBC, Fast Company and Forbes.
Questions this talk answers
Is unconscious bias training or awareness enough to remove bias from hiring?
How can job advertisements attract a broader pool of applicants?
How can interviews be made less biased?
Why does the composition of a shortlist matter?
How can resume screening be de-biased?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Work sample tests
Structured interviews
Resume anonymization
Independent evaluation before group discussion
Joint and simultaneous evaluation
Resources mentioned
Gender Action Portal
Cite this talk
Siri Chilazi. "De-Biasing Recruitment in Tech." WomenTech Network, 2020. https://www.womentech.net/video/siri-chilazi-de-biasing-recruitment-in-tech