Does Artificial Intelligence (AI) Support Diversity, Equity and Inclusion?
Patrice Caire23:032022
A job ad hidden from anyone over 54 shows how AI quietly discriminates
A job ad sat open on one woman's screen while her 60-year-old friend could not see it at all: it was shown only to profiles aged 18 to 54. Computer scientist Patrice Caire, a consulting scientist at Stanford University's Robotics Lab, traces cases like this, and a husband granted 20 times his wife's credit limit, back to the data AI learns from. What can users, companies and governments actually do about it?
Learn how biased training data skews AI and what users, companies and governments can do
Computer scientist Patrice Caire argues that AI systems often reinforce society's inequalities because the data they learn from, and the teams that build them, lack diversity. Using cases such as age-targeted job ads, mislabelled images, CEO image searches and Apple Card credit limits, she explains why training data matters. She calls for vigilant users, transparent companies, diverse teams and enforceable AI standards.
“No wonder then that the point of view of women is not included in the design and development of the artificial intelligence tools and systems that we use every day.”
“That was a bit of a quick fix, but probably it was cheaper than to address systematically the issue.”
What you will learn
Explain how the size and diversity of a training data set shape what an AI system can recognize and decide.
Recognize everyday cases where AI discriminates by age, gender or race, from job ads to image search and credit limits.
Distinguish the statistical and computational biases that company toolkits address from the human and systemic biases that NIST says remain below the surface.
Apply practical steps as a user: report ethical issues, demand data set transparency and ask companies to disclose automated decision making.
Follow the regulatory and standards work of bodies such as NIST and IEEE to help close the legal void around AI.
Key takeaways from the speaker
Understand that today, AI impacts almost everyone, but that the negative impacts are particularly affecting women and minorities.
Learn why scientific experts are urging governments and companies to take action and develop AI standards and regulations.
See how critical it is for all citizens be aware of AI biases and learn how to identify them.
Grasp the significance of the input data used to train AI models.
Realize that to prepare the future, women and minorities should become full participants in the design and development of the AI technologies that shape our world.
Details
Level
Beginner
Format
Talk
Technical depth
Introductory technical
For
All career stages
Length
23:03
Published
July 2022
Questions this talk answers
Why do AI systems produce biased results?
They learn from training data, so when a group is missing or mislabelled in that data, the system cannot handle it correctly. Caire calls this the zebra problem.
Are quick fixes enough to remove AI bias?
No. Google removed the gorilla label instead of fixing the data, and the patched CEO image search fell back on outdated results with no women or people of color after a slight change in the search term.
What can ordinary users do about biased AI?
Stay on the lookout, report ethical issues, talk to each other and demand that companies be transparent about their data sets and their use of automated decision systems.
How can AI be regulated when there is no clear legal framework?
Caire advises following what governments and bodies such as the IEEE publish, joining petitions and discussions, and staying informed about advances in AI standards.
Why did digital assistants have female voices?
According to the social science research Caire cites, female voices seemed more passive and amiable, which people preferred in an assistant. She notes Apple only added a gender-neutral Siri voice that year.
About this talk
Today, Artificial Intelligence (AI) is deployed everywhere—from smartphones to search engines—raising ethical questions particularly concerning women and minorities. However, governments and companies have been slow to address AI biases and discriminations. In this talk, I focus on the question whether AI supports diversity, equity, and inclusion. Using daily life use cases and examples, I illustrate the significance of the input data while training models. Then I stress the importance to recognize, understand and remove biases in this input. All along, I present recommendations like establishing AI standards and regulations in order to fill the legal void in which companies operate and including diverse demographics that reflect the composition of our society, to design, develop and test current and future AI technologies.
Why a husband got 20 times the credit limit (89 s)
Why digital assistants had female voices (52 s)
Thank you very much for having me. I'm very happy to be here with you today. I am Dr. Patrice Caire and I do research in artificial intelligence and robotics. I have a PhD in computer science and I've worked extensively in Silicon Valley and in academia. As you know everywhere very well today only 23% of women are working in the tech industry today. No wonder then that the point of view of women is not included in the design and development of the artificial intelligence tools and systems that we use every day. Instead this system often reinforce these inequalities and stereotypes of our society. It is exactly for these reasons that I have decided to focus my work on this question. How can artificial intelligence systems support diversity, equality and inclusion? Let me start with
the story. I have a friend Jane who was looking for a job. So I see an ad, I sent it to her and she contacts me right back and says, "Patrice, I cannot see the ad." And I said, "But Jane, I'm looking at it right now. I have it under my eyes. What was happening?" Well, what was happening was that Amazon, who was trying to hire people between the age of 18 and 54, try to, which is illegal, of course, arranged something with, at the time, Facebook, now Meta, to bypass the law. So how did they do that? Well, they displayed the ad only to people, to users whose profiles were indicating that they were between the age of 18 and 54. This is illegal. This is a way to bypass the law, and this is not
fair. My friend Jane was 60 at the time, so she could not see the ad. She could not see it because it was not shown to her. And I was below 54, and I could see the ad. So we can see that it is in the interactions between AI systems and humans that often the ethical issues are taking place. But we use so today, many, many different kinds of systems. For example, when we log on to our phone, we use image recognition. When we also dictate, we use NLP, natural language processing. And of course, today, there are more and more robots with embedded AI systems. But what are AI systems today? Well, there are systems which display intelligent behaviors, which display intelligent behaviors. And they do that by analyzing
their surrounding and taking actions to fulfill some goals. What is behind that? Well, today, the main approach is neural network, machine learning, and deep learning, and so forth. So for people who are not familiar with these topics, here is a very high level example, let's imagine that we have our system and there is a new data, in this case, this black cat, and we want to know what it is. The system outputs. It's a black cat. It's a cat. So what was happening? How did was that possible? Because in the training model, we had used a data set, which was a data set, which was containing lots of cats, labeled cats, and lots of dogs, labeled dogs. So the bigger the data set, and also the more diverse the data set, the more refined will be the training model, and the more precise will be the output.
So we see this data set is important. So let's consider now that we put as new data, another animal, whatever, zebra. So we have a zebra. And what is our system going to do? Well, it is not going to recognize it, of course, because there was no zebra in the training data set. All right. All right. All right. Well, we see, again, the importance of this. So let's look in real life how this can play out. In fact, I'm going to show you an image. There was this engineer, who was looking at images, and suddenly, he sees his girlfriend.
He sees his girlfriend. But, oh, shockingly, she was labeled as a gorilla. This was terrible, of course, this is not possible. And what was happening there? Well, the data had been wrongly labeled. And this happens sometimes, very often, because it is still done manually, this labeling, and by so-called Amazon Mechanical Turks. And this sometimes causes difficulties when, in these cases, wrong labels are used in images. So this is like our zebra before. A zebra problem again. The data set was not of higher quality.
So we can see. So we can see that this was an issue. So, of course, Google tried to address this issue. And how did they do that? Well, a couple of years ago, they just removed the gorilla label. That was a bit of a quick fix, but probably it was cheaper than to address systematically the issue. But we have other examples. For example, image search is very important. And when you looked at image search a few years ago, you could return, it would return for the term CEO, it would return this output.
So, only white male. Only white male. How is that possible? Our zebra problem again. Yes, the data set didn't include many women, probably any, for the CEO term. So many that maybe the team, which was working on this issue was not combining many diverse population or didn't have many women among them. But in any case, it was not addressed. And it was in all the different search engines.
So what happened was that to check on the issue, which had made quite a scandal, the University of Washington earlier this year decided to do a project to see what was happening there. So they decided to check and what was the issue when you were searching for CEO. And very nicely, it displayed here on Google, the more statistical representation of what today the CEO can be.
Well, they decided then the University of Washington decided to go further and to use a technique used by security and privacy researchers when they do test crashes for cars for safety of the cars. In these cases, what do they do? They test, they challenge the computer systems by a number of examples and they choose to see how they hold up the system hold ups.
So they decide, the University of Washington decided to do a very, very slight change to start with. And so they just added a search term to this very slight change. An enormous difference happened in the output. And here is what we got. Our zebra problem again. There was no women again in this in this output, no people of color. It was outdated data which had been used for to fix the issue of the first page. And nothing had been done to do a systematic approach to the whole problem.
So this is an example of these are a few examples there, but we can also see sometimes where there are very, very, the impact can be very dramatic on our lives. For example, when we use automated decision systems, it is used very pervasively today for many issues for housing and for loan, for example. And there was this couple who was filing husband and wife. They were filing together on their tax return and they were actually the wife had the better tax credit tax history than her husband. But they had a new project and they wanted to check their credit line on the Apple card.
Now, what happened when they saw that their husband had 20 times the credit limit than his wife? How was that possible? They contacted Apple and, you know, requested to have some explanation. And as an explanation, Apple returned the answer that their automated decision algorithm was actually based on a 1974 law, which was, of course, bringing some totally outdated output when the data set was used in that way.
Because the situation of women in 1974, of course, especially the economic situation was very different from today. So our zebra problem again, wrong data set and wrong outcome. But we saw now many, many different, different, different, different examples of biases and different way to deal with it. But how was, how can we manage these, these biases? So, this was what the NIST, the US National Institute for Standard and Technology, addressed this year. They published a report in March. And in this report, they showed this picture.
So, what is pretty interesting there is that what we are starting, just starting to address and to look at is the tip of the iceberg, meaning that we are looking at, you know, the problems, which are biases, which are statistical and computational biases. So, the rest of it is not addressed at all. And the rest is pretty important. So, the point, the tip of the icebergs that we're addressing today is main, is addressed by many companies by providing toolkits.
Toolkits, they are often, which are often called Python code, using Python code, and providing a number of metrics for fairness, and also providing decision, debiasing methods, and as well as visualization. So, the rest of it, what is below the surface, the human biases, the systemic biases, is not taken into account, as they say, and it is still something to remain to be done.
But, we still can do something. We can do something. In fact, what we can do is we can be on the lookout. Whenever we see some ethical issues with AI, we have to report them. We have to talk to each other. We have to try to find a solution and not let it go. We have to be aware of this situation.
And, of course, we have to ask and demand that companies be responsible and they can, they are transparent. For example, they should be giving access to their data sets so that we know the composition, if they are diverse, if they are gender balanced. And, of course, the responsibility is enormous for these companies, because in the automated decision making algorithm that we see more and more in our lives, and which impacts us directly, this should be really, companies should be really writing right up the sites that they are using these automated systems.
And so that we as users have the opportunity to address and to contact and to say, well, there are some issues there. We have to fix something because the result was not right. And, of course, we have seen that we are only 23% of women. So, yes, we have to make sure that we have some teams which are diverse and balanced, gender balanced in the whole AI lifecycle.
They have to be, they have to be diverse and representing society. We cannot just have a few part, a few percentage of the society being there doing all this work that we use all the time. And we have to make sure so that we have multi-factor and multi-stakeholders to test, to validate, to evaluate, to do the testing scenario for all these systems.
And finally, we have to make sure that companies are not operating in the legal void that they are operating in today. This is not possible because the companies are not there for philanthropic goals. They are there for business. And we have to make sure that our governments and our institutions are really there working for us to make sure that there are laws and standards, AI standards, to give a framework to all the companies producing AI systems.
And we have to make sure that all these are enforced and that we have set up also some legal ways to audit algorithms, all algorithms. So we see we can do a lot. And to conclude, I'm going to let you read this sentence. So what do we want to do? Do we want to let AI systems reinforce the stereotypes and inequalities in our society?
Or do we want to make sure that we work together and we try to address these issues and work towards AI systems, which are supporting diversity, equality and inclusion? This is our choice. Thank you very much for your attention. And they'll be taking questions now. So there is a question actually here for from Paula Castro. There is if there is no clear legal frameworks, how can we make sure AI is being regulated?
Well, that is really a good question because we have to follow the fact that the government are publishing some results all the time and some laws. So we have to follow the issue and see and see what are the the bodies who are actually working on this. So that way we can make sure that we can also when there is some petitions to us, because, for example, there were many petitions a few years ago from scientists who were trying to regulate for AI.
For example, the weapon industry, the autonomous weapon industry, which is a really critical area. So we have to make sure there that something is happening and maybe join these discussions for the free frontier or from the electronic frontiers and all the AI bodies which are working on this topic. So there are there are some they are starting the IEEE, for example, has been putting a report together, starting with the looking at different areas such as the area of trust.
So how can how can how can trust be a fundamental area to be able to build some more gender equality and also more some some specific standard for that. So there are some some some some area every country actually has started to set up to be in movement to have something like this. So it is for us. And that was my first point. It is from from us to also try to get informed and and try to make sure that we follow the advances which are being done in this area.
If there are other questions, please don't hesitate. I mean, gender equality is a is a huge question there. I mean, we cannot being only 23 percent, 23 percent of women in the tech industry, of course, you know, we are not represented. And it's not to say that the white males who are working on these products and on these softwares are not biased, consciously biased.
It's we have a lot of biases which are just in conscience. We have we have interiorized them. So we don't know. I mean, people often don't know that they have that they are biased until they are being shown that this is at the expense of other people, other groups, other minorities or other, you know, ethnic or other, you know, people from the society.
Often when you are in a part of the control group, you don't see that actually others do not have these the problem that you you don't see because you don't have them. So this is why the NIST was showing this importance of the human biases and this and the problem of of of the systemic biases. We all have biases. So the interesting part with AI's tools and systems is that they show us that we have biases because they are like reinforcing these biases.
And we see them even more because it's so powerful. I mean, there are many instances of that in natural language processing, where if you translate one language into another, for example, it was done with them with the doctors and nurse in some in some languages, the doctor is neither feminine or masculine. And in some languages where it has a gender, it was translated by a male automatically, even though sometimes it was that it was a female. So female are considered nurses and doctors and males are considered nurse are considered doctors, excuse me.
So it is all these biases, which are not good because they reinforce also what is happening in our society. For example, another example, for the past 10 years, we have been having this dictation on the on the phone, like Siri, like what we see on Amazon, I mean, everybody now as a as a digital assistant. Well, from the start, the digital assistant, where feminine voice, female voice, and they have been even now for some digital assistant, it's still female voice. Do you know why that is? Well, simply because the research social science research has has shown that social that female voices appear to be more passive, and more amiable.
And so people preferred having a digital assistant, which would be passive, rather than some thing they would perceive as more controlling or more, more dictatorial or more, you know, male oriented if it was a voice. In fact, in fact, it's only this year that Apple added to its Siri voice, gender neutral voice. So we are making progress because people have been complaining for ages to see that these voice assistants would be more gender bias, less gender bias. And this is has finally occurred.
But it takes a long time and it takes a long time and it takes people to be on the lookout. That's what we have to be on the lookout. Whenever we see something which doesn't feel right, where you feel that you are being being not favored, but on the contrary, we are being discriminated, then you should just speak out and demand that things change, because it's only like this, that they will change.
Thank you.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why AI reinforces bias against women and minorities, and how to stop it
Computer scientist Patrice Caire traces AI discrimination back to its training data and sets out what users, companies and governments must do.
In short
AI does not support diversity, equity and inclusion by default. According to Patrice Caire, a computer scientist working in AI and robotics, it often reinforces the inequalities already present in society, because the data it learns from and the teams that build it do not reflect the people it serves. The fix, she argues, needs vigilant users, transparent companies, diverse teams and enforceable AI standards.
Key takeaways
Understand that AI today impacts almost everyone, but its negative impacts particularly affect women and minorities.
Learn why scientific experts are urging governments and companies to take action and develop AI standards and regulations.
See how critical it is for all citizens to be aware of AI biases and learn how to identify them.
Grasp the significance of the input data used to train AI models.
Realize that, to prepare the future, women and minorities should become full participants in the design and development of the AI technologies that shape our world.
A job ad that only some people could see
Caire opens with a gap and a story. As she cited, only 23% of the people working in the tech industry are women, so their point of view rarely shapes the AI tools everyone uses. "No wonder then that the point of view of women is not included in the design and development of the artificial intelligence tools and systems that we use every day." (0:00)
The story makes the point personal. Caire sent a job ad to her friend Jane, who replied that she could not see it. The ad had been set up to display only to users whose profiles indicated an age between 18 and 54. Jane was 60; Caire was younger and could see it. For Caire, this is where the real ethical risk sits: "So we can see that it is in the interactions between AI systems and humans that often the ethical issues are taking place." (2:21)
The zebra problem: AI only knows its data
For viewers new to the field, Caire explains AI in plain language. Today's systems mostly rely on neural networks, machine learning and deep learning. Train a model on many labelled cats and dogs, and it will recognize a new black cat. Show it a zebra, and it fails, because no zebra was ever in the training data. "So the bigger the data set, and also the more diverse the data set, the more refined will be the training model, and the more precise will be the output." (3:20)
She then returns to this "zebra problem" throughout the talk. One example is an engineer who found his girlfriend's photo labelled as a gorilla. Caire explains that labelling is still often done manually, through Amazon Mechanical Turk workers, and that wrong labels lead to wrong outputs.
Quick fixes leave the real problem in place
The responses to these failures, Caire shows, often patch the symptom rather than the cause. Google's answer to the gorilla error was to remove the label altogether. "That was a bit of a quick fix, but probably it was cheaper than to address systematically the issue." (6:01)
Image search told a similar story. A search for "CEO" once returned only white men across search engines. After the issue drew attention, the first results page improved, but researchers at the University of Washington, borrowing a stress-testing approach from security research, made a very slight change to the search term. The output reverted to outdated data with no women and no people of color, which Caire describes as evidence that nothing systematic had been done.
The stakes rise with automated decision systems used for housing and loans. Caire describes a married couple who filed taxes jointly, with the wife holding the better credit history, yet the husband received 20 times her Apple Card credit limit. As she cited, Apple attributed its algorithm to a 1974 law, from an era when women's economic situation was very different. "So our zebra problem again, wrong data set and wrong outcome." (10:19)
Below the tip of the iceberg
To explain why these problems persist, Caire turns to a report published by the US National Institute of Standards and Technology (NIST), which pictures AI bias as an iceberg. Companies are starting to tackle the visible tip with toolkits, often Python code offering fairness metrics, debiasing methods and visualization. The larger mass stays out of sight. "So, the rest of it, what is below the surface, the human biases, the systemic biases, is not taken into account, as they say, and it is still something to remain to be done." (11:44)
In the Q&A she returns to this point: most bias is unconscious and internalized, and people in the dominant group often cannot see problems they do not experience. "So the interesting part with AI's tools and systems is that they show us that we have biases because they are like reinforcing these biases." (19:27) She points to machine translation that turned gender-neutral doctors into men, and to digital assistants given female voices because, according to the research she cites, those voices seemed more passive and amiable.
What users, companies and governments can do
Caire's recommendations work at three levels. Users should stay alert: "Whenever we see some ethical issues with AI, we have to report them. We have to talk to each other." (12:19) Companies should be transparent, giving access to their data sets so people can judge whether they are diverse and gender balanced, and clearly disclosing when automated decision systems are in use so users can challenge wrong results. Teams across the whole AI lifecycle should be diverse and gender balanced, with multiple stakeholders testing and validating systems.
Finally, governments must end what she calls the legal void. "This is not possible because the companies are not there for philanthropic goals. They are there for business." (14:29) She calls for AI laws and standards, their enforcement, and legal ways to audit algorithms. Answering a question from Paula Castro on regulation, she advises following what governments and bodies such as the IEEE publish, and joining petitions and discussions. Change takes a long time, she notes, citing Apple's addition of a gender-neutral Siri voice after years of complaints, and it only happens when people speak out.
Watch the talk
Does Artificial Intelligence (AI) Support Diversity, Equity and Inclusion? by Patrice Caire, 23 minutes: https://www.womentech.net/video/does-artificial-intelligence-ai-support-diversity-equity-and-inclusion-patrice-caire-0
About the speaker
Patrice Caire is a consulting scientist, writer and ethical-AI advocate based in Luxembourg, with a PhD in computer science from NYU and the University of Luxembourg. She is a Consulting Scientist at Stanford University's Robotics Lab, has worked extensively in Silicon Valley and founded the Social Robotics Lab at the University of Luxembourg.
Questions this talk answers
Why do AI systems produce biased results?
Are quick fixes enough to remove AI bias?
What can ordinary users do about biased AI?
How can AI be regulated when there is no clear legal framework?
Why did digital assistants have female voices?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
NIST iceberg model of AI bias (statistical, human and systemic biases)
Resources mentioned
NIST report on AI bias
IEEE report on trust in AI
Cite this talk
Patrice Caire. "Does Artificial Intelligence (AI) Support Diversity, Equity and Inclusion?." WomenTech Network, 2022. https://www.womentech.net/video/does-artificial-intelligence-ai-support-diversity-equity-and-inclusion-patrice-caire-0