Does Artificial Intelligence (AI) Support Diversity, Equity and Inclusion?
Patrice Caire23:082022
A friend aged 60 never saw the job ad, and that was by design
A job ad sits on one screen and is invisible on another, because the platform shows it only to users aged 18 to 54. Biased data and design choices like this decide who sees jobs, what search returns for CEO and how much credit a wife gets. Patrice Caire, a computer scientist and consulting scientist at Stanford University's Robotics Lab, asks what it would take for AI to support diversity instead of reinforcing inequality.
See where AI bias comes from and the steps users, companies and governments can take against it
Computer scientist Patrice Caire argues that AI systems often reinforce society's inequalities because they are trained on biased, incomplete or outdated data and built by teams that do not reflect society. Using everyday cases such as age-targeted job ads, mislabeled images, CEO image search and Apple Card credit limits, she shows where bias enters and calls for transparency, diverse teams, user vigilance and AI standards and regulation.
“So we can see that it is in the interactions between AI systems and humans that often the ethical issues are taking place.”
“So the bigger the data set, and also the more diverse the data set, the more refined will be the training model, and the more precise will be the output.”
What you will learn
Explain how training data size, diversity and labeling quality shape what an AI system outputs.
Spot everyday cases where AI reinforces bias, from ad targeting and image search to credit decisions and translation.
Distinguish the statistical and computational biases that toolkits address from the human and systemic biases they miss.
Report and challenge AI outcomes that feel discriminatory instead of letting them go.
Advocate for transparent data sets, gender-balanced AI teams, multi-stakeholder testing and enforceable AI standards.
Key takeaways from the speaker
Understand that today, AI impacts almost everyone, but that the negative impacts are particularly affecting women and minorities.
Learn why scientific experts are urging governments and companies to take action and develop AI standards and regulations.
See how critical it is for all citizens be aware of AI biases and learn how to identify them.
Grasp the significance of the input data used to train AI models.
Realize that to prepare the future, women and minorities should become full participants in the design and development of the AI technologies that shape our world.
Details
Level
Beginner
Format
Talk
Technical depth
Introductory technical
For
All career stages
Length
23:08
Published
July 2022
Questions this talk answers
Why do AI systems end up biased against women and minorities?
They learn from training data that may be incomplete, wrongly labeled or outdated, and are built by teams in which women make up only about 23% of tech workers, as the speaker cited. What is missing from the data, like the zebra in her example, cannot be recognized correctly.
Do companies' quick fixes solve AI bias?
Often not: Google removed the gorilla label instead of fixing the problem, and University of Washington researchers found that a slight change to a CEO search term brought back outdated results with no women or people of color.
What part of AI bias are current fairness toolkits missing?
Citing a NIST report, Caire explains that toolkits address only the tip of the iceberg, statistical and computational bias, while human and systemic biases below the surface remain unaddressed.
What can individuals do about biased AI?
Stay on the lookout, report ethical issues, talk to each other and speak out when something feels discriminatory, while demanding transparency from companies about their data sets and automated decisions.
How can AI be regulated when legal frameworks are unclear?
Follow what governments and bodies such as the IEEE are publishing, join petitions and discussions, and press governments to set and enforce AI standards and ways to audit algorithms.
About this talk
Today, Artificial Intelligence (AI) is deployed everywhere—from smartphones to search engines—raising ethical questions particularly concerning women and minorities. However, governments and companies have been slow to address AI biases and discriminations. In this talk, I focus on the question whether AI supports diversity, equity, and inclusion. Using daily life use cases and examples, I illustrate the significance of the input data while training models. Then I stress the importance to recognize, understand and remove biases in this input. All along, I present recommendations like establishing AI standards and regulations in order to fill the legal void in which companies operate and including diverse demographics that reflect the composition of our society, to design, develop and test current and future AI technologies.
Thank you very much for having me. I'm very happy to be here with you today. I am Dr. Patrice Caire and I do research in artificial intelligence and robotics. I have a PhD in computer science and I've worked extensively in Silicon Valley and in academia. As you know everywhere very well today only 23% of women are working in the tech industry today. No wonder then that the point of view of women is not included in the design and development of the artificial intelligence tools and systems that we use every day. Instead this system often reinforce these inequalities and stereotypes of our society. It is exactly for these reasons that I have decided to focus my work on this question. How can artificial intelligence systems support diversity, equality and inclusion? Let me start with
the story. I have a friend Jane who was looking for a job. So I see an ad, I sent it to her and she contacts me right back and says, "Patrice, I cannot see the ad." And I said, "But Jane, I'm looking at it right now. I have it under my eyes. What was happening?" Well, what was happening was that Amazon, who was trying to hire people between the age of 18 and 54, try to, which is illegal, of course, arranged something with, at the time, Facebook, now Meta, to bypass the law. So how did they do that? Well, they displayed the ad only to people, to users whose profiles were indicating that they were between the age of 18 and 54. This is illegal. This is a way to bypass the law, and this is not
fair. My friend Jane was 60 at the time, so she could not see the ad. She could not see it because it was not shown to her. And I was below 54, and I could see the ad. So we can see that it is in the interactions between AI systems and humans that often the ethical issues are taking place. But we use so today, many, many different kinds of systems. For example, when we log on to our phone, we use image recognition. When we also dictate, we use NLP, natural language processing. And of course, today, there are more and more robots with embedded AI systems. But what are AI systems today? Well, there are systems which display intelligent behaviors, which display intelligent behaviors. And they do that by analyzing
their surrounding and taking actions to fulfill some goals. What is behind that? Well, today, the main approach is neural network, machine learning, and deep learning, and so forth. So for people who are not familiar with these topics, here is a very high level example, let's imagine that we have our system and there is a new data, in this case, this black cat, and we want to know what it is. The system outputs. It's a black cat. It's a cat. So what was happening? How did was that possible? Because in the training model, we had used a data set, which was a data set, which was containing lots of cats, labeled cats, and lots of dogs, labeled dogs. So the bigger the data set, and also the more diverse the data set, the more refined will be the training model, and the more precise will be the output.
So we see this data set is important. So let's consider now that we put as new data, another animal, whatever, zebra. So we have a zebra. And what is our system going to do? Well, it is not going to recognize it, of course, because there was no zebra in the training data set. All right. All right. All right. Well, we see, again, the importance of this. So let's look in real life how this can play out. In fact, I'm going to show you an image. There was this engineer, who was looking at images, and suddenly, he sees his girlfriend.
He sees his girlfriend. But, oh, shockingly, she was labeled as a gorilla. This was terrible, of course, this is not possible. And what was happening there? Well, the data had been wrongly labeled. And this happens sometimes, very often, because it is still done manually, this labeling, and by so-called Amazon Mechanical Turks. And this sometimes causes difficulties when, in these cases, wrong labels are used in images. So this is like our zebra before. A zebra problem again. The data set was not of higher quality.
So we can see. So we can see that this was an issue. So, of course, Google tried to address this issue. And how did they do that? Well, a couple of years ago, they just removed the gorilla label. That was a bit of a quick fix, but probably it was cheaper than to address systematically the issue. But we have other examples. For example, image search is very important. And when you looked at image search a few years ago, you could return, it would return for the term CEO, it would return this output.
So, only white male. Only white male. How is that possible? Our zebra problem again. Yes, the data set didn't include many women, probably any, for the CEO term. So many that maybe the team, which was working on this issue was not combining many diverse population or didn't have many women among them. But in any case, it was not addressed. And it was in all the different search engines.
So what happened was that to check on the issue, which had made quite a scandal, the University of Washington earlier this year decided to do a project to see what was happening there. So they decided to check and what was the issue when you were searching for CEO. And very nicely, it displayed here on Google, the more statistical representation of what today the CEO can be.
Well, they decided then the University of Washington decided to go further and to use a technique used by security and privacy researchers when they do test crashes for cars for safety of the cars. In these cases, what do they do? They test, they challenge the computer systems by a number of examples and they choose to see how they hold up the system hold ups.
So they decide, the University of Washington decided to do a very, very slight change to start with. And so they just added a search term to this very slight change. An enormous difference happened in the output. And here is what we got. Our zebra problem again. There was no women again in this in this output, no people of color. It was outdated data which had been used for to fix the issue of the first page. And nothing had been done to do a systematic approach to the whole problem.
So this is an example of these are a few examples there, but we can also see sometimes where there are very, very, the impact can be very dramatic on our lives. For example, when we use automated decision systems, it is used very pervasively today for many issues for housing and for loan, for example. And there was this couple who was filing husband and wife. They were filing together on their tax return and they were actually the wife had the better tax credit tax history than her husband. But they had a new project and they wanted to check their credit line on the Apple card.
Now, what happened when they saw that their husband had 20 times the credit limit than his wife? How was that possible? They contacted Apple and, you know, requested to have some explanation. And as an explanation, Apple returned the answer that their automated decision algorithm was actually based on a 1974 law, which was, of course, bringing some totally outdated output when the data set was used in that way.
Because the situation of women in 1974, of course, especially the economic situation was very different from today. So our zebra problem again, wrong data set and wrong outcome. But we saw now many, many different, different, different, different examples of biases and different way to deal with it. But how was, how can we manage these, these biases? So, this was what the NIST, the US National Institute for Standard and Technology, addressed this year. They published a report in March. And in this report, they showed this picture.
So, what is pretty interesting there is that what we are starting, just starting to address and to look at is the tip of the iceberg, meaning that we are looking at, you know, the problems, which are biases, which are statistical and computational biases. So, the rest of it is not addressed at all. And the rest is pretty important. So, the point, the tip of the icebergs that we're addressing today is main, is addressed by many companies by providing toolkits.
Toolkits, they are often, which are often called Python code, using Python code, and providing a number of metrics for fairness, and also providing decision, debiasing methods, and as well as visualization. So, the rest of it, what is below the surface, the human biases, the systemic biases, is not taken into account, as they say, and it is still something to remain to be done.
But, we still can do something. We can do something. In fact, what we can do is we can be on the lookout. Whenever we see some ethical issues with AI, we have to report them. We have to talk to each other. We have to try to find a solution and not let it go. We have to be aware of this situation.
And, of course, we have to ask and demand that companies be responsible and they can, they are transparent. For example, they should be giving access to their data sets so that we know the composition, if they are diverse, if they are gender balanced. And, of course, the responsibility is enormous for these companies, because in the automated decision making algorithm that we see more and more in our lives, and which impacts us directly, this should be really, companies should be really writing right up the sites that they are using these automated systems.
And so that we as users have the opportunity to address and to contact and to say, well, there are some issues there. We have to fix something because the result was not right. And, of course, we have seen that we are only 23% of women. So, yes, we have to make sure that we have some teams which are diverse and balanced, gender balanced in the whole AI lifecycle.
They have to be, they have to be diverse and representing society. We cannot just have a few part, a few percentage of the society being there doing all this work that we use all the time. And we have to make sure so that we have multi-factor and multi-stakeholders to test, to validate, to evaluate, to do the testing scenario for all these systems.
And finally, we have to make sure that companies are not operating in the legal void that they are operating in today. This is not possible because the companies are not there for philanthropic goals. They are there for business. And we have to make sure that our governments and our institutions are really there working for us to make sure that there are laws and standards, AI standards, to give a framework to all the companies producing AI systems.
And we have to make sure that all these are enforced and that we have set up also some legal ways to audit algorithms, all algorithms. So we see we can do a lot. And to conclude, I'm going to let you read this sentence. So what do we want to do? Do we want to let AI systems reinforce the stereotypes and inequalities in our society?
Or do we want to make sure that we work together and we try to address these issues and work towards AI systems, which are supporting diversity, equality and inclusion? This is our choice. Thank you very much for your attention. And they'll be taking questions now. So there is a question actually here for from Paula Castro. There is if there is no clear legal frameworks, how can we make sure AI is being regulated?
Well, that is really a good question because we have to follow the fact that the government are publishing some results all the time and some laws. So we have to follow the issue and see and see what are the the bodies who are actually working on this. So that way we can make sure that we can also when there is some petitions to us, because, for example, there were many petitions a few years ago from scientists who were trying to regulate for AI.
For example, the weapon industry, the autonomous weapon industry, which is a really critical area. So we have to make sure there that something is happening and maybe join these discussions for the free frontier or from the electronic frontiers and all the AI bodies which are working on this topic. So there are there are some they are starting the IEEE, for example, has been putting a report together, starting with the looking at different areas such as the area of trust.
So how can how can how can trust be a fundamental area to be able to build some more gender equality and also more some some specific standard for that. So there are some some some some area every country actually has started to set up to be in movement to have something like this. So it is for us. And that was my first point. It is from from us to also try to get informed and and try to make sure that we follow the advances which are being done in this area.
If there are other questions, please don't hesitate. I mean, gender equality is a is a huge question there. I mean, we cannot being only 23 percent, 23 percent of women in the tech industry, of course, you know, we are not represented. And it's not to say that the white males who are working on these products and on these softwares are not biased, consciously biased.
It's we have a lot of biases which are just in conscience. We have we have interiorized them. So we don't know. I mean, people often don't know that they have that they are biased until they are being shown that this is at the expense of other people, other groups, other minorities or other, you know, ethnic or other, you know, people from the society.
Often when you are in a part of the control group, you don't see that actually others do not have these the problem that you you don't see because you don't have them. So this is why the NIST was showing this importance of the human biases and this and the problem of of of the systemic biases. We all have biases. So the interesting part with AI's tools and systems is that they show us that we have biases because they are like reinforcing these biases.
And we see them even more because it's so powerful. I mean, there are many instances of that in natural language processing, where if you translate one language into another, for example, it was done with them with the doctors and nurse in some in some languages, the doctor is neither feminine or masculine. And in some languages where it has a gender, it was translated by a male automatically, even though sometimes it was that it was a female. So female are considered nurses and doctors and males are considered nurse are considered doctors, excuse me.
So it is all these biases, which are not good because they reinforce also what is happening in our society. For example, another example, for the past 10 years, we have been having this dictation on the on the phone, like Siri, like what we see on Amazon, I mean, everybody now as a as a digital assistant. Well, from the start, the digital assistant, where feminine voice, female voice, and they have been even now for some digital assistant, it's still female voice. Do you know why that is? Well, simply because the research social science research has has shown that social that female voices appear to be more passive, and more amiable.
And so people preferred having a digital assistant, which would be passive, rather than some thing they would perceive as more controlling or more, more dictatorial or more, you know, male oriented if it was a voice. In fact, in fact, it's only this year that Apple added to its Siri voice, gender neutral voice. So we are making progress because people have been complaining for ages to see that these voice assistants would be more gender bias, less gender bias. And this is has finally occurred.
But it takes a long time and it takes a long time and it takes people to be on the lookout. That's what we have to be on the lookout. Whenever we see something which doesn't feel right, where you feel that you are being being not favored, but on the contrary, we are being discriminated, then you should just speak out and demand that things change, because it's only like this, that they will change.
Thank you.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why AI reinforces inequality, and what it takes to change that
Computer scientist Patrice Caire traces AI bias back to training data, team composition and a legal void, and lays out what users, companies and governments can do.
In short
AI does not automatically support diversity, equity and inclusion. Patrice Caire argues that, left alone, AI systems often reinforce society's stereotypes because they learn from incomplete, mislabeled or outdated data and are built by teams that do not reflect society. Change, she says, depends on vigilant users, transparent companies, diverse teams and real AI standards and regulation.
Key takeaways
Understand that today AI affects almost everyone, but its negative impacts fall particularly on women and minorities.
Learn why scientific experts are urging governments and companies to act and develop AI standards and regulations.
See how critical it is for all citizens to be aware of AI biases and learn how to identify them.
Grasp the significance of the input data used to train AI models.
Realize that, to prepare the future, women and minorities should become full participants in the design and development of the AI technologies that shape our world.
Bias happens where AI meets people
Caire opens with a statistic she uses throughout the talk: as she cited, only 23% of the people working in the tech industry are women. With so few women involved, she argues, their point of view is missing from the design of the AI tools everyone uses daily.
She makes this concrete with a personal story. She forwarded a job ad to her friend Jane, who could not see it. The employer, Caire explains, wanted to hire people aged 18 to 54 and arranged with Facebook (now Meta) to show the ad only to users whose profiles fell in that range. Jane was 60; Caire was younger and could see it. Her conclusion frames the rest of the talk: "So we can see that it is in the interactions between AI systems and humans that often the ethical issues are taking place." (2:21)
The zebra problem: training data decides the output
For viewers new to the subject, Caire gives a plain-language explanation of how today's AI works. Systems built on neural networks, machine learning and deep learning are trained on labeled examples, such as many pictures of cats and dogs, and then classify new inputs. "So the bigger the data set, and also the more diverse the data set, the more refined will be the training model, and the more precise will be the output." (3:20)
Show that same system a zebra, and it fails, because there was never a zebra in the training data. Caire turns this into a running motif, "the zebra problem", and applies it to real cases. An engineer found his girlfriend labeled as a gorilla in an image system, a result Caire attributes to mislabeled data, since labeling is still often done manually by Amazon Mechanical Turk workers.
Quick fixes that hide systemic problems
The company's response, she says, was to remove the gorilla label entirely. "That was a bit of a quick fix, but probably it was cheaper than to address systematically the issue." (6:01)
Image search followed the same pattern. A few years earlier, searching for "CEO" returned only white men. After the scandal, the first page was corrected, but University of Washington researchers, borrowing a stress-testing technique from security and privacy research, made a very slight change to the search term and saw the old results return, with no women and no people of color. In Caire's reading, outdated data had been patched on the surface and the underlying problem never addressed.
The stakes rise with automated decision systems used for housing and loans. Caire describes a married couple who filed taxes jointly, with the wife holding the better credit history, yet the husband received 20 times her credit limit on the Apple Card. According to the speaker, Apple explained its algorithm was based on a 1974 law, from a time when women's economic situation was very different. "So our zebra problem again, wrong data set and wrong outcome." (10:19)
The iceberg: what fairness toolkits miss
To explain why these problems persist, Caire points to a report published in March by the US National Institute of Standards and Technology (NIST). It pictures AI bias as an iceberg. Above the surface sit statistical and computational biases, which companies now tackle with toolkits, often Python code with fairness metrics, debiasing methods and visualization. Below the surface sit human and systemic biases, which, she says, are not yet taken into account.
In the Q&A she adds that most bias is unconscious and internalized, and that AI makes it visible by amplifying it. She cites machine translation that turns gender-neutral words into male doctors and female nurses, and digital assistants given female voices because, as she cited social science research, those voices seemed more passive and amiable.
What users, companies and governments can do
Caire's recommendations work at three levels. Individuals should stay alert: "Whenever we see some ethical issues with AI, we have to report them." (12:19) Companies should be transparent about the composition of their data sets and clearly disclose when automated decisions are being made, so users can challenge wrong results. AI teams should be diverse and gender balanced across the whole AI lifecycle, with multiple stakeholders testing and validating systems.
Finally, she argues, governments must end the legal void companies operate in. "This is not possible because the companies are not there for philanthropic goals. They are there for business." (14:29) That means AI standards, enforcement and legal ways to audit algorithms. Asked how to act without clear legal frameworks, she urges viewers to follow what governments and bodies such as the IEEE publish, and to join petitions and discussions. She closes with a choice: let AI reinforce inequality, or work together towards AI that supports diversity, equality and inclusion.
Watch the talk
"Does Artificial Intelligence (AI) Support Diversity, Equity and Inclusion?" by Patrice Caire, 23 minutes: https://www.womentech.net/video/does-artificial-intelligence-ai-support-diversity-equity-and-inclusion-patrice-caire
About the speaker
Patrice Caire is a consulting scientist, writer and ethical AI advocate with a PhD in computer science from NYU and the University of Luxembourg, and she is a consulting scientist at Stanford University's Robotics Lab. She has worked extensively in Silicon Valley, founded the Social Robotics Lab at the University of Luxembourg, and speaks in plain language about tech topics.
Questions this talk answers
Why do AI systems end up biased against women and minorities?
Do companies' quick fixes solve AI bias?
What part of AI bias are current fairness toolkits missing?
What can individuals do about biased AI?
How can AI be regulated when legal frameworks are unclear?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
NIST iceberg model of AI bias (statistical, computational, human and systemic biases)
Resources mentioned
NIST report on AI bias (published in March)
IEEE report on AI and trust
Cite this talk
Patrice Caire. "Does Artificial Intelligence (AI) Support Diversity, Equity and Inclusion?." WomenTech Network, 2022. https://www.womentech.net/video/does-artificial-intelligence-ai-support-diversity-equity-and-inclusion-patrice-caire