Round-trip 'she's a doctor' through Hungarian and AI makes her the nurse
Type 'she's a doctor, he's a nurse' into Google Translate, send it to Hungarian and back, and the roles swap. Models trained on historical data quietly carry old assumptions into tools people use every day, from photo tagging to resume screening. Jyotika Singh, Director of Data Science at Placemakr and an inventor of machine learning patents, traces where that bias enters and what it would take to break the cycle.
See where bias slips into AI models and the practical steps that reduce it
Jyotika Singh, Director of Data Science at Placemakr, explains how AI already shapes everyday products and how historical, societal and cognitive biases get into the models behind them. Using examples from Google Translate, Amazon's resume screening and text-to-image systems, she argues that diverse teams, careful testing and ongoing monitoring for data drift are needed to break the cycle of biased data.
“We often hear the argument that computers are impartial, but unfortunately, that may not be the case.”
“So overcoming bias is important and AI models can only be as good as the person behind it and the data behind it.”
What you will learn
Recognize the everyday products, from email autocomplete to shopping recommendations, that run on AI behind the scenes.
Explain how models learn from historical data and the difference between supervised and unsupervised models.
Identify where bias enters an AI system: data sampling, feature engineering, the people building it and the applications that feed data back.
Apply practical bias-reduction habits such as building diverse teams, testing models on varied inputs and monitoring for data drift.
Question the assumption that computers are impartial when using or building AI products.
Key takeaways from the speaker
Impact of AI on everyday lives
Influence of past bias in AI models
Overcoming bias
Details
Level
Beginner
Format
Talk
Technical depth
Introductory technical
For
All career stages
Length
21:43
Published
July 2022
Questions this talk answers
Where do people interact with AI in everyday life?
In autocorrect, Gmail sentence completion and spam filtering, photo tagging, Google Translate, robotic vacuums, chatbots and product recommendations. Brands also use AI to classify product reviews by topic such as fit.
How does bias get into AI models?
Through historical, societal, gender, racial and cultural patterns in the training data, through unrepresentative data samples, and through the cognitive biases of the people who engineer features and build the models.
Why does AI bias tend to reinforce itself?
Biased models shape what people see and do, which produces new data that strengthens the original patterns, so each model update repeats the cycle.
What are real examples of biased AI?
The speaker cites Google Translate swapping gendered roles, a portrait app that could not render people of color well, Amazon's resume screener penalizing the word 'women's', and Zoom cropping out a Black person's face.
How can teams reduce bias in AI?
Build diverse data science teams, understand the underlying data and what is missing from it, test models with many types of input, and monitor models over time for data drift.
About this talk
Today, Artificial Intelligence technologies are at the forefront across the globe and across industries. Each of us interact with a product or service powered by AI on a daily basis. AI and Machine Learning have provided us with a range of useful and efficient products in the last decade. However, there is a bias created alongside that has been carried forward in forms not directly visible to consumers of such technologies. Historical, societal, gender, racial, and cultural biases have made their way into models we consume each day impacting our everyday decisions. A note on overcoming such biases for the future generations of researchers, practitioners, and consumers will help the world move towards a much-needed equilibrium.
Amazon's resume screener learned to prefer men (60 s)
How to overcome bias in AI models (65 s)
Speaker: All right. Well, hello, everyone. I'm super excited to be here. It's been a fun event so far, and I've had the opportunity to look at some sessions live and watch a recording of some others. Without further ado, let me start sharing my screen. All right. Please let me know if there's any trouble looking at my screen or anything like that. Awesome. So I'm going to be talking about the impact of AI on the society. Before diving right in, I want to just quickly introduce myself. My name is Jyodhika Singh, and I work as a director of data science at PlaceMaker, which is a tech-enabled, flexible hospitality living platform. I am also a volunteer at Data Science Nigeria and Women Impact Tech, and I'm linking my social media handles here in case anybody has any questions or wants to discuss anything on this talk or otherwise. Please feel free to connect with me and reach out to me on Twitter if there's anything I can help answer.
I'm also working on a book right now, which is titled Natural Language Processing in the Real World, which is expected to release sometime in 2023. So before actually getting into the impact of AI and understanding how we use it and the very basic things that we interact with every single day have the impact on us. What is artificial intelligence? Now, it's been a buzzword for a long time now, for more about 10 years, approximately. It's been in the limelight. And there are several other terms that are associated with it that, you know, we hear very often.
So at a high level, artificial intelligence is just intelligence which is demonstrated by a computer or by a machine. So as humans, we have a particular type of intelligence, and that's because we have a brain. We have this neural network connectivity inside our bodies, inside our brain that emits signals. And we have a total neuron structure in our brain. And, you know, that sort of is part of reason why humans have this intelligence. Now, trying to give a machine a similar amount of intelligence or, well, not the same, but some amount of intelligence is essentially artificial intelligence.
Now, there are a lot of other terms which are associated with artificial intelligence, like data science, machine learning, deep learning, big data, predictive analytics, recommendation systems, computer vision. Now, there are all these terms, it's not an exhaustive list, but all of this is under AI, it's under the umbrella of artificial intelligence. So when you hear these terms, this is really like artificial intelligence and people talking about artificial intelligence through a different lens, through a different bucket.
So we actually interact with a lot of the products of AI on an everyday basis in our lives. Now, we may not realize all the things that we are engaging with that have a machine learning or an AI powered backend behind the scenes. So, for example, you know, when you're typing and you have these auto spelling corrections, well, that's learning from your data for sure.
When you're typing emails, if you've noticed very recently on Gmail, if you're typing an email, it tries to complete your sentences just based on the first word that you're putting in. The classification of email as spam versus not spam and several other folders that especially Google has been creating, like social. Promotions all of that. It's all powered by data and AI.
Furthermore, if you have noticed any social media platforms where you post images and how you see, you know, people are getting automatically tagged, maybe you posted other pictures with your friends and tag them manually before. And then the next time you upload a picture, it's able to give you a suggestion or recommendation of the tag and identify people.
And these features are also available on Google Photos and many other platforms. Well, that is also powered by AI. Robotic vacuum cleaners is a big one. Big time AI in the back end. A little helper helping you vacuum and clean your house. Google Translate. Any time we see anything that's of interest to us in a different language, we just type it on Google and translate it to another language.
Well, that is also powered by data and AI. And then a lot of other things too. So anything you search on social media, you see these recommendations that come on the side of very similar type of content. Well, all of that is also powered by AI. And similarly, when you're shopping for something, you click on a product, you have all these recommendations that come up that are relevant or similar to the product that you've been searching for or any other products that you have bought before and so on.
So it looks at consumers behaviors as well. So another example, which is slightly more detailed is going to be something like this. So, you know, during the pandemic, online shopping increased a lot. So you may have seen something like you're clicking on a product. Reviews are important. So you check out the reviews. What companies and brands have started doing is classify their comments that they receive on the products into different categories.
So this not only helps the consumer filter to the comments that are most relevant to them. So in this case, if you care about the fit of the pants the most, then you can just click on that and filter to the comments that are going to be talking about the fit. But then it also helps the brands themselves to understand how customer sentiment impacts the sales of their products or what are the limitations of their products?
How's the sizing like? What are people sentiment behind it? So that's also another another example of AI. So we've seen all these examples of what you may have seen or you do see every day in your lives that you interact with that have some sort of an artificial intelligence backend. It's important to know how these AI models are built.
And this is because we are going to dive into all the great things about, you know, AI as it relates to our use cases in the society. And then some potential biases and improvement factors. So one common approach is looking at data that already exists and then training your machine to learn from that data so that it's able to make predictions for the future unseen data as it comes.
So that is a very common pattern. And what happens is the data that already exists is sometimes very old or it's something that's already happened. So we call it historical data here. Basically, we information extract from the data and create an algorithm. And that's your model that is making these recommendations or predictions. There are two types of models.
We have supervised AI models and unsupervised AI models. In reality, there are more types as well, but this is a very popular classification of these AI models. In the supervised model, we're learning from a known input output relationship. And in an unsupervised, we are automatically detecting patterns from unlabeled data. So one example would be we saw the image tag recommendations that we have seen on social media.
So if you already have tagged a lot of your images for a particular one of your friends or yourself, the next time that it can recommend that this particular person in this image is so and so. So that is basing analysis on a supervised AI model because we know this particular face belongs to which person and unsupervised model would be just detecting faces from images or detecting other patterns from images.
So we would say that this particular face might be same in all these 10 different images, even though we don't know who that is, and the touches spaces on the different facial features that people represent. So often there are popular patterns in the data that are used to make inferences and inform model. But then they're also used to inform model applications, and we'll see what that means.
There's a lot of good that comes from AI. There's so much advanced research that is happening in the medical field and in a lot of educational research, which is tremendously helpful to mankind. One popular example is image processing of x-ray images and CT scans to actually detect things that a human eye might miss. So there have been great advances in research.
Many brands are able to automate a lot of their processes with the help of machine learning and make sense of the large volumes of data that is being produced to help businesses grow and automate. So also another factor is reduction of manual efforts. A popular example is chatbots. You may have noticed on an e-commerce website when you click on the chat option.
At first, you're not talking to a human necessarily, and it's just a computer trying to understand your questions and answer whatever it's able to before actually even connecting you to an actual person. There are a lot of other applications in climate forecasting, medical advancements, especially creating one of the recent ones, the last few years has been creating models for speech impaired individuals to understand speech and any other, you know, translation issues.
A lot of new technologies have been coming up as we hear about it all the time. We've all already seen of some basic everyday items like your email assistance, robotic cleaners, home security, personalized recommendation and whatnot. So it's really touching a lot of the industry verticals and spheres of life for individuals. Now, because we are using a lot of data, there's potential for bias to get introduced in AI models.
We often hear the argument that computers are impartial, but unfortunately, that may not be the case. How upbringing experiences and culture shapes people and they internalize certain assumptions about the world around them accordingly. AI can be looked at through the same lens. So it doesn't exist in a vacuum, but it's built out of algorithms and tweaked by actual human beings, people.
So it tends to think the way it's been taught. Now, there are a few different types of biases, including historical biases. So things that were true a decade ago that are not necessarily true today. Societal biases, gender, racial and cultural biases that if present in the data that can surface to your model. And then in the end, any application using that model will be impacted as well.
So AI bias bias in general is an anomaly in the output of a model that is built on top of machine learning and data. Now, this can be due to any cognitive biases. So biases coming from the person building the model and their understanding of the world and their data representation and the model features. And the second would be the data set itself.
So the data is not very well represented and it only represents a certain part or subset of what actually exists in the world that can introduce another set of biases as well. So this is an image that I got from the British Medical Journal, but this can actually be applied to any application that you think about. So there exists a world and there's a lot of data in the world.
Now we take that data, but we can't get all of the data that exists in every part of the world and every single user experience. So there is a subset of data, like a sample of data, which can sometimes make room for any discriminatory data biases, lack of representative, well-representative data sets. Now, once there is data, there's a lot of manipulation, feature engineering, and a lot of steps that go into model building.
So that's where any biases coming from people building models or any data cleaning factors or data filtering factors that are applied there can also introduce the bias. Then once a model is ready, it's actually used for several different applications that people, the world interacts with. So when those biases make their way into those applications in those models, people using it are actually exposed to that data.
And as a result, it further strengthens the very patterns that the data sample represented. So sometimes there is that lack of representation and this is a cycle. Now, once you have such data in the world, anytime you update your model or take another sample, it really depends on what are the strengths in the pattern scene and the cycle repeats.
So we look at some examples of AI bias. So one popular one is that if you on Google translate type, she's a doctor, he's a nurse and translate that to Hungarian and then translate the very same output back, it kind of translates that to he's a doctor, she's a nurse. Now, for people who want to change this result, there are options available to define what should be your default pronouns for any languages that have sentences that are not -- that are essentially gender neutral.
But this is because in the past, it's referencing a lot of past data where it was more common for men to be doctors and women to be nurses. But, you know, that's not necessarily true. I think these sort of biases flow into a model and this is one perfect example. Another one would be this portrait AI R generator app where you feed in a picture and it makes a portrait of you in a very authentic manner that looks very similar to how portraits were painted all the way back in the mid centuries.
So the issue there was that most well-known paintings of the era were white Europeans resulting in a database of primarily a particular color of people in the training data. And the algorithm thus was not able to take images of like Obama, Oprah, Labyrinth Cox and make appropriate portraits. So the portrait AI, of course, acknowledges this problem. The technology behind is good, but it's just an example of a data set that's not well represented.
There's several other examples with actually detecting faces and constructing images. So one of them is OpenAI released this new system that essentially turns text to image. So you input text and it will actually return an image. So it really helps you visualize what something could look like. And it's a really cool technology. But it was reported that eight out of eight attempts to generate images of words like a man sitting in a prison cell or a photo of an angry man return images of men of color.
Another example would be in a zoom background, one black individual's face was actually considered part of the background and was cropping it off. One popular one is Amazon's resume screening model back in 2014. So it was built from data from the prior 10 years. Now, historical data contained biases against women since there was male dominance across the tech industry and men were forming 60% of Amazon's employees.
So therefore, Amazon's recruiting system incorrectly learned that the male candidates were preferable. And it was penalizing resumes that included words such as women, like women's chess club champion, for example. There are several other examples across healthcare, across, you know, ads. In 2021, a child asked Alexa for a challenge to do. And Alexa said, plug in a phone charger half into a wall and then touch a penny to the exposed prong.
So it's a pretty risky thing. Now, talking about these biases is not just to point out the issues in AI, but to understand the opportunities to detecting these issues and working accordingly. So, for example, historically, men have been more into sports and the participation trends today have shifted considerably, especially in the recent Olympics. There was almost an equal representation of men and women, but the advertisement and content targeting strategies are taking longer.
So as a man, you may see more ads for sports and sports products. So you would, so would your male peers, right? Now, when you see on social media and what you see on social media and around you, further aids in solidifying your interests. And the most popular and the general trends for content classifications and content consumptions further inform more and new content creation.
So thus it strengthens the original trend and biases that leads to, you know, longer lingering ignorance around some outlier interest that should be considered in models. And of course, bias reduction is going to not, it's going to help every aspect that we considered here. So it's going to help a research technology advancements and the products that society interacts with every single day.
You know, basically speaking, understanding the context is important in these models and ensuring that you have a well represented data set. So overcoming bias is important and AI models can only be as good as the person behind it and the data behind it. Right? So it's only good as the brain behind it. There's a very urgent need of awareness and knowledge about these models, how they are built and what should be the consideration factors as a consumer of these models and also as individuals who are building these models.
We need diverse AI and data science teams so that there's good representation within each team such that the team collectively can build diverse representative models. It's very important to understand the underlying data and think of any cases that might be missing in the data and acknowledging the problem accordingly. Together, we must work towards reducing bias in models of data professionals need to test models for different types of inputs so that they're covering all the use cases.
A very important concept is data drift when past data no longer applies to current situations. So models need to be monitored over time and updated and responsibility is a big factor, not just people building the models, but people consuming the models. You know, often a lot of people say that a lot of underrepresented groups say that they didn't dream big to begin with because successful people didn't look like them.
And artificial intelligence has the potential to do so much good in the world. But when it's built on top of bias data and assumptions, it can harm how people live, work and progress through their lives. So we can together work towards reducing these biases by being attuned to the biases of the world we live in and challenging the assumptions that underpin the data sets we are working with and the outcomes that they offer.
So we can start by reading widely engaging with progressive ideas and sharing helpful articles and research that can be used to educate others. Thank you very much. It was a pleasure talking at the session. Thank you. I think we do have some time. So I'm going to see what questions are here. That is true. So I'm reading Patricia's comment here about the race corrections applied to kidney transplants.
I believe I did read this document as well. And that's a perfect example of, you know, where these biases impact models, especially when they are starting to get very widely used for applications that cater to different individuals, but it's not built with considerations of different types of individuals. Absolutely. One thing I forgot to mention is that I am going to share my slide deck on Twitter.
I'm going to tweet it out today itself. So definitely feel free to download it from there and also feel free to reach out to me if there is anything that I can help answer or anything anybody would like to discuss.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why AI is not impartial, and how to stop it repeating the past
Jyotika Singh traces how historical data carries old biases into everyday AI products and what builders and users can do about it.
In short
AI models learn from historical data, so they inherit the gender, racial and cultural patterns of the past. Jyotika Singh, Director of Data Science at Placemakr, shows how that bias flows through data samples, model building and real products, then feeds back into new data. Her answer is diverse teams, deliberate testing, monitoring for data drift and shared responsibility between the people who build models and the people who use them.
Key takeaways
The impact of AI on everyday lives
The influence of past bias in AI models
Overcoming bias
AI is already part of daily life
Singh starts by defining artificial intelligence simply as intelligence demonstrated by a machine, and places data science, machine learning, deep learning, big data, predictive analytics, recommendation systems and computer vision under that umbrella. She then points out how often people touch AI without noticing it: spelling correction, Gmail completing sentences, spam and promotions folders, automatic photo tagging, robotic vacuum cleaners, Google Translate and shopping recommendations.
"Now, we may not realize all the things that we are engaging with that have a machine learning or an AI powered backend behind the scenes." (3:04)
A more detailed example comes from online shopping, which, as she notes, grew during the pandemic. Brands now classify product reviews into categories such as fit, which helps shoppers filter comments and helps the brand understand how customer sentiment relates to sales.
How models learn, and the good they do
To understand bias, Singh explains how models are built. A common approach trains a machine on data that already exists, which she calls historical data, so that it can predict outcomes for new data. She distinguishes supervised models, which learn from a known input and output relationship (recognizing that a tagged face belongs to a specific friend), from unsupervised models, which detect patterns in unlabeled data (noticing the same face appears in ten images without knowing who it is).
She is clear that AI does a great deal of good: image processing of x-rays and CT scans that can catch what a human eye might miss, automation that helps businesses make sense of large data volumes, chatbots that reduce manual effort, climate forecasting, and models that help people with speech impairments.
Where bias enters
Because these systems run on data, bias can slip in. Singh rejects the idea that machines are neutral observers.
"We often hear the argument that computers are impartial, but unfortunately, that may not be the case." (10:37)
AI, she argues, is shaped the way people are shaped by upbringing and culture. "So it tends to think the way it's been taught." (11:03) She names historical biases (things true a decade ago that are no longer true), societal, gender, racial and cultural biases in the data, and cognitive biases from the people choosing features and representations. A data sample that covers only part of the world adds further distortion.
Using an image she credits to the British Medical Journal, she describes a cycle: a sample is drawn from the world, people clean, filter and engineer it into a model, the model is deployed in applications, and people using those applications generate new data. "And as a result, it further strengthens the very patterns that the data sample represented." (13:16)
Bias in real products
Singh walks through examples, as she cited them. Translating "she's a doctor, he's a nurse" into Hungarian and back in Google Translate returns "he's a doctor, she's a nurse", reflecting past data in which men were more often doctors. A portrait generator app trained mostly on paintings of white Europeans could not produce appropriate portraits of Barack Obama, Oprah Winfrey or Laverne Cox. An OpenAI text-to-image system was reported to return images of men of color in eight out of eight attempts for prompts such as "a photo of an angry man". A Zoom background treated a Black person's face as part of the background and cropped it out.
The best-known case is Amazon's resume screening model from 2014. Built on ten years of hiring data from a male-dominated industry, where men formed 60% of Amazon's employees, it learned to prefer male candidates and penalized resumes containing words such as "women's". She also mentions a 2021 incident in which Alexa suggested a dangerous challenge to a child.
The feedback loop in ads and content
Singh stresses that the point is not just criticism. "Now, talking about these biases is not just to point out the issues in AI, but to understand the opportunities to detecting these issues and working accordingly." (17:02)
Her example is sports. Participation has shifted, with almost equal representation of men and women at the recent Olympics by her account, yet ad targeting still leans toward men. Men see more sports ads, their interests are reinforced, and popular content trends inform new content, so outlier interests stay ignored for longer.
Overcoming bias
Her prescription combines people and process. "We need diverse AI and data science teams so that there's good representation within each team such that the team collectively can build diverse representative models." (18:52) Practitioners should understand their data, think about which cases are missing, and test models on many kinds of input. Models also need monitoring over time. "A very important concept is data drift when past data no longer applies to current situations." (19:24)
Responsibility extends to consumers too. Singh closes by urging everyone to challenge the assumptions under the datasets they use, read widely, engage with progressive ideas and share research that educates others. In a brief exchange afterwards, she agrees with an audience member that race corrections applied to kidney transplants are a clear case of bias in widely used models.
Watch the talk
Impact of AI on Society, Jyotika Singh, 22 minutes: https://www.womentech.net/video/impact-ai-society-jyotika-singh
About the speaker
Jyotika Singh is the Director of Data Science at Placemakr, where she develops pricing algorithms that optimize operations and revenue using machine learning, deep learning and AI. She holds multiple patents on classification and machine learning algorithms, volunteers as a mentor at Data Science Nigeria and Women Impact Tech, and received the Toronto Womxn in Data Science Leadership Award in 2022.
Questions this talk answers
Where do people interact with AI in everyday life?
How does bias get into AI models?
Why does AI bias tend to reinforce itself?
What are real examples of biased AI?
How can teams reduce bias in AI?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Supervised and unsupervised learning
Data drift
AI bias cycle (British Medical Journal)
Resources mentioned
Natural Language Processing in the Real World (book by Jyotika Singh)
Speaker's slide deck, shared on Twitter
Cite this talk
Jyotika Singh. "Impact of AI on Society." WomenTech Network, 2022. https://www.womentech.net/video/impact-ai-society-jyotika-singh