Default AI settings can hide bias behind high accuracy scores
A face classifier can report high accuracy while the images it gets wrong belong to people of Asian background or Black people, simply because nobody tuned it beyond its default settings. Anyone uploading data to an off-the-shelf AI tool risks shipping that. Marjory Da Costa Abreu, who has spent around 20 years researching artificial intelligence, asks what it really takes to use these black boxes responsibly.
Learn five practical rules for using off-the-shelf machine learning without reinforcing bias
Marjory Da Costa Abreu, a Senior Lecturer at Sheffield Hallam University, argues that the push-button popularity of machine learning tools hides a serious risk: people use black-box models with default settings they do not understand. She shows how untuned models can report high accuracy while their errors fall on specific groups, and closes with practical rules for understanding data, techniques and parameter tuning.
“But worst of all, when you don't change and you don't refine the parameters, you can create solutions that look fine.”
“So each technique is dedicated. It should be fine tuned to the data set to the problem that is trying to solve.”
What you will learn
Distinguish artificial intelligence, machine learning and deep learning as nested concepts rather than competing ones.
Spot how default settings in off-the-shelf tools can produce models that look accurate but do not really classify.
Check which groups a model's errors fall on, not only its overall accuracy, to catch reinforced bias.
Apply five rules before using machine learning: understand your data, understand the technique, change the defaults, be creative and treat black boxes with care.
Challenge the belief that AI is a panacea that will solve every problem or replace people.
Key takeaways from the speaker
gender equality
artificial intelligence
bias in data
fine tuning
Details
Level
Beginner
Format
Talk
Technical depth
Introductory technical
For
Student, Early career, Mid-career
Length
20:20
Published
June 2021
Questions this talk answers
What is the difference between AI, machine learning and deep learning?
Deep learning is a set of techniques inside machine learning, and machine learning is a subset of techniques within artificial intelligence. They are nested, not separate things.
Why are default settings in off-the-shelf machine learning tools risky?
Defaults are not optimized for your data or problem. Using them can lock a model in a local optimum, reinforce bias and produce results that look good without really classifying anything.
How can a model with high accuracy still be biased?
In a gender-from-faces example, most images may be classified correctly while the misclassified ones belong to people of Asian background or Black people, so the model reinforces bias despite its accuracy rate.
Will AI replace people in the workplace?
The speaker argues it will not: well-tuned solutions are built to do specific tasks really well, not to become or replace human beings.
What should you do before using a machine learning model?
Understand your data first, understand the overall workings of the technique, change the default settings through fine tuning, be creative, and use black boxes only with careful analysis.
About this talk
The understanding that the widespread use of technology by the citizen will always benefit of the wider society is conditioned by this use and dependent on it. Countless of the most basic and fundamental public and private activities, nowadays, necessarily pass through access to tangible and intangible computational innovations.
However, there has been an increase in social inequalities which can be linked to limited access to educational opportunities, public health inefficiency, institutionalised violence (gender, race, for instance), irrationality in the exploitation of natural resources, corruption, lack of transparency and abuse of power and so on which can inadvertently powered by the popularisation of Social Media platforms.
On top of all of this, the use of artificial intelligence has become the panacea that will solve all the issues of the world. At a first glance, it might look like a good and fair solution, but the indiscriminate use of black boxes (Python scikit learn/Tensorflow, Weka, MATLAB ML toolbox, Google AI, etc) without the AI models understating and the human bias in data can make this use a powerful enforcement of the current inequalities.
Thus, this talk with discuss the social related issues of using AI without understanding it.
Speaker: Well, thank you very much for attending this talk. I have to say, I'm so excited to be here. And, oh, okay. You can't hear me. How about now? Testing, testing. Can you hear me? Fabulous. Okay. Right. So I'm so excited to be, you know, attending and presenting in this amazing event. I have heard some really inspiring women so far, and I hope to, you know, keep interacting with everybody. So let's start talking about my topic. So, first of all, I would like to present myself. So my name is Marjorie. I'm originally from Brazil, but currently I am working as a senior lecturer in Applied Artificial Intelligence in Sheffield Hallam University, the United Kingdom. And my work, I'm working as a lecturer in Sheffield Hallam and part of my research. So I have been working in academic in settings for around 20 years. And all my research has always focused on artificial intelligence, the theoretical part.
Fabulous. So I have supervised research students and worked developing new models. And my work focuses on creating new algorithms. So I evaluate the current ones. I apply the current algorithms to, you know, specific application areas, and I evaluate how they behave and how they work. Fabulous. It's interesting. You know, in the last, I would say five to six years, it has become very, very evident that we have a whole new understanding of, you know, intelligence systems, artificial intelligence, and its application. This is possibly due to having, you know, more data available. So big data has a big impact on this. And also having more processing power. So we have stronger, stronger, stronger, stronger, more powerful computers that can deal with that specific amount of data. And therefore, it can, we can use intelligent systems to, you know, process and understand the data.
So this has caused some good, but also some bad in the world. It's very interesting that I was listening to some of the other talks and bias is something that a lot of you presenters are talking about. So when we talk about using data, big data, huge amounts of data, it's very common that the real data will have some sort of bias. And what I wanted to discuss with this talk today is actually the popularization of the use of artificial intelligence, which in my understanding, it can be a good thing, but also the bad understanding and the misuse of this kind of techniques. And I'm going to explore more of that in a bit. Okay, so what we have, as I said, it's lots of data. So data is available. And that brings a few different issues. So you have problems
regarding access to data. So you have lots of systems, people don't really understand what is happening with their own data. And then they, you know, give their data willingly. So it's not unusual to have people sharing lots of personal things on social network, and on, you know, with their peers or their friends or whatever. And then this data is being used for these big tech companies. But not only that, you have a lot of people, like surfing like surfing in the trend of using AI. And then using AI demands from the people who are developing new models, a very specific kind of training. So it's not, it's quite easy to go to a Python platform, or to, you know, a Microsoft one, or a Google one, press some buttons, and then getting some results for a neural network, a decision tree, a random forest, a convolutional neural network.
And that sounds really amazing. And that can, you know, make people have more access to these kind of techniques. But understanding how they work, would have a much bigger impact on the actual results. So the first thing that I want to talk is this misunderstanding of the concept. So artificial intelligence is everything that is intelligent. So machine learning is a set of techniques which are inside artificial intelligence. It's not everything. It's not the same thing. Artificial intelligence is different from machine learning. Machine learning is a subset of techniques within artificial intelligence.
And then you have deep learning, which is something that became popular and became possible because of the deep learning of the big data evolution. So we have deep learning, which is something inside machine learning. And then you have machine learning, which is something inside artificial intelligence. So it's not uncommon to see people saying, oh, machine learning is different from artificial intelligence. Are you using artificial intelligence? It doesn't exist. So understanding what you have, I think it's a very important step. So the first step is understanding that you have lots of data.
Understanding that you have lots of data. The data has bias and you should be aware of the bias. Second step is understanding that machine learning is not the solution for all our problems. And then you have different levels of techniques that you can use within machine learning and then within artificial intelligence. So not everything that is intelligence is machine learning. And then deep learning is not something different from machine learning. OK, so this is the second point. Then you have lots of models. You have something, some of the techniques, they could be potentially well explored using this big data reality.
And then you have lots of technology. And then you have lots of other techniques that can be used, not necessarily exploring the big data reality, but that could cause very much similar results without the need for creating those big, you know, very powerful solutions that don't really deal with lots of variations in the data and so on and so forth.
So the second thing, as I said, it's important to understand the whole context and then what you have available. But not only that, and this is the main point of my talk today that I would like to talk to you. So my main concern with the widespread use of machine learning specifically, which is what is mostly available on the, you know, solutions off the shelves that you can find online is the misuse of the black box.
And why do I say the black boxes are misused? Because the fact that you make this so accessible, anyone can go to Python, to a platform, to Google, to Microsoft or whatever, and then upload some data, press some buttons and get some results. But understanding what those results are is so important. And it's so important to understand how to fine tune or parameter tune the techniques.
So more often than not, and I'm going to ask you this, and I would like you to reply on the chat box. Have you ever used any black box, any machine, you know, any technique which is off the shelf that you didn't implement? Can you say yes in the chat box or no, if you haven't? So can you, no, Savita didn't use, who, everybody else, who, who used it?
Can you tell me yes or no quickly? No. Oh, very good. I have a very good audience. So what happens is when you have a technique, for instance, you have a technique called neural network, which is something that is very popular. A neural network has several different parts on it that you could potentially. What happens is in the black boxes in the off the shelf solutions is that they come with what we call default settings.
So they have predefined numbers for these parameters for these, you know, configurations of the technique. And what most people do, most people who don't have the understanding, the training to, you know, organize this, they just use the default settings and then they get the results and then off they go, you know, presenting that, selling that solution. And this is a very big problem because, you know, the default settings, they are not there to be used.
They are there to be changed. And then when, when you get a set of data and you use the default settings, you are by no means optimizing the results of that technique. When you are not optimizing it, you can reinforce bias. You can be locked in a, what we call a local, you know, optimum. So you don't have the full potential of that technique.
But worst of all, when you don't change and you don't refine the parameters, you can create solutions that look fine. So they look, they have what we call, you know, high acceptance rate. They have lower arrows, high accuracy and so on and so forth. But they don't really classify anything because you haven't fine tuned the technique. So this is a major problem.
And I can give you some examples of what can cause, what, what, what are the impacts. So this is completely directly linked to the actual data set. So imagine that you have a data set of faces. So you want to recognize face faces. So you want to say if this person is a female and then this person is a male.
So for instance, you want to identify gender from faces and then you want to use CCTV camera videos. OK, to do that. If you select a group of pictures or frames from a video and then you do it in a very small. So you have a very small selection of videos of pictures from these videos. You might have a very good result.
You might have, you know, very high accuracy. So the the the amount of images that are correctly classified classified is high. But if you don't know how to fine tune and you don't know how to analyze the results, the actual in actual fact, you might be excluding the arrows, even though you have a high accuracy rate, you have a high acceptance rate.
The arrows that are caused, they exist, they might be linked to specific groups. So if you want to identify genders, so male and female, for instance, you might get lots of accepted, you know, classified images. But the wrongly classified ones might an Asian background or they might be black people. And then you are creating, you are reinforcing the bias.
Another example, one of my research students, he's doing he's doing mining on judicial sentences. So if you are if you want to classify, you know, judicial judicial sentences, so law stuff, you normally use a processing language. So natural language processing techniques, which is one kind of, you know, machine learning model. Then what happens is if you don't fine tune, if you don't optimize the techniques, the models, what happens is, is that you reinforce the words that are mostly linked.
They are mostly mostly present on the specific sets of sentences that you are analyzing. So you don't really understand what is going on with the technique. You don't really understand what is the relationship between the technique and the data set and the results. So if you just go to the black box, upload the data set, press a button and then get the results.
That's not enough. We should be, you know, focusing on getting people to understand how to fine tune machine learning models. So we could have better quality, you know, solutions, and then everybody would benefit it. So society would have better solutions. People would be able to improve the quality of their solutions. And then the update process of everything that we are using can be easier as well.
The default settings, they are quite, you know, interesting. I have been to different talks where people would say to me, but, you know, the default settings, they are there because they are the best optimized settings that you can get for the techniques. And that's really not the case, you know, some people, they had this illusion that the default settings, they were selected, but there is no such a thing as a good single configuration of a technique that will solve everybody's problem.
So each technique is dedicated. It should be fine tuned to the data set to the problem that is trying to solve. This brings me to the next problem. And the next thing that I wanted to dismystify today with this of this talk, which is the sensation and the feeling that people from all the areas, apart from, you know, computer science, computer engineering, looking at machine learning as the, you know, panacea, the holy grail, the only possible solution that can, you know, make our lives better.
This is better, which is again, absolutely not the case. So you have two issues with people from other areas. One of the issues is people thinking that the technology, which is really bad because they think that solution, which is no very magical and very hidden and very complicated, which is not true. But they think that it's going to replace them in the workplace.
This is never the case. Another thing that I want to dismystify is the fact that, you know, the current power power that we have, you know, the technological power and then the dedicated solutions that really work because they went through the process of fine tuning. They solve specific problems. They are not there to solve, you know, to become a human.
They are not going to become, you know, to replace human beings. That's not the goal. The goal is to have, you know, specific tasks done really, really well. The other thing that happens is because people don't really understand, you actually have the opposite reaction. So people in higher, you know, roles in institutions, being it public or private institutions, they see potentially, you know, increase in gain in profit because they have, you know, a more accurate, a more precise running of their businesses.
So another example that I can give you is, you know, in the Brazilian judicial system, they want to replace, they want to use a decision support system that does classification of, you know, sentences. So it gives you a suggestion of the outcome of the sentence to a judge, for instance. And this was done not following fine tuning, but the judges, they want to make their lives easier, even though this might have a really big impact on the lives of the citizens.
They are, you know, opening those cases. So they don't really care about, oh, what is the impact of this badly, badly designed AI to the, you know, the population. They don't really care about that. They want, you know, facility, they want to have everything made quicker. And this is a really big problem. So this is something that we must fight as well as a group, as a team, as a community that, you know, works and studies artificial intelligence, machine learning models and so on and so forth.
So just to wrap up my talk today, I hope you are following the idea that I wanted to bring today was this mystifying the use of AI. So AI is not going to solve everything. AI can be a very powerful solution, can be a very supporting tool, but it's not going to save everything and to deal with every single problem that we have.
The first point that I wanted to raise is understand your data. So if you want to use machine learning, make sure that, first of all, before you even get to the model they want to use, you must understand the data they want to use. So if you understand the data, if you are not this area, come and find someone, go and talk to that person today, especially so you will understand the data.
The second one is do not use techniques that you don't understand. If you obviously you don't need to understand to be a, you know, a mathematician, a specialist on the workings around of a support vector machines or deep learning neural network, for instance. But you should understand the overall workings of that technique that will give you understanding and power and knowledge to be able to do the next thing.
Change the default setting. So you will be able to do the fine tuning, the parameter tuning that we all that I like to talk to, you know, my students, my people that I talk to and everybody else. So, oh, Paula, if you want to have to talk to me about that, I have a few, you know, research on gender bias in data.
I'm more than happy to collaborate with you to have to, you know, exchange some ideas. The next thing is be creative. Don't use what everybody else is using. Make sure that you know what is available and then, you know, go and try, but always changing the default settings. And the next thing is be aware of black boxes, you know, off the shelf solutions.
They are very good, very useful. They can be used. No problem whatsoever with that. But they must, they must be used with careful analysis and understanding. And then if you want, you can talk to me. I'm more than happy to, you know, discuss this specific specifics. I am on Instagram. I'm going to post my profile on Instagram.
You can find me there. You can find me on LinkedIn. You can find me on, you know, Facebook. You can find me on Twitter as well. I'm always, you know, at Prof Marjorie. I'm doing research on theoretical AI, not only machine learning, but other techniques as well. I'm more than happy to discuss, you know, I'm very willing to, you know, get involved with the social part of the ethical part of using AI, because I'm very worried about, you know, this kind of misuse.
You know, just want to, if you want to talk to me, get in touch. I'm more than happy to talk to you. And that's it. This is what I wanted to talk today. And thank you again for coming. And please, please enjoy the rest of the conference. It's amazing. We can meet lots of amazing women. And thank you very much for attending. Bye-bye.
Bye-bye.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why default settings make AI tools quietly reinforce bias
Marjory Da Costa Abreu explains why push-button machine learning needs understanding, fine tuning and careful analysis before anyone trusts its results.
In short
Machine learning has never been easier to use: upload data to a platform, press a few buttons and a neural network or random forest returns results. Marjory Da Costa Abreu, Senior Lecturer at Sheffield Hallam University, argues that this ease is exactly the danger. Models run on default settings, built on biased data and never fine tuned, can report high accuracy while concentrating their errors on specific groups of people.
Key takeaways
Gender equality
Artificial intelligence
Bias in data
Fine tuning
Why AI suddenly seems everywhere
Da Costa Abreu has worked in academic settings for around 20 years, with research focused on the theoretical side of artificial intelligence: creating new algorithms and evaluating how existing ones behave in specific application areas. She traces the boom of the last five to six years to two forces, as she describes them: far more data and far more processing power to handle it.
That combination brings benefits and harms. People share personal information on social networks without understanding what happens to it, and that data feeds large technology companies. More importantly for her argument, big data is rarely neutral.
"So when we talk about using data, big data, huge amounts of data, it's very common that the real data will have some sort of bias." (2:26)
At the same time, many people are riding the trend of using AI without the specific training that building good models demands.
AI, machine learning and deep learning are not rivals
Before tackling misuse, Da Costa Abreu clears up a common confusion. People often talk as if machine learning and AI were different camps. In her framing they are nested: deep learning sits inside machine learning, and machine learning sits inside artificial intelligence.
"Artificial intelligence is different from machine learning. Machine learning is a subset of techniques within artificial intelligence." (4:25)
This matters because machine learning is not the solution to every problem. Other AI techniques, she notes, can achieve very similar results without building large, powerful solutions that cope poorly with variation in the data.
The black box problem
The heart of the talk is what she calls the misuse of the black box. Off-the-shelf tools, whether from Python libraries or large cloud platforms, ship with default settings: predefined values for a technique's parameters. Many users never touch them. They take the results and go on to present or sell the solution.
Da Costa Abreu is blunt that defaults are not there to be used but to be changed. Without tuning, a model is not optimized for the data, can get stuck in a local optimum and can reinforce bias. Worse, it can look like a success.
"But worst of all, when you don't change and you don't refine the parameters, you can create solutions that look fine." (9:40)
She also rejects the belief, which she has heard at other talks, that defaults are the best settings a technique can have.
"So each technique is dedicated. It should be fine tuned to the data set to the problem that is trying to solve." (13:49)
When accuracy hides bias
To show the stakes, she describes classifying gender from faces in CCTV footage. A small selection of frames can produce very high accuracy. But if nobody knows how to tune the model or analyse its results, the misclassified images may belong to people of Asian background or Black people.
"And then you are creating, you are reinforcing the bias." (11:17)
A second example comes from one of her research students, who mines judicial sentences with natural language processing. Without optimization, the model simply reinforces the words most present in the sentences analysed, and nobody understands the relationship between technique, data and results.
She then points to the Brazilian judicial system, where, as she describes it, a decision support system suggesting sentence outcomes to judges was built without fine tuning. The appeal is speed and convenience for decision makers, while the impact falls on the citizens whose cases are decided. She calls this something the AI community must fight together.
She also addresses two opposite misreadings from people outside computing: fear that AI will replace them, and executive enthusiasm for profit from supposedly more precise operations. Working solutions, she says, do specific tasks really well.
"They are not going to become, you know, to replace human beings. That's not the goal." (15:11)
Five rules for using AI well
Da Costa Abreu closes with practical guidance. First, understand your data before choosing any model, and if it is not your area, find someone who knows it.
"The first point that I wanted to raise is understand your data." (17:18)
Second, do not use techniques you do not understand. You need not be a mathematician specialising in support vector machines, but you should grasp a technique's overall workings. Third, change the default settings and do the parameter tuning. Fourth, be creative rather than copying what everyone else uses. Fifth, treat black boxes with care: they are useful, but must be used with careful analysis.
"The next thing is be creative. Don't use what everybody else is using." (18:29)
Watch the talk
Demystifying Artificial Intelligence, by Marjory Da Costa Abreu (about 20 minutes): https://www.womentech.net/video/marjory-da-costa-abreu-demystifying-artificial-intelligence
About the speaker
Marjory Da Costa Abreu is a Senior Lecturer at Sheffield Hallam University with more than 15 years of experience in higher education, spanning teaching, research student supervision and science dissemination. She has led a Computer Science undergraduate course, is a Senior Fellow of the Higher Education Academy and is an activist for women in science.
Questions this talk answers
What is the difference between AI, machine learning and deep learning?
Why are default settings in off-the-shelf machine learning tools risky?
How can a model with high accuracy still be biased?
Will AI replace people in the workplace?
What should you do before using a machine learning model?
Drafted with AI from the talk transcript. Editor review pending.
Cite this talk
Marjory Da Costa Abreu. "Demystifying Artificial Intelligence." WomenTech Network, 2021. https://www.womentech.net/video/marjory-da-costa-abreu-demystifying-artificial-intelligence