Hybrid Decision making: Black box ML ft. explainable AI
Laisha Wadhwa18:172022
An image model told wolves from dogs by looking only at the background
An image classifier turns out to tell wolves from dogs by the background alone, and a digit model is 50% sure an eight is a three. Laisha Wadhwa, a software engineer on the Goldman Sachs data team, argues that when models decide loans or read cancer scans, accuracy without explanation is a risk. So how can anyone, expert or not, see what a black-box model actually learned?
See how LIME explains a single prediction in terms a non-expert can follow
Laisha Wadhwa, a software engineer on the Goldman Sachs data team, argues that a model's accuracy says nothing about whether it learned what you intended, and that explainable AI should sit alongside every ML pipeline. She explains how the LIME library builds simple local explanations for any model, and uses image, digit and text examples to show how these explanations expose bias, guide debugging and help non-experts trust a model.
“But what good is a churn prediction model if it cannot tell you why so many people are leaving?”
“So essentially, how my model is differentiating between wolves and dogs is only the background.”
What you will learn
Explain why a high-accuracy model can still be learning the wrong thing
Describe how LIME fits a simple local model to explain one prediction of any black-box model
Use explanations to decide whether to fix training data or tune hyperparameters
Present model decisions to business stakeholders without walking through the mathematics
Weigh LIME's limitations and recognise when SHAP suits tabular data better
Key takeaways from the speaker
Get a chance to learn a hybrid way of decision making, while building ethical models on the way!
At the end of the talk you’ll know how to explain your model to possible anyone! This makes it easier for you to sell your Business idea.
You’ll be equipped in creating a model agnostic easy to understand explainations for possible any ML model
There will be hands on for multiple datasets to build a profound understanding of the concepts discussed and equip you to start building explainable AI!
Details
Level
Beginner
Format
Talk
Technical depth
Introductory technical
For
Early career, Mid-career
Length
18:17
Published
July 2022
Questions this talk answers
Why isn't a high-performing model enough?
A model that works well is not necessarily working as expected. In critical cases such as declined loans or cancer scans, you need to know why it made a decision.
What is LIME and how does it work?
LIME stands for Local Interpretable Model-agnostic Explanations. It learns a simple model on top of your existing model by sampling around one instance and minimizing a locality-aware loss that measures unfaithfulness.
How do explanations help debug a model?
They show which pixels or words drove a prediction. When an eight was read as a three, the explanation suggested augmenting data rather than spending time on hyperparameter optimization.
What are LIME's current limitations?
You need to know the complexity of your model beforehand, and sampling from a Gaussian distribution introduces bias that you tune by choosing how many samples the local model uses.
When should you consider SHAP instead of LIME?
The speaker says SHAP is better suited to tabular data, where explanations are more involved.
About this talk
With wide no. of libraries & frameworks available for building ML models ML has become a black-box these days Thus model interpretability is vital. But its hard to define a model’s decision boundary in simple terms. With LIME(an open source library) its easy to produce faithful explanations and decrypt any ML model.
In today’s business centric world, there’s a renewed focus on model interpretability. With ML finding multiple use cases for elevating businesses, it has become vital to Interpret the model, build trust in the model (because there’s money at stake!) and understand how it works for any given data.
The most common all ML enthusiasts have is: why was this prediction made or which variables caused the prediction?
Model interpretability tricks like CV and grid search only answer the question from the perspective of the entire data set. It is hard to diagnose specific model predictions.
How many times have we all been stuck when our model performs well for some labels while fails for spurious data. We are left thinking what went wrong during training? That’s where the magical libraray LIME comes into the picture. LIME is a python library which tries to solve for model interpretability by producing locally faithful explanations thus explaining the decision boundaries of our model in human understandable form.
If you are a novice or an expert in ML, LIME is just the tool for everyone. It explains a prediction so that even the nonexperts could compare and improve on an untrustworthy model through feature engineering.
What will you learn?
- The talk will primarily highlight the necessity of understanding the outputs of ML models to build a profound understanding of how ML works.
- You'll learn a new way of hybrid decision-making.
- At the end of the talk you’ll know how to explain your model to possible anyone! This makes it easier for you to sell your business idea and build transparent and responsible models.
In order to create trust in our model, we need to explain the model not only to ML experts but also to domain experts which require a human understandable explanation. You’ll be equipped in creating a model agnostic locally faithful explanation set which helps even the non experts in understanding how the original model is making it’s decision.
All this sounds exciting, to add to this, a demonstration on the classic TITANIC data-set will be given to build a profound understanding of the concepts discussed.
The wolf detector that only saw the background (42 s)
Don't trust models without explanations (51 s)
Today we're going to talk about an interesting, I would say, aspect of AI that a lot of people are talking about and working on. So what is it? Before we delve into it, I'll give a quick introduction about myself. I currently work as a software engineer at Goldman Sachs and I love participating in hackathons and I've won couple of them in the past. Apart from that, I also like to write and I'm a technical writer with Omdena. I've been a podcast host as well. So let's dive in.
So what are, what all are we going to talk about? And, you know, we will start with an introduction. What, what is hybrid decision making? Then we'll dive into understanding this whole concept of interpretable AI that the entire AI community is talking about. We'll be looking at Lime, which is a great library, which is going to make our lives easier. And we can also be looking at really interesting examples of how you can make your pipelines, existing ML pipelines more explainable.
So let's begin. So honestly, I'm pretty sure we all have been here, right? We've trained models, we have built amazing models, but when it comes to testing them out, right, there are very ambiguous and spurious results. And we're just thinking like, what just went wrong? We've all been in his position. And what we're trying to do is trying to, you know, take smarter and better decisions. So if we look at today's world, right? AI with increasing compute power and more and more algorithms and an ever increasing amount of data, right? AI has come out of the laps and more and more businesses, companies are adopting and joining the bandwagon. But as we're moving towards more and more and more better models, we're also moving away from explainability and
interpretability. So if you look at this, right, this is where we started, this is where we are. We have amazingly well performing models, but we do not understand how they work. I mean, sometimes it so happens that your iPhone is able to recognize you without a mask, but sometimes with a mask, but sometimes it's not even able to recognize you without a mask. So that's just one application, right? But if you think of it from a perspective of, let's say the medical domain or the fintech domain, where your loan is getting declined or the scans say that you have cancer, you don't want false positives in those critical cases, right? That's why this whole concept of interpretability is very important. Another aspect that people miss out a lot is that if a model is working very well, it doesn't mean it's working as we expected
it to be. Now, this might sound very trivial, but as we look at certain examples as we go forward, you'll be shocked to see how AI can actually go haywired. But before we go forward, right, let's just understand explainability. From both the data and an algorithmic perspective, when, you know, all along, when we've built data pipelines, we've understood the data really well.
When it comes to algorithms, we just use whatever best is available, right? That's why ML has become actually a black box. And we're trying to move towards a world where it becomes a glass box. And then finally, we're able to understand it, how we understand the problems as a human mind. So when it comes to algorithm, right, we should be very thorough with, you know, how it works, how it's trained, how it's tested. And when we're sending in the data, right, what data points are contributing and influencing the outcome. So these are the questions we are trying to get answers for. So if you look at how ML pipelines are designed in today's world, this is how we do it. But with the entire training ML process and the test
functions in place, we still have these questions unanswered. We don't know why it worked, why did it fail in a certain case. And we always think, can I trust this AI, you know, if your loan is getting declined in the bank, you don't know why it got declined, you don't have answers to your questions. Right. And if you think from a developer perspective, right, many a times the models go haywire, and you're just left thinking, how do I correct this error? Where do I start? Explainable AI or XAI, as we call it, is a way that answers all of these questions for you and helps you take better decisions. So you have a path to start from. And essentially, you're not doing something groundbreaking, you're just
augmenting it into your existing pipeline. So you can consider explainable AI as an auxiliary support that helps you take smarter decisions, helps you navigate your way with complex models, right. And essentially, how it fits in the current pipeline is that when we are taking data from the real world, and you know, giving it any black box model between the model and when humans interpret that results, we're trying to fit in this, we're trying to, you know, use maths, again, to explain all that statistical interpretation that goes inside the model. And the idea is not to like, you know, again, have certain metrics ideas to give explanations that even, you know, a non-technical person or someone who's not a subject matter expert in whatever you're building can understand. So the explanations
have to be that simple. And people would say that, you know, what are you going to get out of making AI explainable? The idea is to make AI more responsible, right? We are advocating for better practices and methodologies in building and deploying these MAs. So a lot of things come under this umbrella. And most important of them being secure ethical AI. And explainable and interpretable AI actually drives the adoption of all of these things, because we are building this AI around, you know, it's a human computer interaction model. So it has to be centered around the people who are going to use it as well as, you know, in the process of making it more secure and ethical. It's also a very good approach at achieving data transparency. You know, we are living in a tech savvy world and everybody
is aware of how their data is going to be used. So you build an AI that's trustworthy, you build systems that are more resilient, they protect their privacy. But at the same time, when you build systems that are explainable, you ensure that AI has learned what you expected it to learn. Now, you all understood what explainability is, you know, you understood why is it important. But how do we create it? And I mean, you've already invested so much time into building a machine learning model, and you've tested it rigorously, and it still doesn't work. And now you don't want to do much of, you know, you don't want to delve deep into, again, doing that, repeating that cycle. Well, we have LINE for us. And what LINE stands for?
LINE stands for Local Interpretable Model Agnostic Explanations. Now, why? Why do we use LINE? It's because if I'm saying that it's a cat, and I'm 93% sure it's a cat, I want to know why it's a cat. Because it has fars, it has whiskers and claws. That's how my mind would work, right? Now, this might be a very trivial example. Think of it, that your company is building a churn prediction model, and it's saying that, okay, 30% of the employees are going to leave in the month of July.
But what good is a churn prediction model if it cannot tell you why so many people are leaving? That's what we're trying to do. We're going towards a future where we get output plus reasons for that output. And how it does this is very simple. Essentially, I mean, all you do is you learn a simple model on top of your existing model, by taking certain samples around your existing sample. I won't delve deep into the mathematics of this given the paucity of time. But the essence is, we're trying to minimize a loss function. And that loss function is trying to measure the unfaithfulness of your model.
And how it does that? It's very simple. I mean, we call it locality-aware loss. It takes into consideration the neighborhood of what you're trying to explain for, the simple model it has learned, and the complexity of the model. And there are various links that I've linked into the presentation. So you can go further and read about it. But, you know, before we understand, you know, and start implementing, there's a very fundamental question that will come in each one of yours mind, is that I was doing good already with all my EDA and my, you know, model performance evaluation metrics. Why should I use line? How is it advantages? It's because it's explaining decision boundary for you, which no, um, which none of the metrics can do. I mean, you could argue that simple models does it, does give you explanations like
decision trees and other models, but it does not, um, you know, this is not valid when you move towards more complex and bad models. Like, you know, when you combine XGBoost and your classifiers, you cannot expect that someone, um, who's not an expert can understand it. Also, the best thing about line is that the explanations are crisp, they're short, and it's actually helping you approximate your global model using very simple local explanations. Let's see it in action, right? But if you consider the classic problem of digit classification, this is how traditionally people evaluate their outputs.
They'll do a DSNE plot. But if I do not understand digit classification, I cannot understand what's happening here. I mean, I do see clusters are scattered, there's something definitely wrong, but what, right? When we, when we want the answers for what is wrong and where do I start, line comes into picture. Let's take an example that we're trying to classify this number eight. My human mind knows it's an eight.
My model says, okay, I'm 50% sure this is a three. And you're like, what is happening? Why is it even calling it a three? Now, what line helps you do is it helps you trace the boundary of the model, why it thought certain pixels were important in contributing it to a three. And you actually, if you look at it closely, it does resemble three. And this tells you that you need to probably augment more data. Instead of wasting your time in hyperparameter optimization, you might want to go back to your data, correct it, come back, test it. So it's helping you take decisions in a much more informed manner. Because there's any plot, you would naturally be a little lost. And it's not just text
that line, it's not just image that line helps you, you know, understand it's even text and even your tabular data. So any kind of ML application that you're building line actually helps you achieve like, you know, build explanations which anybody can understand. There's a couple of collab notebooks that are present here at this particular link, you can go and try them out, you know, see this in practice. And it is an application of all the three cases that I've discussed, which includes text, ML and deep learning. And I'll also skip this part, since it's more deep into how it actually works. But if you see four lines of code and you're done, you have an explainer module with you, which actually helps you understand. So essentially, if you look at it, right, if six is getting identified as zero, and it's also
getting identified as five and six, you see that model has not trained properly, you probably need to train more iterations of it. So this is a model which I just stopped at five iterations. So it has essentially learned nothing. So if you see the process is really simple, right, you have your model, Lime is training another model on top of your model, but it's a very simple model, because it's trying to explain for one explanation. And it's working in a local linear space, it's not working in the non-linear space. So it simplifies the scope of your problem. Now, let's, you know, look at this application, I'm trying to differentiate between wolves and dogs. And this is what happened with my model.
If you look at the gray area, that is the decision boundary the model has learned. So essentially, how my model is differentiating between wolves and dogs is only the background. The reason for this bias is because wolves are generally present in the wild, while dogs are present primarily in households as as well as wild. So one fine day, if you give an image of a dog in a wild area, it might just get identified as wolf. Now, this is a classic example of, you know, what you were trying to do would actually happen. And you do not want that when you're dealing with critical applications, you know, where let's say you're trying to summarize legal documents and that summarization goes haywire and the end users are
reading something that's not even relevant to your documents. Or you're working in a fintech domain, we are trying to predict the risk involved in a particular transaction and you let that bypass. That will be a false negative. And in domains like this, it becomes very critical. So essentially, LIME is helping you do validation. It's helping you decide your next path. It's also helping you understand what what went wrong. And it's also answering all of your hows and where's and what's and what's better than that. I mean, if we look at this particular model differentiating between dogs and cats, you can easily see why a dog is a dog. Even I as a human would see that left if part of the image is a dog and LIME helps you validate your decision boundaries. And essentially, if you're
trying to, you know, sell your idea, sell your algorithm to someone or you're trying to pitch it to someone, it's very easy to explain what your AI does rather than delving into the mathematics, the functionality of it, you can explain them from a working perspective. So when you're proposing this to someone who's from the business aspect of it, they need to understand it in a more abstracted manner.
And even for text, right, the explanations are very, very simple. Let's say I'm just trying to classify questions on Stack Overflow. If I'm saying that the question belongs to the SQL category, it's because these keywords were important while these were not. And in a similar way, you can also highlight in the same, you know, at the same time, what what part of the text was important? Where did it occur? How many times did it occur? So it helps you in easy dashboarding as well. And, you know, you can do pretty much anything. If you're doing sentiment analysis, this is a great way to demonstrate why your sentiment analysis works. You can, you know, talk about how it's able to identify the proper words with proper
importance and their occurrence. So you can talk about their correlation with other words in the text. And while LINE is a great library, you also have SHAP as a library available. It's more suited for, you know, tabular data because the explanations are much more involved for tabular data. And if you're someone who's interested to explore this further, I would recommend reading this book by Christopher Molnar. It's a great book for someone who's, you know, trying to understand more interpretable ML. And here we are coming towards the conclusion. You know, LINE as a library, but rather explainable AI as a concept is very, very important because it's helping you induce trust and transparency in your existing applications while helping you increase the public perception of what you are building. Because essentially AI being a black box,
people would necessarily doubt that should they really use it? How are, how is my data going to be used? Is it really safe? Are the decisions it's going to make? Do I know how it works? So, you know, you are using an auxiliary support to make informed decisions. And at the same time to people who are your end users, you're able to give explanations which are very short and selective. And while everything is really great, there are a few drawbacks currently since it's in development is that, you know, you need to know the complexity of your model beforehand, as in how deep is your neural network or how deep is your decision tree. And since the sampling is happening from a Gaussian distribution, it does introduce certain
bias in your models, which you have to tune by, you know, by tuning how many number of samples you want to take for your local model. And this brings us towards the end of the session. The most important thing that we've learned is that do not trust your models without explanations because you do not know what they're actually learning, right? And we all know interpreting machines intelligence is tough, even for us humans. So it's a, it's a process, it's a journey to understand, you know, to understand how these explanations can be made more and more verbose and simpler. And the most important thing is that for any business centric use case, right, where there are stakeholders, be it regulators or consumers, it's very, very essential that we ensure model is learning what we expected it to, irrespective of the
accuracy. So accuracy shouldn't be the only metric that we track at any given time. And while we've come a long way, what the future holds for us is that we are looking at a world where focus will be on model agnostic interpretability tools. Right now, how we do is we analyze the data, we do not analyze models from a qualitative perspective much. So we'll have partial dependence functions, like line helps us do to, you know, say model A is better than model B in certain cases, or for a certain test set, right. And with all of this happening, we are essentially going to automate ourselves, the auto ML has done that for us, but that has introduced another problem, which interpretability is going to solve. And since all of this is happening,
there's a great boom in machine intelligence research that's happening. And there are a couple of references for you to look at. This is the original paper by the authors of this library, and certain blogs. So that brings us to the end of the session. Thank you for being patient audience. And I'll be open for questions, you can put your questions in the chat. You can also reach out to me either on Twitter or on LinkedIn. If you have any further questions, if you're working on a project, and you would like to know more, so I'd be happy to answer your questions. All right, then that's a wrap. Thank you so much for having me today. Bye everyone.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why accuracy alone can't tell you what your ML model learned
Laisha Wadhwa shows how LIME turns black-box predictions into explanations that developers, stakeholders and end users can understand.
In short
A model that scores well can still be learning the wrong thing, and metrics alone will not show it. Laisha Wadhwa, a software engineer on the Goldman Sachs data team, makes the case for explainable AI as a standard layer in every ML pipeline. She walks through LIME, an open source library that explains individual predictions of any model in terms a non-expert can follow.
Key takeaways
Learn a hybrid way of decision making while building ethical models along the way.
By the end of the talk, you will know how to explain your model to almost anyone, which makes it easier to sell your business idea.
You will be equipped to create model-agnostic, easy-to-understand explanations for almost any ML model.
Hands-on examples with multiple datasets build a deeper understanding of the concepts and equip you to start building explainable AI.
From black box to glass box
Wadhwa opens with a situation most practitioners know: a model trains well, then produces ambiguous or spurious results in testing, and nobody knows why. She argues that as compute, algorithms and data have grown, models have improved while moving further away from explainability. Data teams usually understand their data well, she notes, but tend to take whichever algorithm performs best without understanding how it reaches its outputs.
"That's why ML has become actually a black box. And we're trying to move towards a world where it becomes a glass box." (3:07)
The stakes rise with the domain. A phone that fails to recognise a face is an annoyance. A declined loan or a cancer scan is different, and in those cases, she says, false positives are not acceptable. Users want to know why the decision was made, and developers want to know where to start when a model goes wrong.
Explainable AI as an auxiliary layer
Wadhwa is clear that explainable AI (XAI) does not replace an existing pipeline. It sits between the black-box model and the people interpreting its results, translating the statistics inside the model into explanations simple enough for someone who is not a subject matter expert.
"So you can consider explainable AI as an auxiliary support that helps you take smarter decisions, helps you navigate your way with complex models, right." (4:34)
She links explainability to the wider goals of responsible AI: secure and ethical systems, data transparency and trust. Most importantly, explanations let teams confirm that a model has learned what they expected it to learn.
How LIME explains a single prediction
LIME stands for Local Interpretable Model-agnostic Explanations. Wadhwa uses everyday reasoning to frame it: if a model is 93% sure an image shows a cat, a person wants to know why, for example the fur, whiskers and claws. The same applies to business models such as churn prediction.
"We're going towards a future where we get output plus reasons for that output." (7:18)
The approach, as she describes it, is to learn a simple model on top of the existing one by sampling around the instance being explained. The method minimises a locality-aware loss that measures unfaithfulness, taking into account the neighbourhood of the instance, the simple model and its complexity. Because it works in a local linear space for one prediction at a time, it simplifies the problem. She stresses that this explains decision boundaries, something evaluation metrics cannot do, and that simple interpretable models such as decision trees stop being enough once teams combine complex models like XGBoost with other classifiers.
What the explanations revealed
The examples are where the talk comes alive. With digit classification, a t-SNE plot shows scattered clusters but not what is wrong. LIME, by contrast, showed why a model was 50% sure an eight was a three: certain pixels did resemble a three. That pointed to a data problem rather than a tuning problem.
"Instead of wasting your time in hyperparameter optimization, you might want to go back to your data, correct it, come back, test it." (9:42)
In another case, a six identified as zero, five and six revealed a model stopped after only five iterations that had learned nothing. The most striking example involved telling wolves from dogs.
"So essentially, how my model is differentiating between wolves and dogs is only the background." (11:52)
Because wolves usually appear in the wild, a dog photographed outdoors could be labelled a wolf. Wadhwa connects this to legal document summarisation and fintech risk prediction, where a missed risky transaction becomes a costly false negative. LIME also works for text: classifying Stack Overflow questions, it highlights which keywords drove a SQL label, which helps with dashboarding and with demonstrating sentiment analysis. Explanations like these, she adds, make it far easier to pitch an algorithm to business stakeholders without delving into the mathematics. For tabular data she points to SHAP, and recommends Christoph Molnar's book for anyone exploring interpretable ML.
Limits and what comes next
Wadhwa is candid about the drawbacks. LIME requires knowing a model's complexity in advance, such as the depth of a neural network or decision tree, and its sampling from a Gaussian distribution introduces bias that has to be tuned through the number of samples. Her closing message is direct.
"The most important thing that we've learned is that do not trust your models without explanations because you do not know what they're actually learning, right?" (15:57)
For any business use case with regulators or consumers as stakeholders, she argues, teams must make sure the model learns what they expect, whatever its accuracy. Looking ahead, she expects more model-agnostic interpretability tools, qualitative comparison of models, and interpretability as the answer to problems introduced by AutoML.
Watch the talk
"Hybrid Decision making: Black box ML ft. explainable AI" by Laisha Wadhwa, 18 minutes: https://www.womentech.net/video/hybrid-decision-making-black-box-ml-ft-explainable-ai-laisha-wadhwa
About the speaker
Laisha Wadhwa is a Software Engineer (Data Team) at Goldman Sachs in Bengaluru, India, where she builds data systems for Marcus by Goldman Sachs and has worked on computer vision and music genre analysis. A serial hackathon winner often called "The Hackathon Girl", she is also a tech speaker, blogger, technical content creator at Omdena and Global Ambassador at Women Tech Network.
Questions this talk answers
Why isn't a high-performing model enough?
What is LIME and how does it work?
How do explanations help debug a model?
What are LIME's current limitations?
When should you consider SHAP instead of LIME?
Drafted with AI from the talk transcript. Editor review pending.
Laisha Wadhwa. "Hybrid Decision making: Black box ML ft. explainable AI." WomenTech Network, 2022. https://www.womentech.net/video/hybrid-decision-making-black-box-ml-ft-explainable-ai-laisha-wadhwa