Okay. So so I'm Brandon Kramer. So I have twenty years of experience in data operations, analytics, and governance.I have my MBA as well as a postgraduate from Purdue University in advanced AI and generative AI. And last year, I had the pleasure of attending the ISACA advanced audit training week in Vegas. And so I wanted to provide just a little bit of background as I talk about AI auditing so you guys can know where I'm coming from. So the the real question with AI is, you know, it's no longer about how organizations are going to use AI, but whether or not those organizations are governing AI responsibly. And so that's what we're gonna take a peek at today and try and understand. AI creates new risks, right?

And so it's it's new risks and new risk categories, you have to deal with bias and ethical concerns that you may have not had to deal with before. There's a lot of regulatory uncertainty with new regulations being passed depending on which country you're in. Other ones coming on board, draft regulations out there. There's always a concern with sensitive data exposure. Are you exposing your data to other people? Are you exposing data you shouldn't be? Is your AI model making wrong assumptions? Is it giving wrong answers? The lack of transparency with models. Where did it get the assumptions? Where did it get the answer? And then model drift. Right? Like, it's just is that model slowly leaning one way or leaning the other way? So these are just a couple of the AI category risks that you have to think about when you're thinking about auditing. So what are auditors looking for specifically?

I bucket into three different things. They're looking for governance, controls, and compliance. For governance, they're looking at who owns it and what's the oversight. Who is owning those models? Who's owning those risks that we talked about? And what's the oversight being placed on top of that? In addition, it's controls. Right? What are the policies? What are the monitoring? How are you maintaining those AI models? How are you maintaining the drift? And then compliance. Compliance is one I think people don't dive into enough because it's not just the regulations that exist today, but it's the regulations that may exist within your business unit, may exist within third parties. Are they following the regulations? Are they following compliance? The compliance is an area where I feel needs more focus and needs more dedicated. And then essentially, when it comes to AI auditing, it all comes down to, you know, can your organization prove it?

Can you prove that you have governance, that you have controls in place, and that you are following compliance guidelines? So I came up with three recommendations since this is kind of a really short session on on AI auditing. So just real quick three, recommendations I have. The first one is to make sure you have established formal AI governance. So like I mentioned, have clear ownership over your processes, your data, and your models. Make sure you have established policies and then you can audit those policies. What are the policies for accessing AI? What are your policies for data risk? What are your policies for data access? Right? Make sure you have those written policies. And then I always recommend an approval pass approval process with an AI governance board. There should always be a board that oversees everything you do when it comes to AI, and they're helping make those decisions whether they're risk based or privacy based.

They're helping make those decisions. Documenting those decisions gives you a better foundation for your AI governance. The second recommendation is to implement risk based controls. If you take a look at your use cases and you classify all of your AI models and all of your AI software by risk, it helps you implement those risk based controls. Whether you're doing privacy controls, in-depth monitoring, you have different security rules, access rules, those all being sort of categorized by risk makes it easier. If it's a high risk, it has more controls, low risk, okay, maybe it's more open, it's more apt to, user experience and that type of stuff. And then keep human insight. Right? If there's not a human that's constantly validating the data and the privacy and the controls, you're you're you're losing out. The third recommendation I have is build defensible audit evidence. And so what I mentioned in my second slide there, it's like, can you prove it?

Just because you did build a robust system, just because you did test it, you did make sure the security and privacy controls are in place. Can you prove it? Right? So maintaining an inventory of your tools, processes, audits, SOPs, training datasets, Maintain that inventory even if it's in, like, a cold storage. And then any testing records, UAT, test scripts, right, your bugs, your defects, all those things are things that you wanna make sure that you have in evidence collection. And then vendor reviews, you know, critical things like that, monitoring logs. Right? It's essentially you're building your defensible evidence, your folder of evidence for your AI audit. This slide gets really busy and so do my next couple of slides. I didn't have a lot of time to present on this, so I wanted to give you guys some real examples of what I use, but I I'm not able to go into super detail.

But essentially, this is an Excel audit toolkit that I have, and it has a comprehensive library of eight tabs that are audited. So you have rationale, responsibility, data fairness, safety and performance, impact, your library, and taxonomy. So in each of these tabs, there are literal audit scores that happen on the different areas. So you can see here I included responsibility. And so it literally says, hey. What's the description? What's the evidence of the deliverable? Why are we assessing this? Same when it comes to fairness. You know? The example I provided is, like, the electoral influence. So some politics. Right? Is my AI model taking politics in account or not in account? And so things like that are built into each one of those individual tabs that allow you to actually do that auditing.

And I can share this toolkit if you if anybody reaches out to me on, LinkedIn, I can share my Excel toolkit example that I have here. Another one I mentioned to you guys that I did go to the Vegas, AI audit training. It was a week long training put on by ISACA. They provide an extensive toolkit. They have 22 elements of audit. So where I only have eight elements of audit that sort of hit what I think are the broad buckets, they provide something that has 22 elements. And so it's not as detailed when you go into their toolkit specifically, but if you just take a look at the control family and you look at the different highlights, it can make sense to you. Right?

Like, you look at AI operations, asset management, governance, life cycle, risk, it makes sense why they have it separated by control family, it's just an added layer of detail even beyond what I showed in the first example. And then the third example here I just will show real quick is if you're starting from the beginning, if you're starting from scratch, just having an AI audit model card is very helpful and these you can just get off of Google, literally Google images. And so it's just a model card for each one of your AI models or each one of your AI softwares that just goes into enough detail that at least you have a starting point. So you know where did the training data came from? You know what? Were ethical considers take ethical considerations taken into account? What metrics, factors? What's the intended use of that AI model? Having that card, I think of it like a business card or a resume for your AI model, I think sets you up for success from the beginning, even if that's at a minimum all that you have and that you're deploying.

So I know I rushed through that a little bit because we got a a late start, but, you know, if you guys have questions on some of the AI audit tools or toolkits, I can provide those to you if you wanna reach out to me on LinkedIn or if you have any questions on AI auditing or any feedback.

Thank you. Thanks everyone for your time.