So since Chegg GPT launched in 2022, three and a half years later, the conversation around AI has moved beyond demos to productions.And as we know, the reality of AI in production is probably way more complex, messier than the headlines, and also more way more interesting. Today, we're gonna have three CTOs who have actually shipped it, scaled it, and have been keeping it running. Please welcome Caroline Dubey, few CTO and cybersecurity AI strategist at Caldera. And we also have Kelly Buchanan, divisional chief technology, data, and operations officer for wholesale banking and truist. And we are expecting, one more CTO, Bratna Singh, former CTO and customer identity at Okta, and she probably gonna join us shortly. But I think we can get started. When we talk about AI in production, we all know that, oh, Bartna is here. Hi, Bartna. So Hi.

Bartna is a former CTO of customer identity and Okta, now the CTO at Lightspeed, I believe.

That's right.

Yeah. Thank you. Well, let's get started. Since you all have put, AI in production, I'm gonna ask each of you to share, AI use case in production at your company or your client's company that works exceptionally well. And how do you know actual work? Kelly, I'm gonna put you on the spot. Let you start first.

Sure. Look. The thing that I think I'm most excited about that we're doing here within the wholesale bank is what we it's a multiyear program that we call credit modernization, and it's really fundamentally changing how we do wholesale credit from origination all the way to underwriting, decision, and fulfillment.

Today, that is a highly manual process, that takes a lot of different systems. And we're on this journey to really transform all of that. And what I really think is cool about it is it goes back to we're not just putting kind of AI or or pieces of automation in little pieces of it, but really starting with fundamentally changing how we do the work. It starts with everything from how we revamp our credit policies and our workforce to actually work different. And then we're bringing in technology like, you know, GenAI that's generating credit memos, with policy overlays and operational capabilities. We we're using, you know, data and signaling and models for, data driven dashboards that can predict defaults, that can look at, you know, high risk cases and really give us the information that we need. You know, the benefits here are automating the work, reducing kind of, you know, the the cost it takes to to fulfill and service each loan, reducing our risk when we have so I mean, this is a highly we're in a highly regulated business.

So being able to reduce that risk, through through AI and and different technology. And then really, you know, it's it's decreasing our our credit exposure and our credit loss. So it has through ROI capabilities.

Right. So it's really you're taking a holistic approach talking about end to end. Correct. Got it. That's amazing. Bratna, how's it what's your, success story in in in Okta for AI in production?

Well, there's multiple products that we have rolled out. So we have used AI both in the in the production as well as, you know, for internal productivity as as many companies are doing it. We certainly see a lot of, I would say, value add of how fast we are able to roll out things, how fast we are able to generate code, and, of course, you know, the the usage of it in driving our productivity overall. And then on the on the, rolling our products, we have rolled out intelligence products to help our customers, you know, drive better, insights from their data to make decisions on, you know, what's the attack surface, how are they getting attacked, how to kinda, solve for, you know, credential stuffing, and many other aspects of identity threats, let's say.

And then, we have also rolled out platform that helps, customers who are building agentic solutions or agentic applications. They can build that on our product very safely. And that has get gotten a lot of traction primarily because, as you know, that's a conversation we'll get into as we continue, which is more of the development is happening, but, getting it into production is where the hesitation comes in. So the platform allows that secure development to make sure that that hesitation can doesn't have to be part of the process. So lot of, I would say, value add, but but I would also call out that there are areas where there are concerns, and we are still working on, you know, resolving those, wrinkles. I see. And certainly, many learnings are on,

on the way. We're gonna talk a lot more about that, and it sounds like both of you have pointed out the the customers and success. It's the way that signal your success. Caroline, can you share us some of those? And I assume that you work with clients every day. So what is your, experience in the use

cases with my work? With clients across multiple industry verticals, but a lot of a lot of what we're doing with AI is helping with customer experience. So, you know, internally, we've been doing a lot with processing. It doesn't sound very sexy, but, like, document processing. Right? Like, taking one of the things that I found that AI is really good at is taking notes. Like, it's it's very good at taking notes and helping, and that helps our, you know, that that's that's a that's a product that we use that's kind of embedded within our within our, our video conferencing system. But it really helps our our customer, facing folks to to focus on the conversation that they're having at hand rather than taking notes. So and then a lot of times, it's like, you know, did you get that? Do you you know, did you did you hear that?

Did you get that document right? You know? So it's I think that's really helping us to focus more. And then, also, we have a lot of a lot of documents that we had to process within sales to help understand our customers better so that when we go into a meeting, we're not asking them, you know, what is your business? What are what are your pain points? You know, those sorts of things. We already go in with that in you know, knowing that by having it from public public basing sources. Like, so if you're going with a company, you know, looking at all their 10 k's, looking at all their you know, that's a lot to read. Right? Like, it's a lot to read and process for all the different we have, you know, thousands of customers.

So I think, you know, it's it's really we're not looking at AI ourselves as, you know, cutting people or getting rid of people, but what we're doing instead is to be able to do more, to have better relationships with our customers, to kind of what they call it, like, punch above your weight.

Right? Like, we could do we could do the work of a bigger company because we're leveraging these things in AI. And we have things that we didn't do before that now that now we can do. So I think that's, you know, I think that's the positive the positive side. I see I see a lot of our customers kind of focusing on that customer experience

and

trying to help, like, internally their their folks to manage all of the information that they have to manage to help them serve their customers better.

So I think we're gonna dive deep into each of those topics. So very soon you talk about the success story, and and some of those might not sound sassy, but it's actually provide a lot of values. And so what did you have to build or have your team to build or fix whenever it is that have nothing to do with AI models that to make AI work in production? Bratna, when you want to start?

Yeah. I think that's a great question. And, you know, there was early stage where we were trying to understand the models and make it better or value add for us. I think now we're at a stage of ROI, figuring out the ROI, and that's where you need to make sure that for us to build AI and agentic and even autonomous solutions, it's important that we have the tools integrated because AI is only as smart as the knowledge it has from the data that these tools carry.

Think Jira. Think Confluence. Think, documents that you have, as Carolyn called out. So that's number one. Number two is making sure that all these controls are visible, especially your data controls. Do you have critical data identified properly? Do you have the right kind of data being used by these AI tools with the right access? We we talked about multiple use cases here. You wanna make sure that the right customer's data is being used by the right account holder or account representative and not everybody. Right? These are these are there's PII data. These are sensitive datasets. So those two are very important aspect. And and the third aspect I would say I would call out that my team spent on is observability and traceability.

It's important that we can observe what the agent is doing, when it's doing, and be able to come back and trace it if we need to, understand and unlock a certain situation. And lastly, governance. And the important part of governance, of course, people call out is end to end life cycle. But the most important is when the agent is running and not doing something that you want them to do, can you kill it right away? That's an important aspect of the setup. So building all of this, what I would call it harness end to end to make sure that your agentic solution is working well and as expected, because, you know, you you know that agentic solutions can be unpredictable. That is the big part where team ends up spending time, and I I believe lot of development or or innovation is happening in this space at this time.

Mhmm. Kelly, do you have anything to add? And I assume that in a rec regulated industry, you know, you need to do many of you.

Yeah. No. I I echo everything she said, but in a I would say, you know, a a different lens on it is we have to have all of that and the traceability, but we're in a highly regulated organization. So I have to be able to get my risk partners really understanding. We had to look at kind of what are our policies around some of these activities. How do we get our risk folks comfortable with what we're doing? Where do you need a human in the loop to make sure of that? How can we make sure that we stand up to FINRA audits, regulate you know, conversations with regulators around what we're doing and why we can control it? That was a really I mean, we probably spent a lot more time I always tell people, you know, like, building the tech is actually the pretty the easy part.

It's all the other stuff around it. We're also when you're talking about a bank, I mean, we're we're in, you know, everything from you know, we we're doing, you know, Carol, what you said around, you know, looking at pitch book creation and how we're doing investment banking. Like, some of that stuff is really cool and sexy. When we're talking about back end operations for payment operations or loan processing. Right? This is knowledge that people have have in their heads for long periods of time. So I've got tenure up to twenty years on the teams. Right? And and getting people to think differently about how not just kinda go, like, I could automate this job, but really pull up and say, how do we think differently about this and how would we deploy this technology? I think that's the other big piece of it. Right?

Of of really kinda sometimes we say, you know, is this something that we just wanna a small thing that, like, I now have this tool that I can now create pitch books quicker and I need less, you know, analysts or or I can get more more work out of them? Or is this an a blank sheet exercise? Mhmm. Where we wanna really just kinda go, let's rethink everything that we're doing, and and how do we how do we look at that? Those are a lot of things we spend time on. A lot.

Yeah. So we we kinda naturally get into the topic of a trade off. Right? When you're putting AI into production, how are you managing the trade off between, let's say, trust, speed, and cost? And which one tends to give? Caroline, since you work with quite a bit, a lot of clients, how would you advise them? How do you design? So

I think at this point, there is a lot of what I would call FOMO. Everyone every it there's it it's almost it's almost crazy. Right? You know? It feels like, oh, I only have four agents in production, and somebody else has 200 or whatever. You know? There's a lot of FOMO in the industry. There's a lot of, you know, oh, I have to show cool things with AI. Everybody's doing AI. Right? But it's to me, it's how do you do it in a way that is consistent with your brand that gives your customers and your employees the best experience. It's not about technology. You know, Kelly Kelly hit the nail right on the head. This is not about technology. This is about looking at what is your company doing. How can I make it better for the customer and for the employees? And it's not about the technology.

And, you know, everybody's focused on, oh, how many GPUs do you have, or how many tokens have you used this month. Right? It's it's not it's not about that. It's about how are you using those tokens? How are you using those GPUs? What is your road map? How are you going to integrate this into the whole process? And how are you gonna do it safely? So you don't wanna build up. You don't wanna be be cool and and doing all these cool new things and then end up, you know, like, in in financial trouble. Right? Like, something like this could could put you out of business if you're in a regulated industry or if you're, you know, you're doing things that are that could be harmful harmful or have provide you know, having a loss. Mhmm. So I think that you need to be very thoughtful about it, but you need to have a balanced approach. Right?

If you go too slow, people are gonna just start using AI within your organization, and they're not gonna you know, they're gonna be like, oh, I need to do this. I'm gonna get on OpenAI. I'm gonna use this. You know, they call it shadow shadow AI. Right? Mhmm.

If you

go too slow, you got shadow AI. If you go too fast, now you have a lot of risk. So you have to find that middle ground. It it's it's a little challenging, but I think once you find that ground, once you figure out, you know, here's the safe progression and the the logical progression of the things that I can deliver in a way that's consistent with my brand and also safe. I think that that's, you know, that's the that's the balance that you have to strike. Mhmm. You have to go get the right pace, like the Goldilocks, you know, the Goldilocks AI pace.

Yeah.

I I wanna double down on what Carolyn called. This is very important to emphasize that, you know, the the point of view of, like, if you're using more AI, is that good? I think we need to every organization should evaluate and define what is the value add that you're looking for, and then AI or any solution of AI should be in service of. And and that is that's where, you know, as Kelly called out, what's the business goal? What's the business outcome that you're looking for? And, hence, we are building something to that goal. It's super important, and that gets missed out in so many places, in so many conversations I hear. You know, there's there's also overusing of tokens just because, you know, that's how the the the value of a per for a particular person is evaluated.

These these are all bad habits that we need to, you know, start to share and start to build the value, that's when we will start to, see ROI. The key point I wanna call out here is as you are leveraging it, I'll give you my team's example. We started using AI, and we were using a lot of tokens. But as the team built the understanding, we started to see of course, the token usage continued, but the cost started to drop. And that is the the key part, you know, that the the Goldilocks or you can say the best understanding and depth of of AI usage is also something that has all kind organizations we need to look for. It's a very important point that I wanna double down and emphasize that it's important to think about value.

So I'm hearing all three of you sorry. I apologize. Go ahead. Go ahead.

Oh, no. I was just gonna say, look. When I think about it, when you ask kinda, like, trust versus speed versus cost, right, it kinda depends on sometimes what you're trying to do. But, like, at working at a financial institution, like, trust is a nonnegotiable. Trust and safety, like, if people don't can't trust us with our money, it doesn't matter. It doesn't matter what products you can put out. Right? So that that's my my business. But I think that there's also the ability that that's when we're talking about production level stuff. We should be able to, though, create the space for our teammates, business and tech across the board, right, with the right product, with the right tools and guardrails to experiment, to fail, to break things along the way.

Right? We have this bad habit, I think, when you're in kind of broader legacy type organizations that everything is a project and you have to that's how we do funding and you have to work through it all, but we have to break away from that. And, you know, look, you know, I always like to remind my teams and stuff, you know, where they say 90% of AI, ideas fail. Like, where are we building that in at? Right? And I think that especially as the tools and things change, that's really important because that's where you can break more eggs to then get to what should the thing that's going into production be, where the thing in production has many more, nonnegotiables than than what we would POC.

Yeah. So I'm here all three of you are landing on a similar place around trust as a nonnegotiable, but also recognize speak is also important too. You know? You know, also really looking at what kind of problem and what kind of solution is is, you know, maybe some internal things that you could actually speed up. By also putting in galleries, we can also see the cost surprisingly. And when those tokens are not cheap, but the cost could potential drops as you actually focus on the outcomes, that the impact of the project. That sounds fantastic. I'm so glad to hear it. This this seems like a consistent scene. So if we talk about continuing on on this AI journey, though, in in your experience, what actually become a bottleneck when AI hit the real the production system?

And how much how much is that is really technical versus organizational? I know some of you already mentioned the the technical. It's not the most critical part, but I really want to hear what is the bottleneck in in the journey to putting AI in production? Callie, you want to start?

For yeah. For me, it's it's organizational. Right? Like, TechWorks. It's it's not, I mean, it's not that difficult, on some of these if you have the right tool sets to to develop all kinds of AI products and AI capabilities and automation and so on. But it it really is from an organization standpoint. You know, we're still learning. Not only, you know, the the tech kind of the tech teams, I'd adopt it pretty quick. They love kind of the new technology. When you're talking about our the business line folks of them understanding, okay, what does this need? How does this work? How do they talk to clients about it and sell the different things that we have? Right?

If we're gonna revamp how financial advisors are able to pull information, I do think that's a whole new training and understanding around it. And then there's the governance. Right? Where do we put AI? What does it mean? Who's developing it? Do we have the right governance around it? Do we understand the fully cost of it? It's just a real mind shift, I think, when we're when for an organization like a bank to take on. I think that's that's everybody wants to move really quick, and everyone likes the idea of it. But as we're going through, there's it's getting everyone comfortable with what we're doing can sometimes be the harder part.

I I would add, to, you know, certainly organizational. I think we we call out. Right? The culture is a big part to drive. The other thing I would call out here is, you know, the quality does become a bottleneck. If you're not getting the right quality or outcome from your AI solution, then what do you do? You bring human in the loop. And now with that speed aspect of what you are looking for, now you have a human who who is at the door or control, evaluating it. I'll give you an example for internal productivity. We have, been leveraging AI to drive code development. Certainly, you know, engineers are evaluating it, but then PR reviews become a bottleneck because we wanna make sure that the human does the review before it goes into production.

So it's it's that aspect which is, which is where we will see and, start to see the shift, but also start to see innovation as well to to unlock that, I would say, blocker or human in the loop, concept. But to get the human human out of the loop, you need to make sure the quality is at par and at expectation. And, Kelly, what you pointed out is it's true for every enterprise organization, which is trust and safety is utmost priority. So when we have the right governance, the checks, the controls, that's when the the comfort level will come to, you know, not have

you so

you Yeah. Did I just

Yeah. Make me we just lost you the last five seconds. Yeah.

I I I was just saying, like, once we have the right controls, the right governance, the right visibility, and, of course, quality in the AI solutions, then we will start to see less human in the loop and then, of course, the less bottleneck.

You see the human in the loop is actually a way to ensure quality and also build trust and safety, but it's more of the interim solution. Ultimately, you want to continue drive the technology to make it more automated.

Which is where we'll need humans too. Right. Right. Exactly.

Yes. Exactly. Anything to add, Caroline?

Sorry. What what was the question about

So, the the question is about, you know, how much of these AI, challenges is really, technical versus organizational. So everything has to work together.

You can automate something, but if your process is not is not if you're automating something that's that's not good to start with, then automating it isn't gonna solve your problem. Right? So I I read a I read a an article in reworked the other day that was kind of funny. It was kind of a funny article, and they were saying your your processes are Rube Gold Goldberg machines. You know? Rube Goldberg machines like the like the game mousetrap. You know? Like, the ball comes down, and then it hits something else, and then something else hits something else. Right? If your processes are convoluted and they don't make any sense, then automation is not gonna fix that for you. And if you don't have the trust within your organization, with your employees and with your customers, that you're gonna do the right thing with their data, that you're going to be sensitive and that you're going you know, you're not gonna just have them training training the AI that's then going to send them out the door.

Right? There's a lot of there's a lot that goes into building the psychological safety within the organization of your employees who are right on the front lines working with your customers and making your brand. Right? So if you're not if you're not building that trust up within your organization, then customers customers and employees are not gonna use your AI, you know, your AI systems. And to to Vonna's point, if they don't work and they don't and you gotta do the work anyways, you gotta do it over, then, then they're not gonna use it. So it has to be well thought out.

It has to have high quality, and you have to build that psychological safety to enable your organization to be productive and your customers to be productive with this technology. So there's a lot more than just like, oh, I've got the coolest, you know, large language model. I'm using, like, g p GPT, like, 8,000,000,000 or whatever. You know? It's, it's it's really about how do you think about the whole the whole situation and the whole organization and the culture. There's a lot that goes into it.

Right. So y'all talk about the organizational design matters, more than just tools. Building of the trust is important. Where do you see the organizational design actually make or breaks it in in in this AI journey? Kelly, you want to

I don't think that there's one place that makes or break it. I I I think that, you know, one of the things that we're working on right now is is education and training and helping people, you know, kind of, like like I say, always get on the bus. Right? It's moving at a at, you know, as we know, AI and technology, the clip of movement and change is, like, it's crazy. It used to be crazy. It's, like, super crazy now.

Mhmm.

And those are for technologists. Like, think about the all the people that we talked about that need to make this happen when we're talking about governance, risks, even our own regulators that we talk to and others of helping them understand all of those pieces. So I, you know, from working in the wholesale bank, I sit at the technology leadership table and I sit at the business leadership table. So my job is to really sit with the business leaders and help understand where are they going, where is their revenue going, where do they need where where is our strategy, and how does technology really enable that? But in order for me to do that, I have to enable them with what does this mean to them, how should they be using things. You know, one of my I love getting multiple emails a day from different people around, like, Salesforce has an AI or this this vendor, this vendor.

Can I use this vendor? This person said they have this. Have you seen this tool? And I'm like, oh my gosh. AI tool overload. Right? So it it's really helping us get back to how should they be thinking about this? How are they transforming their individual work and how they're actually working and and and then, you know, how do they move forward? But I I think to me, it's not one place of it, but this is constant education and transparency about what what it is and where we're going so that we all move together. Because I I run payments here. It all has to work together for it to be successful, and that's how I look at it here.

Right. So when you're driving organizational changes in the technical environment, you're CTOs, so I I can only imagine that all three of you have rooms and where you have have to earn trust and credibility fast within your organization or with your clients. So where does that tension land hardest in AI adaption? You share some of your experience? And I'm gonna Let

me start. Yeah. Yeah. I will call out. Look. When we started the AI journey, there there was concerns about, you know, how it will get used and who will use it, and do they do they have the right control. And this is true for many organizations, which is, and because of that fear, the feeling was that, okay. Let's let's tell everybody to not use it or slowly use it or gradually use it. And that's a big no no because, you know, when innovation comes and hits your door, everybody wants to jump on it. Everybody wants to try it. And that's where, you know, Karen called out, shadow IT starts to start to show up. And avoiding it is very important for an organization, especially in an enterprise, company.

So that's number one. Number two is building that relationship with your IT and security and legal team to help help partner very closely. Then, Kelly, you called out it called it out as a trust team, but, you know, very working very closely to make sure that the legals are in place, the IP is in is is protected. We have the right visibility in the right places to make sure that we our implementation is per the terms and conditions and expectations of our customer. And if and and if there's certain experiments that we wanna do that we we bring our terms and condition up to mark. So there's a lot of things that needs to happen to unlock the usage of AI. And while the train is running very fast and everybody wants to jump on it, and if you don't let them jump on it, they call out that, oh, maybe my organization is not innovative.

So there's a lot of pressure from from the leadership side to make sure that we do it right. At the same time, we do it very, very fast. So that's the big part of organizational change, you can say, or expectation. On the on the side of, you know, bottoms up, I would say, as you give these technology in the hands of people to leverage, are they using it in the right ways? And as they're building applications, where do these applications live? Now we are not talking about five applications that IT helped integrate and build. We are talking about one person creating 80 to 100 applications. Now you're you're you're an enterprise company. You got thousands and thousands of employees. Where do they live? Are all of these important? Which ones are important? Which ones should we should we squash?

These are all the conversation, and you can say process and setup that we have not had before. And we are now having those conversations to say, where should they live? Who should own it? Who should support it? And now imagine overhead of each applications of security and governance and scanning and all of that stuff. So I I would say as organizations in the tech industry, we are all together evolving very fast at a very high pace to make sure that we ensure the trust, we ensure the expectations, and meet our our customers' expectation. At the same time, we drive innovation at the fastest pace. It's a fun time to be in the space, to be honest, but also, you know, it's a high demand time as well.

Alright. I totally see that. Since there's so much to do, and thank you for all the sharing. So many of the audience here right now, they are different, stage of this AI in production journey. Right? What is one thing that you can advise them right now? The one thing that they should stop doing right now if they are serious about putting AI into production? Knowing there's so many to dos, you know, technical and organizational. What's the one thing that you would suggest stop doing?

I would say a little bit what Carolyn said before. It's the FOMO and trying to keep up with the Joneses. Right? Every organization is different. I I get from my executives too. Like, did you see what x, y, and z is doing? And I'm like, you know, they're not a regulated bank. Right? So it's like, you know, it's it's it's not about, you know, who like, yes. You should absolutely explore how it's being used because and and get into it and and ask questions because there's so many things I'm like, wow. I never would have thought about that. I wouldn't have thought thought about using it that way and so on. So it's not it's not of keeping curious in that, but it has to work for you, your organization, and your ROI, and at the pace that is deemed within whatever transformation you're trying to do.

So just be careful with getting sucked up into that, you know, how how many agents do I have? How many how many, as I said, tokens am I using? Well, so and so is doing this. Like, get specific, you know, just like anything else. Have a strategy around what you're trying to accomplish, and AI is just another tool in that to make sure that you're accomplishing your strategy.

So this is not a competition. Right? How about the current one? Yeah. Where would you, suggest your clients to stop doing something?

I I would say stop being unrealistic. So a a lot of people are like, oh, you're not using enough tokens. Why haven't you you know, like, I was speaking with someone who's in the law profession, and their organization is, encouraging them to leverage more GenAI in their work. And, you know, when you're in the legal profession, we've already seen some crazy things happen with, you know, GPT, hallucinating cases, and that sort of thing. Like, you don't you don't want that. You don't want the quality of the work to go down. You don't wanna be like, oh, use GenAI. Go faster. But then you don't want the quality of the work to to be, to be degraded. So I think focus on what you're trying to achieve, not how many tokens you're using.

Like, if you can achieve the same amount of things in the same amount of time with less tokens, like, that's great. Don't focus on how much you're consuming. Focus on the output.

That's fantastic. Rana, what what's your take?

Well, I'll I'll call out. Have a AI adoption goal mapped to a business goal. Otherwise, there's no point of adopting AI if you don't know which needle do you wanna move. And I also see organizations pushing for speed, speed, speed, but speed to drive what value is super duper important because otherwise Exactly.

If you're using a lot of tokens and you're not delivering anything to the business, you're not making it safer. You're not making it better. You're not Exactly.

Yeah. You're just doing a negative, value add to yourself, not to mention you're spending money. You're not getting any business outcome, and then you question ROI. So that's an important thing that I would call out.

That's great. Be mindful of the time. So last question, and I'm gonna ask, each of you to keep it to one sentence. What does AI in production done right actually looks like to you? Rana, you want to get started in this?

Well, AI in production done right means it's doing the job it was put in production to do with absolute high quality, 100% visibility, and traceability and ability to govern on govern it end to end.

Great. Caroline, what's your take? It's safe, and it's human centered, and it performs what it needs to do in a way that's almost invisible. It makes our lives better. It helps us with our experience, but it's not it's not the end to the means. Right? It's it's it's doing its job, and it may not even we may not even see it.

Yeah. I love it. Almost invisible. Carol, Kelly.

Those were those are amazing. I would say the same. It's it's it's really, creating new client consumer experiences, or even employee experiences, that kinda delight, but in a in a safe governed way. It it's really just doing what we've done today, but much better.

That's fantastic.