My name is Rudima. I am a senior staff engineer at Instacart.I have been in the Instacart company for more than ten years, around ten years now, I would say. And I've worked on, many customers and shopper facing products. I'm thrilled to be here in Women in Tech conference. This year, this is my first year and it's amazing to see all the energy, the different sessions, and I was waiting for my turn. So I'm excited. In this session, I'll share a lot insight from our journey in Instacart on, like, how we build Instacart's customer care support chatbot. We also call it as clear bot, and we'll explore how combining the LLM models and with the predefined logic tree helped our, helped our keyboards responses significantly to be more reliant, more, accurate and more actionable.

Actionable. So let's dive in. So to let to start with, let me introduce what Carebot is. It's Instacart, Gleneye chatbot specifically created for customer's care support. We're also working now after doing this, we also supported the same for our shopper support, and it's doing great as well. So Carebot utilizes our existing help center articles for there. We summarize and show it in a more personalized way for the customers. We provide the interactive, follow-up questions just like the chatbot. Most of the chatbot does, and it also supports the direct actions which customers can take within the chatbot. So if a customer is stuck on some of their order issues, or if they wanna cancel or reschedule, they don't have to leave the chatbot. They actually can perform the action within the chatbot itself. And then there is, like, a seamless human escalation, so also supported.

So, if there is a need for customer to hand it over to, for us to hand over the customer to an agent, There is a full context shared with the agent. So that that also from within the chatbot, they don't have to leave. There is a direct, support of that as well. So before we launched Carebot, our admins our customer and shopper help center articles were entirely static pages. So you can think of, like, normal. Like, you have to click one page, and that gives a list of options. From that, you select something. So they were very general article, very informational and accurate, but very generic article lacking any personalization. So if you are in a certain state, stuck because of some issue, the page doesn't know, and it might just give you a generic, solution, which might not work for you.

And also it did not have any interactive with me. So it does not understand where, what you want exactly. Again, it help being a generic, it only gives you a a a very common problem solution. So there was because of that, there were, like, high frustration point sometimes for customers when they are not able to, proceed in their, journey, whether it might be placing an order or, like, taking action on an existing order or, because of, any other reason they want to take action with their account.

So this led to, also high volume on the human escalation. Of course, that, includes comes with the cost, like agent cost, and also waiting time is high always when you have to reach out to customer care. We all have faced that. Recognizing these limitations and with the rise of AI, we we saw that AI chatbot now is much easier to build. And so as an opportunity, we improved our customer interaction significantly. The AI net the chatbot offered natural conversation. It more like human, it enhanced the contextual understanding because now you know exactly you can fetch that from the current state of where customer is, what is trying to do, what its, current profile looks like. And then based on that, you get responses again because of that, it's more personalized.

And so a customer likes it, you know, and then, of course it's scalable because now adding a new flow to an AI is much more faster, compared to creating those pages and then asking people or, like, you're making that pages findable for customers. So, those were some key benefits of our AI chatbot, and, of course, much better than static stuff. However, adopting, and purely generative AI solution brought some challenges, which we all will point out. So LLM sometimes produces incorrect information with confidence. It's like, you know, toddlers to big one tell the lie with a lot of confidence, and, we call it hallucination. So sometimes you feel like, well, the answer is correct, but then is it correct in context of my use case? Is it correct in concept context of Instacart? We even support this flow. So, that those were some challenges. Of course, the prompt improvement and all can fix most of it.

But we felt that these, kind of issues when every time new use cases are, encountered, it causes a lot of confusion, in the customers. And because of this, there is also lack of consistency, which can lead to diminishing user trust. They they're like, well, I I don't trust this. It's sometimes right. Sometimes it doesn't work. So I would rather just skip it and go somewhere else, for the help. So, so it wasn't that reliable. Furthermore, purely generative chatbot, lacks actionability. And why is it lack here is that you can add to some extent, but not all, because there is, like, what amount of information you want to share with them to, take an action. Right? So you have to give the context and there comes the data privacy thing. How much of the user information you want to share with the chatbot to build its context so that it can take some action on behalf of user.

So keeping that all in mind to end addressing all these challenges, we've adopted a hybrid approach. So we combine the strength of AI and MLM models, there with the structured logic trees. So people who don't know about the structured logic trees, it's more like a defined tree. You can think of in your computer science and you might've seen, where there is a node and then it has a children node. So there is like an option. Ship. Then from there, you have another offset of option till you reach reach the least note. So, you can think of an example of, like, a cancel order. If someone says that I want to cancel an order, okay, what are the reasons? We'll show you the list of reasons, and then from there we go.

This is how the creation of chatbots were made. So, this this, combining our AI conversation, strength with the piece, defined tree definitely helped us, move faster. So we were more confident on pushing, more actionable things on the predefined part versus more conversation, things on the LLM. Bit. Then the result was more accurate, reliable. We know exactly the customer wants to cancel the order, more flow they have to go. So if they're clicking the cancel button from the cancel flow somewhere, all they're typing in the cancel order, they exactly know how the flow should be. So that also gives them more confidence that, okay, this is how the flow would look like. That also gives you as an team, engineering team, a lot of flexibility you need to run, not on engineering, but product team to run an experiments on these, flows.

So if you want to tomorrow, add a new friction flow in the cancel order in this, I'm using the cancel order, for example, here much easier. So, add, new friction flow. Are you sure you want to cancel? It's much easier adding into the predefined flow rather than changing, in in, like, complete, GenAI, prompt thing. And, it was just much faster. Same as the way you wanna remove something, it's much easier. So experimentation is much faster, easier. So before we get into the technical design, I'll just show you guys a demo. So this is what I was telling you guys about. There is a flow here saying to cancel order from a card or if you go via a chat, which is here. So here we do say cancel or reschedule. Here's a list of predefined reasons. You select one of it and do, or you say I wanna chat instead.

So you chat and then, in chat, you say, well, cancel my order. Instead of being too chatty, here I understand the intent that it is want to cancel the order. We ask them which order you want to cancel. This is the order. And then the flow is consistent. So you know what I'm saying that, okay, here we are using to refine. We use AI to understand the intent whether it wants to cancel. And if there is a predefined flow, we just switch to that. So let's go to the tech design. So, bidding on that example, so this is how the flow looks like. So when a user comes in, you see that there are, like, multiple predefined, steps, or I would say parallel, checks we, run.

Here, we check that whether whatever the text, message or the query has come in. It doesn't match with an existing predefined flow. If it does, we click a a shortcut here as you saw in the cancel example. We will pass it on to the predefined flow. And from there, the predefined flow will take the charge. If there is, another check is more is the message as a very clear intent. So the if you write cancel my order, the intent is very clear. You want to cancel. The question here would be which order, right? Is this your actual or existing order? So that is a, and clarifying question. Are you talking about this order? But if there is a very unclear intent, say, someone just type this, cancel. Cancel my order. Do you wanna cancel your account? Do you wanna cancel your membership?

It's not clear. So unclear intent would be the flow will be triggered here, that it will ask the clarifying question. I understand what you want to say, but can you be more clear? Are you thinking of these options or anything else you have in mind? So that is just more like a better follow-up question. Another, check here is like, okay, there are some sensitive use cases we want our agents to handle. Like they cannot have a human intervention there. So say something around, some fraud has happened, or they want to do something with the payments, which is more like a sensitive issue. We would want, customers request directly instead of being engaged into that NAI exactly decimal. We better escalate them to, the Asian therapy so that they get more better, help.

Or, if there are sometimes, like, it's hard for us to find, the understand the impact. If customer is just after four or five back and forth unclear questions, we have been asking them, are you you mean this? They're like, no. Doing this. And then like, whatever, rather than being too chatty and frustrating customer, it's better to understand the threshold or fix a threshold, I would say, within your system. And then from there, like, escalate to patients so that customer can handle, talk to human and get them to resolve much faster rather than frustrating someone who is already in trouble more and not able to express it or, just, not, we are not able

to understand it, you know?

And, then if we don't have a predictive mind flow, but we do have a help center article, which we could actually with the help of Jenny and I, add more context and be more precise about their profile. We would summarize that health center article to help them and then show them the response in the chatbot itself. So if, you are a mobile user and your order is in the hidden menus, we will say that when you go into my accounts and do the orders and do this, but on web, it might be different. So that that's kind of more personalized response you'll get, which you don't get it on the static pages. So those are the kind of responses, comes back, as I mentioned, these things, all these prompts are like, they are AI prompts and they do the different functions. They all run-in and based off, there is one more prompt about it, which actually, make sure that we we understand which one to which response to prioritize and then, respond to the customer.

One important practical learning I wanna share here from the air force implementation was that approach to prompt design. So as we discovered when we were working on this, that breaking a complex, larger prompt to a smaller prompt did wonders. So we were able to move much faster when a single prompt had a single responsibility. And, it well, we were able to, even, you know, break it into smaller pieces where we yeah. We we it you just that's not new exactly what to do. This is your work and just do that rather than giving one big push from a lot of responsibilities where it's struggling sometimes saying that, oh, well, if you try to get it in some aspect, it was too regress. So, and it was too debug.

Like, why did it was it

a problem or is it

made from a problem? So here it made it much easier. It after doing that, is that making the prompts into a smaller. Prompts smaller prompts, help us also with the activity of selecting the model prompt. So some models, even example, etcetera, the summarizer, I would use a particular model which is more better at writing, versus one which is used for, for our clear clear clarification, asking clarification questions. It is more one for the chatty or chat. So you based on your need, you can select the model. You can go deeper model with the ones where it's only basic responsibility that there, where you have to write and curate or things like that. You can use more, you know, in advance. So that actually added us with the cost and operational, efficiency as well.

Because now you can, do the mixed resources in the one, big, large model prompt. We had only stuck with one model. And then at least there is, things which came with it was better stability. So, you guys, many of you guys might know about LRT URLs. Those are like the test cases for your prompts. And so writing those test, emails for per prompt was much better. You could test which use cases are not working and then we kind of fix the prompt in that way, even when newer, cheaper models keep coming, we can test it, test that prompt against those models and then switch the models if that is doing equivalent work and if it's cheaper for us.

So, lot of win win situation on the prompt design here. The real world impact of this was, of course, customer satisfaction, static pages. There was a big win here. We, our customer satisfaction metric opened up, our, support, escalation. So agent, handoffs went down because we were able to support most of our important use cases within the chatbot. It says, setting up the system like this also held, in clarity. So are you able to hear me now, Terry? Hope hope it fixed it. So, setting up our system in this way, improve this, kind of, put sets us up for, scalability as well. And, as I mentioned, the cost saving, benefit which came with it. So key takeaway, for everyone here is, hybrid approach win. We replicate the same for our, from the customer care support to the shopper care support, and we are doing great there as well.

So hybrid approach definitely work better for us. Prompt design matters. We took this learning from the care board across the company because, that really, one key learning for us. And, of course, setting up emails is something which, which is current with initial time, risk taking part, but it helps you move faster a lot. So, yeah, we we are going to add a a lot of more more, capability on our care bot. And, so if you guys are interested, actually interested, would love to chat more about it. And, the end of the slide, thank you. They are it has been really good. I'm very I wanna call that out, and it's a new design. I got a new so yeah. Feel free to connect with me on the LinkedIn. My barcode is here. If you wanna have, more chat about this topic. Yeah. Happy to talk about it. Sure, Christy.

Happy to connect. Thanks, Matthew. Thank you. Thank you, guys. It's it's nice sessions. Cool. Alright. I think my time is almost up. If anybody has any question, I can answer from the chat itself. Thank you. Alright. You guys see you guys on LinkedIn. Alright. Bye bye.