I'll start now. Hi, everyone. Thank you so much for being here. I'm Pragya Keshav.I'm a technical architect, and I'm a Google Developer Expert in Google Cloud and also a Docker Captain. Today, I'm going to talk about, something that I kept running into some issues that I kept kept running into at work. And, I'm not going to talk about any new framework or a new model, but the the problem that we all keep facing. So the products that we build are actually good. We always have the features that we want. The documentation is there, but the users users still can't find what they need.

So that's that's where the pro that is that's what this talk is going to be about. What if the problem isn't the product? The problem is how people access the knowledge about it. Okay. Let me move on to the next screen. Okay. So we ship some features. We write docs about it. We put in Wiki help portal and maybe a PDF. And then users still raise support tickets asking how to do the things that that we just documented. So research show that less than 40% of the users discover the entire product features. So less than 40% of the product features are actually discovered by the users. So less than half. That means more than half of what your team built, the engineering investment, the design decisions, the documentation effort goes unnoticed. The issue is not the product. It is not even the docs. It is that the documentation is static. It requires users to already know what they're looking for.

And here is the shift I want you to make today. The gap between what your product can do and what user thinks it can do is not a platform problem. It is a discovery problem, and that is what we are going to solve. Okay. Let's see the old way versus new way. How does the problem look like? In the old way, users search for a keyword, get a list of documentation, read through them, and hope they find the right one, and repeat. The system has no idea what the user user actually needs. It just matches the words. So the static FAQs do not understand the intent. Search returns documents, not the answers, which which is pretty similar to how initially the Google search used to work.

But that's when the docs go stale, and they always go stale. The search engine has no way of knowing the difference. Now let's talk about the new pattern. The new pattern flips this completely. Instead of asking users to search for a document, we let them ask a question in plain language. And instead of instead of returning a list of links, the system finds the most relevant piece of your actual product knowledge and generates a direct grounded result. And that shift is powered by RAG, which is also called as retrieval augmented generation. And let me be clear. RAG is not a AI magic button. It is an architecture, and let us see how it looks like. So I guess the point. Okay. I'm sorry. There's some problem with my mouth. Let's talk about the three layer blueprint. The architecture I use is the simple or regular three layer model, which is ingest, retrieve, and respond.

Let's talk about layer one, which is which is the ingestion layer. This is where you prepare your product knowledge. You take your documentation, FAQs, release notes, feature guides, and you chunk them into meaningful pieces, not too big, not too small. And I'll come back to why that matters. Critically, you add metadata to each chunk, which product, which version, which user role. This matter metadata is what makes retrieval intelligent later. And then you generate vector embeddings for every chunk using Vertex AI. Think of these embeddings as mathematical representation of meaning. Two chunks, that means similar things, will have similar embeddings even if the words are different. And you store all of those in the AlloyDB with p g vector. That's just one option. If we are using GCP, then we can use LRD with with the PG vector. Let's go to the second layer, which is retrieve.

When a user asks a question, you embed that question the same way. Then you run a semantic similarity search against your vector store to pull the top matching chunks, the most relevant pieces of your product knowledge for that exact question. Next is the third layer, which is the respond layer or the response layer. You took the those retrieve chunk chunks and pass them to Gemini via Vertex AI. Along with the user's questions, the model generates an answer, but it is constrained to what you gave it. It cannot make up things from its general training. It answers from your product knowledge and excites the sources. So this is the architecture deployed on deployed and scale on scale on g GKE with APIs on top. Now let's see what runs where on Google Cloud. So if we are using Google Cloud, then what are the resources or applications we should we should be using? Yeah.

So on the control plane, Vertex AI handles the embeddings and the Gemini calls. AlloyDB is the vector store. It has native p g vector support and scales with with the needs. Cloud storage holds raw documents and the outputs of the ingestion pipeline. On the runtime side, GKE runs your chat service, your retriever, and your orchestration logic. Apigee sits at the API layer for both rate limiting policy enforcement, etcetera. And cloud logging plus BigQuery give you the observability you need. The reason I like this tag is that every piece is managed, scalable, and part of the same ecosystem, which is the Google Cloud ecosystem, and you don't have to glue together multiple different vendors. Now let's talk about the real challenge, the grounding layer. Now here is the thing I most want you to walk away with.

Most chatbot projects fail when we dig into why. It is always never the model. The model is fine. What fails is the ground grounding layer. These failure modes, I see over and over. First, unstructured documentation, raw HTML, inconsistent heading, missing metadata. The retriever cannot distinguish signal from noise. So it returns vague chunks and the model generates vague answers. Second problem is wrong chunk site size. If it is too large, the context gets diluted. If it is too small, you lose the meaning that spans across a few questions or few sentences. Chunking strategy is an architectural decision, not a detail you leave to defaults. Third one is no versioning. Still doc chunks silently corrupt corrupt the answers. The users ask about a feature that changed three releases ago and still get the wrong answer, but confidently delivered. So we should treat the knowledge base like an API.

It needs structure, versioning, and contracts. Like, you would not ship an unvalidated API endpoint. You should not ship an unvalidated retrieval process. And this is where the architect adds the most value, not in picking the model, but the design, the pipeline but in designing the pipeline that feeds it. Let's talk about guard guardrails and production readiness. Before you ship your product to the users, there are four things that you need to build. First is observability. Log everything, the questions, what was retrieved, what prompt was sent, the response, and the latency. You cannot improve what you cannot see. So a chatbot without observability is like a black box, and users will stop trusting it the moment it gets something wrong, and you cannot explain why. So use cloud logging and BigQuery here. Next is fallback handling.

If the retrieval confidence is low, if the system cannot find a good match, do not let the model hallucinate. Return a documentation link and ask clarifying questions or route to a human. Set confidence confidence thresholds per query type. That graceful degradation is what separates a trustworthy production system from a prototype or a demo. Next is API gateway policy. We can use Apigee to enforce authentication rate limits and tenant boundaries, especially if different users or organizations should only see the content relevant to them. For scalability, we can use GKE. So keep your ingestion ingest pipeline and your serving pipelines as separate deployments on GKE because they have different load patterns, and mixing them leads to resource contention at the worst possible time.

So trust is not a feature that we can add later. It is a design decision you have to make on day one. So the key take takeaways that I want you to leave with is number one, the blueprint. Ingest, retrieve, respond. This is the simplest mental model for the product of your chatbot. If you remember nothing else, just remember these three words. Number two is ground in real knowledge. Use your actual product content, not generic model training. Vertex AI embeddings, alloy vector search, Gemini. The power is how well you connect those systems and not in which model you pick. The number three is that it is a grounding and guardrails problem. The hardest part of building this is not the AI. It is the architecture of the knowledge layer underneath it.

Structure it, version it, and govern it, and this is where we, as architects, have the most leverage. So from docs to dialogue is not just a title. It is a design principle. When your AI system knows your product as well as your best engineer does, you do not just improve the user experience, you build scale trust at scale. So that's my presentation. Thank you so much for watching and listening to me. I've shared certain links here. I'll share the presentation later on. Thanks, everyone.