Let's get started. My name is Jadishree. I'm a tech tech lead manager here at Fireworks, and I lead part of our applied machine learning team.Today, I'm going to talk to you about an interesting topic called make RFP boring, in a good way. This whole at the end of this presentation, I hope that you have a predictable and a practical way to do, fine tuning or training on Fireworks or any other platform. So the agenda for today is three, four things. So one, we'll get started with an introduction where we learn about different fine tuning techniques, then we'll move on to how RFD works, including evaluation, which is a big part of RFT, look at a customer use case, and leave some time for q and a. There are mainly three different types of fine tuning. One being super vised fine tuning, second being reinforcement fine tuning, and then there is TPO.

Supervised fine tuning or training itself is used when you actually have, say, high quality input output examples. Sometimes and it and it is, really useful when you have, like, a clear gold standard. It's easy, fast, and predictable. But there are sometimes when you wanna teach a model your brand voice or style or alignment or even, guardians in terms of safety. That's where DPO comes into the picture, where you can train a model based on preferred versus nonpreferred output pairs. Lastly, there is reinforcement fine tuning where you can train a model with the help of an evaluator on a real task. This is mainly useful, especially in today's agentic system.

And throughout the presentation, we will be focused on laser focused on the middle one because, to be quite honest, not only does it help and and improve the model performance, it's also quite, misunderstood today. As part of my job, because I work with customers, one thing I see again and again is customers come to us, and they say that their agent is working fine. They've tested it. And in the dev environment, it passes all their evaluation criterias. But then they roll it out into production, and they slowly start noticing that it won't follow any format. It picks the wrong tools or even at the wrong time. It passes incorrect arguments or tools, and it can't even learn, conventions. In fact, it's just too slow. And all these problems, they might look at, look like SFT problems. But the usual fixes that you can think of when you think of fixing an SFT problem, probably breaks down. Why?

Because most customers or you don't even have a 100 k labeled examples. You might have around 200 or sometimes even none. And in fact, most of the time, happens is even if you have, even even if you have, like, labeled dataset or not, what you don't have is a verifiable, what you do have is a verifiable task. So you have, like, test validators, business metrics. So a way to, like, easily evaluate whether your model is improving or not, during each run. So the real question then doesn't become how do I get more dataset. It more so becomes how do I make my agent reliable when I cannot rely on these big dataset? But remember, we do have a verifiable task. This is exactly the kind of question that evaluator driven training, answers.

You have an agent that interacts with the environment, then the environment responds, and you then evaluate the performance of the agent using an evaluation or a reward function, and then you retrain the model. These tight loops happen again and again, until the model improves. Let's dive a little bit deeper into RFT itself. In terms of, what RFT is under the hood, it is mainly three different steps and a very, very tight loop. In the first step, let's assume that you've given a prompt to an LLM, and it generates multiple responses. Let's take a, b, and c. And step two, what you do is assign scores or rewards to each of this response. Say, over here, response a got 0.6, and then b got 0.2, and then c got negative 0.1.

And the step three is where the LLM's weights then get updated based on these rewards that we are seeing in step two. So it's a easy three step process. You let the LLM generate a response. You assign a a score, to these responses using a reward function or an evaluation technique. And step three, these LLM weights get updated based on these scores. Now that we've gone through, RFD itself and the different kinds of fine tuning, a lot of people have questions around what kind of fine tuning should I choose. And here are some examples. If you have a simple classification or domain adaptation task, then SFT is a good fit. But if you have things like code generation, text to SQL, or deep research, or even customer support agents, then RFT might be a better bet.

We will go into a a customer use case later in this presentation. Let's dive a little bit more deeper into RFT and see how it works on Fireworks, especially the evaluation itself. So the the process on Fireworks for RL is includes five steps five simple steps. One is a user creates a JSONL file with prompts. And then in step two, they define an evaluator. So this is the function that scores the model output anywhere between zero or one. For an evaluator, you can use a rule based technique, a more deterministic technique, or you can even have LLM as a judge, which is a little more nondeterministic. Once you have both the dataset and evaluator ready, you can then connect. When you say connect, there are multiple ways of doing it.

One, you can upload the dataset and it to Fireworks and then run a training job. Or if you are an advanced researcher or ML employee, then you can actually even use our training API, which lets you define your own training loops and use the dataset and the evaluator that you have. The step four is actually creating the RFD jobs. Over here, you can define different hyperparameters, as part of this training job, and then Fireworks handles the rules the rollout for you. And step five, this is where it gets really, it gets real. You not only are able to train the model on the platform, you can also now deploy. And not only deploy, but also let it scale into production. And one important thing in this whole process is actually step two, defining your evaluator.

Because the final model you get is only as good as your dataset evaluated and and hyperparameters. So let's look at how we make the evaluation process simple. We make that simple using something called an eval protocol. Eval protocol is is is like a standard interface, where you can plug your, agent and eval into the RFT itself. Why actually use eval protocol? This actually lets you reuse your reward signals in your agentic workflows, instead of having to rewrite them. And how it does it, it's very simple. It runs some rollouts, collects the results and traces, and then trains using, different algorithms such as gRPO, DAPO, GSPO. If all of that went over your head, don't worry. I will simplify it in the next few slides. But the result of, using eval protocol as part of your RL is quite straightforward. You have faster iteration.

You have more reliable agents. Let's walk through the same example we were looking at. Let's take this there's an agent that interacts with an environment that then gives a response, and the response is evaluated using an evaluation or reward. In and let's assume this agent itself is a Python file that just runs a loop and responds to different user queries. Let's call this agent dot py. And let's now assume there's an environment that could be a cloud's SQL service that takes in the query that the agent gives, executes it, and returns the result back to the agent. Let's call this SQL, service dot py. In a typical example, as we were talking about it, the agent makes the tool call. The call gets sent to the the SQL service on cloud, the query run, and the result come back to the agent.

But in a classic RL environment, the environment needs to return a reward. But in real systems, that can be messy. For example, over here, you're returning a SQL result, and that's quite messy to actually use as a reward function to update your model weights. So given things like the SQL result, conversation history, or a ground truth, you need to be able to turn all of that into one reliable reward signal, which is sometimes hard to define and even harder to maintain. And that's why you get eval protocol. Eval protocol itself wraps the agent and the environment into one component. So it like, we are seeing two components. A lot become, like, one component, and it's totally nonintrusive. So you don't even have to write your entire, agentic loop itself. It'll be, RL ready using the functions that eval protocol gives.

And now what you do have an agent in environment becomes one block, and then there is an eval dot pi that just takes in the output and then turns it such that it can be rewarded itself. So what changes is that you run you run your task normally, and then you send the outputs to eval, eval dot py function. And the using eval protocol, we run the evaluator, feed this course into the training loop. So now you can do RL without actually having to rebuild your entire, agent or an environment itself. Now let's talk about a customer use case where this was really, really helpful. So something similar was used, eval protocol and RL, while actually building Composer two for Cursor. And at a high level, four main things went into building the Composer tool model, especially as part of the RL flow.

One was, we didn't actually focus on, delta updates, like small we focus on small, delta updates instead of full model weight changes. When we do RL, most of the time, not all the weights, all the layers change in the model. Instead, what changes are small weights. So instead of actually updating the whole model each time between each epoch, we just updated, updated smaller weights. Next is actually training and generation can, and rollout can happen in in simultaneously. We modified our code such that both of these can happen simultaneously, especially because this helps us fully utilize the GPUs. The two other things we focused on was something called hot loading where the new weights can up be applied instantaneously to a deployment that is running.

Usually, when weights are updated, there is some amount of downtime for the deployment before it is actually rolled out. But this entire process lets us eliminate the whole downtime and because the new weights can be instantly applied. Another technique we use is called distributed GPU training where there were GPUs in multiple location instead of using, like, one massive cluster because our capacity can become a bottleneck really soon. And, this is the exciting news that Koso published about, Composer two model itself using RL. That's all we have for today. Thanks for attending.