A model built in three weeks can still take 11 months to deploy
There are fewer perfumers in the world than astronauts, yet Firmenich has a robot that composes fragrances on request, and building that kind of intelligent product takes far more than model code. Teams can build a model in weeks and still wait months to deploy it. Priyanka Sharma, an enterprise architect who began as a developer 17 years ago, asks what organizations must change so models, data and code reach production together.
Learn the three organizational changes that let MLOps, DataOps and DevOps work together
Priyanka Sharma, an enterprise architect at Firmenich, traces how waterfall and agile led to DevOps and distills the general idea of ops: break a life cycle into steps, assign tools to each step and automate the workflow. She argues that intelligent applications rest on three pillars (model, data and code), explains how MLOps and DataOps differ from DevOps, and outlines the team structure, tooling and KPI changes organizations need to get models into production.
“And what we all have observed now that no matter in what industry or what domain you belong to, being digitally successful can make or break your enterprise.”
“each ops is very very distinct discipline and they require different people and tools so don't push the people to do all the tasks together because the skill sets are very very different”
What you will learn
Explain how waterfall, agile and DevOps tooling led to the general concept of ops.
Apply the ops pattern to any domain: map its life cycle, assign a tool to each step and automate the workflow.
Distinguish MLOps from DevOps and DataOps in testing, monitoring, users served and skills required.
Compare product squad and centralized team structures for building intelligent applications.
Define model KPIs up front so teams stop tuning models indefinitely.
Key takeaways from the speaker
complete understanding of the model lifecycle
Understanding of Devops and how does regular devops differ from modelOps
ML model code is only a small part (~5–10%) of a successful ML system, and the objective should be to create value by placing sustainable ML models into production .
Details
Level
Intermediate
Format
Talk
Technical depth
Introductory technical
For
Mid-career, Senior individual contributor, People manager
Length
30:04
Published
July 2022
Questions this talk answers
What does ops mean beyond DevOps?
Ops means identifying the life cycle of a domain such as code, models, data or a business process, breaking it into small steps, assigning a tool to each step and letting the tools talk to each other so the workflow runs on autopilot.
How is MLOps different from DevOps?
Model training and testing differ from code building, models depend on data quality and accuracy requirements, and model monitoring tracks model and data quality rather than CPU and latency. MLOps also serves data scientists, who need self-service tools rather than complex developer tooling.
Are MLOps and DataOps the same thing?
They go together but are distinct disciplines: DataOps relies on pipelines, Spark clusters, SQL, ETL tools and database administration, while ModelOps needs data engineering and data scientist profiles.
How should an organization structure teams for intelligent applications?
A product squad puts a data scientist, data engineer and software engineer together for fast iteration but risks losing expertise, while a centralized core team model iterates more slowly but centralizes skills, governance and best practices. She also recommends separate infrastructure teams for app, ML and data services.
Why do models take so long to reach production?
Because organizations need structural changes to onboard ops: the right team structure, the right tools for collaboration, discoverability, security and self-service (buying rather than building where possible), and clear ownership with defined KPIs.
About this talk
It took a while for the IT industry to realize that Development and deployment goes hand in hand and There could be no successful project without Devops. We should take this learning and adopt MLOPs as a basic element for any AI/ML project.
My talk will consist of with
1: The explaination of development process and introducing devops , and importance of devops
2: Understanding what are the curent trends and best practises in devops.
3: explaining how the development of AI /ML models differs from traditional application develoment
4: How to adopt devops in machine learning annd introducing modelops
5: What is models ops
6: lifecycle of preditive models
7: Best practises Pre and Post model deployment
8: How modelops cover the model lifecycle
Hello, everybody. My name is Priyanka Sharma. I am an enterprise architect in Firmenich. So I started my career as a developer around 17 years back, and I have seen the evolution of this IT industry very, very closely. And what we all have observed now that no matter in what industry or what domain you belong to, being digitally successful can make or break your enterprise. And the concept of ops is so important. It's the key factor in the digital evolution or your digital success. So today I'm talking about what is this ops and why does it really matter? So before we dive into the project or into the presentation, I would like to give you a brief introduction to Firmenich. So Firmenich is one of the major giants in the
perfumery business. We cater to fine fragrance and consumer fragrances. We actually manufacture the fragrances which goes into each and every end product, like body home care, like your creams and shampoos and so on. It's a very, very niche skill. And the fun fact that I want to share with you here is that there are less perfumer in the world than astronauts. So we are working with a very, very specific and unique expertise. And that expertise also goes into all the application development.
And we are very proud to announce that we now have a digital perfumer. So there is a robot to whom you can tell that what kind of fragrance you desire and it can develop that fragrance for you. So that kind of digitalization we are talking about. OK, so let us go into the agenda today. So I'm going to talk about the brief history of the DevOps. We'll deep dive a little bit on the waterfall and agile practices, come to the concept of DevOps and derive the ops concept from it. Then we understand what does the future application demands, what is model ops and data ops and the key organizational practices for a successful op? So we see that there are plethora of ops, right? There is DevOps, there's design ops, there's products
ops and so on. And these ops are not any longer just technical. They're also going into the business domain. So you also see like marketing ops, business ops, people's ops, sale ops. And with all these ops, what you also see is plethora of tools. You know, with each ops, there comes support of tool and we are flooded with so many tools. And so we'll understand why with the concept of DevOps, why did we come across so many tools and why there is suddenly so much surge of tools with DevOps, right?
So before that, we see how the DevOps came into concept, right? So it all started with application development. So to begin with, we were doing agile, we were doing ad hoc application development in the beginning, right? In 1970s to 2000, waterfall came into practice. So we were doing a sequential development, I would say. Then the agile practices came into picture. So agile concept actually brought a lot of things on the table, which had upper hand over the waterfall approach. And then Flickr announced that they could do 1000 deployment per day, which was really unheard of in that era. And to support this 1000 deployments per day, lot of open source DevOps tools came into practice. And these tools were born or basically envisaged to help this 1000 deployment per day. And then, you know, there came a Phoenix
Project. So Phoenix Project, I'm sure you guys must have read the books called Phoenix Project. If not, I really, really recommend if you're starting on the DevOps practices. And with Phoenix Project, all of this came in into a workflow to my automatization, and then a lot of enterprise started to adopt the DevOps practices. And what next now? So we will talk about what next now. But before that, let's dive a little bit into the waterfall model. So waterfall model is a sequence sequential software development practice, right? So you have different boxes for each phase. So you have requirement phase, you have analysis phase, you have design phase, you have implementation phase, and everything works in silo. So you don't talk to each other. So developer works in his own box in his own machine,
operations prepare their environment for production, tester prepare their use cases, and so on. And then, you know, you suffer from something called the local responsibility syndrome, where, you know, developer says, Okay, it works on my machine, I don't know why it is not working on production, and operations that I have done exactly what developer has said. Now I don't know how the work, the code works, and so on. So we were always in this flux. And, you know, there was always this wall of confusion. Because you know, they were separate boxes not talking to each other. So while all this was happening in waterfall, there came agile concept. Now what happened with agile, that these boxes started to interact with each other, and this wall of confusion was broken. So now developer has the
understanding of the business requirement, developer was being connected to the operational guys, infra guys, and the functional and the non functional requirement were working in tandem. The code, whichever code was ready was being deployed. And you know, the small iterations, workable code, frequent releases, it all came to reality. But this was still a concept. And there is a very funny analogy that I have figured out that I really want to share with you, because it's, it's quite apt. So let me know if you can hear it.
Well, you fill in the blanks. Okay, so now, we're gonna start discussing men's brains, women's brains, and how they're very different from each other. Now, I want to start with men's brains. All right. Now, men's brains are very unique. Men's brains are made up of little boxes. And we have a box for everything. We've got a box for the car. We've got a box for the money. We've got a box for the job. We've got a box for you. We've got a box for the kids. We've got a box for your mother somewhere in the basement. We got we got we got boxes everywhere.
And the rule is, the boxes don't touch. Sorry, my Catholic upbringing got in there for a minute. I'm not a Catholic, but I went to Catholic school when I was little. I had a nun who taught on hell like she was born and raised there. I mean, I'll never forget it. But it did be good. Actually, it was a good thing.
Now, women's brains are very, very different from men's brains. Women's brains are made up of a big ball of wire. And everything is connected to everything. The money is connected to the car and the car is going to do your job. And your kids are connected to your mother and everything. It's connected to everything. So I think we got the concept. So I would like to say that, you know, a waterfall model is like this men's brain where we have boxes and not connected to each other. And then the agile came into picture where it's like woman's brain. Everything is connected to everything, which makes things quite quick and quite easy. And everybody everybody is aware of what is happening in totality and not in in a separate
box. Right. So when agile came at this point, agile was good in concept, but definitely the tools and technologies were not matured enough to help agile practices. And what DevOps brought into the table is this tool and technology to support the agile way of working. So there were tools designed to cater the specific task and the tool was supposed to communicate to the next tool so that the workflow could be achieved. So if we look at the development life cycle, we could commit the code in Git. Git can actually trigger the Maven build. The build can actually trigger the test cases. The test cases after being finished can build the release. The release can automatically be deployed. Now this deployment can be on infrastructure which could be automated.
And then once everything is in production, it could be monitored and so on. So what we had the seamless flow of things without intervention. And this is what actually DevOps brought into the picture. So there are two main concepts. First is tools for the individual steps and the workflow automation. So from this DevOps, if we derive the concept of pure Ops, right, if we talk about operations in any context, what does it off means? So Ops basically means that you identify the life cycle of whatever domain you're working on, be it code. So here we are talking about code and development, but it could be models, it could be data, it could be your business process, you break everything into small, small pieces and derive the life cycle
for it. Then you assign tool to that individual steps. And then these tools can talk to each other and can do your workflow automation. So that is basically the core concept of Ops. And why is Ops so important? Work Ops is so important because it takes a lot of responsibilities out of your head. And these all responsibilities can run in an autopilot mode so that you can focus on your business area and not actually think about operations because operations can run automatically. And these responsibilities include analytics and analysis. So all these tools are available which can give you analytics and analysis. They can give you alerts, notifications of what is going wrong and so on. We talked about automation, we talked about enablement you can put your governance on on top of it all the orchestration tools are available
you have you know your workflow can be representative of your process so you can do process control you can configure your own processes you can set up monitoring on your infrastructures and tools so everything can run in autopilot mode so basically if you do ops means you have spent a lot of time in planning how your operations would work and then from there on it works in an autopilot mode so that is basically the concept of ops and it can be applied anywhere so uh if we just recap a little bit so what happened in a waterfall early step of uh industrialization of the core developer operations were working in a pre-silo manner agile software development everybody started to work together but the tools and technology was not matured then
the concept agile concept was put into practice then tools and technology came to support it the workflow automation came because tools started to talk to each other and then the enterprise adoption of devops happened and now we have seen recently that in past decades that the applications are becoming stuck so the what is future so future here is intelligent you know it's not only smart applications it is intelligent applications so in next 10 years every piece of software that we are going to interact with is going to be intelligent it's going to have intelligence built in so it is already available in youtube it is available in emails we write because it auto correct your sentences for you it is available in our inboxes it is available in the various appliances that we are buying for our
household and um so every software component will be intelligent and we as a community are working towards it for that goal for for sure right uh so uh here i'm taking example of the perception system of a self-driving car and uh basically idea here is to to uh highlight that there is a lot that goes into building an intelligent system and it's not the old concept of only coding right so in the case of the car uh the car will have sensors and the cameras in order to figure out what is the obstacle on the road what does landmarking look like and so on and so forth and uh uh you know uh if you are uh taking all this so first step would be you know to take data from all the cars that is on the street uh in different
conditions then you build a model to train on that data and probably most of the time goes into this model preparation then you run that model on a specific test set so your test set could be a golden test set uh where you have uh the exact conditions uh like you know uh the low light condition the rain condition or the road on the snow so you have different test sets for different kind of condition to test your model and once your model is tested and validated on this local test set you actually work with the code together and work this whole system into a simulator because you will not test your car on production the first time right so you make this uh this simulating environment where you actually are
driving simulator where you will test your code and then once everything is okay you package your code here is all ci cd and everything that comes into picture you deploy the model and the surrounding code together then you could put code into production you make predictions the car actually drives now you are collecting more and more data so your code actually has to be built to uh to capture the predictions of what the model is predicting because you need data for the future in order to improve your model then you use all that data together and do the whole exercise all over again because your correct prediction your incorrect prediction everything goes back into the model to train the model further otherwise your model will get obsolete so uh again the thing i want to highlight here is building an intelligence system
is way more than building just a model or building just a code or working just on data it is really going to be able to look at the problem so it is going to be able to work in order and then you need to work in order to run this into autopilot mode you need ops for all these three pillars and you want to make these three pillars working as efficiently as possible so all ml and ai basically picked up the concept so ml ops and data ops picks up the concept of this ops from devops right what does this mean so if we go and look at the ml life cycle um so there is again uh data preparation there is training and there is another the whole loop of collecting the data doing feature engineering training the model
evaluating the model and once this model is prepared you operationalize the model you deploy the model you serve the model you think about auto scaling resource utilization high availability sl maintenance and all these things you monitor your prediction you analyze the insights and and go back into the loop so again this entire model life cycle has to be broken and made efficient in ml ops so what do we do again you define the ml into small tasks you set up tool for the individual task and you set up the workflow automation but looking at the complexity of the life cycle you're going to find that there is not just one single silver bullet that will solve your problem you're going to find tools that will
help you in specific pieces or specific tasks and they might do one one or more but they have to interact to the next tool in order to actually make the workflow automation and that is why you see that there are so many tools available there because the tools does not mean that it has to fill ml ops only the tool also need to fit into your enterprise the tool must fit into your processes as well right so there is no no just one set of tools that say if you want to do ml ops go ahead with this tool and and you'll be efficient it depends on your business process what are your workflows what are your kpis and so on so now we have seen similarities between ml ops and data ops right let's let's touch upon uh data ops also
so data ops is actually a bit tricky uh because data ops is quite new and quite large and the definition are still being processed it's it's not complete now we are evolving on data ops so data ops is basically the combination of uh information or data architecture to ensure the data quality data security data integration data governance and all those things and um so data integration comes with tool the etl tools and so on you will have data governance where you will see who is responsible for the data if i have to add a field i have to delete a field which domain is responsible for which kind of data what data goes into which application and so on and once you have figured out all of this information architecture the
data integration and data governance you have to put automation so that all these things whatever is is decided works into into the workflow so basically what data ops engineer does is supporting the tools that follow this works flow and and you know uh for example setting up the a4 pipelines uh or providing data in the data lake and make them available to the various bi tools and so on so all this goes basically in into data ops so uh we talked about similarities so far so let's look at a little bit of differences as uh so as i mentioned that ml ops uh and data ops basically reuses the devops tools and principles where possible but they're also distincts why because domains and activities are different so model training and testing is very different from code building
or testing right and uh does the test case even exist for a model okay what data goes into the model is very important if you uh you know you say you fill uh you give crappy data you will give the crappy you will get the crappy results so here uh it is very much dependent on data where code uh is a separate identity in itself and um there is also accuracy requirement for the model there are outliner settings and and the concept all these concept is quite difficult different from a typical development mindset right similarly uh when you do uh model monitoring it is very different from apm monitoring so in apm what monitoring what you'll uh monitor for you will monitor for uh cpu utilization latencies while model
monitoring will look at what is your model quality what is your data quality um and you know you will trigger alert based on that then we should also understand that people we are serving are very different right so here uh in model ops we are serving data scientists rather than software developer which are less on technical tools and so on so that is why the toolset needs to be different a developer can work on a tool which is really complicated but data scientists have to have more power you know to do self serving rather than self coding or things like that right um so mlops also need tight integration with data but mlops is very different from data ops generally people say that data ops mlops
goes together yes it goes together but mlops is different data ops is different and the skills and requirement for these ops are very different data ops skills would be mostly the fo pipelines the spark clusters a lot of sql queries a lot of etl tools and and so on you might also be working with administrating the caches and uh administering the databases and so on uh whereas uh for model ops we would probably require data engineering and data scientist uh job profiles so uh after we have understood this whole concept of ops we still get tweets like this right so you build a model which is in three weeks but to deploy a model still take 11 months because uh your model ops need some organization level
changes to onboard ops so i here list the most three important changes that i have observed in my life that an organization must do in order to embrace ops so first and foremost is structuring your team you know there are two popular approaches of structuring an application development team or to build an intelligent application of course so first one is the product squad structure as i would call it so what does that mean uh is that you have a product team so you have a data scientist you have a data engineering uh data engineer and a software engineer together in a team and they work together for that product but it is still a silo development these people are very skilled in this requirement and will
efficiently deliver this project very closely talk to each other and so on so here the iteration uh cycle is quite fast but the problem here is that you know the core expertise is only within that project and unless you have a governance and the documentation model and your knowledge sharing model within your organization uh these skills will be lost the other kind of structure uh that that works is the coa structure so you have a core data science team you have a core data engineering team and then you have you know resources from these team uh being assigned to the products here the problem is the iteration cycle goes a little longer because there is no sorry because there is no uh a dedicated resource for this
particular project but in this case your skills are centralized um the problem which a person will face in product one the same problem will not occur in product two so probably you can save time there plus everything would be centralized and governed and you can put best practices in terms of architecture and everything if you have a coa structure um and with development you also need to think about the infrastructure now again as i said that uh for developing an intelligent app it is different uh from code development so it also impacts your infrastructure team and it really makes sense these days to uh you you know have a dedicated team for each service so you should have or you must consider having an app
services infra team an ml service in stream and a data service uh infra team uh because uh the skill set the tools for these three uh are absolutely different and then within your organization you will also have you know the data data services team uh using the capabilities of app service team app service team using the capability of data services uh the machine learning code has to be deployed in the application so these are so within uh within your organization and outside your organization a modular component and then you can structure govern and make vertical organization to derive the three infra team and can have uh the devops data ops and model ops put in place together right the second thing that i would say is
as i have been emphasizing on tools the right tool is needed for the right task so you need team for collaboration uh you need collaboration between the teams so how to do data science team will collaborate with each other and the ability to discover what model exists what data exists is the data critical is the model available for every user or is it built for a specific user and so on and same is the discoverability so any data scientists should know that this this complex model how it is working what data it needs and so on so everything has to be transparent otherwise you will have this intelligence but nobody knows how this intelligence work and next time you have to do something on it or if the person leaves you have
no knowledge around it and you have to build everything from scratch so collaboration discoverability and documentation are extremely important same is in terms of security your model has to be secured when uh so the the users of the model is very different from uh the people who have delivered the model so in terms of security you can't even imagine how people uses these model and they will push in um phishing data they will uh they will try to break your model and so on so this has to be thought thought thought thought through and you have to put proper tools in place so that your models are secure secure and uh what really works is self-service wins as i said that your data scientist team
should be capable of serving themselves they should not struggle to get the data that i need data from application a application b no you should have data warehouse you should have data lake concept uh you should have data catalog where they can think about what are my data objects where what is the master data and how and where should i get this data from and so all these capabilities should be built with the right tools and uh with the right tools the most important decision that you have to take is whether you have to build or buy you have to focus you know uh whether you want to focus your team's time and energy in building a tool for their task or actually doing the task so uh recommendation is by the tool
if it is available rather than trying to build it yourself and the third and last thing is ownerships and kpi so this is really the key piece so uh if you have the ownership you decide the kpis of course right so no matter if you're working on for a company or an academic organization you always want to build the best model but you know there is always a limit your model can be improved and improved and improve but you need to define your kpis and and say that you know if i meet this kpis this model is done for me or this these kpis are important for me and the rest of the kpis are not important for me because what i have also seen that a lot of organization waste a lot of time in just tuning
the model and by that time something else comes in the market so you can't keep tuning the model indefinitely you have to derive you have to define your kpis and you have to meet those kpis so in some cases like healthcare uh the model failure can be life threatening so you have to be really really careful in defining your kpis there whereas some in some cases like recommendation engine it might not um be that critical so you can let go uh some kpis right so with that i have covered a bunch of ground today so let's recap quickly the takeaways so first and foremost that you know if you're working on a data science or data engineering team uh you must understand that developing a futuristic applications
stand on pillars model data and code and you should think of it holistically as an organization when you're bringing or building your organization you must think of these three pillars and not just follow the concept of code development these ops are made to optimize uh for iteration speed and high quality so you can bring your product to market quickly each ops is very very distinct discipline and they require different people and tools so don't push the people to do all the tasks together because the skill sets are very very different as i mentioned you should have various infrastructure teams and various you should have a data scientist data engineering and a software developer as the key roles and not try to intermingle these team structure and tool as we talk about these can expedite your
collaboration or actually can bring you down if you don't have the right set of people and tools at the right places it could be very difficult so choose wisely and last but not least the kpis are very important before you think of any intelligent application you should put a stop right like we used to call in the code that you should define the scope so you should also define the scope for your model right so uh that's all i have and what i would say is just keep calm and let the operations handle it and now i would open it for any questions if you have i see a question here which talks about what is data governance okay so data governance is basically suppose you're working in a financial domain and you have key data objects like customers you have
objects like credit card information and all these things so your data object must be defined and must be defined what that object concerns of and who is responsible for that for example customers can have customer address right but do you want to put this customer address in your core customer as a data object so all these things have to be defined and there has to be a data owner there has to be a data stivered so that your data object does not fall out of place it stays what it is defined for and you have a person responsible for that data so it is a big exercise these days going on in all enterprises where you have data catalog where you catalog all your main business objects you define who is the business
owner for it who is the technical owner for that object and if i have to add a field for example in this table for that object it has to be approved by a workflow so you can't just go along and add any field in the customer and so on so the data governance is whole practice is put around it and there are a lot of tools with support data governance as well can you have ah of course why not so uh i am i will post my linkedin profile here give me a minute or you can search by priyanka sharma fermanish and you'll find it okay fantastic guys i think i have taken more time than i'm prescribed to so thank you very much for
whoever was listening to me and i would love to connect to you send me a invite on linkedin and please don't hesitate if you have even any more questions on linkedin i'll be very happy to answer okay thank you very much and thank you woman tech for giving me this opportunity to speak thank you bye
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why machine learning models need ops, not just good code
Priyanka Sharma shows how the lessons of DevOps carry over to models and data, and what organizations must change to get intelligent applications into production.
In short
Priyanka Sharma, an enterprise architect at Firmenich, argues that intelligent applications stand on three pillars (model, data and code) and that each pillar needs its own ops discipline. Drawing on the history of DevOps, she explains what ops really means, how MLOps and DataOps differ from it, and the three organizational changes that stop models from stalling between training and production.
Key takeaways
A complete understanding of the model lifecycle.
An understanding of DevOps and how regular DevOps differs from ModelOps.
ML model code is only a small part (about 5 to 10%) of a successful ML system, and the objective should be to create value by placing sustainable ML models into production.
From silos to DevOps
Sharma opens with a reminder of why this matters to every organization, not only to technology companies: "And what we all have observed now that no matter in what industry or what domain you belong to, being digitally successful can make or break your enterprise." (0:00) Her own employer is a case in point. Firmenich creates fragrances for consumer and fine fragrance products, and now has a digital perfumer: a robot that develops a fragrance from a description of what you want.
To explain where ops comes from, she walks through the history of software delivery. In the waterfall model, requirements, analysis, design and implementation lived in separate boxes, and developers, testers and operations prepared their work in isolation. The result was what she calls a wall of confusion, where code that worked on a developer's machine failed in production and nobody owned the gap. Agile broke down those walls, connecting developers to business requirements and to operations, with small iterations and frequent releases. She illustrates the shift with a short comedy clip about boxes that never touch versus everything being connected to everything.
But agile, she notes, was ahead of its tools. Flickr's announcement of 1,000 deployments per day triggered a wave of open source DevOps tooling, and The Phoenix Project helped bring these practices into enterprises. "And what DevOps brought into the table is this tool and technology to support the agile way of working." (8:00) A commit in Git triggers a Maven build, the build triggers tests, tests trigger a release, and the release deploys to automated infrastructure that is then monitored.
What ops really means
From DevOps, Sharma distills a general pattern that applies far beyond code. "So Ops basically means that you identify the life cycle of whatever domain you're working on, be it code." (8:42) The same approach works for models, data or a business process: break the life cycle into small pieces, assign a tool to each step, and let the tools talk to each other so the workflow is automated.
The payoff is that operations run on autopilot. Analytics, alerts, governance, orchestration, process control and monitoring are handled by the toolchain, freeing teams to focus on the business. Much of the work, she explains, goes into planning how operations should run; after that, the system largely runs itself.
Intelligent applications stand on model, data and code
Sharma's view of the future is clear: "in next 10 years every piece of software that we are going to interact with is going to be intelligent" (11:00). She uses the perception system of a self-driving car to show how much more this involves than writing code. Data is collected from cars in many conditions, a model is trained, then validated against golden test sets for low light, rain or snow. The model and surrounding code are tested in a driving simulator, packaged through CI/CD and deployed. In production, the system must capture its own predictions, correct and incorrect, so they can feed the next round of training. Without that loop, the model becomes obsolete.
That is why MLOps and DataOps borrow from DevOps. The machine learning life cycle (data preparation, feature engineering, training, evaluation, deployment, serving, scaling and monitoring) has to be broken down and automated in the same way. There is no single silver bullet tool, she cautions; the right tools must also fit the organization's processes, workflows and KPIs.
How MLOps and DataOps differ
Similar principles do not make these disciplines the same. Model testing is not code testing, results depend heavily on the data going in, and models carry accuracy requirements. Model monitoring watches model and data quality rather than CPU utilization and latency. The users are different too: "here uh in model ops we are serving data scientists rather than software developer" (18:26), which means tools must support self-service rather than heavy coding.
DataOps, which she describes as still evolving, combines data architecture with data quality, security, integration and governance. Its skills center on pipelines, Spark clusters, SQL and ETL tools, while ModelOps calls for data engineering and data science profiles.
Three organizational changes
Sharma points to a familiar complaint: a model built in three weeks that takes 11 months to deploy. Fixing that requires organizational change. First, team structure. A product squad of data scientist, data engineer and software engineer iterates fast but risks losing expertise without strong documentation and knowledge sharing. A centralized core team structure iterates more slowly but centralizes skills, governance and best practices. She also recommends separate infrastructure teams for app, ML and data services.
Second, the right tools for the right task, with a strong emphasis on transparency: "so collaboration discoverability and documentation are extremely important" (23:38). Models must be secured against people who try to break them, data scientists should be able to serve themselves through data lakes and catalogs, and teams should generally buy a tool rather than build one.
Third, ownership and KPIs. Organizations waste time perfecting models while the market moves on: "so you can't keep tuning the model indefinitely you have to derive you have to define your kpis and you have to meet those kpis" (25:42). How strict those KPIs are depends on the stakes, from life-critical healthcare to more forgiving recommendation engines.
In the Q&A, she explains data governance as defining core data objects, assigning business and technical owners and data stewards, cataloging them, and routing changes through an approval workflow.
Watch the talk
"MLOps and ModelOps - Why does it matter?" by Priyanka Sharma (30 minutes): https://www.womentech.net/video/mlops-and-modelops-why-does-it-matter-priyanka-sharma
About the speaker
Priyanka Sharma is an Enterprise Architect at Firmenich and describes herself as an enthusiastic technologist and strategist. Her work focuses on empowering industries through enterprise architecture practices.
Questions this talk answers
What does ops mean beyond DevOps?
How is MLOps different from DevOps?
Are MLOps and DataOps the same thing?
How should an organization structure teams for intelligent applications?
Why do models take so long to reach production?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
DevOps
MLOps
ModelOps
DataOps
Agile
Waterfall model
CI/CD
Product squad team structure
Resources mentioned
The Phoenix Project (book)
Cite this talk
Priyanka Sharma. "MLOps and ModelOps - Why does it matter?." WomenTech Network, 2022. https://www.womentech.net/video/mlops-and-modelops-why-does-it-matter-priyanka-sharma