Before cloud pipelines, film footage travelled between countries by hand
Footage shot in a Mexican jungle may need an editor in Toronto, sound and VFX artists in India and a creative team in Los Angeles, and it once travelled as physical tapes carried between places. Netflix senior software engineer Anvita Jain works on the post-production pipelines that replaced that hand-off. The question is how a single platform keeps terabytes of media moving through jobs that run for seconds or for years.
See how Netflix builds media pipelines as state machines on its open source Conductor
Anvita Jain, Senior Software Engineer at Netflix, walks through how footage travels from a film set to a streamable title by way of automated, cloud-based post-production pipelines. She explains the stages of a media workflow, the microservices and "pipeline as a platform" layer behind them, and how Netflix models each pipeline as a state machine running on its open source Conductor orchestrator.
“A media workflow is a representation of a media exchange between different teams.”
“So every pipeline is in a manner, a state machine running on top of a process orchestrator and has certain resources that are being managed on it.”
What you will learn
Map the post-production stages a title passes through, from on-set ingest to editorial, VFX, sound, picture finishing, localization and archival.
Design long-running workflows as state machines with explicit states such as draft, in progress, failed and redeliver.
Apply a pipeline-as-a-platform approach so teams can run pipelines without managing scale, scheduling or cloud resources.
Use an orchestrator such as Conductor to handle retries, back-off policies, priorities, queuing and asynchronous tasks.
Explain why streaming services generate multiple encodes so quality fits each device and connection.
Key takeaways from the speaker
Cloud Technologies
Application of Cloud
Everything as a Service
Media Workflows
Pipelines
Details
Level
Intermediate
Format
Tech talk
Technical depth
Practitioner
For
Mid-career, Senior individual contributor
Length
25:05
Published
July 2022
Questions this talk answers
What is a media workflow?
It is a representation of a media exchange between different teams, such as editors, VFX artists, sound artists and picture finishing facilities working on the same title.
How do production partners get footage into Netflix's systems?
Technically equipped vendors connect through an external API, while others use a UI tool called Content Hub to import different types of media.
How does Netflix model its media pipelines?
Each pipeline is a state machine on a process orchestrator, moving through states such as draft, in progress, failed or redeliver, while a resource manager tracks the files and assets attached to it.
What does the Conductor orchestrator take care of?
Conductor, Netflix's open source orchestrator, provides workflow, task, decider and queuing services that handle scheduling, retries, back-off and priorities, with workflows defined as JSON blueprints and visible in a UI.
Why does Netflix video quality vary by device?
Netflix generates several encodes for different kinds of devices, so a small phone on a low-bandwidth connection does not receive the encode meant for a 4K TV.
About this talk
This talk is about how Digital Assets are handled from when a show idea is pitched to a video from a shooting location to making it streamable media on Netflix app by leveraging cloud. Often we enjoy our favorite show on Netflix but wonder how did it all come together. Get a sneak peak of what happens to each and every shot, how the VFX are reviewed and finalized, how it is processed to be of highest quality on different devices, etc. How we automated each and every post production process in cloud and scaled for the amount of shows and movies produced by Netflix every year!
One show, five countries: why footage needed pipelines (75 s)
Pipeline as a platform and device-specific encodes (83 s)
Every pipeline is a state machine (71 s)
Speaker 1: Hello, everyone. Thanks for joining my session. I'm Anvita Jain, and I'm going to present Media Workflows Leveraging Cloud, and you'll see what this actually is going to cover. So just a little background about me, first of all, that I'm going to talk about, then I'll talk about what are the workflows and the pipelines and deep dive into the architectural and how we are implementing and why we are doing what we're doing. And then again, I'll probably leave some time to take some questions. So I'm a senior software engineer working at Netflix. I work in the Studio Technology org. So we typically enable all the creators and the content creators with technology so that they can improve their time and use their resources efficiently so that they can clearly focus on their creative in depth. And the main technologies
that I've used is Kotlin, Java, Python, a lot of SQL, AWS products, we use Spinnaker for our deployments and a couple of other in-house tools. Now, if you talk about a life cycle of a content, right? So whatever favorite show you see on Netflix, it's not something that's just there like a magic, right? A lot of processing goes behind. A couple of shows even take like years to reach to the state where they are. So how does the whole process come into play? It starts with a pitch. So say, suppose tomorrow you have an idea that you think it could be converted into a show or a series or a movie, someone would come to Netflix and pitch their ideas. After that, our development team takes a look at it,
we do some kind of business negotiations and finally come up with the contract saying, okay, this can be formulated into a series or say 15 episodes, right? So those kinds of negotiations happen. And then comes the actual production state. And after that, it is finally launched on the service. Now, if I zoom in a bit into production state, there are a couple of steps involved. Some is called pre-production, then production, and then the post-production. So pre-production involves tasks like writing a script or, you know, finding the locations where you want to do the show. Actual production means you are out there on the sets with the camera and recording your scenes and music and say dubbing.
Sorry, not dubbing exactly, but maybe dialogues. And then comes the whole post-production architecture. So this is where our team, my team comes into play. So once everything is shot from there, how to make it streamable on the service requires a lot of post-production processes. And that is what I'm going to deal about today. So now what is a media workflow? A media workflow is a representation of a media exchange between different teams.
So for a movie to come together, there could be like a VFX artist who wants to get some visual effects done. For example, a VFX shot of a Demogorgon scaring Eleven in Stranger Things. There is nothing called a real Demogorgon, right? It's all visual effects. So lights, camera and media. How does it all happen? So for a global company like Netflix, even our production systems are globally distributed.
So it can happen that your onset locations are somewhere in New Zealand and then there could be a location in, say, Mexico. But your editor is sitting in Toronto and your sound artists and the picture finishing people are somewhere in India, followed by the VFX artists in India. So now all these people have to collaborate together, whereas our Netflix creative team is sitting in Los Angeles to figure out that, OK, I like the shot.
This is the final cut and it is survival. So what happens in these kind of scenarios? These are all live action productions. So first of all, a media is offloaded from a camera and the sound recorders which are on the set. Someone would copy over files between this and take it to different departments. So earlier it used to be like a manual process.
There will be actually a person will be carrying your huge tapes of recording from, say, one country to another country or one place to another. So we wanted to automate that whole process. And in this entire process, there are like terabytes of data that is produced and different partners across the globe interact with them. So we wanted to come up with a globally distributed processing using the pipelines.
Don't get scared with this diagram. I know it's quite intense, but we'll zoom into each one of the tasks. So when I talk about media workflows, there are several things that happen. So first few things would happen at the on-set location or near-set. So when I say near-set, just imagine if narcos is being shot in the jungles of Mexico, there might not be enough bandwidth or there might not be enough internet or manpower to download the information from camera, upload it somewhere into the cloud and no one can work.
So either they would do it exactly on set or they will have nearby locations where they will do that kind of stuff. Second comes the editorial. Now, on the editorial side, what we see is given that, say, one hour of tape out of that one hour, only a small portion is what editor should be looking at, because there could be someone clapping the window.
There could be some, you know, spot boys running around and that was recorded. So those kind of things are done at the editorial side. After that, there is a lot of like visual effects that are applied and there could be some sound and music kind of things that could be needed. Followed by picture finishing. So sometimes the light on the set is not sufficient.
You want to enhance some colors or, you know, add more brightness. So all that happens at a picture finishing facility. And finally, it comes down again to Netflix where someone would do localization in different languages. They will do dubbing, subtitling, et cetera, and finally make it streamable and also archive it. So now I'll get into one of these, right?
What happens when we have to collect camera from the media? So for that, we build a UI as well as some API. So a lot of companies who are actually doing these kinds of productions are very technology friendly. So they would have like a technical department who would write certain scripts or, you know, certain programs running on Linux machines.
So for them, we have provided an external API for which they can connect and start sending information information or their data or the files into our ecosystem. Or the second way we gave them by creating a UI tool called as Content Hub. If you just Google for it, you would be able to see it as well. And using that tool, they can import certain different types of medias.
Now, second part is the editorial part. So we have the media once, okay, in the first step. Second step is how to conform it to the editorial workflow. So typically what happens is there is a EDL, which is called Editorial Decision List. It's a timeline file. So it will have time code saying at time 000.1, the cut starts and it runs for, say, next 20 seconds.
After that, there is, say, a 10-minute break. And then another shot comes in. And those kind of information, it is guided by an open I/O standard, open timeline I/O standard. And using that, we produce an editorial decision list. Not reproduced, but usually the vendor would give it to us. Once that is done, we have processes in place to match those time codes with exact frames that we have got.
And when I say frames, it could be any kind of file that we got. It could be just a plain image sequence or it could be like a video or an MOV or MP4. And once that is all done, we would also have to look at some kind of transcoding because to interact between different systems, there is some kind of transcoding needed.
And then finally, a VFX plate or a VFX facility will actually get all the VFX plates, for example, if there are any involved. So same kind of process could span across different kind of assets. It could be VFX assets, it could be just raw camera footage, etc. So similar process happens for the conform pool. So when I say the term conform pools, it means that once you extract a video out of a camera, after that, how you match and trim it across the entire tape, so that you match exact time codes and you send only the relevant information to the next component in the pipeline, which could be like your editor who would be interested only in those shots.
And on top of that, they will start doing whatever cutting or, you know, additional review they want to do. Once this is all trim, they also send it to a picture finishing facility for reviewing, which will say, oh, the colors are not good, or I want to increase the lightness or someone would say the dress of the actor, the color of the dress of the actor is not bright enough, let's change the color.
So all these kinds of things start happening here. Now, once these things happen, there's a heavy need of visual effects. So I'll give you very basic examples of when visual effects come into picture. Say, suppose you're watching a scene where people are communicating over chat on their WhatsApp or on their phone, and you see those chat pop up bubbles popping up all over the screen.
You see, I'll give you a little bit more, but I'll give you a little bit more, let's see. I'll give you a little bit more. I'll give you a little bit more, let's see. One is called Media Review, for example. So in Media Review, what we do is we work with several different vendors. So just to create Demogorgon for Stranger Things, we could have worked with, say, three different vendors, ask them to submit, you know, different versions of Demogorgons to us.
And someone from the Netflix creative team would confirm that, okay, this is the final version that we want to go with. And these kind of things happen with a heavy integration with a lot of third party creative tools such as ShortGrid, which is a product by Autodesk. And it is heavily used in the creative review processes. I covered VFX short delivery and also the studio archival a bit in my previous slide.
So now let me talk about the picture finishing part. So how does picture finishing actually happen, right? You need certain ingredients or some recipe to generate a particular way. So suppose if there is a movie which is entirely shot in night or, you know, it's just like one night corresponding to entire movie. So typically they won't end up shooting everything in the night.
They would be doing these kind of shootings at stages or, you know, at certain set locations, but will have enough light. And then there could be certain filters. It's almost similar to your putting a picture up on Instagram. And then you want to put a filter on it, say, a Paris or LA. So they are, these are kind of different filters that you can put, right?
So once that is done, then we do the timeline media is verified and exact points are extracted. That out of, say, one hour of video, only five minute worth of video needs some kind of color correction, for example. So these kind of transitions happen in the picture finishing workflow side of things. And then there could be a VFX plate related deliveries available.
So just imagine if you see an, you know, especially during COVID times, right? They could not shoot on real locations. So it would be all on set or nowadays, there is a huge demand for virtual production. So what happened is you will have a big screen. Say, suppose it's having a background traffic and an artist would just stand ahead of it and do some acting.
Or the artist might even just sit at their home, do some acting and later someone will merge the two. So this background plate, which was just giving you a, like a scene of a busy street is like one plate on which you will overlay an artist. So all those kind of things could happen through our pipelines. Similarly, there are things that would happen for sound.
So it could be like, you know, generating audio stems separately, generating some mixes, doing dubbing. There's a lot of machine learning kind of thing involved to generate dubs and subtitles before a manual person would go and do a QC. So all these processes have been starting to get more and more automated in the industry now. Now, that was a lot to take on the media side, right?
But if you see technically, this all cannot be done by like one single monolith system. So we came up with different systems behind each of these systems are microservices in place. So to start with, if you see our first exchange about how we get the media, right? It could be by a partner API or it could be using our UI tool called as content hub.
And we, behind the scenes, we use GraphQL services to interact with our other downstream processes or microservices that can do some kind of say authentication, like about who is allowed to access what or what kind of processes are allowed on certain type of assets. So beneath this, we have created a platform called as media workflow platforms or pipeline as a platform.
And people can just come right there pipeline and start running. They don't have to worry about scale. They don't have to worry about scheduling. They don't have to worry about resource management, seeking an instance in AWS running. All that is managed by our platform. Beneath that, we also interact a lot with our encoding and inspection teams. So what happens basically is once a shot is shot, it needs to be encoded so that the quality remains correct on a tool like on, say, suppose your iPhone or if you're having a small Android phone in a low bandwidth internet connectivity area.
So you might have noticed that the quality of Netflix content that you see always goes up and down with the kind of device you are in. But that quality is still good quality. So this happens because we generate several encodes for different kinds of devices. We don't try to use an encode for a 4K TV on like a very bare minimum smartphone.
And then we also have an asset management platform, which basically what it does is it takes up all our files and we try it. We have certain schema. So we create assets against certain schema. So tomorrow if I have to go look up at footage and trigger some other workflow based on availability of footage, I can tie it all together using my asset management platform.
And underlying, we have a huge global storage service that takes care of what data gets pushed into AWS, how to secure an S3 bucket, how to store things, how to make sure you do a checksum and validate that what we delivered in AWS is exactly what we were supposed to deliver. And then there are other teams with which we interact are like Netflix's internal platform tools and data science platform to run a lot of analytics on top of it.
Now, how is the whole pipeline composed together? So every pipeline is in a manner, a state machine running on top of a process orchestrator and has certain resources that are being managed on it. So when I say a state machine, it means that when I trigger a pipeline or a workflow, it will be first of all, say in a draft state, someone would submit it giving all the inputs that are supposed to go in.
Once that happens, we turn the state to say in progress and we start processing each of the node in that workflow. As part of that process, sometimes the pipeline could fail and we can mark the state as failed. So on the UI, the user would know that the pipeline is in a failed state. Similarly, we can say that, hey, your data that you provided was not sufficient, please redelive.
So we can change the state to redelive. And all these kind of states are visible to the end user via our UI tool. And at the same time, we have a resource manager. So for a pipeline, there could be, say, five VFX shots that are attached as a resource. And the state of resource could change throughout the pipeline process.
So at some point, it could be just a simple file. After a couple of minutes, it could turn into an asset. After a couple of minutes, it could turn into an external file being delivered to someone else. So we keep track of all those stages as part of our resource manager process. And to run everything, of course, we need an orchestrator.
An orchestrator is a process which would make sure that everything gets scheduled, it gets processed, calling to different services and things like that. So for that, we use Conductor. Conductor is a Netflix's open source orchestrating system. And it allows you to come up with, like, different kind of, you know, workflow definitions. It gives certain inbuilt tasks that you can use.
And then it takes care of scheduling and queuing everything. So what are the properties of a media workflow or a pipeline, right? It has its own unique and persistent entity. It can be started manually or automatically as a trigger based on one pipeline. Certain pipelines run for, like, minutes to seconds. And some can even run for months or years depending on the kind of work they are supposed to do.
All the execution happens as part of a predefined set of nodes, which could be sequential. At certain stages, there could be some parallel computing involved as well. And if you look at a high level, it would look like a graph. So how would multiple nodes or a directed graph in certain manner? It can interact with other systems that I covered about, like, you know, how our entire ecosystem is made for.
We can have certain await tasks in it. I can wait for, you know, generation on one kind of asset. And till that happens, my pipeline is in a pause state. As soon as that happens, it releases a lock or it somehow informs my pipeline saying, I'm ready for you to review. Once that is done, you wait for, you know, its own states and data.
It can be queried at any point. And then we publish several events also. Now I'll dig in into how our process orchestrator or a conductor is composed and how we use it. So at a high level, if we talk, right, like, of course, it needs some kind of storage to store all the definitions. But on top of that, it gives you certain services.
So one of the services is called workflow service. A workflow is a composition of several tasks. So using workflow service, you can create a workflow. You can start a workflow. You can stop a workflow. You can retry. You can fail a workflow. Define the workflow definitions and things like that. Similarly, there are task services. So using a task service, you can define what would be my retry strategy on a task.
So if, suppose, a task fails in the first attempt, do you retry for a few times? Or do you, you know, can you, like, restart after a couple of minutes? What is, like, exponential back off, kick off policy that you would use before retrying it and those kind of things. Then it is accompanied by a decider service and a queuing service.
So using the decider service and queuing service, you can also define certain priorities and then conductor will queue your tasks accordingly. So the end user like us, we don't have to worry about, oh, how would my, you know, hundreds of instances would be used? How does all my tasks get allocated and how do they run? So that, so basically for an end user, you don't have to worry about that kind of granularity about how your distributed computing is going to happen.
So at a high level, a workflow can be defined using a JSON blueprint, which obviously can be interacted using a workflow API and some metadata API to define the blueprints. You can have abilities to, like, resume, restart process. And there is a user interface also involved. So you can always see the progress of your workflow. It clearly shows what are the inputs, what are the outputs, what are the logs.
If it fails in the first attempt, it will tell you, like, first attempt failed, what were the logs for the first attempt. Now if it is retrying, you can also see that kind of information on the UI. You can synchronously or asynchronously process things, which obviously is a great deal because sometimes you will want your workflow to kick off certain process and you don't want your workflow to wait for that.
And you can just catch that result later at some point in the workflow. So all those kind of event handling and that can be done using a combination of the event handler that we build as part of a pipeline platform as well as the conductors ability to asynchronously process things. We have been running, like, millions of workflows till date, concurrently, a lot of them, and it's been a seamless experience for the end user.
So it's all backed by a queuing service, so everything is extracted from the client and we don't have to worry about how does your task get scheduled. So one thing is, of course, because now then it's like indeterministic behavior for client user, right? So if I want my task to get picked up right away, but if for some reason there's a huge load going on, I might have to wait.
And the best thing is it does allow a lot of communication over HTTP or GRPC. So a lot of our new services are now moving with GRPC. So our interaction still remains seamless. And so now what are the technologies that we have been using? So a lot of technologies that we have used are like, of course, Conductor that I talked about.
We use Python, we use Kotlin, we use Espera Connect from IBM for doing a lot of data transfer. So when someone has to upload a file, so it creates a session for you and then it takes care of getting the uploads and the downloads done. We use a lot of Java. Our UI uses JavaScript. Everything, every service into our system is a microservice.
So we do not have any kind of monolith architecture and we try keeping everything separate as much as possible. And we use Spinnaker for interacting with our AWS instances and taking care of our deployments in the build cycles. So as part of this presentation, I would definitely like to give credits to my entire team and all the partner teams that we have worked with.
So the video engineering team, our media workflows team, Studio Technology, Art, and animation that I use in this from storieset and slidescore.com. And that is it. So if there are any questions, I would be happy to take it now or if you feel free to connect with me over LinkedIn and I would be also happy to chat with you if you are more interested in anything.
Thank you everyone for joining the session.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
How does Netflix turn raw set footage into streamable media in the cloud?
Netflix engineer Anvita Jain shows how microservices, a pipeline platform and the Conductor orchestrator automate post-production at global scale.
In short
Anvita Jain, a Senior Software Engineer in Netflix's Studio Technology organisation, explains how footage moves from a film set to the Netflix app without anyone carrying tapes from one country to another. Her answer is a set of microservices, a "pipeline as a platform" layer and Netflix's open source Conductor orchestrator, which together turn each post-production step into an automated, trackable workflow that runs in the cloud.
Key takeaways
Cloud technologies
Application of cloud
Everything as a service
Media workflows
Pipelines
From pitch to post-production
Jain opens with the life cycle of a title. A show starts as a pitch, moves through development and business negotiation into a contract, then into production, and finally launches on the service. Production itself splits into pre-production (scripts, locations), production (cameras, sound and dialogue on set) and post-production, which is where her team works.
She defines the core idea simply: "A media workflow is a representation of a media exchange between different teams." (2:33) Those teams are rarely in one place. In her example, sets might be in New Zealand or Mexico, the editor in Toronto, sound, picture finishing and VFX artists in India, and the Netflix creative team in Los Angeles, all needing to agree on the final cut.
Why the work had to leave the tape era
Every shoot produces terabytes of data that partners around the world must touch. Jain recalls how manual this used to be: "There will be actually a person will be carrying your huge tapes of recording from, say, one country to another country or one place to another. So we wanted to automate that whole process." (4:19)
The goal became globally distributed processing through pipelines. The stages she walks through are on-set or near-set ingest (for remote locations with little bandwidth), editorial, visual effects, sound and music, picture finishing, and finally localization, dubbing, subtitling, streaming preparation and archival back at Netflix.
Inside the pipeline: ingest, conform, review and finishing
To collect media from cameras, Netflix offers two doors. Technically equipped production companies can call an external API from their own scripts, while others use a UI tool called Content Hub to import different media types.
Editorial work is driven by an edit decision list, a timeline file built on the OpenTimelineIO standard and usually supplied by the vendor. Netflix's processes match those time codes to the exact frames received, whether image sequences or video files, transcode where systems need it, and trim footage in conform pools so only relevant shots reach the editor and the picture finishing facility.
For visual effects, Netflix may ask several vendors for versions of the same creature and let its creative team choose, with heavy integration into third-party review tools such as Autodesk's ShotGrid. Picture finishing then adjusts colour and light, which Jain compares to a familiar habit: "It's almost similar to your putting a picture up on Instagram." (11:56) She also describes VFX background plates used in virtual production and sound work such as audio stems, mixes and machine-learning-generated dubs and subtitles that a person then checks.
Microservices and pipeline as a platform
None of this runs in a monolith. Media arrives through the partner API or Content Hub, and GraphQL services connect to downstream microservices that handle concerns such as who may access which assets. Beneath them sits the media workflow platform, which Jain calls pipeline as a platform: "They don't have to worry about scale. They don't have to worry about scheduling." (14:44)
The platform works alongside encoding and inspection teams, an asset management platform that stores assets against a schema so new workflows can be triggered when footage is available, and a global storage service that pushes data into AWS, secures S3 buckets and validates deliveries with checksums. Encoding explains something every viewer notices: "So this happens because we generate several encodes for different kinds of devices. We don't try to use an encode for a 4K TV on like a very bare minimum smartphone." (15:21)
Pipelines as state machines on Conductor
The technical heart of the talk is how a pipeline is composed. "So every pipeline is in a manner, a state machine running on top of a process orchestrator and has certain resources that are being managed on it." (16:32) A pipeline moves from draft to in progress, and can be marked failed or set to redeliver when inputs are insufficient, with every state visible to users in the UI. A resource manager tracks attached items, such as VFX shots, as they turn from plain files into assets and then into external deliveries.
The orchestrator is Conductor, Netflix's open source system. Its workflow service creates, starts, stops, retries and fails workflows; its task service defines retry strategies and exponential back-off; and its decider and queuing services handle priorities and scheduling so engineers need not manage how hundreds of instances are allocated. Workflows are defined as JSON blueprints, can run tasks synchronously or asynchronously, can pause while awaiting another asset, and can be queried at any time. Durations vary widely: "Certain pipelines run for, like, minutes to seconds. And some can even run for months or years depending on the kind of work they are supposed to do." (18:28)
Jain notes one trade-off: because everything is queued, a task may wait under heavy load. As the speaker cited, the team has run millions of workflows to date. She closes with the stack (Python, Kotlin, Java, JavaScript, IBM's Aspera Connect for transfers, Spinnaker for deployments) and a firm design rule: "So we do not have any kind of monolith architecture and we try keeping everything separate as much as possible." (23:36)
Watch the talk
"Media Workflows Leveraging Cloud" by Anvita Jain, 25 minutes: https://www.womentech.net/video/media-workflows-leveraging-cloud-anvita-jain
About the speaker
Anvita Jain is a Senior Software Engineer at Netflix who came to the United States from India for a master's in computer science at NC State University and later worked at Bank of America, Goldman Sachs and eBay. She loves solving complex distributed systems problems and building pipelines at Netflix scale, and she is always open to mentoring junior engineers and college graduates.
Questions this talk answers
What is a media workflow?
How do production partners get footage into Netflix's systems?
How does Netflix model its media pipelines?
What does the Conductor orchestrator take care of?
Why does Netflix video quality vary by device?
Drafted with AI from the talk transcript. Editor review pending.