How a data flywheel cut document turnaround from 90 minutes to 10
Loan applications still arrive as paper-based documents, and lenders type each data point by hand into their systems. A product and machine learning team from Ocrolus, including Senior Technical Product Manager Evan Zhang, shows how over 700 human verifiers and machine learning models feed one loop that holds accuracy above 99%. The open question is when a machine can safely take a human out of that loop.
See how ground-truth testing shows a team when its model beats the average human labeler
Evan Zhang and her Ocrolus colleagues Sylvia and Kirsten explain how a data flywheel automates lending document processing: processed documents become labeled data, better machine learning models and a better product that attracts more users. The talk covers document classification and extraction, why diverse high-quality data matters, how human verifiers and models work together, and how ground-truth accuracy measurement shows when human steps can safely be skipped.
“Lots of high-quality data is mandatory for a good model.”
“And then it's the idea that with the more documents you process, the more data you have, you can build better machine learning models.”
What you will learn
Explain how a data flywheel turns more users and documents into better models and a better product.
Recognize why diverse, representative and correctly labeled training data matters for model quality.
Design human-in-the-loop review screens that make model errors easier and faster to spot and fix.
Measure model accuracy against multiple human labelers and an expert ground truth.
Decide which parts of a workflow are safe to automate using historical accuracy and model confidence.
Key takeaways from the speaker
Optimizing Machine Learning
Details
Level
Intermediate
Format
Panel
Technical depth
Introductory technical
For
Early career, Mid-career
Length
16:57
Published
June 2021
Questions this talk answers
What is a data flywheel?
It is the idea that the more documents you process, the more data you have to build better machine learning models. With human verification, that produces a better product, which brings more users, more documents and more data.
Why do machine learning models need so much high-quality data?
Real examples vary widely in background, cropping, breed and position, so a model needs diverse, representative and correctly labeled data to learn abstract rules. Mislabeled or narrow data, such as only grey cats, teaches the wrong thing.
Why combine image and text models for document classification?
Because Ocrolus has both image data and OCR text, it builds an ensemble that combines a text model (BERT) with an image model. Together they give better accuracy than either model alone.
How do you measure accuracy in a combined human and machine system?
The same document is sent to several people, their labels are compared with each other and with an expert's ground truth to set a baseline. The model's output is then compared with those human labels to see when it beats the average human.
When can human review steps be skipped?
When pages have historically very high accuracy and the model shows very high precision and confidence, those pages can bypass human review for that step. The remaining pages still go to a person.
About this talk
The Machine Learning Fly Wheel: Optimizing the process of creating a set of training for continuous machine learning
Speaker 1: Hi, everyone, and welcome to our presentation on how Ocrolus uses a data flywheel to modernize financial services. We're really grateful for the chance to be here today and to share some of the problems that we're solving at Ocrolus and how machine learning and data play a critical role. Here's our agenda for today. We'll first introduce ourselves and then move into an overview on the problems faced in financial services and how Ocrolus fits in.
After that, we'll give an overview of machine learning at Ocrolus and how we enable humans and machines to work together to build a better product. For introductions, my name is Evan, and I'm joined by my teammates, Kirsten and Sylvia, for today. I'm a senior technical product manager at Ocrolus building out our core client and API platforms. Hi, everyone. I'm Kirsten. I'm a technical product manager here, and I work on our document classification and paste of data capture products.
Hey, everyone. I'm Sylvia and a senior machine learning engineer at Ocrolus. Thanks, Kirsten and Sylvia. So to jump right in, the lending industry is full of manual work. Especially when it comes to processing documents now, loan applications are made up of multiple documents that come from several different sources. And these different lending industries require different sets of documents that are specific to the borrowers or the lending products.
So for example, you'll need to give your lender different documents, depending on if you're applying for a mortgage or a small business loan. But regardless of the type of lending, lenders need to review these documents throughout several steps of their process. And there are lots of inefficiencies here. There's a lot of back and forth between the lender and the applicant through email to get all the documents in one place.
Lenders are then manually checking to see if the applications have all the required documents in them, in the event that something was incorrectly uploaded by accident. From there, they're looking at the data on these documents and then typing each data point into their internal systems. And finally, any kind of analytics or data verification is done with back of the napkin calculations or checking values from one PDF to another.
So overall, it's safe to say that it's time consuming and error prone. And it comes down to the point that these documents are fundamentally paper based, while lenders have moved their workflows and processes online. So there's the problem statement. How can we make document processing more efficient while maintaining high levels of accuracy? And that's where Oculus comes in.
Oculus is a fintech infrastructure company that takes in financial documents and returns structured data for what these documents are, whether it's a bank statement, a pay stub, a tax form, et cetera, and also what data is on these documents with over 99% accuracy. This allows financial services companies to make high quality decisions with trusted data without compromising for efficiency.
We've been able to cut down the time it takes to underwrite loans and process applications for clients like Bluevine, Brex, Square, SoFi, and PayPal. One area of impact that we're really proud of is our work with the Paycheck Protection Program or PPP. PPP is a program that we launched last year in the US by the Small Business Administration that offers loans to small businesses to help keep their workers on payroll during the COVID-19 pandemic.
And some really cool stats that I can share with everyone today, over 3.6 million small businesses receive PPP loans that were processed by Oculus lenders. This means that we helped save more than 5 million estimated jobs through these loans by processing over 8.7 million PPP supporting documents. Now, how do we do that? Our product is based on the idea of a data flywheel.
And you might be familiar with what a regular flywheel is, which is essentially a really heavy wheel that takes a lot of force to get it to start spinning around. And just as a flywheel needs a lot of force to start it off, it also needs a lot of force to make it stop. So as a result, when it's spinning at a high speed, it tends to want to keep on spinning because it has a ton of momentum pushing it forward.
The data flywheel is similar to that. And then it's the idea that with the more documents you process, the more data you have, you can build better machine learning models. And with the help of human verification processes, ultimately a better product that will lead you to more users, more documents, more data, and the cycle continues. And if you don't know what machine learning is, Sylvia will go into this in a bit more detail in just a minute.
At Oculus, we have over 700 data verification employees who, along with our machine learning models, have verified over 80 million financial documents to generate 750 million data labels. All of which go back through our data flywheel for a continuous improvement loop. So throughout this presentation, we'll be talking about each component or step of the data flywheel. So to kick off, this first step is pretty straightforward.
The more users we have, the more data we have. And now I'll hand it over to Sylvia to talk about how we can use that data to build machine learning models. Thanks, Evan. So now that we have a sense of what Oculus does, I'm going to be talking specifically about machine learning at Oculus. Machine learning is an essential step in automating document processing for millions of documents.
But before we talk about machine learning at Oculus, I wanted to start by reviewing what machine learning is. Machine learning refers to algorithms that use lots of data to make predictions. In the last few years, machine learning has become a big part of all of our lives. It's used to predict what shows we'll like, what we should buy, what words we'll type next, and what our Google searches should return.
The recent boom in machine learning comes from a boom in data, big data. As technology has advanced, the amount of data we're able to store has increased exponentially. This graph shows the total amount of data in the world over the last 10 years. As we'll discuss, big data is needed for most machine learning algorithms to work well.
Meanwhile, without machine learning, a lot of data isn't necessarily very valuable. Oculus is like a dream come true for a machine learning engineer because we have a lot of very interesting problems to solve and an ever-growing amount of high-quality hand-labeled data. At Oculus, we use machine learning for many different types of problems, but two of the problems, as mentioned, are particularly well-suited.
The first is that lenders often upload incorrect documents, and also that data entry is error-prone. The first problem to identify document type is what's known as a classification problem. Given a few thousand pictures of cats and dogs, a model can learn to classify an unseen image as a cat or a dog. Image models use image data to classify images, and likewise, text models can use text data to classify text.
An example you might be familiar with is emails which are often classified into spam or not spam. So let's look a little closer into this image classification example. As humans, it might seem easy to distinguish a cat or a dog, but in these photos we can see how tricky the problem is. We have cats with humans, multiple cats, different backgrounds, cropping, different breeds and ages, and the same for dogs.
So this example illustrates why these models need a lot of data, and why the quality of the data is important. Imagine if we only had images of grey cats, or if they were all sitting in the same position. Or if some pictures of cats were mislabeled as dogs. For a model to learn the abstract rules of what a cat looks like, it needs to see diverse data that's representative.
Lots of high-quality data is mandatory for a good model. In the previous slide, we talked about image classification and text classification. At Aqualist, we have a unique advantage in that we have both image data and text data. So we use an algorithm called OCR, which stands for Optical Character Recognition, that can get all of the text from an image.
OCR isn't perfect, but it's a great starting point, and using our large team of over 700 verifiers, we can get perfectly clean data to our clients and for our models. OCR will go into more detail on our unique human-in-the-loop approach shortly. I'm sorry, can you go one back, Kirsten? Okay. So because we have image data and text data, we can build what's called an ensemble model that combines multiple machine learning models into one, allowing us to use the best models for text, something called BERT, and for images efficient.
So we can use those together to get better accuracy than either model gives individually. With ensemble classification algorithms, we can classify documents, we can verify that the correct document type was uploaded, we can verify all required documents have been uploaded, and we can handle trickier problems like when a user uploads all of their documents in one file.
We can then split them up and label each of them. We can also extract key information. So we can use machine learning to pull specific data from the page, such as account number or begin date, so we don't need to do tedious data entry. In addition to high accuracy, our models are dynamic. We can train specific models, so for a specific form or a specific client, or we can leverage a diverse data set.
We continually integrate new data so that the models never fall behind if a new form or template is introduced. And finally, hand labeling allows us to create feedback for our models to further improve accuracy. Now that we've gotten into some of the details of how we use machine learning at Oculus, Kirsten is going to guide us from where we take it from there.
Thanks, Sylvia. That's very exciting stuff in our machine learning space. And what I'll be talking about next is what do we actually do when we have a machine learning model in hand? And how do we optimize the way that humans and computers interact to be as productive as possible? And by productive, we mean as accurate and efficient as possible.
So one of the first questions that we ask ourselves with an ML model in hand is can we make humans more accurate? And our sort of minimum threshold is 99% and we really do reach over 99% accuracy. And one of the things that we want to continue to do is have machine learning leverage that accuracy first. So just taking a look at this image, this is actually two machine learning models plugged in.
The first machine learning model is where is the data on this pay stub? And the second is what does the actual text say in that location? And so what we've tried to do is try to make it as easy as possible for a human to interact with these machine learning models and to correct anything plus get suggestions for things that might be wrong.
So for example, first the location model has a line between the bounding box or location on the left and where the actual text is populated on the right. So a person can basically see does this location look right? And if not, they can correct that location. The second bit is more of the optical character recognition that Sylvia was talking about where we have the actual text in that box and we're reading Tyrion Lannister.
And so what a human can do is actually edit the text right underneath that bounding box on the left so that they don't have to move their eyes from left to right across the page, which makes it easier to see errors and it also makes it easier to correct errors. Last, it makes it faster to correct errors.
So that's the first component is how do we make humans and computers interact? And then the second component is making sure that our accuracy is where we say it is. So the first idea that we needed to sort out as a company was how do we actually measure accuracy between a machine and a human system? And how do those tie out together when it comes to building some kind of automation infrastructure?
So how do we do it? We basically drop the same document kind of over and over and over again to our system and have multiple people answer the document. And by answer the document, I mean extract the data from the document. And we basically compare those labels against each other. We then compare it to someone's expert opinion, which is what we call ground truth in-house.
And that sets us a baseline for what our accuracy should be. Once we get to a point where we have a machine learning model plugged in, we can actually compare how well the machine learning model has done against those human labels and see at what point does the machine learning model beat out the average human. And when we do that, it gives us the ability to do something really cool, which is actually starting to skip some human steps or all of them.
So once we figure out how we do on accuracy, we can actually do a bit of analysis. So here I'm referencing a classification that we have for machine learning, where I know that the first four pages have been historically extremely accurate, and we are also getting extremely high confidence from our machine learning models. So very high precision, very high confidence, and historically very high accuracy basically means that with a four-page document that has these factors successful, we can actually skip those four pages.
So we can actually pull out those four pages and have no humans touch these particular pages for this step. Now the other ones still have to go to a human, but what we're basically doing is starting to pull apart the problem and figuring out what we can automate first so that we can start to move a lot more efficiently for our customers.
So what does this actually look like in terms of data? The result of the flywheel over the course of four years is honestly a series of really pretty graphs. We've reached over 120 document types, and if you look at that bottom graph, we have gone from about an hour and a half from upload to returning data back to about 10 minutes, which basically means that what other companies will do is significantly slower than what we'll do with that high over 99% accuracy.
And basically what we can say is that this efficiency is truly the most disruptive part of our product. The most important part of our product obviously is accuracy because people are trying to decision on this data, but with that efficiency comes the sort of like inability to say no. And this basically just gives us a huge competitive advantage and basically brings me back around to what Evan was talking about, which is more users.
So more and more people are getting convinced that automation is a very, very good way to go for document processing, and our company is playing a pretty critical role in that, which is pretty awesome. Okay. So this is the cool thing about working in FinTech at Oculus. There's a huge gap in loan underwriting efficiency that's largely based on getting data from paper documents.
We can help solve these real-world problems with our technology infrastructure that provides a systematized step-by-step approach to continuous improvement. And overall, our theme is iteration. We improve our ML models regularly. We improve our accuracy constantly. We improve our user interfaces, and we even improve the way that clients interact with our API.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
How a data flywheel makes document automation more accurate over time
Ocrolus product and machine learning leads show how human verifiers, labeled data and careful accuracy measurement let a lending platform automate step by step.
In short
Evan Zhang, Sylvia and Kirsten of Ocrolus argue that the way to automate slow, error-prone lending paperwork is a data flywheel: every processed document becomes labeled data, better models make a better product, and a better product brings in more users and more documents. Their talk explains the machine learning behind document classification and data extraction, and shows how the team measures accuracy so it knows when a machine can safely take over a step from a person.
Key takeaways
Optimizing machine learning
Explain how a data flywheel turns more users and documents into better models and a better product.
Recognize why diverse, representative and correctly labeled training data matters for model quality.
Design human-in-the-loop review screens that make model errors easier and faster to spot and fix.
Measure model accuracy against multiple human labelers and an expert ground truth.
Decide which parts of a workflow are safe to automate using historical accuracy and model confidence.
Why lending still runs on paper
Evan Zhang opens with the problem. A loan application is made of several documents from different sources, and the set changes depending on whether someone applies for a mortgage or a small business loan. Lenders chase missing files by email, check by hand that the right documents were uploaded, retype each data point into internal systems and verify figures with "back of the napkin calculations". Her verdict is blunt: "So overall, it's safe to say that it's time consuming and error prone." (2:28)
The root cause, she explains, is a mismatch: the documents are fundamentally paper based while lenders have moved their workflows online. Ocrolus positions itself as fintech infrastructure that returns structured data on what each document is and what it contains. As the speaker cited, lenders using Ocrolus processed PPP loans for over 3.6 million small businesses, handling over 8.7 million supporting documents.
The data flywheel
Zhang borrows the image of a mechanical flywheel: a heavy wheel that takes a lot of force to start, but once spinning has so much momentum it wants to keep going. The data version works the same way. "And then it's the idea that with the more documents you process, the more data you have, you can build better machine learning models." (4:49)
Human verification is part of the loop, not an afterthought. As the speaker cited, over 700 data verification employees working with the models have verified over 80 million financial documents and generated 750 million data labels, all fed back into continuous improvement.
Why data quality makes or breaks a model
Sylvia, a senior machine learning engineer, steps back to define machine learning as algorithms that use lots of data to make predictions. She notes that data alone is not the prize: "Meanwhile, without machine learning, a lot of data isn't necessarily very valuable." (7:00)
Using the classic cats-and-dogs classification example, she shows how messy real images are: multiple animals, people in the frame, different breeds, ages, crops and backgrounds. If the training set held only grey cats, or some cats were mislabeled as dogs, the model would learn the wrong rules. "Lots of high-quality data is mandatory for a good model." (8:50)
At Ocrolus the same logic applies to documents. OCR extracts text from images, verifiers clean it, and the team builds an ensemble that combines a text model (BERT) with an image model for better accuracy than either alone. That lets them verify document types, check that all required documents are present, split a single uploaded file into labeled documents, and extract fields such as account numbers so nobody has to do tedious data entry.
Designing for humans and machines together
Kirsten, a technical product manager, picks up the question of what to do once a model exists. "And by productive, we mean as accurate and efficient as possible." (11:16) Her example is a pay stub review screen powered by two models: one predicts where the data sits on the page, the other reads the text in that location. A line links each bounding box to its extracted value, and the editable text sits right under the box, so reviewers do not have to move their eyes across the page. The design makes errors easier to see and easier to fix. "Last, it makes it faster to correct errors." (12:48)
Measuring accuracy, then automating step by step
The harder problem is proving accuracy in a system that mixes people and machines. The team sends the same document to several people, compares their labels with each other, and then checks them against an expert answer. "We then compare it to someone's expert opinion, which is what we call ground truth in-house." (13:32) That baseline lets them see when a model beats the average human.
Once that is known, automation becomes targeted rather than all or nothing. Kirsten describes a classification case where the first four pages of a document have historically been extremely accurate and the model reports very high confidence, so those pages skip human review while the rest still go to a person. As the speaker cited, over four years the flywheel took Ocrolus to over 120 document types and cut turnaround from about an hour and a half to about 10 minutes, while keeping accuracy above 99%. "And basically what we can say is that this efficiency is truly the most disruptive part of our product." (15:46)
The closing message ties the loop together: faster, accurate results win more users, which feeds more data back into the wheel. "And overall, our theme is iteration." (16:31)
Watch the talk
Demonstrating the Machine Learning Cycle, Evan Zhang with Sylvia and Kirsten, 17 minutes: https://www.womentech.net/video/demonstrating-machine-learning-cycle
About the speaker
Evan Zhang is a Sr. Technical Product Manager at Ocrolus, building out the Core Client Platform and Suspicious Activity Detection offering, and previously helped launch You Invest on the product team at JP Morgan Chase. She presents with Ocrolus colleagues Sylvia, a Senior Machine Learning Engineer who works on language-based models and privacy research, and Kirsten, a Technical Product Manager who owns the DocClassify and Paystub Capture offerings.
Questions this talk answers
What is a data flywheel?
Why do machine learning models need so much high-quality data?
Why combine image and text models for document classification?
How do you measure accuracy in a combined human and machine system?
When can human review steps be skipped?
Drafted with AI from the talk transcript. Editor review pending.