0:0016:57

Demonstrating the Machine Learning Cycle

Evan Zhang 16:57 2021

How a data flywheel cut document turnaround from 90 minutes to 10

Loan applications still arrive as paper-based documents, and lenders type each data point by hand into their systems. A product and machine learning team from Ocrolus, including Senior Technical Product Manager Evan Zhang, shows how over 700 human verifiers and machine learning models feed one loop that holds accuracy above 99%. The open question is when a machine can safely take a human out of that loop.

See how ground-truth testing shows a team when its model beats the average human labeler

Watch on YouTube

In short

Evan Zhang and her Ocrolus colleagues Sylvia and Kirsten explain how a data flywheel automates lending document processing: processed documents become labeled data, better machine learning models and a better product that attracts more users. The talk covers document classification and extraction, why diverse high-quality data matters, how human verifiers and models work together, and how ground-truth accuracy measurement shows when human steps can safely be skipped.

“Lots of high-quality data is mandatory for a good model.”
“And then it's the idea that with the more documents you process, the more data you have, you can build better machine learning models.”

What you will learn

  • Explain how a data flywheel turns more users and documents into better models and a better product.
  • Recognize why diverse, representative and correctly labeled training data matters for model quality.
  • Design human-in-the-loop review screens that make model errors easier and faster to spot and fix.
  • Measure model accuracy against multiple human labelers and an expert ground truth.
  • Decide which parts of a workflow are safe to automate using historical accuracy and model confidence.

Key takeaways from the speaker

  • Optimizing Machine Learning

Details

Level
Intermediate
Format
Panel
Technical depth
Introductory technical
For
Early career, Mid-career
Length
16:57
Published
June 2021

Questions this talk answers

What is a data flywheel?

It is the idea that the more documents you process, the more data you have to build better machine learning models. With human verification, that produces a better product, which brings more users, more documents and more data.

Why do machine learning models need so much high-quality data?

Real examples vary widely in background, cropping, breed and position, so a model needs diverse, representative and correctly labeled data to learn abstract rules. Mislabeled or narrow data, such as only grey cats, teaches the wrong thing.

Why combine image and text models for document classification?

Because Ocrolus has both image data and OCR text, it builds an ensemble that combines a text model (BERT) with an image model. Together they give better accuracy than either model alone.

How do you measure accuracy in a combined human and machine system?

The same document is sent to several people, their labels are compared with each other and with an expert's ground truth to set a baseline. The model's output is then compared with those human labels to see when it beats the average human.

When can human review steps be skipped?

When pages have historically very high accuracy and the model shows very high precision and confidence, those pages can bypass human review for that step. The remaining pages still go to a person.

About this talk

The Machine Learning Fly Wheel: Optimizing the process of creating a set of training for continuous machine learning

Topics

Related talks

Browse the library