0:0019:33

Traditional Software vs Machine Learning Software

Filipa Castro 19:33 2022

If two human labelers disagree, don't expect your model to beat them

Give the same tire image to two human labelers and they will not always agree on whether it is damaged. That gap, Continental data scientist Filipa Castro argues, sets the ceiling for any model trained on their labels, and it is one of many ways machine learning products break the planning habits of traditional software. So how do teams set realistic goals and design around errors they cannot eliminate?

Learn to set ML expectations from your data and design apps that make model errors forgivable

Watch on YouTube

In short

Filipa Castro, a data scientist at Continental, explains why machine learning products (software 2.0) are harder to plan than traditional software, because they learn from data and behave probabilistically. Through a tire inspection case study and an activity-detection app, she shows how deep data exploration, human-label baselines and error-tolerant design help teams set realistic expectations and protect the user experience.

“So while in software 1.0, it's a deterministic behavior of the product here, it's a probabilistic behavior because it's a model that is answering or outputting something.”
“So to sum up, I would say that it's very important to dive deep into the data understanding and show what is feasible, what is not according to the data we have.”

What you will learn

  • Explain the difference between deterministic software 1.0 and probabilistic, data-trained software 2.0 to stakeholders.
  • Plan for an ML lifecycle that loops back between business understanding, data preparation, modeling and deployment.
  • Use data distributions to show stakeholders which cases a model can and cannot be expected to handle.
  • Set realistic performance targets by measuring agreement between human labelers.
  • Design ML features so errors are reversible and confirmed by users, feeding corrections back into the model.

Key takeaways from the speaker

  • Learn to identify and recognize the main challenges of building ML software, in contrast to traditional software
  • Learn how to deal with the uncertainty of building ML software, especially get some tips on how to manage expectations and set realitistic goals for the project
  • Find which type of approaches big companies can follow to manage ML projects, from consulting to prototyping and deployment of real solutions
  • Learn about the main differences on designing a ML product vs a traditional software product, and understand the impact in the user experience
  • Get introduced to data protection and model bias issues, which need to be intensively discussed and taken into account when building ML software

Details

Level
Intermediate
Format
Talk
Technical depth
Introductory technical
For
Mid-career, Senior individual contributor, People manager
Length
19:33
Published
July 2022

Questions this talk answers

Why is machine learning software harder to plan than traditional software?

Traditional software follows explicit instructions and behaves deterministically, while ML software learns from examples and produces probabilistic outputs. Its development cycle also involves many steps that often loop back to earlier ones.

How can a team set realistic expectations for an ML model?

Explore the data distribution to show which defects or cases are missing, break the broad problem into smaller priorities, and use agreement between human labelers as a performance baseline.

How should an app be designed when its model will sometimes be wrong?

Prefer designs where errors are transparent and reversible, such as detecting events after recording and asking the user to confirm them, rather than acting irreversibly in real time. User confirmations can later improve the model.

How does a corporate data science team engage with business units on AI projects?

At Continental the team acts as a pool of data experts: it can consult, explore data for potential, build an MVP prototype using methods such as Scrum, or deliver an end-to-end product into production.

How much data does an image or activity recognition model need?

Typically in the region of thousands of examples, around 1,000 to 3,000; about 1,000 is a good reference for a binary problem, and multi-class problems need more.

About this talk

Smart digital products, such as Alexa and autonomous cars, are taking over our lives. The development of such products, which are based on machine learning (ML) and artificial intelligence (AI) features, brings new challenges to the table.

In contrast to traditional software, the results of a ML model are not deterministic. Thus, managing expectations and setting clear requirements is key for the project kick-off. The expected performance of the ML models must be realistic and well defined. State of the art methods and human performance should be used as baselines.

Just as humans do, smart products might fail sometimes. It’s then essential to carefully design these applications, so that errors are transparent, well understood and tolerated by the user.

More than ever, ethics is a real concern. Topics such as data protection need to be intensively discussed and taken into account. Model bias needs to be measured and mitigated.

In this talk, we’ll go through real use cases that prove how challenging it is to build ML products. This will include tips and best practices for dealing with such challenges.

Topics

Related talks

Browse the library