0:0016:34

Seeing the unseen: inferring unobserved information from multi-modal data

Adriana Romero Soriano 16:34 2021

When AI fills in what it cannot see, each guess could mean a different diagnosis

Show a model a single photo of a bottle, or a fast, low-resolution knee MRI, and it has to guess everything it cannot see. In medical imaging, those guesses could point to different diagnoses. Adriana Romero Soriano, a research scientist at Facebook AI Research and adjunct professor at McGill University, asks how machines can narrow the many possible answers down to the right one.

See how touch data and active acquisition cut uncertainty in 3D reconstruction and MRI

Watch on YouTube

In short

Adriana Romero Soriano, a research scientist at Facebook AI Research and adjunct professor at McGill University, explains why partial observations leave machine learning models uncertain about what they cannot see. She presents two ways to reduce that uncertainty: combining vision and touch to reconstruct 3D object shapes, and actively acquiring the most valuable measurements to produce faster, personalised MRI reconstructions.

“As humans, we never fully observe the world around us. And yet we are able to build remarkably useful models of it.”
“Therefore, vision and touch provide complementary global and local information about an object.”

What you will learn

  • Explain why partial observations create uncertainty whenever a model must infer missing information.
  • Recognize how vision supplies global context while touch supplies precise local structure for 3D shape reconstruction.
  • Describe a visual-tactile pipeline that combines touch charts, vision charts and graph neural network mesh deformation.
  • Understand how active acquisition lets an MRI system pick the next most valuable measurement and personalise scans per patient.
  • Apply the idea of complementary data and targeted data collection to reduce uncertainty in your own models.

Key takeaways from the speaker

  • Partial observations naturally induce uncertainty when attempting to recover full observations of the world.
  • With the current technology, we are able to build reasonably good models of the world, which infer unobserved information from the observed one.
  • Leveraging complementary data modalities helps reduce the uncertainty induced by the partial observations.
  • Equipping the models with active acquisition also helps reduce this uncertainty.
  • Building accurate models of the world unlocks the potential to many applications.

Details

Level
Intermediate
Format
Talk
Technical depth
Practitioner
For
Senior individual contributor, Mid-career
Length
16:34
Published
June 2021

Questions this talk answers

Why do partial observations make machine learning predictions uncertain?

Because there are many possible ways to fill in the missing information, each leading to a different full observation. A partial digit, for example, could be completed as a seven, a one or a four.

Why combine vision and touch for 3D shape reconstruction?

Vision gives a strong global sense of an object's shape and position but suffers from occlusion and bas-relief ambiguity. Touch gives highly detailed local structure and positional grounding, so the two are complementary.

How was the visual-tactile model trained and built?

The team simulated grasps of thousands of objects with an Allegro hand fitted with vision-based touch sensors, then predicted touch charts, initialised vision charts as a sphere and deformed all charts with graph neural networks while keeping touch charts fixed.

How can MRI scans be made faster without losing quality?

By collecting fewer measurements and using a reconstruction network plus a second network that scores which missing measurement is most valuable to acquire next, until the image reaches high fidelity.

What does active acquisition mean for patients?

The number of measurements adapts to each patient, small in some cases and larger in others, which the speaker frames as a way of personalising MR scans.

About this talk

As humans we can never fully observe the world around us and yet we are able to build remarkably useful models of it from our limited sensory data. Machine learning systems are often required to operate in a similar setup, that is the one of inferring unobserved information from the observed one. Partial observations entail data uncertainty, which may hinder the quality of the model predictions. In this talk, we will discuss two strategies to mitigate this problem: (1) leveraging the complementarity of different data modalities, and (2) actively acquiring additional information from the same data modality.

Topics

Related talks

Browse the library