Seeing the unseen: inferring unobserved information from multi-modal data
Adriana Romero Soriano16:342021
When AI fills in what it cannot see, each guess could mean a different diagnosis
Show a model a single photo of a bottle, or a fast, low-resolution knee MRI, and it has to guess everything it cannot see. In medical imaging, those guesses could point to different diagnoses. Adriana Romero Soriano, a research scientist at Facebook AI Research and adjunct professor at McGill University, asks how machines can narrow the many possible answers down to the right one.
See how touch data and active acquisition cut uncertainty in 3D reconstruction and MRI
Adriana Romero Soriano, a research scientist at Facebook AI Research and adjunct professor at McGill University, explains why partial observations leave machine learning models uncertain about what they cannot see. She presents two ways to reduce that uncertainty: combining vision and touch to reconstruct 3D object shapes, and actively acquiring the most valuable measurements to produce faster, personalised MRI reconstructions.
“As humans, we never fully observe the world around us. And yet we are able to build remarkably useful models of it.”
“Therefore, vision and touch provide complementary global and local information about an object.”
What you will learn
Explain why partial observations create uncertainty whenever a model must infer missing information.
Recognize how vision supplies global context while touch supplies precise local structure for 3D shape reconstruction.
Describe a visual-tactile pipeline that combines touch charts, vision charts and graph neural network mesh deformation.
Understand how active acquisition lets an MRI system pick the next most valuable measurement and personalise scans per patient.
Apply the idea of complementary data and targeted data collection to reduce uncertainty in your own models.
Key takeaways from the speaker
Partial observations naturally induce uncertainty when attempting to recover full observations of the world.
With the current technology, we are able to build reasonably good models of the world, which infer unobserved information from the observed one.
Leveraging complementary data modalities helps reduce the uncertainty induced by the partial observations.
Equipping the models with active acquisition also helps reduce this uncertainty.
Building accurate models of the world unlocks the potential to many applications.
Details
Level
Intermediate
Format
Talk
Technical depth
Practitioner
For
Senior individual contributor, Mid-career
Length
16:34
Published
June 2021
Questions this talk answers
Why do partial observations make machine learning predictions uncertain?
Because there are many possible ways to fill in the missing information, each leading to a different full observation. A partial digit, for example, could be completed as a seven, a one or a four.
Why combine vision and touch for 3D shape reconstruction?
Vision gives a strong global sense of an object's shape and position but suffers from occlusion and bas-relief ambiguity. Touch gives highly detailed local structure and positional grounding, so the two are complementary.
How was the visual-tactile model trained and built?
The team simulated grasps of thousands of objects with an Allegro hand fitted with vision-based touch sensors, then predicted touch charts, initialised vision charts as a sphere and deformed all charts with graph neural networks while keeping touch charts fixed.
How can MRI scans be made faster without losing quality?
By collecting fewer measurements and using a reconstruction network plus a second network that scores which missing measurement is most valuable to acquire next, until the image reaches high fidelity.
What does active acquisition mean for patients?
The number of measurements adapts to each patient, small in some cases and larger in others, which the speaker frames as a way of personalising MR scans.
About this talk
As humans we can never fully observe the world around us and yet we are able to build remarkably useful models of it from our limited sensory data. Machine learning systems are often required to operate in a similar setup, that is the one of inferring unobserved information from the observed one. Partial observations entail data uncertainty, which may hinder the quality of the model predictions. In this talk, we will discuss two strategies to mitigate this problem: (1) leveraging the complementarity of different data modalities, and (2) actively acquiring additional information from the same data modality.
Personalising MRI scans with active acquisition (75 s)
All right. So hi, everyone. Welcome to this session of WomenTech. I hope you're enjoying the conference. My name is Adriana Romero Soriano and I'm a research scientist at Facebook AI Research. I also hold an appointment as an adjunct professor at McGill University. And today I'm going to briefly discuss the research that I've been doing over the past few years on the topic of seeing the uncertainty.
In particular, I'll be discussing how to infer unobserved information from multi-modal data. As humans, we never fully observe the world around us. And yet we are able to build remarkably useful models of it. Let's see a very simple example of this. Here we have a partial observation and we know that this partial observation corresponds to a digit.
There are many ways in which we could fill in this partial observation such that it would lead to fully observed, but different digits. For example, some of you might fill in the partial observation in such a way that it becomes a seven. Some others might fill in such a way that it becomes a one or even a four. And so partial observations naturally induce uncertainty when attempting to recover the full observation due to this number of possible ways in which we can fill in the missing information.
And so in order to improve the models that we build, we reduce the many ways in which we can fill in the missing information by acquiring new observations. These new observations can potentially complement the previous information that we already had. For example, we improve our understanding of the outer space by acquiring new observations. Similarly, we gather information from underwater by taking deep dives.
And we also improve our understanding of our own inner space or the one of other organisms on Earth by exploring new observations. And by studying the newly acquired information, we increase our understanding and are able to build models of what once was invisible to us. Similarly, if we go back to our simple example in the world of the digits, if we collect additional information, say we have this additional stroke shown now in the slide in the partial observation, then the number of possible ways in which we can fill in this missing information reduces.
And so in this particular case, we reduce the uncertainty in such a way that the most likely digit to be recovered is digit number seven. And so we keep improving our models of the scene and the unseen by acquiring and leveraging information to complement the previous information that we already had. Machine learning models are often required to operate in a similar set up.
That is the one where they have to infer observed information from the observed one. Let's see a couple of examples of such systems. So one might have this image. It's an image of an object, a bottle in particular, that is being handheld. And one question that we could ask is, can we recover the full 3D shape of the object given this partial view on the object?
So here the partial observation would be the single view image of the object. And the full observation that we would want to recover is its full 3D shape. And so because we only have a single view of the object, there might be different ways in which we can fill in the information that we don't see in order to infer the 3D shape.
As a second example, one could show you this image. This image corresponds to an MR image of a knee, but it has a very low resolution. So one possible question would be, what can we do or how can we address the problem of making this high resolution, this image high resolution? So what would the high resolution version of this image look like?
In this talk, I will not only answer these two previous questions, but also present approaches that allow us to get improved results. Let's start with the first question that is, can we infer the 3D shape of the object in view? So this is the problem of 3D shape reconstruction. And this problem has been tackled in most cases by the computer vision community from a single or multiple view images.
In this work, we will focus on the single view formulation. So the goal of 3D shape reconstruction is to infer the whole 3D shape of the object from a single view image of the object. And as mentioned before, because we only have a single view and so there are parts of the object that we don't see and that are occluded.
And there might be different ways in which we might fill in this 3D shape. And current 3D shape reconstruction methods result in relatively successful reconstructions. If we consider that we have a single view of the chair depicted on the slide, we see that the current model, the one that I'm using here is a state of the art model, obtains a relatively high quality reconstruction.
However, as we can observe, there are still some missing details such as the ones in the top back of the chair. However, in different settings, if we consider more challenging scenarios where the ambiguities may arise from the unseen parts, there might be many more ways in which we can fill in the missing information about the shapes. And in this case, we see, for example, the couch or the car, we don't have the full idea of how these objects look like in practice.
And so inferring the full 3D shape becomes more challenging. In order to limit the possible ways in which we could fill in this missing information and recover better 3D object shapes, we explored how to leverage the complementarity of vision in tactile data modalities. So let me try to explain why these two modalities. So if we start with vision, although we only have a single view image of the object that we want to reconstruct, vision already provides a strong global context for the object in view.
That is, it provides us with a good grasp on the general shape of the object and also about its rough position in space. However, vision has some ambiguities, which may hinder the quality of the reconstructed objects. For example, vision suffers from occlusions and from bas-relief. If we think of tactile signals, on the other hand, they provide highly detailed information about the local structure of an object being touched.
They also provide important positional information about the object since this object will be very well grounded within the reference frame of the hand touching it. However, touch information is also limited in that the local information that it provides will very often fail to extrapolate well to global object understanding, even if we have strong object priors. Therefore, vision and touch provide complementary global and local information about an object.
In order to explore the complementarity of these two kinds of signals of the visual and the tactile signals, we started by building a visual tactile dataset of object grasp interactions. To do so, we simulated object grasp interactions using an allegro hand, which is equipped with high resolution vision based touch sensors on each of its three fingers and thumb.
We used objects from publicly available computer vision data sets, given their ubiquitous use in research. And once we performed the object grasp interactions, we obtained an image of the interaction, which we use as a visual signal and depicted on the slide as the vision signal. And we also obtained four haptic signals, four tactile signals as depicted on the slide.
And we performed these object grasp interactions from thousands and thousands of objects. So this led to our simulated data set. And once we had access to the simulated data set, we designed an approach to fuse visual and tactile signals for 3D shape reconstruction. In this approach, we represent a 3D object with a collection of the joint mesh surface elements called charts and here depicted as the triangles composing the sphere.
And some of the charts are reserved for the tactile signals and some of the other charts are reserved for the visual information. In the first step of our model of our pipeline, we predict the touch charts from the touch recordings. For that, we use a neural network to predict the position of the vertices composing each chart in the touch charts.
Then we initialize the vision charts as a sphere and we project the visual signal, so the image of the object, onto all charts. And then we feed all the charts, the vision and touch charts into a mesh deformation process. This mesh deformation process leverages recent advances in graph neural networks. And at the end of this process, we enforce touch consistency.
That is that at the end of the deformation process, the touch charts are copy pasted back to their original position. And so we use the visual information to fill in the blanks and recover the full 3D shape of the object. As a result, we obtain a global prediction of the deformed charts representing the object. Let me now show you a visual result depicting what the proposed approach leveraging vision and touch can achieve.
On the left column, we see the input vision signal of the bottle. Then from left to right, we have reconstructions from touch only with a single grasp, followed by vision and touch with a single grasp, and then vision and touch, but with increasing number of grasps. So this means that we perform several object grasp interactions in order to obtain our final reconstruction.
The top results depict the predicted shape and the bottom result visualizes the error on top of the predicted surface. From these experiments, we visually confirm that the model gains additional insight into the nature of the object by touching new areas on its surface. And if we look into the details of this, if we see in the case of one grasp without vision, we see that there is quite a bit of error around the object surface highlighted in red.
As soon as the vision signal kicks in, so we have one grasp plus vision, we see that the error on the surface is mainly reduced around the top of the bottle. And as we keep increasing the number of grasps performed, the error on the surface of the object is further decreased. Moving on to the second question that I posed in the beginning of the presentation, let's see what does the high resolution version of this MR imaging look like.
And before we dig into this problem, just let me give you a brief overview on how MRI works. MRI scanners work by successively acquiring measurements in the frequency domain. And from these acquired measurements, we can reconstruct an image using the inverse Fourier transform. And generally speaking, the more measurements we take with the MR machine, the better the resulting image quality.
However, taking many measurements is slow. And so a simple way to accelerate the MRI acquisition process is to collect less measurements. For example, we could tell the MR machine to follow the sampling trajectory depicted under the MR machine on the slide, where each of the vertical lines of the white vertical lines represents one measurement to be acquired.
So if we acquire those measurements from those acquired measurements, we can then apply the inverse Fourier transform and obtain a reconstruction of the image. As you see on the slide, this results in low quality images because we have limited the number of measurements that we have acquired. And so there is a whole field devoted to improving the quality of MR image reconstruction by leveraging machine learning and more recently deep learning.
And therefore, our goal in this case is to infer the unobserved measurements from the observed one, ensuring high fidelity image reconstructions. Once again, there might be many ways in which we can fill in these missing measurements that could potentially lead to different diagnosis. Hence the importance of inferring the right ones. We tackle the MRI reconstruction problem from a perspective of active acquisition as a way to reduce the uncertainty by acquiring new measurements.
And as a result, we hope to increase the quality of the reconstructed images in the active acquisition scenario. If we believe that the quality of the reconstructed image is not good enough, we would like to keep acquiring additional measurements in order to ensure that we end up with a high fidelity image reconstruction. This means that in some cases, the number of required measurements will be small.
And in some other cases, the number of required measurements will be high. The number of required measurements will be high. Note that designing active acquisition strategies allows us to adapt the measurements taken to each patient. And so we can think of this as a way of personalizing MR scans. Here is a brief overview on how to learn these active acquisition trajectories.
So we start with a reconstruction model, which takes as input partial measurements taken by the MR machine and which infers the missing ones. In order to reduce the uncertainty or the many ways in which we can fill in the missing measurements, we design a second neural network, which takes as input the output of the reconstruction network. This second network outputs a score for each reconstructed measurements and the score tells us how good it is to acquire this measurement in the long run.
In practice, we choose to acquire the measurements which have the highest added value according to the second network. And after acquiring it, we can go back to the first network with the new information and predict the new image reconstruction. I will now show a short video on how this active acquisition that we have learned works. So from left to right, you will see the active acquisition trajectory.
As we keep adding new measurements, new vertical lines will appear. Then you have the ground truth image and followed by the reconstructed image and its associated error maps. So as we see in the video, as we keep adding new measurements, the errors are decreasing. The quality of the reconstructed image is improving and it's getting closer and closer to the ground truth, which is the first image showing.
From the left. So to wrap up this presentation, I started the talk arguing that partial observations entail data uncertainty and that there are many possible ways in which we can fill in the missing information. We then saw two examples of machine learning systems which attempt to infer unobserved information. And although current models are able to provide somewhat useful reconstructions of what is unseen, they might be hindered by the uncertainty that arises from it.
And so to address this issue, we discussed two different strategies to mitigate the number of possible ways in which we can fill in the missing information, which lead to improved results. In particular, we discussed the ability of current models to infer the unobserved information and we saw that this could be improved by leveraging complementary data modalities, which was the case of leveraging vision and touch to improve 3D reconstruction systems.
And we also discussed how to equip the models with active acquisition capabilities in order to determine the amount of observations needed to recover the missing information with high fidelity in the context of MRI reconstruction. I'll wrap up the presentation here. I'll stop sharing my slides and take any questions that you might have on the chat.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
How can AI recover what it cannot see?
Adriana Romero Soriano shows how complementary data and smart data collection reduce uncertainty in 3D shape reconstruction and MRI.
In short
Machine learning systems, like people, rarely get a complete view of the world, and every gap leaves room for many possible answers. Adriana Romero Soriano, a research scientist at Facebook AI Research and adjunct professor at McGill University, argues that this uncertainty can be reduced in two ways: by combining complementary data modalities, such as vision and touch, and by actively acquiring the most useful new measurements, as in accelerated MRI.
Key takeaways
Partial observations naturally induce uncertainty when attempting to recover full observations of the world.
With current technology, we are able to build reasonably good models of the world, which infer unobserved information from observed information.
Leveraging complementary data modalities helps reduce the uncertainty induced by partial observations.
Equipping models with active acquisition also helps reduce this uncertainty.
Building accurate models of the world unlocks the potential of many applications.
Partial observations mean many possible answers
Romero Soriano opens with a simple observation about human perception: "As humans, we never fully observe the world around us. And yet we are able to build remarkably useful models of it." (0:30)
To make the problem concrete, she shows a fragment of a handwritten digit. Some viewers would complete it as a seven, others as a one or even a four. That ambiguity is the heart of the talk: "And so partial observations naturally induce uncertainty when attempting to recover the full observation due to this number of possible ways in which we can fill in the missing information." (0:54)
Adding a single extra stroke changes the picture, making seven the most likely answer. She links this to how science advances more broadly, from exploring outer space to deep-sea dives: "And so we keep improving our models of the scene and the unseen by acquiring and leveraging information to complement the previous information that we already had." (2:28) Machine learning models face the same setup, and she introduces two examples: recovering the full 3D shape of a handheld bottle from one image, and producing a high-resolution version of a low-resolution knee MRI.
Vision sees the whole, touch feels the detail
Single-view 3D shape reconstruction already works reasonably well. Using a state of the art model, a chair can be reconstructed with relatively high quality, though details such as the top of the chair back are missing. For harder objects, such as a couch or a car where much is hidden from view, the number of plausible shapes grows.
Romero Soriano's answer is to pair two senses. Vision provides strong global context about an object's general shape and rough position, but it suffers from occlusion and bas-relief ambiguity. Touch provides highly detailed local structure and is well grounded in the reference frame of the hand, yet on its own it rarely extrapolates to the whole object. As she puts it: "Therefore, vision and touch provide complementary global and local information about an object." (6:46)
To test this, her team simulated grasps of thousands of objects from public computer vision datasets using an Allegro Hand fitted with high-resolution, vision-based touch sensors on each of its three fingers and thumb. Each grasp produced one image and four tactile signals.
How the model fuses the two signals
The approach represents an object as a collection of mesh surface elements called charts. Some charts are reserved for touch and others for vision. A neural network first predicts the touch charts from the touch recordings. The vision charts start as a sphere, the image is projected onto all charts, and a mesh deformation process based on graph neural networks shapes them. At the end, touch consistency is enforced so touch charts return to their original positions while visual information fills in the blanks.
The results, shown as error maps on a reconstructed bottle, tell a clear story. Touch alone with one grasp leaves substantial error; adding vision cuts error around the top of the bottle; more grasps reduce it further. "From these experiments, we visually confirm that the model gains additional insight into the nature of the object by touching new areas on its surface." (10:18)
Personalised MRI through active acquisition
The second problem moves to medical imaging. MRI scanners acquire measurements in the frequency domain, and images are reconstructed with the inverse Fourier transform. More measurements mean better images, but taking many measurements is slow. Collecting fewer measurements speeds things up but produces low-quality images, which is why a whole field uses machine learning and deep learning to improve reconstruction.
The stakes here are higher than a misshapen bottle: "Once again, there might be many ways in which we can fill in these missing measurements that could potentially lead to different diagnosis. Hence the importance of inferring the right ones." (12:30)
Her team frames the problem as active acquisition. A reconstruction network takes partial measurements and infers the missing ones. A second network scores each reconstructed measurement by how valuable it would be to acquire in the long run, and the system acquires the highest-value one before reconstructing again. The process continues until the image is good enough, so some patients need few measurements and others need more. "Note that designing active acquisition strategies allows us to adapt the measurements taken to each patient. And so we can think of this as a way of personalizing MR scans." (13:23) In the demonstration video, errors shrink and the reconstruction moves steadily closer to the ground truth as measurements are added.
Two strategies, one principle
Romero Soriano closes by tying the examples together. Current models already produce somewhat useful reconstructions of what is unseen, but uncertainty holds them back. Leveraging complementary modalities, as with vision and touch, and equipping models to decide what to observe next, as with MRI, both reduce the number of plausible ways to fill in missing information and lead to better results.
Watch the talk
"Seeing the unseen: inferring unobserved information from multi-modal data" by Adriana Romero Soriano, 17 minutes: https://www.womentech.net/video/adriana-romero-soriano-seeing-unseen-inferring-unobserved-information-multi-modal-data
About the speaker
Adriana Romero Soriano is a Research Scientist at Facebook AI Research and an Adjunct Professor at McGill University, whose research focuses on models and algorithms that learn from multi-modal and real-world data, reason about conceptual relations and recognize their uncertainties. She completed her postdoctoral studies at Mila, advised by Prof. Yoshua Bengio, and received her Ph.D. from the University of Barcelona with a thesis on assisting the training of deep neural networks for computer vision.
Questions this talk answers
Why do partial observations make machine learning predictions uncertain?
Why combine vision and touch for 3D shape reconstruction?
How was the visual-tactile model trained and built?
How can MRI scans be made faster without losing quality?
What does active acquisition mean for patients?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Graph neural networks
Inverse Fourier transform
Active acquisition
Cite this talk
Adriana Romero Soriano. "Seeing the unseen: inferring unobserved information from multi-modal data." WomenTech Network, 2021. https://www.womentech.net/video/adriana-romero-soriano-seeing-unseen-inferring-unobserved-information-multi-modal-data