Computer Vision for Augmented Reality & Virtual Reality
Tilottama Goswami24:242023
Without computer vision, augmented and virtual reality cannot exist
A digital ring lands on a finger the camera has just located, a dinosaur appears in a garden, and a treadmill tilts as an on-screen guide leads the climb, yet every effect depends on a machine first seeing the world. Computer vision researcher Professor Tilottama Goswami, with 23 years across academia, research and industry, unpacks what sits underneath. Which vision tasks must run before any digital object can sit convincingly in a real scene?
Learn the four types of AR, the vision tasks behind them and the tools to start building
Tilottama Goswami, Professor at Vasavi College of Engineering, gives a broad introduction to how computer vision underpins augmented and virtual reality. She defines the metaverse, VR and the four types of AR, walks through the computer vision tasks and SLAM that make immersive content possible, and shows applications and development tools, arguing that AR and VR cannot exist without computer vision.
“After that, I'll provide the list of the fruits. So the whole concept of the computer vision is see, observe, understand, and then act.”
“Now coming to the augmented reality, it is basically your real environment and the digital content is overlaid at the top of it to get an interactive experience. So it is augmenting the real world.”
What you will learn
Distinguish the metaverse, virtual reality, augmented reality and extended reality, and name the three pillars of presence.
Identify the four types of augmented reality: marker-based, markerless, projection-based and superimposition-based.
Map core computer vision tasks such as detection, tracking, segmentation and feature extraction to what AR and VR software must do.
Explain how SLAM lets markerless AR track a user continuously and build a virtual map of the environment.
Choose a starting point among common AR and VR development tools such as Unity, Vuforia, ARKit, ARCore, Wikitude and OpenCV.
Key takeaways from the speaker
The talk is about the emerging technology of the future
The session will discuss potential application domains of AR/VR Technology
The talk will focus on how Computer vision along with virtual and augmented reality work together to build a system for achieving sophisticated interactive content.
The session briefs about the AR/VR Development environment
Details
Level
Beginner
Format
Tech talk
Technical depth
Introductory technical
For
Student, Early career
Length
24:24
Published
June 2023
Questions this talk answers
What is the metaverse?
It is a virtual universe, beyond the universe, where humans and avatars coexist as a community. It rests on three pillars: the feeling of presence of oneself, of the space around you and of others.
What types of augmented reality are there?
Goswami describes four types: marker-based, markerless, projection-based and superimposition-based. She also shows a GPS location-based example from tourism.
Why does AR and VR depend on computer vision?
AR and VR software must first classify, detect, recognize, track, segment and extract features from a scene before any digital content can be placed. In her words, computer vision is the fundamental of it.
How does tracking differ between marker-based and markerless AR?
Marker-based AR tracks a fixed user location by looking for markers, while markerless AR continuously tracks the user wherever they go. With SLAM, that continuous tracking can also build a virtual map of the environment.
Which tools are used to develop AR and VR applications?
She names Unity and Vuforia as popular choices, ARKit from Apple, ARCore from Google, Wikitude, and OpenCV for the computer vision side.
About this talk
The objective of the talk "CV for AR/VR" is to give a broad overview of the role of computer vision in Augumented Reality and Virtual Reality. Various E-Commerce industries are paving ways towards mass adoption of AR and VR technologies to add the real-time visual effects to their products to provide better customer experience, improve brand perception and improve sales. Participants will learn the basic concepts of the technologies and the tools involved. The talk will focus on how Computer vision and virtual reality work together to build a system for achieving sophisticated interactive content.
An AR treadmill that simulates a mountain trek (67 s)
Namaste. Greetings, one and all. I am Tilottama Goswaami, working as a professor in Department of Information Technology, Vasavi College of Engineering from Hyderabad, India. I have overall 23 years of experience, including industry, academics and research. My areas of interest are computer vision, image processing and machine learning. I feel honored to be a part of WomenTech Global Conference 2023. Thank you for giving me this opportunity. The title of my talk is Computer Vision for Augmented Reality and Virtual Reality. The world of today is changing at a rapid pace. Advances in technology is helping us to explore and interact with the digital environments in ways that were unheard of few years ago. The potential of revolutionizing how we interact with the world around us is tremendous. So let's take a brief dive into the key concepts
surrounding this topic. I will walk through the basic terminology and the concepts to get you familiar with the subject. So now let's get started. The topics which related to this title is metaverse, virtual reality, augmented reality, computer vision, computer vision algorithms which will be used for the AR and the VR, the applications of the virtual reality and augmented reality and very brief about the development environment.
So in this figure we are moving from reality to the virtual reality. So there were sensory organs, the sight, taste, smell, touch, hearing, all of these can be felt in a simulated environment and that we call as immersive environment. So the reality which we are getting through these gadgets, smartphones, haptics, the VR goggles, speech recognition, interacting with the machine, that gives us a feeling of a reality which is an augmenting the universe in which we are now.
So that is a virtual world. So that is a virtual world. What is metaverse? Meta means beyond and verse is universe. So something beyond universe. And we mean by the virtual universe where the humans and avatars co-exist as a community. So that is a virtual world. And the presence is felt and the presence is felt by communicating with each of the stakeholders in this ecosystem that is the superset of the virtual reality.
There are three pillars of the metaverse. The first one is feeling of the presence of oneself. Second is feeling of the spatial presence that is of the environment. The third one is creating an impression of presence of the others. It is not an isolated world where many of the stakeholders like avatars, humans are coexisting. So let us see here. The types of metaverse are augmented reality, virtual worlds and extended reality.
The augmented reality is basically over playing the digital into the user's own world. The virtual world is totally immersive into the 3D environment and the extended reality is the, including the virtual reality, augmented reality and mixed reality. So the more we, the word is metaverse, which is the super, the superset where we are going beyond the universe.
That is the virtual universe where humans, avatars co-exist and the presence is also felt. So there are three pillars of the metaverse. The one is feeling of oneself as a jumbo or a jami or a ruta or a goswami. And then there is a feel of spatial presence that is an immersive experience. And though I may not be reality, in reality, I may not be venturing out in this valley, but I may have a virtual feeling of being in the valley.
And the third one is making presence of the others by communicating with them, whether it is avatar or whether it is a human. So the virtual reality, it is a computer generated simulation and it gives you an immersive experience and the user can move around in all the 360 degrees, can manipulate the objects. So there are two types of virtual reality.
One is the non-immersive, which is used for training simulation, where we are focusing only on a part of the machine where we want to study and we want to know the, understand the know-how of that equipment. The second one is the fully immersive, where it is a gaming and entertainment. It is not only the goggles, but we will be having many gadgets like haptics, sound and some sensors to make us feel the maximum number of sensory organs can be felt using this fully immersive virtual reality.
Now coming to the augmented reality, it is basically your real environment and the digital content is overlaid at the top of it to get an interactive experience. So it is augmenting the real world. Here the real world is having, so there are augmented realities having four types, marker-based, markerless, projection-based and superimposition-based. So in this, if you see, this is the marker and hence that marker is replaced by a digital information.
Here on this particular location of the garden, the dinosaur is getting overlaid. Here the bone structure of the arm is overlaid after the arm has been identified or detected. So just projecting into it. The superimposition one creates an illusion of mixed reality. So you are having one template like a Snapchat and then you have a mixed reality where the virtual object is placed on the top of the real scenes.
So the question was, any guess on what type of AR is A, B, C, and D? In B, it is here, it is a marker. So the B is a marker-based. Whereas A, it is just identifying the wrist and putting the overlay of the digital watch. That how this watch is a marker-based, that how this watch is suiting or not.
So this is a marker-less. Okay. This one is projecting a dinosaur on the top of this particular diagram. And this one is basically your superimposition of your digital at the top of the real world. So this is what is some guess on some more examples I gave. I'll go to the chat box and just check. And kind of check.
Is it clear now? I gave a quick recap. Can I proceed? Oh, thank you, thank you, thank you. So I'll again share. I don't know whether it is. Okay. So this is what one example we have done. So now coming to the computer vision. The topic is computer vision for AR and VR. So computer vision is actually a part of a field of artificial intelligence.
And it allows us to see, observe, understand, and make an informed decision. So if you see this figure, the task is there are some fruits on the plate. And I want to list down all the fruits. If as a human being, I see the fruits, I identify and recognize the fruits, my brain processes them by recognizing them.
And finally, I list it out. Now, if I want to automate this, if I want to automate this, then I need a sensing device, first of all, to capture what is in the plate. So that is called the image, the sensing device. Now, I want to have an interpreting device or the algorithm to recognize each of these fruits.
After that, I'll provide the list of the fruits. So the whole concept of the computer vision is see, observe, understand, and then act. So these are some of the computer vision tasks. We'll go one by one. Okay. Let us start with here. Here, I'm trying to detect the vehicles. Whatever vehicles are there, I'm just detecting. So I'm putting a bounding box.
If it is an object recognition, I will say this is a car or this is a truck or this is a bus, something. What is object tracking? Tracking is if I pinpoint this particular car and I will track where it is going, whether it has met with an accident, what is the speed limit. So tracking is usually with the videos coming to this image, a one image will be classified as one label.
A label will be given. So it is identifying that this is a cat. It can be a multi-class classification as well. If you have a cat and a dog in one image, then it will show, it will tell you there are two classes. Okay. Next one is classification and localization. Localization means putting a bounding box for the object.
So it is localizing it. Next is semantic segmentation. Semantic segmentation, if in the left-hand side, if you see here, then this is the cat. This is the grass. The trees are shown here and the sky is this blue color. So semantic meaningful segmentation has been done. So it is not only objects, the pixels are getting clustered in one meaningful manner.
Here we are not saying that this particular thing is a cat or a tree or a sky, but all of them are given a different color. If we want instant segmentation, then apart from segmenting, there are three cat classes, multi-classes, I told you. Cat, dog, duck. So the cats are in red color boundaries. Duck is in green color and the dog is in blue color.
Next is feature extraction. So these are all computer vision tasks which are used for the AR-VR. In the feature extractor, it is trying to take all the gradients, wherever the edges are there or wherever there is a curvature. Now there is an optical character recognition where depending on anyone's handwriting, I am able to identify which character or alphabet the person has written.
Any questions, kindly put it in the chat box. Otherwise, I will proceed. Shall I proceed? Catalina, can I proceed please? Any doubts you have? Okay. Next, another computer vision task with natural language processing. So here we are not only giving one word, but we are generating a sequence of words which is a meaningful sentence. So if you see the classification at the left-hand side, it is for the whole image.
It is for the whole image. Girl. For the whole image, this is another task called captioning. The girl is playing tennis. And the girl is playing with black box. So now coming to this hand recognition and marker list. So now when we develop the software for augmented reality and your virtual reality, then it has to do all these things.
Classified, classify, detect, recognize, track, segment, feature detection. All of these have to be done at the preliminary stage. So now if we see this as an example of markerless augmented reality, where you want to see whether the ring suits you or not at the ring finger. Then it identifies the ring finger and the overlays this digital ring at the top of it.
The second example is marker based. So this is the marker. This is the marker we are having here. And after that, the product has to be displayed in the pamphlet. So this is the marker based. Where the product is in the pamphlet is shown as a 3D. It is rendered as a 3D. Another very important computer vision task is simultaneous localization and mapping, also called SLAM.
So the VR will be giving you a virtual map of the environment. For marker based augmented reality, a tracking can be done for a fixed user location. For markerless AR, you can track a user wherever he is going. So in marker based, you will look for only the markers where the user is going because it's a fixed for markerless irrespective of any marker.
It is, it will be continuously tracking the user. While continuously tracking the user, you can make a virtual map of the environment. So this is one case study of health and fitness. This is a treadmill, the conventional treadmill, which is mechanical. Nowadays, companies are coming up with IoT and augmented reality treadmill, where you can train yourself for mountaineering and trekking at your home itself.
So if you see here, if you see the picture, this is a guide, there is a software where you will have a guide and the guide will take you to the mountains and the slope, the alignment will be simulated similarly in the treadmill. So if it is a steep climb, the same IoT based treadmill will make it steep and in the screen, you can see that you are walking in the mountains and that's why it is steep.
When you come down, when you climb down, then the IoT again makes in the treadmill a slope type so that you do not feel any problem while climbing down. But this is not an immersive experience because this immersive experience can happen only if you see the last one where this is the virtual world in which you feel as if you are going inside it.
So it is fully immersive and you will have so many gadgets at the hand, the sound, the goggles, so many like as if there will be many sensors and IoT sensors to make you feel fully immersive. This is fully immersive and you will have so many of them, so many of them are using this IoT and the augmented reality based treadmills.
This is AR application in tourism industry. So this is, can you tell this is marker based or markless based or it is marker or markless? Okay, so here this is GPS location based. So in the GPS location based, depending on this GPS location, it knows through the GPS that this is a restaurant, this is a copy shop.
So this is augmented reality. Why? Because this information is being overlaid in the real world. So this is an AR application for an architect and a clientele. So here it is superimposing from the plan, 2D plan. The 3D is getting rendered in the screen. This is virtual reality for interactive learning. So both the humans are having, both the humans are having goggles and they are interacting with each other with a virtual world where this plane is there.
Again, this is the virtual rendering where they both are able to interact. The software development environment are various, like Unity and Buforia are very much commonly when popularly used. Arcade is from Apple. AR core from Google. Wikitude and OpenCV is all about the computer vision. So let's be ready for the future because the Gartner 2023 strategic technology trends have been released where they have already mentioned metaverse as the ninth.
In the ninth, the topic is already there, the metaverse technology. So can AR VR exist without CV? No, because the CV is the fundamental of it. So these are all the references. I would like to thank and acknowledge Vasavi College of Engineering, Hyderabad, India and Women Tech Network for giving me this opportunity. Thank you. Namaste. Any questions, please feel free to ask.
First name I am able to see. Okay, this is at a very nascent stage. So I have collected, I have researched through various blogs and the software manuals and this, I couldn't find really a one shot place where I can get each and everything. And it's evolving. And I have put all the references anyway. So those references are mostly blogs or some research papers.
Thanks a lot. Any other questions?
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why augmented and virtual reality depend on computer vision
Tilottama Goswami maps the metaverse, the four types of AR and the vision tasks that make immersive content work.
In short
Tilottama Goswami argues that augmented and virtual reality cannot exist without computer vision, because every immersive effect begins with a machine detecting, recognizing, tracking and segmenting what is in front of it. Her talk is a beginner-friendly map of the field: what the metaverse is, how VR and AR differ, which vision tasks power them, where they are already used and which tools developers reach for.
Key takeaways
The talk is about an emerging technology of the future.
The session discusses potential application domains of AR/VR technology.
The talk focuses on how computer vision, along with virtual and augmented reality, works together to build a system for achieving sophisticated interactive content.
The session gives a brief overview of the AR/VR development environment.
From the metaverse to virtual reality
Goswami opens by moving from reality toward the virtual: sight, taste, smell, touch and hearing can all be felt in a simulated, immersive environment through smartphones, haptics, VR goggles and speech interfaces. She then unpacks the word metaverse, "meta" meaning beyond and "verse" meaning universe.
"So something beyond universe. And we mean by the virtual universe where the humans and avatars co-exist as a community." (2:52)
She describes three pillars of the metaverse: the feeling of presence of oneself, the feeling of spatial presence in an environment, and an impression of the presence of others, whether humans or avatars. Its types include augmented reality, virtual worlds and extended reality, which spans VR, AR and mixed reality.
Virtual reality itself she defines plainly:
"So the virtual reality, it is a computer generated simulation and it gives you an immersive experience and the user can move around in all the 360 degrees, can manipulate the objects." (5:24)
She separates non-immersive VR, used in training simulations that focus on one part of a machine, from fully immersive VR for gaming and entertainment, where goggles are joined by haptics, sound and sensors to engage as many senses as possible.
Four ways to augment the real world
Augmented reality starts from the real environment rather than replacing it.
"Now coming to the augmented reality, it is basically your real environment and the digital content is overlaid at the top of it to get an interactive experience. So it is augmenting the real world." (6:29)
Goswami names four types: marker-based, where a marker is replaced by digital information; markerless, such as detecting a wrist and overlaying a digital watch to see if it suits you; projection-based, such as a dinosaur projected onto a spot in a garden or the bone structure overlaid on a detected arm; and superimposition-based, which creates an illusion of mixed reality, as with Snapchat templates. She turns this into a short guessing exercise with the audience before moving on.
Computer vision: see, observe, understand, act
Computer vision, she explains, is a field of artificial intelligence. Using a plate of fruit as an example, she contrasts how a person identifies and lists each fruit with how a machine would need a sensing device to capture the image and an algorithm to recognize each item.
"After that, I'll provide the list of the fruits. So the whole concept of the computer vision is see, observe, understand, and then act." (10:53)
She then walks through the core tasks: object detection with bounding boxes, object recognition, tracking a specific car through video, single and multi-class image classification, classification with localization, semantic segmentation, instance segmentation, feature extraction from gradients and edges, optical character recognition and image captioning that combines vision with natural language processing. Her point is that these are not academic extras.
"Classified, classify, detect, recognize, track, segment, feature detection. All of these have to be done at the preliminary stage." (15:28)
Tracking, mapping and real applications
Goswami illustrates markerless AR with a ring try-on that identifies the ring finger and overlays a digital ring, and marker-based AR with a pamphlet whose marker renders the product in 3D. She highlights simultaneous localization and mapping (SLAM): marker-based AR tracks a fixed user location, while markerless AR continuously tracks the user and can build a virtual map of the environment.
Her case study is health and fitness:
"Nowadays, companies are coming up with IoT and augmented reality treadmill, where you can train yourself for mountaineering and trekking at your home itself." (18:18)
As an on-screen guide leads the trek, the treadmill steepens for climbs and adjusts for descents. She notes this is not yet fully immersive, which would require goggles, sound and many more sensors. Further examples include GPS location-based tourism overlays labelling restaurants and coffee shops, an architect's 2D plan rendered in 3D for a client, and VR classrooms where two people in goggles interact in the same virtual world.
Tools and what comes next
For developers, she lists Unity and Vuforia as popular environments, ARKit from Apple, ARCore from Google, Wikitude, and OpenCV for the computer vision layer. As the speaker cited, Gartner's 2023 strategic technology trends include the metaverse. She closes by asking whether AR and VR can exist without computer vision, and answers:
"No, because the CV is the fundamental of it." (22:43)
In the brief Q&A she adds that the field is at a nascent stage and still evolving, and that her own sources were mostly blogs, software manuals and research papers because no single resource covered everything.
Watch the talk
"Computer Vision for Augmented Reality & Virtual Reality" by Tilottama Goswami, 24 minutes: https://www.womentech.net/video/computer-vision-augmented-reality-virtual-reality
About the speaker
Tilottama Goswami is a Professor in the Department of Information Technology at Vasavi College of Engineering in Hyderabad, India, with 23 years of experience across academia, research and the IT industry and research interests in computer vision, machine learning and image processing. She holds a PhD in Computer Science from the University of Hyderabad, has been granted an Australian patent for her research and serves as Chairperson of the ACM Hyderabad Deccan Professional Chapter.
Questions this talk answers
What is the metaverse?
What types of augmented reality are there?
Why does AR and VR depend on computer vision?
How does tracking differ between marker-based and markerless AR?
Which tools are used to develop AR and VR applications?
Drafted with AI from the talk transcript. Editor review pending.