Designing the Future: A Journey into the Revolutionary UX of the Mercedes Vision AVTR
Vera Schmidt16:532023
Mercedes wants the car to reach out to you, not the other way round
Raise your hand inside the Vision AVTR and its interface lands on your palm, following you as you move, so you feel your own warm skin instead of cold glass. Vera Schmidt, Director Advanced UX Design at Mercedes-Benz, walks through the concept car built with the Avatar team to show where in-car interaction is heading. What does it take to design a machine that adapts to people, rather than people adapting to the machine?
See how merge control, palm projection and biometric sensing rethink the human-car interface
Vera Schmidt, Director Advanced UX Design at Mercedes-Benz, takes viewers inside the Vision AVTR, a concept car created with the Avatar film team. She argues that car interfaces must move from people adapting to machines toward a "human machine merge," and shows how merge control, palm projection, biometric sensing and spatial sound make that vision tangible.
“So, it's not longer that the man or woman adapting to technology, but the technology is embracing the mankind.”
“We have an explosion of features due to the technology that brings about the need for radical simplification on the user side.”
What you will learn
Reframe interface design from users adapting to technology toward technology adapting to users.
Apply radical simplification when new technology causes an explosion of features.
Use concept prototypes to make future ideas tangible and experienceable.
Combine haptics, spatial sound, projection and biometrics to extend what users can perceive.
Draw on cross-industry collaboration, such as with film creatives, to stretch design thinking.
Key takeaways from the speaker
Transforming ambitious ideas into reality.
Facilitating and showcasing natural technology interaction.
Pushing the boundaries of automotive UX design.
Details
Level
Intermediate
Format
Talk
Technical depth
Introductory technical
For
Mid-career, Senior individual contributor
Length
16:53
Published
September 2023
Questions this talk answers
What is the Mercedes-Benz Vision AVTR?
It is an autonomous concept vehicle created in collaboration with Avatar, whose name stands for advanced vehicle transformation. It explores a symbiosis between the human, the machine and nature.
What does moving from HMI to human machine merge mean?
Instead of the user reaching out to touch screens and knobs, the machine adapts to and reaches out to the user, so the person becomes part of the interface.
How can you drive a car without a steering wheel?
With merge control: you place your palm on it to synchronize and have your heartbeat detected, then move it to accelerate, brake and turn left or right.
How does the hand projection interface work?
When you raise your hand, the interface is projected onto your palm and follows it as you move. Closing your hand selects the desired icon.
How does Vera Schmidt envision in-car entertainment?
She sees the car's movement through the environment as what makes it special, so entertainment should use the fact that you are moving rather than sitting on a couch.
About this talk
Join me on a journey into the world of the Mercedes Vision AVTR, where we unite design and technology to create a revolutionary user experience. From the car's advanced interface and intuitive gestures to its hand projection feature, immerse yourself in a world where the vehicle is an extension of your senses. With augmented reality and a biometric authentication system, you'll feel like you've entered a new dimension of driving, experiencing the world around you in a fascinating way.
With its pioneering design and futuristic features, Mercedes offers a new level of immersive and intuitive user experience. The car's advanced technologies and innovative materials are a testament to Mercedes' commitment to pushing the boundaries of digital innovation and luxury.
Driving virtually through places you can't visit (63 s)
Well, it's a pleasure for me to give you a glimpse into our design and innovation approach at Mercedes-Benz. Currently, I'm located here in our design headquarter in Sindelfing, which is near Stuttgart in Germany. And I want to give you a small transparency about what we do here. In our center, we are focusing on the user interface of the car with our engineering colleagues. I'm a designer and it's our design department and design center. And we are focusing on the interface. How do you interact with the car? What's the input? What's the output?
What's the interface? It's for the production cars customers can buy. But at the same time, and it's especially in my department, we are focusing also on the future vehicles. So how do you as a human interact with the car in the future? What kind of technologies will be available? We heard about AI today already, what kind of impact will have AI on our designing the human user experience within the car, but also beyond the car. So today, and for in this part, especially we are constantly push the boundaries to find and visualize new technological solutions for desirable future. And we are creating cars like you will see in a moment to make our ideas and prototypes experienceable. So in my talk today, I will introduce you to the vision AVTR. It's a car
where our vision of human centered design becomes very visible and tangible. In short, it becomes experienceable. The Mercedes AVTR is the result of a very extraordinary global collaboration between Avatar. It's you guys might know Avatar. It's one of the most innovative film brands in the entertainment industry. And of course, Mercedes Benz. The name of this groundbreaking concept vehicle stands for advanced vehicle transformation. And I have to say from my personal experience, it's one of the best best collaboration I ever experienced, such a creative people from the film industry to collaborate with them from a very different field. It was a very highlight. It's one of my highlights in my career, to be honest. And in some of the slides I will show you in a second, you will see how we got inspired by the Avatar movie and the ideas of the creatives. So, the vision AVTR is about a symbiosis between the human, the machine and the nature around us.
And the biometric connection, the autonomous concept vehicle, it's driving autonomous, enables a complete new interaction between man, machine and the nature. And we ask ourselves, so what if our vehicle would extend our senses and make us aware of our world's beauty, which we might not see or might not experience in this way. Looking ahead, it's all about reducing the whole complexity by expanding the functionalities that matter. Our method of choice is human centered innovation. We have an explosion of features due to the technology that brings about the need for radical simplification on the user side.
And to UX and UI design is one of the major driver for this innovation here. We have to simplify the information, the content you want to interact with. The user on the car is often interacting as an operator. You are approaching the machine. So, you are reaching out to a display. You are touching the screen. You are approaching the machine.
So, this is something you are touching called metal, the glass. But our vision here is how could the machine let us explore and support us in our desires, adjusting to how we interact with our environment. So, it's not longer that the man or woman adapting to technology, but the technology is embracing the mankind. It's as if the machine is reaching out to you. So, you will be part of the interface. And we called it from HMI to HMM, human machine merge.
So, for us, digital luxury also means that the digital luxury also means that we can support each individual needs and personal preferences and create an interface that can adapt, that can extend your senses, and it can generate an incredible immersive experience. It's very personalized. It's adapting to your needs as human. Well, the experience, of course, already starts before you enter the car. You're approaching the car. It recognizes you. It wakes up. It's signaling that it becomes ready to go on the ride with you.
And you see also the interior is lighting up. The sensors are getting ready. There are a lot of sensors in there, and it's introducing you really to enter the car. So, for us, for us, it really means that we are designing kind of a living organism. If you look at the back of the car, you see like these 33 movable flags. They're integrated. And what they can do is they're reacting off your movement. So, if you're approaching the car from the left or the right, the flaps start moving. So, it's almost like this creature can detect you and starts communicating with you.
And, of course, it can not only sense you. You see the different colors here. But also, if you're sitting inside a vehicle, you can communicate to your environment, to other cars. It's not like a stiff metal anymore, but it starts really reacting to environment, communicating. And especially if the car in the future can drive autonomous, this would be a very interesting feature to develop.
If you are looking into the interior, you integrated very different technology. So, it really feels like this is an organism you can communicate to. The car can sense you, but it also can react to you. So, the whole interior becomes really an immersive experience space. And I want to explain you what's inside the car because it's full of technology. We have, for example, inside the seats, we have haptic perception in the seats.
There are sensors detecting your heartbeat. There are sensors or technology which is vibrating you, giving you haptic feedback. We have a curved dashboard for a lot of space for projection. There are sensors. And we have a very natural interface, which is completely new. And visual extension for the dashboard screen. We have 3D spatial sound. It's also very, very interesting. So, it's also for your senses.
And we have a very interesting magic pool for the child connection. And I will jump to some of these to explain you further what our ideas were, how we can design visual experience in the future to a more human-centered approach. So, if you are entering the car, you saw there was no steering wheel. It looked pretty empty. The only thing you saw was like, we call it merge control. And once you are approaching the car, the car starts to wake up.
We call it wake up. It's more like an organism. And this merge control started to pulsate. It's like you are breathing. It starts to become alive. And it's showing you it's ready. It's awake. It's detecting you. It's sensing the environment. Once you sit inside the car, you can put your palm on top of the merge control. And that way, you are synchronizing yourself. And you see the moment here with the car.
So, your profile is getting set up. Your heartbeat is detected. The car can see if you are nervous, if you are under stress. It can then adapt to your needs. But at the same time, we said the car is driving mostly autonomous. But if you don't want to drive autonomous, you want to steer the car, you can use this merge control to move forward, to accelerate or to brake, to turn left, to turn right.
That's our new human machine. And we call this element merge control. It's for us more adaptive to how we could interact with the car. It's not the steering wheel anymore. You have to turn. If you move on. If you move on, one of the very interesting features is one of my favorites in this car is that we integrated the hand projection. Before, remember, you said we said usually you are the one who is reaching out to the machine.
You touch a touch screen or you're trying to turn a knob. You're trying to turn the steering wheel. But how about if the car is an interface that really is approaching? It's more reaching out to you. So you don't have to reach out to stretch out, especially in a car. You know, while you are driving, it's difficult anyways. But for example, if you raise your hand, the projection, the interface will be projected onto your palm.
And even if you move your hand left and right, the interface will follow you. It will stay and stick to your hand. And for us, it's very interesting because suddenly you are not touching a cold glass or like a material, but you are touching or you're feeling your own warm, soft skin, which is very different in how we interact usually.
And by if you want to select the icon, you can just close your hand and select the desired icon for the interface, which you see in the front. Moving forward, we see a very luxurious dashboard. This dashboard is really huge, not only in front, but you can see and it's really expanding into the back of the of the rear.
So to the children's seats. And that way, the whole information, which is coming in the front, can float into the magic pool in the rear where the kids can interact and play with it. But it was very interesting to create an immersive ecosystem within the car where the information can really come onto the back of the little seat you see on the left, on the magic pool you see on the left side.
And the kids can play with it and really experience the environment around them with very interesting information. For us, it's also that the whole interior should become kind of a digital membrane. So we want that the car is extending our senses. There are things we might not see. There are things we might not hear. Let's reintegrate the 3D spatial sound. So for example, if an element, maybe an deer or another animal can or is really walking close to us or flying, maybe this is something we can hear because the car can give the sound to it.
Or the seat is vibrating on the sound to know, hey, on the left side there is something happening. So let's see what we created for the huge dashboard. We think it can be very interesting to use the car, as we said, as a membrane to visualize maybe information we usually are not able to see. So maybe you can see and the car is detecting with all the sensors the car has the environment, for example, you can see nature forces, things like the magnetic field or the air flow, energy flows, which you are usually not seeing with your own eyes.
And the car can see and the car can see and maybe it's something interesting. It can detect to you and show you the environment. And together with the sound and with the vibration in a seat, it would get really an immersive experience. At the same time, we said, well, we are driving around with the car, but maybe there are areas you can't drive around just because there are no roads or it's a secure nature.
You shouldn't destroy. But how about your drive through it virtually? So what we could imagine is to extend the view to places which you can't really visit. So you are visiting, for example, the pink lake or different, very special natures and places within the within the earth. And of course, because we are collaborating with Avatar, we said maybe there are views usually or perspectives you can't see.
As an example from nature, we have bees and bees see very different colors and structures on the flower or an environment which we as human can't see. How about we can jump into this perspective? And in the collaboration we used from the Pandora, the dire horse or, for example, the mountain Benji. And then you can fly through the area like the mountain Benji or you run through the area like a dire horse.
For us just to experience this and show what kind of ideas there could be, how you could really use the car to drive through the environment would be very interesting. At the end, I would say for all these interactions you see in the car and the designs, the new approaches for the human center design and the aesthetics we generated, it's almost like a piece of art and you almost sit inside it as a piece of art.
And it was a great collaboration and gave us a huge impact also on what we could do and how we can design and engineer our cars in the future. Thank you for your time. Thank you for listening. It was a pleasure. Thank you so much for your presentation. I was very impressed by the design and the presentation and I just see how much effort and thought you put into coordinating different parts of the product project together and truly inspiring.
It's like you are the, you are at the forefront of the transformation, innovation and the way changing the way we drive and the driven and, and, and, um, you know, um, thought of everything in the future. Thank you so much. Um, here's a question. Do you, how do you, how do you envision in car entertainment by, uh, how do you envision in car entertainment by Anke Elena Petracje?
So it's, it's, it's, it's really what we just said, the car entertainment. I mean, the, the, the, the special thing on a car is it's, it's moving, right? You are not sitting on a couch and watching TV. You are moving for environment. So for us, it would be really interesting to include the movement of the car to any entertainment features.
If it's around you, if you're wearing glasses, whatever it is to really use the situation that you are in a car moving around would be for us, uh, very interesting. Yeah. Thank you so much. Do we have any more questions? Let me also have a look. I saw that people were thanking you in the chat and, um, we're truly honored and happy to have you with us today at the chief in tech summit.
What is the best way for you to fall to follow up with you? People have more questions. So maybe we're a bit shy to ask them now in the chat. You guys, I'm, I'm on LinkedIn and you can reach out to me anytime. And, uh, we have different design studios around the world. We have huge engineering parts. So we would love to interact and exchange. And of course we're also looking for talents. So connect. I'm very happy about it.
Yeah. So connect with Vera Schmidt. Don't be shy. And maybe who knows tomorrow you're working at Mercedes and it's you presenting at the women in tech global conference. Yeah. It's, I think it's very exciting to be working on the team and I'm sure many people will, you know, explore this opportunities and yeah. Thank you so much for being with us today. Truly impressed with your design and presentation and everything that you do.
Thank you. So thank you so much Vera. Bye. Take care. Have fun.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
What happens when the car adapts to you instead of you adapting to the car
Vera Schmidt of Mercedes-Benz shows how the Vision AVTR concept car turns the interface into something that senses, follows and extends the person inside it.
In short
Vera Schmidt, Director Advanced UX Design at Mercedes-Benz, argues that the future of in-car experience lies in reversing the usual relationship between people and machines. Using the Vision AVTR, a concept car created with the Avatar film team, she shows how a vehicle can recognize its driver, read biometric signals, project its interface onto a palm and extend human senses. The result is what her team calls a shift from HMI to HMM, human machine merge.
Key takeaways
Transforming ambitious ideas into reality.
Facilitating and showcasing natural technology interaction.
Pushing the boundaries of automotive UX design.
A concept car born from an unusual collaboration
Schmidt works at the Mercedes-Benz design headquarters near Stuttgart, where her department designs both the interfaces customers can buy today and the ways people might interact with cars in the future. Her team builds concept vehicles so that ideas and prototypes can be experienced rather than only described. The Vision AVTR, whose name stands for advanced vehicle transformation, came out of a global collaboration with the creative team behind Avatar.
She describes the partnership as a career highlight, and explains the concept as "a symbiosis between the human, the machine and the nature around us" in the vehicle's design brief. The guiding question was whether a car could extend human senses and make people aware of beauty in the world they might otherwise miss.
Underpinning the work is a method she calls human-centered innovation. As she puts it: "We have an explosion of features due to the technology that brings about the need for radical simplification on the user side." (2:26) In her view, UX and UI design are among the major drivers of that simplification.
From HMI to human machine merge
Today, Schmidt notes, the driver acts as an operator: reaching for a display, touching glass, turning a knob. The Vision AVTR flips that relationship so the machine reaches toward the person. "So, it's not longer that the man or woman adapting to technology, but the technology is embracing the mankind." (3:44) Her team gave the idea a name: "And we called it from HMI to HMM, human machine merge." (3:44)
For Mercedes-Benz, she explains, digital luxury means supporting individual needs and preferences with an interface that adapts, extends the senses and creates an immersive, personalized experience.
A car that behaves like a living organism
The experience begins before anyone gets in. As a person approaches, the car recognizes them, wakes up and lights its interior. At the back, 33 movable flaps react to movement from the left or right, so the vehicle seems to notice and communicate with the person nearby. "So, for us, for us, it really means that we are designing kind of a living organism." (4:55)
Inside, the cabin is full of technology: haptic feedback and heartbeat sensors in the seats, a curved dashboard with space for projection, 3D spatial sound and a rear "magic pool" for children. Schmidt describes the whole interior as an immersive experience space that both senses and responds.
Merge control and the interface on your palm
There is no steering wheel. In its place sits merge control, which pulsates as if breathing when the car wakes. Placing a palm on it synchronizes driver and vehicle: a profile is set up, the heartbeat is detected and the car can tell if the driver is stressed and adapt. Although the car is designed to drive mostly autonomously, merge control also lets the driver accelerate, brake and turn.
Schmidt's favorite feature is hand projection. Raise a hand and the interface appears on the palm, following it as it moves; closing the hand selects an icon. "And for us, it's very interesting because suddenly you are not touching a cold glass or like a material, but you are touching or you're feeling your own warm, soft skin, which is very different in how we interact usually." (9:23)
Extending the senses beyond the windshield
The dashboard extends into the rear, where information can float into the magic pool so children can play with it. Schmidt wants the interior to act as a digital membrane. Spatial sound could let passengers hear an animal moving nearby, while seat vibration signals that something is happening on one side. The large dashboard could visualize forces normally invisible, such as magnetic fields, air flow and energy flows.
The team also imagined virtual journeys through places that cannot or should not be driven, such as protected nature. Drawing on how bees see colors humans cannot, they asked: "How about we can jump into this perspective?" (13:08) Through the Avatar collaboration, passengers could fly through a landscape like a mountain banshee or run like a direhorse.
She closes by describing the car as almost a piece of art you sit inside, and says the project shaped how Mercedes-Benz might design and engineer future cars. In the Q&A, asked about in-car entertainment, she points to what makes a car unique: "You are not sitting on a couch and watching TV." (15:07) Future entertainment, she suggests, should build on the movement of the car through its environment.
Watch the talk
Designing the Future: A Journey into the Revolutionary UX of the Mercedes Vision AVTR, by Vera Schmidt, about 17 minutes: https://www.womentech.net/video/designing-future-journey-revolutionary-ux-mercedes-vision-avtr
About the speaker
Vera Schmidt is Director Advanced UX Design at Mercedes-Benz AG, heading Advanced Experience Design at the Mercedes-Benz R&D and Design headquarters after more than 15 years of experience in New York City, Silicon Valley and the Stuttgart area. With her cross-functional department, she envisions the digital vehicle of the future and has helped shape core UX design strategies, concepts and aesthetics for current and next-generation Mercedes experiences.
Questions this talk answers
What is the Mercedes-Benz Vision AVTR?
What does moving from HMI to human machine merge mean?
How can you drive a car without a steering wheel?
How does the hand projection interface work?
How does Vera Schmidt envision in-car entertainment?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Human-centered innovation
Human machine merge (HMM)
Cite this talk
Vera Schmidt. "Designing the Future: A Journey into the Revolutionary UX of the Mercedes Vision AVTR." WomenTech Network, 2023. https://www.womentech.net/video/designing-future-journey-revolutionary-ux-mercedes-vision-avtr