Voice AI: The Future is By Everyone and For Everyone
Joan Bajorek19:302020
Your voice assistant may grade you a D plus, and it isn't your fault
Read the same paragraph to a voice assistant and a white man might score an A minus while a mixed-race woman scores a D plus. Many users blame themselves when the system fails them, and companies lose buyers who give up on the product. Joan Bajorek, a speech technology PhD and CEO of Women in Voice, asks what it would take to build voice AI that truly works for everyone, everywhere.
See the evidence, the business case and the projects building voice AI for every speaker
Joan Bajorek, CEO of Women in Voice and Head of Conversational Research and Strategy at VERSA, shows how voice AI has spread into search, homes and multimodal devices worldwide. She argues that speech recognition still carries significant race, gender and dialect bias, and that building for every speaker is both a moral obligation and a large financial opportunity.
“And I think one of the things I struggle with terribly is that people believe it's their own fault, but the tech doesn't understand them. And that's utterly not the case.”
“I really don't believe that there are people at these companies being like, Oh, let's design something that's horribly biased against people.”
What you will learn
Explain what voice AI is and where multi-turn systems like Google Duplex still fall short.
Recognise how multimodal interfaces combine voice, vision, gesture and augmented reality to fit a use case.
Spot race, gender and dialect gaps in speech recognition instead of blaming users for being misunderstood.
Build a financial as well as a moral case for supporting more languages, dialects and speakers.
Find projects such as Mozilla Common Voice and communities such as Women in Voice that widen who voice tech serves.
Details
Level
Beginner
Format
Talk
Technical depth
Introductory technical
For
All career stages
Length
19:30
Published
July 2020
Questions this talk answers
What is voice AI?
It is a system where a computer interprets spoken input, processes it and may answer by voice. Google Duplex, which phones a salon to book an appointment in a multi-turn conversation, is one of the most sophisticated consumer examples.
How widespread is voice technology already?
As the speaker cited, Google reports that around 20% of its searches are made by voice, and smart devices see hundreds of millions of weekly interactions. Many people use voice tech without realising it.
Does speech recognition work equally well for everyone?
No. Research she cites shows accuracy gaps by gender, race and dialect; in her scorecard example a white man gets an A minus while a mixed-race woman gets a D plus reading the same paragraph.
Why should companies invest in voice tech that works for more speakers?
Beyond the moral obligation, there is money at stake: a coworker skipped buying a smart fridge because it would not understand him and his wife, and the speaker cited $1.7 trillion in Latinx purchasing power in the United States.
How can people help make voice AI more inclusive?
They can donate their voice to Mozilla's Common Voice, which makes voice data openly available with attention to demographics, and join communities such as Women in Voice.
About this talk
Voice AI is increasingly ubiquitous and democratized, voice AI is shifting from a glitchy IVR world to more and more human-like, sophisticated, multi-turn interactions. The ubiquity of these products cannot be underestimated, i.e. Google Assistant will replace voice search in the Chrome browser in 2020 (Voicebot.ai, 2020). Yet it is an open question whether voice AI supports users everywhere and how far we are from supporting a mission of everyone and everywhere. While significant financial opportunities alone may be enough for some companies to consider demographics during research and design phases (Bajorek, 2019), multilingual and multi-dialectal roadmap creation can be a tricky feat to prioritize numerous scripts, low-resource datasets, and models trained with English as the default framework that extrapolate poorly (Bender, 2019; van Esch, 2019). This talk provides concrete examples of robust fiscal and moral reasons to support a wide variety of languages and dialects internationally and the evolution of our builds to support voice technology of the future.
Why your voice assistant understands Gary, not you (74 s)
The smart fridge sale bias cost a company (75 s)
Using Alexa with American Sign Language (53 s)
So this talk is called Voice AI: The Future is by Everyone and for Everyone. My name is Joan Paul-Andre Bjorik, she/her. I'm the Head of Conversational Research and Strategy at VIRSA, which is a voice agency. I'm also the CEO and founder of Women in Voice, and I'll talk about both of those things today. I'm so jazzed to be here, especially during this tumultuous time.
It is amazing that the Women Tech Network is here and to meet and talk to people from all around the world brings me a lot of joy. So thank you so much for coming. Let's move into it. So perhaps you know about Mark Cuban, but he said, "There's no future that doesn't have ambient computing or voice activation. None." And so I think, especially he's a futurist, he's an investor, but thinking about the future of technology and what voice technology has to play a part in this whole ecosystem.
So a little bit about me, since you probably don't know me. Like I said, my name is Joan. I'm a linguist, and I'm a researcher. I live in Seattle. Greetings from Seattle, wherever you are around the world. I previously worked at Nuance. I got my PhD at the University of Arizona. A lot of my research publications have been published in high profile places like Adobe XD, Cambridge University Press, Harvard Business Review. I got to speak at CES this year, which was super awesome. I participate in Voice Summit, and I'm also an Alexa Champion. All these things to say is that I'm really connected with the voice ecosystem. I'm really thinking about and talking to different organizations who are building the future of voice technology and what technology looks like in the voice conversational
ecosystem. So as I mentioned before, I'm the CEO and founder of Women in Voice, which is an empowerment organization internationally for women and gender diversity in voice technology. I'm also the head of conversational research at Versa, which is the largest voice agency in the world. I'll talk a little bit more about our projects as well. The outline of today's talk. So I'm going to be talking about voice AI, I'm going to be talking about innovation and ubiquity of voice technology. I'm also going to be talking about multimodal and integrated builds for all. And I think when we think about tech, we talk a lot about impact. And I think I'm going to I'm going to talk a little bit about problematizing who is able to use these
tools right now and who might be in the future. OK, what is voice AI here? So voice AI. So when we think about voice, we think about communication through an oral modality. So a conversation, there's listening and there's hearing with humans. And so when we think about artificial intelligence or augmented information, how I think about it, we have the computer both interpreting words and also potentially sending words out so that the input is voice and then processing on the computing side and the output potentially is also voice having this interaction. And one of the most sophisticated versions of this to date for end consumers is Google AI's project, Google duplex. If you're in the voice ecosystem, maybe, you know, but duplex is a system where the Google assistant can interact with someone over
the phone so that, you know, how can I help? And the person would ask, make me a haircut appointment on Tuesday morning anytime between 10:00 and 12:00. And the Google duplex will actually call the hair salon on the phone and make that reservation for you having what we call a multi-turn interaction. So there are different pieces of the conversation going back and forth. And this has beaten the Turing test that people actually think that they're speaking to another human. Google duplex puts in pauses and puts in different ways of iterating the information back and forth to each other. So it's a really cool technology. It scares people because it sounds so realistic that, um, a voice assistant, the computer can sound so realistic.
Um, but I think the implications of this technology, does it report that it's a robot and not a human? Um, it, we, we talk about this is that Google duplex is a pretty narrow use case. Um, it's a very specified, like you're looking to book an interaction or you're looking to go to a restaurant. Um, it cannot support all the conversations of the world quite yet. Um, but this has really pushed the bounds of what voice AI is today. Um, and maybe you haven't heard of Google duplex, but the ubiquity of voice technology is right. Um, for example, search by voice Google reports that around 20% of their searches are made by voice query today. So that people are speaking into their Chrome or into their, um,
mobile devices. And I've already seen this in mind, but Google assistant will replace voice search in Chrome in 2020. So that instead of a search bar, right, you can voice query material into Google. Um, similarly with Google and Amazon being the behemoths in the United States, um, Alexa devices, there are hundreds of millions of IOT purchased. Um, they announced at CES that hundreds of millions of weekly interactions with smart devices. So here on the right, we have, you know, the, the Google, um, assistant as well as a different, um, Amazon IOT. There are so many around the world.
Um, maybe you've tried them. Um, at one point, one in six, um, people in the United States had one. There's just a lot of people, um, focusing on and just the ubiquity of these devices. The fact that you can literally ask for search things in the future. Um, and now not only in the future, uh, but how common this is, I think the ubiquity I'm often told like voice tech, is that a thing? Do a lot of people interact with that? In fact, you may already, and you don't know that you are potentially. So the future also is not only with voice tech specifically, as we think about here's a Google home product, um, where you're literally looking at what is basically a black box,
right? That is listening to you. It's a smart speaker. Um, but in the future, um, we will be thinking about multimodal. And this is a common term in my field. Multimodal is thinking about leveraging multiple modalities, modes, and that could be all, all kinds of different inputs and outputs, um, with the computer. So text, this could be audio. This could be visual video. Um, this could be augmented reality, virtual reality, gesture, voice inputs, outputs. Right now you are hearing my voice.
Hopefully you're seeing the slides visually, um, and how that is interpreted by the computer that you might be seeing on your screen. So here are some other examples. And we have some Google hardware here with, um, smart home devices, as well as on mobile. Um, on the top right, we have examples of gesture and vision so that by using computer vision, potentially in a medical setting, you could augment images to see them differently. Um, and that the computer is interpreting your gestures to, in these different multimodal, um, environments where you could also be speaking to the computer as well. On the bottom right, we have augmented reality where the device augments, right? What is in your reality? This was, you know, famously done with Pokemon Go, right? Augmenting the reality. Um, there have
been problems with hardware is not prepared to do the live streaming and a huge amount of data. I mean, computing that is expected with a lot of these systems, but this is the future of our tech that really fits the use case, right? If it's a furniture buying adventure, you know, it really makes sense for that. If it's in a medical environment and so forth. So this is multimodal. Uh, here's some more really cool examples of a voice technology in a multimodal world. So on the top left here, I believe this is from Hong Kong. These are other kinds of smart devices, um, that came from Alibaba and Baidu, um, which are big companies out there. Um, they're really thinking about, do you see your media in this? Are you consuming
news? How are you interacting with these systems on the top right? Here's an example from Kuwait. Um, using Alexa to, um, this woman who's a technology blogger, um, help uses Alexa for her prayer practices. So using these in different ways, my company at Versa in Australia, we, um, worked on, um, a collaboration with Domino's where you could on the Google assistant, you can ask Domino's to send you a pizza. And if you're an authenticated user, it'll send you a pizza literally. Uh, and my CEO there has done demos with this where, you know, she orders a pizza, the pizza comes so quickly. People are shocked. Um, and think that was like a prank. It was not a prank where it's just, it goes that well
in the system that it's basically a send and the pizza gets ordered through the Domino's system. So check it out, um, on Google assistant. There's also other systems on the bottom right here where the, there's a smart device in a home environment that turns to see you in a different thing. So it's using not only computer vision as well as speech, but it's really being thoughtful about, you know, is the screen pivoted toward you? Can he hear you well? Is the microphone a 180 degree mic and so forth? So I think really thinking, these are some really cool examples, I think all around the world of people using devices and different ways. Uh, lastly on this one, thinking about multimodal and those inputs and outputs that I was talking about can be in different modalities. Uh, this is an example of
American sign language using Alexa devices as well. So in this case, um, Abhishek Singh, um, used computer vision to interpret signs from the user. So American sign language uses, um, gesture in that sense that that is put into the Alexa system, um, and then voice and screen and back to text so that someone who, um, communicates through American sign language can use the Alexa system, which is a really cool use case. Um, and also thinking about when we're blending, what does, what are these inputs and outputs, what different languages are being used in this multiplicity of environments? Uh, and so this is where tech for all, especially with, um, American sign language is one example of something that it is a language, but maybe it's not one we traditionally think of when we're thinking about
different languages. Um, in my work, I really think about how tech is or isn't working for lots of different people. So tech for all, um, specifically my, um, most famous publication and the one I'm deeply proud of is was published last year in Harvard business review. Um, it was so well loved that they put it on the homepage and they even translated into Brazilian Portuguese, which just, this was one of my comps papers for my PhD. And I'm just deeply proud of being able to hopefully help people interpret better what bias really means in these contexts. So the article is titled voice recognition still has significant race and gender biases. And specifically it looks at some research from the national, um, NAACL, um, as well
as there's been an updated report of this, um, for, um, at Stanford looking at black and white speakers. But this idea that we have significant race and gender biases of what current computers can understand. Um, and one of the big things I talk about in this paper is that it's not intentional. I really don't believe that there are people at these companies being like, Oh, let's design something that's horribly biased against people. No. In fact, there are huge incentives for people to be, um, making the best systems possible, um, economically. Um, yes. Um, moderation panel. Oh, um, I'm not sure what's going on there. Um, I'll continue. Oh, Oh, Shawnee, maybe this is the next speaker getting ready. Um, looking at, sorry, we're looking at many,
many screens. I'm not a speaker. Sorry. Oh, are you the moderator for this panel? No, no, no. Oh, okay. I'm just joining here. Lovely. Okay. Well, um, I am just continuing to talk about, uh, gender and dialect bias. Um, this is a very famous paper from 2017 that looks at accuracy. Um, I don't know if you play with GG plot two and that's, I, I, I, my, my coding languages are, but this is what this was made in. Um, and you can see different accents from California, New Zealand, um, New England, Georgia, and Scotland. So different people speaking these languages, speaking these dialects, excuse me, and that it's 13% across these more accurate for women than it is for men. Um, and for mixed race speakers, it's potentially 10% lower in accuracy. So this is the 50% line.
Basically these systems are not performing that well is the answer to most of this. Um, and if, um, this is not exactly interpretable for everyone, I made a scorecard that's potentially a little bit more interpretable. Um, so for example, I, if, um, three people are all native speakers of English, um, and the only real difference is who they are as a speaker. Um, and they're reading the exact same paragraph, my friend, Josh, who's a white male might get an A minus on how well the system understands him and understands almost everything. He only needs to correct 8% of what's going on.
If I speak in the same system with the same paragraph as a white female, I might get a C plus. So a 79 of how well it understands me. Um, and even worse than this, Jada, a mixed race female might get a D plus of how well the system understands her. So I think I talk about this type of research, um, at tech events and at dinner parties as well. And people say, Oh, this explains it. It always understands Gary and never understands me. And I think one of the things I struggle with terribly is that people believe it's their own fault, but the tech doesn't understand them. And that's utterly not the case. Um, most of these systems are just deeply biased, unfortunately. And we are working
on that in the speech, um, ecosystem. Um, and when we think about why to make a system better, not only is there an obvious moral obligation, um, but there's a huge amount of money in these type of IOT performing well for everyone. So, um, for example, um, one of the examples I think about often is my coworker who, um, speaks Spanish and English and he and his wife have a significant amount of purchasing power. Uh, they considered buying a smart fridge, which I don't know if you realize exactly how much smart fridges cost. They cost a lot of money. Um, but he decided not to buy a smart fridge because he knew his wife and he would not be understood by the smart fridge and the voice
recognition in it. Um, and I think this is, especially in the United States, 1.7 trillion dollars in purchasing power for the Latinx community in the United States. So there's a huge amount of money that they have to purchase different IOT that they want to. And also there's a huge growth index that the Latinx community, um, accounts for almost half of the growth of the US population. So really think about who are you catering to? Is it prepared for them? Will they be buying it? Are there preventing factors to that? Um, and this also is spoken about by, um, Melinda Gates, huge missed financial opportunities, financial blind spots. I think I'm preaching to the choir. You all are at the women tech global conference, but I think it's really important when we think about incentives to other
people who might not come from our perspective or really need, um, the case to be made for why to consider these opportunities, financial blind spots. So she says, we care about diversity, but we really care about how much money we make. Women are 85% of consumer dollars spent. Women control 70% of the financial decisions in the house. So you're missing an opportunity because you don't see it. You, so you're leaving money on the table. You're not in the deal flow. I think this is extremely compelling, especially from someone as successful as Melinda. But I think really articulating in that way can be extremely powerful. Um, with, we started at 20, it ends at 40. I have three minutes left. I have some examples of different, um, cool startups in the ecosystem that are working on voice technology for
kids. Check out soapbox labs. They just raised 6.5 million, um, novel effect, especially if you have kids. Um, this company does soundscapes behind, um, your reading of a book. That's really cool. Um, they're based in Seattle. I know the founders, they're phenomenal. There's also, when we think about voice technology for an elderly population, you know, are we designing tech that is for the 50 and over cohort or depending on what you think of elderly as clearly, um, that's some people live much, much longer. Um, but really thinking about tech that is designed for an older population and what their needs might be and how voice tech or technology in general is prepared to support them. Um, lastly, on some of these example ones, I think, um, common voice is a project out of Mozilla, Mozilla Firefox,
you may have heard about, um, Mozilla is working on a project that teaches machines how real people speak instead of learning how to speak. So Alexa understands you, for example, um, and they make voice data freely and publicly available, open source. You can donate your voice and they, um, really think about what demographics are compiled in these data sets. So I've donated my voice, check it out.
Common voice is the name of the project, um, through Mozilla. Um, lastly, I of course have to also talk about my amazing company, Versa based in Australia. Um, we've done projects with Huggies, with Domino's, with Beyond Blue, which is around mental health. Um, we've worked on a maternal, maternal and child's healthcare apps and conversational bots, as well as Coca-Cola. We've done some promotions with them.
Um, so if you're interested in hearing more about conversational AI, about voice technology, about how to support your brand, or you're just interested in this, definitely get in touch. Um, Versa, we do some amazing work. Um, lastly, as the CEO of Women in Voice, I have to tell you about my amazing community. We have 12, I think almost 13 chapters around the world, um, all across the globe.
Here's an example from a Mexican hackathon. Um, here are some of my Japanese ambassadors, and here is a tech event in Madrid, um, at the Google for startups. So there's a lot of cool work we're doing. If you have interest in this, if you're pivoting into voice, um, we're an international empowerment organization for women and minority genders in voice tech. We're all about building community, amplifying, providing professional development, empowering, and celebrating people, diverse people in this field. So check it out. Um, our website is womeninvoice.org. We're typing women in voice. We're really active on Twitter and LinkedIn, as well as Slack and some other forums. So this has been Voice AI. The future is by everyone and for everyone. I hope you have enjoyed this. Um, I look forward
to being in touch. Again, um, you can find me and all of this work, uh, on social media. Um, I've put my details in the chat and I'll do that before I leave as well. It was lovely speaking with all of you, and I look forward to continuing the discussion.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why voice AI still fails many speakers, and why that costs companies money
Linguist Joan Bajorek shows how speech recognition gaps by gender, race and dialect hurt users and leave revenue on the table.
In short
Voice AI is spreading quietly into search, phones, homes and multimodal devices, but it does not hear everyone equally. Joan Bajorek, a linguist and CEO of Women in Voice, argues that speech recognition still carries real race, gender and dialect gaps, that users wrongly blame themselves when it fails, and that fixing it is both a moral duty and a large missed business opportunity.
Key takeaways
Explain what voice AI is and where multi-turn systems like Google Duplex still fall short.
Recognise how multimodal interfaces combine voice, vision, gesture and augmented reality to fit a use case.
Spot race, gender and dialect gaps in speech recognition instead of blaming users for being misunderstood.
Build a financial as well as a moral case for supporting more languages, dialects and speakers.
Find projects such as Mozilla Common Voice and communities such as Women in Voice that widen who voice tech serves.
Voice AI is already everywhere
Bajorek opens with a simple definition: voice AI is a computer interpreting spoken words, processing them and potentially answering by voice. Her showcase is Google Duplex, in which the Google Assistant phones a hair salon and books an appointment through a back-and-forth, multi-turn conversation. She notes that it sounds so human that people believe they are talking to another person, which raises its own questions about whether it discloses that it is a machine. Yet she is clear about its limits: "Um, it cannot support all the conversations of the world quite yet." (4:13)
The bigger point is ubiquity. As the speaker cited, Google reports that around 20% of its searches are made by voice, the Google Assistant was set to replace voice search in Chrome in 2020, and hundreds of millions of weekly interactions with smart devices were announced at CES. At one point, she says, one in six people in the United States owned a smart speaker. When people ask her whether voice tech is really a thing, her answer is that they may already be using it without knowing.
Beyond the smart speaker
The future, Bajorek argues, is not a black box listening on a kitchen counter but multimodal design. "Multimodal is thinking about leveraging multiple modalities, modes, and that could be all, all kinds of different inputs and outputs, um, with the computer." (6:14) That can mean text, audio, video, gesture, augmented reality and virtual reality, chosen to fit the use case, whether furniture shopping or a medical setting where computer vision interprets a clinician's gestures.
Her examples span the globe: smart devices from Alibaba and Baidu, a technology blogger in Kuwait who uses Alexa for her prayer practice, and a VERSA project in Australia that lets authenticated users order Domino's pizza through the Google Assistant so quickly that audiences thought the demo was a prank. The example she lingers on is Abhishek Singh's work using computer vision to interpret American Sign Language so that signers can use Alexa. It is a reminder, she says, that sign language is a language too, even if it is not the first one people think of.
Who the machines do not hear
This is the heart of the talk. Bajorek's Harvard Business Review article, "Voice Recognition Still Has Significant Race and Gender Biases", drew on research presented at NAACL and later work from Stanford on Black and white speakers. She stresses that the bias is not deliberate: "I really don't believe that there are people at these companies being like, Oh, let's design something that's horribly biased against people." (11:22) Companies have every economic incentive to build systems that work well.
She walks through a 2017 study comparing accents from California, New Zealand, New England, Georgia and Scotland, which she cites as showing a 13% accuracy gap by gender and potentially 10% lower accuracy for mixed-race speakers. To make it tangible she turns the numbers into a scorecard. Three native English speakers read the same paragraph: her friend Josh, a white man, gets an A minus and corrects only 8% of the output; she, a white woman, gets a C plus; and Jada, a mixed-race woman, gets a D plus. People hearing this often say it explains why the device always understands Gary but never them. Bajorek's response is firm: "And I think one of the things I struggle with terribly is that people believe it's their own fault, but the tech doesn't understand them. And that's utterly not the case." (13:37)
The money left on the table
For audiences who need more than a moral argument, Bajorek makes the financial case. A coworker who speaks Spanish and English, with real purchasing power, decided not to buy an expensive smart fridge because he knew it would not understand him or his wife. As the speaker cited, the Latinx community in the United States holds 1.7 trillion dollars in purchasing power and accounts for almost half of US population growth. Her challenge to product teams: "So really think about who are you catering to? Is it prepared for them?" (14:58)
She reinforces the point with Melinda Gates's remarks on financial blind spots, citing figures that women account for 85% of consumer spending and control 70% of household financial decisions. In the line Bajorek reads out, companies are "missing an opportunity" because they do not see it, and so they leave money on the table.
Building voice tech for everyone
Bajorek closes with people already widening who voice tech serves. SoapBox Labs builds voice technology for children, Seattle's Novel Effect adds soundscapes to books read aloud, and she urges designers to think about older users and their needs. Mozilla's Common Voice project collects openly available voice data with attention to which demographics are represented, and anyone can contribute: "So I've donated my voice, check it out." (17:16)
She also describes VERSA's work with Huggies, Domino's, Beyond Blue on mental health, maternal and child health bots, and Coca-Cola, and the growth of Women in Voice to around 13 chapters worldwide, from a hackathon in Mexico to ambassadors in Japan and an event at Google for Startups in Madrid. "We're all about building community, amplifying, providing professional development, empowering, and celebrating people, diverse people in this field." (18:25)
Watch the talk
"Voice AI: The Future is By Everyone and For Everyone" by Joan Bajorek, about 20 minutes: https://www.womentech.net/video/joan-bajorek-voice-ai-future-everyone-and-everyone
About the speaker
Dr. Joan Palmiter Bajorek is CEO & Head of Conversational Research & Strategy at Women in Voice/VERSA, leading conversational research and strategy at the voice agency VERSA and founding Women in Voice, an international organization empowering women and gender diversity in voice tech. She holds a PhD in speech language technology from the University of Arizona and an MA in Linguistics from the University of California, Davis, and her research on bias in AI and the future of voice products has appeared in Harvard Business Review and Cambridge University Press.
Questions this talk answers
What is voice AI?
How widespread is voice technology already?
Does speech recognition work equally well for everyone?
Why should companies invest in voice tech that works for more speakers?
How can people help make voice AI more inclusive?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Turing test
Multimodal design
Resources mentioned
Harvard Business Review article "Voice Recognition Still Has Significant Race and Gender Biases"
Mozilla Common Voice
womeninvoice.org
SoapBox Labs
Novel Effect
Domino's ordering on Google Assistant
Cite this talk
Joan Bajorek. "Voice AI: The Future is By Everyone and For Everyone." WomenTech Network, 2020. https://www.womentech.net/video/joan-bajorek-voice-ai-future-everyone-and-everyone