Most hostile technology was never meant to be hostile
A teaching hospital's ICU model said nothing about asthma patients, because policy had always sent every one of them straight to intensive care: no malice, yet potentially deadly. Thoughtworks CTO Rebecca Parsons argues that as technology reaches credit, criminal justice and medicine, unintended harm is the real risk. So how can teams spot the people their products quietly fail?
Learn how to surface unintended consequences and look beyond your target audience
Rebecca Parsons, CTO of Thoughtworks, argues that most technology that ends up hostile to people was never intended to be, and that technologists have a responsibility to anticipate unintended consequences. Using examples from a biased hospital ICU model, facial recognition, credit scoring and autonomous driving, she shows why teams must look beyond their target audience and introduces the free Responsible Tech Playbook of facilitation techniques.
“But the vast majority of technology that ends up being hostile often isn't intended to be.”
“We need to lift our head up and instead of just focusing on our target audience, look around.”
What you will learn
Distinguish intended consequences of a technology from unintended and second order effects.
Recognise how hidden bias in training data can make a well-meant model harmful, as in the asthma ICU example.
Question what counts as fair in your own context, for example when using proxies in credit scoring.
Look beyond your target audience to stakeholders who are invisible during design.
Apply facilitation techniques such as the tarot cards of tech and consequence scanning from the Responsible Tech Playbook.
Key takeaways from the speaker
Technologists need to know how to identify unintended consequences of their tech choices and how to mitigate the risks or amplify the opportunities those consequences present.
We have a playbook outlining different approaches to identifying these consequences.
Team diversity helps a lot in simplifying the process, but it's insufficient.
Details
Level
Intermediate
Format
Talk
Technical depth
Introductory technical
For
Mid-career, Senior individual contributor, People manager
Length
16:50
Published
June 2022
Questions this talk answers
What is responsible technology?
Parsons treats it as an umbrella term covering ethics and equity, and the responsibility of creators and deployers to consider the broad consequences of their technology, intended and unintended.
Is hostile technology always malicious?
No. Beyond hackers and ransomware, much technology becomes hostile unintentionally, as when a hospital's ICU model learned nothing about asthma patients because of a hidden bias in its data.
Why is responsible tech more urgent now than in the past?
Early systems handled ledgers and payroll with few grey areas, but technology now shapes criminal justice, medical decisions, credit scoring and physical systems, so the footprint of what can go wrong is enormous.
What can teams do if they cannot build a fully diverse team?
They can use facilitation techniques that prompt them to look at problems from different perspectives, collected in Thoughtworks' free Responsible Tech Playbook.
About this talk
The reach of technology is extending into more and more sensitive and complex arenas, from credit decisions and medical diagnoses to sentencing — to name just a few. In addition, there continues to be little discussion around what boundaries should exist to protect individuals and organizations, although legislation and regulation is increasing coming into play. While technologists don’t have an analogue to the Hippocratic oath, we still need to consider what responsibilities we have to those who use the technology that we build, as well as those who are invisible during the design process. We will discuss approaches to help technologies examine their product and technology choices to be more inclusive, aware of bias, transparent and to mitigate negative inadvertent consequences.
From Volkswagen to looking beyond your audience (86 s)
Inside the free Responsible Tech Playbook (65 s)
I'm going to start with a few quotes about technology, because when we think about technology, we all have different kinds of perspectives. This first one comes from William Gibson. I think that technologies are morally neutral until we apply them. It's only when we use them for good or evil that they become good or evil. If you'll notice, I emphasize that word we, because technologies are not just inert things.
They happen because we interact with them. We create them. We bring them into the world. And then the next quote is from Tim Harford from the Financial Times. "New inventions do not appear in isolation. They profoundly shape the societies around us. And I would assert they also, this is now Rebecca talking and not Tim Harford, they also are shaped by those societies.
So how we react to technology, the influence that that technology has is shaped by the societies around us. And then one of my personal favorites is from Marc Andreessen. "The historical track record of technology innovators predicting the consequences of their innovations is very poor. So is everybody else's predictions." As technologists, we are not very good at really understanding what is going to happen with the result of our technology. Because as problem solvers, we have this laser focus on here's our audience. We have this entire discipline of user-centered design and user research and accessibility to understand how our users are going to interact with the solution that we've given them to their problem. But we have blinders because we're looking at our problem and we're looking at our target audience and we aren't always looking at the other people around us. The other
people that we aren't thinking about right now. And that's where this notion of responsible technology technology comes into play. And we can have a discussion, is it responsible technology? Is it ethical technology? I consider responsible technology to be kind of an umbrella term that will take in ethical considerations as well as considerations of equity. Are we bettering or making worse societal or economic inequities? But it also is talking about what do we as technologists need to do to think about what are the consequences of the technology that we are creating? There's clearly an intended consequence. The problem that we want to solve, the solution, the service that we want to deliver to our customers or to society. But then there can be unintended consequences. And there might be second order effects of those technologies as well.
And what responsible tech gets at is our responsibility as the creators and the deployers and the implementers of technology to take into account what are the broad consequences of our technologies, even if, as Mark Andreessen points out, we're not very good at predicting what those are. Now let's talk a little bit about what I mean by hostile tech. Because there's a whole category of technology that people would easily recognize as hostile. Hackers, ransomware, disinformation bots, all of those things are easily identifiable as hostile tech.
But what about, say, tracking algorithms? I was talking to a marketing professional not all that long ago, and she said no one cares about being tracked online if the recommendations and the information that they get back as a result of that tracking is useful. And I disagreed with her. I said it might be that many people don't mind, or even most people don't mind, but a lot of people don't mind.
People don't mind. But a lot of people do mind. They don't care what they are giving up. They don't want to be tracked. To them, they consider that tracking to be hostile. And so we need to take into account the perspectives of the people who are involved. Sometimes this issue with the technology isn't even intentional. One of my favorite examples that I try to use when I talk about bias in AI is a teaching hospital. And they tried to develop a model that would help them with recommendations on when patients should be put into the intensive care unit after a particular procedure.
And they used two different algorithms to train on the same data set to see what kind of models they got. And the first model was very good at making predictions that tracked with the data set. But they couldn't really tell what was the basis for the recommended actions. The other technique they used didn't come up with quite as good of a model, but they could actually interrogate the model and understand what was the rationale for the recommendations being made.
And what they learned is that they had a bias in their data set. Their policy had always been for this particular procedure that all people with asthma would immediately be put into intensive care. And so their data set had nothing to say about people with asthma. So not surprisingly, the model that resulted said nothing about people with asthma. Were those modelers malicious? Did they have ill intent against people with asthma? Of course not. What they didn't understand was that their data set had an underlying bias that they didn't recognize.
Now, I'm sure people with asthma would have felt, well, if they died as a result of that model, well, that feels pretty hostile to, you know, to me. But it certainly wasn't malicious. And it certainly wasn't intentional. Fortunately that they had the foresight to use those different models to try to determine what the basis of that recommendation was so that they could understand that and could possibly account for it.
So then what kinds of areas might we be thinking about? Let's start with the environment. When you're developing a product, particularly a physical product, are you thinking about the communities around where that product will be manufactured? Are you thinking about the community and the environment around where the raw materials will be sourced from? What about a quality? Does your product in some way discriminate against people? Again, facial recognition software, facial recognition software is well known to not necessarily do as good of a job on women black faces as they do on white male faces. A lot of that has to do with the training set that's used. And so if you're deploying facial recognition software, the possibility, the probability of a false positive will be much greater for certain demographic groups than others.
Is that fair? What about a credit scoring algorithm? As a bank, for example, how should I make the determination of whether or not I use proxies for particular attributes of people who are applying for credit? Because it might be that if I use and find some kind of proxy for socioeconomic status or or educational upbringing or race, that I might get more profit because the model is making recommendations that will tend to steer me away from people who might be not have quite a good credit risk. But is that fair? Are you improperly labeling people as a credit risk based on factors that don't apply to them?
Yeah. So as organizations, we need to come to terms with what constitutes fair in my context. It gets even more complex when you start talking about autonomous driving. Who do you really want to have making the decision about whether or not the car prioritizes the life of the driver or the pedestrian? We need to come to terms with what constitutes fair in my context. It gets even more complex when you start talking about autonomous driving. Who do you really want to have making the decision about whether or not the car prioritizes the life of the driver or the pedestrian?
Personally, I'm a software developer. I don't want to have the responsibility of deciding what that happens. And so we have this broad range of areas where technology is having an increasing impact on our life. And we have to figure out how, as technologists, we respond to that. So I've given a lot of examples about how this can play out in different realms.
But I want to talk a little bit about why is this such an important topic now? And if you think back to when people like me with this amount of gray hair got involved in technology, we were doing things like account ledgers, accounts payable, payroll. There weren't many gray areas with respect to what was the right thing to do.
No, you don't cheat. No, you don't break the law. Yes, you pay, you know, you pay in the withholding taxes you're supposed to pay in. So many of those things, it was very straightforward what the right thing to do was. But the surface area of where technology is impacting our lives, both in the software realm, things like criminal justice and medical decision making and, you know, credit scoring and all of that.
But also in the way that it's starting to impact us physically, more and more wearables, more and more medical technology. And more and more automation in things like manufacturing and chemical processing and such. And so the footprint of what can go wrong if we don't take particular problems into account, if we don't take particular failure modes into account, that footprint is enormous.
And so we need to spend more time thinking about how do I do a better job of predicting what are the possibly unrecognized, positive, unintended consequences of my technology, but also what are the unintended negative consequences and what might I do to be able to mitigate against those. And so what should we do? I don't think it's terribly controversial to say that the individual software developer working for Volkswagen, who are the individuals, if it was a team, who made the decision to write the code that said, if I'm hooked up to an exhaust testing machine in the United States of America, the engine runs this way.
And if I'm not hooked up to that machine, the results should look like this. I don't think anybody would argue with the fact that that team made a bad choice. But past those obvious cases, we have a responsibility to consider the broader range of stakeholders. We need to lift our head up and instead of just focusing on our target audience, look around.
And maybe there's another target audience out there that with a very minor tweak, either to our product or to our message, we might have vastly expanded opportunities. Or maybe if we think about how somebody who doesn't look like us or comes from a different culture or comes from a different socioeconomic background or a racial background, how they might feel about that.
But maybe we can make decisions that help our technology be less hostile to people who weren't in our original target audience. We can't have a representative from every culture and every language and every religious background and every sexual identity and race and country and socioeconomic. But we do have to do what we can to try to bring diverse perspectives into the discussion.
And where you can't do that with building a diverse team, there are strategies that will allow you to do this. There have been many teams in different organizations who have developed facilitation techniques to allow us to help prompt us to look at problems differently and from different perspectives. And by getting permission from these different groups, we've collected these techniques into something we call the Responsible Tech Playbook.
And it's available from our website, perfectly free, no paywalls, no email, you've got to give an email address kind of thing. But it's a variety of facilitation techniques, some modeled after things like threat modeling, but there's the tarot cards of tech, there's something called a consequence scanner. So all of these things helped prompt you as you're trying to look at your technology and understand what this technology might represent to help you step back and think about, well, what about this?
What about people from here? What about someone who wants to do this? And it's that diversity perspective, getting our head up, that will allow us to start looking more broadly at what impact our technology is having. Because although, yes, there are hackers out there, there are people who are malicious, who are intentional about creating hostile technology. But the vast majority of technology that ends up being hostile often isn't intended to be.
And it's our responsibility as technologists to do what we can to not create hostile technology when we don't intend to. And I want to go back to that quote from William Gibson, "I think that technologies are morally neutral until we apply them. It is only when we use them for good or evil that they become good or evil."
It's our responsibility as technologists to make that determination and to decide for ourselves, we are going to be on the side of the responsible technologist. We are going to do what we can to ensure that the technology we create has as broad a positive impact as possible and as small a negative impact as possible. And we do what we can to mitigate that risk.
Thank you so much for your attention.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why most hostile technology is built by accident
Thoughtworks CTO Rebecca Parsons explains how well-meaning teams create harmful technology, and what technologists can do to see the consequences coming.
In short
Most technology that harms people was never designed to, argues Rebecca Parsons, CTO of Thoughtworks. Because technologists focus tightly on their target audience, they miss the people around it and the unintended consequences of what they build. Her answer is to deliberately widen the view: bring in diverse perspectives and use structured facilitation techniques, such as those collected in the free Responsible Tech Playbook.
Key takeaways
Technologists need to know how to identify unintended consequences of their tech choices and how to mitigate the risks or amplify the opportunities those consequences present.
There is a playbook outlining different approaches to identifying these consequences.
Team diversity helps a lot in simplifying the process, but it is insufficient.
Technology is shaped by the people who build it
Parsons opens with three quotes. William Gibson's view that technologies are morally neutral until applied leads her to stress the word "we": technology exists because people create it and bring it into the world. From Tim Harford of the Financial Times she takes the idea that inventions shape society, and adds her own corollary: "So how we react to technology, the influence that that technology has is shaped by the societies around us." (1:04)
The third quote, from Marc Andreessen, is about how poorly innovators predict the effects of their work. Parsons agrees: "As technologists, we are not very good at really understanding what is going to happen with the result of our technology." (1:04) Disciplines such as user-centered design and user research help teams understand their intended users, she notes, but they also create blinders. Responsible technology, in her definition, is an umbrella term covering ethics and equity, and the duty to consider intended consequences, unintended ones and second order effects.
Hostile tech is often unintentional
Hackers, ransomware and disinformation bots are easy to recognise as hostile. Parsons is more interested in the grey areas. She recalls disagreeing with a marketing professional who claimed nobody minds being tracked if the recommendations are useful. Many people may not mind, she says, but some do: "They don't want to be tracked. To them, they consider that tracking to be hostile." (4:26)
Her central example is a teaching hospital that trained two models to recommend when patients should go to intensive care after a procedure. The more accurate model could not explain itself; the more interpretable one revealed that the data said nothing about asthma patients, because hospital policy had always sent them straight to intensive care. The modelers meant no harm. "What they didn't understand was that their data set had an underlying bias that they didn't recognize." (5:48) A patient harmed by that model would still experience it as hostile.
Fairness questions every team must answer
Parsons walks through areas where these questions arise. Physical products affect the communities where they are manufactured and where raw materials are sourced. Facial recognition software, as she points out, is known to perform less well on women and Black faces than on white male faces, largely because of training data, which raises the risk of false positives for some groups. A bank might find proxies for socioeconomic status or race that make a credit model more profitable while unfairly labelling people as risky. Autonomous driving raises the question of who decides whether a car prioritises the driver or the pedestrian.
Her conclusion: "So as organizations, we need to come to terms with what constitutes fair in my context." (9:10)
Why the stakes are higher now
When Parsons started in technology, the work was account ledgers, accounts payable and payroll, with few grey areas about the right thing to do. Today software influences criminal justice, medical decisions and credit, and it increasingly touches people physically through wearables, medical technology and automation in manufacturing and chemical processing. "And so the footprint of what can go wrong if we don't take particular problems into account, if we don't take particular failure modes into account, that footprint is enormous." (11:05)
Lifting your head up: the Responsible Tech Playbook
Some cases are obvious, Parsons says, such as the Volkswagen team that wrote code making engines behave differently when connected to an exhaust testing machine. Past those, technologists must consider a broader range of stakeholders: "We need to lift our head up and instead of just focusing on our target audience, look around." (12:31) That wider view can reveal both harms and opportunities, such as a new audience reachable with a minor tweak to a product or message.
No team can include someone from every culture, language, religion, identity, race and background, so diversity alone is not enough. Thoughtworks gathered facilitation techniques developed by other organizations, with their permission, into the Responsible Tech Playbook, available free on its website without a paywall or email sign-up. It includes approaches modelled on threat modeling, the tarot cards of tech and a consequence scanner, all designed to prompt teams to ask what happens for people they had not considered. "But the vast majority of technology that ends up being hostile often isn't intended to be." (15:10)
She closes by returning to Gibson and calling on technologists to choose to be responsible: to maximise positive impact, minimise negative impact and mitigate the risk.
Watch the talk
Responsible Tech in an Era of Hostile Tech, Rebecca Parsons, 17 minutes: https://www.womentech.net/video/responsible-tech-in-era-hostile-tech
About the speaker
Dr. Rebecca Parsons is Chief Technology Officer at Thoughtworks, with many years of experience in technology and large-scale software development, and co-author of the book Building Evolutionary Architectures with Neal Ford and Pat Kua. Before Thoughtworks she was an assistant professor of computer science at the University of Central Florida after a Director's Post Doctoral Fellowship at the Los Alamos National Laboratory.
Questions this talk answers
What is responsible technology?
Is hostile technology always malicious?
Why is responsible tech more urgent now than in the past?
What can teams do if they cannot build a fully diverse team?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Responsible Tech Playbook
Tarot Cards of Tech
Consequence scanning
Threat modeling
User-centered design
Resources mentioned
Responsible Tech Playbook (free on the Thoughtworks website)
Tarot Cards of Tech
Consequence scanner
Cite this talk
Rebecca Parsons. "Responsible Tech in an Era of Hostile Tech." WomenTech Network, 2022. https://www.womentech.net/video/responsible-tech-in-era-hostile-tech