Are We Building Products that are Ruining the World?
Carey Jenkins20:382021
A startup's $1 million prescribing alert fired 230 million times
A free health records platform let a pharmaceutical company help write the unbiased-looking alerts doctors saw while prescribing pain drugs, and the product did exactly what it was built to do. Carey Jenkins, CEO of the research, innovation and build studio Substantial, argues this was no unintended consequence but the result of how tech teams are trained and rewarded. So what would it take for the people building products, including you, to choose differently?
Learn five practices that help teams build products without trading users' wellbeing for revenue
Carey Jenkins, CEO of Substantial, argues that the tech industry is building products that harm the world because it trains people to chase growth and revenue at any cost. Through a case study of a health records startup that let a pharmaceutical company shape prescribing alerts, she shows how ethical lapses happen in small steps, and offers practices for more ethical innovation: change the operating system, lead from why, reduce blind spots, think in systems and create with foresight.
“Technology is capable of making the world a better place. But the technologists have to choose that.”
“This wasn't an unintended consequence. This was very intended and very effective.”
What you will learn
Recognize how growth targets, equity-heavy pay and revenue-only bonuses push teams toward ethical shortcuts.
Distinguish legal compliance, a minimum bar for risk mitigation, from ethics as a daily practice.
Ask who will be most affected by a product, not only who pays for it or uses it.
Design research free of a customer agenda and bring in independent experts to reduce blind spots.
Map the system a product lives in, identify its key actor and anticipate how it will evolve.
Key takeaways from the speaker
The enormous power technology wields requires us to hold ourselves accountable for the role technologists play every day in blurring ethical boundaries.
Ethical practices cannot be siloed into a process or team - it must be embedded and empowered throughout your culture.
The path to revenue does not have to be paved with ethical trade-offs - how we build is as important as what we build.
Details
Level
All levels
Format
Talk
Technical depth
Non-technical
For
All career stages
Length
20:38
Published
June 2021
Questions this talk answers
Are tech companies building products that are ruining the world?
Jenkins says yes, because the industry has trained creators to value growth and engagement at any cost, but she insists they do not have to.
Why isn't a strong design process enough to make a product ethical?
Design is a skill that can serve many purposes and usually reports in to the business, so ethics must be embedded in every department and every step of creation.
Is regulatory compliance the same as acting ethically?
No. Compliance is a minimum bar mostly about risk mitigation, while ethics is about how you contribute, influence and lead, and it takes practice and courage.
What is the most important question to ask about a product?
Who will be the most affected by it, not who will pay for it or necessarily who will use it.
How can a company do thorough research and still build an unethical product?
When research is driven by the customer's point of view or an existing agenda, teams miss important information while conducting or synthesizing it.
About this talk
Innovators and technologists are in an extraordinary position to create experiences that impact how society interacts with nearly every facet of our world. In this talk, Carey Jenkins - CEO of Substantial - shares her perspective on how the intersection of technology and ethics leads to the development of more compelling products and more conscious innovation cultures. She unpacks how technologists have not just a responsibility, but also a mission to create more intentionally with creativity, diversity, and deeper purpose. Intertwining insights from over 20 years of professional experience, Carey overviews her five key frameworks for ensuring a more ethical approach to innovation: Lead from Why, Change the OS, Reduce Blind Spots, Think in Systems, and Create with Foresight.
Yes, we are ruining the world, but we don't have to (74 s)
A $1 million alert that fired 230 million times (74 s)
Compliance is not ethics (79 s)
Just because we're all in the same boat, at least for a little while longer, I am broadcasting directly from my daughter's bedroom. There's stuffed animals back there, her clothes. So I just thought you all would appreciate that, just in case you're wondering. So are we building products that are ruining the world? Yes, we are. In case you were thinking I was going to take a long time to get to the answer to that question, we can all leave now. Just kidding.
There's more to it. I used to have a long preamble that I would give at the beginning of this talk about the current state of technology, the unintended consequences, the harmful effects. But honestly, it was exhausting. And we all know them. And I suspect you knew how I was going to answer this question when you came to this talk. Because are we building products that are ruining the world? Yes. Yes, we are. But we don't have to. Technology is capable of making the world a better place. But the technologists have to choose that. We have trained a generation of entrepreneurs and product creators that the only thing that matters is massive growth and engagement at any cost. We have taught them to work for the highest bidder and that they
bear little responsibility for their individual contributions. We have crafted an environment where VCs make some of the most important early product decisions. And the quest for attention erodes exploration in the areas that most need our creativity. We have to untrain ourselves. Now, if you're curious who I am, I am the CEO of Substantial, which is a research, innovation and build studio. We're headquartered in Seattle, Washington. It is closed right now with many, many businesses, although soon to reopen. But until that moment, we are a fully remote and distributed workforce workforce of 50 strategists, designers, product managers and developers. And for 14 years, substantial has been conducting research, creating innovation and building digital products and services for startups, foundations and enterprises. We created products for some of the largest companies
in the world and some really small ones. And occasionally we even build products for ourselves. So I consider myself optimistic about what technology can do for the world. But these are trying times. And I find myself desperate to use whatever voice I'm given to convince people we can change. So this talk is not about passing judgment. It's about fostering a sense of individual responsibility and purpose that we can bring to every product we create. So today I'm going to share a case study with you. And then we're going to talk about how we can approach our work with integrity and a more ethical mindset.
So there are a lot of really amazing technology companies out there. I get to work with many of them, many I hope to work with someday. These are companies that are really trying to do everything right. Companies finding remarkable success building products we use every single day. But even the companies doing everything right can lose sight of the power imbalance that technology presents.
I'm encouraged that design is becoming a more prevalent voice and advocate for equity and responsibility. A constant reminder to understand the customer. But design is not a field that is immune from doing harm. And placing the burden of ethical interrogation on one department or role or even on a process ignores the realities of the decisions big and small that happen outside of design. Design is a skill and an approach. And it can be employed for many purposes. Remember, most design departments still report in to the business. So we shouldn't expect products to be inherently ethically designed just because we put them through a process. We can't expect ethical products to be created without embedding the practice of ethics throughout every department and in every step of creation. A singular focus on the
customer will not ensure technology that is responsible without interrogating who the customer is and why. So I'm going to tell you the story of a startup that did everything right. Acme HealthTech is a startup that has made a name for itself by tapping into the enormous demand for digital health information, particularly health records. Their free platform allows physicians to document patient visits, order prescriptions and lab tests, and easily access their records online.
And this is very useful for doctors, particularly in rural and more small offices that really needed this kind of infrastructure and support. Acme had experimented with other revenue models, but research oriented businesses in the healthcare space take years to develop. And in the meantime, pharma advertising is a huge pool of spend and a straightforward sales opportunity. But once you choose that ad driven path, commercial gravity tugs in the direction of selling more ad inventory and creating more effective advertising tools.
Backed by a high profile venture firms, Acme had reeled in over 150 million in private funding. And they had some really big mouths to feed. So what if they turned all of their data into insightful information for their advertisers, particularly pharmaceutical companies that never had access to granular weekly data on their drugs? Utilizing something called clinical decision support is not new. It can be applied to patient care in an infant number of ways and it has been. Vaccinations is one such way. Previously, Acme worked with a pharma company to develop workflow tools based on CDC guidelines that would try to boost pneumonia vaccinations in the elderly. And that work was viewed very positively. So why not apply it to other uses?
Partnering with a pharma company, they started with a research study to see what kind of CDS would work with patients suffering from chronic pain. The idea was to get the pharma company's pain drugs to certain kinds of patients. Patients who weren't currently taking pain drugs or those being prescribed the company's less profitable products. It also aimed to secure longer or consistent prescriptions.
Their research found some very clear and actionable insights. One of the things they found is that overuse of medication and pain treatment was already a national issue in gaining attention. They also found that reminders have a sustained influence on the rate of opioid prescribing. So the Acme design team got to work creating a seamless UI to help doctors in real-time patient care by providing helpful suggestions based on certain triggers. Despite the suggestions looking like unbiased medical information, Acme allowed the pharmaceutical companies to participate in designing the CDS alerts, including selecting the guidelines used to develop the alerts, setting the criteria that would determine when a healthcare provider received an alert and in some cases even drafting the language used in the alert.
For this product, the drug maker paid Acme a million dollars. And from 2016 to 2019, it went off 230 million times. It may be difficult to imagine the internal decision cycle that approved this work without triggering some ethical or legal alarm bells. In a business of this size, perhaps a one million dollar deal might not have been discussed at the board level, but 14 of them certainly would.
I want you to think about how many leaders, managers, analysts, researchers, consultants, designers, developers, QA people worked on this product. Imagine all the meetings to discuss KPIs and customer journeys and use cases and features. This wasn't an unintended consequence. This was very intended and very effective. And in how many of those meetings were the objectives, intent and success metrics deeply discussed or even questioned?
In how many meetings did anyone raise objections? We don't honestly know. Exploitation is a really strong word. But technologists laugh when we say if the product is free, then you are the product. I have a lot of concerns about technology companies, but my biggest concern is an industry culture that prioritizes revenue and speed above all else and deprioritizes the lives of users in general and vulnerable populations in particular.
And if you're thinking regulatory progress and legal requirements are going to solve ethical friction, know that compliance and ethics are not the same thing. Compliance is a minimum bar for social responsibility, but it's mostly about risk mitigation. Ethics is not as black and white as regulations and law, but it is about how you contribute, influence and lead. Acting ethically takes practice and courage.
This is why ethical mishaps are thornier and frankly more damaging. They can happen in such tiny steps. We don't know we are moving until the line we have crossed is miles behind us. Now it's worth mentioning that the employees of Acme HealthTech seem just like you and me. I don't get the sense they chose worsening opioid addiction as their career paths. And for what I could tell, their design team seemed diverse and close knit and focused on quality design practices. Their design principles, while this product was being created and deployed, were published on their website as three words: simple, focused, and trusted. And I imagine the pharmaceutical companies would wholeheartedly agree that they were those three things. There's no point hoping to burn this whole thing down. Technology
companies are amassing more and more power. Startups are being created every hour. The economy and marketplace reflect the quest for the next unicorn and users increasingly reliant on products. But companies are made up of individuals. What if we try to convince each and every technologist that they have a personal responsibility to practice more ethical decision making? How might we rethink our everyday decisions to ensure critical thinking is the most important objective?
An integrity framework is broader, deeper, and more demanding than a legal compliance initiative. It doesn't replace design thinking or compliance. I am not an ethicist, and it's very possible that your work could benefit from that expertise. But I believe strongly that every technologist should be empowered to think and act critically. So what might that look like in practice?
In many companies, the very structure employees and teams are in diminishes the feeling of agency and personal responsibility. That is not a coincidence. We need to change the operating system that products that products are being built in. Are directives and mandates coming from above without discussion or involvement at every level. What is the message being sent from incentives in your company? Is a large percentage of compensation in equity, for instance? Are people bonus for hitting only revenue and profits? The result is going to be a bias towards shortcuts, speed, status quo, and poorly thought out and poorly thought out implications. What kind of accountability is framed in your structure?
Huge competitive atmosphere? Growth and high turnover all the time? Constant reorgs and leadership changes? We may be telling ourselves that the meritocracy will serve us, but it has proven not particularly adept at ethical behavior. If there are competing priorities, competition for resources, a very clear response would be to value only objectives without factoring in the larger picture.
Now, the most important question to answer up here is actually who will be the most affected by your product? Not who will pay for the product. Not necessarily who will use the product. And to condition ourselves to ask that question above all else requires rethinking the operating system that most companies create products in. So should we sell more pain meds or gain efficiency for physicians?
You can imagine this might have been the question asked early on at Acme Health Tech. And it seems an easy question to answer if the pharmaceutical company is paying for the product. But remember, efficiency for physicians is why the company was founded. And you can't increase efficiency for physicians without keeping one very important mandate in mind. First, do no harm.
So selling more pain meds was the easy answer and the road to more revenue. But it lost sight of their company mission and traded in their values for cash. Had they chosen efficiency for physicians, realized the connection to positive patient outcomes, that would have created a very different product. And if the leaders in the company don't set a very clear example of what is important, teams will make their own assumptions.
Every use case, every KPI, every feature is an opportunity to lead from why. What does successful engagement mean in your product? Are you engaging in open dialogue and critical thinking across all of your teams? How do you define success? And does it factor in humanity, community, and society? And most importantly, could you disclose your intentions openly to all of your teams, all of your customers, and all of the users?
If the answer is no, then it's pretty obvious something is amiss. You need to reduce your blind spots and believe me, you have them. ACME did robust research, but they did it with an agenda to support an already troubling objective with a customer that had an obvious conflict of interest. What if their research had focused on doctors and patients?
Many within the tech industry position themselves as the actors best suited to address ethical challenges, rather than less technically inclined stakeholders and advocacy groups. This can manifest in relying on technologists to use their personal judgment to discern and evaluate the ethical mistakes of their own products. And no matter how skilled you are in critical thinking, there are times when getting neutral expert opinions is warranted.
You could certainly hire an internal expert that immediately creates a conflict of interest and fiction. The most public recent example of this is with the Google AI team. So how much do you know about the most vulnerable people in your system? Is your research agnostic of any agenda? Because if your customer POV is driving what you look for, you will miss important information, either when you're conducting research or when synthesizing.
And that's how you're conducting research. And that is how you can do hardy research and still not create an ethical product. And for many companies, independent specialists empowered to do equity-centered audits, research, and systems thinking is the most effective solution. So products are not isolated. They are part of an ecosystem along with the intended users like doctors and other actors like patients, insurance companies, and existing infrastructure and protocols.
So this is a very rudimentary version of a map of a healthcare system. But do you understand the system your product is living in? Can you map out all the dependencies, all of the connections? Because system changes take place gradually and are interconnected. Expectations of speed and scale and adoption can cause imbalances and unexpected consequences. What is the key to the system your product is living in?
The entity whose existence all other actors depend. In the case of Acme HealthTech's product, the key was prescribers. And if we're asking ourselves if the product improves the system, changes it, or eliminates part of the system, In the case of Acme HealthTech's product. In the case of Acme, the product changed the way prescribers made decisions and improved the speed of those decisions.
But if we're thinking about how the product might change the system in unintended ways, the Acme product made the quality of the doctor's decisions worse. And it deprived the patients of an opportunity for different kinds of care and tipped the scales even further in addiction to painkillers. So what if the key to the Acme product system had actually been the patients?
And it could be argued that any product in healthcare should center the patient, because that is, in fact, the whole point of healthcare. So once you understand the system, you can map out how it might evolve. And this isn't an empty exercise. There are trends that are affecting the system locally and globally. You need to know what the system's going to look like in one, three, five years.
You need to know how your product's going to grow within that system and how your success metrics will work within that. And you need to know how much runway you have to actually bring value. Because I can't tell you how many products are vastly not built the way they should be, because they did not have enough time and money to really work on the value stream that was most helpful for their users.
So we all know good products have been built in bad environments. There is a cost to product builders or to the users, but someone paid. For better or worse, technologies are driving the economy. The way we approach our work, our responsibility, our accountability will affect research, innovation, regulations, and the very future of work. We owe it to ourselves, our customers, our teammates to create an environment where the richest, thorniest, and most important questions can be explored without impunity.
That is how you change the world with great products.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
How do well-meaning tech teams end up building harmful products?
Carey Jenkins, CEO of Substantial, traces how a health startup's prescribing alerts went wrong and what every technologist can do differently.
In short
Carey Jenkins, CEO of the research, innovation and build studio Substantial, answers her own question without hesitation: yes, the tech industry is building products that are ruining the world, but it does not have to. Through the story of a health records startup that let a pharmaceutical company shape the alerts doctors saw when prescribing pain medication, she shows how ethical lapses grow out of ordinary incentives and structures. Her remedy is not a single process or department but a set of practices every technologist can carry into daily work.
Key takeaways
The enormous power technology wields requires us to hold ourselves accountable for the role technologists play every day in blurring ethical boundaries.
Ethical practices cannot be siloed into a process or team; they must be embedded and empowered throughout your culture.
The path to revenue does not have to be paved with ethical trade-offs: how we build is as important as what we build.
A generation trained for growth at any cost
Jenkins opens by skipping the usual catalogue of technology's harms, because, as she says, everyone already knows them. Instead she points to the culture that produces them: entrepreneurs and product creators taught that massive growth and engagement matter above all, that they should work for the highest bidder, and that venture investors make many of the most important early product decisions. Her stance is optimistic but firm: "Technology is capable of making the world a better place. But the technologists have to choose that." (0:38)
She also warns against handing ethics to designers alone. Design is increasingly a voice for equity and responsibility, but it is not immune from doing harm, and most design departments still report in to the business. "Design is a skill and an approach. And it can be employed for many purposes." (3:32)
The startup that did everything right
The heart of the talk is a case study of Acme HealthTech, a startup offering physicians a free platform to document visits, order prescriptions and lab tests and access records online, which was especially valuable for rural and small practices. Backed by high profile venture firms and over 150 million in private funding, as the speaker cited, the company turned to pharma advertising as a straightforward revenue path.
After earlier positive work using clinical decision support to boost pneumonia vaccinations among the elderly, Acme partnered with a pharmaceutical company on alerts for patients with chronic pain. The pharma company helped select guidelines, set triggers and in some cases draft the alert language, even though the suggestions looked like unbiased medical information. According to Jenkins, the drug maker paid Acme a million dollars and the alert went off 230 million times between 2016 and 2019.
She asks the audience to picture every leader, analyst, designer, developer and QA person who worked on it, and every meeting about KPIs and customer journeys. "This wasn't an unintended consequence. This was very intended and very effective." (8:49) The employees, she notes, seemed just like everyone else, with a diverse, close-knit design team whose published principles were simple, focused and trusted.
Compliance is not ethics
Jenkins cautions against expecting regulation to resolve ethical friction. "Compliance is a minimum bar for social responsibility, but it's mostly about risk mitigation." (10:06) Ethics is less black and white and takes practice and courage, which is why ethical mishaps are so damaging: "They can happen in such tiny steps. We don't know we are moving until the line we have crossed is miles behind us." (10:33)
Since companies are made up of individuals, she argues, the goal is to convince each technologist that they carry personal responsibility for more ethical decision making, supported by an integrity framework that is broader and more demanding than legal compliance.
Change the operating system and lead from why
Many company structures diminish people's sense of agency, and Jenkins says that is no coincidence. She invites leaders to examine mandates that come from above without discussion, compensation weighted toward equity, bonuses tied only to revenue and profit, constant reorganizations and high turnover. Those conditions breed a bias toward shortcuts, speed and poorly thought-out implications.
The central question, she says, is different from the one most teams ask: "Now, the most important question to answer up here is actually who will be the most affected by your product? Not who will pay for the product." (13:42) For Acme, choosing to sell more pain medication rather than to make physicians more efficient meant losing sight of its mission and, in her words, trading its values for cash. Every use case, KPI and feature is a chance to lead from why, and a simple test applies: could you disclose your intentions openly to your teams, customers and users?
Reduce blind spots, think in systems, create with foresight
Acme did robust research, Jenkins notes, but with an agenda that served a customer with an obvious conflict of interest. "You need to reduce your blind spots and believe me, you have them." (15:42) She recommends research that is agnostic of any agenda and, for many companies, independent specialists empowered to run equity-centered audits.
She then maps the healthcare system around the product, asking teams to identify the key actor on whom all others depend. For Acme it was prescribers, and the product made their decisions faster but worse. Had the key been patients, the product would have been very different. Finally, she urges teams to anticipate how the system will look in one, three and five years, and how much runway they have to deliver real value. She closes by calling on technologists to create environments where the richest and thorniest questions can be explored, because that is how great products change the world.
Watch the talk
"Are We Building Products that are Ruining the World?" by Carey Jenkins (about 21 minutes): https://www.womentech.net/video/are-we-building-products-are-ruining-world
About the speaker
Carey Jenkins is CEO of Substantial, with almost 20 years of experience leading large interdisciplinary teams and multi-channel digital projects. She is passionate about the intersection of ethics and technology and committed to leading a technology company that is thoughtful and intentional about its people, products and community.
Questions this talk answers
Are tech companies building products that are ruining the world?
Why isn't a strong design process enough to make a product ethical?
Is regulatory compliance the same as acting ethically?
What is the most important question to ask about a product?
How can a company do thorough research and still build an unethical product?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Lead from Why
Change the OS
Reduce Blind Spots
Think in Systems
Create with Foresight
Clinical decision support
Integrity framework
Design thinking
Cite this talk
Carey Jenkins. "Are We Building Products that are Ruining the World?." WomenTech Network, 2021. https://www.womentech.net/video/are-we-building-products-are-ruining-world