Not Another AI talk - A PM's practical tips to building AI Products
Navdeep Martin18:392022
Your next AI product might not need machine learning at all
An engineering team spends four months building functionality on deadline, and at the last minute nobody can explain it to users. Navdeep Martin, VP of Product Management at Blackbird.AI, who led personalization at The Washington Post, has watched AI products stall on poor design, feuding teams and runaway infrastructure costs. For product managers shipping machine learning, the question is which habits separate a working AI product from an expensive experiment.
Get six field-tested tips for designing, prototyping, staffing and costing AI products
Navdeep Martin, VP of Product Management at Blackbird.AI, shares six practical tips for product managers building AI products, drawn from her work at The Washington Post, Comcast and AI startups. She argues that design, parallel prototyping, team trust, honest cost estimates and hands-on data tagging matter as much as the models, and that teams should first ask whether a problem needs machine learning at all. She closes by encouraging women in AI to ask their questions out loud.
“My point here is take a look at the problems you're trying to solve and determine the fast and most effective solution. You might not even need machine learning.”
“It doesn't matter how good your machine learning models are if no one wants to use your product.”
What you will learn
Run an ugly machine learning prototype and a design prototype in parallel, then validate both in back-to-back user interviews.
Judge whether a problem truly needs machine learning or whether regular expressions, rule-based logic or open source datasets will do.
Estimate model, storage and GPU costs on the back of an envelope before committing to costly infrastructure.
Build respect and trust across product, data science, design and front-end engineering from the start of a project.
Tag data yourself to uncover edge cases and make product decisions such as renaming or retraining a classifier.
Key takeaways from the speaker
Helping women find creative solutions to building a well-functioning product.
Discovering a new field of technology they may be excited to dive into.
Sharing best practices on tagging data, prototyping and productionizing the AI work coming out of your AI team.
What the future looks like for women in AI.
Details
Level
Intermediate
Format
Talk
Technical depth
Introductory technical
For
Early career, Mid-career, Senior individual contributor
Length
18:39
Published
July 2022
Questions this talk answers
How should a product manager prototype an AI product?
Have the machine learning team deploy an ugly prototype with no design effort while the designer builds wireframes in parallel. Then show both in back-to-back user interviews, updating the design between sessions.
When should a team use machine learning at all?
Navdeep Martin advises looking for the fastest, most effective solution first, since regular expressions, simple if-then rules, open source datasets or published papers often solve the problem. Engineers can then focus machine learning on the hard problems.
Why should product managers tag data themselves?
Classifier results always contain edge cases that are not clear from the start, and tagging data helps the PM understand them. That knowledge supports product decisions, such as renaming an elections classifier to elections and voting.
How do you keep AI product costs under control?
Do back-of-the-envelope cost estimates before going too far, to make sure customer demand will justify the cost of models, data storage, transfer and GPUs.
What advice does she give women advancing in AI careers?
Begin asking your questions, even in large settings. Tech keeps evolving, you gain more by pushing the conversation, and someone else in the room usually has the same question.
About this talk
Building AI-products requires a completely different process and mindset. There are creative solutions to building a well-functioning product. In this session, I'll run through examples of AI in different products. Then we'll go through tricks of the trade including best practices on tagging data, prototyping and productionizing the AI work coming out of your AI team.
Does this problem even need machine learning? (90 s)
Why PMs should tag data themselves (88 s)
The class where she stopped asking questions (75 s)
Navdeep Martin: My name is Navdeep Martin. I am today going to talk about specifically towards product managers and direct this talk on how to build out AI products. So while this is for a product manager, I think this could be applicable to basically any of you guys building out products. These are tips of the trade that I've kind of learned along the way. A little bit of my background. I have a seven and eight year old. I practice this deck on front of their stuffed animals. So hopefully that should have led to some good content.
You should have seen them all. They were all watching me. But on a more serious note, I have a computer science background. I did an engineering path right out of school. I worked at the CIA for 10 years. And then my machine learning product management expertise actually really started at the Washington Post. I was there at the right place at the right time. Bezos had wanted to basically start a personalization program and I got the opportunity to lead that. And then moved over to Comcast to try out machine learning product management in a new vertical, which was TV and movie recommendations. And after that, moved over to two startups in the space at Primer AI was one of them. I led the machine learning strategy with our flagship product. And then joined Blackbird AI
just recently Q4 of last year as their VP of product. We are using AI to detect online narratives and risks for portion 500 companies and national security agencies. To give you a sense of where I'm headed, I'm going to start out with some definitions of terms that I mentioned in my deck. So just want to make sure we're all speaking the same language. After that, I'll go through my five tips for AI product management. My ninth grade teacher, you always say to show and not tell. So I promise to give as many practical examples as I can. And finally, I'll leave you with my own reflections for the future of women leading breakthroughs in AI. So let's get started on my definitions here. First off, just kind of get us
on the same playing field machine learning. Machine learning is a branch of AI. You're giving a computer examples of data and it's deriving inferences to make predictions in the future. This is an example of what machine learning could look like in a product. This is an example from my own experience and recommendations. What you see here is a recommendation module and everything you see here is a recommendation from the products chosen. In this case, the products here are news articles. Also, the images are recommendations as well as the headings. So the titles, you know, the titles of the articles themselves.
A machine learning algorithm is determining the optimal recommendation for these three different situations to yield a click from users, which would be you and I. Backing up though, how do we get to this? First, you need training data. This is kind of a crude example here where I have a list of news articles, their titles. You see a first paragraph and a URL to go skim the article if I desire.
So what I would do here is have a subject matter expert to go in and tag whether each article is about fashion, parenting, or national security. The three columns to the right. The finished version of this spreadsheet is essentially my training data. A data scientist will take this and train a supervised machine learning model to classify if these articles are about fashion, parenting, or national security.
And the finished result is a classifier for these categories. Machine learning is prevalent in your everyday life too. Here are some examples kind of from my own feeds. So the first one to the left is a grocery store app. If you think about it, grocery stores only have a certain amount of real estate in their apps. And so they really need to make the right recommendation at the right time. And then the middle section is actually my Pinterest board.
What I found really interesting about this is about in January or so, my husband and I decided to go vegetarian. And I didn't like mark some setting in Pinterest to let Pinterest know I had gone vegetarian. But somehow over time, the recommendations shown to me were actually began to be vegetarian. And I mean, what's happening here behind the scenes is there's a recommendation module and that's taking the input, my data input of not clicking on meat recipes as data when deciding to decide on the optimal recommendation for me. On the right, you'll see an example from the company I work for, Blackbird AI. This is an example of image analysis. So what can happen here is that customer searches for brands and risks, and through our machine learning image analysis, we can show images that match with that search.
Alright, those definitions were not comprehensive, obviously, but it'll give you kind of a starting point for my five tips, which I'll go into next. First off, design matters a lot. How big you make your text? What did you name your heading? Most importantly, how will customers use this? Some industry examples from my own past. Companies like Comcast found that titles like More Like This and People Also Watched were much better headings for the recommended program sections over something like You Might Like. The Washington Post found that having an image preview next to each article actually increased their click-through rates.
Another company found that if they showed their work and allowed users to understand how they came up with the categorization, even if they got that categorization wrong, users were forgiving of that error because they understood how the machine came up with it. The point is your designer needs to play center stage in building out your product, which leads me to my second tip. You want to design and build your ugly prototypes in parallel. What I like to do is actually have my designer tinkering away while my machine learning solution is being built. My process that I follow is actually to work with my machine learning team to deploy an ugly prototype. I call this an ugly prototype because no thought should be put into the design
at all. The point is to showcase your solution. In parallel, I'll provide my designer with some loose requirements and ask them to develop wireframes of what the workflow could be. So let's give an example here. I think this makes this make more sense. Let's say I'm building a news article virality predictor. My marketing team wants to amplify possible viral articles and they want an alert every two hours of the day to get a ranked list of anything they should act upon. And they want some level of explainability here too. They want to understand why we came up with the recommendations on this list.
So what I would do here is work with my machine learning team to explain the need by writing a one pager. My one pager might describe my users, the use case, what else is on the market that might do this. I might also include how my users do this work today and how they could do that work in the future. At the same time, I'll begin working with my designer to begin creating design prototypes to share.
When I finally have something ready to show, I'll begin my roadshow. I'll essentially schedule user interviews back to back as much as I can and conduct user interviews where I'm showing both the ugly prototype and the design prototype. I'll change the design prototype, incorporating users feedback in between interviews. And at the end of this exercise, I have validation that the machine learning model is doing what my user needs and I know I've built the right user experience. Let's move on to tip number three. You kind of want a team that likes one another. You might be thinking, how do I control for this? Or this isn't my job. But as a PM, I would argue your job is the success of the product. If your
team doesn't have basic respect for one another, you'll never be great. A few examples from my past, I've seen situations where a PM and designer cooked something up for a couple months without ever involving an engineer. When the engineers were finally brought in, they tore it apart, creating a permanent finger pointing scenario. Or I've seen where the team self-selected into the cool kids versus everyone else. The environment deteriorated before our eyes. In this case, the PM began to assume the designer role. And if you remember my tip number one, designers go to school and have reasons for everything that they do. Design matters a lot. A good team will appreciate the value of good design.
It doesn't matter how good your machine learning models are if no one wants to use your product. Or the worst one I've experienced was when the engineering team spent four months building out new functionality on a deadline. There wasn't enough time, I'll put that in quotes, enough time for the PM to do product discovery. And at the last minute, the PM and designer were brought in out of realization that no one could even explain the functionality to external users. There were countless design emergency meetings where the team tried to determine the UI based on a very limited backend.
Because the team, the engineering team had spent four months building out the functionality, it was really too late to reverse product decisions that had already been made. My point is here that you'll never build something great without a positive relationship of all four roles you see here. But let me kind of dig into that because I've been to organizations where it's actually not clear who does what.
So as a PM, as I mentioned, you'll need to use your soft skills, figure out how to break down barriers. Whether that's virtual team building that you tag an organization development person to help you with, if you have someone like that at your organization, or you can conduct them yourself. In the height of the pandemic, I've introduced doing Latin dance workouts off screen, of course, scavenger hunts, casual phone walk and talks where I just have a one on one with someone while we're walking outside. I've also done outdoor in person gatherings wherever I could.
My hope is always that I can get everyone's guard down and just have natural conversations flow where people are just talking to one another without my needing to be there. On a team, the PM also needs to set realistic and inspiring goals to build great products for customers and work with each team member to get that product to fruition.
At the end of the day, the PM is the decision maker, but we're essentially trying to bring everyone along the same ride. So let's hit each of the other roles here. The data scientist, for me, has often been the originator of the great ideas that I've ended up putting into products. You want them with you on your customer calls to hear directly from customers to improve their own machine learning models. Meanwhile, your designer, they're making large amounts of data accessible to a human being. They need to be involved right from the start. If you're B2B, that means they should be involved with all user meetings. If you're B2C, you'll want them intimately familiar with regards to what people are clicking on so that the user metrics. Your front end engineer is bringing your
designer sketches to life and they'll offer your designer additional ideas. My favorite kind of relationship that I've seen between the front ender and designer is when they're actually kind of pair programming, you know, elbowing one another. Hey, try this, try that in order to get to the best possible solution. Let's go to tip number four. This is really important. Does this problem even mean machine learning? I have to say, oftentimes when you've hired a machine learning team, the mindset goes to how can I build the coolest thing to solve this problem versus the more basic.
What's the best path to solve this? The best times to use machine learning is when a person can complete the same actions the machine would. I found that regular expressions, for instance, which is a rule-based search pattern, can be used in place of machine learning with very positive results. I found that there are often open source data sets, open source algorithms or papers written solving the same problems a new team might be facing. I found that customer or interns can be your subject matter experts tagging data with positive results. I found that you can use rule-based logic to remove things that are problematic. For example, in a previous company, we were analyzing news articles. We found that stock ticker trades were showing up in our results. These were articles that had a very peculiar pattern and there
were thousands of them and they were messing up our data results. We found that writing a simple rule-based if-then statement allowed us to remove these articles from our analysis and we could spend our machine learning engineers to solve the more hard problems. My point here is take a look at the problems you're trying to solve and determine the fast and most effective solution. You might not even need machine learning. Moving on, tip number five. Machine learning models are costly. Storing and transferring data is costly. You really need to look at whether you have the customer demand to support this. I'll give you a couple of examples in my past life. I've seen a situation where the company's leadership initiated a very costly replatforming effort. This effort involved the entire engineering team and it took
a year to do. The team did not conduct a cost analysis until the effort was nearly complete. What ended up happening is once deployed, they ended up with a very costly infrastructure to maintain and they couldn't find customers to cover the costs. I've seen another situation where GPUs are used in development to improve features, but they never get into the product because of the expense of running a GPU. My point, my conclusion here for this is do some back-of-the-envelope cost estimates before going too far to ensure your customer demand will justify your costs.
I have a bonus tip here. Tip number six, if you will. As a PM, you really need to roll up your sleeves and tag data. How can you explain to tag how you want data to be tagged without doing it yourself? In my experience looking at classifier results, there are always edge cases that aren't clear from the start. I'll give you a couple of examples here. I've seen a situation where a data scientist on the team trained a finance classifier.
They only use documents with the financial context, unfortunately. And when a completely different customer came on, a national security customer, and they were trying to review those same documents, we found that there were a lot of false positives. And I had to send that model back to the drawing board for retraining. Another situation, I found that we had decided to create an elections classifier.
But the peculiarly, we found that the results were often showing up for articles that were also about polling and voting, and that had nothing to do with elections. In this case, as a product manager, I realized from our users' perspective, they'd be okay with having polling and voting results show up within this classifier. So I renamed it. It was a simple renaming of elections to elections and voting. My point here is there are product management decisions to be made for classifiers, and you'll want to be very in tune with the data, at least from the start, so that you can be educated on what's happening and make decisions.
I wanted to leave you all with a few words on women trying to advance in the career in AI. And I'll have to say, as I sat down to think about what I wanted to say here, I really struggled with not wanting to appear to generalize. I still also wanted to be authentic with my advice. And so what I'm sharing here is my own growth story. And this may or may not apply to you, but wanted to share with you how I've kind of gone through my own challenges.
So I will take you back, if you can, we're going on a trip to college, junior year for me, William and Mary, I decided I was going to major in computer science. So I'm sitting there junior year. This is a college in Williamsburg, Virginia, if you're not aware. And I'm sitting in class, and it's, I don't know, maybe the first day of school, actually.
And our professor asks a question. He says, "What is the meaning of X?" I actually don't remember what his question was, but forgive me for not remembering the exact question. But he was asking, "What's the meaning of X?" And he was challenging the class. And I looked around, and I didn't know the answer, but no one knew the answer. No one was answering this question.
And the professor was like, "No, come on, guys, you know this. What's the meaning of X?" And finally, a guy in the back row, I still remember turning this way to look at him, and I remember exactly what he looks like. He says, "Is it blah?" And sorry, that terrible story, I don't remember what he said either.
But he said something like, "Is it blah?" And the professor turns and says, "Well, that's a stupid answer." The whole rest of the year, I didn't say a word in that class. And in fact, that was my personality anyway. I was afraid to ask questions. I didn't want to ask something that I should already know. So I proceeded and did well, got out of school and my first job. And somehow, over time, I learned to pull a close colleague aside when I had questions, and I'd ask them in a private manner my question.
But over more time, I learned that I do, in fact, know what I'm talking about, and I felt more confident to ask questions in larger settings. I don't want to generalize and state that women are afraid to ask questions. But as I sat thinking back what advice I wanted to give this group, I found that from my own experience, in order for me to thrive in this field, my advice for you is to begin asking your questions.
Tech is constantly evolving in that you gain more from a situation by asking your question and pushing the conversation. And I guarantee you there will be someone in the room who has that same question, and they're happy that you asked it.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Does your AI product even need machine learning?
Navdeep Martin of Blackbird.AI shares six hard-won tips for product managers turning machine learning into products people actually use.
In short
Building AI products takes a different process and mindset, and the model is only one part of it. Navdeep Martin, VP of Product Management at Blackbird.AI, argues that design, parallel prototyping, a team that respects each other, honest cost estimates and hands-on data tagging decide whether an AI product succeeds. Her sharpest advice: before building anything clever, check whether the problem needs machine learning at all.
Key takeaways
Helping women find creative solutions to building a well-functioning product.
Discovering a new field of technology they may be excited to dive into.
Sharing best practices on tagging data, prototyping and productionizing the AI work coming out of your AI team.
What the future looks like for women in AI.
Design first, and prototype in parallel
Navdeep Martin came to product management through engineering, including ten years at the CIA, before leading a personalization program at The Washington Post, working on TV and movie recommendations at Comcast, and leading machine learning strategy at Primer AI. She opens with simple definitions: machine learning gives a computer examples of data so it can make predictions, and training data is often just a spreadsheet tagged by a subject matter expert.
Her first tip is that design matters a lot. She describes how Comcast found that headings such as "More Like This" and "People Also Watched" worked better than "You Might Like", and how The Washington Post saw click-through rates rise when articles carried an image preview. Another company found users forgave wrong categorizations when the product showed how it reached them. "The point is your designer needs to play center stage in building out your product, which leads me to my second tip." (5:42)
That second tip is to build an "ugly prototype" with the machine learning team, with no thought given to design, while the designer works on wireframes in parallel. Using the example of a news article virality predictor, she explains how a one-pager describes the users, the use case and the market. She then runs a "roadshow" of back-to-back user interviews, showing both prototypes and updating the design between sessions. By the end, she has validation that the model does what users need and that the experience is right.
Build a team that likes one another
Tip three may sound outside a PM's job, but Martin argues that a PM's job is the success of the product. She shares cautionary stories: a PM and designer who worked for months without engineers, a team that split into "cool kids" and everyone else, and an engineering team that spent four months building functionality nobody could explain to external users. "It doesn't matter how good your machine learning models are if no one wants to use your product." (8:45)
Her remedies are practical. During the pandemic she ran Latin dance workouts, scavenger hunts, walk-and-talk one-to-ones and outdoor gatherings, hoping people would talk naturally without her needing to be there. She also maps each role's contribution. "The data scientist, for me, has often been the originator of the great ideas that I've ended up putting into products." (10:32) Data scientists should join customer calls, designers should be involved from the start, and front-end engineers and designers do their best work side by side.
Ask whether the problem needs machine learning
Tip four is, in her words, really important. "I have to say, oftentimes when you've hired a machine learning team, the mindset goes to how can I build the coolest thing to solve this problem versus the more basic." (11:20) She has seen regular expressions replace machine learning with very positive results, and found open source datasets, algorithms and papers that already solved a new team's problem. Customers or interns can tag data well.
In one example, stock ticker articles with a peculiar pattern were polluting a news analysis. A simple if-then rule removed them, freeing machine learning engineers for harder problems. "My point here is take a look at the problems you're trying to solve and determine the fast and most effective solution. You might not even need machine learning." (12:45)
Count the costs, then tag the data yourself
Tip five is about money. Models are costly, and so is storing and transferring data. Martin recalls a year-long replatforming effort with no cost analysis until it was nearly complete, which left a costly infrastructure the company could not find customers to pay for. She has also seen GPU-powered features that never shipped because GPUs were too expensive to run. "My point, my conclusion here for this is do some back-of-the-envelope cost estimates before going too far to ensure your customer demand will justify your costs." (13:38)
Her bonus sixth tip: "As a PM, you really need to roll up your sleeves and tag data." (14:14) A finance classifier trained only on financial documents produced false positives for a national security customer and had to be retrained. An elections classifier kept surfacing articles about polling and voting, so she simply renamed it "elections and voting", a product decision that only made sense because she understood the data.
Ask the question in the room
Martin closes with her own growth story rather than generalizations. As a computer science student at William and Mary, she watched a professor call a classmate's answer stupid. "The whole rest of the year, I didn't say a word in that class." (17:20) Over time she learned to ask questions privately, then gained the confidence to ask them in larger settings. Her advice to women in AI is to begin asking: tech keeps evolving, and you gain more by pushing the conversation. "And I guarantee you there will be someone in the room who has that same question, and they're happy that you asked it." (18:19)
Watch the talk
Watch "Not Another AI talk - A PM's practical tips to building AI Products" by Navdeep Martin (19 minutes): https://www.womentech.net/video/not-another-ai-talk-pms-practical-tips-building-ai-products-navdeep-martin
About the speaker
Navdeep Martin is Vice President of Product Management at Blackbird.AI, with over 20 years of experience building products for clients across national security, finance, e-commerce, crisis management, public affairs and media. She spent ten years as an engineer and over ten in product management, holds a B.S. in Computer Science from the College of William and Mary and an MBA from the University of Virginia Darden.
Questions this talk answers
How should a product manager prototype an AI product?
When should a team use machine learning at all?
Why should product managers tag data themselves?
How do you keep AI product costs under control?
What advice does she give women advancing in AI careers?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Ugly prototype
One-pager
Supervised machine learning
Back-of-the-envelope cost estimate
Cite this talk
Navdeep Martin. "Not Another AI talk - A PM's practical tips to building AI Products." WomenTech Network, 2022. https://www.womentech.net/video/not-another-ai-talk-pms-practical-tips-building-ai-products-navdeep-martin