Her AI risk engine was scrapped mid-build over one data category
An AI risk engine built for a bank's relationship managers gets scrapped midway because the legal team rules out the member financial data it depends on. Asma Syeda, a senior product manager with more than 15 years in IT, uses lessons like this to argue that AI products break the habits of traditional product management. What must a PM add before an AI idea deserves a place on the roadmap?
Learn the skills, checks and data habits that keep AI products from stalling before adoption
Asma Syeda, a senior product manager, explains how traditional product management practices must change when the product is an AI model. She walks through the PM role and product lifecycle, then covers the extra skills AI PMs need: vetting whether a problem needs AI, data fluency, realistic expectation setting, explainability, ethics and planning for adoption.
“So AI PMs need to ensure that AI POCs must match the organization's reality. Say no when machine learning is not a good candidate.”
“People need to know why they were rejected for a loan or why a model made a certain decision.”
What you will learn
Assess whether a problem truly needs AI using the five whys, value stream mapping and ROI baselines from comparable applications.
Build enough AI fluency in model training, feature engineering and technique trade-offs to make product calls without coding.
Plan training data, labeling and the model's delivery format early so the model holds up in production.
Set realistic expectations with leadership on timelines, accuracy and ROI, and agree on acceptable risk before deployment.
Clear legal, ethical and ownership questions up front so an AI product is not scrapped or left unmaintained.
Key takeaways from the speaker
Grow the Women in Tech and AI community
Connect with WTN
Evolve my on stage persona
Details
Level
Intermediate
Format
Tech talk
Technical depth
Introductory technical
For
Mid-career, Senior individual contributor
Length
50:25
Published
June 2021
Questions this talk answers
How is AI product management different from traditional product management?
AI products are heavily data driven and less UX focused, so AI PMs need data and AI fluency, use case familiarity and strong cross-functional buy-in. They must also prepare for a constant loop of iteration instead of shipping features every sprint.
How should a product manager decide whether a problem needs AI?
Ask the five whys, make sure proofs of concept match the organization's reality and say no when machine learning is not a good candidate. Use value stream mapping and the results of comparable AI applications as a baseline for ROI.
How technical does an AI product manager need to be?
They do not need to code or be data scientists, but they need a conceptual understanding of how models are trained, how feature engineering works, which AI techniques suit which data, and the AI project lifecycle.
Why do AI models fail when they reach production?
Models trained on general web data or data unlike the production environment perform differently on real data. Training data should be captured in the same conditions as production, and labeled data needs strong quality and privacy checks.
How can a PM drive adoption of an AI product?
Build explainability into model predictions so users trust the decisions, since team members otherwise fall back on their own gut sense. Then evangelize carefully and design for reuse through APIs and microservices.
About this talk
"The Future is Technology "and its not Complicated . The World is undergoing a revolution once again and technology is key driver of advancement in all industry areas , We have a whole new way of doing things now . Words and phrases like Digitization, transformation , startups, Artificial Intelligence , Machine learning , API's etc have become the most common words appearing on every content . As it seems pretty overwhelming new skills and frameworks like Agile, Scrum and product management have become a needed area of expertise to reimagine how we build on ideas to solve real problems looking through the lens of a customer . The big question .. Can product management be a good framework to build AI products with the challenge that they are not heavily UX focused . What specific approaches and skills do Product managers need to get their AI ideas to works and drive adoption
Without explainability, users trust their gut (47 s)
Asma Syeda: Hello, everyone. Hi, I'm Asma Syeda. I'm an AI product manager at a company called Alberta Motor Association. I'm from Canada and I'm here to speak about some product management practices and how they kind of like are relevant with AI because AI is different and it's different from traditional product management. So I'm here to speak about how we need to do things a bit differently when we are building AI products. I've got about 15 years of experience in driving innovation and transformation in different environments and it's all coming along with creating AI models ensemble with my product management experience along with my design, learning, thinking and lean management capabilities.
So I'm really excited to talk about some AI product management stuff. So before we do that, I know the world is undergoing, I just want to give a brief introduction on technology as itself. The world is undergoing a huge revolution once again, and technology, particularly AI, is the key driver of advancement in all the industry areas. And we have a whole new way of doing things now, like words, phrases, like digitization, transformation, startups, artificial intelligence, machine learning, APIs have become the common words on every content.
And it seems pretty overwhelming to write, like along with new skills that we need to kind of acquire, like scrum, agile, product management skills, which we have never heard before have become a needed area of expertise to reimagine how we build on ideas and to solve real problems, looking through the lens of the customer. And that's really very important. And it's what about it's all what product management is about. So what exactly is product management?
Let me share my screen. I think I'm sharing my screen. So yeah. Okay, so everybody must be familiar with this diagram, building the right product management. This one diagram where it involves skills around UX tech and business. But what does it mean to somebody who's really new to product management and has never heard about product management? So product management is all about building the right thing for the right user at the right time.
But we as PMs, who do we need to, who are we allying it for? So I think you have, you have familiar with this type of diagram, which is really very common on the neck when we talk about what customers really want, and what has been envisioned, and what people think about. So a PM is really an ally of customers who advocates for being the voice of the customer, right?
Like it's really important for someone to be in the shoes of the customer, have empathy for users, just because, because I think business who's sponsoring and funding the product, it's always thinking about benefit realization, ROI, or how soon can we get returns. And engineers who want to build the features, want to build it in and build the ones that are really easier and simple to deliver and also don't require a lot of maintenance.
Right? So in this kind of a setup, we need to think about the need of the customer, which is often overlooked. And I think this image really explains well about what a customer wanted, and what was understood and delivered, right? Like, so the customer described it as something else the business understood as the designers designed it as really something different understood different and the engineers built it as a totally different thing.
Right? Like, so I think product management is, is the one that bridges the gap between the customers who are users of the product, the business, and the tech. They strategize the problems into ideas and partner with internal and external stakeholders. to actually bring them to life. Okay, moving on. So a PM needs to understand their customers and their pain points, right? Like, so as a product manager, so what are their pain points? A product manager needs to work with the UX team, who I think they really start with when they are discovering how do they understand their customers and their pain points. So they have to partner starting off with, I am I am just calling it a different hats of a product manager that a product manager needs to wear. So they have to act like a psychologist. It's just not enough to know the customers. So PMs really care deeply about their customers and emphasize with their struggles and desires to get a full view of who they are. Right? So they need to be like a detective, learning to leave no stone unturned.
So PM needs to push to discover the hidden insights so that they have a complete and accurate view of the problems. Further on, they need to do a deep dive like a scientist. PMs need to regularly test their hypothesis using a mixture of structured frameworks and creativity to make connections, invent new ideas and solutions to customer problems.
So I think these are the critical steps. So I think these are the critical steps when PMs need to be aware of when they are kind of like doing user research or doing ideas around discovery. So after the completion of the discovery phase, next is to strategize talking to the business. How do you get your ideas? How do you influence people to kind of like build your ideas to subscribe to your ideas?
That's when I think PMs. That's when PMs need to put the head of a director to leverage creativity and talent to create and tell that story, right? To know when to know when to cut great ideas to keep the experience focused and memorable. That's really important. Also to keep up the momentum, PMs need to actually behave like a diplomat.
They need to drive the product strategy across all the organizations, right? Or to get all the teams involved, to subscribe to the vision, creating elevator pitches, influencing stakeholders for funding, nailing down OKRs and KPIs to show benefit realization, white labeling, or creating even the blue sky solution. You just don't want to create a solution which is just fit for one business use case.
You want to repurpose your solution. That's where your businesses would be interested, right? To actually invest more when they see that scalability and the benefit realization years after years, years after years. So next is the exciting phase. I call it the Artitect hack. That's when idea setting gets formalized into creative designs and blueprints. So it's the connecting the dots phase where PMs need to put on the Artitect hack.
They design and plan for an elegant solution that will delight customers also making necessary arrangements and trade-offs while partnering with departments to orchestrate that pipeline based on the reality of getting it built and delivered. So once they have finished their strategizing phase, what do they do next, right? Like you have all your things ready. You have your teams in play.
So next is, I think, the exciting phase where PMs need to demonstrate servant leadership. You have to really rally around the team by listening, being for them, entering, keeping the momentum high by influencing and motivating them like a coach. These are the people who need the food and water to be brought to them so that, yeah, you can, they can build the solution without any having roadblocks.
So that's where they need to act like an ER doctor, right? Like not everything goes as planned. So PM needs to quickly diagnose and steer the team from a very stressful situation to a positive outcome. Also, it's time to celebrate, right? Like, so like a gardener where PM needs to iterate and nurture and cultivate a delightful experience to see their product grow and scale beyond MVP year after year by monitoring and paying really close attention to the product metrics, the market changes.
So I think, yeah, different hats. I know it's overwhelming because, yeah, it's different hats like a product manager needs to wear when they are kind of like moving around, building things. But what does actually a product manager do? As I said, they, now that we know that the different roles of a product manager, we need to see how they need to leverage all these skills, tie them together and perform.
So what are the key responsibilities? I know they need to strategize when we've learned they need to develop the product. They need to ship it. But how do they do it within the within the realms of their job description or role? So you need to engage with customers on a regular basis. There are multiple channels to understand what the problem they might be, maybe like doing conductive or deductive research, understand what the problem that the product may solve.
So that's very important specifying market and prioritizing market and product requirements feature sets along with key positioning and messaging of elements collaborate with designers and engineers to solve problems. Curate, curate, communicate, and manage long term product relationships or the roadmap. Analyze external and internal key holders. There might be sales people or legal or compliance or design people that you might have to partner and work on on a regular basis.
You need to conduct relevant research and discovery on an ongoing basis. Your discovery doesn't always need to happen at the starting of the product. You need to have the discovery session going on and on to identify what is that aha moment that you need to create for your customers. Support the teams in communication, making sure all the teams are shipped to align your product so that it reaches the direct set of users, right?
Like measuring metrics, drawing insights, creating those feedback loops to help the product grow and mature, right? Like, and also don't forget your, to define your objective and KPIs and set targets so that your team can actually move towards them for success. Without having these, it's very difficult to actually measure what your success looks like. So it might be very ambiguous and cloudy when you have not set them.
So make sure you do that. So now that we have understand, I'll give you a moment to see if anyone has any questions. Okay. So moving on. So these are the product lifecycle stages and I've measured them against time and market, market research, acquisition, adoption, and retention. So you start with your discovery, ideation, and planning. That's where you strategize.
Who are your users of your product? What type of research you need to do? Should it be an inductive research wherein you are creating focus groups, going out, interviewing the customers, asking them, putting themselves in the shoe, in your shoes, looking everyone from a customer per customer lens? Or do you do a deductive research if you are your company or if you don't have resources to do that kind of a UX research?
How do you come up with a hypothesis? Create a lightweight prototype, test it, decide what your product is going to be called, the problem it solves, the business value. So you have to figure out all those steps before you actually decide on what your product would be or what your MVP would be. So planning also includes zooming out, having that long term vision for your product.
So just ask yourself those questions. How will my product look like one year from now? And also zoom in to see that short term vision, which is three to six months away. So what should be our first release, our MVP or minimum viable product as it is called. And now we have another term called as minimum marketable product, which is like a set of features that our product should contain without which the users will not be able to use your product.
So we don't want to build and keep on building until we have defined all the features. But we want to build something that is really quick and which can provide value to the users and the users can start using your product right after so that you can get that feedback and iterate on it. Then you need to figure out what your roadmap should look like.
What are your features that you need to build? And when do you need to build them? How do you want to prioritize and distribute them over your roadmap, right? Like, so what should be your milestones we need to target? And then how do we measure success? How do we kind of like, yeah, move on? What should be our delivery model?
Who should be our team's resources? What kind of expertise we need? Which type of framework should we be following? Would it be the agile, the scrum, Kanban? Do we need to do a PI planning and plan for every 10 weeks? How do you go about working around that? We need to identify when and how we need to work and what framework we would be following so that we can have this alignment with orgs and teams working on the business goal.
So yeah, next comes the building and the testing phase. Yeah, a really critical phase because as product managers, you'll be really very busy creating your product backlog, your sprint backlogs, attending sprint ceremonies, demos with business, getting that feedback, incorporating it, reprioritizing your backlog, iterating until your MVP is ready and good for release. And next, and next would be your introduce and test phase.
So once you are ready, like, you know, work with your peers to kind of like, you know, position your product into the market. You have to identify on who your buyer persona would be, plan your launches, who would be really willing to pay for your product. So identify your users first and then make them your users for your beta release.
Once your product is live, you need to have that feedback loop through various channels, like surveys, calls of like, you know, kind of like any kind of like feedback loops, which can measure the success of your product. And then also you need to actually plan for where would you be seeing this? You can't keep on listening to phone calls or listening to each individual surveys and seeing like, oh, what happened?
Why did my users drop off or what stopped them from reaching that aha moment? You need to tie them all into a dashboard. So it's very critical to work ahead with your data analytics team to make sure that you have you have plans for a dashboard or you have a dashboard in your MVP release, which would help you iterate and grow your product.
So once you have done all that, I think the next one is you have to start thinking as a product manager is how do I improve on my MVP? How do I scale it? How do I get more benefits or how do I get the ROI, most ROI out of my product? Now is that phase that you have delivered your MVP?
How do you scale and grow? So scalability should be the top goal when you start envisioning your idea. You don't want your problem to be limited, your product, sorry, to be limited to a subset of your users. You need to see how you can reach horizons on trying various different models, like, you know, in a B2C or a B2B environment.
Wherein you can partner with either internal businesses within your or external businesses to see how you can grow your product. You don't need to reinvent the wheel again and again. You have to focus on how we can repurpose or reuse our work and drive ROI. So once you've done all that, your product has reached the maturity level.
You have gained your ROI. So now it's the time to think about next steps. So because technology trend and frameworks, existing ones, start becoming a legacy over time. They start losing their shine and wearing out. Now you really have to start to think about how do I kill my first version? Because you already need to start planning on your second one.
So that's the time when you need to see how do you decommission your first version because maintenance of legacy systems becomes harder and we lose out on support because even bureaucracy of new tech kicks in, right? We are forced to migrate to new platforms. We are forced to migrate to new platforms if we need to stay in competition.
Sustainance is expensive. Investment is easier. So plan on your next generation of product and retire the current model. So I know it's hard decisions, but I think, yeah, that's very. So understanding the product lifecycle and then planning in tandem to get your roadmap, it's very critical because you need to have that clear picture of how your product lifecycle would be year after year.
But also keep a sharp focus when you're zooming in and seeing what should be the next set of features that would help my product grow from the version one to a version two or to get to that aha moment. So that's really very critical skill a product manager needs to have. Any questions so far for me? Cool.
Okay, moving on. Oh, sorry. I lost my screen. Okay. It's going to the next screen. So now that we know what product management looks like from a traditional perspective, what does it look like from an AI perspective? Product management, we need to, can product management be a good framework to build AI products with a challenge that they are not heavily UX focused?
What specific approach and skills do product managers need to get their AI ideas to work? Because everyone with a tech mandate is feeling pressure to use AI. So this diagram actually creates that distinguish distinction between what a product, regular product manager does and what AI product manager does. So AI product manager, since AI products are heavily data driven, they need to be really be willing to get their hands dirty in data.
They need a lot of AI fluency. They need a lot of AI fluency, you know, you know, you know, a use case familiarity, AI doesn't work for every and any, any use case. So you have to understand and assess that capability of product capability against which use case your product, your model would work against. Buy in an alignment, it's really very critical for AI product managers to work with cross functional teams and that comes with handling a lot of challenges.
So AI product managers need to be really skilled at that. So AI product management. So we are told that like AI can change the world, unlock hidden business values, solve problems that we are not able to do before. But how do we kind of like, how do we do that? Right? Like, so, but how do you even start?
What sort of considerations you need to make? How do you even go about building your first AI product? So who should first figure out how do we approach my AI product? Because most AI products are built for efficient, optimized operational efficiency or reduction in cycle times. Most of the food and figure out who your customers should be.
Would there be my internal teams within the organization? Or would there be, who would be integrating and using my within my AI product? Right? Like, so would it be my external users? For example, I created a risk engine, which would, because I was working in a bank, where I had to partner with different teams. So that because the product would be not necessarily used by external customers, but it was made for relationship managers who would be working with customer, external customers to manage their loans, to manage their financial portfolio.
So my customers were internal relationship managers who I was building the product for. So it's very necessary for you to understand who your product owner users would be. Who would be your core team? Because you would mostly be working with data scientists, analytics, data warehouse people, because that's where your data comes from. The integrations team, like the APIs or the CDKs, the ones who build your APIs or the CDKs and SDKs.
So until you have your microservices model with the architecture team, it's going to be really hard to scale your AI products. And mostly, AI product, because there is no lot of visual representation, like marks and prototypes, it's really hard to speak to the representation of idea. The identification of problem can be a big challenge, as AI PMs are not directly speaking to customers, so they don't know what the customer wants.
It all basically depends upon the users of your model. So as an AI PM, we always feel left out because we are always working behind the scenes. There's no motivation or influence because there's no spotlight. Nobody recognizes your product, right? So that's a challenge. So how do you feel motivated and keep up with the momentum? And most of all, I think it's really important to be, because we are called technical, because we need to communicate and understand the technical judgments to be the best fit with your technical team.
We don't want to be living with the FOMO effect. That doesn't mean that AI product managers needs to be technical. And it's not, we need to learn rocket science, but not everyone is good at coding. So PMs, and also PMs don't just have one role, right? Like they need to manage everything from inception to launch. So definitely there's some extended skills that AI product managers need to have apart from the regular ones.
So let's found out, find out what they are. Okay. So moving on. So the skills of an AI PM, business case evaluation, you are fit for AI. So start with your problem statements. In this phase, you decide what to do and why. So, right? Like smart product managers combine data along with analysis and judgment. Ask the right questions, right?
Like, so ask those five why. It's like, why do you need AI? Why do we need to build this product? Why do we need AI? Why do we need automation? What's the problem that we are trying to solve? Right? Like having enough depth to filter out the noise and focus on the worthy POCs. That's very important because they are not like, they are not just science experiments.
Executives whim or have really a lot of wishful thinking and they have no data or scaling possibilities. So AI PMs need to ensure that AI POCs must match the organization's reality. Say no when machine learning is not a good candidate. Make a checklist of things that you would need absolutely for your AI model to be a success.
Decide how much machine learning is to be used along with other approaches. You might need to, you know, partner, unite, you might need to combine it with the rules, automation. Right? Like so to enhance that success or to amplify that success. It's just not, AI will not alone give you that key results that you're looking for. Like so, first of all, to need to prioritize and manage the pipeline, we need to have all these things in alignment.
And then also, yeah, figure out what your resources and development looks like. Organization structure really matters. Understand and coordinate organization structure, right? Like so, you need to have data literacy, domain depth, geek creed. All those are important things. We need to partner with the right SMEs at different stages. And then how do we engage or engage support teams so that we are not roadblocked while we are sprinting.
So AI product manager is the glue that keeps all of those AI elements together. Speed and frequency of improvement matters. Be agile, smart. POCs are more important than ever. You can ever start with your POCs, create a series of MVPs, which are actually lightweight models. And also set your priorities and road mapping. Yeah. Do a value stream.
What are the features that we need to build as a part of our MVP? And how long it's going to take? How do we plan? How do we prioritize? The business wants everything to be delivered or every set of features to be included in the MVP. No, have a conversation with them. Set up your backlog, your delivery schedule, your risk that you may encounter, your mitigation plan.
And include trust and empathy right from the beginning. That's really important. So understanding also your ROI, right? Like so do a value stream mapping. So understand what your current process looks like. How long it takes for a stakeholder to complete one, to get from one state to another. Where are the bottlenecks? How much of it is the lead time and cycle time?
Measure all of that. And then come up with, this is a current process and this is like a 2B process. How will the enhanced process look like with incubating it with machine learning models? Are we able to remove bottlenecks? Will it reduce the cycle time or the lead time? Do we need automation to drive efficiencies? And don't reflect to forget on a few other things.
Is the result that we are aiming for really obtainable? That's one thing that you need to ask. And also what have been the reasonable results or the ROIs of other companies who have implemented the same applications of these kinds. So you need a baseline to measure your success against. Sometimes applications, sometimes an application or a model is really first of its kind and there won't be any precedent to it.
So in most cases, there will be some analog AI application that will give you some reasonable ROI. So you have to just, you know, measure it against that. Any questions so far? So I guess I just keep moving on. So another thing is like you have an AI apart from your AI strategy is a big thing is the platform you would be using, whether it would be a hybrid setup, a cloud setup, and then also what kind of data do you have?
So product managers need to have, to be able to understand the fundamental and terminology and concepts of artificial intelligence. It doesn't mean that they need to learn to code. Now does it, that doesn't mean that they need to talk shop with the IT department, but they, and they also don't need to be data scientists, but they have to be managers and leaders at a conceptual level.
At least they need to know how a machine learning algorithm is trained, how feature engineering works. So if the team asks you which model features need to be of high priority, for example, than the others among the enormous set of ones. So you might have a model that is kind of consisting of about thousands of features. So if your team asks, like, how do you prioritize, right?
Like, so how does our model make decisions and what would be the priority of your features? You need to be able to be in a spot to understand those questions. Also, what kind of AI approaches and techniques are out there? There are some predictive analytics, machine vision, natural language processing, computer vision, image processing. So which one actually, which of these techniques and approaches can be applied to your use case?
Because there are tons of ones that are available in the industry. Also, which type of algorithm works best for structured data versus unstructured data? For example, is the BERT model fit for my use case or would it be a regression one? You need to make those trade-offs, right? Like, so that's where a product manager skills need to grow, especially for a AI product.
And also the life cycle and the phases of the AI product. So it's understanding, the business understanding, the data understanding and assessing the product needs. A conceptual understanding of the life cycle of the product, of the AI and data science is one of the skills that I think most of the product managers are missing. Because they go into AI projects just like as if it's a regular one and they fail to prepare for the constant loop of iteration.
So if they do understand this, it's more likely that the C-level executives, they report, don't, they do not. So aligning leadership with reasonable expectations, it's very, for an AI project is a separate skill that needs to be nurtured. So also product managers often have to make sure that they develop or they can perform that incubation period in a way that also doesn't, doesn't disrupt the user's experience in a way that it is detrimental.
So they need to understand the challenges that come along with adopting artificial intelligence in an, in an enterprise. We have already spoken about ROI. Right. Also, it can take months to access and clean the necessary data to begin an AI product project. Really, it's not about the model, right? Like the, where we are shipping features out every sprint as the business is used to seeing in some cases, the business structure needs to be overall, including data infrastructure and, and some protocols.
All of these are subjected to change in the middle of the project. So in any of these challenging situations, which are a big surprise for an AI product manager, they need to be very good at communicating the realistic expectations with the leadership team. And they're also going to manage various different teams of the project. So setting expectations with stakeholders on deliverables, timelines, model accuracy, and ROI, and maintaining a regular cadence of transparent communication right from start.
That's really key. Okay. So next is commercialization. So how do you productize your product? How do you have the right people, the processes and tools? How will you continuously monitor and improve the performance of your product? So involve your subject matter experts right from the beginning. During the training process, if you're doing supervised learning, have your subject matter experts who are the domain experts to label data for you, because that will keep them motivated and involved.
Right? Like so getting business sign off early on the model accuracy and prediction, your subject matter experts might help with that kind of a thing too. I've seen AI artificial intelligence PMs struggle a lot with businesses to reach a target accuracy and precision state. It's hard. We need to negotiate on a sweet spot if you don't want to spend the entire project just attaining accuracy.
Okay. And end users. How will your AI team reach? How will your AI product reach the end users? So most of the times you need to pair up with another PM who needs to house your model inside their application. So what would be the format you would be delivering the model as? Would it be a JSON file or would you push results directly into a dashboard?
Work on those logistics early enough to give you enough time for the end user and the PM to work in tandem to develop those tools that is required to launch your product. Also, don't underestimate the importance of shared vision. Understanding effective communication to do that. AI initiative works best when multiple teams orchestrate work and it also reduces friction.
Develop shared understanding and work through different subcultures or bringing different roles together to learn about each other. So making data technical folks serve as evangelists, they provide raw access. Next is, I think, yeah, model inputs and outputs. There's no way around this. Have your training data as close as to be in production data as possible. So you might need a large amount of data, historical data, for creating training and testing data sets with various compositions.
You might need to have a data set which has 70% of training data and 30% of testing data. Whereas you might need another data set where you have to change it around. You have to test it like because you have to your your your sprinting would be comparing models to models against for accuracy. How would my 70/30 model perform with a 60/40 or a 50/50 or 80/20?
That's exactly what you need to be testing your hypothesis against. You'll need a lot of experiment until you get that accuracy that would fit your ROI. For example, a computer vision use case like the camera used to capture the image, its position, movement, lightening, etc. You need to match that clearly to the in production environment. So it might be tempting to just use publicly available data sets for your use case to kickstart your model training.
But model trained on general data from the web that are built or performs really different on the data that is built for the purposes. And it will definitely fail in production if you are building it from data from the web. So transfer learning or generating generating synthetic data might look at an interesting option when you're doing POCs.
But when you have to put something into production, yeah, you just you really have to be realistic to generate enough realistic results for your model. And same goes with the output of the model. If you're using supervised learning technique, you need annotated data for your model outputs, right? Like so in order to train your model, labeling data is really expensive.
And while multiple companies offer services for annotation, you know, you need to pick one that has got a really strong quality checks in place because you need to be aware of data privacy, security and other concerns. Right. Like so what people misusing your data, which is proprietary data, label data often is more valuable for the model itself because any competitor can get access to your label data and can build a similar product or an offering.
So make sure that your data cannot provide and can provide the same user benefits. So that's that's an insight which I have often got working with different industries in these teams is that product teams have successfully developed AI solutions with an in-house data intern who can actually label the data for them and manage the annotation tools or platforms.
So data is the fuel that actually fuels that your model learns from. And there's no good substitution for using data for training that has been captured in the same exact conditions as the production environment. So on the model output, getting ambiguous high quality data is more important than spending your time on actually improving the model itself. Again, explainability.
Yeah, trust plays a crucial role in customer satisfaction and retention and also overall business success. Explainable AI gives customers insights into AI decision making. So AI product managers are really responsible for including explainability reasons in the model predictions and ensuring trust in the model. People need to know why they were rejected for a loan or why a model made a certain decision.
They need explanation around that and we might be and we as AI product managers need to work around that. We need to respect the reasons for it and have those incorporated in the models. We need to build models more on explainability and it should not be a kind of black box, which most AI models are. Scalability. Yeah.
PMs must evangelize that the product project just more than the business must evangelize the project more just than the business case. They have to understand and promote the merits of AI for adoption and help the company stay competitive in the new AI world. And ethics a big part and it's often discussed whenever we are talking AI because AI PMs need to be ethical of what need to have an ethical consideration and a deep concern when they are building AI or machine learning tools.
It's up to the AI PM to often remember and remind others that with great power comes great responsibility, especially when dealing with data with regulations and privacy issues around it. Make sure you have required the necessary approvals and consensus before you actually dive in. I was in a spot at one time when I was building the same risk engine for my customers when I was not allowed by I was stopped by the legal team for using a certain data category.
Right. Like so I was not it was financial data of a member and it was not supposed to be used apart from whoever like apart from just, you know, using it to make some business rules decisions. And we were not supposed to use that in an AI model. So our product got scrapped right in the middle. So that's very key to understand your ethics and your privacy and legal constraints around it.
So do your homework beforehand. And also maintenance and ownership. AI products constantly need to be tuned and maintained to the right accuracy. So pay attention early on to identify your model ownership and work with your stakeholders towards an engagement business model. To sustain and maintain AI products with the right resources on your team. So either it has to be owned by the data scientists or the AI team or if you are sourcing it off to your or if you are passing the ownership off to your business.
Make sure that the business has the right capability and resources on their team to actually do tweaks and maintain to the model on an ongoing basis so that they don't disrupt the ownership doesn't fall on you. Any questions so far? Okay. Moving on. So building AI products is often referred to as a virtuous AI cycle, because there is no ship features out of the door at regular intervals and it's hard to explain to the stakeholders why AI products need so much of time to be built and implemented.
So these are kind of like the mandatory steps, right? Like that must happen when building AI products. So it starts with the data collection. So preparing customer data from whatever sources that you are using can be really a daunting task due to sheer number of displayed data sources and data silos that exist in organizations. To build an accurate model, it's really critical to do the right selection of data that is likely to be predictive of the target that you are chasing.
The outcome which you hope the model will predict based on the other data inputs, right? Like once you have collected your data, the next step is normalizing your data. So the next step is where the data analysts and the data scientists typically spend most of their time on analyzing the projects, cleaning the and normalizing the dirty data because the data is not stored in clean formats and structures and tables, right?
Like you need to kind of like structure it, add quality to it, make it like usable by kind of removing the noise, removing the things that we don't need. This is often, often requires data scientists to make decisions on data that they may not understand. Like what do they do with the missing data? Sometimes the data is missing, incomplete data, there are some outliers.
So how do we kind of like, do we need to include all of that in our data sets? It's not a good idea, right? Like, so that's not good ethics. So we spend a lot of time with the data scientists actually doing data normalization. The data may not be easily correlated to proper unit of analysis, right? Like, so in order to predict if a single customer will churn, for example, siloed data from disparate sources can't be relied on.
So a data scientist will prepare and aggregate the data from all the data sources into a format wherein the machine can interpret it. So this will end up being a very lengthy process and may require a lot of work before any machine learning project can even start. So next is data modeling. So the next phase of the machine learning project is to model the data that will be used for prediction.
So part of the modeling data for prediction about customers is to combine those disparate data sets to paint a proper picture of a single customer. So this includes blending, aggregating of silos of data, like from web or mobile app or offline data, and then putting it together so that they can come up with those data sets. Like, you know, that's called data modeling.
So once they have done that, they have the training and testing data sets. They start with the model training and the feature engineering. So after a brand has been deployed and collected and enrichment of a meaningful input data, it's time to put the predictive power to it, right? Like, so to test it. So what do data scientists do at this is, like, take representative samples of population, like, for example, all customers who are anonymous visitors or who are known prospects or, you know, set aside portions of data for training.
And the remainder is used for the training and the remainder is used for validating of the models after the training is completed. So the key component of this is to iterate rapidly and to continuously test new data points that can be delivered from various other data sources. So this is called as feature engineering and also prioritizing which data points need to have a higher allocate higher priority when the based on which the model makes predictions.
The last is like deploying models to productions. All of all work to at this point culminates into the final step of deploying a model to production wherein the ability to predict outcomes in the real world is tested. So by this point, models should be able to meet some kind of like threshold or accuracy, right, like that warranties them to be deployed to production.
So for this reason, it's really important to interpret model performance with stakeholders to agree on what is the level of risk that is acceptable for inaccuracy. So some customers' behaviors may not be sufficiently predictable enough, and thus a model may never be able to achieve that accuracy to justify deploying it to production. So in the end, machine learning is just isn't going to replace a digital marketing strategy, but will just rather augment it and enable it, right?
So successful brands will put their customers at the center of what they want to do with machine learning and one of them is machine learning to optimize decision making as a part of that larger initiative. Cool. So we have a couple of questions. Okay. So before I conclude, I just wanted to give some tips for AI product managers.
So AI and machine learning are here to stay and will continue to transform the way we interact and engage with each other and the world at a larger at large. So you need to figure out where your AI and machine learning can give your business a competitive edge. So they need to be AI PMs, need to approach AI initiatives differently by exploring data, first to identify and vet business opportunities, get a solid handle on your data and modeling technologies.
Is your organization really having that appetite to change and transform data, create data lakes, data sources? So first you digitize your data from unstructured data sources, like get all of the data into one place before you start building your AI models. And also get your hands dirty in data, get some data analysis skills, like different tools. There's power BI, Google analytics, which can help actually to kind of like make, make those decisions, help you understand, create dashboards on the spy to see what kind of data is available.
At your disposal. Also communication. AI PMs must understand how to answer, ask the right questions to the customer, right? Like so, and also they must continue to gather that data to fine tune it for ongoing AI executives, like effective cross collaboration with cross functional teams is what AI brings, AI brings in AI initiatives to life. Listen to each other.
Listen to each other, learn from each other. They also need to have that empathy for the customer, right? Like so they might, they must leverage AI and machine learning to deepen the understanding of the customer, have empathy, look at privacy with empathy, explore new solutions such as differential privacy. What will it take for the users to adopt my solution, right?
Like so use a human user centric product centric framework for error measurement and also challenge facilitate and deliver, right? Like so challenge your business. Is this the best use case for AI? Like so do we, how can we set on your objectives and KPIs earlier? Just do your homework, research back and see how many people in the past or how many organizations in the past have benefited from this type of use case or from this type of model.
Do they have metrics? What was the implementation strategy? What was their benchmarking? Challenge your business to get all those answers to you before you start building your AI models and then increase the trust and adoption, right? So it's really very critical to have people trust in AI and adopt AI because without trust, there wouldn't be a lot of adoption.
So I've come across in my experiences people, wherein in my organizations, my team members would not prefer to go by the decisions the AI models make and use their own gut sense and experience to make those decisions. So we are not, we are not like using that adoption, we are not encouraging that adoption if we are keeping AI, if you are not explaining AI to them, right?
Like, so if you don't have the reasons why AI took this decision or why the model failed to kind of like adjudicate a customer or make a decision on the customer, if you're not able to explain that, then our AI adoption will always be slow. And then evangelize carefully, like, you know, you're also look at things that you would like to scale for, right?
Like, so you have built this product for this AI, but then focus on the architecture, focus on the integration, focus on the API. How would this be able to, how can I convert it into microservices? So how would other, other areas of my business benefit from my model? Is there a way wherein I can scale, create that virality, create that network effect, that 10x, right?
Like the 10x experience. So, yeah, you have to evangelize it carefully because it takes a lot of time and effort and money to create AI models and you just don't want them to be sitting on their shelf. And last of all, perhaps most importantly, prepare for failure. Uncertainty is very high in AI initiatives. And above all, keep learning and use that knowledge to make self-improvements.
So, yeah, that brings us to the end of my talk. So connect me on LinkedIn. I'm available on LinkedIn. And then also you can shoot me an email, have any questions regarding AI or product management. Yeah. Feel free to reach out. I'll be most willing to help you. Thanks for taking the time to listen to me. I hope it was beneficial.
Yeah. And then, yeah, keep in touch and happy Wednesday.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
What product managers must do differently when the product is AI
Asma Syeda explains why AI products demand data fluency, hard use case filters, early ethics checks and a plan for trust.
In short
Asma Syeda argues that product management is a sound framework for building AI products, but only if product managers adapt it. AI products are data driven, less visible to end users and slower to ship, so AI PMs need data fluency, a hard filter on use cases, early answers on legal and ethical limits, and a deliberate plan for trust and adoption.
Key takeaways
Assess whether a problem truly needs AI using the five whys, value stream mapping and ROI baselines from comparable applications.
Build enough AI fluency in model training, feature engineering and technique trade-offs to make product calls without coding.
Plan training data, labeling and the model's delivery format early so the model holds up in production.
Set realistic expectations with leadership on timelines, accuracy and ROI, and agree on acceptable risk before deployment.
Clear legal, ethical and ownership questions up front so an AI product is not scrapped or left unmaintained.
The many hats of a product manager
Syeda starts with the fundamentals for anyone new to the role. "So product management is all about building the right thing for the right user at the right time." (2:03) In her framing, the PM is the customer's ally and voice, bridging a business focused on returns and engineers who prefer features that are simple to build and maintain.
To do that, she describes a series of hats a PM has to wear. During discovery the PM acts as a psychologist, a detective and a scientist, empathizing with users, digging for hidden insights and testing hypotheses. In strategy the PM becomes a director who tells the story, a diplomat who rallies the organization around OKRs and KPIs, and an architect who turns ideas into blueprints. During delivery the PM is a coach practicing servant leadership, an ER doctor when things go wrong and a gardener nurturing the product beyond the MVP.
Planning the whole lifecycle, including the end
Syeda walks through the product lifecycle: discovery and planning, choosing between an MVP and a minimum marketable product, building the roadmap and delivery model, building and testing, launching to a beta audience, and tying feedback channels into a dashboard planned with the analytics team. Scalability, she says, should be a goal from the moment an idea is conceived, with products repurposed across internal and external businesses rather than rebuilt.
She also asks PMs to plan for retirement. Legacy technology loses support and becomes costly to maintain, so the next generation should be planned while the current one matures. "Sustainance is expensive. Investment is easier." (17:29)
What changes when the product is AI
AI products, Syeda explains, are heavily data driven and lack the mockups and prototypes that make traditional products easy to discuss. AI PMs need AI fluency, use case familiarity and strong cross-functional buy-in, and they often work far from the spotlight. "So as an AI PM, we always feel left out because we are always working behind the scenes." (22:09)
Her first skill is business case evaluation. She recommends asking the five whys about why AI or automation is needed at all, and filtering out proofs of concept driven by executive whim. "So AI PMs need to ensure that AI POCs must match the organization's reality. Say no when machine learning is not a good candidate." (24:08) She suggests combining machine learning with rules and automation, mapping the current value stream against the improved process, and benchmarking ROI against comparable AI applications.
AI PMs do not need to code, she stresses, but they need a conceptual grasp of model training, feature engineering, AI techniques and structured versus unstructured data, for example whether a BERT model or a regression fits a use case. They also need to manage leadership expectations, since preparing data alone can take months.
Data, explainability and ethics decide success
On commercialization, Syeda advises involving subject matter experts early, including in labeling data for supervised learning, and agreeing on the model's delivery format with the PM whose application will host it. Training data should mirror production, because models trained on general web data behave differently on real data. "And there's no good substitution for using data for training that has been captured in the same exact conditions as the production environment." (36:41)
Explainability is central to trust. "People need to know why they were rejected for a loan or why a model made a certain decision." (37:04) Ethics and legal checks matter just as much. Syeda describes building a risk engine for a bank when the legal team stopped her from using a category of member financial data in the model. "So our product got scrapped right in the middle." (39:01) She also urges PMs to settle model ownership and maintenance early so tuning does not fall back on them.
The virtuous AI cycle and earning adoption
Syeda outlines the steps every AI product goes through: data collection, normalization of messy and siloed data, data modeling, model training with feature engineering, and deployment once stakeholders agree on an acceptable level of risk. Her closing tips include getting hands-on with data through tools such as Power BI and Google Analytics, looking at privacy with empathy and exploring options such as differential privacy, and challenging the business on whether a use case truly suits AI.
Adoption depends on trust. She has seen team members ignore model decisions in favor of their gut when the AI was not explained to them. "So it's really very critical to have people trust in AI and adopt AI because without trust, there wouldn't be a lot of adoption." (47:50) Her final word is realism: "And last of all, perhaps most importantly, prepare for failure. Uncertainty is very high in AI initiatives." (49:22)
Watch the talk
"Product Management for AI" by Asma Syeda, about 50 minutes: https://www.womentech.net/video/asma-syeda-product-management-ai
About the speaker
Asma Syeda is a Sr Product Manager at AMA, based in Edmonton, Canada, who helps shape and execute product vision and strategy using data, technology and clear objectives to solve complex problems and improve customer experience. With more than 15 years in IT, she draws on product management, critical data analytics, design thinking, lean management and AI capabilities, along with a strong ability to empathize with her team and customers.
Questions this talk answers
How is AI product management different from traditional product management?
How should a product manager decide whether a problem needs AI?
How technical does an AI product manager need to be?
Why do AI models fail when they reach production?
How can a PM drive adoption of an AI product?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Agile
Scrum
Kanban
PI planning
OKRs
KPIs
Minimum viable product (MVP)
Minimum marketable product
Five whys
Value stream mapping
Servant leadership
Explainable AI
Differential privacy
Virtuous AI cycle
Design thinking
Lean management
Resources mentioned
Power BI
Google Analytics
Cite this talk
Asma Syeda. "Product Management for AI." WomenTech Network, 2021. https://www.womentech.net/video/asma-syeda-product-management-ai