The Journey of Data Science - How Did It Iterate the Way to AI
Over 85% of siloed data science projects failed, and AI grew from the fix
Data science once ran in silos, and the speaker says over 85% of those projects failed at high cost and low profit. For anyone building a data or AI career, that history explains how today's AI took shape. A strategist with more than 20 years in analytics, data science and AI traces each stage, then asks what the journey teaches the people who will shape what comes next.
Learn the stages, AI waves and analytics levels behind today's AI, plus five career lessons
In short
The speaker (not named in the session record), a data science and AI strategist with over 20 years of experience, traces how data science evolved into today's AI. She describes six foundational pillars, the move from siloed projects to modernization and automation, four AI waves and five levels of analytics maturity, then turns that history into five lessons for building a data science or AI career.
“The progress is not linear, it's zigzag, but it's reached upon a huge breakthrough.”
“When the machines become smarter, actually humans become also smarter.”
What you will learn
- Name the six pillars behind any data science effort: business problem, algorithm, data, computing power, strategy and people.
- Trace how methods moved from frequency counts and linear models to ensembles, neural networks and generative AI.
- Place your organization on a five-level analytics maturity scale, from dashboards to decision intelligence.
- Explain why siloed data science projects stalled and how reusable pipelines, centers of excellence and DevOps, DataOps and MLOps helped.
- Apply five career lessons: build foundations, start small, collaborate, partner proactively and mentor others.
Details
- Level
- Beginner
- Format
- Talk
- Technical depth
- Introductory technical
- For
- Student, Early career, Mid-career
- Length
- 18:43
- Published
- June 2023
Questions this talk answers
What stages has data science gone through on its way to AI?
The speaker describes laying theoretical foundations, a silo-based practice stage, AI emerging through modernization and reuse, automation through the Ops practices, and the current state of AI.
What are the four AI waves?
Rule-based systems where humans hard-code rules, then automated machine learning, then deep learning, and now reinforcement learning that puts humans in the loop to teach machines.
What are the five levels of analytics?
Dashboards show what happened, root cause analysis and A/B testing show why, model fitting predicts what will happen, simulation finds the best solution, and decision intelligence automates decisions.
Why did early data science projects struggle?
People, data processing, algorithms and analytics worked in silos, making projects slow, less efficient and costly, and the speaker says over 85% failed.
What should people building a data science or AI career learn from this history?
Lay a solid data and algorithm foundation, start with something easier, learn to collaborate humbly, partner proactively with others, and once successful coach and mentor others.
Topics
Highlights
- The four AI waves, from rules to humans in the loop (71 s)
- Why siloed data science projects failed (90 s)
- Five lessons for a data science career (84 s)
And 2023 is a year of AI. We all know AI is transforming the world as both generators and consumers of technology. We all interact with chat box and you may have some smart toaster at home, robot cleaner, and also you may take a self-driving car to work someday. We may greeting to Robert, shake hands to them on the street. So AI is not only changing almost every single profession, it's also changing our understanding what work is. AI is redefining every company and our daily life.
So how did AI come to here? Today my talk is really about the journey of data science. How did it iterate the way to AI? Before I jump to the content, first thing, let's have a brief introduction about myself. I have about over 20 years experience in data analytics design, data science, AI. My focus is really develop AI data science strategies for companies, build data-driven products, build teams. I'm passionate about people, a driven leader, with an entrepreneurial spirit to consistently identify opportunities and the challenge and the challenge. I have a different approach. I have a great opportunity for universal. I hold two approved patent, two books. One is co-authored with others in the data area. Another one helps people in their personal growth, growth to your fullest.
My vision is co-authored with others in the future. My vision really helps companies empower data, inform the decisions, accelerate growth, reach optional excellence through data science of AI. In life, I want to help people achieve their fullest in life, in the career data science of AI. My favorite code is to try not become a main success, but rather become a main value.
Okay, this is today's outline. We start with what is AI. The focus is really five stages of data science evolution and the current state of AI lessons we learned. So what is AI? So what is AI? At a very high level, AI is the simulation of human intelligence by machines and computer systems that include learning, reasoning, self-correction. We all know AI is a multidisciplinary field.
So here is the data to share what's the relationship AI with computer science, data science, and machine learning. So what is a robot? In my simple formula, I would like to say robot is put AI, machine learning in the center, data science, hardware. What does a robot do? Basically, they collect the consuming information. Like human, they learn, reasoning, automated process, decision-making, even communicating with machines and people.
So this is the five stages personally, I think data science evolution have gone through. Let's start with six pillars of fundamentals. This includes business problems. We know everyone facing problems in life. So the business problem may be from simple to complicated. We saw one followed another. How can we let machine help us? This is the means. How can we use the right algorithm with the right assumption and conditions to formula the business problem?
So we also need data. Data need to complete relevant. This is a very popular four ways, volume, velocity, velocity, velocity. Also, we also need computing power to process the massive data that comes to distribute parallel computing to the current quantum computing. The strategy is really try to develop systems by leveraging data theory technology altogether to solve a unique business problem, build the product and the service.
Last, not least, is people. People is on the driver's seat on each of these pillars. People collaborated with each other for innovation. So let's zoom in for the algorithm business problem. We start very early, back to the 1863 probability-based method. Really one method, one learning. One England physician actually, using simple frequency count, identified the location in Chlora disease outbreak in 1854 at Linden.
Later on, we used the linear method to nonlinear. Also, numerical forecast is one application. It comes to the assembled method. Assembled method basically is the same method, but through multi-iterature learning by focusing different slides, snapshots of data, such as grinding boosting, random forest. Also around 2000, people think about how can that machine thinking and reasoning like human. That's come to much layered learning using neural network.
So what we've come to now is generative AI. Basically, machines interact with humans. It's not only one method, multiple method, multiple iterations, and also many. One time learning a big model provides many solutions. So if we look at this, it's really from simple pattern detection comes to very human way to solve, to build the product, solve the problem.
So if we look at the speed development, very beginning is very slow. Then comes 10 years, 8 years, 6 years. So let's talk about the four AI waves. So let's talk about the four AI waves. As the very beginning is rule-based method. Basically humans include the rules. Hard-coded machine do things. Thus come to machine learning, automated. Then using deep learning currently is reinforced learning, put humans in the loop.
So humans can teach machines to become smarter. So basically the evolution begins with baby steps crawling, then progress is flying. So the journey of baby steps. So the journey of a thousand miles begins with one step. Now let's look at the five-level analytics, what's the relationship, the analytics with data, and the decision. So we start with the very beginning, no data, no analytics, purely human gut feeling-based decision.
Then we use smaller data to identify what happened. That's dashboard-based approach. Then comes to, okay, we want to know why it happened. That's a root cause analysis. Basically we do A/B testing experience. Level three is really supporting strategy planning. We need to understand what will happen. That's really a model-fitting start. So level four is basically decision support. But what's the best solution? We use simulation-driven analysis approach.
Currently is really decision intelligence. Machines do the self-learning, leverage cognitive computing to upgrade all these kind of steps together to the automated decision-making. So from database perspective, we go through structured database to data warehouse, data lake to data lake house. We start from monitoring to data governance and currently observability, understanding nitty-gritty decay of what happened to the data.
So the decision we are already mentioned. So the decisions we already mentioned, at the very beginning, no insight, no data, just pure human. Some we leverage insight gradually from less insight, more human instinct to become less human instinct, more insight. Some analytics to come to AI humans working together. When the machines become smarter, actually humans become also smarter.
And because humans make machines become smarter. At a very high level, when we reach very high level analytics maturity, a corporate should have a well-established data culture, unlock the full potential of data. Everyone uses data analytics to make decisions. It's not limited, only a small portion of people. The first stage, actually, when we have this, after we laid the foundation theoretical, we go through silo-based practice.
If we look at the data science projects. If we look at the data science project, the life cycle involves eight components and related business objectives. Whether from people, data processing, algorithm, analytics processes, they are all solid. Everyone does their own thing. The challenge really comes slow, less efficient. Over 85% project, they fail. High cost, low profit. How to solve this? This comes to stage three, AI emerging, modernization. Basically, we try to do high level abstraction.
The challenge, such as data pipeline, data process, feature engineering. And we package all for the specific business use case with the different modules pipelines together to make it easier, reuse, accessible. So this is the agile approach, actually. People in here are involved in the data, design, machine learning process. They collab to each other, try to identify reusable templates. And some companies also build a center of silence to communicate the best practice.
Organize the training program. Organize the training program. Create an internal marketplace with reusable template and guidelines. Then we come to automation. Basically, leverage the people process technology to infuse AI machine learning into business process. The automation process. The automation process is really leverage the four OPS. Basically, it's DevOps starting first to build the product. Then Data OPS for the data pipeline and engineer. And there comes the machine learning part, AI, ML OPS, Model OPS.
It includes these six components involved. So I know I go through very fast because there are so many stops. As a summary, this is the four stages of data science evaluation. The progress is not linear, it's zigzag, but it's reached upon a huge breakthrough. Next, I'm trying to talk about what's occurring in the state. Let's focus on challenges, trends, opportunities, future development.
With Tad GTP4 release, this year is a really groundbreaking year for the world and AI, but still there are a lot of challenges. Because Tad GTP4 is a large language model. The training is so expensive, it's over a million dollars, consuming a lot of resources energy, and their potential bias is well known and is hard to explain.
And the number misuse AI is also rising. Maybe because this is a challenge, the US government actually invests more in AI research and design, about $2 billion. Another trend is very interesting. Industry actually takes the lead to produce more machine learning models, because it's easier to access real-time data, computer power with enough money than academic. A lot of startups, a lot of startups, they come up to try to make AI more easier to understand, transparent.
The opportunity is also huge. I would like focus is really not only for companies, it's for the personals, such as how can we use AI-powered automation to do all kinds of things, not only marketing sales, customer assistance. Also think about how we are going to optimize all these kinds of operations. There are a lot, a lot of innovation opportunities.
Future development will continue to improve the AI. It's used in every area, such as edge nodes, smartphones, also help make AI understand human emotions. So the future of AI really looks amazing if we use AI for the benefit of human, put human in the center. At the beginning, I said AI is redefining our company and our everyday life.
The essence is what it means to be a human under the true meaning of work and life. I believe a beautiful future is waiting for us, where we focus on what makes us unique as humans, such as our vision, curiosity, our passion, our dreams, our emotions, intelligence. Finally, let's talk about what we learn. Finally, let's talk about what we learned. Like AI, if we work in data science AI career, we also need to lay a solid foundation to build our data and our algorithm technology foundations.
Secondly, we need to collect this siloed approach to build ourselves, start with the undermined, beginning something easier. Third, learn to collaborate. Be humble. You can't do everything. Stand on the shoulders of the ground. Learn from others, collaborate from others, to adopt others, approach with them. The fourth part is partnership with others. Be proactive. Leverage the power of career, the wisdom.
The last part is when you become expertise, your success. Multiply your success. Be supportive, coach, mentor others. Build a flying wheels. Leo, Leo, Alexei. I think the future is if you follow this, your future is also flying like AI right now. So this is my talk today. I go very fast. Thank you, everyone. I'm not sure is there any questions.
So let's talk. Oh, I can't see any question here. I will stop sharing.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Frameworks and ideas mentioned
- Five stages of data science evolution
- Six pillars of data science fundamentals
- Four AI waves
- Five-level analytics maturity
- Four Vs of big data
- DevOps
- DataOps
- MLOps
- ModelOps
- Agile
- Center of excellence
- Decision intelligence
Cite this talk
WomenTech Network. "The Journey of Data Science - How Did It Iterate the Way to AI." WomenTech Network, 2023. https://www.womentech.net/video/journey-data-science-how-did-it-iterate-way-ai
Related talks
Browse the libraryData Science, Artificial Intelligence and Machine Learning
AI won't take your job, but it will change the role you play
De-Mystifying Machine Learning and Artificial Intelligence
Data science is called the sexiest job, but that is no reason to switch
I know where you wanna go next summer - Demand Forecasting at FlixBus
A forecast nobody trusts creates no value, however good the model
Product Management for AI
Her AI risk engine was scrapped mid-build over one data category
The world of Python
Two lines of Python can do the work of 10 or 15 in other languages
Demonstrating the Machine Learning Cycle
How a data flywheel cut document turnaround from 90 minutes to 10