Data Science, Artificial Intelligence and Machine Learning
Sandhya Krishnan20:412021
AI won't take your job, but it will change the role you play
Every tweet, Facebook post and mobile search leaves a digital footprint, and the speaker's chart shows data volumes climbing exponentially since 2015. For anyone who feels the AI conversation has outrun them, the jargon alone can shut them out. Sandhya Krishnan, a data enthusiast with ten years in IT across development, testing and production support, maps the whole territory: so how does raw, noisy data become a prediction a business can act on?
Get a plain-language map of analytics, AI types and the steps to build a machine learning model
Sandhya Krishnan gives a beginner-friendly tour of data science, artificial intelligence, machine learning and deep learning, from the four types of analytics to the ways AI systems are classified. She walks through supervised, unsupervised and reinforcement learning, the practical steps of building a machine learning model, and everyday applications from Netflix recommendations to fraud detection.
“Human job won't go anyway. But they will change role.”
“So the first two is an reactive method something happened, then we are working on that. Whereas later two is proactive approach.”
What you will learn
Distinguish descriptive, diagnostic, predictive and prescriptive analytics and match each to a business question.
Classify AI systems by capability (narrow, general, super) and by functionality (reactive, limited memory, theory of mind, self-awareness).
Choose between supervised, unsupervised and reinforcement learning based on whether labeled data exists.
Prepare data for a model by handling missing values, inconsistencies, duplicates and categorical variables.
Recognize machine learning at work in recommendations, photo tagging, virtual assistants and fraud detection.
Key takeaways from the speaker
Different types of AI, ML real time uses
Details
Level
Beginner
Format
Talk
Technical depth
Introductory technical
For
Student, Early career, Career changer
Length
20:41
Published
June 2021
Questions this talk answers
What is the difference between business intelligence and business analytics?
Business intelligence mainly covers descriptive and diagnostic analysis of what already happened, while business analytics adds predictive and prescriptive analysis. Business analytics is the bigger umbrella, with business intelligence as a subset.
Will AI take people's jobs?
Krishnan argues human jobs will not disappear but roles will change. People will become more creative and specialized as AI automates tedious work.
How are AI systems classified?
By capability into narrow, general and super intelligence, and by functionality into reactive machines, limited memory, theory of mind and self-awareness.
What are the main types of machine learning?
Supervised learning trains on labeled data, unsupervised learning finds hidden patterns and clusters in unlabeled data, and reinforcement learning learns by acting on an environment and correcting itself over time.
What steps go into building a machine learning model?
Collect data, clean it (missing, inconsistent and duplicate values, categorical encoding), choose a model using scatter plots, split into training and test sets, apply feature scaling, then predict.
About this talk
Data Science, Artificial Intelligence and Machine Learning
Supervised vs unsupervised learning, explained simply (67 s)
I am having around 10 years of experience in IT. Today, I will be dealing with session Data Science, Artificial Intelligence, Machine Learning and Deep Learning. First, we will go through Data Science. As you all know, data is the new oil in digital economy. There are different types of data. It can be categorical or qualitative numeric. But we can classify them into broader way. There is open data and closed data. Open data are those data which anyone can access without any restrictions like survey results, research data and customer studies which are published. Whereas closed data, it will have a restriction and we should have some permission to access those data like sensitive and important record.
Speaker 2: This is a graph where data is plotted with data volume in terabyte, zerabyte and years. And you can see from 2015, there is a massive increase or exponential increase in data. These data can be collected from Twitter or Facebook or whatever you search in from your mobile. So whatever we do in this digital era, we are living a digital footprint.
Speaker 2: And these are raw data. These raw data are inconsistent in nature, incomplete, does not have any certain or particular behavior or trends. And most likely it will have noise. That is error. So these data can be refined and distilled. And after that, if you put some statistical advanced tool in that, then we can get valuable insights.
Speaker 2: And this valuable insights we can convert into information and which in turn can be converted to knowledge. And this knowledge can be used to solve both technical and productivity problem of analytic business business. And these are mainly done by two that is business intelligence and business analytics. Speaker 2: Business intelligence is used to deliver relevant and reliable information to right people in the right format at right time. Here it is very important that we are delivering the information at the right time. So BIE is used to make better decisions faster. Whereas business intelligence, it is having artificial intelligence, statistical and operation research technique, information technology and management strategies.
Speaker 2: And by using these knowledge, we are making some valuable insights. And by using these knowledge, we are making some valuable insights. This can be done by correlation analysis, regression analysis, Speaker 2: Regression analysis, text mining, and many much thing like that. Mainly business intelligence deal with descriptive and diagnostic analysis. Whereas business analytics lead with prospective and predictive analytics.
Speaker 2: So in a nutshell, we can explain these four analytics. That is description analytics is what happened. Diagnostic is why did it happen. And predictive is what will happen. And predictive is what will happen. Speaker 2: And prescriptive is what should I do. So you can see the first two that is descriptive and diagnostic is that something already happened. And then we are analyzing the data where are the later two it is forecasting the future. So the first two is an reactive method something happened, then we are working on that. Whereas later two is proactive approach.
Speaker 2: So we can say that when we put some prediction into business intelligence, we get business analytics. That is business analytics is a bigger umbrella and business intelligence is a subset to that. Speaker 2: Descriptive we can use for revenue of company or sales report and the diagnostic we can use for employee performance analysis by HR department.
Speaker 1: Descriptive we can use for forecasting weather predict predict weather and the eyes of a company customer churn. We can see the likelihood of that. Prescriptive analysis always deals with when a word optimization come. Speaker 2: That is optimization of bed capacity and overtime shift in a hospital price and promotion optimization for retail. Speaker 2: Then we can see artificial intelligence. Artificial intelligence is the science and engineering of making intelligent machine. Any technique which enables computer to mimic human intelligence such as decision making, text processing, visual representation, sort of visual perception. AI is a broader field for a big umbrella, which contains several subfields such as machine learning, robotics, computer vision, etc.
Speaker 2: So we have data science where we deal with data and if we use data science to make intelligent decisions, then we are using artificial intelligence. So we can see that the Venn diagram, the intersection. Speaker 2: AI is now being used in almost every sector like transportation, health care, banking, retail, entertainment, e-commerce. And we already all have an hype that if artificial intelligence come, then our job will be at risk. But actually it is not like that. Human job won't go anyway. But they will change role.
Speaker 2: We all will be more creative and specialized as AI is integrated into workday. Better data lead to better maps, which lead to better prediction. So people using AI can automate the tedious work and track action to the inside. Speaker 2: Artificial intelligence can be classified based on capabilities and functionalities. When we classify it based on capabilities, we have narrow intelligence, general intelligence and super intelligence.
Speaker 2: Narrow intelligence is also known as weak intelligence, and it is only specific to one area and it cannot work beyond that. It is having little limitation and it is widely used for natural language processing. IBM Watson and Siri is best example. And we are also having many other like image recognition system, recommendation system, spam filtering. The natural. When we come when human communicate with each other with the help of Speaker 2: text or by verbal communication we are using natural language.
Speaker 2: The natural. When we come when human communicate with each other with the help of text or by verbal communication, we are using natural language. Speaker 2: The same frontal is used here natural language processing or NLP refers to branch of artificial intelligence that give the machine ability to read, understand and derive meaning from human language.
Speaker 2: Robots such as Sophia or human assistance used used. Speaker 2: Uses natural language in process to sound like human and understand and understand. Speaker 3: The same frontal is used here natural language processing or NLP refers to branch of artificial intelligence that give the machine ability to read, understand and derive meaning from human language. Speaker 3: Robots such as Sophia or human assistance used uses natural language in process to sound like human and understand what we are seeing.
Speaker 3: Then comes general intelligence. It is a strong one. Speaker 3: It can understand and learn any individual intellectual task that a human being can. Speaker 3: And we are not completely developed into it, but we are in the phase of development. Speaker 4: Fijisto has built K computer, which is one of the fastest supercomputer in the world.
Speaker 4: It is a significant attempt towards strong intelligence and it take around 40 minutes to stimulate a single second of neural activity. Speaker 4: Then last come artificial super intelligence. It is only hypothetical. Speaker 5: It should be above human activity in all sense that it should surpass human intelligence and can perform any task better than human.
Speaker 5: It should be smarter than human in every way. Speaker 5: And when we classify intelligent artificial intelligence based on functionality, it can be divided into reactive machine limited theory mind and self awareness. Speaker 5: Reactive machine is like the primary form of artificial intelligence. Speaker 5: It does not store any memory. It does not have any cache. It works on present data.
Speaker 5: Best example is IBM Deep Blue that defeated just Grandmaster Gary Kasparova. Speaker 5: And then comes limited theory. Limit theory is like it has limited memory access, but it is very short term short lived and they can use this past data for specific period by they work on past data and it works only for short period.
Speaker 6: This is best example is like self-driving car. They are included with vehicle devices, which went to change lane, avoid cutting off another driver or hit a nearby vehicle. Speaker 6: Then theory of mind. It is also like not completely developed. Speaker 6: Kismet and can mimic human emotion and recognize them is one of the real world example of this, but it cannot follow grace or convey attention to human.
Speaker 6: Theory of mind. The example is Robert Sophia. She has an combined computer algorithm, which allow in her eye, which allow her to see. Speaker 6: She can sustain eye contact, recognize individuals and follow face. Speaker 6: Self-awareness. It also is an hypothetical one. Speaker 6: It all at such system understand the internal tried state and condition and preserves human emotion.
Speaker 6: These machines will be smarter than human mind. Speaker 6: Then these are the famous quotes of great personality. Speaker 6: Elon Musk says that AI does not have to be able to destroy human. Speaker 6: If AI has a goal and humanity just happens in the way, it will destroy human as a matter of course without even thinking about it.
Speaker 6: No hard feelings. Speaker 6: Whereas Mark Cuban says that artificial intelligence, deep learning and machine learning, whatever you are doing. Speaker 6: If you don't understand it, learn it because otherwise you are going to be a dinosaur within three years. Speaker 6: And father of deep deep learning. Speaker 6: Jeffrey Hinton says that in deep learning, the algorithm which we are using, it is of 1980s and 1990s.
Speaker 6: People were very optimistic about them, but in turn out they did not work too well. Speaker 6: And he has worked lots of lots and lots of work in deep learning and he had published many articles in the same. Speaker 6: We'll learn machine learning. Speaker 6: Machine learning provides AI the ability to learn without explicit programming.
Speaker 6: Here we make machine to work. Speaker 6: Machine will program. Speaker 6: Machine learning is achieved by algorithm that discovers pattern and generate insights from data they are exposed to. Speaker 6: Machine learning access vast amount of data both structured and unstructured and learn it from to predict the future. Speaker 6: When we mark artificial intelligence, machine learning and deep learning by an Venn diagram, we can see that artificial intelligence is a broader umbrella Speaker 6: And machine learning is a way of achieving artificial intelligence.
Speaker 6: Likewise, deep learning is a way of achieving machine learning. Speaker 6: Artificial intelligence. Speaker 6: We know that anything that imitate human behavior and application of AI that allows a system to automatically learn and improve from experience and deep learning limit our neural network of the human brain. Speaker 6: Machine learning algorithm can be classified into supervised learning unsupervised learning and reinforcement learning.
Speaker 6: Supervised learning is like making a child learn. Speaker 6: We want to want them to identify some cat or a dog. Speaker 6: We have to show them your pics. Speaker 6: The same as work for supervised learning. Speaker 6: The data is already labeled. Speaker 6: So we know that what our target is. Speaker 6: It required at least an input and output variable be given to the model for it to be trained.
Speaker 6: It is somewhat like this. Speaker 6: We have input data and annotations that we feed to the machine. Speaker 6: So when we put an input to the machine, it will predict that what the input is. Speaker 6: Example of supervised learning is linear regression, logistic regression, support vector machine, NAV base and decision tree. Speaker 6: Unsupervised learning, it is like we don't have labeled data.
Speaker 6: And we make the machine to understand the design pattern. Speaker 6: The system can identify hidden features from input data provided. Speaker 6: Once the data is more readable, the pattern and similarity become more evident. Speaker 6: It will be like we will be putting lots of input without any annotations and the model will learn.
Speaker 6: And then it can divide and classify into the specific group. Speaker 6: Example of unsupervised learning is K-Mean clustering. Speaker 6: Hierarchical clustering and anomaly detection. Speaker 6: Reinforcement learning. Speaker 6: Here it is a combination of supervised and unsupervised learning. Speaker 6: Here minimum data, training data are available. Speaker 6: At that time we use reinforcement learning and also we don't have much idea of the environment.
Speaker 6: And it learned the environment by interacting with it. Speaker 6: Algorithm act on the environment. Speaker 6: For example, make a trade in financial portfolio. Speaker 6: Algorithm optimizes for best series of action by correcting itself over time. Speaker 6: And when we have to design a machine learning algorithm, we have to first collect the data, data processing, choose the model, train the model, test the model, Speaker 6: Feature scale and predictions.
Speaker 6: We will learn one by one. Speaker 6: First one is data collection. Speaker 6: There are five common methods by which we can collect data. Speaker 6: That is close survey and cues. Speaker 6: Other is open ended survey and questionaries, bar one to one interview, focus group or direct observation. Speaker 6: Once we have collected data, we have to check identifying and handling missing data.
Speaker 6: That is if any numerical value is missing, then we can go for mean of the all value and put it in the column. Speaker 6: Then inconsistence value. Speaker 6: For example, we are having someone is entering the data in computer that time. Speaker 6: They may have made a mistake like address and pin number are not matching.
Speaker 6: In such cases, it becomes inconsistent values. Speaker 6: So we have to treat that manually. Speaker 6: Then comes duplicate value. Speaker 6: There is a chance that two people under the same value. Speaker 6: If duplicate value is there, we have to delete one value that encoding categorical data. Speaker 6: For example, we are taking a population of three main three country.
Speaker 6: For example, India, US and UK. Speaker 6: In that case, we cannot feed the system categorical value. Speaker 6: We have to make better like we have to put India as 100, US as 010 and UK as 001. Speaker 6: Then next is choose the model for choosing the model. Speaker 6: We can we can have a first have a scatter plot of the independent variable each independent variable and dependent variable.
Speaker 6: When we have a scatter plot, we can see the relation between two. Speaker 6: If they are linear related or they are scattered and then we can go based on scatter plot. Speaker 6: We can decide which model we have to take that we have already learned supervised and unsupervised learning many like in a regression logistics.
Speaker 6: So we can choose from one of the model then resampling method. Speaker 7: It is an advanced like Monte Carlo stimulations can be used for this. Speaker 7: Then after that, we have to split the test. Speaker 7: We have having a test data. Say we are having 1000 test data in that we have to split into training set and a set training set.
Speaker 7: We have to divide into 80% test to 20% then training set. Speaker 7: We have to evaluate. Speaker 7: We have to train the system with training set and test set is used for evaluation. Speaker 7: And evaluation of model is done to check the performance of system. Speaker 8: Then come feature scaling for all the algorithm.
We don't go for feature scaling, feature scaling are used such so that all your features that is variable are scaled to make sure that all take same scale. Speaker 8: It is done to prevent that one feature does not dominate the other. Speaker 8: For example, if you are having multiple linear regression, it can be represented by this equation.
Speaker 8: So by using normalization or standardization, we can go for feature scaling. Speaker 8: Then prediction once our model is there, we have trained test and we have feature scale. Speaker 8: Then we can go for prediction, predicting the future value. Speaker 8: This can be done by putting system and live examples of machine learning is automatic tagging of friend in Facebook.
Speaker 8: When we put a photo, we automatically, we know that we get a tag. Speaker 9: This person is name is this. So like it is by using deep face algorithm, then Uber, Uber, calculate shortest distance and the demand. This are surprising everything is calculated, basing on machine learning application. Speaker 9: Then virtual personal assistants, such as Siri, Alexa, Google, Cortana, everything are using ML ML is used for speech recognition, speech to text conversion, match to language processing, text to speech conversion.
Speaker 9: Then Google Translate use neural machine translation and NLP for translation. Speaker 9: Then Netflix, we know that when you log into Netflix, it gives us the recommendation. It is done by collecting the data we do that is searching, browsing, start, stop, rewind, by collecting all of this data, they recommend the film for us. Speaker 9: Then next is fraud detection.
Speaker 9: When we use credit card or debit card, whenever a customer carries out a transaction. Speaker 9: ML model, thoroughly scan x-ray their profile searching for suspicious activity. Speaker 9: Then keynote sports. Here we don't have joystick, but by seeing our actions and movement, ML play with us. Speaker 9: Then it is virtual reality set. As we move, the picture also move.
Speaker 9: Then robot dog. Speaker 9: It is used for making kids learn walking and also it is appears in fiction. Then other example are audible recommendation, Amazon recommendation, Facebook ad recommendation. We use ML for mass exploration program. Then in drugs, we use behavior modeling. Speaker 9: Then in leap learning. Speaker 9: The major difference between ML and a deal is that ML uses our mimic our neural network.
But whereas ML works only on structured data. This is the example of neural network and provides a ability to mimic human brain. Speaker 9: It is like input layer. Speaker 9: We feed and the hidden layer identifies the pattern and get the output. It is a retina by scanning retina. They identify ML. Sorry. DL identifies if the patient is diabetic or not.
Speaker 9: Then it is calculated by when we provide an input. It provide a weightage for each one. Then it goes to an activation function. Activation function identifies the input and provide a bias. It determines whether it the input should be filtered or not. Once I put it according to the activation, the value enters. And once it entered, output layer predict the value.
These are the other applications DLR used. Thank you. You can write down my email ID. If there is any doubt or anything which I can have, please let me know. I will try to contact soon. Speaker 9: Thank you. Speaker 9: Thank you everyone.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
How does raw data become an AI prediction?
Sandhya Krishnan maps the path from noisy data to analytics, artificial intelligence, machine learning and deep learning in plain language.
In short
Raw data is noisy and incomplete, but once it is cleaned and analyzed it becomes insight, then knowledge, then the fuel for artificial intelligence. Sandhya Krishnan, a data enthusiast with ten years in IT, walks through the four types of analytics, the ways AI is classified, the three families of machine learning and the concrete steps of building a model. Her message for anyone worried about AI is reassuring: jobs will not vanish, but roles will change.
Key takeaways
Different types of AI and real-time uses of ML.
From raw data to decisions
Krishnan opens with the premise that frames the whole talk.
"As you all know, data is the new oil in digital economy." (0:00)
She separates open data, such as published survey results and research data that anyone can access, from closed data, which requires permission because it holds sensitive records. A chart in her slides shows data volumes rising exponentially from 2015, collected from platforms like Twitter and Facebook and from everything people search on their phones. In her words, everyone living in the digital era leaves a digital footprint.
That raw data, she explains, is inconsistent, incomplete and full of noise. Refined and run through statistical tools, it yields insights that become information and then knowledge. Two disciplines do this work. Business intelligence delivers reliable information to the right people at the right time and mostly covers descriptive analytics (what happened) and diagnostic analytics (why it happened). Business analytics adds predictive analytics (what will happen) and prescriptive analytics (what should I do).
"So the first two is an reactive method something happened, then we are working on that. Whereas later two is proactive approach." (3:07)
Her examples make the distinction practical: sales reports for descriptive work, HR performance analysis for diagnostic work, weather and customer churn for prediction, and hospital bed capacity or retail pricing for prescriptive optimization.
What AI is, and what it is not
Krishnan defines artificial intelligence as any technique that lets computers mimic human intelligence, such as decision making, text processing and visual perception. AI is a broad umbrella covering machine learning, robotics and computer vision, and when data science is used to make intelligent decisions, the two overlap. She addresses the common fear directly.
"Human job won't go anyway. But they will change role." (5:03)
She then offers two ways to classify AI. By capability, there is narrow intelligence, specific to one area (Siri, IBM Watson, spam filters, recommendation systems); general intelligence, still in development, for which she cites Fujitsu's K computer as an attempt that took around 40 minutes to simulate a single second of neural activity; and super intelligence, which remains hypothetical. By functionality, there are reactive machines with no memory, such as IBM Deep Blue, which defeated Garry Kasparov; limited memory systems such as self-driving cars; theory of mind, illustrated by Kismet and the robot Sophia; and self-awareness, again hypothetical. She closes the section with quotes she attributes to Elon Musk, Mark Cuban and Geoffrey Hinton on the risks of AI and the need to learn it.
How machines learn
Machine learning, in Krishnan's framing, is a way of achieving AI, and deep learning is a way of achieving machine learning.
"Machine learning provides AI the ability to learn without explicit programming." (10:41)
She describes three types. Supervised learning works on labeled data, with examples including linear regression, logistic regression, support vector machines, Naive Bayes and decision trees.
"Supervised learning is like making a child learn." (12:02)
Just as a child learns to identify a cat by being shown pictures, a supervised model learns from inputs paired with known outputs. Unsupervised learning has no labels: the model finds hidden patterns and groups the data, through methods such as K-means clustering, hierarchical clustering and anomaly detection. Reinforcement learning applies when training data is scarce and the environment is unknown; the algorithm acts, for example by making a trade in a portfolio, and corrects itself over time.
Building a model, step by step
Krishnan lays out a practical workflow. Data collection can use closed or open-ended surveys, one-to-one interviews, focus groups or direct observation. Cleaning comes next: filling missing numeric values with the column mean, fixing inconsistent entries such as an address that does not match its pin code, deleting duplicates, and encoding categorical data, for instance turning India, US and UK into separate binary columns.
To choose a model, she recommends plotting each independent variable against the dependent one.
"When we have a scatter plot, we can see the relation between two." (15:37)
The data is then split, typically 80 percent for training and 20 percent for testing, and the model is evaluated on the held-out set. Feature scaling, through normalization or standardization, keeps one variable from dominating the others before the model moves to prediction.
Machine learning in everyday life, and deep learning
The final section grounds the concepts in familiar products: automatic friend tagging on Facebook using DeepFace, Uber's route and demand calculations, virtual assistants like Siri, Alexa and Cortana, Google Translate's neural machine translation, Netflix recommendations built from searching, browsing and rewinding behavior, and fraud detection that scans every card transaction for suspicious activity.
Deep learning, she explains, mimics the neural network of the human brain. Inputs receive weights, pass through hidden layers and an activation function, and the output layer predicts a value. Her example is a deep learning system that scans the retina to identify whether a patient is diabetic.
Watch the talk
"Data Science, Artificial Intelligence and Machine Learning" by Sandhya Krishnan, 21 minutes: https://www.womentech.net/video/data-science-artificial-intelligence-and-machine-learning
About the speaker
Sandhya Krishnan is a Data Enthusiast and a highly organized, detail-oriented software professional well-versed in Java, UNIX, Selenium, VB script and Python, with 10 years of IT experience spanning development, testing and production support. She is one of the contributors to the e-books Testing_Stories and #21stskills4testers.
Questions this talk answers
What is the difference between business intelligence and business analytics?
Will AI take people's jobs?
How are AI systems classified?
What are the main types of machine learning?
What steps go into building a machine learning model?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Descriptive, diagnostic, predictive and prescriptive analytics
Narrow, general and super intelligence
Reactive machines, limited memory, theory of mind and self-awareness
Supervised, unsupervised and reinforcement learning