Transfer Learning With BERT: Building a Text Classification Model
Jayeeta Putatunda45:482020
Most text classifiers don't need to be trained from scratch
An algorithm told "I am a huge metal fan" cannot tell whether you love metal bands or own a metal fan, and teaching it the difference takes enormous amounts of data. Data scientist Jayeeta Putatunda, who specializes in natural language processing, shows why building every text model from zero is costly when labelled data is scarce. So how do you borrow a model pre-trained on billions of words and make it classify your own text?
Follow the steps from cleaning raw text to fine-tuning BERT for news classification in Google Colab
Data scientist Jayeeta Putatunda introduces natural language processing for newcomers, from core text preprocessing to word embeddings, Transformers and BERT. She argues that transfer learning lets teams reuse models pre-trained on huge datasets instead of building text classifiers from scratch, and demonstrates fine-tuning a pre-trained BERT model to classify news text in Google Colab.
“So a lot of the time people feel that NLP is hard. And to start with, it definitely is hard.”
“A lot of people say NLP is magic, but it's not.”
What you will learn
Clean raw text with core preprocessing steps: stripping extra spaces, tokenization, spell checking, contraction mapping, stemming and lemmatization, and stop word removal.
Explain how transfer learning reuses a model trained on a large general dataset and fine-tunes its final layer for a smaller, related task.
Judge when transfer learning fits: labelled data is scarce or expensive, and the source and target tasks share the same input properties.
Describe how BERT's bidirectional training and its token, position and segment embeddings give the same word different meanings in different contexts.
Fine-tune a pre-trained BERT model for news classification in Google Colab by refitting its final dense layer with a softmax activation.
Details
Level
Intermediate
Format
Tech talk
Technical depth
Practitioner
For
Student, Early career, Mid-career
Length
45:48
Published
July 2020
Questions this talk answers
Why is natural language processing hard?
Language is full of ambiguity, so a phrase like "I am a huge metal fan" has two readings. Humans separate the contexts easily, but algorithms need large amounts of data to learn those nuances.
What is transfer learning?
It is a technique where you train a model on a task with plenty of data, then repurpose it for a related task. Typically you keep the learned low-level features and fine-tune the last dense layer on your smaller dataset.
When should you use transfer learning?
Use it when labelled data is scarce, because labelling needs time and subject matter experts. The target task must also share input properties with the source task so knowledge can transfer.
How is BERT different from earlier embeddings like Word2Vec?
Word2Vec read context in one direction, while BERT trains both left to right and right to left. That lets a word like "bank" get different representations next to "river" or "deposit".
Which tasks is BERT pre-trained on?
BERT is pre-trained on masked language modeling, where it predicts hidden words, and next sentence prediction, where it judges whether one sentence logically follows another.
About this talk
The domain of Natural Language Processing have seen a tremendous amount of research and innovation in the past couple of years to tackle the problem of implementing high quality machine learning and AI solutions using natural text. Text Classification is one such area that is extremely important in all sectors like finance, media, product development, etc. Building up a text classification system from scratch for every use case can be challenging in terms of cost as well as resources, considering there is a good amount of dataset to begin training with.
Here comes the concept of transfer learning. Using some of the models that has been pre-trained on terabytes of data and fine-tuning it based on the problem at hand is the new way to efficiently implement machine learning solutions without spending months on data cleaning pipeline.
This talk with highlight ways of implementing the newly launched BERT and fine tuning the base model to build an efficient text classifying model. Basic understanding of python is desirable. Code can be made available via Github for everyone to examine after the talk.
Why "I'm a huge metal fan" confuses algorithms (90 s)
How BERT tells a river bank from a money bank (84 s)
The carbon cost of models and why NLP isn't magic (73 s)
So the topic for today is transfer learning with BERT, NLP 101, as I call it, because this is like a subject that's really close to my heart. I really like researching about NLP, working on it, and it's great to be sharing some of the things that I've learned with you guys. So thank you for joining me in this.
So a quick introduction. My name is Jayeeta Putatunda. I'm a data scientist with Indellient US Inc. We are based out of New York and we do a lot of interesting data analytics projects. So it's been great researching and working with them. You can connect with me in my LinkedIn and Twitter accounts. Follow me, ask me any questions if you have any doubt, and I can address that as well.
Great. So just a quick format. So this is a 40-minute session. So we're going to talk 30 to 35 minutes about the subject. And if you have any questions, we're going to take that at the end, like five to 10 minutes of question answering. I will be able to share the code in my GitHub link. I will update it after the session. So get back to me if you have any questions on the code, or if you need any clarifications, always there to help. No worries.
So going forward, this is a thing that I feel when data science is such a huge field and NLP is such a big area in that. So a lot of the time people feel that NLP is hard. And to start with, it definitely is hard. Let's take a look at this. What do you see when you see this image? It says, I am a huge metal fan. What are the two references you get from this picture?
I get two. Do you guys get two references from this picture? If an algorithm hears this line, I'm a huge metal fan. Does it think about a metal fan? Or it thinks about a metal band and that you're a fan of that band. So there are a lot of complexity in English language. Yeah, I feel this is a great example. It lets me clear out the method of what I think about where to start with NLP. And some of the major complexities of English language is basically ambiguity. There is a lot of synonymity that you do not know. A particular word can mean multiple things.
So it is very easy for us humans to understand and reference between these two contexts. But it's very difficult for an algorithm to actually distinguish between this and they need lots and lots of data to understand the very nuances, differences and give us proper results. So that's why they say NLP is hard. But I guess we need to go a little bit step by step and we will conquer it. So let's go ahead.
So we are going to talk a little bit about what is natural language processing. I know there's a lot of newbies out there who wants to jump into data science and wants to explore the domain of natural language processing. So we're going to talk a little bit about NLP, where NLP is today, where it is applied mostly in the industry, what are the basic pre-processing steps that you've got to do to make sure that your model works fine.
And then we're going to talk a lot about transfer learning bird and the basic and we look into a code that talks about the classification model using bird pre-trained models. So let's jump onto it. So what is NLP? So NLP is basically a subfield of a lot of areas you would see in the chart. Can you guys see clearly the chart, the Venn diagram that shows here that it is that there are a lot of components.
Linguistics comes in because it talks about English language, English literature, what are the grammatical rules, what are the syntax rules for English language? Then it is an amalgamation of a lot of natural language generation, natural language understanding. And with a big chunk of deep learning and machine learning comes in to create a very niche subfield of natural language processing.
I guess the major idea is all of these fields come together to basically help the machines to understand and communicate back in a free flowing human speech so that we can generate or give them direct speech with a direct way and they can give us a very quick feedback without us giving them very manually tagged data. So let's see how that happens. So before that, just look at this chart and see that how far we have come.
Like if you see the exponential growth that has happened from April 2018 to May 2020. Wow. Just look at the jump in parameters. So when I say parameters, those are the parameters that big companies like Google, OpenAI, Microsoft and Facebook, Hugging Face, they are all working with and creating pre-trained models. These models are very, very difficult to train.
They take a lot of time, resources, a lot of GPU, CPU strengths. So there are like multiple GPUs that needs to run to train one particular model. So if you see, as we speak today, end of May 2020, OpenAI developed a new model with 175 billion parameters. Those are huge numbers and we need to understand the significance of what we can achieve with this exponential growth.
So where is the NLP actually getting implemented in all these industry use cases? The major first use case that comes to my mind is definitely machine translation. There are English text if you want to translate any text to a different colloquial language. Like from India, we have multiple languages that people communicate in. So how do you build a one model and then use that to translate between multiple text videos?
Then you translate Spanish audio to say you want to translate it to a German text. The major example is like Google Translate. Another area is chatbots. I'm sure you guys have interacted with the chatbots a lot lately and sometimes you may find them frustrating because you keep asking them a lot of questions and it feels that they keep reiterating the same thing and you're not getting the exact information you want from them.
But you got to know that when you're building a chatbot model, there's a lot of data that goes out in there and the intent classification, the knowledge tree building and this whole question answering needs a lot of data set to train a chatbot to make them as good as a human chat interaction that you would want to have.
So we look where we are in that domain at the end of this presentation. Another area is like definitely natural text generation. Like you see so many big articles. What if you want to understand abstract summary of it? Like you don't want to read through multiple pages of the news, but just want an abstract summary. So creating that has become like a big leap and Google is working.
Just released a paper yesterday that spoke about that and they did like they achieved a lot of accuracy in the new model that they developed. So it is an ongoing field and the research is like heavy in this area. Topic modeling is another area. So topic clustering, entity extraction, sentence chunking. These are very specific cases of where NLP gets to be implemented.
And another area is definitely text classification. We will see an example of text classification today that you have if you have a news and you want to classify that if it's a political news, if it's a sports based news, we will go through that example as well. Yeah. So moving ahead. How do you do it? So I wanted to make sure that before we go and jump into transfer learning, all of you have a basic or a baseline understanding of some of the steps that definitely goes in.
And we cannot miss out on this because the concept of garbage in garbage out comes out. So if you feed the model with very heavy garbage text and it's not clean, not structured, then there is no way that we're going to get a good good model or a good accuracy state. So let's look at some of these areas.
So extra spaces. These are like if you see the example text, it says God is great. I won a lottery. It had a lot of spaces. So how do you fix that? There are very simple ways of fixing this in each of the methodologies that you can use. There are multiple libraries as well. You can explore NLTK.
You can explore spacey. There are a lot of pre-built areas that you can utilize that. But in a simple way, you can just strip it and strip actually basically takes out all the spaces from the trailing as well as spaces anywhere between the sentence. And then you can join it again to get a very clean sentence. The next would be tokenization.
So tokenization is the basis of all natural language processing model. So when you say tokenization, the best idea would be to think about it as like splitting your sentence into particular words. Like you cannot feed a whole sentence to a, you can feed a whole sentence to a model. But the way that it connects and builds those vector representations that it does in NLP is via word embedding.
So you can look into that as well as we go in. So for tokenization, there's a lot of libraries like mentioned. NLTK does a great job. So here, if you just do tokenization and you resub. So resub basically is a reject search and it's taking out any extra characters, any sentences that you do not want. You can create your specific reject searches as you want.
So the final output, if you see there, we deleted the exclamation mark, the full stops and basically get a very clean one saying that God is great. I won a lottery. Great. So spellcheck, this is another very important area. Spellcheck leads to a lot of error in the model because a model is not able to distinguish between two same words even if there is like one very slight or two character slight differences in it.
So how do you, there is a Peter Norvig spellchecker that he used to work with Google brain and they created a very, I would say a very efficient baseline spellchecker. So you guys should take a look at it. If you see the implementation of it here. So correction, the first correction is like the function name. And when I'm calling it as spelling with a one L missing, it gives me back the correct spelling.
So this ensures that we are not feeding the data, at least clarifying or changing it for the basic words that we know that are incorrect. So the here in the next, in the next sentence, it's like called corrected. So corrected means it should be corrected with a C and not a K. So it translated into that. It changed poetry and the last, and the last word quintessential is correct as it is.
So it didn't change anything much in there. The next is contraction mapping. So this is a very, very important thing to do in every NLP project that we do is that in a natural text, like say, if you are talking with a chat bot or you're posting it in your Facebook social, I'm sure all of us use very small words.
We don't want to be writing it with a full version of it. So how do you do that? So basically with contraction mapping, what we are saying is that have nots like have with an apostrophe, we are basically mapping it out into a whole word. So if you see the final result here, it says, hey, I am Jayita.
So it clears it as I am Jayita and takes out the contraction that is I apostrophe M. So these are some of the use cases like for not, for will, for would have, I would have. Those are some of the applications of it. The next is stem and stem limitization. So this is a really important area. We will see that BERT also does a similar feature to train on outer vocabulary words.
So how do we do it? So like say game, gaming, games and games, they all come from a same root word, which is game. So if we want to train a model and also keep it efficient and not reiterate a lot of our words, the best way to do it as a lemmatization and a stemming version of it so that we know that that comes from the particular word.
And even if the model do not have, say, maybe gamer, but it knows that it is related to the word game and maps it into the particular vector space of that. Next is stop words. So stop words are very general words like articles, and there can be use case stop words as well. So example, like say, if you're working with the legal data, the legal data has a lot of legal connotations that do not add any value to the particular model.
So how do you fix that? So you take out, you create a list of all the stop words that you don't want to use. And you basically see here, like say, hey, I am went away because I and am are like very redundant words and you and it doesn't serve any purpose to the meaning of the sentence.
And then you see how and then there are words like this. So these kind of do not have any value that could add into your model. And at the end, there are a lot of case based preprocessing as well. Case based is basically to understand that, say, like if you want to understand, you have a full sentence and you want to see if it's a pronoun or if it's a verb.
So you can create there are a lot of good tools under spacey that you can go and explore and then it will give you a great connotation of all your words. Great. So now that we have like a basic idea of what NLP or the basic preprocessing structure is set up, let's go and take a look at what transfer learning means.
So this is a great quote by Andrew Engie. He's like my favorite person all over the earth. Some of I encourage you guys to go and check out his tutorials. There's a lot of good tutorials by him in Coursera. There are a lot of open YouTube videos. You should definitely check them out. So he mentioned in an IPS conference 2016 that transfer learning will be the next driver of ML success.
And he mentioned in 2016 we are in 2020 and we know that the major models that all these big companies build are based out of transfer learning. So the applications are huge. The areas that we can explore are huge. We'll see how. So in a very quick connotation, I can explain transfer learning as it's just a technique where you train a model on a one particular task for which you have a lot of data set.
And then you use that model or you repurpose that model for a second related task. It doesn't have to be, it doesn't have to be the exact same task. You can definitely find you, which we were going to do today at the end of the session. So let's see how that works. So here's a very simple math of it.
Say you have a data set one, X and Y, and X here is just a general image and Y is the classified object. Like you can have a classification like the images of a tree or a train or a car or anything that you see in a everyday center scenery. So, and this, this data set is coming from image net.
It's a huge data set of like millions of image collected labeled over the years. So that's the perfect, perfect way to start your journey into NLP and any kind of computer vision applications. So now I'm going to put out a big chart. Don't get afraid of it. If you have not seen this before, but we'll go through it step by step.
And it's going to be like really simple to understand. So say when you're feeding X, Y, so this is a tensor diagram where you see that you're putting input in multiple formats. There are multiple layers that you are, that you are training the whole data set on. And then at the final layer, you get an output, which is in Y hat.
So basically it gives you a category implementation of which image, if you want to test it against, it will give you a classification based on all the data set that you have fed it into the model before. So one particular thing that changes with transfer learning is here. Say now you have a data set to X and Y tag and X data set is very small.
So like a very general use case example. Like if you want to classify between image, if it's from a rural or urban image. So how do you describe rural and urban image? In rural, you'll have like nice big lawns. You'll have nice big like areas, open grounds. And in urban space, you'll have like tall buildings. That's like a general nomenclature.
But how many do you think images that you can collect for this particular data set? It's going to be very difficult to collect so many data set, label it, clean it. So how do you achieve that with a higher level of accuracy in the first round? You use the data set from the ImageNet because the baseline parameters remain same, right?
Because the low level features like edges, curves, some of the features like if it's a horse or if it's a ground. These are some of the features or parameters that will remain same from the previous data set. And what we are doing here, if you see number step five in this case, we are basically fine tuning the last dense layer.
I call it dense layer because it's a 1D vector space array. And from there, we are feeding that layer to our data set too. And fine tuning it to make sure that the final output is now our two category definition, which is U and R, which is urban and rural for short. So the idea here is not necessarily all the time you will have the full data set or the full resources and time to train it.
So what do you do? You use data set that's very general and very closer to what the task that you're trying to achieve. And you do a transfer learning and fine tune from there. So let's see some of, so why do we do transfer learning? So I spoke a lot about this already. Say here are three examples where you use, like say you want to do speech recognition in Bengali.
Bengali is an Indian language. It's a regional language and not necessarily there are a lot of, say, documents or scriptures that we can actually talk about it. So how do we use it? Like say there are so many English and Hindi and other languages that are already available. So why not scrape that data from all the audio sources like YouTube and all those areas and then make make it a fine tuned version to understand Bengali because the underlying nomenclature of language remains the same.
And if we feed the pre-trained model and update the last layer with some of the dataset tagged clean dataset of the Bengali version, it will give us a pretty good accurate model for the first try. Another example that we just spoke about is the urban versus rural. Definitely we took image net with which had like a huge amount of datasets.
And then we trained from there. And even for text recognition, like if you want to work with a legal dataset and you want to do data classification, but you do not have a lot of legal data to train your model with. So you can use and scrape all general resources like news, blogs, scrape from websites, Reddit, Twitter.
You can have like abundant of general dataset from there and that can help you build the whole data structure. Let's see what we have next. So when to use. Yeah, we basically spoke about it when you have a scarcity of labeled data to start with. So creating a label data is expensive. Definitely needs time and experts like you need you need subject matter experts to come in, train the data.
If it's a very specific use case involved in the training process that you would need. The other is like you need to make sure that the final goal of your output matches with some of the pre-existing models that exist. It cannot be like too far away. It cannot be like exact complete opposite. The input properties of both task one and task two has to be same.
So the transfer of knowledge would only happen if you are transferring the knowledge from a general base to a very niche base, but they come from the same particular understanding. So. So let's see what's the difference between transfer learning versus traditional ML framework. So if you see from this image, it's very clear that in traditional ML, what we do is like we have multiple different tasks.
We have multiple different data sets for each of those tasks, and we train them to create a learning system. Now, how do we what are the things that we're doing different in transfer learning? So basically you have, say, a source task. Like we said, you have image net, which has all the images from all over your surroundings, and then you create a knowledge base out of that.
Now that when you're creating, now that you have that knowledge base, you have a target task. Like you have to classify between a rule and an urban image, and then you utilize that knowledge base. Do a fine tuning, which that final arrow shows from the target to learning and create a final learning system without creating a method or a model for each specific task.
So two things are happening here. We are reducing resources. We are also ensuring that we are utilizing already trained models that exist, and we don't overdo it, or we don't double our work. So we're utilizing what we already have. So now let's talk about BERT. I find this garden very funny. And if you see some of the models that Ruben have come up with, they named them in a very funny way.
There is Elmo. There is BERT. I guess they're a fan of Sesame Street. I am too. So they said, okay, I am BERT. How may I help you? So let's see what BERT is. When we want to learn about BERT, there is another very important concept that we need to ensure that we have a clear understanding about, which is word embeddings.
Word embeddings are the base of creating any natural language model, be it BERT, be it BERT, be it BERT, be it BERT, any other model that you can think of. So word embedding is the clear or a very simple definition of word embedding is that it's a feature vector representation of a word. Let's see what that means.
So when you see this diagram, what do you see? Do you see a connection? Can you point out one connection if you see what it means to you? Like I see that woman and queen are kindly connected. Then there's man and king. And also some of the other areas like slow, slower, slowest, which is like a different form of a simple word.
There's also mother, boy, daughter. So there are connections here. So we need to create this kind of connections among multiple various words based on where they have occurred in a particular sentence or in a particular document set. So the whole idea of word embedding is to create that whole vector space. You can call it a vector space definitely where you categorize and you cluster basically similar going words.
So let's look at it here. Great. So if you see some of these examples, sorry, if it's a little bit of stretched out image. But if you see the image, you see water, sea and arm in the left top are water and sea are like kind of together. Then you see climate, wind, ice. They're like kind of together.
Then you can see fish, eggs and meat are kind of together. So this is like a 2D vector space where vector space can be of multiple dimensions based on how much parameter of TSH you have. But in a very simple way to explain, the idea is to cluster similar feature occurring words into one cluster so that the machine or the algorithm understands that they come from a similar background or have the similar kind of meaning.
Let's say heat, electricity, oil, energy, fuel. They kind of mean the same, not same, like they kind of come from the same concept. So let's take a quick example in a 3D format. So here you see there are the 3D graph where you have sky, engine and wings. So all of us have. So where does helicopter lie?
So helicopter comes in between sky and engine. That sounds correct, right? Like it has an engine and it also flies in the sky. There is also drone. So it falls a lot between engine and sky, more of an engine, less than the sky. And then you see engine and rocket also falls. Now, if you see the first zero of helicopters, zero to four, zero means that they didn't find or couldn't find the model, any co-occurrences of helicopter and wings.
Yeah, I guess helicopters did deserve wings, but I guess it's not that prominent enough. But it did find wings between eagle and sky. So if you see the left hand side of the green arrows, you see eagle is termed as 303, which means that eagles fall between wings and sky. And it has zero connotation for the word engine.
So that's how basically in the vector space, words are mapped with different vector notations. And each of these becomes features into how we create a model at the end of the day. Great, going forward. So this is a Bert Mountain by Chris McMormick. He has like great YouTube tutorials. I would encourage all of you to go and check them out.
So he explained this in a very great way. So Bert is like the outcome of a lot of underlying technology that has been developed or developing for a long time now. So RNN, LSTM, Encoder by LSTM are some of the baseline things. But if you see these three at the top, if you know what I mean, Attention Transformer and Bert are kind of the baseline of where Bert is based out of.
So I know that was a lot of things. This is like the best way I feel that we should calm down. There are a lot of heavy concepts when we try and learn about NLP. But take a deep breath. Don't get too nervous. And always think of Transformers like I do as like Optimus Prime, that he's there to help us.
He's not there to scare us. So let's learn what Transformer is. It's a very simple concept. So that's the basic concept of a Transformer. What does it do? So here you have a German language input. The idea of Transformer is that it will take each of those words. It will encode it into a system. And then it will come up with an outcome as a decoder.
So there is an encoder decoder work that goes on in all Transformers. That's why they are called Transformers. They're transforming one language to another language by encoding each of those tokens or tokenizations from your particular language to another outcome that you would want. So here we are translating a German to an English language like say please come here.
So the application here seems very simple, but it has a lot of difficulty and a lot of internal neural networks that are actually working through it. So let's see what that means. So BERT basically the full form has a lot of heavy words. It's called bi-directional encoder representation from Transformers. So let's break it down. And what does that mean?
So the basic idea or the base to give you a overall view of the bi-directional encoder model is that it's trained on Wikipedia with almost 200, 2,500 million words. It's a big amount of words, like if you think about it, and also from, excuse me, and also from book corpus, which had like almost 800 million words. Now, the concept of BERT is we are, we are trying to learn from both directions, like Word2Vec.
When Word2Vec came, it was transformative because it was trying to understand what context the word is being used in, but it was still being trained at one level. So it was going left to right from one state. Now, BERT came in with saying that, no, we won't train one particular sentence from only left to right. We will also do it from right to left so that we know exactly which word comes left or right or positions in multiple areas and create vector space from there.
So that's the whole concept of why it's called bi-directional. It creates the encoder, represents it, and then transforms it. That's the whole definition or the short that's called BERT. The architecture layer has almost 12 layers, transformation blocks. It has 12 attention heads. We will just talk about very quickly what attention heads mean. And then it has 110 parameters.
So 110 million parameters. I missed a million in there. Okay. So it's a huge, huge, huge number. So BERT was the first bi-directional unsupervised pre-trained on plain text, basically trained on everything that's available in Wikipedia, which I guess it's huge because it's all the content available. And the final, I wouldn't see the goal as a BERT, but the whole concept of BERT is that it does input representation and word piece embedding.
It tries to find that concept and link it to other concepts of the whole structure so that it can find better meaning of what you're trying to achieve. Now take a look at this example and see how, if you, if you get a hold of it. So like the first sentence says that we went to the river bank and the second sentence says, I need to go to the bank to make a deposit.
So you see the same word that we are using here, which is bank, but it has definitely different context and which leads for it to mean different things in this two different sentences. So when we say we went to the river bank, it's obviously a river bank. So how does the, how does the model understand that it's two different contexts with the word, with the bi-directional word of finding the bank was once used next to river and another way to find that a deposit was used next to a bank.
So there are multiple other examples that would feed in to make that differentiation between river and deposit, which is like in a monetary or a financial, in a construct and create that meaning of bank, which is not necessarily one particular vector, but a multiple particular vector in two different vector spaces. When you remember when we saw the vector space, it won't, the bank word won't be represented as one feature.
It would be represented as multiple feature across the whole, the vector space. I guess it made a little bit clear. Like if you have any questions to let me know. So going ahead, BERT has three embedding layers. Word embeddings are like the same. We are creating layers. So tokens, position and segment. So what this means is if you take a look at this sentence, the input sentences, my dog is cute and he likes playing.
If you see, there are two additional things that BERT does here. It creates the CLS and it creates another token called SCP, which is like the separation. So basically BERT is trying to understand where a word or what a sentence is starting. And if there is a, where it is separating or if there is a full stop or if there is any kind of connotation, comma, exclamation marks, et cetera.
So here the word piece tokenization comes into play basically because it tries to understand and see. If you see the position embedding, it says zero, E0, E1, E2, E10, because there are like 10 tokens that it could find in the sentence. Now, if you see the segment embeddings, it says that EA, EA, and then there is EB.
So basically it categorizes the whole input that you have given to the model into two sentences. So that's great. Now, if you see token embeddings, it means that I see a question from Irina. Irina, I'll get back to you and we'll discuss that at the end of the discussion. Let's go and see that what token embeddings mean.
Now you see that after segments, segment embedding is done. It's because it's a sentence to sentence relation that this sentence was separate. And this is another separate sentence for the token embeddings. You see that it actually creates an embedding for each multiple token that it found. So how many layers you found? There are definitely three. And there are a lot of workings that goes on between all these three embeddings layers that finally creates the BERT layer.
So we'll see that how. So BERT is basically pre-trained on two NLP tasks. If you guys are, if I'm trying to like make it as simple as possible to like talk about the idea first, and then we will go into the code. So like the mask language modeling is basically this example. Like you say a man went to somewhere and he bought a milk.
So the way that BERT is trying to mask these ideas is because if there are multiple other words and because it is bi-directional, in a way BERT in the model will be cheating and would already know in hindsight that what those words are if it is not masked. So that is the purpose of why we need to mask some of the words in the whole sentences and then test the model to see if it can find the labels correctly like store or.
And for the second sentence, it's a gallon. Now the next sentence is like next sentence prediction basically. So if you're saying the man went to a store and the next sentence is he bought a gallon of milk. That makes sense, right? Like because he went to a store and he went to gallon of milk. So the label would be is next sentence.
These are the two separate operations of BERT to understand what is a particular mask words that's missing from the whole concept. And also to understand if one particular sentence is supposed to be a sentence for the next sentence. So if in the second example, you see a man went to the store, but the penguins are flightless. So it's not a next sentence, right?
It does not mean the same thing. So I guess we're going to look at some of the code first and come back to tension. If that makes sense to you guys. Can you guys see my screen and the code? Is it big enough? Okay. So if you, great. So if you see here, so I'm using Google Colab.
Google Colab is great. You guys should check it out. You can utilize GPUs, CPUs because BERT is a big model and your screen or your laptop will freeze if you're using a CPU in your laptop. So I prefer going and working out from Colab, which gives you, so you can change here, if you see, I guess under edit.
Oh, again, under runtime, you can change your runtime type. So you can make it a GPU or a TPU. So based on if it's a heavy task, I get to the GPU for this simple classification. So basically we are downloading all the pre-trained models from Keras. Keras is a great application package built by Facebook. And we're driving, we are mounting all the content in our Google Drive.
We are downloading the data set that we want to use to fine tune some of the models that we are using to understand if it makes sense to use it with the pre-trained model. Here we are downloading and using TensorFlow version one version two has already been released. So you guys can work with that as well. The basic idea, if you see here.
So we basically downloaded all the pre-trained models here. We trained the model on that pre-loaded checkpoint. And if you see the model summary here, basically you see each layer of the birth model. So there is an embedding model, embedding layer. There is an encoder model where it's doing a multi-head attention. And then there is definitely a lot of encoder layers.
So depending on how deep you want to make the network, like how many networks you're creating, you'll have that many level of encoder and decoders. So the next step for us, which is like just us using a very small data set to test if that makes sense, if the pre-trained model makes sense for our data. Like we are creating or importing another news group data from any news source.
You can download it from anywhere, any news set that you want. We are creating and labeling. So here we see that here are the text files. Text files is basically the tags for each of those sentences that we have in the new data file. And then when we are loading it, we are basically loading and creating in a very similar ML way.
We are creating a train and test text files. Here we see that the amount of data for each tag is kind of like in equilibrium. There is not too much discrepancy. Now here is what we are doing. The one final step that we are doing for fine tuning our model. Here it goes. So you're basically calling the last dense layer.
You're getting the layer, which is called get layer NSP dense. And you are fitting that dense layer with a softmax function, which is an activation function. I encourage you guys to go and look at a lot of tutorials. There are two that talks about activation and which functions to use depending on your use case. And then you're fitting that whole model into another compiling that changes into your model to fit it into a new fine tuning section.
Now you have another, you can just create model that's a new model. And here if you say I'm training the model and I'm fitting it with an epoch size of 100. Sorry, there was too much talking. So, yeah. So epoch size of 100. It runs for 100 times with the sequence length and all of the batch size that I provided in the code above.
And at the end, if you see, when you calculate the prediction parameters, it gives you a 0.52. Now think about it. You have like a 50 or maybe say 100 data set column and you want to create a model. It's impossible to start with that kind of less data set. But with the pre-trained model in the first round of iteration, we received almost 52% accuracy, which for me is like pretty good or it's a pretty big deal.
So I guess if we have a little bit of more data to fine tuning this whole method or if we use some other parameters to fine tuning, do some hyper parameter tunings, we're going to get great results. So if you see the final, like I use some additional text that is from outside of the data set that we tested on.
I tested about some Democrats that are having hour long talkathon. So basically it talks about and gives you a thing of classification of politics and guns. So, yeah, it's not too difficult. It's just a step by step understanding of what we want to achieve and going through it by understanding some of these concepts. Now, I guess we're running out of time, but I'll just quickly go through it.
So where we are with NLP now. If you see, Google has released the conversation. This is a very funny conversation. I will like if it's possible when they shared the video, you can take a look at this conversation. It's called Mina and they can basically talk about anything. And then they said at the end, they said that there was a multi turn joke in an open domain and they didn't find any of those words in the training data set.
So basically it came up with its new data set. So it's a huge advance for NLP domain. The final new OpenAI GPT-3 that was released just like 15 days ago had 175 billion parameters. That's a huge number, but there are a lot of things that we need to do to ensure that it's pretty efficient in the way we are applying it into everyday use cases.
So I will end up in a very crucial note that we need to understand that creating a model is very challenging. It takes the resources. If you see this chart that MIT released, training a single model emits as much as like five cars, carbon from five cars in their whole lifetime. So it's a big data. It's a big number.
And we need to be aware that we are not just creating models for the sake of it, but for solving very particular problems. And utilizing or reutilizing already available methods so that we are not redoing it again. So closing remarks. It's been a great discussing with you guys, but NLP is hard, but I guess it's hard because human intelligence have set like a huge benchmark and we need to make sure that we do enough to kind of reach as close as possible.
A few guidelines like bad data cleaning means it will get harder, always know your data better than others. Because if you do not know your data and you just start on creating a word model, it's not going to work out and you would be like frustrated saying that why is it not working. A lot of people say NLP is magic, but it's not.
There is a lot of logic, a lot of concept of neural networks, a lot of understanding of the word embeddings, which takes take a step by step learning at each of it. And these are some of the additional resources that I put it up. If they are going to put the video out, you can all have the resources.
Great. Thank you so much, all of you guys. It was great. If you have any quick question, I can take at the end now. And thanks. And keep learning and take one step at a time. Don't be afraid of learning and thinking that it's not your cup of tea. It's everybody's cup of tea. And we just need to take one step at a time approach.
And if you believe you can do it, everybody will believe that you can do it. So just do it and don't think about it much. Is there any question that I can take or quickly answer at the end? Because I guess we are running out of time kind of. Great. Thank you. Thank you for such great reviews.
Is there any particular question? Or I guess I'm going to see that I upload the code at the end of the session in my GitHub account. And you can follow me there and utilize the code. Let me know if you have any questions. Reach out to me. Yeah, definitely. I can share the slides on the Google Colab notebook.
Definitely. For corrections. For menu, corrections function. It is actually a function that I mentioned in the earlier. It was developed by Google. So if you look at the code, they have, I have mentioned the name of the authors that have built that whole code base. So you can, that's a definition function that you can call from their already open sourced code.
So it's already out there. Are there any other questions that I can answer? Thank you so much, guys. Like, you guys have been, like, so helpful and, like, supportive. And I guess learning anything new needs a lot of making it light and funny. So I get these transformers and cute videos that makes me think that, hey, it's not that difficult.
So great. I guess one last question. Do you think transfer learning is something we could use to replicate something like user personalization and voice assistance? I guess, yeah, that makes a lot of sense. I guess if you have the context built from a very, yeah, very lesser causing some voice. I guess the voice is the area that audio is the area that it's still developing and there is a lot of work that's going on.
But I guess the work or the application that you mentioned is definitely doable. But it would mean that we can do a lot of voice modulations or we can do a lot of voice fine tuning at the end of understanding those models. But we'll see where we go with that. There is a lot of scope, definitely. And, yeah, I guess it's very much doable.
Thank you, everybody, for joining. Do share. I'm going to put my LinkedIn and Twitter for you guys to ask any questions. Join me. Connect with me. I'm happy to learn. This is, like, a great session where we can learn from each other. Thank you. And keep learning. I guess that's the end of the session.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why reuse a pre-trained BERT model instead of building a text classifier from scratch
Data scientist Jayeeta Putatunda walks from text cleaning to transfer learning and fine-tuning BERT, showing how a small dataset can still produce a working classifier.
In short
Building a text classifier from scratch is expensive when labelled data is scarce, and Jayeeta Putatunda, a data scientist at Indellient US Inc., argues it is often unnecessary. Transfer learning lets practitioners take a model such as BERT, pre-trained on billions of words, and fine-tune only its final layer for their own task. Her talk moves from the basics of NLP preprocessing through word embeddings and Transformers to a live Google Colab notebook that classifies news text.
Key takeaways
Clean raw text with core preprocessing steps: stripping extra spaces, tokenization, spell checking, contraction mapping, stemming and lemmatization, and stop word removal.
Explain how transfer learning reuses a model trained on a large general dataset and fine-tunes its final layer for a smaller, related task.
Judge when transfer learning fits: labelled data is scarce or expensive, and the source and target tasks share the same input properties.
Describe how BERT's bidirectional training and its token, position and segment embeddings give the same word different meanings in different contexts.
Fine-tune a pre-trained BERT model for news classification in Google Colab by refitting its final dense layer with a softmax activation.
Why NLP is hard, and why clean data comes first
Putatunda opens with a deliberately ambiguous sentence: "I am a huge metal fan." A person instantly sees two possible meanings, a fan made of metal or a fan of metal music, but an algorithm needs large amounts of data to learn that distinction. She is candid about the difficulty: "So a lot of the time people feel that NLP is hard. And to start with, it definitely is hard." (1:10)
Her answer is to go step by step, starting with data quality. Before any modelling, she covers the preprocessing that every NLP project needs: stripping extra spaces, tokenizing sentences into words with libraries such as NLTK, cleaning characters with regular expressions, correcting spelling with Peter Norvig's spell checker, expanding contractions, reducing words like "gaming" and "games" to the root "game" through stemming and lemmatization, and removing stop words that add no meaning, including domain-specific ones in areas like legal text. The principle behind all of it is familiar: "So if you feed the model with very heavy garbage text and it's not clean, not structured, then there is no way that we're going to get a good good model or a good accuracy state." (7:51)
Transfer learning: reuse before you rebuild
The core idea of the talk is transfer learning, which Putatunda frames with Andrew Ng's prediction, as the speaker cited, at a 2016 conference that it would be the next driver of machine learning success. Her definition is simple: "So in a very quick connotation, I can explain transfer learning as it's just a technique where you train a model on a one particular task for which you have a lot of data set." (14:21)
She illustrates it with images. A network trained on ImageNet already knows low-level features such as edges and curves, so classifying urban versus rural photos only requires fine-tuning the last dense layer on a small labelled set. The same logic applies to language: speech recognition in Bengali can build on abundant English and Hindi audio, and a legal text classifier can start from general news, blogs and social media text. The conditions matter, though. Transfer learning helps most when labelled data is scarce and costly, and when the source and target tasks share the same input properties.
How BERT reads words in context
Before BERT, Putatunda explains word embeddings: feature vectors that place related words, such as king and queen or heat and fuel, close together in a vector space. BERT builds on a lineage of RNNs, LSTMs, attention and Transformers, whose encoder-decoder design she compares to translating German into English.
What sets BERT apart is direction. Earlier approaches like Word2Vec read context one way. "Now, BERT came in with saying that, no, we won't train one particular sentence from only left to right." (28:35) Reading in both directions lets the model separate a "river bank" from a bank where you "make a deposit." She walks through BERT's token, position and segment embeddings, its special CLS and SEP tokens, and its two pretraining tasks: masked language modeling, where hidden words must be predicted, and next sentence prediction, where the model judges whether one sentence logically follows another.
Fine-tuning BERT in Google Colab
The live demo runs in Google Colab, which Putatunda recommends because BERT is large enough to freeze a laptop CPU and Colab offers GPU and TPU runtimes. She loads pre-trained BERT checkpoints, inspects the embedding and multi-head attention encoder layers, imports a small labelled news dataset, and refits the final dense layer with a softmax activation before training for 100 epochs.
The result, from a dataset of roughly 50 to 100 rows, is modest but telling: "But with the pre-trained model in the first round of iteration, we received almost 52% accuracy, which for me is like pretty good or it's a pretty big deal." (38:52) She notes that more data and hyperparameter tuning would improve it, and tests the model on an outside article about a political talkathon, which it places under politics and guns.
The cost of models and the case for patience
Putatunda closes with the state of NLP in mid-2020, including Google's Meena chatbot and OpenAI's GPT-3 with 175 billion parameters, and a warning about cost. As the speaker cited from MIT, training a single model can emit as much carbon as five cars over their lifetimes. Her conclusion: "And we need to be aware that we are not just creating models for the sake of it, but for solving very particular problems." (41:24)
Her final guidance is practical and encouraging. Know your data better than anyone, because poor cleaning makes everything harder, and remember that "A lot of people say NLP is magic, but it's not." (41:58) It takes logic, neural network concepts and a step-by-step approach, and she insists it is within reach for anyone willing to learn.
Watch the talk
"Transfer Learning With BERT: Building a Text Classification Model" by Jayeeta Putatunda (46 minutes): https://www.womentech.net/video/jayeeta-putatunda-transfer-learning-bert-building-text-classification-model
About the speaker
Jayeeta Putatunda is a Data Scientist at Indellient US Inc. with a specialization in Natural Language Processing and a Masters in Quantitative Methods and Modeling. She works on machine learning and big data projects, previously worked as a research analyst at Deloitte, and is actively engaged with organizations that inspire more women to take up STEM.
Questions this talk answers
Why is natural language processing hard?
What is transfer learning?
When should you use transfer learning?
How is BERT different from earlier embeddings like Word2Vec?
Which tasks is BERT pre-trained on?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Transfer learning
BERT
Transformer encoder-decoder architecture
Masked language modeling
Next sentence prediction
Word embeddings
Word2Vec
Garbage in, garbage out
Resources mentioned
Peter Norvig's spell checker
Andrew Ng's tutorials on Coursera and YouTube
Chris McCormick's YouTube tutorials
Google Colab
NLTK
spaCy
Speaker's GitHub code repository for the talk
Cite this talk
Jayeeta Putatunda. "Transfer Learning With BERT: Building a Text Classification Model." WomenTech Network, 2020. https://www.womentech.net/video/jayeeta-putatunda-transfer-learning-bert-building-text-classification-model