NLP 101: Natural Language Processing and The Era of Transformer Models
Ana-Maria Istrate38:102023
Why reading every word at once made transformers beat recurrent networks
AlphaFold cracked a protein-folding problem that biology had not solved in over 50 years, and ChatGPT can plan a trip to Hawaii, yet both rest on the same architecture. Ana-Maria Istrate, a Senior Research Scientist at the Chan Zuckerberg Initiative who works with these models daily, starts from plain definitions for newcomers. The question she builds toward: what changed when models stopped reading text one word at a time?
Learn how embeddings, self-attention and pre-training combine in BERT, GPT and T5
Ana-Maria Istrate, a Senior Research Scientist at the Chan Zuckerberg Initiative, gives a beginner-friendly tour of natural language processing, from common tasks such as question answering and entailment to word embeddings, recurrent neural networks and self-attention. She explains why transformer models overtook recurrent networks and how BERT, GPT and T5 are pre-trained, fine-tuned and best used.
“So this is a model developed by DeepMind and it's really impressive because this is actually a problem that has not been solved in biology for over 50 years.”
“So to recap the reason why these transformer based models have become so popular is because they have reduced training times compared to RNNs and information doesn't get lost.”
What you will learn
Distinguish the main NLP tasks, from named entity recognition and summarization to entailment and text generation.
Explain how word embeddings capture a word's meaning from the contexts it appears in.
Compare why self-attention outperforms recurrent neural networks on training speed and long-range accuracy.
Choose between BERT, GPT and T5 depending on whether a task needs understanding or generation.
Use pre-trained models through fine-tuning, prompting and few-shot examples instead of training from scratch.
Key takeaways from the speaker
Types of NLP tasks
Building Blocks for NLP Models
The Transformer Model Architecture
Examples of transformer-based NLP models
Details
Level
Beginner
Format
Tech talk
Technical depth
Introductory technical
For
Student, Early career, Career changer
Length
38:10
Published
June 2023
Questions this talk answers
What is natural language processing?
It is a branch of AI covering a machine's ability to understand or process written and spoken human language. Not every NLP model is a machine learning or deep learning model.
How does a computer represent the meaning of a word?
As a word embedding, a vector of numbers built from the contexts in which the word appears. Words used in similar contexts end up close together in the embedding space.
Why did transformers replace recurrent neural networks?
Transformers read the whole input at once, so they parallelise well on modern hardware and train faster. Self-attention also preserves relationships between distant words that RNNs tend to lose.
How do BERT, GPT and T5 differ?
BERT uses the transformer encoder with bidirectional context and suits classification and entity extraction, while GPT uses the decoder to predict the next word and suits text generation. T5 uses both encoder and decoder and is pre-trained with span corruption.
How can you use a huge language model without training it from scratch?
Fine-tune it: initialise a new model with the pre-trained weights and train it on a smaller labelled dataset for your task. You can also describe the task in a prompt, optionally with a few examples.
About this talk
Machine Learning, AI and NLP have become part of the daily tech vocabulary. Companies use ML techniques to acquire users, drive revenue and increase profits. It seems that ML is everywhere. But what are the types of things you can do with machine learning, in particularly natural language processing?
What exactly is NLP? How does it work? What are transformer models? How does the chatGPT (Chat Generative Pre-trained Transformer) model work?
If you’re new to the NLP space, and would like to understand more about what is going on under the hood, or the types of things you can use NLP for, come to this talk! We’ll start broad, going over NLP use cases, followed by understanding how some NLP foundational blocks, such as transformers, work. We'll end with looking at a few specific NLP models (including chatGPT!) in more depth.
Prompt engineering replaces task-specific training (81 s)
How self-attention works out what "it" means (69 s)
Language models that read proteins (51 s)
Ana-Maria Istrate: Okay, so maybe we can get started. Hi, thank you so much for being here today. My name is Ana-Maria Istrate and I am a senior research scientist at Chan Zuckerberg Initiative, where I work in machine learning and particularly in natural language processing to support our teams in science. Ana-Maria Istrate: And I am so incredibly excited to be here with you today and talk about natural language processing and transformer models.
Ana-Maria Istrate: I work with some of these models we'll be talking about today on a daily basis as part of my job. So I'm incredibly excited to share some of this information with with you today. Ana-Maria Istrate: So we live in really exciting times. There are many breakthroughs in AI and ML and particularly natural language processing. Surely, many of you have heard of or interacted with with systems like chat GPT, which is an intelligent conversational agent that is able to do things like answer questions intelligently summarize perform chain of thought reasoning.
Ana-Maria Istrate: Perform equations perform equations write code and even as a picture here plan a trip to Hawaii. Ana-Maria Istrate: This is a system that has been developed by the team at open AI, who prior to chat GPT also work on. Ana-Maria Istrate: Dali II, which is another really impressive machine learning generation system. This is a text to image generation system where you can give an input such as a text input, such as an oil pastel drawing of an annoyed cat in a spaceship and then the model is able to give you that and give you back a picture of an oil pastel drawing of an annoyed cat in a spaceship.
Ana-Maria Istrate: Which I find pretty astonishing. So there's other examples such as alpha folk is a machine learning model that has been used to predict the 3D folding structure of a protein from its amino acid sequence. Ana-Maria Istrate: So this is a model developed by DeepMind and it's really impressive because this is actually a problem that has not been solved in biology for over 50 years.
Ana-Maria Istrate: So being able to predict the 3D folding structure of a protein from its one what they call the 1D structure or just the amino acid sequence that's that's just not been solved in machine learning has been able to do it. Ana-Maria Istrate: Alpha code is similarly a transformer based model that's been able to generate code that performs so well that is at the top of coding competitions.
Ana-Maria Istrate: Google search uses birth, which is a transformer based model to understand the meaning behind the users question. Ana-Maria Istrate: So um what all of these models or systems have in common is that they use deep learning model architectures that at the core are based on transformer models, and that is what we will be talking about today.
Ana-Maria Istrate: So today, in the time we will spend together, we will try to understand how we came to have these transformer model architectures and look at some examples and understand how they work. Ana-Maria Istrate: We will start fairly broad and we will end up quite technical towards the end of the end of the talk so first we'll start with.
Ana-Maria Istrate: A broad introduction to natural language processing and what are some common NLP tasks we will continue by looking at some foundational NLP blocks so we'll talk about word embeddings recurrent neural networks this were. Ana-Maria Istrate: These were the state of the art models before transformer based models in natural language processing and we will also look at the mechanism of attention, which is what ultimately led to this.
Ana-Maria Istrate: These really performant transformer models. Ana-Maria Istrate: We will look at the transformer model architecture and that deeper going into some more details on particular some particular large language models. Ana-Maria Istrate: such as birth GPT and T five. Ana-Maria Istrate: So that being said, I am super excited to go on this journey with you today and let's get started.
Ana-Maria Istrate: um. Ana-Maria Istrate: First of all, you know, to start by the definitions, what exactly do we mean by not by natural language processing. Ana-Maria Istrate: or NLP. Ana-Maria Istrate: NLP is a branch of artificial intelligence and if we think about natural language as the language in which. Ana-Maria Istrate: We as humans communicate with each other so whether that's through speech or text.
Ana-Maria Istrate: The natural language processing is just the ability of a machine to understand or process the written and spoken human language. Ana-Maria Istrate: Not all natural language processing models are necessarily machine learning models or even deep learning models. Ana-Maria Istrate: And not all machine learning models are deep learning models either. Ana-Maria Istrate: When we say deep learning models, we usually mean these really massive model architectures that have hundreds of millions or even billions or more recently with charged GPT or GPT for trillions of parameters.
Ana-Maria Istrate: So the intersection of these really large language models, which we call deep learning models and natural language processing, this is where where these large language models live. Ana-Maria Istrate: So now we want to talk about some of the some of the things you can do with these models, so we know what are some of the types of.
Ana-Maria Istrate: of tasks you can train these machine learning learning models to do or to help you with and that's what we're going to go over the next few slides. Ana-Maria Istrate: The first one question answering is here is basically you want to train a model to. Ana-Maria Istrate: To retrieve answers to questions that you're posing in natural language, so this is one of the features that we do see in in chat GPT, for instance.
Ana-Maria Istrate: Here, given a passage of text you can pose the question in natural language to the model to the model and then you retrieve the answer back and hopefully the answer is correct that's the goal. Ana-Maria Istrate: In this example, we have a sports based passage, you know, we have some questions such as which NFL team won Super Bowl 51 this AFC stand for.
Ana-Maria Istrate: What year was Super Bowl 50 and in each case the mother learns to identify the correct answer from text and here we have a passage, but, of course, you can extrapolate this and. Ana-Maria Istrate: and build models that aim to answer questions from documents research papers, you know, maybe chapters of books and other more complicated things like that.
Ana-Maria Istrate: Another types of type of task is called name entity recognition so here given a text you want to locate and classify the name entities mentioned in the text. Ana-Maria Istrate: So let's say we have the sentence on the 50th of September, Tim cook announced that apple wants to acquire ABC group from New York. Ana-Maria Istrate: For $1 billion and we have a number of entities, we are interested in such as date person organization GP, which stands for geopolitical political entity.
Ana-Maria Istrate: or money and we train a model that is able to identify these entities and text and tell us that the 50th of September is a date team cook is a person apple and ABC group are organizations New York is a location and $1 billion is money. Ana-Maria Istrate: Summarization is when you want to add it's very core is when you want to reduce the text size of a document by creating a summary.
Ana-Maria Istrate: And you can do this mainly in two ways so extractive summarization is where you're just extracting various kinds of text. Ana-Maria Istrate: From our various words from the text and patching them together, and we have an example here. Ana-Maria Istrate: Peter and Elizabeth took a taxi to attend the night party in the city, while in the party Elizabeth collapsed and was rushed to the hospital.
Ana-Maria Istrate: So the summary we're getting here from the model is Peter and Elizabeth attend party city Elizabeth rushed hospital, and you can see that you know this is not necessarily wrong it does convey the most important information. Ana-Maria Istrate: And it's probably also similar to how people would text, but it might not make the most grammatical sense and the model is just identify learning learning to identify the most important words and text and patching them together.
Ana-Maria Istrate: abstract the summarization in contrast is where the model learns to paraphrase and generate the summary that makes more logical sense. Ana-Maria Istrate: So you can see that the model here learn to paraphrase more intelligently and that the summary Elizabeth was hospitalized after attending a party with Peter. Ana-Maria Istrate: It makes it sounds sounds better and more coherent.
Ana-Maria Istrate: sentiment analysis is where we want to classify the polarity of a given text so, for example, maybe you want to classify something as positive negative. Ana-Maria Istrate: neutral whatever you wanted to classify as maybe you can even look at political types of quest classifications. Ana-Maria Istrate: We have here two examples from Twitter first one fresh baked bread bread and pastries are some of the great joys of life is clearly a very positive positive take.
Ana-Maria Istrate: A second one i'm not going to read it but it's a little bit more negative so we're able to pick that up. Ana-Maria Istrate: machine translation is something we've all probably been using at least once or twice, especially if you've used something like Google translate or maybe travel and it's where you want to translate a piece of text or speech from one language to another.
Ana-Maria Istrate: entailment is a really interesting task so it's actually one of my favorite tasks it's when you train a model to discern whether one sentence can be inferred from another. Ana-Maria Istrate: Basically, you want to understand do these sentences mean the same thing so in this case we have the sentence an old woman walking down the street, and we have two examples that.
Ana-Maria Istrate: Or we have multiple examples the first two and while an old woman is walking outside and a lady is walking down the street, we can train a model to identify that these two examples can be entailed from our premise. Ana-Maria Istrate: The second two or the basically they mean the same thing the same thing the same to the second pair of sentences a young lady is walking and ladies sleeping on the street.
Ana-Maria Istrate: These are contradictory to our premise, because we know the lady is not young and we know the lady is not sleeping because the lady is walking so we can confidently say these are contradictions. Ana-Maria Istrate: The third pair is even more interesting because we have you know an old woman in red top walking and woman walking with her dog.
Ana-Maria Istrate: These could or could not be true, we just don't know because we don't have a lot of we don't have enough information based on our premise to to be able to tell. Ana-Maria Istrate: So this is super interesting because the model has to be able to understand and not just the meaning of sentences. Ana-Maria Istrate: But it has to reason over them and find find accuracies or inaccuracies between the premise and the sentences that it's looking at.
Ana-Maria Istrate: You can also do things like sentence similarity, which is where you want to determine how similar to pieces of texts are. Ana-Maria Istrate: So, in this case, we have a sentence machine learning is so easy and then a number of other sentences we wanted to compare to rank in order of the similarity score. Ana-Maria Istrate: Of course, if we look at something deep learning is so straightforward that's quite similar to say machine learning is so easy.
Ana-Maria Istrate: If you say something like I can't believe how much I struggled with this that's almost a contradiction so the similarity score will be very low. Ana-Maria Istrate: And you can do this kind of similarity, not just at the sentence level, but you can do it at paragraph level or document level again, you can kind of increase in complexity as much as you want.
Ana-Maria Istrate: The last the last type of task I wanted to talk about this text generation, so this is where you want to produce new text based on a prompt and this is a lot of what chat GPT and these GPT models have been trained to do. Ana-Maria Istrate: So you give to your model a prompt such as once upon a time and then the model learns to generate texts from that prompt.
Ana-Maria Istrate: And one more interest the recent interesting area is this area of prompt engineering, which is where you want to embed. Ana-Maria Istrate: The description of the task you expect the model to do in the prompt itself. Ana-Maria Istrate: So the model is then able to figure out the task it's supposed to do from your natural description of the task so instead of programming the model to.
Ana-Maria Istrate: For a particular task you'd be able to just get tell the model in natural language what that task is. Ana-Maria Istrate: So you know if you go back to our example of on the 15th of September, Tim cook announced. Ana-Maria Istrate: That apple wants to acquire ABC group from New York for $1 billion we were looking at extracting various types of entities from the sentence.
Ana-Maria Istrate: So instead of training a model specifically for that the name and the recognition model, we can. Ana-Maria Istrate: tell a GPT model that we want to you know extract all of the important entities mentioned in the text and tell it first extract all company names and extract all people names and extract what other types of things were more interested in.
Ana-Maria Istrate: So um. Ana-Maria Istrate: The idea here is that your model learns to do that based on just this general knowledge without you having to particularly tune it for that. Ana-Maria Istrate: And this is why it's called one shot learning where the model has not particularly trained for this task but it's still able to do it really well.
Ana-Maria Istrate: And in some cases, you can even give it a few more examples and hope that the model learns from those few examples and that would be called few shot learning. Ana-Maria Istrate: And this is a fairly recent area that sort of became more popular with with these chat GPT models so it would be really, really interesting and exciting to see this kind of where this area goes.
Ana-Maria Istrate: um this kind of concludes the first the first portion of our time today so we've covered what is natural language processing and what are some types of natural language processing tasks. Ana-Maria Istrate: We will now move into the second portion of today's talk so looking at some of foundational NLP blocks and, in particular, looking at word embeddings aren't aren't sorry ever RNNs which are type of machine learning models and also attention.
Ana-Maria Istrate: So earlier in the talk we define natural language processing as the ability of a machine to understand or process written and spoken human language. Ana-Maria Istrate: And a word is one of the very fundamental blocks of natural language so then one of the very first questions we want to ask ourselves is. Ana-Maria Istrate: How can we represent the word in a way that the computer is able to understand.
Ana-Maria Istrate: The computer obviously can't understand human language it doesn't have a notion for that, however, it can handle numbers right so then we want to be able to represent the meaning of a word through what we call a word embed a word embedding or a vector. Ana-Maria Istrate: And these are vectors of numbers that correspond to meaningful representations of a word.
Ana-Maria Istrate: The next question becomes you know how do you choose these numbers such that you get these meaningful representations of these words and here one of the key ideas is that a word's meaning. Ana-Maria Istrate: is given by the words that that word appears frequently by and there then there's this quote of you shall know a word by the company it keeps.
Ana-Maria Istrate: That what that means is that when you're computing a words embedding you're looking at its context or the set of other words that it's surrounded by. Ana-Maria Istrate: So when you build a vector representation of that word you use the many contexts in which you see that word and then you aggregate over them. Ana-Maria Istrate: And then you can expect that words that appear in similar contexts will end up having similar similar word embeddings and this can be measured by doing something like a dot or cosine similarity because again these are just vectors.
Ana-Maria Istrate: numbers so you can actually look at how close they are in the embedding space and we have here just a few examples, so if you look at a cluster of yellow things on on the top, you see that things like oven refrigerated refrigerator microwave. Ana-Maria Istrate: end up being clustered together the maybe the cluster of red things on the bottom finish color and paper are also clustered together and so on, and this is likely because these words had similar context in the sentences in which they appeared when the model was trained.
Ana-Maria Istrate: So, as we know words don't occur in natural language and isolation, they do appear in sentences so then, once we have these word embeddings the next question becomes. Ana-Maria Istrate: How can you model the relationships between words in a way that the computer is able to understand. Ana-Maria Istrate: And one of the most successful models in natural language processing to do this and that was state of the art before transformer models came came to be.
Ana-Maria Istrate: Is is recurrent or are recurrent neural networks or RNNs and the idea here is that you have a neural network that is able to model sequential data. Ana-Maria Istrate: And information is flowing from left to right, so you have acceptance like the students open their books. Ana-Maria Istrate: And what you first do is you compute some embeddings for each of these words so you can do that we do that through a look up or you know some some learning bad things.
Ana-Maria Istrate: And then you pass these so this would be like the very first purple layer that we have here. Ana-Maria Istrate: Then you would pass these embeddings through some hidden layers age and then each word would be connected to the ones that follow it so you can see this kind of arrows from left to right between the.
Ana-Maria Istrate: Between the between these these cells in the in the red the second layer the one that that's more red and these are these arrows what these arrows signify is that information is flowing from left to right so from previous words to to following words. Ana-Maria Istrate: And this is how the model is able to hold information and have this notion of a past and the memory between words so then when we're let's say we're trying to compute the representation of the last word in the sequence here there.
Ana-Maria Istrate: We technically have still the information that's been flowing from each of the previous words, including very first one, which is the word them. Ana-Maria Istrate: So this was a revolutionary idea and it's what made these models become beast becoming I stay state of the art for for a while before transformers. Ana-Maria Istrate: In practice, it is kind of hard to keep track of these long range dependencies between words.
Ana-Maria Istrate: Particularly if your sequences gets really get really get really long and information might get lost, which is why we we have seen kind of like this development of the of new of new and more performant models. Ana-Maria Istrate: So the other kind of the other notion I want to talk about is this groundbreaking which was also groundbreaking concept in natural language processing.
Ana-Maria Istrate: As this notion of self attention so i'm here again we want to computer the representation or embedding of a word and wants to know. Ana-Maria Istrate: What other words in the sequence should I pay attention to to compute my owning my own meaning, so this is kind of this notion of self attention what what else in my.
Ana-Maria Istrate: In my own in my own sentence should I be paying attention to so as an example, we have the sentence the animal didn't cross the street, because it was too wide. Ana-Maria Istrate: And if we want to computer presentation for the word it we want to know. Ana-Maria Istrate: Is it referring to the animal or the street and as humans, we are kind of able to say that okay well it probably refers to the worst street not the animal because the animal was not too wide.
Ana-Maria Istrate: But then, how the model is able to understand this is by computing these attention scores between the word it and every other word in the sentence. Ana-Maria Istrate: So these attention scores end up determining how much of these other words should contribute to the next representation to the word representation of the word it. Ana-Maria Istrate: And we see that, in this case, we have a model that learns to attempt to the word street the most.
Ana-Maria Istrate: So the cool thing here is that unlike these RNNs models we've talked about where we had the sequentiality you don't need to have the words in a sequence to be able to compute these scores, you can you can feed your whole input sequence at once. Ana-Maria Istrate: Instead of one word at a time and this notion of self attention is actually what is at the core of the transformer model architecture, which is what we will go into next.
Ana-Maria Istrate: So um this transfer model architecture is at its core it's it's a neural network architecture and again, unlike RNNs recurring neural networks you don't have this notion of sequentiality. Ana-Maria Istrate: Whereas don't get read once at a one at a time, but the input is get seen by the model at once, and they are based on this idea of self attention.
Ana-Maria Istrate: So we have here kind of like on the left of the slide this this is what the transformer looks like it's based on an encoder decoder architecture. Ana-Maria Istrate: The encoder at least the very first version has six layers and process it's in charge of processing the words in the input sequence. Ana-Maria Istrate: The decoder usually you know how again that the baseline one has six players and it's in charge with outputting the answer.
Ana-Maria Istrate: So you can see that inside of each of these architectures both encoder and decoder you have what it's called multi head attention, and this is where the self attention come comes into play. Ana-Maria Istrate: Because what this what this this represents is you have multiple attention heads where each of these each of these attention heads represents a different type of attention that the model is learning.
Ana-Maria Istrate: So the model is learning different types of relationships between words by attending to different things and different time and learning those things. Ana-Maria Istrate: And you know it may again maybe those can be different types of relationships between words various degrees of semantic meaning and other types of more and more intelligent things. Ana-Maria Istrate: This this notion or this this model architecture has has been used quite widely in a natural language processing and it's at the core of these large language models that we're seeing it's also using computer vision and.
Ana-Maria Istrate: yeah it has surpassed against RNNs in terms of being state of the art. Ana-Maria Istrate: So um what what what why why is that I want to take just a little bit of time to explain that so. Ana-Maria Istrate: For RNNs you have this sequentiality again and reading one word at a time is that if you want to you know compute embeddings or representations for for for words that are very far away from each other.
Ana-Maria Istrate: Then let's say maybe you want to you know computer relationships or like keep track of the relationship relationships between the first and the last word in your sequence. Ana-Maria Istrate: Then that would be an O of sequence or your sequence length that that that's the amount of number of steps you want you have to take to compute those relationships.
Ana-Maria Istrate: And because of the sequentiality these operations are not paralyzable, so you have then you end up with all of sequence of length unparalyzable operations. Ana-Maria Istrate: In contrast again for the transformers the input is read all at once you perform actually a very small constant number of steps every time. Ana-Maria Istrate: And because we have seen these many advancements in machine learning hardware architecture recently that are really good at parallel processing.
Ana-Maria Istrate: That makes transformers better suited for this machine learning hardware because they are they are way easier to paralyze than recurrent neural networks because they're missing the sequentiality. Ana-Maria Istrate: So this in turn leads to faster training up to an order or many, maybe even more of magnitude. Ana-Maria Istrate: The other reason is that in RNNs, we have information we have this information point from left to right right but the information might actually get lost, especially for longer sequences because it's pretty hard to learn these long distance dependencies between words.
Ana-Maria Istrate: And, in contrast, in the transformer architecture, you have again this notion of self attention, which means that. Ana-Maria Istrate: You can model these relationships between words at once, regardless of the respective position of a word in the sentence and information is just better preserved between words in the sentence because of that and it's actually it's been shown to lead to higher accuracy, regardless of you know training time.
Ana-Maria Istrate: So to recap the reason why these transformer based models have become so popular is because they have reduced training times compared to RNNs and information doesn't get lost. Ana-Maria Istrate: In these long term dependencies so you have this perfect combination of you know we have better hardware that's better optimized for paralyzation you have these transformer models that are easier to paralyze and you have you know they're also.
Ana-Maria Istrate: higher performance because they're able to to to use this notion of attention. Ana-Maria Istrate: So this is why we've seen this increase in development of large transformer based models, not just language models, but you know just generally transformer based models. Ana-Maria Istrate: Some of the large language models, such as bird I mean yeah bird GPT T five, which is what we're going to going to talk about are all based on this this notion of of transformers.
Ana-Maria Istrate: So how did these transformer based models learn or these large language models learn um the main way in which these models learn is through this process that's called pre training, so. Ana-Maria Istrate: The idea here is that you start with some large unlabeled corpora so a lot of these models use corpora that's been scraped from the web or maybe.
Ana-Maria Istrate: Large corpora of books or you know i'd like Wikipedia something like that so you want to start with some really large and labor only label corpora. Ana-Maria Istrate: And then the really cool thing is that you don't need to have any sort of supervised or labeled data to to learn from this corpora so you train them in what is called a semi supervised fashion.
Ana-Maria Istrate: And there are multiple strategies just to to I want to go over that I think are some of the most common. Ana-Maria Istrate: So you have the mass language model modeling, which is where you're masking one word in a sentence and you're asked them in, then you ask the model to predict it. Ana-Maria Istrate: So we have here an example students open their books when the teacher entered the room, the teacher started assess the lesson someone raised their hand.
Ana-Maria Istrate: So what you're doing here is you're choosing a number of words so let's say you choose the word books and then the word started. Ana-Maria Istrate: And then you're masking these these these words in the model and you're asking the model to learn to predict them, so you don't need any sort of label data for this.
Ana-Maria Istrate: Another strategy is next sentence prediction, which is where you're asking the model to predict if two sentences are following each other in text. Ana-Maria Istrate: So again, looking at our kind of little paragraph of three sentences, we can see that in the first and the first example the modelers to predict that. Ana-Maria Istrate: These sentences are indeed following each other and in the second one it's it's not so um this is how the modelers these representational distributions between words from a large and labeled corpora.
Ana-Maria Istrate: And you know these large models usually have hundreds of millions or billions of parameters and take a very long time to pre train so if you want to use them and you have a particular task you're interested in. Ana-Maria Istrate: um you can imagine that you don't want to train them from scratch and you might also not really have the resources, whether that's computational resources or money.
Ana-Maria Istrate: And you can still get the benefits of using these models by using transfer learning and finding them on particular tasks. Ana-Maria Istrate: So finding means that you're taking the weights of these pre train model and then you use them as initialization for a new model. Ana-Maria Istrate: That learns from label data, and this is usually a supervised task and you can find you in these models on whatever tasks you want, such as name and recognition question answering others that we've talked about in the beginning of the talk.
Ana-Maria Istrate: And the finetuning takes way less time because it has fewer parameters to learn and the strain data sets are usually smaller and this is how you get this been the benefit of using these you know these super large large models on your own particular task. Ana-Maria Istrate: So now I want to move and we want to spend a little bit of time touching on some of these key model architectures that we've seen in recent years and we're going to go in depth into BERT GPT and T5.
Ana-Maria Istrate: Starting with BERT so BERT stands for bidirectional encoder representations for from transformers so the pre training methodology for BERT. Ana-Maria Istrate: Is that it's considering the bidirectional context what that means is that it looks at the sentence both right to left and left to right when training. Ana-Maria Istrate: So um if we consider back the transformer architecture which we have here on kind of like the lower right corner.
Ana-Maria Istrate: And if you remember the this model architecture the transformer has an encoder and a decoder module. Ana-Maria Istrate: So BERT only uses the encoder transformer architecture for pre training, which is why it's able to look at the entire sequence during pre training. Ana-Maria Istrate: So it will when you're looking at the word presentation aware representation you'll have access to both words that came before.
Ana-Maria Istrate: and after it. Ana-Maria Istrate: And it's between on on two tasks on the past mask language modeling task where that we've talked about we're replacing some of the words with the mass token and predicted as the model to predict the next words. Ana-Maria Istrate: or to predict those words and also on the next sentence prediction task.
Ana-Maria Istrate: it's a train on a combination of the books corpus and English Wikipedia. Ana-Maria Istrate: corpora that have a total of over 3 billion words and because you have because you're able to look at the entire sentence left to right and has this what we call bi directional context. Ana-Maria Istrate: it's better used in tasks where you need to understand the full sentence so let's say, maybe you want to classify a sentence or you want to extract.
Ana-Maria Istrate: The different types of entities and you name them to recognition word classification extractive question answering bird models would be the best suited. Ana-Maria Istrate: Next i'll talk about the chat GPT sorry GP the GPT three model, which really started this family of GPT model started with GPT one GPT two and then they kind of evolved from there and GPT one the first GPT model started around the same time as bird.
Ana-Maria Istrate: So the GPT models are generated models, and if we go back to this transformer architecture that has both an encoder and and the decoder if birds use the encoder to pre train GPT uses the decoder to pre train. Ana-Maria Istrate: So they are actually trained to predict the next word in a sequence. Ana-Maria Istrate: And what that means is that attention only has access to passwords so you cannot condition on future words.
Ana-Maria Istrate: And this is called other aggressive and it's the reason why you know it is it is a generated model because it's only what what's actually learning to do is just predict the next word in a sentence. Ana-Maria Istrate: they are trained on huge amounts of corpora that is largely this corpora is obtained by scraping the web such as there's a number of data sets the common crawl web text to like corporate books Wikipedia.
Ana-Maria Istrate: And again, they are best used for text generation because because this is literally what what their training objective has been and it's also why they're doing so well at generating all this content in in chat GPT. Ana-Maria Istrate: So, then, you, you have you know these iterative models so you have GPT three GPT 3.5 GPT 4 and I want to spend just a little bit of time kind of talking about some of the most interesting particularities between them.
Ana-Maria Istrate: So GPT 3.5 GPT 3.5 GPT 3 model that is further fine tuned using something that's called reinforcement learning from human feedback so what really happens here is that you start with this GPT 3 pre trade model. Ana-Maria Istrate: And then you do if you remember we talked about pre training and fine tuning so then what they do is they do the supervised fine tuning where they basically have curators so use human Ai trainers that.
Ana-Maria Istrate: provide conversations in which they play both sides, so this is the this is the training data they use for supervised learning you have curators that are able to you know give a prompt and then a desired answer to that problem. Ana-Maria Istrate: So once you once you have that supervised model you you create a reward model for reinforcement learning so similarly using curators you have your curators and ask them to converse with chat GPT.
Ana-Maria Istrate: And then have them bring the conversations that or have them bring the answers that they get back and then give the model perfect particular or like specific rewards for each of the conversations. Ana-Maria Istrate: And then using those those those rewards models, you can find in the model again using something called proximal policy optimization. Ana-Maria Istrate: And if you're interested, I highly encourage you to go check out the open Ai blog post, they have a lot of a lot of content and this describing that their methodologies in there also you know point pointers to the like their research papers and reports so it's super interesting to look into.
Ana-Maria Istrate: GPT for is going one step further and it's a large multimodal model and what that means is that it's able to take both the text and image as an input and output a text so. Ana-Maria Istrate: We have an example here, where we have a user that gives an image and a question, and you know these are asked why is the image funny.
Ana-Maria Istrate: And the model is not only able to tell you, but also give you a detailed explanation of why it's funny and this is called chain of thought, where the model is able to explain to you how it arrived to a certain conclusion. Ana-Maria Istrate: And that that that that's super interesting in this in itself um one thing is check us GPT for this model is generally better aligned with human intent so making sure that the answers it gives back it's better aligned with what the human intended through the prompt.
Ana-Maria Istrate: And it achieves really high performance on a number of things. Ana-Maria Istrate: Both on professional academic benchmarks again. Ana-Maria Istrate: If you're more interested, I encourage you to take a look one really interesting thing is that you know it passed the simulated bar exam with a score of around top 10% of test takers versus the previous chat or GPT 3.5 scores was was around the bottom 10%.
Ana-Maria Istrate: So the last type of transform based model, I want to mention very briefly is the T five models, so this again going back to the transformer architecture, this is. Ana-Maria Istrate: uses both the encoder and decoder. Ana-Maria Istrate: modules and it's pre trained through what is called stem corruption, this means that we replace random spans of text that contain several words with a single mask special word.
Ana-Maria Istrate: And then the objective is to predict the test that the text that this mass mass word replace. Ana-Maria Istrate: So you know all of these models we've talked about are really huge once then just one minute talking about the sizes where we're looking at we started talking about birth which has around 300 million. Ana-Maria Istrate: The five has around 11 billion parameters.
Ana-Maria Istrate: gpt one is not on this chart but it started at around 175 million parameters so kind of like similar to birth. Ana-Maria Istrate: You know gpt two ended up with 1.5 billion parameters gpt through 175 million parameters and gpt for it's kind of not really known but it's estimated to have at least 1 trillion parameters.
Ana-Maria Istrate: That kind of concludes the technical portion of our talk I know we're also at time. Ana-Maria Istrate: Last thing I'd like to share is a little bit about exciting applications of these large language models outside outside of natural language processing. Ana-Maria Istrate: And one thing i'm particularly excited about because one thing I care about this science are protein language models so here instead of working with natural language we're actually working with protein sequences that are made up of amino acids.
Ana-Maria Istrate: And the parallelism here is that one word corresponds to one or a sequence of amino acid in a protein sequence so if instead of words in a sentence you end up having amino acids in a protein sequence. Ana-Maria Istrate: So then you can apply these models to protein and other type of related data and make make similar kind of contributions and investment to bio biological models.
Ana-Maria Istrate: So um to sum up you know again really exciting times lots of cool models from intelligent conversation agents to multi modality models text to image. Ana-Maria Istrate: We have these applications to biology coding this is really a very, very exciting time to to do natural language processing. Ana-Maria Istrate: We definitely went on a journey together today we we covered a lot we started with an intro to natural language processing.
Ana-Maria Istrate: place NLP in the broader context of Ai looked at various types of NLP tasks we then moved into looking at some foundational NLP blocks. Ana-Maria Istrate: talked about word embeddings RNNs the notion of attention and particular self attention and last but not least we talked about these transformer based models and looked at the transformer architecture talked about birth GPT and T five.
Ana-Maria Istrate: So I thank you very much for being here here's my contact info I do work at CCI science if you're not familiar with us or our work we are tech based philanthropy. Ana-Maria Istrate: we're focused on science and education, please check us out if you're interested in also feel free to reach out to me I am on linkedin.
Ana-Maria Istrate: And would love to stay in touch if you have questions right now i'd be more than happy to take them I don't think we have much time, but if there are lingering questions i'd love to connect on linkedin or you know through my email or anything like that, thank you. Ana-Maria Istrate: Thank you.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why transformers replaced word-by-word language models
Ana-Maria Istrate builds from NLP basics to self-attention and shows how BERT, GPT and T5 each put the transformer to work.
In short
Transformer models won because they read a whole sentence at once. Ana-Maria Istrate, Senior Research Scientist at the Chan Zuckerberg Initiative, shows that this change, made possible by self-attention, let models train faster on parallel hardware and keep better track of how distant words relate. She builds up to that point from first principles, then explains how BERT, GPT and T5 each use the transformer differently.
Key takeaways
Types of NLP tasks
Building blocks for NLP models
The transformer model architecture
Examples of transformer-based NLP models
From headline breakthroughs to plain definitions
Istrate opens with systems many viewers already know: ChatGPT, OpenAI's text-to-image system DALL-E 2, DeepMind's AlphaFold, AlphaCode and Google Search's use of BERT to understand queries. What they share, she says, is a deep learning architecture built on transformers. She then steps back to define the field. Natural language is how people communicate, in speech or in text, and:
"The natural language processing is just the ability of a machine to understand or process the written and spoken human language." (4:46)
She is careful to separate the layers: not every NLP model is a machine learning model, and not every machine learning model is deep learning. Large language models live where deep learning and NLP overlap.
What you can train a language model to do
A tour of tasks follows, each with a worked example: answering questions from a passage about the Super Bowl, picking out dates, people, organisations, places and money in a sentence about an acquisition, extractive versus abstractive summarisation, sentiment analysis on tweets, machine translation, sentence similarity and text generation. Her favourite is entailment:
"entailment is a really interesting task so it's actually one of my favorite tasks it's when you train a model to discern whether one sentence can be inferred from another." (9:48)
Given the premise "an old woman walking down the street", a model has to decide whether other sentences follow from it, contradict it or cannot be judged, which means reasoning over meaning rather than matching words. She closes the section with prompt engineering: describing the task in the prompt so a GPT model can, for example, extract company and people names without a dedicated entity recognition model, and few-shot learning, where a handful of examples guide the model.
Turning words into numbers
Computers handle numbers, not language, so each word is represented as an embedding, a vector of numbers. The values come from the word's surroundings:
"That what that means is that when you're computing a words embedding you're looking at its context or the set of other words that it's surrounded by." (15:45)
Words that share contexts end up close together, so oven, refrigerator and microwave cluster in the embedding space. Recurrent neural networks then modelled sentences by passing information from left to right, giving the model a kind of memory. That was state of the art for a while, but it had a weakness:
"In practice, it is kind of hard to keep track of these long range dependencies between words." (18:25)
Self-attention and the transformer
Self-attention lets each word ask which other words in its own sentence matter for its meaning. In "the animal didn't cross the street because it was too wide", the model computes attention scores between "it" and every other word and learns to attend most to "street". Because this does not depend on word order, the whole input can be fed in at once. The transformer wraps this idea in an encoder and a decoder, six layers each in the original version, with multi-head attention so that different heads learn different relationships between words.
Istrate explains the advantage in two parts. Relating the first and last word in an RNN takes a number of steps that grows with sequence length and cannot be parallelised, while a transformer takes a small constant number of steps that suits modern parallel hardware. And because every word can attend to every other, information is better preserved.
"So to recap the reason why these transformer based models have become so popular is because they have reduced training times compared to RNNs and information doesn't get lost." (24:33)
Pre-training, fine-tuning and three model families
Large models learn first from huge unlabelled collections of web pages, books and Wikipedia:
"And then the really cool thing is that you don't need to have any sort of supervised or labeled data to to learn from this corpora so you train them in what is called a semi supervised fashion." (25:44)
Two common strategies are masked language modelling, where hidden words must be predicted, and next sentence prediction. Because pre-training is slow and costly, most teams fine-tune instead: they start from the pre-trained weights and train on a smaller labelled dataset for their own task.
BERT uses only the encoder, reads context in both directions and suits classification, entity extraction and extractive question answering. GPT uses only the decoder:
"So they are actually trained to predict the next word in a sequence." (30:36)
She walks through how GPT-3.5 was further tuned with reinforcement learning from human feedback, using human AI trainers, a reward model and proximal policy optimisation, and notes, as the speaker cited, that GPT-4 scored around the top 10% on a simulated bar exam where GPT-3.5 scored around the bottom 10%. T5 uses both encoder and decoder and is pre-trained by predicting masked spans of text.
Beyond language: proteins
Istrate ends with the application she cares most about as a scientist, protein language models:
"And the parallelism here is that one word corresponds to one or a sequence of amino acid in a protein sequence so if instead of words in a sentence you end up having amino acids in a protein sequence." (36:14)
The same ideas that read sentences can read biology, which is why she calls this a very exciting time to work in NLP.
Watch the talk
"NLP 101: Natural Language Processing and The Era of Transformer Models" by Ana-Maria Istrate, 38 minutes: https://www.womentech.net/video/nlp-101-natural-language-processing-and-era-transformer-models
About the speaker
Ana-Maria Istrate is a Senior Research Scientist at the Chan Zuckerberg Initiative, where she supports program teams in the Science Initiative and works on text-mining algorithms that extract entities from scientific journal articles. She previously worked in software engineering roles at Amazon, holds a BS in Mathematical and Computational Science and an MS in Computer Science (Artificial Intelligence track) from Stanford, and co-leads the Women In Tech group at CZI.
Questions this talk answers
What is natural language processing?
How does a computer represent the meaning of a word?
Why did transformers replace recurrent neural networks?
How do BERT, GPT and T5 differ?
How can you use a huge language model without training it from scratch?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Transformer architecture
Self-attention
Multi-head attention
Recurrent neural networks (RNNs)
Word embeddings
Masked language modeling
Next sentence prediction
Transfer learning and fine-tuning
Reinforcement learning from human feedback (RLHF)
Proximal policy optimization
Span corruption
Few-shot learning
Chain of thought
Resources mentioned
OpenAI blog posts and research reports on ChatGPT methodology
Cite this talk
Ana-Maria Istrate. "NLP 101: Natural Language Processing and The Era of Transformer Models." WomenTech Network, 2023. https://www.womentech.net/video/nlp-101-natural-language-processing-and-era-transformer-models