0:0038:10

NLP 101: Natural Language Processing and The Era of Transformer Models

Ana-Maria Istrate 38:10 2023

Why reading every word at once made transformers beat recurrent networks

AlphaFold cracked a protein-folding problem that biology had not solved in over 50 years, and ChatGPT can plan a trip to Hawaii, yet both rest on the same architecture. Ana-Maria Istrate, a Senior Research Scientist at the Chan Zuckerberg Initiative who works with these models daily, starts from plain definitions for newcomers. The question she builds toward: what changed when models stopped reading text one word at a time?

Learn how embeddings, self-attention and pre-training combine in BERT, GPT and T5

Watch on YouTube

In short

Ana-Maria Istrate, a Senior Research Scientist at the Chan Zuckerberg Initiative, gives a beginner-friendly tour of natural language processing, from common tasks such as question answering and entailment to word embeddings, recurrent neural networks and self-attention. She explains why transformer models overtook recurrent networks and how BERT, GPT and T5 are pre-trained, fine-tuned and best used.

“So this is a model developed by DeepMind and it's really impressive because this is actually a problem that has not been solved in biology for over 50 years.”
“So to recap the reason why these transformer based models have become so popular is because they have reduced training times compared to RNNs and information doesn't get lost.”

What you will learn

  • Distinguish the main NLP tasks, from named entity recognition and summarization to entailment and text generation.
  • Explain how word embeddings capture a word's meaning from the contexts it appears in.
  • Compare why self-attention outperforms recurrent neural networks on training speed and long-range accuracy.
  • Choose between BERT, GPT and T5 depending on whether a task needs understanding or generation.
  • Use pre-trained models through fine-tuning, prompting and few-shot examples instead of training from scratch.

Key takeaways from the speaker

  • Types of NLP tasks
  • Building Blocks for NLP Models
  • The Transformer Model Architecture
  • Examples of transformer-based NLP models

Details

Level
Beginner
Format
Tech talk
Technical depth
Introductory technical
For
Student, Early career, Career changer
Length
38:10
Published
June 2023

Questions this talk answers

What is natural language processing?

It is a branch of AI covering a machine's ability to understand or process written and spoken human language. Not every NLP model is a machine learning or deep learning model.

How does a computer represent the meaning of a word?

As a word embedding, a vector of numbers built from the contexts in which the word appears. Words used in similar contexts end up close together in the embedding space.

Why did transformers replace recurrent neural networks?

Transformers read the whole input at once, so they parallelise well on modern hardware and train faster. Self-attention also preserves relationships between distant words that RNNs tend to lose.

How do BERT, GPT and T5 differ?

BERT uses the transformer encoder with bidirectional context and suits classification and entity extraction, while GPT uses the decoder to predict the next word and suits text generation. T5 uses both encoder and decoder and is pre-trained with span corruption.

How can you use a huge language model without training it from scratch?

Fine-tune it: initialise a new model with the pre-trained weights and train it on a smaller labelled dataset for your task. You can also describe the task in a prompt, optionally with a few examples.

About this talk

Machine Learning, AI and NLP have become part of the daily tech vocabulary. Companies use ML techniques to acquire users, drive revenue and increase profits. It seems that ML is everywhere. But what are the types of things you can do with machine learning, in particularly natural language processing? What exactly is NLP? How does it work? What are transformer models? How does the chatGPT (Chat Generative Pre-trained Transformer) model work?

If you’re new to the NLP space, and would like to understand more about what is going on under the hood, or the types of things you can use NLP for, come to this talk! We’ll start broad, going over NLP use cases, followed by understanding how some NLP foundational blocks, such as transformers, work. We'll end with looking at a few specific NLP models (including chatGPT!) in more depth.

Topics

Related talks

Browse the library