0:0045:48

Transfer Learning With BERT: Building a Text Classification Model

Jayeeta Putatunda 45:48 2020

Most text classifiers don't need to be trained from scratch

An algorithm told "I am a huge metal fan" cannot tell whether you love metal bands or own a metal fan, and teaching it the difference takes enormous amounts of data. Data scientist Jayeeta Putatunda, who specializes in natural language processing, shows why building every text model from zero is costly when labelled data is scarce. So how do you borrow a model pre-trained on billions of words and make it classify your own text?

Follow the steps from cleaning raw text to fine-tuning BERT for news classification in Google Colab

Watch on YouTube

In short

Data scientist Jayeeta Putatunda introduces natural language processing for newcomers, from core text preprocessing to word embeddings, Transformers and BERT. She argues that transfer learning lets teams reuse models pre-trained on huge datasets instead of building text classifiers from scratch, and demonstrates fine-tuning a pre-trained BERT model to classify news text in Google Colab.

“So a lot of the time people feel that NLP is hard. And to start with, it definitely is hard.”
“A lot of people say NLP is magic, but it's not.”

What you will learn

  • Clean raw text with core preprocessing steps: stripping extra spaces, tokenization, spell checking, contraction mapping, stemming and lemmatization, and stop word removal.
  • Explain how transfer learning reuses a model trained on a large general dataset and fine-tunes its final layer for a smaller, related task.
  • Judge when transfer learning fits: labelled data is scarce or expensive, and the source and target tasks share the same input properties.
  • Describe how BERT's bidirectional training and its token, position and segment embeddings give the same word different meanings in different contexts.
  • Fine-tune a pre-trained BERT model for news classification in Google Colab by refitting its final dense layer with a softmax activation.

Details

Level
Intermediate
Format
Tech talk
Technical depth
Practitioner
For
Student, Early career, Mid-career
Length
45:48
Published
July 2020

Questions this talk answers

Why is natural language processing hard?

Language is full of ambiguity, so a phrase like "I am a huge metal fan" has two readings. Humans separate the contexts easily, but algorithms need large amounts of data to learn those nuances.

What is transfer learning?

It is a technique where you train a model on a task with plenty of data, then repurpose it for a related task. Typically you keep the learned low-level features and fine-tune the last dense layer on your smaller dataset.

When should you use transfer learning?

Use it when labelled data is scarce, because labelling needs time and subject matter experts. The target task must also share input properties with the source task so knowledge can transfer.

How is BERT different from earlier embeddings like Word2Vec?

Word2Vec read context in one direction, while BERT trains both left to right and right to left. That lets a word like "bank" get different representations next to "river" or "deposit".

Which tasks is BERT pre-trained on?

BERT is pre-trained on masked language modeling, where it predicts hidden words, and next sentence prediction, where it judges whether one sentence logically follows another.

About this talk

The domain of Natural Language Processing have seen a tremendous amount of research and innovation in the past couple of years to tackle the problem of implementing high quality machine learning and AI solutions using natural text. Text Classification is one such area that is extremely important in all sectors like finance, media, product development, etc. Building up a text classification system from scratch for every use case can be challenging in terms of cost as well as resources, considering there is a good amount of dataset to begin training with.

Here comes the concept of transfer learning. Using some of the models that has been pre-trained on terabytes of data and fine-tuning it based on the problem at hand is the new way to efficiently implement machine learning solutions without spending months on data cleaning pipeline.

This talk with highlight ways of implementing the newly launched BERT and fine tuning the base model to build an efficient text classifying model. Basic understanding of python is desirable. Code can be made available via Github for everyone to examine after the talk.

Topics

Related talks

Browse the library