0:0017:28

Using transformer models for your own NLP task - building an NLP model End To End

Ana-Maria Istrate 17:28 2022

A 99% F1 score can still hide a model whose output makes no sense

A model can post a 99% F1 score and still produce output that makes no sense to a human reader, and in a niche field like biomedical research that gap decides whether the work is usable. Ana-Maria Istrate, a Research Scientist at the Chan Zuckerberg Initiative, built a transformer model that pulls datasets and experimental methods out of full-text papers. What does getting it right take, from training data to evaluation?

Learn how to define entities, source training data, fine-tune domain models and check real output

Watch on YouTube

In short

Ana-Maria Istrate, Research Scientist at the Chan Zuckerberg Initiative, explains how transformer models work and how to apply them to your own NLP task end to end. Using her team's named entity recognition model for finding datasets and experimental methods in biomedical papers, she covers defining entities, sourcing or curating training data, fine-tuning domain-specific models and pairing metrics with human evaluation.

“One very important thing that I do want to advocate for is human evaluation. So, just sanity checking the output.”
“So the more niche or specialized your domain or task is, the more likely you are to have to create your data set from scratch.”

What you will learn

  • Explain how self-attention lets transformer models process input all at once, unlike recurrent neural networks.
  • Write clear entity definitions with domain experts before building a training dataset.
  • Choose between an openly available dataset and curating your own, based on how specialized the task is.
  • Fine-tune a pre-trained or domain-specific model such as BioBERT by adding a linear classification layer.
  • Pair quantitative metrics like precision, recall and F1 with human review of model output.

Key takeaways from the speaker

  • Building or using an available training corpus for an NLP task
  • Develop a NLP model for a specific task using state of the art available technology
  • Evaluating an NLP model

Details

Level
Intermediate
Format
Tech talk
Technical depth
Practitioner
For
Early career, Mid-career, Senior individual contributor
Length
17:28
Published
July 2022

Questions this talk answers

Why did transformer models overtake recurrent neural networks in NLP?

RNNs process input one word at a time, while transformers process it all at once. That allows more parallelization and shorter training times, which led to many large pre-trained models such as BERT and GPT.

Do I need to train a transformer model from scratch for my own task?

No. Through transfer learning you take the weights of a pre-trained model as initialization and fine-tune it, for example by adding a linear classification layer on top that gets trained for your task.

Should I use an open dataset or build my own training data?

The more general the task, the more likely an openly available dataset exists, for example through Hugging Face. The more niche the domain or task, the more likely you need to curate one, which can be time consuming, expensive and need domain knowledge.

What should I do when my domain uses specialized vocabulary?

Use models pre-trained on your specialized domain, such as BioBERT, SciBERT or PubMedBERT for biomedicine, or FinBERT for finance. They share BERT's architecture but are trained on domain corpora and tend to perform better on specialized tasks.

Are high evaluation metrics enough to trust an NLP model?

No. A model can show very high accuracy or F1 and still produce output that does not make sense, so Istrate advocates sanity checking output by hand and getting feedback from domain experts where possible.

About this talk

Transformer models have revolutionized the NLP field and are currently state-of-the art on a variety of tasks, such as named entity recognition, language inference or question answering. With new, more performant models being continuously developed (BERT, RoBERTa, AlBERT, ELECTRA, ERNIE, etc), these models are ubiquitous in virtually all domains that make use of natural language processing. 

So how can you apply these models on your own task? In this talk, we will go over the process of using state-of-the-art transformer models for your own NLP task. We will discuss the entire pipeline, from building a training corpus, developing a NLP model and evaluating the model. We will offer an example of building a model to extract mentions of Experimental Methods and Datasets from full-text biomedical papers. Even though our example will focus on an NLP task for the biomedical text, the framework can be applied to any domain.

Topics

Related talks

Browse the library