0:0017:21

Using transformer models for your own NLP task - building an NLP model End To End

Ana-Maria Istrate 17:21 2022

A 99% F1 score can still hide a model whose output makes no sense

A model can post 95% accuracy or a 99% F1 score and still produce output that makes no sense to a human reader. Ana-Maria Istrate, a research scientist at the Chan Zuckerberg Initiative who builds models that mine biomedical papers, shows what that gap means for anyone fine-tuning BERT-style models on specialized text. So where do those hidden failures begin, long before the metrics come in?

Follow a full NER build from entity definitions to fine-tuning and human evaluation

Watch on YouTube

In short

Ana-Maria Istrate, Research Scientist at the Chan Zuckerberg Initiative, walks through building a transformer-based NLP model end to end, using a named entity recognition model that extracts datasets and experimental methods from biomedical papers. She covers self-attention and transfer learning, why clear entity definitions shape the training set, when to use open versus curated data, how domain-specific models like BioBERT help, and why human evaluation must accompany metrics.

“For us, it was very important to know what exactly do we mean by data sets and experimental methods.”
“So the more niche or specialized your domain or task is, the more likely you are to have to create your data set from scratch.”

What you will learn

  • Explain how self-attention lets transformers represent a word in context and why they replaced RNNs as the NLP state of the art.
  • Write clear entity definitions with domain experts before building or labeling a training set.
  • Choose between an openly available dataset, curating one from scratch, or combining both.
  • Fine-tune a pre-trained model with a classification layer and pick domain-specific models such as BioBERT for specialized text.
  • Combine quantitative metrics with human and expert review of model output to catch hidden failures.

Key takeaways from the speaker

  • Building or using an available training corpus for an NLP task
  • Develop a NLP model for a specific task using state of the art available technology
  • Evaluating an NLP model

Details

Level
Intermediate
Format
Tech talk
Technical depth
Practitioner
For
Early career, Mid-career, Senior individual contributor
Length
17:21
Published
July 2022

Questions this talk answers

Why are transformer models faster to train than recurrent neural networks?

RNNs read input one word at a time, while transformers process the whole input at once. That allows more parallelization and reduces training times.

Should I use an openly available dataset or build my own training data?

The more general the task, the more likely an open dataset exists, and the more niche it is, the more likely you must curate from scratch. Her team combined both, further curating openly available Europe PMC annotations to fit their own definitions.

What does fine-tuning a transformer look like in practice?

You initialize the architecture from a pre-trained model such as BERT with frozen weights and add a linear classification layer on top, which is what gets trained. In her example that layer classifies each token as part of a method or a dataset.

Why use a domain-specific model like BioBERT instead of BERT?

Specialized domains have different word distributions and terms that general-domain models may not capture. Models with the same architecture but pre-trained on domain text, such as BioBERT, SciBERT or PubMedBERT, tend to perform better on specialized tasks.

Are high evaluation metrics enough to trust an NLP model?

No. She advocates sanity checking the output by hand, because a model with very high metrics can still produce output that does not make sense, and domain experts can show where the model fails.

About this talk

Transformer models have revolutionized the NLP field and are currently state-of-the art on a variety of tasks, such as named entity recognition, language inference or question answering. With new, more performant models being continuously developed (BERT, RoBERTa, AlBERT, ELECTRA, ERNIE, etc), these models are ubiquitous in virtually all domains that make use of natural language processing. 

So how can you apply these models on your own task? In this talk, we will go over the process of using state-of-the-art transformer models for your own NLP task. We will discuss the entire pipeline, from building a training corpus, developing a NLP model and evaluating the model. We will offer an example of building a model to extract mentions of Experimental Methods and Datasets from full-text biomedical papers. Even though our example will focus on an NLP task for the biomedical text, the framework can be applied to any domain.

Topics

Related talks

Browse the library