Using transformer models for your own NLP task - building an NLP model End To End
Ana-Maria Istrate17:212022
A 99% F1 score can still hide a model whose output makes no sense
A model can post 95% accuracy or a 99% F1 score and still produce output that makes no sense to a human reader. Ana-Maria Istrate, a research scientist at the Chan Zuckerberg Initiative who builds models that mine biomedical papers, shows what that gap means for anyone fine-tuning BERT-style models on specialized text. So where do those hidden failures begin, long before the metrics come in?
Follow a full NER build from entity definitions to fine-tuning and human evaluation
Ana-Maria Istrate, Research Scientist at the Chan Zuckerberg Initiative, walks through building a transformer-based NLP model end to end, using a named entity recognition model that extracts datasets and experimental methods from biomedical papers. She covers self-attention and transfer learning, why clear entity definitions shape the training set, when to use open versus curated data, how domain-specific models like BioBERT help, and why human evaluation must accompany metrics.
“For us, it was very important to know what exactly do we mean by data sets and experimental methods.”
“So the more niche or specialized your domain or task is, the more likely you are to have to create your data set from scratch.”
What you will learn
Explain how self-attention lets transformers represent a word in context and why they replaced RNNs as the NLP state of the art.
Write clear entity definitions with domain experts before building or labeling a training set.
Choose between an openly available dataset, curating one from scratch, or combining both.
Fine-tune a pre-trained model with a classification layer and pick domain-specific models such as BioBERT for specialized text.
Combine quantitative metrics with human and expert review of model output to catch hidden failures.
Key takeaways from the speaker
Building or using an available training corpus for an NLP task
Develop a NLP model for a specific task using state of the art available technology
Evaluating an NLP model
Details
Level
Intermediate
Format
Tech talk
Technical depth
Practitioner
For
Early career, Mid-career, Senior individual contributor
Length
17:21
Published
July 2022
Questions this talk answers
Why are transformer models faster to train than recurrent neural networks?
RNNs read input one word at a time, while transformers process the whole input at once. That allows more parallelization and reduces training times.
Should I use an openly available dataset or build my own training data?
The more general the task, the more likely an open dataset exists, and the more niche it is, the more likely you must curate from scratch. Her team combined both, further curating openly available Europe PMC annotations to fit their own definitions.
What does fine-tuning a transformer look like in practice?
You initialize the architecture from a pre-trained model such as BERT with frozen weights and add a linear classification layer on top, which is what gets trained. In her example that layer classifies each token as part of a method or a dataset.
Why use a domain-specific model like BioBERT instead of BERT?
Specialized domains have different word distributions and terms that general-domain models may not capture. Models with the same architecture but pre-trained on domain text, such as BioBERT, SciBERT or PubMedBERT, tend to perform better on specialized tasks.
Are high evaluation metrics enough to trust an NLP model?
No. She advocates sanity checking the output by hand, because a model with very high metrics can still produce output that does not make sense, and domain experts can show where the model fails.
About this talk
Transformer models have revolutionized the NLP field and are currently state-of-the art on a variety of tasks, such as named entity recognition, language inference or question answering. With new, more performant models being continuously developed (BERT, RoBERTa, AlBERT, ELECTRA, ERNIE, etc), these models are ubiquitous in virtually all domains that make use of natural language processing.
So how can you apply these models on your own task? In this talk, we will go over the process of using state-of-the-art transformer models for your own NLP task. We will discuss the entire pipeline, from building a training corpus, developing a NLP model and evaluating the model. We will offer an example of building a model to extract mentions of Experimental Methods and Datasets from full-text biomedical papers. Even though our example will focus on an NLP task for the biomedical text, the framework can be applied to any domain.
All right. Hi, everyone, and thank you so much for being here. I'm really excited to talk to you today about Transformer Models for Natural Language Processing and building an NLP model end-to-end. First of all, hi. I am really happy to be here today. My name is Ana-Maria Istrate, or Ana. I am a Senior Research Scientist. at Chan Zuckerberg Initiative. And my interests lie at the intersection of NLP and applications of NLP on the scientific domain, in particular, text mining, biomedical literature, predicting impact of research outputs, knowledge graphs, and graph-based models.
This is the outline of what we will be covering today. So I would like to share with you a little bit of a background, some background information about Transformer Models, in particular, how they relate to Natural Language Processing tasks. And then we will go into the process of building a Transformer-based NLP model end-to-end. We will be talking about the training data set and the importance of having good definitions of what to include in the training data set. We will touch on the options of choosing an openly available data set versus curating one from scratch. We will then move into model development and talk about the transfer learning, the idea of fine-tuning model versus pre-training from scratch. And some of the challenges that come with specialized corpora. We will end with model evaluation and talk about quantitative and qualitative evaluation. For the second part of our talk, we will use an example of building a named entity recognition model to mine different types of entities from biomedical text. But if you're not interested in the biomedical domain or you feel you don't have domain knowledge for the biomedical
domain knowledge for the biomedical field, that's totally fine. That's not very relevant to our talk today and you'll still be able to follow along. Awesome. So with that in mind, I am excited to get started and talk about Transformer models in natural language processing. So Transformer models have become really popular mainly in natural language processing and computer vision. And Transformer models are essentially deep learning models that are based on this idea of self-attention. And self-attention is a concept that basically basically wants to, or helps us understand the representation of a current word given its context in the input sequence.
So for instance, we have this example of a sentence on the slide. The animal didn't cross the street because it was too wide. If we want to compute a representation for the word it, as humans, we might know that the word it refers to the street and not to the animal. However, we need to install this sort of information in machine learning models as well. And this is exactly what self-attention aims to do. Basically, when computing a representation for a particular word, we want to know what other words in the sequence we should attend to. What other words are relevant to our current representation?
Okay. And the notion of self-attention is at the core of the Transformer architecture. And the Transformer architecture has firstly been introduced in 2017. Once introduced, Transformer-based natural language processing models pretty much became state-of-the-art in natural language processing. And before that, the state-of-the-art in NLP tasks were recurrent neural networks or RNNs. The difference between RNNs and transformer models is that in RNNs, the input gets processed sequentially. So one word at a time. However, in transformer architectures, we are able to process the input all at once.
This means that transformers allow for more parallelization than recurrent neural networks. And it also means that we get reduced training times. So as an example of how this happens, for instance, we have this diagram of a recurrent neural network at the bottom. When we're feeding input in a recurrent neural network, we're basically reading the input one word at a time.
Because transformer models don't need to follow this sort of like pattern, it means that we get reduced training times. And this led to the development of a really large number of models of pre-trained language models based on the transformer architecture. So maybe you've heard of things such as BERT, which stands for bi-directional encoder presentations from transformers, GPT, which stands for generative pre-trained transformers, T5, Roberta, Electra, Albert, Ernie, and the list can go on and on and on.
There are a lot of really high-performing models based on the transformer architecture. And the reason why, you know, the reason why these models perform really well on a variety of tasks such as named entity recognition, question answering, summarization, or machine learning translation is because these models are being pre-trained on huge amounts of text. So they learn to capture complexities in natural language from the training corpus really well.
However, the really cool thing is that you don't have, if you want to use one of these models, you don't have to train them from scratch every time you want to, you want to do that. You can use them by fine tuning them on your own particular task. And this is behind this concept of transfer learning. So fine tuning means that you're basically taking the weight of a model that has been pre-trained, already pre-trained.
And you can use these weights as initialization for a new model. For instance, we have the BERT model, which has been pre-trained on the books corpus and English Wikipedia data sets. And we can take this model architecture and finding it on our own particular tasks, whether this is named entity recognition, question answering, or sentence classification. All right, so hopefully that gave us a little bit of an, of an understanding of transformer models and how they are being used in natural language processing.
The next thing I want to go over is building a transformer based model, a transformer based NLP model end to end. As mentioned, we will be talking about the training data set, model development and model evaluation. And for this part of the talk, we will focus on an example of building a named entity recognition model that mines different types of entities from biomedical text.
And we build this model at Chan Zuckerberg initiative. Our task with this model was to extract mentions of data sets and experimental methods from biomedical research articles. So essentially, if we have a piece of text that happens to be coming from a biomedical research article, and we have an example of this paragraph on this slide. Our task here is to be able to extract data sets and experimental methods from this piece of text.
Data sets can look like identifiers in a database or repository. So basically these GSE mentions that we see here on the slide. And experimental methods can look at this method that's being found in this paragraph, ChIP-seq, which is the sequencing method. How we end up extracting these mentions of data sets and experimental methods from biomedical research articles is through a named entity recognition model that basically learns to recognize these entity types from text.
And if you're not familiar with the named entity recognition task, named entity recognition aims to locate and classify named entities in unstructured text into predefined categories. So in our case, our predefined categories are data sets and experimental methods. And we want to be able to locate and classify all of the named entities in this piece of text that fall into either one of these two categories.
So the first thing I wanted to talk about in this process is building the training data set. And the first step we took in building our training data set is we asked ourselves what gets included. And I do want to emphasize the importance of having entity definitions. For us, it was very important to know what exactly do we mean by data sets and experimental methods.
What exactly gets included into each of these predefined categories. And definitions are particularly important, especially for specialized domains or tasks or boundaries are not very clear. So for us, you know, we want to be able to identify experimental methods, but it wasn't very clear if we should include things like software, equipment, regions, or other types of entities.
And luckily we do have a team of biomedical experts at CZI that we can collaborate with and that can give us domain expertise. So we collaborated with them to come up with these definitions. Once we have, once we have good definitions for what we want to include in our data set, the actual, the next step is actually creating the data set.
And this, and here we usually have two main options. One, we can use an openly available data set. And there are lots of data sets that are openly available. There are even, there are more than 5,000 data sets available only through the Hugging Face API. And these are easy to load and use with just a few lines of code.
And the more general your task is, the more likely it is that you will be able to find an openly available data set for your particular task. The other option is to create a data set from scratch. So the more niche or specialized your domain or task is, the more likely you are to have to create your data set from scratch.
In our case, we not only dealt with a specialized domain like the biomedical field, but we also have a very specialized task of being able to identify particular mentions of data sets and experimental methods. Creating a data set from scratch comes with its own set of challenges. And it usually involves some type of curation. You know, maybe you have product or user data and you want to be able to build a model that learns from this.
Maybe you have things like user logs or user interaction data, which is likely to be very noisy. And it is very likely you'll have to clean up this data in order to be able to build a high quality training data set for a machine learning model. Or maybe you don't have product or user data and you just want to build your data set from scratch.
In our case, we do have this by a creation team that I mentioned that we can use in order to create these data sets from scratch if we want to. If you don't have access to that type of resource, there are other types of resources such as Mechanical Turk or, you know, that uses crowd sourcing. And there are also curation companies who specialize in building high quality training data sets.
And some of them have also specialized expertise. We've been using some of them for the biomedical field as well. Creating a data set from scratch, again, comes with its own set of challenges because it is time consuming, could be expensive, and it might involve specialized domain knowledge. In our case, we use the combination of both where we use openly available annotations.
There is a platform for the biomedical field. It's called Europe PMC. This platform provides openly available annotations from biomedical research articles. However, their annotations did not exactly fit our own definitions. So, we took this openly available data and we further curated it with the help of our bio creation team. For us, that meant excluding terms that did not fit our own definitions.
Cool. So, we've talked about the training data set. We've talked about the importance of having good definitions, about the options of using an openly available data set. Versus curating one from scratch. The next part of our talk today will focus on model development. So, going back to this notion of transformer models and transfer learning, where you can use one of these pre-trained language models on your own particular task by fine-tuning them without having to train from scratch.
We define fine-tuning again as taking the weights of a trained model and using them as initialization for a new model. How does this look like in practice? So, in practice, we took a pre-trained model such as BERT. This is kind of like the blue module that you see on this diagram. We initialize the model architecture from a pre-trained language model with the frozen weights from this particular model.
And then we add a linear classification layer on top. And this is essentially the fine-tuning part. This is what gets trained. In our case, the linear classification layer was in charge of basically classifying each of the tokens as being part of either a method or a data set. I do want to talk a little bit about specialized domains and mention the fact that specialized domains have different word distributions than general domains.
And specialized domains, that's because specialized domains usually contain specialized terms. So, for the biomedical field, these could be genes, cell lines, or diseases. And there are other specialized terms such as finance or law that have the same types of challenges. The idea here is that representations learned by models from general domain text might not be able to capture all of these complexities for specialized domains.
The solution then is to use models that are pre-trained on your own specialized domain. Because these tend to give better performance on your specialized task. So, for the biomedical field, we have models such as BioBERT, SciBERT, and PubMedBERT. And these are essentially models that have the same model architecture as BERT. The only difference between them and BERT is the corpora they have been trained on.
So, some of these models have been trained on biomedical research articles or scientific articles. And there are similar models for other types of domains such as FinBERT for the financial field, LawBERT for law, and I'm sure there's other for other types of domains. Cool. So, we went over the training dataset, talk about definitions using an openly available dataset versus curating from scratch.
We touched on model development and the idea of transfer learning, fine-tuning versus pre-training a model. And some of the challenges of specialized corpora. The last thing I want to talk about is model evaluation. And talk about quantitative versus qualitative evaluation. So, the first thing that you do when evaluating a machine learning model is you'd look at quantitative measures such as precision, recall, F1 score, accuracy, blue score.
Whatever your task is, there's usually some widely accepted metric, quantitative metric, that you can use to evaluate your machine learning model. One very important thing that I do want to advocate for is human evaluation. So, just sanity checking the output. More often than not, you can get really high metrics such as, you know, 95% accuracy, 99% F1 score.
And when actually looking at the output of the model, it doesn't make sense. Or maybe, you know, it's not exactly what we would expect. So, without sanity checking the output, we would not be able to, like, catch that and go back and fix it. And you don't really have the main expertise for that. You don't really need to have the main expertise for that.
However, sometimes the main expertise, it is helpful to have. So, for us, again, we have this team of biomedical curators that we are able to consult at different stages in model development. So, more often than not, we take output of some of our trained machine learning models and we put it in front of them. And they are able to tell us or, like, to give us feedback on where is the model failing?
What is the model not picking up? Where could the model be improved? This feedback is very valuable for us because we can then go back and incorporate it into the model development. So, this is all I wanted to share with you today as a recap of what we went over. We talked about transformer models for natural language processing.
We then talked about the process of building a transformer-based NLP model end-to-end. We talked about the training dataset and the importance of having good definitions and what to include and what not to include. We talked about using an openly available dataset or curating one from scratch. And then we touched on model development and talked about transfer learning, the idea of fine-tuning versus free training on a model, and some of the challenges of specialized corpora.
We ended by looking at model evaluation and talked about quantitative versus qualitative evaluation. Thank you so much for listening. If you want to check us out, please check out the Chan Zuckerberg Initiative or CCI handles and also check out the Science Initiative within CCI. Thank you so much for being here. I hope you have a great rest of the conference and great rest of the day.
Thank you.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
How do you adapt a transformer model to a specialized NLP task?
Ana-Maria Istrate shows how the Chan Zuckerberg Initiative built a biomedical entity recognition model, from definitions to human evaluation.
In short
You rarely need to train a transformer from scratch: you fine-tune a pre-trained model on your own task. Ana-Maria Istrate, Research Scientist at the Chan Zuckerberg Initiative, argues that success depends just as much on the work around the model: clear entity definitions, a training set that fits them, a domain-appropriate base model and human review of the output. She illustrates the full pipeline with a named entity recognition (NER) model that extracts datasets and experimental methods from biomedical research articles.
Key takeaways
Building or using an available training corpus for an NLP task
Developing an NLP model for a specific task using state-of-the-art available technology
Evaluating an NLP model
Why transformers took over NLP
Istrate begins with self-attention, the idea at the core of the transformer architecture. Using the sentence "The animal didn't cross the street because it was too wide", she explains that humans know "it" refers to the street, and self-attention is how a model learns which other words in a sequence matter when representing a given word.
She notes that the architecture was introduced in 2017 and quickly displaced recurrent neural networks, which process input one word at a time. Transformers take in the whole input at once: "This means that transformers allow for more parallelization than recurrent neural networks. And it also means that we get reduced training times." (3:59)
That efficiency led to a wave of pre-trained language models, including BERT, GPT, T5, RoBERTa, ELECTRA, ALBERT and ERNIE. Because they are pre-trained on huge amounts of text, they capture the complexity of language well. The practical point is transfer learning: you take the weights of an already pre-trained model, such as BERT trained on the Books corpus and English Wikipedia, and use them as initialization for a model fine-tuned on your own task.
Start with definitions, not data
The worked example is a model the Chan Zuckerberg Initiative built to find mentions of datasets (often database identifiers like GSE accession numbers) and experimental methods (such as ChIP-seq) in biomedical papers. Istrate's first step was not collecting data but deciding what counts.
"For us, it was very important to know what exactly do we mean by data sets and experimental methods." (8:10)
In specialized domains the boundaries are fuzzy. Her team had to decide whether experimental methods should include software, equipment, reagents or other entity types. They settled those questions with CZI's in-house biomedical experts, and only then moved on to building the dataset.
Open datasets, curation from scratch, or both
Istrate describes two main options. Openly available datasets are plentiful (she cites more than 5,000 available through the Hugging Face API alone) and load with a few lines of code. But the fit depends on how general your problem is: "So the more niche or specialized your domain or task is, the more likely you are to have to create your data set from scratch." (9:39)
Creating data from scratch means curation. Product or user data such as logs is likely to be noisy and need cleaning. Teams without in-house experts can turn to crowdsourcing through Mechanical Turk or to curation companies, some with specialized expertise. Either way, it is time-consuming, can be expensive and may require domain knowledge.
CZI used a hybrid. Europe PMC provides openly available annotations from biomedical research articles, but those annotations did not match the team's definitions, so the biocuration team further curated them, excluding terms that did not fit.
Fine-tuning, and why domain matters
In practice, the team initialized the architecture from a pre-trained model such as BERT with frozen weights and added a linear classification layer on top. That layer is what gets trained; in this case it classifies each token as part of a method or a dataset.
Istrate then turns to a common trap. Specialized fields such as biomedicine, finance or law have different word distributions, full of terms like genes, cell lines or diseases, and representations learned from general text may miss them. "The solution then is to use models that are pre-trained on your own specialized domain. Because these tend to give better performance on your specialized task." (13:46)
BioBERT, SciBERT and PubMedBERT share BERT's architecture and differ only in the corpora they were trained on. She points to similar models for other fields, such as FinBERT for finance.
Metrics are not the whole story
Evaluation begins with the standard quantitative measures for the task: precision, recall, F1 score, accuracy or BLEU score. But Istrate makes a direct case for more: "One very important thing that I do want to advocate for is human evaluation. So, just sanity checking the output." (15:03)
She warns that a model can report 95% accuracy or a 99% F1 score and still produce output that does not make sense or is not what you expected. "So, without sanity checking the output, we would not be able to, like, catch that and go back and fix it." (15:27) Basic checks need no domain expertise, though experts help. At CZI, biomedical curators regularly review model output and say where the model fails, what it misses and where it could improve, and that feedback goes straight back into model development.
Watch the talk
"Using transformer models for your own NLP task - building an NLP model End To End" by Ana-Maria Istrate (17 minutes): https://www.womentech.net/video/using-transformer-models-your-own-nlp-task-building-nlp-model-end-end-ana-maria-istrate-0
About the speaker
Ana-Maria Istrate is a Research Scientist at the Chan Zuckerberg Initiative Foundation, where she has worked mainly on recommendations, ranking and text mining algorithms that retrieve knowledge from biomedical journal articles. She holds a Bachelor's degree in Applied Math and a Master's degree in Computer Science (Artificial Intelligence track) from Stanford, and her interests lie at the intersection of NLP, text mining and applications that serve the scientific community.
Questions this talk answers
Why are transformer models faster to train than recurrent neural networks?
Should I use an openly available dataset or build my own training data?
What does fine-tuning a transformer look like in practice?
Why use a domain-specific model like BioBERT instead of BERT?
Are high evaluation metrics enough to trust an NLP model?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Transformer architecture
Self-attention
Transfer learning
Fine-tuning versus pre-training
Named entity recognition
Resources mentioned
Hugging Face datasets (via the Hugging Face API)
Europe PMC
Mechanical Turk
BioBERT
SciBERT
PubMedBERT
FinBERT
Cite this talk
Ana-Maria Istrate. "Using transformer models for your own NLP task - building an NLP model End To End." WomenTech Network, 2022. https://www.womentech.net/video/using-transformer-models-your-own-nlp-task-building-nlp-model-end-end-ana-maria-istrate-0