Attention? Attention! - Attention Mechanism In Deep Learning
Nicole Koenigstein35:382022
RNNs falter past 20 time steps; attention learns what to focus on
Sequence-to-sequence models built on recurrent networks work well only for short inputs, under about 20 time steps, before the bottleneck makes them forget. Data science and technology lead Nicole Koenigstein, who has published her own attention-based research on stock price prediction, walks through the math that replaced them. For anyone using transformers without seeing inside them, the question is how a simple weighted sum lets a model decide what deserves its focus.
Follow the self-attention math step by step, then see it in vision, anomaly and stock models
Nicole Koenigstein explains why recurrent sequence-to-sequence models struggle with long inputs and how the attention mechanism overcomes that bottleneck. She walks through soft, hard, global, local and self-attention, the math of queries, keys and values, scaled dot-product and multi-head attention, then shows applications in translation, vision transformers, energy anomaly detection and her own stock price prediction research.
“Thus, the attention is about making the importance of the input adaptive or in machine learning terms, the features.”
“Every output is simply a weighted sum over the inputs. And here the trick is really that the weights in this sum are not parameters, as you might already know from normal neural networks.”
What you will learn
Explain why RNN encoder-decoder models lose information on long sequences and what the bottleneck problem is.
Compare soft, hard, global and local attention by cost, differentiability and training method.
Compute self-attention step by step using queries, keys, values, softmax and the scaled dot product.
Apply pretrained transformer models through Hugging Face pipelines for translation, text classification and generation.
Use attention weights to make time series models for anomalies and stock prices more explainable.
Key takeaways from the speaker
Understand the mathmatics behind transformers and attention mechanism
Learn how to use attention
Comprehend the key benefits and use cases as to why you should use the attention mechanism
Details
Level
Intermediate
Format
Tech talk
Technical depth
Deep technical
For
Early career, Mid-career, Senior individual contributor
Length
35:38
Published
July 2022
Questions this talk answers
Why did sequence-to-sequence RNN models need attention?
They work well only for short sequences of fewer than about 20 time steps, because a single context vector cannot encode all input steps. RNNs forget earlier steps and stacked layers suffer from vanishing gradients.
What is the difference between soft and hard attention?
Soft attention applies learned weights to all inputs and is fully differentiable but costly for large inputs. Hard attention picks one area at a time, needs less computation at inference, but is non-differentiable and requires Monte Carlo sampling to estimate gradients.
How does self-attention compute its outputs?
Every output is a weighted sum of the inputs, with weights derived from dot products between input vectors and normalised with softmax. Each input vector serves as query, key and value.
Why scale the dot product and use multiple attention heads?
Scaling by the square root of the input dimension keeps the weights in range and avoids vanishing gradients from the softmax. Multiple heads capture different relations in a sentence and can be computed in parallel, which RNNs cannot do.
Does a vision transformer beat convolutional networks?
In the paper she presents, it depends on data size: it performs worse on about 1 million images, comparably on 14 million, and outperforms ResNet on 300 million images.
About this talk
The concept of attention was first introduced into machine translation in 2015. Nowadays, attention has acquired considerable prominence in the area of artificial intelligence as an essential component of neural networks for a large number of Computer Vision applications, speech recognition and natural language processing, and more recently for time series prediction. To some extent, the attention concept is driven by how we, as humans, direct our visual attention to various areas of a picture, video, or pair of words in a sentence.
In this session, you will learn the theory behind the attention mechanism and how to apply it through practical use cases and applications.
Why multi-head attention beats RNNs on speed (79 s)
When vision transformers overtake CNNs (64 s)
Welcome everyone to my talk today about the attention mechanism in deep learning and I'm very excited to be part of the Women in Tech global conference this year. And yeah, I'll shortly introduce myself. I'm Nicole Koenigstern and I'm currently working as the data science and technology lead at impactvice and impactvice is a big data and analytics company using machine learning to analyze companies.
All right. So today's talk will be as follows. So I first will give you a short background about sequence to sequence modeling and in general attention mechanism. Then we go over the different types of attention, then we will dive deeper into self and multi head attention. And then we will round up with some practical use cases how you actually can apply the attention mechanism in deep learning.
All right. So before attention and transform a sequence to sequence work pretty much like the illustration on the current slide and the elements of the sequence X1, X2 and up to XN are usually called tokens and they can be literally anything. For instance, text presentations, representations, pixels, or in the case of videos images. And further, the two sequences can be of the same or even arbitrary length. And in case you are now questioning yourself, yes, recurrent neural networks dominated this kind of task. And the reason is really simple. So we just want to treat sequences sequentially. And this sounds straightforward, right? But Well, interestingly, the transformers proved as it is not. And to look a bit from another angle and to get a bit more understanding of the encoder and decoder, there are nothing more than stacked RNN layers, such as LSTMs. And the encoder processes the input and produces then one compact representation called set from all the input time steps.
Well, it's a compressed format of the input. And on the other hand, the decoder receives the context vector set and then generates the output sequence. And the most common application for sequence to sequence is language translation. So we're used to use to work very well, especially with long short term memories and the gated recurrent unit as a component component.
And all right. So, well, now you might ask yourself, so if they worked good, so what's the problem? And the problem is that they just work well for small sequences like shown on the current slide here. So with time steps lesser than 20. And the problem is that the intermediate representation set cannot encode information from all the input time steps. And this is commonly known known as the bottleneck problem. And the bottleneck set needs to capture all the information about resource sentence. And the mathematics in theory indicate that it is possible.
However, in practice, how far we can see in the past, which is the so-called reference window is finite. And RNNs tend to forget information from time steps that are further behind. And moreover, the stack less than 20. And moreover, the stacked RNN layers usually create the well-known vanishing gradient problem as we have visualized on the current slide.
All right. So yeah, Alex Grave, who is a research scientist at DeepMind stated 2020 that memory is attention for time. So let's dive a bit deeper what really attention is all about. And to give you a bit more of an example, you might know that we as humans seldom utilize all available inputs to actually complete a task. So you can think about listening to one of your friends talking.
Maybe you're talking in the coffee shop with many other people talking around you that you will have different conversations going around you, people placing their orders and all of that tends to fall into the background because you, if you're complex and sophisticated brain and ears are capable to pay attention to only what's really important to you. That is in that moment your friend and selectively ignore the things occurring around you that are simply not relevant for you at that moment.
And here the important thing is that your attention is adaptive to the situation. That is, you will ignore the background to just listen to your friend only then there is nothing more important than your friend. So, for instance, if there was going a fire alarm off, you will stop paying attention to your friend and you shift your focus or your attention on this new important thing.
Thus, the attention is about making the importance of the input adaptive or in machine learning terms, the features. All right. So now that we have some basic understanding what attention is all about, let's go over different types of attention. So we have soft and hard attention, also global and local attention. And then lastly, we will look deeper into the self attention.
Right. So with soft attention, the weights for alignment are learned and softly applied, for instance, to all patches in the original picture. And by doing so, the model is smooth and differentiable, which represents an advantage because we can compute with gradient descent because soft attention is fully differentiable because it's a deterministic mechanism. The disadvantage, however, is that it is costly when the source input is large.
And in contrast, if we have hard attention, which just chooses one area of a picture to work on at a time, the advantage is here that less computation is required during the inference process. However, the model here is non-differentiable, which then necessitates the use of the more complex training method. So we have to estimate the gradients by Monte Carlo sampling.
And because we use Monte Carlo sampling, it is now a stochastic process. And you see here on the upper row, the examples with applying a soft attention. And in contrast, on the bottom line, we have the hard attention. And here is just how the formula formulas look like in contrast. All right. So attention often will be calculated across the entire input sequence, which is then just global attention.
And despite the simplicity, this calculation may be computational costly and sometimes really unneeded. And as a consequence, several papers have advocated local local attention as a possible remedy. And obviously, this would be sometimes preferable for extremely lengthy sequences. And local attention can be also seen as kind of hard attention since one must make a choice to reject certain input units.
And the main distinction here between hard and local attention models is that the local attention is nearly always differentiable. While, as you already know now, hard attention is not. All right. So now let's dive a bit deeper into what self attention is all about. And it's really one of the most common attention types used, and it's called self or key value attention.
And self attention is used in transformer based architectures, which are primarily used in modeling language understanding tasks. And with the use of the attention, they assume the use of a recurrence in the neural networks as they trust entirely on the self attention mechanism to draw global dependencies between the inputs and the outputs. So remember, at the beginning of the talk, I showed you how the sequence to sequence models usually work with all of the current models in the cells in between.
So instead of looking for an input output sequence association or alignment, we are now looking for scores between the elements of the sequence, as we see here on the current slide. So we have a probability score matrix and then just look for those relations between them. So all right. So what is then the math behind this? Let's look a bit deeper.
So self attention is also a sequence to sequence operation, and it's composed of the input, which is a sequence of tensors and then of the output, which again is a sequence of tensors. Each one is a sequence of tensors. Each one is a weighted sum of the input sequence. And to produce here the vector yi, the self attention operation simply takes a weighted average over all the inputs.
And here the simplest option to do this is the dot product. And so that is how it looks graphically. So we have our inputs at the bottom and we have our outputs at the top. And in between there will be just some combination of all these inputs. So really at heart, the operation of self attention is very simple.
Every output is simply a weighted sum over the inputs. And here the trick is really that the weights in this sum are not parameters, as you might already know from normal neural networks. They are here derived from the inputs. And in mathematical terms, we do the following here. So we have wij and it is derived, it's a derived value.
And to get it, we first compute the weights for the prime wij. So we take the dot product of the i-th element of the sequence with the i-th element of the sequence. And then after computing the weights, we simply apply the softmax function so that all the values sum up to one. And so that is how it looks graphically.
So we have here a sequence of five inputs and five outputs. And now we are just focusing on the computation of y-three. So to do so, we are taking the corresponding vector x-three to generate the weights. And we do that by taking the dot product with every vector. So we start here with x-three, then we do the same with x-three and x-three, and so on.
And now that we have all these five weights, we take the softmax that it sums up to one. Then we are going to multiply each input vector by the weights we just computed. And then we sum them all up. And this gets us then the vector y-three. And of course, we are going to do the same for all output vectors.
And you might see the problem here. So if we would do that for all the input vectors or output vectors to get the computation done, we would end up with a lot of loops. So obviously, this is something we don't want to do in machine learning or deep learning. So we want to have a vectorized version of it.
So to then use a vectorized version of that computation, we compute all the row weights of w-prime. We can simply compute a large matrix of all dot products of x with itself. So this w-prime matrix contains every dot product of every input vector with every other input vector. Then we apply the softmax function simply row-wise to that matrix.
And then we simply multiply this weight matrix with our input matrix x to give us all the weighted sums in one matrix multiplication. And one thing we need to consider here is that to compute the actual self-attention, we have to acknowledge that every vector occurs in three different positions. So first, the vector that is used to compute the weighted sum that finally provides the output and indicated here in blue, which is then called the value.
Then second, as the input vector that corresponds to the current output matches against every other input vector. And this is called then the query. We have that inter keys. And then lastly, the vector which is matching against the query. This is here in the current slide in orange, and it is called the key. And let's just go over these steps a bit more in isolation to really understand what's going on.
So we have the input vector x of i and it's used as you write now in three different ways. So first, we are going to compute the attention weight for its own output. And it is called the query. So secondly, we are going to compare this to every other vector to compute the attention weight and we can think then of that as the key.
And then lastly, we are going to sum this up with all the other vectors to form a result of the attention weighted sum, which then serves here as the value. And to give you a bit more of an intuition, how you can think of that operation or of the keys, the query and the values, we can compare it here to to a dictionary you use usually in using in any programming language.
And I used here just Python. So we have the values, the key, and we can request it by the by the query. And yeah, mathematical speaking to just summarize it up here. So we can think of these processes as matrix multiplication. And then we just introduce a softmax function to make sure that it sums up to one.
And we do that for each input vector with these three matrices. And then we combine the results in the three aforementioned ways to then produce the output for Y of I. And if we are looking into the scaled self attention, which was used for the original paper attention is all unique, which was kind of a starting for the transformer models.
We will have to use scaled self attention. And in basic, it's really simple. Instead of using the normal dot product, we just use a scaled dot product. And we scale that by the root square of the input dimension. And we do this because the input dimension of the input vector grows, and so does then the size of the dot product.
And this factor is of a sequence of a square root of D. So with that, we normalize the average dot product, which keeps the weights just in a certain range. And wherefore, with that trick, we do not suffer from the vanishing gradient problem from the softmax multiplication, because this one is not linear. And as you remember, I already mentioned that the RNNs do suffer from that particular problem.
And we can remedy kind of that problem with using the scaled dot attention. And yeah, I already mentioned attention is all you need, which started with the error of transformer models. And here you will also encounter multi head attention. And that just means that each head is a set of the three weight majors I just mentioned. So we have to multiply these heads to learn the weights simultaneously.
And here important to note is that if you have three heads, the matrix just becomes three times as big as one head. So we are basically learning those transformations at once. And why might this be interesting or why do we want to do that? Well, in many sentences, there are different relations to consider. And here on the current slide, the meaning of the word bad is inverted by not, right?
So its relation also to the word movie is completely different. It describes the property of the movie. And therefore, the idea behind multi head self attention is that multiple relations in that sentence are best captured by different self attention. And of course, since these heads are independent from each other, we can perform the self attention computation in parallel with different workers.
We might use worker one here, then worker two for that one, and then worker three for that last attention matrix. And this is really speed up in the computational terms. And it's really an advantage in regards to the RNNs. Because with the RNNs, we can't do parallel computing. All right. So now that we know and do understand the basics of the attention, let's go into some use cases, how you actually use the attention mechanism.
All right. So I already mentioned attention is all you need. And this work really represented the first sequence transduction model based entirely on the attention mechanism. So by replacing the recurrent layers, most commonly used in the encoder decoder architecture, which I introduced at the beginning with the multi headed self attention. And to give you a bit more intuition, how that actually work, let's look at a bit short at some code and we see here.
Yeah, I added just the paper here with some links to more resources and I will skip the positional encoding used here. You can look that up later on as a link to a pretty good blog post about that. So we have here the step one. So we have the word embeddings to get the positional information about that.
And then we stick those together in a single array to get them here, the matrix. Then step two, we have the weight matrixes. So note that I just assigned them some numbers and in theory or in practice, you would actually train that during the training process of the network and the weights will then be learned by the network.
And you will usually assign that with random numbers, which are very small at the beginning. And then step three. And then step three, we are going to do the mentioned matrix multiplication to obtain then the query, the keys and the values. And then we can process each input vector to fulfill the different roles I just mentioned to do the computation.
And then we apply the scaled attention scores. So to keep the weight matrices within the certain range, then we apply the softmax. And then step five, we use here the scaled softmax attention for each score. And I just do it here a bit more granular that you get a bit more of an intuition and insight how it's really done instead of using the aforementioned vectorized version.
So it's just a bit more granular. And then we do the computation for the score one. And then as the last step, we just sum that up to get the first line of the output matrix. And of course, for step eight, we repeat all the steps from one to seven to get it for all the inputs. Then here is the example how we use the multi head attention.
So we just assume that we have trained these eight heads of the attention layers. And what we then do, we concatenate these heads together to obtain then the output of the model. And we have that here. And to give you a bit more thing to play around with and to get a bit more of a feeling what you really can achieve with the transformer models in a very easy way, I provide you some code from the having face API.
And you can just run that within the Jupyter notebook. And you just use the pipeline, what you want to do. And then you enter, for example, the translation from English to German and put in the sentence you want to translate. And then you get out the result. And as my mother tongue is German, I can assure that it's the correct translation.
And as the next example, I provided you a text classification. And here we have the just the sentence. I enjoyed the movie and the model correctly indicated that this is a positive review. So with over 99% certainty. And another also interesting thing is that we can use it for text generation. And here we have the question and some sort of context.
And for you, those who already had read Hitchhiking through the galaxy know that the number 32 is the answer to life, the universe and everything. And I will provide you the source code or the link to that notebook later on. So let's look at some more examples and use cases. So I want to also show you the vision transformer.
And this work is based upon the theory that image patches are basically the sequence tokens like words. So like I mentioned at the beginning of that presentation. And in fact, the decoder block is the same as the original transformer model I just introduced to you. And if the vision transformer is trained on a data set with more than 14 million images, it can approach or even beat state of the art convolutional neural networks.
And let's dive in a bit deeper and look how it actually works. So we split an image into patches. Then we do flatten those patches. Then we produce a lower dimensional linear embedding from the flattened patch. Then we add some positional embeddings. And then we feed the sequence as an input to a standard transformer encoder. Then as usual, we pretrain the model with image labels, which is fully supervised on the large data set.
And then as a last part, we fine tune the downstream on the data set for image classification. We actually want to classify. And here we see on the left, the performance of the vision transformer then pre-trained on different data sets. And on the right, we see a comparison of the performance on the computational trade off between ResNet, which is really one of the state of the art big transfer convolutional neural networks to classify images.
And to investigate the impact of the different data sizes on the model, the researchers or the authors of the paper trained the vision transformer on different image data sets. And they used ImageNet 21K, which consists of those 14 million images and 21,000 classes. And they used also JFT, which has 300 million images and 18,000 classes. And then they compare the results of the state of the art CNN.
So convolutional neural networks and big transfer like a ResNet. And they trained those just on the same data sets. And interestingly here, the vision transformer performs significantly worse than the CNN or the big transfer. And just trained on the smaller data set with 1 million images. However, now, then we use the ImageNet 21K with 14 million images.
The performance is comparable. And if we use the large data set with the 300 million images, it now outperforms the state of the art ResNet. And another very interesting use case is here to use it for time series anomaly detection. And here, the authors of that paper used it in combination with the long short term memory, which we have here in the architecture.
And they used it in within the rational recurrent outer encoder. I won't go into what a rational outer encoder is. I will provide research to the paper and you can read it up later on. So I just want to focus here on the attention part. And what they did, they had here the energy time series, and that is how the time series looks without any faults or anomalies.
And we have here just a simple example of an inverter fault. And if we now look what they did, so they used the attention weights generated by the model to get more information about the anomaly in that time series. So if we compare, that is just the plot from a normal time series. And here is the accompanying attention weight.
And here we have just a minor fault, just a brief shading, so it's not that big of an anomaly. And we see there's almost no difference, just here a little bit and here a little bit. But if we then go to the anomaly of having a snow, we see that the attention is distributed quite differently. So that really gives an enhancement to actually better understand how the anomaly is going on in that time series and to enhance and enable to earlier on predict really the anomaly in that time series.
Because it's an unsupervised task usually, and it's very hard to do because the anomaly is usually very sparse in that time series. And so you need to have more information how you really can predict that anomaly. And as a last example, I want to show you how to use that also for stock price prediction. And this is a work I recently published a paper on.
So that's something I did by myself. And I used the attention mechanism to modulate information between a modular structure of a network. And this is how the network structure actually looks like. So we have here active and inactive modules that don't go very deep into how the modules are and what they are. Just note that we have here sparse communication and the communication is moduled by the key value attention.
So the attention and the attention scores decide or help the network decide which information should be transferred between the previous and the next time step. So further, the network has also the input modulated by attention. So also the importance of the input and the activation of those models will be modeled by using the key value attention here.
And here are just an excerpt of the results from training that network. So I used here the closing, just the history of closing prices. And I used it here in the combination with closing prices and new sentiment. And if we see here the prediction and the combination to the state of the art, a long short term memory and just a simple recurrent neural network.
So we see that the network here, which uses the attention mechanism has quite an improvement in its capability to predict the stock prices. And if we even add the second feature, which is the new sentiment, which is especially hard because it's an alternative data set and it has a lot of noise in it. And most networks really struggle with handling that we see that if we use the attention mechanism, we really can secure significantly outperform the state of the art network by using the attention mechanism.
And moreover, as just showed for the example, the energy time series, we also can use here the attention and the attention weights to enhance the explainability of the network. To really analyze what actually is going on in the network and to better understand what decisions the network makes to predict the time series in that certain way. And then to look further into. And here we just see on the current slide, a plot of the activation pattern for that network, which is driven by the input attention mechanism.
And here we have the different modules, which get then activated by the importance of that feature time steps. And here we have the time and you can easily observe that we have quite a different pattern between just using the closing price, which is here in black and use the combination of the closing price of that stock and the accompanying recording sentiment score.
And then we can really use that to then see what event was driven by that decision and learn or try to give the network more to learn about that event. All right. So, yeah, as mentioned, here are the sources to the use cases, all of the links to the papers. And here is the link to my GitHub repo. So you find all the code for the notebooks, for the notebook about the attention mechanism explanation in NumPy.
And I provided you. So the vision transformer notebook with an example, you just can run by yourself and train it and also two notebooks about the energy time series along with the code I implemented from from that paper. So feel free to explore. And if you have any questions, feel free to ping me on LinkedIn, just send me a private message.
I'm happy to answer any questions you might have.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
How attention lets deep learning models decide what matters
Nicole Koenigstein traces the path from forgetful recurrent networks to the self-attention math behind transformers, and shows it at work in vision, anomaly detection and finance.
In short
Attention solves the core weakness of recurrent sequence models: they compress everything into one context vector and forget what came earlier. Nicole Koenigstein shows that self-attention replaces that bottleneck with a surprisingly simple idea, a weighted sum of the inputs whose weights are computed from the inputs themselves. She then shows how that idea powers transformers in language, vision, energy anomaly detection and her own stock price research.
Key takeaways
Understand the mathematics behind transformers and the attention mechanism.
Learn how to use attention.
Comprehend the key benefits and use cases, and why you should use the attention mechanism.
Why recurrent models hit a wall
Koenigstein starts with classic sequence-to-sequence modelling. Before transformers, an encoder built from stacked recurrent layers such as LSTMs read the input and produced one compressed representation, the context vector, which a decoder then turned into an output sequence. Language translation was the textbook application, and recurrent networks dominated it because treating sequences sequentially seemed the obvious approach.
The trouble, she explains, is that this works well only for short sequences, fewer than about 20 time steps on her slide. A single intermediate representation cannot encode information from every input step, which is known as the bottleneck problem. The reference window is finite in practice: "And RNNs tend to forget information from time steps that are further behind." (3:47) Stacking recurrent layers also brings the familiar vanishing gradient problem.
Attention as adaptive focus
To build intuition, Koenigstein uses a coffee shop. You can follow a friend's voice while orders and other conversations fade into the background, yet a fire alarm instantly takes over your focus. "And here the important thing is that your attention is adaptive to the situation." (5:20) Translated into machine learning: "Thus, the attention is about making the importance of the input adaptive or in machine learning terms, the features." (5:46)
She then compares the main types. Soft attention applies learned weights to all inputs, is deterministic and fully differentiable, so it trains with gradient descent, but becomes costly for large inputs. Hard attention picks one region at a time, saving computation at inference, but is non-differentiable, so gradients must be estimated with Monte Carlo sampling. Global attention covers the whole sequence; local attention restricts the window for very long sequences and, unlike hard attention, is nearly always differentiable.
The math: a weighted sum with data-driven weights
Self-attention, also called key-value attention, is the type used in transformer architectures. Instead of aligning an input sequence with an output sequence through recurrent cells, it scores the relations between all elements of the sequence. "So really at heart, the operation of self attention is very simple." (10:19) She makes the key point explicit: "Every output is simply a weighted sum over the inputs. And here the trick is really that the weights in this sum are not parameters, as you might already know from normal neural networks." (10:45)
She walks through one output, taking the dot product of one input vector with every other vector, applying softmax so the weights sum to one, multiplying and summing. Because looping over every vector is slow, she shows the vectorised version as matrix multiplications. Each input plays three roles: query, key and value, which she compares to looking up a value by key in a Python dictionary.
Scaling up: scaled and multi-head attention
The original transformer paper, Attention Is All You Need, uses a scaled dot product, dividing by the square root of the input dimension. This keeps the weights in a stable range and avoids vanishing gradients from the softmax. Multi-head attention learns several sets of query, key and value matrices at once. Using the sentence where "not" inverts the meaning of "bad" in a movie review, she explains: "And therefore, the idea behind multi head self attention is that multiple relations in that sentence are best captured by different self attention." (18:45)
Because the heads are independent, they can run on separate workers in parallel, a clear advantage over recurrent models: "Because with the RNNs, we can't do parallel computing." (19:18) She then walks through NumPy code step by step and shows Hugging Face pipelines in a Jupyter notebook for English to German translation, sentiment classification and text generation.
From translation to images, energy and stock prices
The vision transformer treats images the same way: "And this work is based upon the theory that image patches are basically the sequence tokens like words." (25:16) As she cited from the paper, data size decides the outcome: the model performs worse than convolutional networks on about 1 million images, comparably on ImageNet-21k with 14 million, and outperforms ResNet on JFT with 300 million.
For energy time series, researchers combined attention with LSTMs, and the attention weights shifted clearly for a snow-related fault while barely changing for a brief shading fault. In her own published work on stock price prediction, attention modulates information between modules of a network, and she reports a significant improvement over LSTM and simple RNN baselines, especially when adding noisy news sentiment. Attention weights also aid explainability: "To really analyze what actually is going on in the network and to better understand what decisions the network makes to predict the time series in that certain way." (33:07) Her notebooks are available on her GitHub repository.
Watch the talk
"Attention? Attention! - Attention Mechanism In Deep Learning" by Nicole Koenigstein, 36 minutes: https://www.womentech.net/video/attention-attention-attention-mechanism-in-deep-learning-nicole-koenigstein
About the speaker
Nicole Koenigstein is Data Science/Technology Lead at impactvise.com, working as a data scientist and quant and data engineer, and also with Nectar Financial. She reviews machine learning books and online courses for Manning Publications and is committed to making STEM more attractive to women.
Questions this talk answers
Why did sequence-to-sequence RNN models need attention?
What is the difference between soft and hard attention?
How does self-attention compute its outputs?
Why scale the dot product and use multiple attention heads?
Does a vision transformer beat convolutional networks?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Sequence-to-sequence encoder-decoder
Soft and hard attention
Global and local attention
Self-attention (key-value attention)
Scaled dot-product attention
Multi-head attention
Transformer
Vision Transformer
Monte Carlo sampling
Resources mentioned
Attention Is All You Need (paper)
Hugging Face pipeline API
Jupyter notebooks on the speaker's GitHub repository (NumPy attention explanation, vision transformer, energy time series)
Blog post on positional encoding linked in the slides
Cite this talk
Nicole Koenigstein. "Attention? Attention! - Attention Mechanism In Deep Learning." WomenTech Network, 2022. https://www.womentech.net/video/attention-attention-attention-mechanism-in-deep-learning-nicole-koenigstein