0:0035:38

Attention? Attention! - Attention Mechanism In Deep Learning

Nicole Koenigstein 35:38 2022

RNNs falter past 20 time steps; attention learns what to focus on

Sequence-to-sequence models built on recurrent networks work well only for short inputs, under about 20 time steps, before the bottleneck makes them forget. Data science and technology lead Nicole Koenigstein, who has published her own attention-based research on stock price prediction, walks through the math that replaced them. For anyone using transformers without seeing inside them, the question is how a simple weighted sum lets a model decide what deserves its focus.

Follow the self-attention math step by step, then see it in vision, anomaly and stock models

Watch on YouTube

In short

Nicole Koenigstein explains why recurrent sequence-to-sequence models struggle with long inputs and how the attention mechanism overcomes that bottleneck. She walks through soft, hard, global, local and self-attention, the math of queries, keys and values, scaled dot-product and multi-head attention, then shows applications in translation, vision transformers, energy anomaly detection and her own stock price prediction research.

“Thus, the attention is about making the importance of the input adaptive or in machine learning terms, the features.”
“Every output is simply a weighted sum over the inputs. And here the trick is really that the weights in this sum are not parameters, as you might already know from normal neural networks.”

What you will learn

  • Explain why RNN encoder-decoder models lose information on long sequences and what the bottleneck problem is.
  • Compare soft, hard, global and local attention by cost, differentiability and training method.
  • Compute self-attention step by step using queries, keys, values, softmax and the scaled dot product.
  • Apply pretrained transformer models through Hugging Face pipelines for translation, text classification and generation.
  • Use attention weights to make time series models for anomalies and stock prices more explainable.

Key takeaways from the speaker

  • Understand the mathmatics behind transformers and attention mechanism
  • Learn how to use attention
  • Comprehend the key benefits and use cases as to why you should use the attention mechanism

Details

Level
Intermediate
Format
Tech talk
Technical depth
Deep technical
For
Early career, Mid-career, Senior individual contributor
Length
35:38
Published
July 2022

Questions this talk answers

Why did sequence-to-sequence RNN models need attention?

They work well only for short sequences of fewer than about 20 time steps, because a single context vector cannot encode all input steps. RNNs forget earlier steps and stacked layers suffer from vanishing gradients.

What is the difference between soft and hard attention?

Soft attention applies learned weights to all inputs and is fully differentiable but costly for large inputs. Hard attention picks one area at a time, needs less computation at inference, but is non-differentiable and requires Monte Carlo sampling to estimate gradients.

How does self-attention compute its outputs?

Every output is a weighted sum of the inputs, with weights derived from dot products between input vectors and normalised with softmax. Each input vector serves as query, key and value.

Why scale the dot product and use multiple attention heads?

Scaling by the square root of the input dimension keeps the weights in range and avoids vanishing gradients from the softmax. Multiple heads capture different relations in a sentence and can be computed in parallel, which RNNs cannot do.

Does a vision transformer beat convolutional networks?

In the paper she presents, it depends on data size: it performs worse on about 1 million images, comparably on 14 million, and outperforms ResNet on 300 million images.

About this talk

The concept of attention was first introduced into machine translation in 2015. Nowadays, attention has acquired considerable prominence in the area of artificial intelligence as an essential component of neural networks for a large number of Computer Vision applications, speech recognition and natural language processing, and more recently for time series prediction. To some extent, the attention concept is driven by how we, as humans, direct our visual attention to various areas of a picture, video, or pair of words in a sentence.

In this session, you will learn the theory behind the attention mechanism and how to apply it through practical use cases and applications.

Topics

Related talks

Browse the library