InLine
Stable Attention through Score Scaling
Abstract
Attention mechanisms aggregate information using query–key similarities. We investigate how score scaling affects the concentration of attention weights as the key dimension increases.
1. Introduction
Attention mechanisms allow sequence models to combine information from different input positions. Early work in neural machine translation introduced learned alignment to select relevant source representations during decoding (Bahdanau et al., 2014). Subsequent attention-based models explored alternative scoring functions and alignment strategies, illustrating how the choice of similarity computation shapes the information available to a decoder (Luong et al., 2015).
The Transformer placed attention at the center of a highly parallel sequence architecture (Vaswani et al., 2017). In its dot-product formulation, queries and keys determine similarity scores, while values provide the representations that are combined. This separation makes the mechanism efficient to implement, but it also raises a numerical question: how should the magnitude of a score change when the dimensionality of its inputs increases?
2. Method
The matrices Q, K, and V represent queries, keys, and valuesdenote the query, key, and value matrices. Our algorithm computes pairwise dot products and normalizes the final outputscales the scores before softmax.
The computation implemented in our algorithm is expressed as:
Here, dk is the key dimension. The scale is applied before normalization, and the resulting weights to aggregate the value representations.
We evaluate key dimensions of 16, 64, and 128 while keeping the remaining settings fixed. Attention-weight entropy is recorded for both variants, with higher entropy indicating a less concentrated distribution.
3. Results
The two variants behave differently as the key dimension increases . The unscaled variant produces more concentrated weights, whereas the scaled variant maintains a more distributed pattern.
| dk | Unscaled entropy | Scaled entropy |
|---|---|---|
| 16 | 0.86 | 0.95 |
| 64 | 0.31 | 0.94 |
At dk = 64, scaled attention yields an entropy of 0.94, compared with 0.31 for the unscaled variant. These results support the use of score scaling in the evaluated conditions.
This comparison demonstrates improved downstream task performanceindicates less concentrated attention weights in the evaluated conditions.
4. Conclusion
Score scaling moderates the effect of increasing key dimensionality on attention concentration. Future work will examine whether this behavior persists across longer sequences and different input distributions.
References
Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint. arXiv:1409.0473.
Luong, M.-T., Pham, H., & Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. arXiv preprint. arXiv:1508.04025.
Vaswani, A., et al. (2017). Attention is all you need. arXiv preprint. arXiv:1706.03762.
A larger key dimension can increase the magnitude of unnormalized dot products. The is therefore central to explaining how scores become attention weights. Building on the formulation of Vaswani et al. (2017), we compare scaled and unscaled variants under matched conditions. Our aim is to characterize changes in weight concentration, rather than to infer downstream performance from attention patterns alone.