Inner Products Measure Alignment in Attention
Compute a dot product, read its sign and magnitude geometrically, and see how query-key scores become attention weights.
Category
3articlesin this category.
Compute a dot product, read its sign and magnitude geometrically, and see how query-key scores become attention weights.
Compare L1, L2, and L-infinity vector size, see their unit balls change, and distinguish parameter penalties from input-robustness budgets.
Calculate L1, L2, and L-infinity vector norms, verify their defining rules, and see what an embedding magnitude can—and cannot—tell you.