Scaled Dot Product Attention — Machine Learning/Neural Networks

Scaled Dot-Product Attention

Machine Learning / Neural Networks

H attention heads over a window of T positions:

Q = X·W_q, K = X·W_k, V = X·W_v, A = softmax(Q·K' ÷ √d_k) row-wise, Y = A·V – each head over its own slice of the three projections, and the heads' outputs concatenated across columns.

Each output row is a weighted average of the value vectors, with the weights decided by how well that position's query matches every position's key. It is the mechanism a transformer is built from, and the reason a model can relate two positions directly rather than passing information along a chain of recurrences.

The window arrives as a matrix rather than being buffered inside the block, which is what lets Positional Encoding feed straight into it – attention alone is order-blind, so that block exists to mark position, and the two are meant to be used together.

Ports

• X – the window, [T, d_model]: one row per position, one column per feature. T and d_model are read from the signal, not configured.
• Y – the attended window, [T, H·d_v] – the heads side by side, head n writing columns [n·d_v, (n+1)·d_v). That is all of W_v's columns, so the output width does not change with the head count: splitting a projection into more heads changes how the window is mixed, not its shape.

Parameters

• Query Weights – W_q, [d_model, H·d_k]: the heads' query projections side by side, head n taking the contiguous column range [n·d_k, (n+1)·d_k). That is how PyTorch's nn.MultiheadAttention packs in_proj_weight, so a trained set slices in unchanged. Its width must divide by Heads.
• Key Weights – W_k, the same shape as W_q and sliced the same way: queries and keys are compared by dot product, so they must live in the same space. A mismatch is reported rather than broadcast.
• Value Weights – W_v, [d_model, H·d_v], sliced the same way. Its full width is the output width and is free of the other two; it too must divide by Heads.
• Heads – H, how many attention heads share the three projections; a whole number ≥ 1, and 1 by default. ⚠ d_k is per head, so the scale 1 ÷ √d_k moves when you change this even though no matrix changed shape: two heads over a [d_model, 4] W_q have d_k = 2, not 4. Each head attends independently – its own scores, its own softmax – which is the whole point: one head can only express one weighting of the window per position, and several let different heads track different relations.
• Masking – whether a position may attend to later ones.
  • None – every position attends to the whole window. The encoder-style reading, and the right one when the window is a complete snapshot.
  • Causal – position i attends only to j ≤ i, so no output row depends on a position later than itself. The decoder-style reading, and the one to use when the window is a sliding history and a later row must not leak into an earlier one.
• Sampling Time (s) – zero or less inherits the solver's rate; a positive value runs the block at that period.

Code export

All ten targets: Python, MATLAB, Java, Rust, C, C++, VHDL, Verilog, SystemVerilog and PLC Structured Text. Every projection, score and weight is unrolled: T, d_model, H, d_k and d_v are known at export time, so no backend contains a loop, an index type or a mask array – and no head index either: each head is emitted as its own straight-line block of statements. The scale 1 ÷ √d_k is folded into the score's coefficients once, and under Causal the masked terms are simply not emitted – there is no −∞ and no comparison anywhere.

⚠ The emitted body grows as H·T²·d_k terms. That is inherent to unrolling attention, and it is the cost of having no loops: a window of 4 is a few hundred lines per backend, a window of 32 is tens of thousands. Splitting a fixed projection into more heads is close to free (H·d_k is unchanged); raising T is not. Keep T to what the model actually needs.

The three HDL targets are simulation-only real arithmetic – the softmax needs exp and a division, neither of which belongs in a Q16.16 datapath – quantizing only at the port boundary, exactly as Dense Layer does for its Tanh and Sigmoid activations.

Simulink bridge

None. Simulink's attention layers live inside its Deep Learning blocks, which take a trained network object rather than three weight matrices, and no config value can carry an object across the bridge. The bridge reports the block rather than dropping it silently, and it has no parity testbench, which is the documented consequence of Support::None rather than a gap.

Notes

• Algebraic and stateless: the whole window comes from the current input, so nothing is carried between steps. Buffering the window is the upstream block's job.
• The row maximum is subtracted before exponentiating, inherited from Softmax for the same reason: without it a logit a user will really produce overflows in single precision and in the HDL paths. It cancels exactly in the ratio, so it changes the arithmetic's range and not its value.
• Multi-head attention is the Heads parameter, not a second block and not a wiring pattern: the heads share one input window and one set of matrices, so carrying them here costs one integer and keeps the three projections in the shape a trained model exports them in. What this block does not carry is the output projection W_o that a transformer applies to the concatenated heads – that is a plain [H·d_v, d_model] matrix multiply, which Dense Layer already is.