Scaled Dot Product Attention — Machine Learning/Neural Networks

Scaled Dot-Product Attention

Machine Learning / Neural Networks

One attention head over a window of T positions:

Q = X·W_q, K = X·W_k, V = X·W_v, A = softmax(Q·K' ÷ √d_k) row-wise, Y = A·V

Each output row is a weighted average of the value vectors, with the weights decided by how well that position's query matches every position's key. It is the mechanism a transformer is built from, and the reason a model can relate two positions directly rather than passing information along a chain of recurrences.

The window arrives as a matrix rather than being buffered inside the block, which is what lets Positional Encoding feed straight into it – attention alone is order-blind, so that block exists to mark position, and the two are meant to be used together.

Ports

• X – the window, [T, d_model]: one row per position, one column per feature. T and d_model are read from the signal, not configured.
• Y – the attended window, [T, d_v], where d_v is the width of the value projection.

Parameters

• Query Weights – W_q, [d_model, d_k].
• Key Weights – W_k, the same shape as W_q: queries and keys are compared by dot product, so they must live in the same space. A mismatch is reported rather than broadcast.
• Value Weights – W_v, [d_model, d_v]. Its width sets the output width and is free of the other two.
• Masking – whether a position may attend to later ones.
  • None – every position attends to the whole window. The encoder-style reading, and the right one when the window is a complete snapshot.
  • Causal – position i attends only to j ≤ i, so no output row depends on a position later than itself. The decoder-style reading, and the one to use when the window is a sliding history and a later row must not leak into an earlier one.
• Sampling Time (s) – zero or less inherits the solver's rate; a positive value runs the block at that period.

Code export

All ten targets: Python, MATLAB, Java, Rust, C, C++, VHDL, Verilog, SystemVerilog and PLC Structured Text. Every projection, score and weight is unrolled: T, d_model, d_k and d_v are known at export time, so no backend contains a loop, an index type or a mask array. The scale 1 ÷ √d_k is folded into the score's coefficients once, and under Causal the masked terms are simply not emitted – there is no −∞ and no comparison anywhere.

⚠ The emitted body grows as T²·d_k terms. That is inherent to unrolling attention, and it is the cost of having no loops: a window of 4 is a few hundred lines per backend, a window of 32 is tens of thousands. Keep T to what the model actually needs.

The three HDL targets are simulation-only real arithmetic – the softmax needs exp and a division, neither of which belongs in a Q16.16 datapath – quantizing only at the port boundary, exactly as Dense Layer does for its Tanh and Sigmoid activations.

Simulink bridge

None. Simulink's attention layers live inside its Deep Learning blocks, which take a trained network object rather than three weight matrices, and no config value can carry an object across the bridge. The bridge reports the block rather than dropping it silently, and it has no parity testbench, which is the documented consequence of Support::None rather than a gap.

Notes

• Algebraic and stateless: the whole window comes from the current input, so nothing is carried between steps. Buffering the window is the upstream block's job.
• The row maximum is subtracted before exponentiating, inherited from Softmax for the same reason: without it a logit a user will really produce overflows in single precision and in the HDL paths. It cancels exactly in the ratio, so it changes the arithmetic's range and not its value.
• Single head. Multi-head attention is this block several times over with different weights and its outputs concatenated – which Matrix Concatenate already does – so the head count is a wiring decision rather than a parameter here.