Positional Encoding — Machine Learning/Neural Networks
Machine_Learning/Neural_Networks/Positional_Encoding · 1 input / 1 output port(s) at insert · exports to Python, MATLAB, Java, Rust, C, C++, VHDL, Verilog, SystemVerilog, PLC Structured Text
Description#
The block's own DESCRIPTION_HTML, rendered verbatim — the same text the config dialog's info panel and the library navigator show. Fix a wrong sentence in the block's .cpp (R-D9), never here.
Positional Encoding
Machine Learning / Neural Networks
Adds a position term to a buffered sequence: y[p,d] = u[p,d] + PE[p,d], where p is the position (row) and d the feature (column).
Attention is a weighted sum, and a sum does not care what order its terms arrive in – so a model built from it cannot tell position 3 from position 5 until something in the vector itself says which is which. This block is that something, and without it Scaled Dot Product Attention is order-blind.
The default table is the sinusoidal one: PE[p,2i] = sin(p / base2i/D) and PE[p,2i+1] = cos(p / base2i/D). Each feature pair turns at its own rate, from one radian per position at the fastest down to almost nothing at the slowest, so every position gets a distinct signature and nearby positions get similar ones.
Ports
- u – the buffered sequence, a [T,D] matrix: one row per position, one column per feature. This is the transformer convention (sequence length × model dimension).
- y – the same [T,D], with the position term added. The model dimension does not grow: this is an add, not a concatenation.
Parameters
- Encoding – which table to add.
- Sinusoidal – the fixed formula above. Needs no training and extends to any sequence length.
- Learned Table – use Position Table verbatim, which is what a trained model ships.
- Base – the wavelength base of the sinusoidal formula, greater than one; 10000 is the published value. Larger spreads the rates further apart, which suits longer sequences. Ignored for Learned Table.
- Position Table – the [T,D] table used by Learned Table, which must match the input exactly. Ignored for Sinusoidal.
- Sampling Time (s) – zero or less inherits the solver's rate; a positive value runs the block at that period.
Code export
All ten targets: Python, MATLAB, Java, Rust, C, C++, VHDL, Verilog, SystemVerilog and PLC Structured Text.
The table is computed once at export and baked in as constants, so no generated core evaluates a sine. The emitted body is one addition per element, which is why the three HDL targets are genuinely synthesizable Q16.16 despite the block being made of trigonometry. The table is therefore not retunable on the generated core – re-export to change an encoding, a base, or a sequence length.
The emission is unrolled over T×D, as elsewhere in this family, so a long sequence produces a correspondingly large core.
Simulink bridge
No equivalent (Support::None), as for every block in this
family: the weights and tables are baked here, and Simulink has no
positional-encoding block to map onto. It is straightforward to rebuild there as
a Constant of the same table into a Sum.
Notes
- Algebraic and stateless. The whole sequence is expected to be present in the input matrix – this block adds nothing across time, so a model that buffers its sequence elsewhere loses nothing by placing this anywhere in the chain.
- Position 0 is the first row, and its sinusoidal term is (0, 1, 0, 1, …) – sin(0) and cos(0). A common surprise: the first position is not left untouched.
- An odd model dimension is allowed; the final column then carries a sine with no matching cosine, which is the usual convention.
- It is an add, not a concatenation – the position term shares the feature space with the data, which is what the transformer literature does and what Scaled Dot Product Attention will expect.
Code facts#
| Fact | Value |
|---|---|
| registered type | Machine_Learning/Neural_Networks/Positional_Encoding |
| family | Machine_Learning/Neural_Networks |
| solver environment class | ICoreBlock_0_Machine_Learning_1_Neural_Networks_2_Positional_Encoding |
| source | src/ICoreSDK/ICoreBlockLibrary/Blocks/Machine_Learning/Neural_Networks/Positional_Encoding/ICoreBlock_0_Machine_Learning_1_Neural_Networks_2_Positional_Encoding.cpp |
| header | src/ICoreSDK/ICoreBlockLibrary/Blocks/Machine_Learning/Neural_Networks/Positional_Encoding/ICoreBlock_0_Machine_Learning_1_Neural_Networks_2_Positional_Encoding.h |
| default size on canvas | 140 × 80 px |
| ports at insert | 1 in, 1 out |
| code generators implemented | Python, MATLAB, Java, Rust, C, C++, VHDL, Verilog, SystemVerilog, PLC Structured Text |
Ports#
| # | Direction | Signal type | Description label |
|---|---|---|---|
| 1 | in | ICoreDouble | u |
| 2 | out | ICoreDouble | y |
Ports the constructor creates. A block whose port list changes with its configuration adds or removes ports at load time; the count above is the one a freshly inserted block has.
Configuration variables#
| Config variable | Default | Simulink parameter |
|---|---|---|
Encoding | Sinusoidal%~%Learned Table~~Sinusoidal | — |
Base | 10000 | — |
Position Table | 0 | — |
Every block also carries Sampling Time (s) from ICoreBlockSolverEnvironment: zero or less inherits the solver's rate, a positive value runs the block at that period.
Simulink bridge#
| support | Support::None |
| Simulink path | — |
| port-count rule | PortsParam::None |
SampleTime parameter | yes |
Caveat (shown to the user): no Simulink equivalent available, as for every block in this family: the table is baked here and Simulink ships no positional-encoding block to map onto. It is trivial to rebuild there as a Constant of the same table into a Sum
Catalog contract: src/ICoreSDK/ICoreCoder/ICoreCommandSystem/SimulinkBridge/ICoreSimulinkBlockCatalog.h
Description vs code#
The lists agree. check_block_descriptions.py finds no disagreement between the description's Ports, Parameters, Code export and Simulink bridge lists and the code's.
The verdict above is
tools/docs/check_block_descriptions.py(P7.1), which compares LISTS. It cannot read a sentence: "stateless" on a block with a state, an initial-value semantic the recursion does not implement, a "not synthesizable" caveat the HDL banner contradicts. That is the agent audit (P7.3) on BLOCK_DESCRIPTION_AUDIT.md, and this tool's green is not a substitute for one.
File banner (developer view)#
The top comment of the block's .cpp — the maths, the realization and the export strategy, addressed to whoever changes it. It must not contradict the description above (P7.5).
Positional Encoding — telling position 3 from position 5 y[p, d] = u[p, d] + PE[p, d]
PE[p, 2i] = sin(p / base^(2i/D)) PE[p, 2i+1] = cos(p / base^(2i/D))
Attention is a weighted sum, and a sum does not care what order it is given -- so a model built from it cannot tell one position from another until something in the vector says so. This block is that something.
⚠ THE TABLE IS COMPUTED AT EXPORT, NOT AT RUN TIME. It depends only on the input SHAPE and the config, both of which are known before a single sample flows, so every backend receives it as constants and emits one add per element. All ten targets are exact, and the three HDL ones are GENUINELY SYNTHESIZABLE Q16.16 even though the block is made of sines -- the trigonometry happens here, in C++, once.
⚠ NO MODE: see the header. Both encodings fill the same table and are then added identically, so there is no second code path for a second rig to reach.
Sample results#
| t | in ICoreDouble-Out-0 | out ICoreDouble-Out-0 |
|---|---|---|
| 0 | -2 | -2 |
| 0.4 | 0.5 | 0.5 |
| 0.8 | -2 | -2 |
| 1.2 | 0.5 | 0.5 |
| 1.6 | -2 | -2 |
| 2 | 0.5 | 0.5 |
| 2.4 | -2 | -2 |
| 2.8 | 0.5 | 0.5 |
| 3.2 | -2 | -2 |
| 3.6 | 0.5 | 0.5 |
| 4 | -2 | -2 |
| 4.4 | 0.5 | 0.5 |
| 4.8 | -2 | -2 |
| 5.2 | 0.5 | 0.5 |
Every 4th of 60 samples, from the table stimulus.
The same rig also ran:
| Stimulus | What it is | Output range |
|---|---|---|
impulse | Impulse: one sample of 1 at k = 5, 0 elsewhere (Repeating Sequence Stair) | 0 … 1 |
ramp | Ramp: slope 1 from t = 0 | 0 … 5.8 |
sine | Sine Wave: amplitude 1, 2 rad/s, no phase, no bias | -1 … 0.9996 |
step | Step: 0 -> 1 at t = 1 s | 0 … 1 |
Plotted: table — Repeating Sequence Stair: [-2 -1 -0.5 0 0.5 1 2 3], one entry per sample
Category static · sample time 0.1 · 60 steps · commit ccf005c8 · produced by docsSample --out <folder> --steps 60 · data docs/generated/samples/Machine_Learning__Neural_Networks__Positional_Encoding.json · the SVG is generated from those numbers by tools/docs/plot_svg.py, so it is a run and not a drawing (R-D10).