Layer Normalization — Machine Learning/Neural Networks
Machine_Learning/Neural_Networks/Layer_Normalization · 1 input / 1 output port(s) at insert · exports to Python, MATLAB, Java, Rust, C, C++, VHDL, Verilog, SystemVerilog, PLC Structured Text
Description#
The block's own DESCRIPTION_HTML, rendered verbatim — the same text the config dialog's info panel and the library navigator show. Fix a wrong sentence in the block's .cpp (R-D9), never here.
Layer Normalization
Machine Learning / Neural Networks
Normalizes the input vector against its own mean and variance, then applies a learned per-element scale and offset:
mean = Σu ÷ m, var = Σ(u − mean)² ÷ m, yi = (ui − mean) ÷ √(var + ε) × gammai + betai
This is torch.nn.LayerNorm / Keras
LayerNormalization at inference. It is what stabilizes a
transformer or a deep MLP without any dependence on a batch: the statistics come
from the sample in front of it, so the block behaves identically whether it sees
one sample or a million.
The variance divisor is m, not m−1 – the biased estimator, which is what both frameworks use here. At m = 3 the unbiased one would differ by 22 %, so this is not a rounding detail.
Ports
- u – the layer input, a column [m,1]. m must equal the height of Gamma and Beta, and it is checked rather than broadcast. The whole vector is reduced, so every element affects every output.
- Output – y, the same [m,1] column: the normalization never changes the signal's shape.
Parameters
- Gamma – the learned scale, an [m,1] column
(
layer.weight). Use a column of ones forelementwise_affine=False. - Beta – the learned offset, an [m,1] column
(
layer.bias). Use a zero column forelementwise_affine=False. - Epsilon – added to the variance before the square root, so a constant input vector (var = 0) produces zeros rather than a division by zero. Default 1e-5, matching PyTorch; it must match the value the layer was trained with.
- Sampling Time (s) – zero or less inherits the solver's rate; a positive value runs the block at that period.
Code export
All ten targets: Python, MATLAB, Java, Rust,
C, C++, VHDL, Verilog, SystemVerilog and
PLC Structured Text. Gamma, Beta and Epsilon are baked in as literals at
full setprecision(17); the mean, the variance and the square root
are computed at run time, because they belong to the sample.
The three HDL targets are simulation-only: they carry the
statistic in real arithmetic and quantize only at the port
boundary. A per-sample square root and reciprocal have no place in a Q16.16
datapath. This is the opposite of the sibling Batch Normalization block,
whose frozen statistics fold to a multiply-add and which is therefore genuinely
synthesizable.
Simulink bridge
None. Simulink's Deep Learning blocks take a trained network
object, not a set of arrays, and no config value can carry an object across
the bridge. The bridge reports the block rather than dropping it silently, and
it has no parity testbench, which is the documented consequence of
Support::None rather than a gap. Code export verification still
covers it across all ten languages.
Notes
- Algebraic and stateless: the output depends only on the current input, so the layer cannot break an algebraic loop.
- Nonlinear, and deliberately carries no state space. Dividing by a square root of the input's own variance is not an affine map in u, so no A/B/C/D describes it and model reduction correctly refuses the block.
- A constant input vector normalizes to zeros (before beta): every element equals the mean, so the numerator vanishes and epsilon alone keeps the denominator finite. That is the defined behaviour, not a degenerate case.
Code facts#
| Fact | Value |
|---|---|
| registered type | Machine_Learning/Neural_Networks/Layer_Normalization |
| family | Machine_Learning/Neural_Networks |
| solver environment class | ICoreBlock_0_Machine_Learning_1_Neural_Networks_2_Layer_Normalization |
| source | src/ICoreBlocks/ICoreBlockLibrary/Blocks/Machine_Learning/Neural_Networks/Layer_Normalization/ICoreBlock_0_Machine_Learning_1_Neural_Networks_2_Layer_Normalization.cpp |
| header | src/ICoreBlocks/ICoreBlockLibrary/Blocks/Machine_Learning/Neural_Networks/Layer_Normalization/ICoreBlock_0_Machine_Learning_1_Neural_Networks_2_Layer_Normalization.h |
| default size on canvas | 120 × 80 px |
| ports at insert | 1 in, 1 out |
| code generators implemented | Python, MATLAB, Java, Rust, C, C++, VHDL, Verilog, SystemVerilog, PLC Structured Text |
Ports#
| # | Direction | Signal type | Description label |
|---|---|---|---|
| 1 | in | ICoreDouble | u |
| 2 | out | ICoreDouble | y |
Ports the constructor creates. A block whose port list changes with its configuration adds or removes ports at load time; the count above is the one a freshly inserted block has.
Configuration variables#
| Config variable | Default | Simulink parameter |
|---|---|---|
Gamma | [1.2; 0.8; 1.5] | — |
Beta | [0.05; -0.1; 0.3] | — |
Epsilon | 0.00001 | — |
Every block also carries Sampling Time (s) from ICoreBlockSolverEnvironment: zero or less inherits the solver's rate, a positive value runs the block at that period.
Simulink bridge#
| support | Support::None |
| Simulink path | — |
| port-count rule | PortsParam::None |
SampleTime parameter | yes |
Caveat (shown to the user): no Simulink equivalent that could carry the learned arrays: its Deep Learning blocks take a trained network OBJECT rather than gamma/beta columns, and no config value crosses the bridge as an object. Re-create the layer on the Simulink side and paste the same gamma, beta and epsilon in
Catalog contract: src/ICoreBlocks/ICoreCoder/ICoreCommandSystem/SimulinkBridge/ICoreSimulinkBlockCatalog.h
Description vs code#
The lists agree. check_block_descriptions.py finds no disagreement between the description's Ports, Parameters, Code export and Simulink bridge lists and the code's.
The verdict above is
tools/docs/check_block_descriptions.py(P7.1), which compares LISTS. It cannot read a sentence: "stateless" on a block with a state, an initial-value semantic the recursion does not implement, a "not synthesizable" caveat the HDL banner contradicts. That is the agent audit (P7.3) on BLOCK_DESCRIPTION_AUDIT.md, and this tool's green is not a substitute for one.
File banner (developer view)#
The top comment of the block's .cpp — the maths, the realization and the export strategy, addressed to whoever changes it. It must not contradict the description above (P7.5).
Layer Normalization — statistics taken from the CURRENT sample, every step mean = sum(u)/m, var = sum((u-mean)^2)/m, y_i = (u_i-mean)/sqrt(var+eps)*g_i + b_i
Nothing here folds. Batch_Normalization's four arrays are frozen, so its square root is evaluated once at export time; this block's mean and variance belong to the sample in front of it, so all ten backends carry the reduction and the root.
The three HDL targets are therefore SIMULATION-ONLY
realarithmetic, the same choice Recursive IIR and Dense_Layer make: a square root and a reciprocal per sample do not belong in a Q16.16 datapath. The values are read with to_real() at the port boundary, the whole statistic is computed inreal, and only the result is quantized with to_fx().⚠ There is no way to declare a
realscratch variable in a VHDL block body -- acc/acc2 are Fx and iacc is an integer -- so the mean and the variance are INLINED into each output expression rather than accumulated. That is why the VHDL/Verilog/SystemVerilog bodies below build their text out ofmeanExprandvarExprstrings instead of emitting statements.
Sample results#
Plotted: vector — Sine Wave, [3,1]: amplitudes 1/2/3 at 2 rad/s (tried only because every scalar stimulus was refused)
Category dynamic · sample time 0.1 · 60 steps · commit c01902987 · produced by docsSample --out <folder> --steps 60 · data docs/generated/samples/Machine_Learning__Neural_Networks__Layer_Normalization.json · the SVG is generated from those numbers by tools/docs/plot_svg.py, so it is a run and not a drawing (R-D10).