Layer Normalization — Machine Learning/Neural Networks

Machine_Learning/Neural_Networks/Layer_Normalization · 1 input / 1 output port(s) at insert · exports to Python, MATLAB, Java, Rust, C, C++, VHDL, Verilog, SystemVerilog, PLC Structured Text

Description#

The block's own DESCRIPTION_HTML, rendered verbatim — the same text the config dialog's info panel and the library navigator show. Fix a wrong sentence in the block's .cpp (R-D9), never here.

Layer Normalization

Machine Learning / Neural Networks

Normalizes the input vector against its own mean and variance, then applies a learned per-element scale and offset:

mean = Σu ÷ m, var = Σ(u − mean)² ÷ m, y_i = (u_i − mean) ÷ √(var + ε) × gamma_i + beta_i

This is torch.nn.LayerNorm / Keras LayerNormalization at inference. It is what stabilizes a transformer or a deep MLP without any dependence on a batch: the statistics come from the sample in front of it, so the block behaves identically whether it sees one sample or a million.

The variance divisor is m, not m−1 – the biased estimator, which is what both frameworks use here. At m = 3 the unbiased one would differ by 22 %, so this is not a rounding detail.

Ports

u – the layer input, a column [m,1]. m must equal the height of Gamma and Beta, and it is checked rather than broadcast. The whole vector is reduced, so every element affects every output.
Output – y, the same [m,1] column: the normalization never changes the signal's shape.

Parameters

Gamma – the learned scale, an [m,1] column (layer.weight). Use a column of ones for elementwise_affine=False.
Beta – the learned offset, an [m,1] column (layer.bias). Use a zero column for elementwise_affine=False.
Epsilon – added to the variance before the square root, so a constant input vector (var = 0) produces zeros rather than a division by zero. Default 1e-5, matching PyTorch; it must match the value the layer was trained with.
Sampling Time (s) – zero or less inherits the solver's rate; a positive value runs the block at that period.

Code export

All ten targets: Python, MATLAB, Java, Rust, C, C++, VHDL, Verilog, SystemVerilog and PLC Structured Text. Gamma, Beta and Epsilon are baked in as literals at full setprecision(17); the mean, the variance and the square root are computed at run time, because they belong to the sample.

The three HDL targets are simulation-only: they carry the statistic in real arithmetic and quantize only at the port boundary. A per-sample square root and reciprocal have no place in a Q16.16 datapath. This is the opposite of the sibling Batch Normalization block, whose frozen statistics fold to a multiply-add and which is therefore genuinely synthesizable.

Simulink bridge

None. Simulink's Deep Learning blocks take a trained network object, not a set of arrays, and no config value can carry an object across the bridge. The bridge reports the block rather than dropping it silently, and it has no parity testbench, which is the documented consequence of Support::None rather than a gap. Code export verification still covers it across all ten languages.

Notes

Algebraic and stateless: the output depends only on the current input, so the layer cannot break an algebraic loop.
Nonlinear, and deliberately carries no state space. Dividing by a square root of the input's own variance is not an affine map in u, so no A/B/C/D describes it and model reduction correctly refuses the block.
A constant input vector normalizes to zeros (before beta): every element equals the mean, so the numerator vanishes and epsilon alone keeps the denominator finite. That is the defined behaviour, not a degenerate case.

Code facts#

Fact	Value
registered type	`Machine_Learning/Neural_Networks/Layer_Normalization`
family	`Machine_Learning/Neural_Networks`
solver environment class	`ICoreBlock_0_Machine_Learning_1_Neural_Networks_2_Layer_Normalization`
source	`src/ICoreSDK/ICoreBlockLibrary/Blocks/Machine_Learning/Neural_Networks/Layer_Normalization/ICoreBlock_0_Machine_Learning_1_Neural_Networks_2_Layer_Normalization.cpp`
header	`src/ICoreSDK/ICoreBlockLibrary/Blocks/Machine_Learning/Neural_Networks/Layer_Normalization/ICoreBlock_0_Machine_Learning_1_Neural_Networks_2_Layer_Normalization.h`
default size on canvas	120 × 80 px
ports at insert	1 in, 1 out
code generators implemented	`Python`, `MATLAB`, `Java`, `Rust`, `C`, `C++`, `VHDL`, `Verilog`, `SystemVerilog`, `PLC Structured Text`

Ports#

#	Direction	Signal type	Description label
1	in	`ICoreDouble`	u
2	out	`ICoreDouble`	y

Ports the constructor creates. A block whose port list changes with its configuration adds or removes ports at load time; the count above is the one a freshly inserted block has.

Configuration variables#

Config variable	Default	Simulink parameter
`Gamma`	`[1.2; 0.8; 1.5]`	—
`Beta`	`[0.05; -0.1; 0.3]`	—
`Epsilon`	`0.00001`	—

Every block also carries Sampling Time (s) from ICoreBlockSolverEnvironment: zero or less inherits the solver's rate, a positive value runs the block at that period.

Simulink bridge#


support	`Support::None`
Simulink path	—
port-count rule	`PortsParam::None`
`SampleTime` parameter	yes

Caveat (shown to the user): no Simulink equivalent that could carry the learned arrays: its Deep Learning blocks take a trained network OBJECT rather than gamma/beta columns, and no config value crosses the bridge as an object. Re-create the layer on the Simulink side and paste the same gamma, beta and epsilon in

Catalog contract: src/ICoreSDK/ICoreCoder/ICoreCommandSystem/SimulinkBridge/ICoreSimulinkBlockCatalog.h

Description vs code#

The checker has a blind spot here — it could not resolve something (a grouped port bullet, a computed config name), which is reported and never counted as a pass. A reader has to settle it:

B0 every stimulus in the sample errored — cross-checks skipped

The verdict above is tools/docs/check_block_descriptions.py (P7.1), which compares LISTS. It cannot read a sentence: "stateless" on a block with a state, an initial-value semantic the recursion does not implement, a "not synthesizable" caveat the HDL banner contradicts. That is the agent audit (P7.3) on BLOCK_DESCRIPTION_AUDIT.md, and this tool's green is not a substitute for one.

The top comment of the block's .cpp — the maths, the realization and the export strategy, addressed to whoever changes it. It must not contradict the description above (P7.5).

Layer Normalization — statistics taken from the CURRENT sample, every step mean = sum(u)/m, var = sum((u-mean)^2)/m, y_i = (u_i-mean)/sqrt(var+eps)*g_i + b_i

Nothing here folds. Batch_Normalization's four arrays are frozen, so its square root is evaluated once at export time; this block's mean and variance belong to the sample in front of it, so all ten backends carry the reduction and the root.

The three HDL targets are therefore SIMULATION-ONLY real arithmetic, the same choice Recursive IIR and Dense_Layer make: a square root and a reciprocal per sample do not belong in a Q16.16 datapath. The values are read with to_real() at the port boundary, the whole statistic is computed in real, and only the result is quantized with to_fx().

⚠ There is no way to declare a real scratch variable in a VHDL block body -- acc/acc2 are Fx and iacc is an integer -- so the mean and the variance are INLINED into each output expression rather than accumulated. That is why the VHDL/Verilog/SystemVerilog bodies below build their text out of meanExpr and varExpr strings instead of emitting statements.

Sample results#

No stimulus produced a sampled output in this rig — Invalid input size at: ICore Blocks/Home/Layer Normalization. That is a fact about the single-block rig, not a verdict on the block: an offline batch fit, a block whose output only appears at onSolverFinish, or one that needs a driven environment cannot be exercised alone.

Category unsampled · sample time 0.1 · 60 steps · commit ccf005c8 · produced by docsSample --out <folder> --steps 60

Sample data: docs/generated/samples/Machine_Learning__Neural_Networks__Layer_Normalization.json

Layer Normalization — Machine Learning/Neural Networks

Description#