Batch Normalization — Machine Learning/Neural Networks

Batch Normalization

Machine Learning / Neural Networks

Batch normalization evaluated at inference, elementwise over the input vector:

y_i = (u_i − mean_i) ÷ √(var_i + ε) × gamma_i + beta_i

At training time the statistics are taken from the batch and move on every step; at inference they are frozen – running_mean/running_var in a trained torch.nn.BatchNorm1d, moving_mean/ moving_variance in Keras – so all four arrays are constants you paste in. This block is the inference form only; it never estimates anything from the signal.

Because the four arrays are constants, the layer is exactly a per-element affine map, and the block folds it to one once per configuration: scale_i = gamma_i ÷ √(var_i + ε) and shift_i = beta_i − mean_i · scale_i, leaving y_i = scale_i·u_i + shift_i.

Ports

• u – the layer input, a column [m,1]. m must equal the height of the four arrays, and it is checked rather than broadcast.
• Output – y, the same [m,1] column: the map is elementwise, so it never changes the signal's shape.

Parameters

• Running Mean – mean, an [m,1] column, one entry per channel. layer.running_mean in PyTorch.
• Running Variance – var, an [m,1] column of variances, not standard deviations. layer.running_var in PyTorch. Every entry must be ≥ 0.
• Gamma – the learned scale, an [m,1] column (layer.weight). Use a column of ones for a layer trained with affine=False.
• Beta – the learned offset, an [m,1] column (layer.bias). Use a zero column for affine=False.
• Epsilon – the numerical guard added to the variance before the square root. Default 1e-5, matching PyTorch and Keras; it must match the value the layer was trained with, since the fold bakes it in.
• Sampling Time (s) – zero or less inherits the solver's rate; a positive value runs the block at that period.

Code export

All ten targets: Python, MATLAB, Java, Rust, C, C++, VHDL, Verilog, SystemVerilog and PLC Structured Text. The folded scale and shift are baked in as literals at full setprecision(17), so the exported layer is bit-comparable with the in-app run; there is no tunable parameter, because a trained statistic is not something to retune on the target.

The three HDL targets are ordinary Q16.16 fixed point and are offered for synthesis, unlike the rest of this family. That follows from the fold: with the square root evaluated at export time the emitted datapath is one multiply and one add per element, with no transcendental anywhere in it.

Simulink bridge

None. Simulink's Deep Learning blocks take a trained network object, not a set of arrays, and no config value can carry an object across the bridge. The bridge reports the block rather than dropping it silently, and it has no parity testbench, which is the documented consequence of Support::None rather than a gap. Code export verification still covers it across all ten languages.

Notes

• Algebraic and stateless: the output depends only on the current input, so the layer cannot break an algebraic loop.
• No state space, deliberately. The shift makes the map affine rather than linear, and ICoreStateSpace's feed-through y = D·u has nowhere to put it. Model reduction reports the block as unmergeable, which is the honest answer.
• Distinct from Layer Normalization, which computes its mean and variance from the input vector on every step. Here they are frozen constants, which is why this block folds and that one cannot.