Activation Function — Machine Learning/Neural Networks

Activation Function

Machine Learning / Neural Networks

Applies one elementwise nonlinearity to every entry of the input, with no change of shape. This is the activation as its own node, for a graph that places it between layers rather than inside them – a residual branch that adds before activating, layers exported unactivated from the framework, or simply a network you would rather read off the canvas.

The Dense Layer block carries four activations internally; use those when the activation belongs to the layer, and this block when it does not.

Ports

• u – the input, any size [r,c]. The function is applied to each entry independently.
• Output – y, the same [r,c]: an elementwise map never changes the signal's dimensions.

Parameters

• Activation – which function to apply:
  • ReLU – max(0, x). The default, and the usual hidden-layer choice.
  • LeakyReLU – x for x > 0, Leaky Slope·x otherwise; keeps a small gradient on the negative side.
  • ELU – x for x > 0, ELU Alpha·(e^x − 1) otherwise; saturates smoothly at −alpha instead of at 0.
  • Tanh – squashes to (−1, 1).
  • Sigmoid – 1 ÷ (1 + e^−x), squashes to (0, 1).
  • Softplus – ln(1 + e^x), a smooth ReLU.
  • SiLU – x · sigmoid(x), also called swish.
  • GELU – 0.5·x·(1 + tanh(√(2/π)·(x + 0.044715·x³))). See the Notes: this is the tanh approximation, deliberately.
• Leaky Slope – the negative-side slope used by LeakyReLU only, and ignored by every other setting. Default 0.01, PyTorch's negative_slope.
• ELU Alpha – the saturation level used by ELU only, and ignored by every other setting. Default 1.0, PyTorch's alpha.
• Sampling Time (s) – zero or less inherits the solver's rate; a positive value runs the block at that period.

Code export

All ten targets: Python, MATLAB, Java, Rust, C, C++, VHDL, Verilog, SystemVerilog and PLC Structured Text. The chosen activation is baked into the body at export time – it is structural, not something to retune on the target – and so are the two slope constants.

The three HDL targets are simulation-only: they carry the function in real arithmetic and quantize only at the port boundary. Six of the eight activations are transcendental and have no place in a Q16.16 datapath; ReLU and LeakyReLU alone would be synthesizable, and a block whose synthesizability changed with a combo value would be worse than one that is honest about the whole of itself.

PLC Structured Text has no TANH, so Tanh and GELU are emitted through 2 ÷ (1 + EXP(−2z)) − 1, which is the same function written in terms of EXP alone.

Simulink bridge

None. Simulink has no single elementwise-activation block to map to: its Deep Learning blocks take a trained network object, and the nearest primitives (Saturation, Math Function, Trigonometric Function) each cover one of these eight at best and none of the others, so no ParamRule could carry the combo across. The bridge reports the block rather than dropping it silently, and it has no parity testbench, which is the documented consequence of Support::None rather than a gap. Code export verification still covers it across all ten languages.

Notes

• Algebraic and stateless: the output depends only on the current input, so the block cannot break an algebraic loop.
• Nonlinear, and deliberately carries no state space – every one of the eight is nonlinear, so no A/B/C/D describes it and model reduction correctly refuses the block.
• GELU is the tanh approximation (nn.GELU(approximate='tanh')), not the exact 0.5·x·(1 + erf(x/√2)). This is a portability decision: erf exists in none of VHDL, Verilog, SystemVerilog or IEC 61131-3, so the exact form could not be exported to four of the ten targets at all. The two differ by up to about 1e-3 of the output; if your trained network used the exact form and that matters, place the difference in your error budget rather than assuming these agree.
• Every activation is written in an overflow-safe form – the exponent argument is never positive in ELU or Softplus – so a large input saturates rather than producing an infinity or a NaN.