Incremental Regression Kernel Fit — Machine Learning/Incremental Learning

Incremental Regression Kernel Fit

Machine Learning / Incremental Learning

Fits a kernel regression model online, one observation per step, and puts the model on a bus every step: y ≈ z(x)·β + b, where z(x) is a fixed random expansion of the predictors into D features that approximates a Gaussian kernel. This is MATLAB's incrementalRegressionKernel learner with its default scale-invariant solver (Statistics and Machine Learning Toolbox), as Simulink's IncrementalRegressionKernel Fit block runs it: standardize x (when Standardize is on), expand it, z = (1/√(o·b))·[cos T, sin T] with T the Fastfood map of x (o the Hadamard order, b the number of blocks), then fit a linear model on z. The first Estimation Period observations are used only to estimate what the learner estimates (the predictors' means and standard deviations, and Epsilon); every later one updates β and b. Feed the bus to an Incremental Regression Kernel Predict block that carries the same feature map.

Ports

• x – one observation of the P predictors, a row [1,P]. P is read off this width.
• y – the response of that observation, [1,1]. A NaN in x or y skips the observation.
• mdl (bus) – the learner after this observation, a bus of seven elements: Beta [D,1], Bias [1,1], IsWarm (bool, true once Metrics Warmup Period observations have been trained on), CanPredict (bool, false only while standardization is still being estimated), Mu [P,1] and Sigma [P,1] (the standardization, 0 and 1 when there is none) and Epsilon [1,1] (0 for the leastsquares learner).

Parameters

• Learner – the loss the solver minimizes:
  • svm (the default) – the epsilon-insensitive loss of a support vector machine regression;
  • leastsquares – the squared error.
• Epsilon – half the width of the svm loss's insensitive band, 0.1 by default; not used by leastsquares.
• Estimate Epsilon – on (the default, MATLAB's 'auto'): Epsilon is re-estimated from the responses of the estimation period, as their interquartile range divided by 13.49 (0.1 when that is zero); off: Epsilon stays as set.
• Standardize – off (the default) or on: center and scale each predictor by its mean and standard deviation over the estimation period, before the expansion.
• Estimation Period – how many observations the estimation uses, 1000 by default as in MATLAB. It applies only when something is estimated (Standardize on, or the svm learner with Estimate Epsilon on); otherwise every observation trains.
• Metrics Warmup Period – how many trained observations make the model warm (IsWarm), 1000 by default.
• Num Expansion Dimensions – D, the number of expanded features and the length of Beta, the learner's NumExpansionDimensions; 16 by default, at most 256, and larger than P.
• Kernel Scale – the learner's KernelScale, a positive number; the map divides by it. 1 by default.
• Feature Map S, Feature Map G, Feature Map B and Feature Map P – the learner's random Fastfood map, four [b,o] matrices, one row per Hadamard block: o = 2^{ceil(log2 P)} and b = ceil(D/(2o)). Take them from s = toStruct(Mdl.Impl.FeatureMapper) as s.S, s.G, s.B and s.P (P one-based, each row a permutation of 1..o). The defaults are a real map MATLAB drew for 3 predictors and 16 features.
• Sampling Time (s) – zero or less inherits the solver's rate; a positive value runs the block at that period.

When a run is refused

An x that is not a row, a y that is not [1,1], a negative or fractional period, a negative Epsilon, Standardize on with an Estimation Period of 0, more than 8 predictors, or a feature map that does not fit P and D.

Code export

Six targets: Python, MATLAB, Java, Rust, C and C++, each carrying the learner's whole state and running the same step as the live block, in the same arithmetic order; the feature map is written out feature by feature, which is why D is capped. The hardware targets (VHDL, Verilog, SystemVerilog) and PLC Structured Text carry no bus, so an export to one stops and names the block.

Simulink bridge

None. Simulink's block takes its learner as InitialLearner, the name of an incrementalRegressionKernel object in the MATLAB workspace, rather than as dialog parameters, so there is nothing in its dialog to map these configs onto or read them back from.

Notes

• Stateful: the state is the solver's (four running sums per coefficient and the largest response seen), the estimation sums, and the counts. The feature map does not change.
• The bus at a step has already learnt that step's observation, as in Simulink: a Predict block fed the same x sees a model trained on it.
• Timing follows Simulink's block, not MATLAB's fit(): the estimated Mu, Sigma and Epsilon appear one step after the estimation period ends, and IsWarm turns true one step after fit() would set it.
• Kernel Scale is fixed: MATLAB's 'auto' estimates it with a random heuristic, which no other run could reproduce, so it is not offered. One observation per step; Simulink's optional weights and reset inputs are not offered.
• A NaN is skipped, where Simulink's block lets it in (measured on R2026a): there a NaN predictor turns Beta into NaN for the rest of the run, and a NaN response still advances the solver's running maxima; here either one skips the observation.
• Verified against R2026a, by the incremental_kernel_learners regression suite: driven for 120 steps with the inputs Simulink's block was run on, in four configurations (svm with Epsilon estimated; leastsquares standardized; svm with a fixed Epsilon; svm standardized with Epsilon estimated), with a response of 25 at step 40 that rescales the solver mid-run, every element of the bus agrees with what Simulink's block logged within 10⁻¹¹ at ten checkpoints around the estimation hand-over, and IsWarm and CanPredict at every step.