Trainer and validation module

Train an ensemble on a dataset while holding out some observations to check how well it predicts data it has not trained on. This is useful for a dataset supplied with study.add_many(). The split_ratio parameter specifies the fraction held out for validation; the remaining observations are used to update the network weights.

Validation loss guides early stopping and the selection of saved weights for each ensemble member. checkpoint_validation_tolerance allows a small increase in validation loss in exchange for a better fit to the training data. Optionally, num_expel_NNs removes members with the largest training loss after their saved weights have been restored.

num_epochs (int)

Maximum number of training epochs when starting from newly initialized weights. Each epoch processes the whole training dataset once, in batches of batch_size_train.

Default: 800

num_expel_NNs (int)

Number of members to remove from the trained ensemble. Each member’s saved weights are first selected using checkpoint_validation_tolerance. The members with the largest training loss at those saved weights are then removed. Zero keeps all members.

Default: 0

learning_rate (float | None)

Initial step size for training from newly initialized weights. Larger values can speed up learning but may cause unstable training. If None, a rate is selected automatically from the network’s sensitivity to weight changes on the training data, scaled by learning_rate_safety. The rate can decrease during training when the loss stops improving.

Default: None

num_epochs_warm (int)

Maximum number of epochs when reusing weights from the preceding training run (warm_start=True). Usually fewer epochs are needed because the network already fits most of the data.

Default: 200

learning_rate_warm (float | None)

Initial training step size when warm_start is enabled. If None, a rate is selected automatically after applying shrink and perturb, using the same procedure as for learning_rate. The rate can decrease during training when the loss stops improving.

Default: None

warm_start (bool)

Reuse weights from the preceding training run to reduce the training effort as new observations arrive. Before training, the weights are multiplied by shrink and a random perturbation controlled by perturb is added. This allows the network to adapt to new data. Fresh training is performed when no trained weights are available and periodically according to cold_start_interval.

Default: False

shrink (float)

Multiplier applied to the reused weights before a warm start. A value of one retains their magnitude; a value below one shrinks them. Used together with perturb.

Default: 0.9

perturb (float)

Multiplier for freshly initialized random weights added to the reused weights before a warm start. Larger values change the starting network more strongly. Zero disables the random perturbation.

Default: 0.1

cold_start_interval (int)

Number of iterations after which cold (fresh) ensemble training is invoked.

Default: 20

min_loss (float)

Minimum value of average loss after which the training is stopped.

Default: 0.0

scale_grad_loss (float)

Hyperparameter used to scale the loss induced by the gradients with respect to the input parameters.

Default: 0.5

optimizer (str)

Algorithm used to minimize the training loss. Both choices adapt the step sizes for individual weights. Choose 'AdamW' to apply weight_decay regularization; 'Adam' trains without weight decay.

Default: 'Adam' Choices: 'AdamW', 'Adam'.

weight_decay (float | None)

Strength of regularization for optimizer='AdamW'. Weight decay shrinks network weights during training to limit overfitting; biases are excluded. Larger values give stronger shrinkage, and zero disables it. If None, the strength is chosen automatically from the supplied observation errors, noise estimated when detect_noise is enabled, and the kernel’s hidden-weight variance. This option has no effect with optimizer='Adam'.

Default: None

learning_rate_safety (float)

Multiplier for automatically selected learning rates. Smaller values give more cautious training steps, which can improve stability but slow learning. Must be greater than zero and at most one. Only used when the corresponding learning_rate or learning_rate_warm is None.

Default: 0.25

learning_rate_plateau_patience (int)

Number of epochs to tolerate without a relative loss improvement of at least 0.01% before reducing the learning rate by learning_rate_decay_factor. Uses the mean training loss across ensemble members. Must be nonnegative.

Default: 100

learning_rate_decay_factor (float)

Multiplier applied to the learning rate when the loss stops improving. For example, 0.5 halves the rate after learning_rate_plateau_patience is exceeded. Must be greater than zero and less than one.

Default: 0.5

minimum_learning_rate_factor (float)

Minimum learning rate as a fraction of the initial rate selected for the current training phase. For example, 0.001 allows reductions down to one thousandth of the initial rate. Must be greater than zero and at most one.

Default: 0.001

block_warm_training (bool)

When warm starting with full_data_trainer, first train only the output layer, keeping hidden layers fixed. If the average training loss reaches min_loss, no further training is needed; otherwise all layers are trained. This can reduce the effort of incorporating new observations. Requires warm_start=True and a positive head_epochs_warm. Has no effect with validation_trainer.

Default: True

head_epochs_warm (int)

Maximum number of epochs spent training only the output layer when block_warm_training is enabled, capped by num_epochs_warm. Set to zero to skip this phase. If further training is needed, all layers can then be trained for up to num_epochs_warm additional epochs.

Default: 15

loss_function (str)

Loss function which is minimized during ensemble training.

Default: 'MSE' Choices: 'MSE', 'L1'.

train_mean_grad (bool)

If True, the gradient averaged over all ensemble members is trained to match observed gradients. This leads to a loss in accuracy, because each ensemble member can have a gradient different from the observation. But it speeds up the training time significantly.

Default: True

batch_size_train (int | None)

Number of training points processed in each weight update. Smaller batches require less memory and give more updates per epoch. If None, the batch size targets about four updates per epoch, with at least two and at most 32 points per batch, capped by the dataset size. Large datasets therefore require more than four updates. Set to -1 to use all training data in one batch, or provide a positive batch size.

Default: None

split_ratio (float)

Fraction of the whole dataset held out for validation. For example, 0.15 reserves 15% for validation and uses the remaining 85% for training. Must be greater than zero and less than one. If the dataset is too small to leave at least one observation in each subset, all data is used for training without validation.

Default: 0.15

batch_size_test (int | None)

Number of validation points processed at once. Smaller batches reduce memory use. Set to None or -1 to process the entire validation subset in one batch, or provide a positive batch size.

Default: None

split_strategy (str)

How validation points are selected. With 'shared', all ensemble members use the same training and validation split. With 'per_member', each ensemble member gets its own validation split so that no fixed subset is held out from the whole ensemble.

Default: 'shared' Choices: 'shared', 'per_member'.

early_stopping_patience (int | None)

Stop training after this many epochs without a significant improvement of the mean validation loss. Set to None to disable early stopping.

Default: 200

checkpoint_validation_tolerance (float)

Relative tolerance on validation loss when selecting the saved weights for each member. For example, 0.01 prefers weights with lower training loss while keeping validation loss within 1% of that member’s lowest validation loss so far. Zero selects weights strictly by validation loss. Must be nonnegative and less than one.

Default: 0.01