Trainer module

In this module, the ensemble is trained on the whole available dataset. This is convenient for active learning purposes since the points are expensive to obtain and the datasets are usually very small. The ensemble weights yielding the smallest loss are saved, which helps in reducing the oversampling over the existing training points. This can be further refined by expelling the worst performing ensemble members according to the training loss by adjusting num_expel_NNs parameter. The training of the ensemble in the context of active learning can be significantly accelerated in each iteration by invoking warm starting process.

num_epochs (int)

Maximum number of training epochs when starting from newly initialized weights. Each epoch processes the whole training dataset once, in batches of batch_size_train.

Default: 800

num_expel_NNs (int)

The ensemble members are sorted according to the final training loss. According to that loss, the num_expel_NNs worst performing ones are removed.

Default: 0

learning_rate (float | None)

Initial step size for training from newly initialized weights. Larger values can speed up learning but may cause unstable training. If None, a rate is selected automatically from the network’s sensitivity to weight changes on the training data, scaled by learning_rate_safety. The rate can decrease during training when the loss stops improving.

Default: None

num_epochs_warm (int)

Maximum number of epochs when reusing weights from the preceding training run (warm_start=True). Usually fewer epochs are needed because the network already fits most of the data.

Default: 200

learning_rate_warm (float | None)

Initial training step size when warm_start is enabled. If None, a rate is selected automatically after applying shrink and perturb, using the same procedure as for learning_rate. The rate can decrease during training when the loss stops improving.

Default: None

warm_start (bool)

Reuse weights from the preceding training run to reduce the training effort as new observations arrive. Before training, the weights are multiplied by shrink and a random perturbation controlled by perturb is added. This allows the network to adapt to new data. Fresh training is performed when no trained weights are available and periodically according to cold_start_interval.

Default: False

shrink (float)

Multiplier applied to the reused weights before a warm start. A value of one retains their magnitude; a value below one shrinks them. Used together with perturb.

Default: 0.9

perturb (float)

Multiplier for freshly initialized random weights added to the reused weights before a warm start. Larger values change the starting network more strongly. Zero disables the random perturbation.

Default: 0.1

cold_start_interval (int)

Number of iterations after which cold (fresh) ensemble training is invoked.

Default: 20

min_loss (float)

Minimum value of average loss after which the training is stopped.

Default: 0.0

scale_grad_loss (float)

Hyperparameter used to scale the loss induced by the gradients with respect to the input parameters.

Default: 0.5

optimizer (str)

Algorithm used to minimize the training loss. Both choices adapt the step sizes for individual weights. Choose 'AdamW' to apply weight_decay regularization; 'Adam' trains without weight decay.

Default: 'Adam' Choices: 'AdamW', 'Adam'.

weight_decay (float | None)

Strength of regularization for optimizer='AdamW'. Weight decay shrinks network weights during training to limit overfitting; biases are excluded. Larger values give stronger shrinkage, and zero disables it. If None, the strength is chosen automatically from the supplied observation errors, noise estimated when detect_noise is enabled, and the kernel’s hidden-weight variance. This option has no effect with optimizer='Adam'.

Default: None

learning_rate_safety (float)

Multiplier for automatically selected learning rates. Smaller values give more cautious training steps, which can improve stability but slow learning. Must be greater than zero and at most one. Only used when the corresponding learning_rate or learning_rate_warm is None.

Default: 0.5

learning_rate_plateau_patience (int)

Number of epochs to tolerate without a relative loss improvement of at least 0.01% before reducing the learning rate by learning_rate_decay_factor. Uses the mean training loss across ensemble members. Must be nonnegative.

Default: 100

learning_rate_decay_factor (float)

Multiplier applied to the learning rate when the loss stops improving. For example, 0.5 halves the rate after learning_rate_plateau_patience is exceeded. Must be greater than zero and less than one.

Default: 0.5

minimum_learning_rate_factor (float)

Minimum learning rate as a fraction of the initial rate selected for the current training phase. For example, 0.001 allows reductions down to one thousandth of the initial rate. Must be greater than zero and at most one.

Default: 0.001

block_warm_training (bool)

When warm starting with full_data_trainer, first train only the output layer, keeping hidden layers fixed. If the average training loss reaches min_loss, no further training is needed; otherwise all layers are trained. This can reduce the effort of incorporating new observations. Requires warm_start=True and a positive head_epochs_warm. Has no effect with validation_trainer.

Default: True

head_epochs_warm (int)

Maximum number of epochs spent training only the output layer when block_warm_training is enabled, capped by num_epochs_warm. Set to zero to skip this phase. If further training is needed, all layers can then be trained for up to num_epochs_warm additional epochs.

Default: 15

loss_function (str)

Loss function which is minimized during ensemble training.

Default: 'MSE' Choices: 'MSE', 'L1'.

train_mean_grad (bool)

If True, the gradient averaged over all ensemble members is trained to match observed gradients. This leads to a loss in accuracy, because each ensemble member can have a gradient different from the observation. But it speeds up the training time significantly.

Default: True

batch_size_train (int | None)

Number of training points processed in each weight update. Smaller batches require less memory and give more updates per epoch. If None, the batch size targets about four updates per epoch, with at least two and at most 32 points per batch, capped by the dataset size. Large datasets therefore require more than four updates. Set to -1 to use all training data in one batch, or provide a positive batch size.

Default: None

save_best_train_weights (bool)

This flag enables saving the ensemble weights according to the best train loss. This can help oversampling issue in the course of optimization.

Default: True