.. _ValidationTrainer:


Trainer and validation module
-----------------------------


Train an ensemble on a dataset while holding out some observations to check how well it predicts data it has not trained on. This is useful for a dataset supplied with ``study.add_many()``. The ``split_ratio`` parameter specifies the fraction held out for validation; the remaining observations are used to update the network weights.  

Validation loss guides early stopping and the selection of saved weights for each ensemble member. ``checkpoint_validation_tolerance`` allows a small increase in validation loss in exchange for a better fit to the training data. Optionally, ``num_expel_NNs`` removes members with the largest training loss after their saved weights have been restored.

.. _ActiveLearning.NN.validation_trainer.num_epochs:


num_epochs (int)
""""""""""""""""
   Maximum number of training epochs when starting from newly initialized weights. Each epoch processes the whole training dataset once, in batches of ``batch_size_train``. 

   Default: ``800``

.. _ActiveLearning.NN.validation_trainer.num_expel_NNs:


num_expel_NNs (int)
"""""""""""""""""""
   Number of members to remove from the trained ensemble. Each member's saved weights are first selected using ``checkpoint_validation_tolerance``. The members with the largest training loss at those saved weights are then removed. Zero keeps all members. 

   Default: ``0``

.. _ActiveLearning.NN.validation_trainer.learning_rate:


learning_rate (float | None)
""""""""""""""""""""""""""""
   Initial step size for training from newly initialized weights. Larger values can speed up learning but may cause unstable training. If ``None``, a rate is selected automatically from the network's sensitivity to weight changes on the training data, scaled by ``learning_rate_safety``. The rate can decrease during training when the loss stops improving. 

   Default: ``None``

.. _ActiveLearning.NN.validation_trainer.num_epochs_warm:


num_epochs_warm (int)
"""""""""""""""""""""
   Maximum number of epochs when reusing weights from the preceding training run (``warm_start=True``). Usually fewer epochs are needed because the network already fits most of the data. 

   Default: ``200``

.. _ActiveLearning.NN.validation_trainer.learning_rate_warm:


learning_rate_warm (float | None)
"""""""""""""""""""""""""""""""""
   Initial training step size when ``warm_start`` is enabled. If ``None``, a rate is selected automatically after applying ``shrink`` and ``perturb``, using the same procedure as for ``learning_rate``. The rate can decrease during training when the loss stops improving. 

   Default: ``None``

.. _ActiveLearning.NN.validation_trainer.warm_start:


warm_start (bool)
"""""""""""""""""
   Reuse weights from the preceding training run to reduce the training effort as new observations arrive. Before training, the weights are multiplied by ``shrink`` and a random perturbation controlled by ``perturb`` is added. This allows the network to adapt to new data. Fresh training is performed when no trained weights are available and periodically according to ``cold_start_interval``. 

   Default: ``false``

.. _ActiveLearning.NN.validation_trainer.shrink:


shrink (float)
""""""""""""""
   Multiplier applied to the reused weights before a warm start. A value of one retains their magnitude; a value below one shrinks them. Used together with ``perturb``. 

   Default: ``0.9``

.. _ActiveLearning.NN.validation_trainer.perturb:


perturb (float)
"""""""""""""""
   Multiplier for freshly initialized random weights added to the reused weights before a warm start. Larger values change the starting network more strongly. Zero disables the random perturbation. 

   Default: ``0.1``

.. _ActiveLearning.NN.validation_trainer.cold_start_interval:


cold_start_interval (int)
"""""""""""""""""""""""""
   Number of iterations after which cold (fresh) ensemble training is invoked. 

   Default: ``20``

.. _ActiveLearning.NN.validation_trainer.min_loss:


min_loss (float)
""""""""""""""""
   Minimum value of average loss after which the training is stopped. 

   Default: ``0.0``

.. _ActiveLearning.NN.validation_trainer.scale_grad_loss:


scale_grad_loss (float)
"""""""""""""""""""""""
   Hyperparameter used to scale the loss induced by the gradients with respect to the input parameters. 

   Default: ``0.5``

.. _ActiveLearning.NN.validation_trainer.optimizer:


optimizer (str)
"""""""""""""""
   Algorithm used to minimize the training loss. Both choices adapt the step sizes for individual weights. Choose ``'AdamW'`` to apply ``weight_decay`` regularization; ``'Adam'`` trains without weight decay. 

   Default: ``"Adam"``
   Choices: ``"AdamW"``, ``"Adam"``.

.. _ActiveLearning.NN.validation_trainer.weight_decay:


weight_decay (float | None)
"""""""""""""""""""""""""""
   Strength of regularization for ``optimizer='AdamW'``. Weight decay shrinks network weights during training to limit overfitting; biases are excluded. Larger values give stronger shrinkage, and zero disables it. If ``None``, the strength is chosen automatically from the supplied observation errors, noise estimated when ``detect_noise`` is enabled, and the kernel's hidden-weight variance. This option has no effect with ``optimizer='Adam'``. 

   Default: ``None``

.. _ActiveLearning.NN.validation_trainer.learning_rate_safety:


learning_rate_safety (float)
""""""""""""""""""""""""""""
   Multiplier for automatically selected learning rates. Smaller values give more cautious training steps, which can improve stability but slow learning. Must be greater than zero and at most one. Only used when the corresponding ``learning_rate`` or ``learning_rate_warm`` is ``None``. 

   Default: ``0.25``

.. _ActiveLearning.NN.validation_trainer.learning_rate_plateau_patience:


learning_rate_plateau_patience (int)
""""""""""""""""""""""""""""""""""""
   Number of epochs to tolerate without a relative loss improvement of at least 0.01% before reducing the learning rate by ``learning_rate_decay_factor``. Uses the mean training loss across ensemble members. Must be nonnegative. 

   Default: ``100``

.. _ActiveLearning.NN.validation_trainer.learning_rate_decay_factor:


learning_rate_decay_factor (float)
""""""""""""""""""""""""""""""""""
   Multiplier applied to the learning rate when the loss stops improving. For example, 0.5 halves the rate after ``learning_rate_plateau_patience`` is exceeded. Must be greater than zero and less than one. 

   Default: ``0.5``

.. _ActiveLearning.NN.validation_trainer.minimum_learning_rate_factor:


minimum_learning_rate_factor (float)
""""""""""""""""""""""""""""""""""""
   Minimum learning rate as a fraction of the initial rate selected for the current training phase. For example, 0.001 allows reductions down to one thousandth of the initial rate. Must be greater than zero and at most one. 

   Default: ``0.001``

.. _ActiveLearning.NN.validation_trainer.block_warm_training:


block_warm_training (bool)
""""""""""""""""""""""""""
   When warm starting with ``full_data_trainer``, first train only the output layer, keeping hidden layers fixed. If the average training loss reaches ``min_loss``, no further training is needed; otherwise all layers are trained. This can reduce the effort of incorporating new observations. Requires ``warm_start=True`` and a positive ``head_epochs_warm``. Has no effect with ``validation_trainer``. 

   Default: ``true``

.. _ActiveLearning.NN.validation_trainer.head_epochs_warm:


head_epochs_warm (int)
""""""""""""""""""""""
   Maximum number of epochs spent training only the output layer when ``block_warm_training`` is enabled, capped by ``num_epochs_warm``. Set to zero to skip this phase. If further training is needed, all layers can then be trained for up to ``num_epochs_warm`` additional epochs. 

   Default: ``15``

.. _ActiveLearning.NN.validation_trainer.loss_function:


loss_function (str)
"""""""""""""""""""
   Loss function which is minimized during ensemble training. 

   Default: ``"MSE"``
   Choices: ``"MSE"``, ``"L1"``.

.. _ActiveLearning.NN.validation_trainer.train_mean_grad:


train_mean_grad (bool)
""""""""""""""""""""""
   If True, the gradient averaged over all ensemble members is trained to match observed gradients. This leads to a loss in accuracy, because each ensemble member can have a gradient different from the observation. But it speeds up the training time significantly. 

   Default: ``true``

.. _ActiveLearning.NN.validation_trainer.batch_size_train:


batch_size_train (int | None)
"""""""""""""""""""""""""""""
   Number of training points processed in each weight update. Smaller batches require less memory and give more updates per epoch. If ``None``, the batch size targets about four updates per epoch, with at least two and at most 32 points per batch, capped by the dataset size. Large datasets therefore require more than four updates. Set to ``-1`` to use all training data in one batch, or provide a positive batch size. 

   Default: ``None``

.. _ActiveLearning.NN.validation_trainer.split_ratio:


split_ratio (float)
"""""""""""""""""""
   Fraction of the whole dataset held out for validation. For example, 0.15 reserves 15% for validation and uses the remaining 85% for training. Must be greater than zero and less than one. If the dataset is too small to leave at least one observation in each subset, all data is used for training without validation. 

   Default: ``0.15``

.. _ActiveLearning.NN.validation_trainer.batch_size_test:


batch_size_test (int | None)
""""""""""""""""""""""""""""
   Number of validation points processed at once. Smaller batches reduce memory use. Set to ``None`` or ``-1`` to process the entire validation subset in one batch, or provide a positive batch size. 

   Default: ``None``

.. _ActiveLearning.NN.validation_trainer.split_strategy:


split_strategy (str)
""""""""""""""""""""
   How validation points are selected. With ``'shared'``, all ensemble members use the same training and validation split. With ``'per_member'``, each ensemble member gets its own validation split so that no fixed subset is held out from the whole ensemble. 

   Default: ``"shared"``
   Choices: ``"shared"``, ``"per_member"``.

.. _ActiveLearning.NN.validation_trainer.early_stopping_patience:


early_stopping_patience (int | None)
""""""""""""""""""""""""""""""""""""
   Stop training after this many epochs without a significant improvement of the mean validation loss. Set to ``None`` to disable early stopping. 

   Default: ``200``

.. _ActiveLearning.NN.validation_trainer.checkpoint_validation_tolerance:


checkpoint_validation_tolerance (float)
"""""""""""""""""""""""""""""""""""""""
   Relative tolerance on validation loss when selecting the saved weights for each member. For example, 0.01 prefers weights with lower training loss while keeping validation loss within 1% of that member's lowest validation loss so far. Zero selects weights strictly by validation loss. Must be nonnegative and less than one. 

   Default: ``0.01``