Neural Network Ensemble

Ensembles of neural networks provide a prediction from each ensemble member allowing to compute statistical measures like mean and standard deviation of the predictions. In this way it can approximate Bayesian posteriors like those coming from Gaussian processes.

While for medium number of data points, the training of the ensemble is computationally more demanding than that of a Gaussian process, for large data sets the linear scaling of the training effort becomes beneficial since the effort of Gaussian processes scales at least quadratically with the number of data points.

Another advantage of neural networks over Gaussian processes is that they can learn more complex functions with non-stationary behaviour, i.e. different correlation lengths in different regions of the parameter space.

optimization_step_min (int)

Minimum number of observations of the objective before the hyperparameters are optimized.

Default: Automatic choice according to number of dimensions.

Note

Each derivative observation is counting as an independent observation.

optimization_step_max (int)

Maximum number of observations of the objective after which no more hyperparameter optimization is performed.

Default: 300

Note

Each derivative observation is counting as an independent observation.

min_optimization_interval (int)

Minimum number of observations of the objective after which the hyperparameters are optimized.

Default: 2

Note

Each derivative observation is counting as an independent observation.

max_optimization_interval (int)

Maximum number of observations of the objective after which the hyperparameters are optimized.

Default: 20

Note

Each derivative observation is counting as an independent observation.

optimization_level (float)

Controls how often the hyper-parameters are optimized. Small values (e.g. 0.01) lead to more frequent optimizations. Large values (e.g. 1) lead to less frequent optimizations.

Default: 0.2

num_samples_hyperparameters (int)

Number of local searches for optimal hyperparameters.

Default: Automatic choice min(15, max(5, 2 * num_dim)) according to number of dimensions num_dim.

min_dist (float)

To speed up the hyperparameter optimization the surrogate model can be sparsified such that data points with a distance (in terms of the length scales of the surrogate) below min_dist are neglected.

Default: 0.0

name (str)

The name of the surrogate model.

Default: "loss"

output_dim (int)

Dimensionality of the output of the model, i.e. dimensionality of data added to observations.

Default: 1

output_names (cell{str})

Allows to assign names to each of the output_dim outputs of the surrogate. The length of output_names must be equal to output_dim. By specifying output names they can be accessed in variables as {output_names[0]}, {output_names[1]}, etc.

Default:

By default, the variables can be accessed as {name}0, {name}1, etc. where {name} is the name of the surrogate.

horizon (int)

Specifies the number N of the lastest observations which are used for surrogate model training. This decreases the computational effort of training at the cost of a decreased prediction accuracy.

Default: Infinite horizon.

transformation (str)

Specifies the name of the transformation applied to the parameter space.

Default: No transformation.

detect_noise (bool)

If true, the noise of the function values and function value derivatives are estimated by means of two hyperparameters.

Default: false

mean_function (str)

The name of the method to parameterize the mean function \(\mu(x^*)\) of the surrogate model. The constant method models \(\mu(x^*)\) by a constant which can be determined by an analytic expression. The linear method models \(\mu(x^*)\) by a linear function with \(1+D\) coefficients. The expansion coefficients are determined by a maximum likelihood estimate. Likewise, the quadratic method expands the mean function up to quadratic order with \(1+D+D^2\) coefficients. In order to scale better with the number of dimensions \(D\), the semiquadratic method neglects mixed terms in the quadratic expansion leading to only \(1+2D\) expansion coefficients.

Using a mean function other than constant increases the computational effort of hyperparameter optimization but can also lead to better predictions of the surrogate model.

Default: "constant" Choices: "constant", "linear", "semiquadratic", "quadratic".

warping_function (str)

The name of the warping function \(w\). A warping function performs a transformation \(y \to w(y,\mathbf{b})\) of any function value \(y = f(x)\) by means of a strongly monotonic increasing function \(w(y,\mathbf{b})\) that depends on a set of hyperparameters \(\mathbf{b}\). The hyperparameters are chosen automatically by a maximum likelihood estimate. The choice identity leads to no warping of the function values while sinh uses a sinus hyperbolicus function for warping. Using sinh can result in better predictions at the cost of increasing the computational effort. It should be therefore only applied for more expensive black box functions.

Default: "identity" Choices: "identity", "sinh".

joint_hyperparameters (bool)

If false, mean functions and warping functions learn their hyperparameters for each output of the surrogate model separately. This increases the overall predictive power of the surrogate model, but it also requires more input data and larger computational effort to determine all hyper parameters accurately.

Default: true

correlate_outputs (bool)

If true, a global covariance matrix between all outputs of a multi-output surrogate model is determined and used for the predictions. In cases, where some outputs are linearly dependent, this approach fails since the the covariance matrix becomes ill-conditioned.

Default: false

tolerance (float)

Numerical tolerance parameter. Used as additional term in diagonal entries of covariance matrices to ensure positive definitness.

Default: 1e-20

tolerance_hypercov (float)

Numerical tolerance parameter for covariance matrix hyperparameter between multiple outputs of Gaussian process.

Default: 1e-08

covariance_matrix (struct)

Default: struct('resize_step', 200, 'iterative_start_size', 500, 'cuda_start_size', 2000, 'num_iterative_solver_steps', 0, 'online_start_size', 500, 'max_data_hyper_derivs', 1000)

The object holds and processes information about the covariance \(\left(\Sigma\right)_{i j})\) between all data points \(\mathbf{x}_1, \ldots, \mathbf{x}_N\). One important purpose is to determine the Cholesky decomposition of the covariance matrix, \(\Sigma=L L^T\) which is used to make predictions or optimize hyperparameters. The decomposition takes usually \(\mathcal{O}(N^3)\) steps. By using the Cholesky decomposition of a previous iteration, an iterative update can be computed in only \(\mathcal{O}(N^2)\) steps.

See matrix configuration for details.

num_NNs (int)

Number of neural networks in the deep ensemble. The bigger the number, the better the approximation of the underlying probability distribution related to the final objective.

Default: 10

hidden_layers_arch (cell{int})

The architecture of the hidden layers of each ensemble member. The number of neurons in each hidden layer is specified by an entry in the corresponding list. The architecture for all the ensemble members has to be the same.

Default: {200,200}

activation_fun (str | cell{str})

Activation function or activation schedule for hidden layers. A string uses the same activation for every hidden layer. A list must have one entry per hidden layer, for example [“sine”, “tanh”, “tanh”]. The output layer is always linear. Supported activations are [‘sine’, ‘tanh’, ‘erf’, ‘sigmoid’, ‘relu’, ‘combined’, ‘rational’, ‘gelu’].

Default: "sine"

seed (bool)

Seed flag is used to enforce the same initial weight distribution over the whole deep ensemble. Used for the purpose of reproducibility.

Default: false

trainer (struct)

The training of the network can be either performed on all available data points (full_data_trainer) or on a subset of points while the best configuration is determined based on a set of validation points (validation_trainer).

Default: struct('type', "full_data_trainer", 'num_epochs', 800, 'num_expel_NNs', 0, 'learning_rate', None, 'num_epochs_warm', 200, 'learning_rate_warm', None, 'warm_start', false, 'shrink', 0.9, 'perturb', 0.1, 'cold_start_interval', 20, 'min_loss', 0.0, 'scale_grad_loss', 0.5, 'optimizer', "Adam", 'weight_decay', None, 'learning_rate_safety', 0.5, 'learning_rate_plateau_patience', 100, 'learning_rate_decay_factor', 0.5, 'minimum_learning_rate_factor', 0.001, 'block_warm_training', true, 'head_epochs_warm', 15, 'loss_function', "MSE", 'train_mean_grad', true, 'batch_size_train', None, 'save_best_train_weights', true) The struct entry ‘type’ specifies the type of the module. The remaining entries specify its properties.

Trainer module (type "full_data_trainer"): In this module, the ensemble is trained on the whole available dataset. This is convenient for active learning purposes since the points are expensive to obtain and the datasets are usually very small. The ensemble weights yielding the smallest loss are saved, which helps in reducing the oversampling over the existing training points. This can be further refined by expelling the worst performing ensemble members according to the training loss by adjusting num_expel_NNs parameter. The training of the ensemble in the context of active learning can be significantly accelerated in each iteration by invoking warm starting process.

See full_data_trainer configuration for details.

Trainer and validation module (type "validation_trainer"): Train an ensemble on a dataset while holding out some observations to check how well it predicts data it has not trained on. This is useful for a dataset supplied with study.add_many(). The split_ratio parameter specifies the fraction held out for validation; the remaining observations are used to update the network weights.

Validation loss guides early stopping and the selection of saved weights for each ensemble member. checkpoint_validation_tolerance allows a small increase in validation loss in exchange for a better fit to the training data. Optionally, num_expel_NNs removes members with the largest training loss after their saved weights have been restored.

See validation_trainer configuration for details.

kernel (struct)

Covariance model used to fit input length scales and other hyperparameters that guide ensemble training. The analytic kernel describes the correlations implied by the network’s hidden layers, activation functions, and weight distribution.

Default: struct('min_ls_blocks_per_dim', 2.0, 'max_ls_blocks', 100000000, 'target_ls_blocks', 100, 'joint_ls_prior', true, 'scale_hidden_weights', false, 'weight_degrees_of_freedom', 5.0)

The analytic kernel describes how strongly neural-network outputs at two input points are expected to vary together before training. Inputs with strongly correlated outputs share more information when fitting the surrogate. The kernel is used to choose hyperparameters, such as the input length scales, which then guide the training of the ensemble.

The idea is to average over the possible weights of networks with the configured hidden layers and activation functions, in the limit of infinitely many neurons per hidden layer. If \(f(x)\) denotes the output of such a network, its kernel is the covariance \(k(x,x') = \mathbb{E}[(f(x)-\mathbb{E}[f(x)]) (f(x')-\mathbb{E}[f(x')])]\), where the expectation is over the weights. The trained ensemble itself still uses the finite layer sizes specified by hidden_layers_arch.

Input length scales control how rapidly the model can vary along each parameter dimension. The activation functions and weight distribution determine the shape of the correlations; these can depend on location in the parameter domain as well as on the distance between points.

See analytic configuration for details.