Analytic neural network kernel

The analytic kernel describes how strongly neural-network outputs at two input points are expected to vary together before training. Inputs with strongly correlated outputs share more information when fitting the surrogate. The kernel is used to choose hyperparameters, such as the input length scales, which then guide the training of the ensemble.

The idea is to average over the possible weights of networks with the configured hidden layers and activation functions, in the limit of infinitely many neurons per hidden layer. If \(f(x)\) denotes the output of such a network, its kernel is the covariance \(k(x,x') = \mathbb{E}[(f(x)-\mathbb{E}[f(x)]) (f(x')-\mathbb{E}[f(x')])]\), where the expectation is over the weights. The trained ensemble itself still uses the finite layer sizes specified by hidden_layers_arch.

Input length scales control how rapidly the model can vary along each parameter dimension. The activation functions and weight distribution determine the shape of the correlations; these can depend on location in the parameter domain as well as on the distance between points.

min_ls_blocks_per_dim (float)

Minimum number of length-scale blocks for each parameter dimension. I.e. the length scale must be smaller than domain size divided by min_ls_blocks_per_dim.

Default: 2.0

max_ls_blocks (float)

Maximum number of length-scale blocks of design parameters after hyperparameter optimization. I.e. the hyper-prior goes to zero for smaller length scales.

Default: 100000000

target_ls_blocks (int)

Targeted number of length-scale blocks of design parameters, i.e. length-scale prior density tends to zero for larger length scales.

Default: 100

joint_ls_prior (bool)

If true, max_ls_blocks are penalized, otherwise effective max_blocks_1d are penalized for each parameter independently.

Default: True

scale_hidden_weights (bool)

Whether to optimize the spread of weights in the hidden layers after the input layer. A single variance multiplier is shared by these layers and also scales their biases, except for the phases of sine activations. This changes the range of inputs seen by the activation functions and hence the shape of the kernel. If false, the multiplier is fixed at one. Enabling this adds flexibility and increases hyperparameter optimization effort; it has no effect for a network with only one hidden layer.

Default: False

weight_degrees_of_freedom (float | None)

Controls how frequently unusually large weights occur in the Student-t weight distribution used by the kernel and to initialize the ensemble. The value must be greater than two. Smaller values give heavier tails, allowing more neurons with large weights; larger values approach a Gaussian distribution. The variance is kept fixed, so this parameter changes the shape of the distribution rather than its overall scale. The incoming weights of each neuron share a random scale, as does its bias for activations other than sine. Sine phases are independent. Set to None to fit the value by hyperparameter optimization within the interval [2.001, 16].

Default: 5.0