.. _benchmark:

Benchmarking different studies against each other
==========================================================================================


:Drivers: :ref:`BayesianOptimization`, :ref:`ActiveLearning`, :ref:`CMAES`, :ref:`DifferentialEvolution`, :ref:`ScipyMinimizer`


**Download script**: :download:`benchmark.m`


In some cases, it is not clear, which driver or which driver configuration is able to minimize an objective function with the smallest number of iterations. If similar minimizations are performed on a regular basis, it can be worthwhile to find the best choice by benchmarking several studies with different drivers or configurations against each other.

As an example, the usual 2D `Rastrigin function <https://en.wikipedia.org/wiki/Rastrigin_function>`_ on a circular domain shall be minimized:

.. math::

   &\text{min.}\,& f(x_1,x_2) = 2\cdot10 + \sum_{i=1,2} \left(x_i^2 - 10\cos(2\pi x_i)\right)
   
   &\text{s.t.}\,& \sqrt{x_1^2 + x_2^2} \leq 1.5.

We compare the heuristic minimization methods :ref:`CMAES` and :ref:`DifferentialEvolution`, the multi-start local minimization method :ref:`ScipyMinimizer`, and Bayesian optimization. For the latter, we use the standard :ref:`BayesianOptimization` driver and the :ref:`ActiveLearning` driver that is configured to use Neural Networks instead of a Gaussian process to learn the objective function.

We assume that we can evaluate the function 2 times in parallel and that each evaluation takes 2 seconds. Hence, the driver has to compute new samples in less than a second in order to avoid significant overhead. This is an edge case for the Bayesian optimization approach, that has a considerable overhead typically larger than a second. I this case the other methods with much faster sample computation times can be more appropriate. Nevertheless, in this case of a multi-modal objective Bayesian optimization reaches the global minim still well before the other optimization methods. Gaussian process regression is computational efficient for this small number of iterations. In comparison the Neural Network training time of the `ActiveLearning` driver leads to a larger overhead.



.. literalinclude:: ./benchmark.m
   :language: matlab
   :linenos:


 

.. figure:: images/benchmark/benchmark.svg
   :alt: Benchmark results

   **Left:** The left graph shows the cummulative minimum as a function of the number
   of evaluations of the objective function. The results are averaged over six
   independent runs of each driver. The shaded area shows the uncertainty of the mean
   value. Clearly, the ``BayesianOptimization`` driver performs the best and finds
   the global minimum zero after about 70 iterations while the other methods did not
   converge to zero even after 250 iterations.
   **Right:** The convergence of the cummulative minimum as a function of the total
   optimization time looks slightly different. As expected, the ``ActiveLearning``
   and ``BayesianOptimization`` drivers show an overhead for computing new samples.
   If one would have an optimization budget on less than 50 seconds, the multi-start
   local ``ScipyMinimizer`` performs better. It can be also seen that the training of
   Neural Networks leads to a larger overhead compared to the Gaussian process
   regression used by the ``BayesianOptimization`` driver.