Interface for Matlab

The following section contains the documentation for using the Python interface that allows to set up, run studies and analyze their results.

Overview of Matlab classes

The interface consists of the following Python classes

Class	Purpose
`Server`	Class for for starting and stopping the optimization server.
`Client`	Class for connecting to server, creating studies, and creating benchmarks.
`Study`	Class for configuring, running and analyzing results of a study
`Observation`	Container for observations to be sent to the server.
`Suggestion`	Suggestion retrieved by a study to be evaluated.
`Driver`	Interface to driver-specific methods.
`Benchmark`	Class for benchmarking different studies against each other

Documentation

Benchmark

class Benchmark(host, verbose, id, num_average)

This class provides methods for benchmarking different optimization studies against each other Example:

benchmark = Benchmark('num_average',6);
benchmark.add_study(study1);
benchmark.add_study(study2);
studies = benchmark.studies();
for i_study = 1:length(studies)
    study = studies{i_study};
    while(not(study.is_done))
        suggestion = study.get_suggestion();
        observation = evaluate(suggestion.sample);
        study.add_observation(observation,suggestion.id);
     end
     benchmark.add_study_results(study);
end
data = benchmark.get_data('x_type','num_evaluations','y_type','objective',...
              'average_type','mean');
plots = [];
for ii=1:length(data.names)
     X = data.X(ii,:);
     Y = data.Y(ii,:);
     plots(end+1) = plot(X,Y,'LineWidth',1.5);
end
legend(plots, data.names{:});

add_study

Benchmark.add_study(study)
Adds a study to the benchmark. Example:
benchmark.add_study(study1);
Parameters:

study (Study) – A Study object for which a benchmark should be run.

add_study_results

Benchmark.add_study_results(study)
Adds the results of a benchmark study at the end of an optimization run. Example:
benchmark.add_study_results(study1);
Parameters:

study (Study) – A Study object after the study was run.

get_data

Benchmark.get_data(varargin)
Get benchmark data. Example:
data = benchmark.get_data('x_type','num_evaluations','y_type','objective',...
              'average_type','mean');
plots = [];
for ii=1:length(data.names)
     X = data.X(ii,:);
     Y = data.Y(ii,:);
     plots(end+1) = plot(X,Y,'LineWidth',1.5);
end
legend(plots, data.names{:});
Parameters:

x_type (str) – Data on x-axis. Can be either ‘num_evaluations’ or ‘time’. The time data is given in units of seconds. Options are ‘num_evaluations’, ‘time’. Default is ‘num_evaluations’.

y_type (str) – Data type on y-axis. Options are ‘objective’, ‘distance’, (i.e. accumulated minimum distance off all samples to overall minimum), or ‘min_distance’ (i.e. distance of current minimum to overall minimum). Default is ‘objective’.

average_type (str) – Type of averaging over study runs. Options are ‘mean’ w.r.t. x-axis data or ‘median’ w.r.t. y-axis data. Default is ‘mean’.

invert (logical) – If true, the objective is multiplied by -1. (Parameter not available for distance average types). Defaults to false.

log_scale (logical) – If true, the output of Y and stdev are determined as mean and standard deviations of the natural logarithm of the considered y_type. Defaults to false.

minimum (array) – Vector with minimum position. (Only available for distance average types)

scales (array) – Vector with positive weights for scaling distance in different directions. (Only available for distance average types)

norm (int) – Order of distance norm as defined in numpy.linalg.norm. (Only available for distance average types) Defaults to 2.

num_samples (int) – Number of samples on y-axis. (Only available for median average type or time on x-axis). Defaults to 100.

run

Benchmark.run()
Run the benchmark after the evaluator has been set (see Benchmark.set_evaluator()). Example:
benchmark.set_evaluator(evaluate);
benchmark.run()

set_evaluator

Benchmark.set_evaluator(evaluator)
Set the function that maps design parameters to an Observation. Example:
function observation = evaluate(study, sample)
    observation = study.new_observation();
    observation.add(sample.x1^2 + sample.x2^2);

benchmark.set_evaluator(evaluate);
benchmark.run();
Note

Call this function only after all studies have been added to the benchmark.

Parameters:

evaluator (function_handle) – Function handle for a function of the variable parameters that returns a corresponding Observation object. The function must accept a "study" argument as well as a struct argument which maps each design parameter and fixed environment parameter to its value.

studies

Benchmark.studies()

Returns: cell: A list of studies to be run for the benchmark.

Client

class Client(host, verbose, check)

This class provides methods for creating new optimization studies. Example:

client = Client("http://localhost:4554")
design_space = {...
    struct('name', 'x1', 'type', 'continuous', 'domain', [-1.5, 1.5]), ...
    struct('name', 'x2', 'type', 'continuous', 'domain', [-1.5, 1.5]) ...
};

study = client.create_study('design_space', design_space, 'name', 'basic example');

Parameters:

host (string) – The host name under which the optimization server can be contacted. For example, 'http://localhost:4554'
verbose (logical) – If true, messages from the optimization server are printed out.
check (logical) – If true, check if the optimization server can be contacted via the given host name.

check_server

Client.check_server()
Checks if the optimization server is running. Example:
>> client.check_server()
Polling server at http://localhost:4554
Optimization server is running

create_benchmark

Client.create_benchmark(varargin)
Creates a new Benchmark object for benchmarking different optimization studies against each other. Example:
benchmark = client.create_benchmark('num_average', 6);
Parameters:

benchmark_id (string) – A unique identifier of the benchmark.

num_average (int) – Number of study runs to determine average study performance. Default is 6.

remove_completed_studies (logical) – If true, studies that are completed (i.e. some stopping criterion was met) are removed from the server as long as no other client holds a handle to the study. Default is false.

create_study

Client.create_study(varargin)
Creates a new Study instance. Example:
study = client.create_study( ...
    'design_space', design_space, ...
    'name', 'example', ...
    'study_id', 'example_01' ...
);
Parameters:
design_space (cell) –
List of domain definitions for each parameter. A design space definition consists of a list of dictionary with the entries

name:

Name of the parameter. E.g. 'x1'. The name should contain no spaces and must not be equal to function names or mathematical constants like 'sin', 'cos', 'exp', 'pi' etc.

type:

Type of the parameter. Either 'continuous', 'discrete', or 'categorial'.

domain:

The domain of the parameter. For continuous parameters this is a tuple (min, max). For discrete parameters this is a list of values, e.g. [1.0,2.5,3.0]. For categorial inputs it is a list of strings, e.g. ["cat1","cat2","cat3"]. Note, that categorial values are internally mapped to integer representations, which are allowed to have a correlation. The categorial values should therefore be ordered according to their similarity. For fixed parameters the domain is a single parameter value.

Example:
design_space = { ...
    struct('name', 'x1', 'type', 'continuous', 'domain', [-2.0,2.0]), ...
    struct('name', 'x2', 'type', 'continuous', 'domain', [-2.0,2.0]), ...
    struct('name', 'x3', 'type', 'discrete', 'domain', [-1.0,0.0,1.0]), ...
    struct('name', 'x4', 'type', 'categorial', 'domain', ["a", "b"]) ...
};
If not specified, the design space configuration from the study history is used. If no historic information exists, an error is raised.
environment (cell) –
Environment parameters are those which influence the behavior of the system, but are not design parameters. Examples are uncontrollable environment parameters (e.g. temperature, time) or parameters that are scanned in each evaluation (e.g. wavelength, angle) for each run of the system can be described by environment parameters. Alternatively, scans can be described by surrogates that are trained on multiple inputs (one for each scan value). The environment definition consists of a list of dictionary with the entries:

name:

Name of the parameter. E.g. 'x1'. The name should contain no spaces and must not be equal to function names or mathematical constants like 'sin', 'cos', 'exp', 'pi' etc.

type:

Type of the parameter. Either 'variable' or 'fixed'. Fixed parameters can be used in the constraint functions or other expressions.

domain:

The domain of the parameter. For fixed parameters, this is a single value, for variable parameters this can be a tuple (min, max). If no bounds are specified for variable parameters, the environment is considered to be unconstrained. In this case the scale has to be set.

scale:

The scale at which environment parameter changes affect the system. If the environment parameter describes unknown drifts and aging effects, the length scale is equal to the timescale at which the system behavior changes due to drifts or aging.

Example:
environment = { ...
    struct('name', 'wavelength', 'type', 'continuous', 'domain', [300, 600]), ...
    struct('name', 'time', 'type', 'continuous', 'scale', 0.1), ...
    struct('name', 'radius', 'type', 'fixed', 'domain', 1.5) ...
};
If not specified, the environment configuration from the study history is used. If no historic information exists, the environment is assumed to be empty (environment = {}).
constraints (cell) –
List of constraints on the design space. Each list element is a dictionary with the entries

name:

Name of the constraint.

constraint:

A string defining an inequality constraint in the for a <= b or a >= b. The following operations and functions may be for example used: +,-,*,/,^,sqrt,sin,cos,tan,exp,log,log10,abs,round,sign,trunc. E.g. 'x1^2 + x2^2 <= sin(x1+x2)'. For a more complete list of supported functions, see the expression variable reference.

Example:
constraints = { ...
    struct('name', 'circle', 'expression', 'x1^2 + x2^2 <= radius^2'), ...
    struct('name', 'triangle', 'expression', 'x1 >= x2') ...
};
If not specified, the constraints configuration from the study history is used. If no historic information exists, the list of constraints is assumed to be empty (constraints = {}).
study_id (string) – A unique identifier of the study. All relevant information on the study are saved in a file named study_id+’.jcmo’ If the study already exists, the design_space, environment, and constraints do not need to be provided. If not set, the study_id is set to a random unique string.

name (string) – The name of the study that will be shown in the dashboard.

save_dir (string) – The path to a directory, where the study file (jcmo-file) is saved. Default is a directory in the system’s temporary directory.

driver (string) – Driver used for the study. Default is ‘ActiveLearning’. For a list of drivers, see the driver reference.

output_precision (double) –
Precision level for output of parameters.

Note

Rounding the output can potentially lead to a slight breaking of constraints.

dashboard (logical) – If true, a dashboard will be served for the study. Default is true.

open_browser (logical) – If true, a browser window with the dashboard is started. Default is true.

shutdown_server

Client.shutdown_server(force)
Shuts down the optimization server. Example:
client.shutdown_server(true)
Parameters:

force (logical) – If true, the optimization server is closed even if a study is not yet finished. Defaults to false.

Driver

class Driver(host, verbose, study_id)

This class provides methods for retrieving and setting driver-specific information of the study. Depending on the chosen driver of Client.create_study() the driver provides specific methods. More details for each driver are available in the driver reference. Example:

driver = study.driver;
min_objective_values = driver.historic_parameter_values( ...
    "acquisition_function.min_objective");

The constructor should not be used directly since it does not create a driver on the server side. Instead, one should use Study.driver.

describe

Driver.describe()
Get description of all modules and their parameters that are used by the driver. Example:
description = driver.describe();
fprintf(description.members.surrogates{1});
Returns:

A nested struct with description of submodules consisting
of a name and a descriptive text. If the entry describes a module, it has an additional "members" entry with struct describing submodules and parameters.

Return type:

struct

get_state

Driver.get_state(path)
Get state of the driver. Example:
best_sample = driver.get_state("best_sample");
Parameters:

path (str) – A dot-separated path to a submodule or parameter. If none, the full state is returned.

Returns:

If path is None, a struct with information of driver state.

Return type:

struct

Note

A description of the meaning of each entry in the state can be retrieved by describe().

historic_parameter_values

Driver.historic_parameter_values(path)
Get the values of an internal parameter for each iteration of the study. Example:
min_objective_values = driver.historic_parameter_values(...
    "acquisition_function.min_objective");
Parameters:

path (str) – A dot-separated path to the parameter.

Returns:

Array of parameter values

Return type:

array

Note

A description of the meaning of each parameter can be retrieved by describe().

Observation

class Observation: This class provides a container to collect data to be sent to the optimizer. This includes scalar or vectorial inputs for the objective or surrogate models of an active learning-based driver.

add

Observation.add(value, varargin)
Add data to observation. Example:
obs.add(val);
obs.add(dval_dx1, 'derivative', 'x1');
obs.add(dval_dx2, 'derivative', 'x2');
Parameters:

value (double) – Numerical value or list of values.

derivative (str) – If the values are derivatives w.r.t. a design or environment parameter, this should be the name of the parameter.

uncertainty (double) – The uncertainty of the provided values, i.e. the estimated standard deviation of multiple observations for same parameter.

model_name (str) – Name of the surrogate model that is trained on the data. Can be omitted for drivers that do not support multiple models.

environment_value (cell) – Cell array of environment value for all entries of the environment of the study. Optionally, entries for different environment values can be added consecutively, such that also scans of environment values can be collected into one observation.

Server

class Server(varargin)

This class provides methods for starting and stopping a self-hosted optimization server on the local computer. Example:

server = Server();
client = Client(server.host);
study = client.create_study(...);

Parameters:

jcm_optimizer_path (str) – The path of the JCMoptimizer installation. If not specified, is is assumed that the Matlab file is that of the JCMoptimizer installation.
port (int) – The port that the optimization server is listening to. If not specified, the port is chosen automatically.
persistent (logical) –
If true, the server continues to run even after the Python script has finished. To shutdown a local server later on, one can reconnect to it:
```
client = Client(host="http://localhost:4554");
client.shutdown_server();
```

host

The host name of the server.

Type:: str

pid

The process id of the server.

Type:: str

port

The port that the server is listening on.

Type:: str

shutdown

Server.shutdown(force)
Shuts down the optimization server. Example:
server.shutdown(true);
Parameters:

force (logical) – If true, the optimization server is closed even if a study is not yet finished. Defaults to false.

Study

class Study(host, verbose, id, driver_name)

This class provides methods for controlling a numerical optimization study. Example:

study.configure('max_iter', 80);

function observation = evaluate(study, sample)
    observation = study.new_observation();
    observation.add(sample.x1^2 + sample.x2^2);

%Start the optimization loop
study.set_evaluator(@evaluate)
study.run()

%Alternatively, one can explicitly define an optimization loop
while(not(study.is_done))
    sug = study.get_suggestion();
    obs = evaluate(study, sug.sample);
    study.add_observation(obs, sug.id);
end

driver

of the corresponding driver.

Type:: Driver

add_many

Study.add_many(observations, design_values, varargin)
Adds many observations to the study. Example:
study.add_many(observations, design_values, ...
               'acquisition_times', acquisition_times);
Parameters:

observations (cell) – List of Observation objects for each sample (see new_observation())

design_values (cell) – List of design values. E.g. {{0.1, 1.0, 'cat1'}; {0.2, 2.0, 'cat2'}}

environment_values (cell) – Optional list of environment values. If not specified, the last known environment values are taken. E.g. {{1.0, 2.0}; {1.1, 2.3}}

acquisition_times (cell) – Optional list of times required to acquire each observation. E.g. {0.11, 0.12}

check (logical) – If true, the validity of the observation is checked

Warning

The purpose of this function is to speed up the process of adding many observations to the study. To this end, the intermediate driver states are not computed. This means that all driver-specific historic data (any path of historic_parameter_values() starting with driver…) is incorrect. The same holds for most of the data shown on the dashboard. To avoid this, one has to add the observations one by one using add_observation().

add_observation

Study.add_observation(observation, suggestion_id, varargin)
Adds an observation to the study. Example:
% Add an observation for a given suggestion
study.add_observation(observation, suggestion.id);
% Add an observation for a given design value
study.add_observation(observation, NaN, 'design_value', {1.0, 2.0});
Parameters:

observation (Observation) – Observation object with added values (see new_observation())

suggestion_id (int) – Id of the corresponding suggestion if it exists, otherwise NaN.

design_value (cell) – If the observation does not belong to an open suggestion, the corresponding design value must be provided as a list of floats for continuous and discrete parameters or string for categorial parameters. E.g. {0.1, 2.0, 'cat1'}.

environment_value (cell) – If an environment parameters are specified, this specifies the value of variable environment parameters as a list of floats that is valid for all values added to the observation. E.g. {1.0, 2.0}. Alternatively, one can also set different environment values for each entry of the observation (see add()).

acquisition_time (double) – If the observation does not belong to an open suggestion, it is possible to specify the time it took to retrieve the observation (e.g. the computation time). This information can be used to adapt the effort of computing the next suggestions.

check (logical) – If true, the validity of the observation is checked

add_observation_data

Study.add_observation_data(data)
Add data from another study to the study. Example:
obs_data = study.get_observation_data();
other_study.add_observation_data(obs_data);
Parameters:

data (struct) – Struct with observation data. See get_observation_data() for the details.

clear_all_suggestions

Study.clear_all_suggestions()
Clear all open suggestions. Example:
study.clear_all_suggestions();
Note

The study only creates num_parallel suggestions (see configure()) until it waits for an observation to be added (see add_observation()) or a suggestion to be cleared.

clear_suggestion

Study.clear_suggestion(sug_id, msg)
If the evaluation for a certain suggestion fails, the suggestion can be cleared from the study. Example:
study.clear_suggestion(suggestion.id, 'Computation failed');
Note

The study only creates num_parallel suggestions (see configure()) until it waits for an observation to be added (see add_observation()) or a suggestion to be cleared.

Parameters:

suggestion_id (int) – Id of the suggestion to be cleared.

message (str) – An optional message that is printed out.

configuration

Study.configuration()
Return the current configuration for the driver. Example:
config = study.configuration();
study2.configure(config{:});
Returns:

Nested struct with configuration.

Return type:

struct

configure

Study.configure(varargin)
Configures the study for its run. Example:
study.configure('max_iter', 100);
Parameters:

max_iter (int) – Maximum number of evaluations of the evaluator 2 function (default: inf).

max_time (double) – Maximum optimization time in seconds (default: inf).

num_parallel (int) – Number of parallel observations of the evaluator function (default: 1).

Note

The full list of parameters depends on the chosen driver. For a parameter description, see the corresponding driver in the driver reference.

describe

Study.describe()
Get description of all modules and their parameters that are used by the study. Example:
description = study.describe();
fprintf(description.observation.acquisition_time);
fprintf(description.driver.members.surrogates{1});
Returns:

A nested struct with the root field names:

driver:

Nested struct with description of submodules consisting of a name and a descriptive text. If the entry describes a module, it has an additional "members" entry with structs describing submodules and parameters.

observation:

Struct with a description of the parameters of an observation.

suggestion:

Struct with a description of the parameters of a suggestion of the driver.

Return type:

struct

get_observation_data

Study.get_observation_data()
Get table with data of the observations. This can be used to copy the data manually to another study. Example:
obs_data = study.get_observation_data();
other_study.add_observation_data(obs_data);
Returns:

Struct where each entry holds an equally long cell array of observation data.

The field names of the struct are:

value:

Observed value of black-box function

derivative:

Name of derivative parameter or None for non-derivative observation

uncertainty:

Uncertainty of observed value or None if no uncertainty

model_name:

Name of the surrogate model that is trained on the data or None

design_value:

Value of design parameters

environment_value:

Value of environment parameters or None if no environment is specified

Return type:

struct

get_state

Study.get_state(path)
Get state of the study. Example:
acquisition_time = study.get_state('observation.acquisition_time');
Parameters:

path (str) – A dot-separated path to a submodule or parameter. If none, the full state is returned.

Returns:

If path is None, a dictionary with the following entries
is returned:

driver:

Dictionary with information of driver state.

observation:

Dictionary with information of the latest observation.

suggestion:

Dictionary with information about the suggestion that corresponds to the last observation

Return type:

struct

Note

A description of the meaning of each entry in the state can be retrieved by describe().

get_suggestion

Study.get_suggestion(environment_value)
Get a new suggestion to be evaluated by the user. Example:
sug = study.get_suggestion();
val = objective(sug.sample.x1, sug.sample.x2);
obs = study.new_observation();
obs = obs.add(val);
study.add_observation(obs, sug.id);
Parameters:

environment_value (array) – If an environment is specified, this optional argument specifies the list of variable environment parameter values, for which a suggestion should be computed. E.g. [0.1, 1.2]. If an environment exists and no values are specified, the last known environment values are used.

Warning

The function has to wait until the number of open suggestions is smaller than num_parallel before receiving a new suggestion. This can cause a deadlock if no observation is added by an independent thread.

historic_parameter_values

Study.historic_parameter_values(path)
Get the values of an internal parameter for each iteration of the study. Example:
acquisition_times = study.historic_parameter_values(...
    'observation.acquisition_time');
Parameters:

path (str) – A dot-separated path to the parameter.

Note

A description of the meaning of each parameter can be retrieved by describe().

is_done

Study.is_done()
Checks if the study has finished. Example:
if study.is_done(): break; end;
Returns:

True if some stopping criterion set by configure() was met.

Return type:

logical

Note

Before returning true, the function call waits until all open suggestions have been added to the study.

new_observation

Study.new_observation()
Create a new Observation object that allows to add data via add(). Example:
observation = study.new_observation();
observation.add(1.2);
observation.add(0.1, 'derivative', 'x1')

run

Study.run()
Run the acquisition loop after the evaluator has been set (see set_evaluator()). The acquisition loop stops after a stopping criterion has been met (see configure()). Example:
study.run();
Warning

The automatic study run is only supported for num_parallel = 1. For parallel evaluations of the evaluator, a custom acquisition loop has to be implemented.

set_evaluator

Study.set_evaluator(evaluator)
Set the function that maps design parameters to an Observation. Example:
function observation = evaluate(study, sample)
    observation = study.new_observation();
    observation.add(sample.x1^2 + sample.x2^2);

study.set_evaluator(@evaluate)
Parameters:

evaluator (function_handle) – Function handle for a function of the variable parameters that returns a corresponding Observation object. The function must accept a "study" argument as well as a struct argument which maps each design parameter and fixed environment parameter to its value.

start_clock

Study.start_clock()
The optimization stops after the time max_time (see configure()). This function resets the clock to zero. Example:
study.start_clock();
Note

The clock is also set to zero by calling configure().

open_jobs

Study.open_jobs(job_ids, suggestion)
Collects all ids of open jobs belonging to a suggestion. Example:
study.open_jobs(job_ids, suggestion);
Parameters:

job_ids (array) – List of job ids as returned from jcmwave_solve in daemon mode.

suggestion (Suggestion) – A Suggestion object.

gather_results

Study.gather_results()
Waits for open jobs to finish until all the jobs from one of the open suggestion are completed. Example:
[sug_results, sug_id] = study.gather_results();
val = objective(sug_results);
obs = study.new_observation();
obs.add(val);
study.add_observation(obs, sug_id);
Returns:

Cell array of results from the corresponding jobs. sug_id (int): Id of the corresponding suggestion. sug_logs (cell): Cell array of logging messages from the corresponding jobs

Return type:

sug_results (cell)

Suggestion

class Suggestion(id_, sample_)

This class provides the sample to be evaluated and the id which is required to identify results for this suggestion. Example:

sug = study.get_suggestion();
val = objective(sug.sample.x1, sug.sample.x2);
obs = study.new_observation();
obs.add(val);
study.add_observation(obs, sug.id);

The constructor should not be used directly. Instead, one creates suggestions using Study.get_suggestion().

id

The id of the suggestion.

Type:: int

sample

Struct which maps names of design parameters to their values for which an observation is suggested.

Type:: struct