Code Documentation (API)

Here we give detailed code documentation of the classes, constructors, and functions within MÆSTRO. Here is an index of all the code documentation presented in this page.

Surrogate Model API

ModelConstruction (maestro/model.py)

class maestro.ModelConstruction(state, mc_run_folder=None)

Bases: object

Construct and interface with all possible surrogate models

__init__(state, mc_run_folder=None)

Initialize the surrogate model object

Parameters

state (maestro.Settings) – algortithm state containing algorithm, configuration and other parameters and state information
mc_run_folder (str) – MC run folder path

static get_model_object(function_str, data)

Get surrogate model object based on previously calculated model parameters

Parameters

function_str (str) – function string for the specific data name
data (Any) – previously calculated model parameters. The data type of this argument should make sence to the the class that this data structure is passed to

Returns

model object

Return type

Any

calculate_minimum_tr_radius_bound(data_name_index=0)

Calculate minimum TR radius bound

Parameters: data_name_index (int) – data name index (defaults to 0)
Returns: minimum TR radius bound depending on the type of model
Return type: float

construct_models(): Contruct the surrogate model. This function will read the function specified in the configuration input:model:function_str and call that function from within this class

appr_pa_m_construct(data_name)

Contruct apprentice.PolynomialApproximation model for order of numerator m

Parameters: data_name (str) – data/term name
Returns: polynomial approximation surrogate model object
Return type: apprentice.PolynomialApproximation

appr_ra_m_n_construct(data_name)

Contruct apprentice.RationalApproximation model for order of numerator m and denominator n

Parameters: data_name (str) – data/term name
Returns: rational approximation surrogate model object
Return type: apprentice.RationalApproximation

appr_ra_m_1_construct(data_name)

Contruct apprentice.RationalApproximation model for order of numerator m and denominator 1. This is a special case of the the model construction function appr_ra_m_n_construct

Parameters: data_name (str) – data/term name
Returns: rational approximation surrogate model object
Return type: apprentice.RationalApproximation

appr_appx_construct(data_name)

Contruct apprentice.PolynomialApproximation or apprentice.RationalApproximation model.

Parameters: data_name (str) – data/term name
Returns: polynomail or rational approximation surrogate model object
Return type: apprentice.RationalApproximation or apprentice.PolynomailApproximation

Function Structure API

Fstructure (maestro/fstructure.py)

class maestro.Fstructure(state)

Bases: object

Construct and interface with all possible objective function structures

__init__(state)

Initialize the function structure object

Parameters: state (maestro.Settings) – algortithm state containing algorithm, configuration and other parameters and state information

static projection(X, MIN, MAX)

Project gradient onto a box

Parameters

X (list) – graient to be projected
MIN (list) – minimum bounds of the box
MAX (list) – maximum bounds of the box

Returns

projected gradient

Return type

list

check_close_to_optimal_conditions(): Helper function to check whether the current trust region center is close to optimal. If projected gradient is small enough at maximum fidelity or fixed fidelity is in used, then we claim that we are close to minmum without the ability to improve. Otherwise, if projected gradient is small enough, then the trust region radius is halved to try to certify the minimum since we are dealing with a noisy function

solve_f_structure_subproblem(f_structure_subproblem_result_file)

Solve the subproblem within the trust region by calling minimize on the function structure function handle referrenced in the configuration input:f_structure:function_str

Parameters: f_structure_subproblem_result_file (str) – subproblem result file path in which an object with minimum and argmin of the subproblem optimization solution is stored

appr_tuning_objective(parameter=None, use_scaled=False, approx_iteration_minus_no=0)

Construct apprentice.appset.TuningObjective2 objective function with MC standard deviation values

Parameters

parameter (list) – (optional) if parameter is provided, then the tuning objective recurrence is set to that parameter (default: None)
use_scaled (bool) – true is scaled parameters are to be used, false otherwise (default: False)
approx_iteration_minus_no (int) – number of previous iterations to deduct from the current iteration from which to use the model files

Returns

TuningObjective2 objective function object

Return type

apprentice.appset.TuningObjective2

appr_tuning_objective_without_error_vals(parameter=None, use_scaled=False, approx_iteration_minus_no=0)

Construct apprentice.appset.TuningObjective2 objective function without MC standard deviation values

Parameters

parameter (list) – (optional) if parameter is provided, then the tuning objective recurrence is set to that parameter (default: None)
use_scaled (bool) – true is scaled parameters are to be used, false otherwise (default: False)
approx_iteration_minus_no (int) – number of previous iterations to deduct from the current iteration from which to use the model files

Returns

TuningObjective2 objective function object

Return type

apprentice.appset.TuningObjective2

MC Task API

MCTask (maestro/mc/mctask.py)

class maestro.mc.MCTask(mc_working_directory, mc_parameters=None)

Bases: object

MC Task base class

__init__(mc_working_directory, mc_parameters=None)

Create an MC Task object

Parameters

mc_working_directory (str) – MC working directory path
mc_parameters (dict) – MC parameters set in the configuration input

run_mc(): Run the MC (abstract). This function needs to be implemented in a class that inherits this class

merge_statistics_and_get_max_sigma()

Merge MC output statistics and find the maximum standard deviation of the MC output (abstract). This function needs to be implemented in a class that inherits this class

Returns: class that inherits this class should return the maximum standard deviation of the MC output
Return type: float

convert_mc_output_to_df(all_param_directory)

Convert MC output to a pandas dataframe (abstract). This function needs to be implemented in a class that inherits this class

Example

Example of the returned pandas dataframe is given below. In the example below, there are three parameters and two terms of the objective function. Terms that end with .P are the parameters and those ending with .V are the values associated with either the MC, i.e., MC sample values or the DMC, i.e., MC standard deviation values. You can add more rows, i.e, more sets of parameter and values for additional terms in the objective function or more columns, i.e., more components of the each term of the objective that come from the MC simulator:

>df
                MC                      DMC
Term1.P        [[1,2],[3,4],[6,3]]     [[1,2],[3,4],[6,3]]
Term1.V        [19, 18, 17]            [99, 98, 97]
Term2.P        [[1,2],[3,4],[6,3]]     [[1,2],[3,4],[6,3]]
Term2.V        [29, 28, 27]            [89, 88, 87]

Parameters

all_param_directory (str) – MC outout directory path

Returns

pandas dataframe formatted MC output

Return type

pandas.DataFrame

check_and_resolve_nan_inf(data, all_param_directory, term_names, group_names=None)

Check for NaNs/\(\infty\) and try to resolve them.

If group_names is not None i.e., groups are specified, then try to interpolate NaNs/\(\infty\) for terms within each group for each parameter
If group_names is None i.e., groups are not specified or if interpolation does not completely resolve the issue, then:
- If the NaNs/\(\infty\) was encountered in the start parameter, then raise an exception
- If the NaNs/\(\infty\) was encountered in the iterate (subproblem argmin), then use the surrogate models from the current iteration to fill in the missing values
- If the NaNs/\(\infty\) was encountered in the adaptive sample, then drop the parameter from the corresponding term

Resolved NaNs/\(\infty\) are overwritten in the data argument (as data is passed as a reference to this function).

Example

Example of the data passed in as argument to this function is shown below. In the example below, there are three parameters and two terms of the objective function. Terms that end with .P are the parameters and those ending with .V are the values associated with either the MC, i.e., MC sample values or the DMC, i.e., MC standard deviation values:

>data

{
    "MC":{
            "Term1.P":[[1,2],[3,4],[6,3]],
            "Term1.V": [19, 18, 17],
            "Term2.P":[[1,2],[3,4],[6,3]],
            "Term2.V": [29, 28, 27]
        },
    "DMC":{
            "Term1.P":[[1,2],[3,4],[6,3]],
            "Term1.V": [99, 98, 97],
            "Term2.P":[[1,2],[3,4],[6,3]],
            "Term2.V": [89, 88, 87]
        }
}

Parameters

data (dict) – MC output data formatted into a dictionary
all_param_directory (str) – MC outout directory path
term_names (list) – term names e.g., [Term1, Term2]
group_names (list) – groups of terms (if any) (default: None). If this argument is None, then this function will not try to interpolate missing values between terms

static read_params_file(path)

Read parameters from parameters file.

Example

Example of the format of parameter file on the file path path is given below. The first column is the parameter name and the second column is the value associated with the parameter:

dimension1 0.2
dimension2 0.4

Parameters

path (str) – parameter file path

Returns

list of parameter values in the same order as found in the parameter file

Return type

list

get_param_from_directory(param_directory, fnamep='params.dat')

Get all parameters from parameter file with name fnamep in directory param_directory

Example

Example of the format of parameter file with file name fnamep is given below. The first column is the parameter name and the second column is the value associated with the parameter:

dimension1 0.2
dimension2 0.4

Parameters

param_directory (str) – path of the parameter directory
fnamep (str) – name of the parameter file

Returns

list of parameter values in the same order as found in the parameter file

Return type

list

get_param_from_metadata(param_metadata)

Get parameters from the metadata object

Parameters: param_metadata (dict) – parameter metadata object
Returns: list of parameter values
Return type: list

write_param(parameters, parameter_names, at_fidelities, run_fidelities, mc_run_folder, expected_folder_name, fnamep='params.dat', fnamerf='run_fidelity.dat', fnameaf='at_fidelity.dat')

Write parameters to parameter directory and generate parameter metadata

Parameters

parameters (list of lists) – list of parameters points
parameter_names (list) – names of the parameter dimensions
at_fidelities (list) – current fidelity of the parameters
run_fidelities (list) – expected fidelity of the parameters i.e., these are the fidelities at which the MC should be run at the corresponding parameters
mc_run_folder (str) – MC run folder path
expected_folder_name (str) – expected MC run folder path with the type of run (single or sample) and iteration number
fnamep (str) – name of the parameter file (default: params.dat)
fnamerf (str) – name of the run fidelity file (default: run_fidelity.dat)
fnameaf (str) – name of the at fidelity file (default: at_fidelity.dat)

Returns

parameter metadata object

Return type

dict

set_current_iterate_as_next_iterate(current_iterate_parameter_data, next_iterate_parameter_data=None, next_iterate_mc_directory=None)

Copy current iterate as the next iterate

Parameters

current_iterate_parameter_data (dict) – parameter metadata of the current iterate (from the current iteration
next_iterate_parameter_data (dict) – parameter metadata of the next iterate (from the next iteration (default: None)
next_iterate_mc_directory (str) – MC directory path of the next iterate (default: None)

Returns

updated current iterate parameter metadata

Return type

dict

update_current_fidelities(param_metadata)

Update current fidelity in parameter metadata of all parameters as the sum of current fidelity and run fidelity

Parameters: param_metadata (dict) – parameter metadata
Returns: list of updated fidelities
Return type: list

get_updated_current_fidelities(param_metadata)

Get updated current fidelities after adding the run fidelity

Parameters: param_metadata (dict) – parameter metadata
Returns: list of updated fidelities
Return type: list

get_current_fidelities(param_metadata)

Get current fidelity from parameter metadata

Parameters: param_metadata (dict) – parameter metadata
Returns: list of current fidelities
Return type: list

get_parameter_data_from_metadata(param_metadata, param_index, include_parameter=False)

Get parameter data at a particular index in the metadata

Parameters

param_metadata (dict) – parameter metadata of all parameters
param_index (int) – index of the parameter of interest
include_parameter (bool) – true if parameter value at index should also to be included (default: False)

Returns

parameter metadata at the index of interest

Return type

dict

get_fidelity_from_directory(param_directory, fnamef='run_fidelity.dat')

Get fidelity (current or run fidelity) from a parameter directory

Parameters

param_directory (str) – path of the parameter directory
fnamef (str) – path of the fidelity file (current or run fidelity) (default: run_fidelity.dat)

Returns

fidelity value

Return type

int

get_param_directory_array(all_param_directory)

From the MC run directory, get a list of parameter directory

Example

If <path>/MC_RUN is the MC run directory (all_param_directory), and it has the the following contents:

<path>/MC_RUN/00
<path>/MC_RUN/01
<path>/MC_RUN/02

Then this function return:

[
    <path>/MC_RUN/00,
    <path>/MC_RUN/01,
    <path>/MC_RUN/02
]

Parameters

all_param_directory (str) – MC run directory path

Returns

paths of sub directories within the MC run directory

Return type

list

write_fidelity_to_metadata_and_directory(param_metadata, fidelities, metadata_file_key='run fidelity', fnamef='run_fidelity.dat')

Write (updated) current or run fidelity to metadata and in the fidelity file in the parameter directory

Parameters

param_metadata (dict) – all parameter metadata
fidelities (list) – fidelity values for all parameters
metadata_file_key (str) – metadata file key (corresponding to current or run fidelity) (default: run fidelity)
fnamef (str) – file name of fidelity file (corresponding to current or run fidelity) (default: run_fidelity.dat)

get_run_fidelity_from_metadata(param_metadata)

Get run fidelity from parameter metadata

Parameters: param_metadata (dict) – parameter metadata
Returns: run fidelity values
Return type: list

save_mc_out_as_csv(header, term_names, data, out_path)

Save MC output in CSV format

Example

If:

header = ["name", "MC", "DMC"]
term_names = ["Term1", "Term2"]
# data: two arrays for MC and DMC respectively and within each
# array, two values for the two terms
data = [[19,29],[99,89]]

Then CSV output is:

name,MC,DMC
Term1,19,99
Term2,29,89

Parameters

header (str) – CSV header line
term_names (list) – objective function term names to use in the
data (list) – parameter and value data
out_path (str) – path of the output CSV file

inject_nan(data, all_param_directory, term_names)

Inject NaNs into the MC data.

Example

Example of the data passed in as argument to this function is shown below. In the example below, there are three parameters and two terms of the objective function. Terms that end with .P are the parameters and those ending with .V are the values associated with either the MC, i.e., MC sample values or the DMC, i.e., MC standard deviation values:

>data

{
    "MC":{
            "Term1.P":[[1,2],[3,4],[6,3]],
            "Term1.V": [19, 18, 17],
            "Term2.P":[[1,2],[3,4],[6,3]],
            "Term2.V": [29, 28, 27]
        },
    "DMC":{
            "Term1.P":[[1,2],[3,4],[6,3]],
            "Term1.V": [99, 98, 97],
            "Term2.P":[[1,2],[3,4],[6,3]],
            "Term2.V": [89, 88, 87]
        }
}

Parameters

data (dict) – MC output data formatted into a dictionary
all_param_directory (str) – MC outout directory path
term_names (list) – term names e.g., [Term1, Term2]

convert_csv_data_to_df(all_param_directory, mc_out_file_name)

Read CSV data from all parameter directories and convert them into pandas dataframe

Example

Example of the returned pandas dataframe is given below. In the example below, there are three parameters and two terms of the objective function. Terms that end with .P are the parameters and those ending with .V are the values associated with either the MC, i.e., MC sample values or the DMC, i.e., MC standard deviation values. You can add more rows, i.e, more sets of parameter and values for additional terms in the objective function or more columns, i.e., more components of the each term of the objective that come from the MC simulator:

>df
                MC                      DMC
Term1.P        [[1,2],[3,4],[6,3]]     [[1,2],[3,4],[6,3]]
Term1.V        [19, 18, 17]            [99, 98, 97]
Term2.P        [[1,2],[3,4],[6,3]]     [[1,2],[3,4],[6,3]]
Term2.V        [29, 28, 27]            [89, 88, 87]

Parameters

all_param_directory (str) – MC run directory path
mc_out_file_name (str) – MC out CSV file name

Simple App API

SimpleApp (maestro/mc/simpleapp.py)

class maestro.mc.SimpleApp(mc_working_directory, mc_parameters=None)

Bases: MCTask

Simple but noisy MC Task

__init__(mc_working_directory, mc_parameters=None)

Create an MC Task object

Parameters

mc_working_directory (str) – MC working directory path
mc_parameters (dict) – MC parameters set in the configuration input

run_functions(param, term_names, term_class, term_counts)

run_mc(): Run the simple app MC task

get_functions()

Get the class handle for the term names given in the configuration input:mc:parameters:functions array

Returns: term names, the corresponding class handles for the term names, and counts
Return type: list,list,dict

check_df_structure_sanity(df)

Check the sanity of the pandas data frame created from the MC output

Parameters: df (pandas.DataFrame) – pandas data frame created from the MC output
Returns: corrected structure of the data frame
Return type: pandas.DataFrame

convert_mc_output_to_df(all_param_directory)

Convert CSV MC output to a pandas dataframe.

Example

Example of the returned pandas dataframe is given below. In the example below, there are three parameters and two terms of the objective function. Terms that end with .P are the parameters and those ending with .V are the values associated with either the MC, i.e., MC sample values or the DMC, i.e., MC standard deviation values. You can add more rows, i.e, more sets of parameter and values for additional terms in the objective function or more columns, i.e., more components of the each term of the objective that come from the MC simulator:

>df
                MC                      DMC
Term1.P        [[1,2],[3,4],[6,3]]     [[1,2],[3,4],[6,3]]
Term1.V        [19, 18, 17]            [99, 98, 97]
Term2.P        [[1,2],[3,4],[6,3]]     [[1,2],[3,4],[6,3]]
Term2.V        [29, 28, 27]            [89, 88, 87]

Parameters

all_param_directory (str) – MC outout directory path

Returns

pandas dataframe formatted MC output

Return type

pandas.DataFrame

merge_statistics_and_get_max_sigma()

Merge MC output statistics and find the maximum standard deviation of the MC output.

Returns: maximum standard deviation of the MC output
Return type: float

DeterministicApp (maestro/mc/simpleapp.py)

class maestro.mc.DeterministicApp(mc_working_directory, mc_parameters=None)

Bases: SimpleApp

Simple & Deterministic MC Task

__init__(mc_working_directory, mc_parameters=None)

Create an MC Task object

Parameters

mc_working_directory (str) – MC working directory path
mc_parameters (dict) – MC parameters set in the configuration input

run_mc(): Run the deterministic app MC task

merge_statistics_and_get_max_sigma()

Merge MC output statistics and find the maximum standard deviation of the MC output.

Returns: maximum standard deviation of the MC output
Return type: float

SumOfDiffPowers (maestro/mc/simpleapp.py)

class maestro.mc.SumOfDiffPowers

Bases: object

Sum of different powers from https://www.sfu.ca/~ssurjano/sumpow.html

static mapping(x, **kwargs)

y = f(x)

:param x:parameter :type x: list :return: function value :rtype: float

RotatedHyperEllipsoid (maestro/mc/simpleapp.py)

class maestro.mc.RotatedHyperEllipsoid

Bases: object

Rotates Hyper Ellipsoid from https://www.sfu.ca/~ssurjano/rothyp.html

static mapping(x, **kwargs)

y = f(x)

:param x:parameter :type x: list :return: function value :rtype: float

Sphere (maestro/mc/simpleapp.py)

class maestro.mc.Sphere

Bases: object

Sphere from https://www.sfu.ca/~ssurjano/spheref.html

static mapping(x, **kwargs)

y = f(x)

:param x:parameter :type x: list :return: function value :rtype: float

SumSquares (maestro/mc/simpleapp.py)

class maestro.mc.SumSquares

Bases: object

Sum of Squares function from https://www.sfu.ca/~ssurjano/sumsqu.html

static mapping(x, **kwargs)

y = f(x)

:param x:parameter :type x: list :return: function value :rtype: float

MiniApp (maestro/mc/miniapp.py)

class maestro.mc.MiniApp(mc_working_directory, mc_parameters=None)

Bases: MCTask

MC task for a minified problem with a small subset of observables generated using https://pythia.org

__init__(mc_working_directory, mc_parameters=None)

Create an MC Task object

Parameters

mc_working_directory (str) – MC working directory path
mc_parameters (dict) – MC parameters set in the configuration input

run_mc(): Run the miniapp app MC task

merge_statistics_and_get_max_sigma()

Merge MC output statistics and find the maximum standard deviation of the MC output.

Returns: maximum standard deviation of the MC output
Return type: float

check_df_structure_sanity(df)

Check the sanity of the pandas data frame created from the MC output

Parameters: df (pandas.DataFrame) – pandas data frame created from the MC output
Returns: corrected structure of the data frame
Return type: pandas.DataFrame

convert_mc_output_to_df(all_param_directory)

Convert CSV MC output to a pandas dataframe.

Example

Example of the returned pandas dataframe is given below. In the example below, there are three parameters and two terms of the objective function. Terms that end with .P are the parameters and those ending with .V are the values associated with either the MC, i.e., MC sample values or the DMC, i.e., MC standard deviation values. You can add more rows, i.e, more sets of parameter and values for additional terms in the objective function or more columns, i.e., more components of the each term of the objective that come from the MC simulator:

>df
                MC                      DMC
Term1.P        [[1,2],[3,4],[6,3]]     [[1,2],[3,4],[6,3]]
Term1.V        [19, 18, 17]            [99, 98, 97]
Term2.P        [[1,2],[3,4],[6,3]]     [[1,2],[3,4],[6,3]]
Term2.V        [29, 28, 27]            [89, 88, 87]

Parameters

all_param_directory (str) – MC outout directory path

Returns

pandas dataframe formatted MC output

Return type

pandas.DataFrame

A14App (maestro/mc/a14app.py)

class maestro.mc.A14App(mc_working_directory, mc_parameters=None)

Bases: MCTask

MC task for Pythia 8 Monte Carlo event generator’s [https://pythia.org] parton shower and multiple parton interaction parameters to a range of data observables from ATLAS Run 1 from 2014 (A14) [https://cds.cern.ch/record/1966419/files/ATL-PHYS-PUB-2014-021.pdf]

__init__(mc_working_directory, mc_parameters=None)

Create an MC Task object

Parameters

mc_working_directory (str) – MC working directory path
mc_parameters (dict) – MC parameters set in the configuration input

rivett_analysis = {'qcd': ['ATLAS_2011_S8924791', 'ATLAS_2011_S8971293', 'ATLAS_2011_I919017', 'ATLAS_2011_S9128077', 'ATLAS_2012_I1125575', 'ATLAS_2014_I1298811', 'ATLAS_2012_I1094564'], 'ttbar': ['ATLAS_2012_I1094568', 'ATLAS_2013_I1243871'], 'z': ['ATLAS_2011_S9131140', 'ATLAS_2014_I1300647']}

run_mc(): This method cannot be used with the A14 MC. See documentation on how to use the MC task using the script run``or ``workflow caller_type to run the this task for the A14 MC

merge_statistics_and_get_max_sigma()

Merge MC output statistics and find the maximum standard deviation of the MC output.

Returns: maximum standard deviation of the MC output
Return type: float

check_df_structure_sanity(df)

Check the sanity of the pandas data frame created from the MC output

Parameters: df (pandas.DataFrame) – pandas data frame created from the MC output
Returns: corrected structure of the data frame
Return type: pandas.DataFrame

convert_mc_output_to_df(all_param_directory)

Convert CSV MC output to a pandas dataframe.

Example

Example of the returned pandas dataframe is given below. In the example below, there are three parameters and two terms of the objective function. Terms that end with .P are the parameters and those ending with .V are the values associated with either the MC, i.e., MC sample values or the DMC, i.e., MC standard deviation values. You can add more rows, i.e, more sets of parameter and values for additional terms in the objective function or more columns, i.e., more components of the each term of the objective that come from the MC simulator:

>df
                MC                      DMC
Term1.P        [[1,2],[3,4],[6,3]]     [[1,2],[3,4],[6,3]]
Term1.V        [19, 18, 17]            [99, 98, 97]
Term2.P        [[1,2],[3,4],[6,3]]     [[1,2],[3,4],[6,3]]
Term2.V        [29, 28, 27]            [89, 88, 87]

Parameters

all_param_directory (str) – MC outout directory path

Returns

pandas dataframe formatted MC output

Return type

pandas.DataFrame

write_param(parameters, parameter_names, at_fidelities, run_fidelities, mc_run_folder, expected_folder_name, fnamep='params.dat', fnamerf='run_fidelity.dat', fnameaf='at_fidelity.dat')

Write parameters to parameter directory and generate parameter metadata Additionally, also write the pythia parameter configuration files for the three categories of A14 observables with all relevant and pertinent information as required by phytia8-diy

Parameters

parameters (list of lists) – list of parameters points
parameter_names (list) – names of the parameter dimensions
at_fidelities (list) – current fidelity of the parameters
run_fidelities (list) – expected fidelity of the parameters i.e., these are the fidelities at which the MC should be run at the corresponding parameters
mc_run_folder (str) – MC run folder path
expected_folder_name (str) – expected MC run folder path with the type of run (single or sample) and iteration number
fnamep (str) – name of the parameter file (default: params.dat)
fnamerf (str) – name of the run fidelity file (default: run_fidelity.dat)
fnameaf (str) – name of the at fidelity file (default: at_fidelity.dat)

Returns

parameter metadata object

Return type

dict

State API

Settings (maestro/settings.py)

class maestro.Settings(working_dir=None, algorithm_parameters_file=None, config_file=None, param_matadata_file=None)

Bases: object

Settings and state object

__init__(working_dir=None, algorithm_parameters_file=None, config_file=None, param_matadata_file=None)

Initialize the algorithms state and settings. Three main settings/state file include the algorithm file, configuration and parameter metadata file

Parameters

working_dir (str) – working directory path
algorithm_parameters_file (str) – algorithm parameter file path
config_file (str) – configuration file path
param_matadata_file (str) – parameter metadata file path

property N_p

Get number of parameters to sample

Returns: number of parameters to sample
Return type: int

property algorithm_status

Get algorithm status object

Returns: algorithm status
Return type: maestro.AlgorithmStatus

change_mc_ran(val: bool)

Set whether MC ran (for script run caller type)

Parameters: val (bool) – true if MC ran, false otherwise

property close_to_min_condition

Get close to minimum condition

Returns: close to minimum condition
Return type: bool

copy_type_in_paramerter_metadata(k, curr_type_key, new_type_key)

Copy parameter metadata from one parameter type to a new parameter type

Parameters

k (int) – iteration number
curr_type_key (str) – current parameter type key name
new_type_key (str) – new parameter type key name

property data_names

Get names of the components of the objective function generated from the MC

Returns: names of the components of the objective function generated from the MC
Return type: list

delete_data_at_index_from_paramerter_metadata(k, p_type, index)

Delete data at an index of the parameter type and iteration number from parameter metadata

Parameters

k (int) – iteration number
p_type (str) – parameter type
index (int) – index of the entry to delete

property dim

Get dimensionality of the problem

Returns: dimensionality of the problem
Return type: int

property f_structure_function_handle

Get function structure function handle

Returns: function structure function handle
Return type: function

property f_structure_parameters

Get function structure parameters

Returns: function structure parameters
Return type: dict

property fidelity

Get current fidelity

Returns: current fidelity
Return type: int

get_model_function_handle(data_name)

Get surrogate model function handle

Parameters: data_name (str) – names of the component of the objective function generated from the MC for which the surrogate model function handle is desired
Returns: surrogate model function handle
Return type: function

get_model_function_str(data_name)

Get surrogate model function name

Parameters: data_name (str) – names of the component of the objective function generated from the MC for which the surrogate model function handle is desired
Returns: surrogate model function name
Return type: str

get_paramerter_metadata(k, p_type)

Get parameter metadata

Parameters

k (int) – iteration number
p_type (str) – parameter type

Returns

parameter metadata

Return type

dict

increment_k(): Increment current iteration number by 1

increment_no_iters_at_max_fidelity(): Increment number of iterations where the fidelity have been at maximum fidelity

initialize_algorithm_parameters(file: str)

Check whether all required algorithm parameter keys are defined and initialize algorithm parameters

Parameters: file (str) – algorithm file path

initialize_config(file: str, working_dir: Optional[str] = None)

Check whether all required configuration keys are defined and initialize configuration parameters

Parameters

file (str) – configuration file path
working_dir (str) – working directory path

property k

Get current iteration number

Returns: current iteration number
Return type: int

property kappa

Get algorithrm parameter \(\kappa\)

Returns: algorithrm parameter \(\kappa\)
Return type: float

log_objective_function_values(oj_fn_val_model, oj_fn_val_mc)

Log the current objective function value in the algorithm state

Parameters

oj_fn_val_model (float) – model objective function value
oj_fn_val_mc (float) – MC objective function value

property max_fidelity

Get maximum fidelity value

Returns: maximum fidelity value
Return type: int

property max_fidelity_iteration

Get iteration allowed at max fidelity

Returns: iteration allowed at max fidelity
Return type: int

property max_iterations

Get maximum iteration value

Returns: maximum iteration value
Return type: int

property max_param_bounds

Get maximum parameter bounds

Returns: maximum parameter bounds
Return type: list

property max_parameter_bounds_scaled

Get maximum parameter parameter bounds (scaled)

Returns: maximum parameter parameter bounds (scaled)
Return type: list

property max_simulation_budget

Get maximum simulation budget

Returns: maximum simulation budget
Return type: int

property mc_call_on_workflow

Is the MC Task call on workflow?

Returns: true if MC Task call on workflow
Return type: bool

property mc_call_using_function_call

Is the MC Task call using function call?

Returns: true if MC Task call using function call
Return type: bool

property mc_call_using_script_run

Is the MC Task call using script run?

Returns: true if MC Task call using script run
Return type: bool

property mc_caller_type

Get MC caller type

Returns: MC caller type
Return type: str

property mc_object

Get MC object

Returns: MC object
Return type: maestro.mc.MCTask

property mc_objective_function_value

Get the MC objective function value. If not defined, np.infinity is returned

Returns: MC objective function value
Return type: float

property mc_parameters

Get MC parameters

Returns: MC parameters
Return type: dict

property mc_ran

Get whether MC ran (for script run caller type)

Returns: whether MC ran (for script run caller type)
Return type: bool

property mc_ranks

Get MC ranks

Returns: number of ranks to use in script run
Return type: int

property mc_run_folder_path

Get MC run folder path (that the MC run sees)

Returns: MC run folder path
Return type: str

property meta_data_file

Get meta data file path

Returns: meta data file path
Return type: str

property min_fidelity

Get minimum fidelity value

Returns: minimum fidelity value
Return type: int

property min_gradient_norm

Get minimum norm of projected gradient

Returns: minimum norm of projected gradient
Return type: float

property min_param_bounds

Get minimum parameter bounds

Returns: minimum parameter bounds
Return type: list

property min_parameter_bounds_scaled

Get minimum parameter parameter bounds (scaled)

Returns: minimum parameter parameter bounds (scaled)
Return type: list

property model_objective_function_value

Get the model objective function value. If not defined, np.infinity is returned

Returns: model objective function value
Return type: float

property model_parameters

Get model parameters

Returns: model parameters
Return type: dict

property next_step

Get next step of the optimization task

Returns: next step of the optimization task
Return type: str

property no_iters_at_max_fidelity

Get number of iterations where the fidelity have been at maximum fidelity

Returns: number of iterations where the fidelity have been at maximum fidelity
Return type: int

property optimization_parameters

Get trust region subproblem optimziation parameters

Returns: trust region subproblem optimziation parameters
Return type: dict

property output_level

Get output level

Returns: output level
Return type: int

property param_names

Get names of the parameter dimensions

Returns: names of the parameter dimensions
Return type: list

property previous_tr_center

Get previous trust region center

Returns: previous trust region center
Return type: list

property previous_tr_radius

Get previous trust region radius

Returns: previous trust region radius
Return type: float

property proj_grad_norm

Get norm of the projected gradient

Returns: norm of the projected gradient
Return type: float

property proj_grad_norm_of_next_iterate

Get norm of the projected gradient of the next iterate (argmin of the TR subproblem)

Returns: norm of the projected gradient at the next iterate
Return type: float

property radius_at_which_max_fidelity_reached

Get trust region radius when the fidelity reached the maximum fidelity

Returns: trust region radius when the fidelity reached the maximum fidelity
Return type: float

read_setting_file(file)

Read settings and state JSON file

Parameters: file (str) – file path
Returns: JSON structure from the file
Return type: dict

save(next_step=None, to_log=False)

Save all setting/state parameters and variables

Parameters

next_step (str) – next step in the optimization algorithm
to_log (bool) – true if the save is to log iteration state at the end of the iteration and if false the save is for dumping the state within an iteration

set_data_names(dn)

Set names of the components of the objective function generated from the MC

Parameters: dn (list) – names of the components of the objective function generated from the MC

set_end_time(): Set end time

set_radius_at_which_max_fidelity_reached(tr_radius)

Set trust region radius when the fidelity reached the maximum fidelity

Parameters: tr_radius (float) – trust region radius when the fidelity reached the maximum fidelity

set_scaled_min_max_parameter_bounds(min_bounds_scaled, max_bounds_scaled)

Set minimum and maximum parameter bounds

Parameters

min_bounds_scaled (list) – minimum parameter bounds
max_bounds_scaled (list) – maximum parameter bounds

set_start_time(): Set start time

set_tr_center_scaled(tr_center_scaled)

Set trust region center (scaled)

Parameters: tr_center_scaled (list) – trust region center (scaled)

property simulation_budget_used

Get simulation budget used

Returns: current simulation budget
Return type: int

property theta

Get algorithm parameter :math:` heta`

Returns: algorithm parameter :math:` heta`
Return type: float

property thetaprime

Get algorithm parameter :math:` heta’`

Returns: algorithm parameter :math:` heta’`
Return type: float

property tr_center

Get trust region center

Returns: trust region center
Return type: list

property tr_center_scaled

Get scaled trust region center

Returns: scaled trust region center
Return type: list

property tr_eta

Get trust region parameter \(\eta\)

Returns: \(\eta\)
Return type: float

property tr_max_radius

Get maximum allowed trust region radius

Returns: maximum allowed trust region radius
Return type: float

property tr_min_radius

Get minimum allowed trust region radius

Returns: minimum allowed trust region radius
Return type: float

property tr_mu

Get trust region parameter \(\mu\)

Returns: trust region parameter \(\mu\)
Return type: float

property tr_radius

Get trust region radius

Returns: trust region radius
Return type: float

update_close_to_min_condition(value=False)

Update close to minimum condition condition

Parameters: value (boolean) – close to minimum condition condition

update_f_structure_model_parameters(model_scaling_key: str, ds: object)

Update function parametrers with scaled and unscaled model artifacts

Parameters

model_scaling_key (str) – scaled or unscaled model key
ds (dict) – model artifacts

update_f_structure_parameters(key, value)

Update function structure parameters. If the key does not exist, a new one is created and if the key exists, the value associated with the value is updated

Parameters

key (str) – key to be added or updated
value (Any) – value to be added or updated

update_fidelity(new_fidelity)

Update fidelity (add run fidelity to current fidelity)

Parameters: new_fidelity (float) – run fidelity to add

update_parameter_metadata(k, p_type, param_ds)

Update parameter metadata

Parameters

k (int) – iteration number
p_type (str) – parameter type
param_ds (dict) – parameter metadata

update_proj_grad_norm(pgnorm)

Update norm of projected gradient

Parameters: pgnorm (float) – norm of projected gradient

update_proj_grad_norm_of_next_iterate(pgnorm)

Update norm of projected gradient of the next iterate (argmin of the TR subproblem)

Parameters: pgnorm (float) – norm of projected gradient at the next iterate

update_simulation_budget_used(simulation_count)

Upadte used simulation budget by adding simulation_count to current simulation budget

Parameters: simulation_count (int) – simulation count to be added to current simulation budget

update_tr_center(new_center)

Update trust region center

Parameters: new_center (list) – new trust region center

update_tr_radius(new_radius)

Update trust region radius

Parameters: new_radius (float) – new trust region radius value

property usefixedfidelity

Get use fixed fidelity boolean

Returns: true if fixed fidelity to be used, false for multi fidelity run
Return type: bool

property working_directory

Get working directory path

Returns: working directory path
Return type: WorkingDirectory

write_setting_file(ds, file)

Write the settings/state file

Parameters

ds (dict) – Dictionary of state/settings
file (str) – output file path

WorkingDirectory (maestro/settings.py)

class maestro.WorkingDirectory(working_dir)

Bases: object

Working directory utility

__init__(working_dir)

Initialize working directory utility object

Parameters: working_dir (str) – working directory path

get_log_path(path)

Get log path within working directory

Parameters: path (str) – path within log directory of the working directory
Returns: path consisting of working directory, log and path within log directory
Return type: str

get_conf_path(path)

Get conf path within working directory

Parameters: path (str) – path within conf directory of the working directory
Returns: path consisting of working directory, conf and path within conf directory
Return type: str

AlgorithmStatus (maestro/settings.py)

class maestro.AlgorithmStatus(status, msg=None, tr_radius_messaage=None, tr_center_messaage=None, tr_update_code=None)

Bases: object

Algorithm status class

__init__(status, msg=None, tr_radius_messaage=None, tr_center_messaage=None, tr_update_code=None)

Initialize algorithm status object

Parameters

status (int | dict) – algorithm status value
msg (str) – algorithm status message
tr_radius_messaage (str) – trust region status message
tr_center_messaage (str) – trust region center message
tr_update_code (str) – trust region update code

as_dict()

Get algorithm status object as dictionary

Returns: algorithm status object as dictionary
Return type: dict

from_dict(ds)

Construct algorithm status from dictionry

Parameters: ds (dict) – from dictionary

update_tr_status(tr_radius_messaage, tr_center_messaage, tr_update_code)

Update trust region status

Parameters

tr_radius_messaage (str) – trust region radius status message
tr_center_messaage (str) – trust region center status message
tr_update_code (str) – trust region update code

update_status(status, msg='')

Update status

Parameters

status (int) – new status
msg (str) – status message

static get_status_dict()

Get algorithm status to definition mapping

Returns: algorithm status to definition mapping
Return type: dict

property status_def

Get algorithm status definition

Returns: algorithm status definition
Return type

property status_val

Get status value

Returns: status value
Return type: int

Utility API

OutputLevel (maestro/util.py)

class maestro.OutputLevel

Bases: object

Output level utility class

__init__()

static get_out_level_dict()

Get output level dict

Returns: output level dict
Return type: dict

static get_first_out_level_with_option(option)

Get the first output level that contains the option in output level dictionary

Parameters: option (str) – option to check
Returns: first output level that contains the option in output level dictionary
Return type: str

static get_output_level_def(output_level)

Get output level definition given the output level

Parameters: output_level (int) – output level at which to get the definition
Returns: output level definition given the output level
Return type: str

static is_debug(output_level)

Is output level at debug

Parameters: output_level (int) – output level to check for at debug
Returns: true if output level to check for at debug, false otherwise
Return type: bool

static is_one_line_output(output_level)

Is output level at one line output

Parameters: output_level (int) – output level to check for whether at one line output
Returns: true if output level is at one line output, false otherwise
Return type: bool

static is_param_kp1(output_level)

Is output level at print iterate parameter values of iteration k+1

Parameters: output_level (int) – output level to check whether to print iterate parameter values of iteration k+1
Returns: true if output level at print iterate parameter values of iteration k+1, , false otherwise
Return type: bool

static is_norm_of_step(output_level)

Is output level at printing norm of step

Parameters: output_level (int) – output level to check whether to print norm of step
Returns: true if output level at print norm of step, false otherwise
Return type: bool

DiskUtil (maestro/util.py)

class maestro.DiskUtil

Bases: object

Disk operation utility

__init__()

static remove_file(file)

Remove file

Parameters: file (str) – file path to remove

static moveanything(src, dst)

Move file or folder from source to destination

Parameters

src (str) – source path
dst (str) – destination path

static copyanything(src, dst)

Copy file or folder from source to destination

Parameters

src (str) – source path
dst (str) – destination path

static remove_directory(d)

Remove directory

Parameters: d (str) – direcotry path to remove

static copy_directory_contents(dir_from, dir_to, exclude=None)

Copy directory contents with the option to exclude paths

Parameters

dir_from (str) – source directory
dir_to (str) – destination directory
exclude (list) – files and folder paths to exclude

ParameterPointUtil (maestro/util.py)

class maestro.ParameterPointUtil

Bases: object

Parameter point utility

__init__()

static check_if_not_in_min_seperation_dist(point1, point2, min_dist)

Check if two parameters are not at minimum distance from each other (in infty norm)

Parameters

point1 (list) – first parameter
point2 (list) – second parameter
mindist (float) – minimum distance

Returns

true if infty norm of parameter distance is >= minimum distance, false otherwise

Return type

bool

static check_if_point_in_TR(point, tr_center, tr_radius)

Check if parameter is within the trust region

Parameters

point (list) – parameter value
tr_center (list) – trust region center
tr_radius (float) – trust region radius

Returns

true if parameter is within trust region, false otherwise

Return type

bool

static is_close(a, b, rel_tol=1e-09, abs_tol=0.0)

Is a point close to another point

Parameters

a (float) – first parameter
b (float) – second parameter
rel_tol (float) – relative tolerance
abs_tol (float) – absolute tolerance

Returns

true if first parameter is close enough to the second parameter, false otherwise

Return type

bool

static check_if_same_point(point1, point2)

Check if two parameters are the same with the relative tolerance of 1e-09

Parameters

point1 (list) – first parameter
point2 (list) – second parameter

Returns

true if the parameters are the same, false otherwise

Return type

bool

static get_infinity_norm(point)

Get infinity norm

Parameters: point (list) – parameter point
Returns: infinity norm
Return type: float

static order_of_magnitude(number)

Get order of magnitude of a number

Parameters: number (float) – number of which the order of magnitude is desired
Returns: order of magnitude
Return type: int