This module generates model configurations using grid search or random search.
Once the configurations are defined, they can be used by the fit function in Train Model Configurations. By model configurations we mean both hyperparameters and model architectures. The output table from this module defines the combinations of model architectures, compile and fit parameters to be trained in parallel.
generate_model_configs( model_arch_table, model_selection_table, model_id_list, compile_params_grid, fit_params_grid, search_type, num_configs, random_state, object_table )
Arguments
VARCHAR. Table containing model architectures and weights. For more information on this table refer to the module Define Model Architectures.
VARCHAR. Model selection table created by this module. If this table already exists, it will be appended to. A summary table named <model_selection_table>_summary is also created. Contents of both output tables are described below.
INTEGER[]. Array of model IDs from the 'model_arch_table' to be included in the run combinations. For hyperparameter search, this will typically be one model ID. For model architecture search, this will be the different model IDs that you want to compare.
VARCHAR. String representation of a Python dictionary of compile parameters to be tested. Each entry of the dictionary should consist of keys as compile parameter names, and values as a Python list of compile parameter values to be passed to Keras. Also, optimizer parameters are a nested dictionary to allow different optimizer types to have different parameters or ranges of parameters. Here is an example:
$$ {'loss': ['categorical_crossentropy'], 'optimizer_params_list': [ {'optimizer': ['SGD'], 'lr': [0.0001, 0.001, 'log'], 'momentum': [0.95, 0.99, 'log_near_one']}, {'optimizer': ['Adam'], 'lr': [0.01, 0.1, 'log'], 'decay': [1e-6, 1e-4, 'log']}], 'metrics': ['accuracy'] } $$
The following types of sampling are supported: 'linear', 'log' and 'log_near_one'. The 'log_near_one' sampling is useful for exponentially weighted average types of parameters like momentum, which are very sensitive to changes near 1. It has the effect of producing more values near 1 than regular log-based sampling.
In the case of grid search, omit the sampling type and just put the grid points in the list.
VARCHAR. String representation of a Python dictionary of fit parameters to be tested. Each entry of the dictionary should consist of keys as fit parameter names, and values as a Python list of fit parameter values to be passed to Keras. Here is an example:
$$ {'batch_size': [32, 64, 128, 256], 'epochs': [10, 20, 30] } $$
VARCHAR, 'grid' or 'random'. Search strategy for generating model configurations. For grid search, will generate all combinations of model IDs + compile params + fit params. For random search, specify the number of configs you want to generate using the 'num_configs' parameter below. Note that you can also use short forms for the 'grid' or 'random' keywords, e.g.,'rand' or 'r' instead of writing out 'random' in full.
INTEGER, default: NULL. Number of model configs to generate. Only applies when search_type='random'.
INTEGER, default: NULL. Pseudo random number generator state used for random uniform sampling from lists of possible values. Pass an integer for reproducible output across multiple function calls. Only applies when search_type='random'.
VARCHAR, default: NULL. Name of the table containing Python objects in the case that custom loss functions or custom metrics are specified in the 'compile_params_grid'. Note that this table has to be created by the Define Custom Functions method. It is not allowed to pass a schema name, since it will be automatically pulled from this functions associated madlib schema.
Output table
The model selection output table contains the following columns:
mst_key | INTEGER. ID that defines a unique tuple for model architecture-compile parameters-fit parameters. |
---|---|
model_id | VARCHAR. Model architecture ID from the 'model_arch_table'. |
compile_params | VARCHAR. Keras compile parameters. |
fit_params | VARCHAR. Keras fit parameters. |
A summary table named <model_selection_table>_summary is also created, which contains the following column:
model_arch_table | VARCHAR. Name of the model architecture table containing the model architecture IDs. |
---|---|
object_table | VARCHAR. Name of the object table containing the serialized Python objects for custom loss functions and custom metrics. If there are none, this field will be blank. |
This method is deprecated and replaced by the method 'generate_model_configs' described above.
load_model_selection_table( model_arch_table, model_selection_table, model_id_list, compile_params_list, fit_params_list, object_table )
Arguments
VARCHAR. Table containing model architectures and weights. For more information on this table refer to Define Model Architectures.
VARCHAR. Model selection table created by this utility. A summary table named <model_selection_table>_summary is also created. Contents of both output tables are the same as described above for the method 'generate_model_configs'.
INTEGER[]. Array of model IDs from the 'model_arch_table' to be included in the run combinations. For hyperparameter search, this will typically be one model ID. For model architecture search, this will be the different model IDs that you want to test.
VARCHAR[]. Array of compile parameters to be tested. Each element of the array should consist of a string of compile parameters exactly as it is to be passed to Keras. For custom loss functions or custom metrics, list the custom function name in the usual way, and also provide the name of the table where the serialized objects reside in the parameter 'object_table' below.
VARCHAR[]. Array of fit parameters to be tested. Each element of the array should consist of a string of fit parameters exactly as it is to be passed to Keras. Callbacks are not currently supported except for TensorBoard which you can specify in the usual way, e.g., callbacks=[TensorBoard(log_dir="/tmp/logs/fit")]
VARCHAR, default: NULL. Name of the table containing Python objects in the case that custom loss functions or custom metrics are specified in the parameter 'compile_params_list'.
from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense model1 = Sequential() model1.add(Dense(10, activation='relu', input_shape=(4,))) model1.add(Dense(10, activation='relu')) model1.add(Dense(3, activation='softmax')) model1.summary()Export the model to JSON:_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense_1 (Dense) (None, 10) 50 _________________________________________________________________ dense_2 (Dense) (None, 10) 110 _________________________________________________________________ dense_3 (Dense) (None, 3) 33 ================================================================= Total params: 193 Trainable params: 193 Non-trainable params: 0
model1.to_json()
'{"class_name": "Sequential", "keras_version": "2.1.6", "config": [{"class_name": "Dense", "config": {"kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "scale": 1.0, "seed": null, "mode": "fan_avg"}}, "name": "dense_1", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "dtype": "float32", "activation": "relu", "trainable": true, "kernel_regularizer": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "units": 10, "batch_input_shape": [null, 4], "use_bias": true, "activity_regularizer": null}}, {"class_name": "Dense", "config": {"kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "scale": 1.0, "seed": null, "mode": "fan_avg"}}, "name": "dense_2", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "activation": "relu", "trainable": true, "kernel_regularizer": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "units": 10, "use_bias": true, "activity_regularizer": null}}, {"class_name": "Dense", "config": {"kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "scale": 1.0, "seed": null, "mode": "fan_avg"}}, "name": "dense_3", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "activation": "softmax", "trainable": true, "kernel_regularizer": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "units": 3, "use_bias": true, "activity_regularizer": null}}], "backend": "tensorflow"}'Now use Keras to define a model architecture with 2 hidden layers:
model2 = Sequential() model2.add(Dense(10, activation='relu', input_shape=(4,))) model2.add(Dense(10, activation='relu')) model2.add(Dense(10, activation='relu')) model2.add(Dense(3, activation='softmax')) model2.summary()Export the model to JSON:_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense_4 (Dense) (None, 10) 50 _________________________________________________________________ dense_5 (Dense) (None, 10) 110 _________________________________________________________________ dense_6 (Dense) (None, 10) 110 _________________________________________________________________ dense_7 (Dense) (None, 3) 33 ================================================================= Total params: 303 Trainable params: 303 Non-trainable params: 0
model2.to_json()
'{"class_name": "Sequential", "keras_version": "2.1.6", "config": [{"class_name": "Dense", "config": {"kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "scale": 1.0, "seed": null, "mode": "fan_avg"}}, "name": "dense_4", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "dtype": "float32", "activation": "relu", "trainable": true, "kernel_regularizer": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "units": 10, "batch_input_shape": [null, 4], "use_bias": true, "activity_regularizer": null}}, {"class_name": "Dense", "config": {"kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "scale": 1.0, "seed": null, "mode": "fan_avg"}}, "name": "dense_5", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "activation": "relu", "trainable": true, "kernel_regularizer": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "units": 10, "use_bias": true, "activity_regularizer": null}}, {"class_name": "Dense", "config": {"kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "scale": 1.0, "seed": null, "mode": "fan_avg"}}, "name": "dense_6", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "activation": "relu", "trainable": true, "kernel_regularizer": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "units": 10, "use_bias": true, "activity_regularizer": null}}, {"class_name": "Dense", "config": {"kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "scale": 1.0, "seed": null, "mode": "fan_avg"}}, "name": "dense_7", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "activation": "softmax", "trainable": true, "kernel_regularizer": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "units": 3, "use_bias": true, "activity_regularizer": null}}], "backend": "tensorflow"}'
DROP TABLE IF EXISTS model_arch_library; SELECT madlib.load_keras_model('model_arch_library', -- Output table, $$ {"class_name": "Sequential", "keras_version": "2.1.6", "config": [{"class_name": "Dense", "config": {"kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "scale": 1.0, "seed": null, "mode": "fan_avg"}}, "name": "dense_1", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "dtype": "float32", "activation": "relu", "trainable": true, "kernel_regularizer": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "units": 10, "batch_input_shape": [null, 4], "use_bias": true, "activity_regularizer": null}}, {"class_name": "Dense", "config": {"kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "scale": 1.0, "seed": null, "mode": "fan_avg"}}, "name": "dense_2", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "activation": "relu", "trainable": true, "kernel_regularizer": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "units": 10, "use_bias": true, "activity_regularizer": null}}, {"class_name": "Dense", "config": {"kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "scale": 1.0, "seed": null, "mode": "fan_avg"}}, "name": "dense_3", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "activation": "softmax", "trainable": true, "kernel_regularizer": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "units": 3, "use_bias": true, "activity_regularizer": null}}], "backend": "tensorflow"} $$ ::json, -- JSON blob NULL, -- Weights 'Sophie', -- Name 'MLP with 1 hidden layer' -- Descr ); SELECT madlib.load_keras_model('model_arch_library', -- Output table, $$ {"class_name": "Sequential", "keras_version": "2.1.6", "config": [{"class_name": "Dense", "config": {"kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "scale": 1.0, "seed": null, "mode": "fan_avg"}}, "name": "dense_4", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "dtype": "float32", "activation": "relu", "trainable": true, "kernel_regularizer": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "units": 10, "batch_input_shape": [null, 4], "use_bias": true, "activity_regularizer": null}}, {"class_name": "Dense", "config": {"kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "scale": 1.0, "seed": null, "mode": "fan_avg"}}, "name": "dense_5", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "activation": "relu", "trainable": true, "kernel_regularizer": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "units": 10, "use_bias": true, "activity_regularizer": null}}, {"class_name": "Dense", "config": {"kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "scale": 1.0, "seed": null, "mode": "fan_avg"}}, "name": "dense_6", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "activation": "relu", "trainable": true, "kernel_regularizer": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "units": 10, "use_bias": true, "activity_regularizer": null}}, {"class_name": "Dense", "config": {"kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "scale": 1.0, "seed": null, "mode": "fan_avg"}}, "name": "dense_7", "kernel_constraint": null, "bias_regularizer": null, "bias_constraint": null, "activation": "softmax", "trainable": true, "kernel_regularizer": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "units": 3, "use_bias": true, "activity_regularizer": null}}], "backend": "tensorflow"} $$ ::json, -- JSON blob NULL, -- Weights 'Maria', -- Name 'MLP with 2 hidden layers' -- Descr ); SELECT model_id, name, description FROM model_arch_library ORDER BY model_id;
model_id | name | description ----------+--------+-------------------------- 1 | Sophie | MLP with 1 hidden layer 2 | Maria | MLP with 2 hidden layers (2 rows)
DROP TABLE IF EXISTS mst_table, mst_table_summary; SELECT madlib.generate_model_configs( 'model_arch_library', -- model architecture table 'mst_table', -- model selection table output ARRAY[1,2], -- model ids from model architecture table $$ {'loss': ['categorical_crossentropy'], 'optimizer_params_list': [ {'optimizer': ['Adam', 'SGD'], 'lr': [0.001, 0.01]} ], 'metrics': ['accuracy']} $$, -- compile_param_grid $$ { 'batch_size': [64, 128], 'epochs': [10] } $$, -- fit_param_grid 'grid' -- search_type ); SELECT * FROM mst_table ORDER BY mst_key;
mst_key | model_id | compile_params | fit_params ---------+----------+---------------------------------------------------------------------------------+-------------------------- 1 | 1 | optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 2 | 1 | optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 3 | 1 | optimizer='SGD(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 4 | 1 | optimizer='SGD(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 5 | 1 | optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 6 | 1 | optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 7 | 1 | optimizer='SGD(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 8 | 1 | optimizer='SGD(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 9 | 2 | optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 10 | 2 | optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 11 | 2 | optimizer='SGD(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 12 | 2 | optimizer='SGD(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 13 | 2 | optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 14 | 2 | optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 15 | 2 | optimizer='SGD(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 16 | 2 | optimizer='SGD(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 (16 rows)Note that above uses the same learning rate for the two optimizers. If you wanted to use different learning rates and different parameters for different optimizers (common):
DROP TABLE IF EXISTS mst_table, mst_table_summary; SELECT madlib.generate_model_configs( 'model_arch_library', -- model architecture table 'mst_table', -- model selection table output ARRAY[1,2], -- model ids from model architecture table $$ {'loss': ['categorical_crossentropy'], 'optimizer_params_list': [ {'optimizer': ['SGD']}, {'optimizer': ['SGD'], 'lr': [0.0001, 0.001], 'momentum': [0.95]}, {'optimizer': ['Adam'], 'lr': [0.01, 0.1], 'decay': [1e-4]}], 'metrics': ['accuracy']} $$, -- compile_param_grid $$ { 'batch_size': [64, 128], 'epochs': [10] } $$, -- fit_param_grid 'grid' -- search_type ); SELECT * FROM mst_table ORDER BY mst_key;
mst_key | model_id | compile_params | fit_params ---------+----------+-----------------------------------------------------------------------------------------------+-------------------------- 1 | 1 | optimizer='SGD()',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 2 | 1 | optimizer='SGD()',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 3 | 2 | optimizer='SGD()',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 4 | 2 | optimizer='SGD()',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 5 | 1 | optimizer='SGD(lr=0.0001,momentum=0.95)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 6 | 1 | optimizer='SGD(lr=0.0001,momentum=0.95)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 7 | 1 | optimizer='SGD(lr=0.001,momentum=0.95)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 8 | 1 | optimizer='SGD(lr=0.001,momentum=0.95)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 9 | 2 | optimizer='SGD(lr=0.0001,momentum=0.95)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 10 | 2 | optimizer='SGD(lr=0.0001,momentum=0.95)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 11 | 2 | optimizer='SGD(lr=0.001,momentum=0.95)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 12 | 2 | optimizer='SGD(lr=0.001,momentum=0.95)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 13 | 1 | optimizer='Adam(lr=0.01,decay=0.0001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 14 | 1 | optimizer='Adam(lr=0.01,decay=0.0001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 15 | 1 | optimizer='Adam(lr=0.1,decay=0.0001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 16 | 1 | optimizer='Adam(lr=0.1,decay=0.0001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 17 | 2 | optimizer='Adam(lr=0.01,decay=0.0001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 18 | 2 | optimizer='Adam(lr=0.01,decay=0.0001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 19 | 2 | optimizer='Adam(lr=0.1,decay=0.0001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 20 | 2 | optimizer='Adam(lr=0.1,decay=0.0001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 (20 rows)
DROP TABLE IF EXISTS mst_table, mst_table_summary; SELECT madlib.generate_model_configs( 'model_arch_library', -- model architecture table 'mst_table', -- model selection table output ARRAY[1,2], -- model ids from model architecture table $$ {'loss': ['categorical_crossentropy'], 'optimizer_params_list': [ {'optimizer': ['SGD'], 'lr': [0.0001, 0.001, 'log'], 'momentum': [0.95, 0.99, 'log_near_one']}, {'optimizer': ['Adam'], 'lr': [0.01, 0.1, 'log'], 'decay': [1e-6, 1e-4, 'log']}], 'metrics': ['accuracy']} $$, -- compile_param_grid $$ { 'batch_size': [64, 128], 'epochs': [10] } $$, -- fit_param_grid 'random', -- search_type 20 ); SELECT * FROM mst_table ORDER BY mst_key;
mst_key | model_id | compile_params | fit_params ---------+----------+-----------------------------------------------------------------------------------------------------------------------------+-------------------------- 1 | 1 | optimizer='SGD(lr=0.000195784477708685,momentum=0.9768159513291526)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 2 | 2 | optimizer='SGD(lr=0.0002499200066875511,momentum=0.9807877269510826)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 3 | 1 | optimizer='SGD(lr=0.0009097798285407916,momentum=0.9706029152411938)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 4 | 1 | optimizer='SGD(lr=0.0001272842475986666,momentum=0.9858583458057799)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 5 | 1 | optimizer='SGD(lr=0.0001367874444015989,momentum=0.9772674033475668)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 6 | 2 | optimizer='SGD(lr=0.0002233708561319785,momentum=0.9743315606145182)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 7 | 1 | optimizer='SGD(lr=0.0009066689970530365,momentum=0.9835897505288803)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 8 | 1 | optimizer='SGD(lr=0.0007589416356572876,momentum=0.958751411608181)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 9 | 2 | optimizer='Adam(lr=0.057814228170084386,decay=1.0641718595377929e-06)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 10 | 1 | optimizer='Adam(lr=0.01927466297833838,decay=1.039476442716842e-06)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 11 | 2 | optimizer='Adam(lr=0.014718555287257804,decay=9.947768661882175e-05)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 12 | 1 | optimizer='Adam(lr=0.010397686133595378,decay=2.5730580994358942e-05)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 13 | 1 | optimizer='SGD(lr=0.0008624562426613621,momentum=0.989134963527059)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 14 | 2 | optimizer='SGD(lr=0.00010555974470031461,momentum=0.980489419269402)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 15 | 2 | optimizer='Adam(lr=0.05041699703418617,decay=4.685540619995589e-05)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 16 | 1 | optimizer='Adam(lr=0.034295140601304126,decay=1.6034699865163222e-06)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 17 | 1 | optimizer='Adam(lr=0.06888969005355218,decay=1.6318109152382423e-05)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 18 | 2 | optimizer='SGD(lr=0.0008225712651952847,momentum=0.9819748008695103)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 19 | 1 | optimizer='Adam(lr=0.0819110285922332,decay=1.6912312124827899e-06)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 20 | 1 | optimizer='Adam(lr=0.011688026325555774,decay=2.9315437856404027e-05)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 (20 rows)
DROP TABLE IF EXISTS mst_table, mst_table_summary; SELECT madlib.generate_model_configs( 'model_arch_library', -- model architecture table 'mst_table', -- model selection table output ARRAY[1,2], -- model ids from model architecture table $$ {'loss': ['categorical_crossentropy'], 'optimizer_params_list': [ {'optimizer': ['Adam', 'SGD'], 'lr': [0.001, 0.01]} ], 'metrics': ['accuracy']} $$, -- compile_param_grid $$ { 'batch_size': [64, 128], 'epochs': [10] } $$, -- fit_param_grid 'grid' -- search_type );Now add to the existing table and note that mst_key continues where it left off:
SELECT madlib.generate_model_configs( 'model_arch_library', -- model architecture table 'mst_table', -- model selection table output ARRAY[1,2], -- model ids from model architecture table $$ {'loss': ['categorical_crossentropy'], 'optimizer_params_list': [ {'optimizer': ['SGD'], 'lr': [0.0001, 0.001, 'log'], 'momentum': [0.95, 0.99, 'log_near_one']}, {'optimizer': ['Adam'], 'lr': [0.01, 0.1, 'log'], 'decay': [1e-6, 1e-4, 'log']}], 'metrics': ['accuracy']} $$, -- compile_param_grid $$ { 'batch_size': [64, 128], 'epochs': [10] } $$, -- fit_param_grid 'random', -- search_type 20 ); SELECT * FROM mst_table ORDER BY mst_key;
mst_key | model_id | compile_params | fit_params ---------+----------+-----------------------------------------------------------------------------------------------------------------------------+-------------------------- 1 | 1 | optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 2 | 1 | optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 3 | 1 | optimizer='SGD(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 4 | 1 | optimizer='SGD(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 5 | 1 | optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 6 | 1 | optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 7 | 1 | optimizer='SGD(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 8 | 1 | optimizer='SGD(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 9 | 2 | optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 10 | 2 | optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 11 | 2 | optimizer='SGD(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 12 | 2 | optimizer='SGD(lr=0.001)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 13 | 2 | optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 14 | 2 | optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 15 | 2 | optimizer='SGD(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 16 | 2 | optimizer='SGD(lr=0.01)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 17 | 2 | optimizer='SGD(lr=0.00013996842804647915,momentum=0.9677072493281305)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 18 | 1 | optimizer='Adam(lr=0.04252873277972123,decay=9.503983307511243e-05)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 19 | 2 | optimizer='Adam(lr=0.06666969394323848,decay=1.5626668941131748e-05)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 20 | 2 | optimizer='SGD(lr=0.00016137313867804707,momentum=0.954293112127019)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 21 | 2 | optimizer='Adam(lr=0.019443570245321506,decay=1.2882524497407873e-06)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 22 | 2 | optimizer='Adam(lr=0.06302317748060839,decay=6.238009849562074e-05)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 23 | 2 | optimizer='SGD(lr=0.00010890482493011119,momentum=0.9826239169968034)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 24 | 1 | optimizer='SGD(lr=0.0009201966766121783,momentum=0.9896730563556151)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 25 | 2 | optimizer='SGD(lr=0.00028961522836420906,momentum=0.9859394149216544)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 26 | 1 | optimizer='SGD(lr=0.0001503249757866609,momentum=0.9777816636354879)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 27 | 2 | optimizer='SGD(lr=0.0008405326172626768,momentum=0.9538686498263182)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 28 | 1 | optimizer='SGD(lr=0.00011926989091387571,momentum=0.9876746918399469)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 29 | 1 | optimizer='Adam(lr=0.018794361633022855,decay=9.387826286694454e-06)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 30 | 2 | optimizer='SGD(lr=0.0009692977025027591,momentum=0.9878758592330659)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 31 | 2 | optimizer='SGD(lr=0.0006671929498585603,momentum=0.9786502962872058)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 32 | 2 | optimizer='Adam(lr=0.03948766165185474,decay=3.056584635386748e-06)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 33 | 2 | optimizer='Adam(lr=0.020343961099103417,decay=1.183810228780669e-05)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 34 | 1 | optimizer='Adam(lr=0.016854644990148417,decay=3.561117893117444e-06)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 35 | 2 | optimizer='SGD(lr=0.0004620089560788749,momentum=0.9887310587871919)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 36 | 1 | optimizer='SGD(lr=0.0002493912675066962,momentum=0.9892077270385708)',metrics=['categorical_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 (36 rows)
DROP TABLE IF EXISTS mst_table_manual; CREATE TABLE mst_table_manual( mst_key serial, model_id integer, compile_params varchar, fit_params varchar ); INSERT INTO mst_table_manual(model_id, compile_params, fit_params) VALUES (1, $$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['categorical_accuracy']$$, 'batch_size=4,epochs=1'), (1, $$loss='categorical_crossentropy',optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy']$$, 'batch_size=4,epochs=1'), (1, $$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy']$$, 'batch_size=4,epochs=1'), (2, $$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['categorical_accuracy']$$, 'batch_size=8,epochs=1'), (2, $$loss='categorical_crossentropy',optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy']$$, 'batch_size=8,epochs=1'), (2, $$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy']$$, 'batch_size=8,epochs=1'); SELECT * FROM mst_table_manual ORDER BY mst_key;
mst_key | model_id | compile_params | fit_params ---------+---------------+---------------------------------------------------------------------------------+----------------------- 1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['categorical_accuracy'] | batch_size=4,epochs=1 2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy'] | batch_size=4,epochs=1 3 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy'] | batch_size=4,epochs=1 4 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['categorical_accuracy'] | batch_size=8,epochs=1 5 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy'] | batch_size=8,epochs=1 6 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy'] | batch_size=8,epochs=1 (6 rows)Create the summary table which must be named with the model selection output table appended by "_summary":
DROP TABLE IF EXISTS mst_table_manual_summary; CREATE TABLE mst_table_manual_summary ( model_arch_table varchar ); INSERT INTO mst_table_manual_summary(model_arch_table) VALUES ('model_arch_library'); SELECT * FROM mst_table_manual_summary;
model_arch_table --------------------+ model_arch_library (1 row)
DROP TABLE IF EXISTS mst_table, mst_table_summary; SELECT madlib.generate_model_configs( 'model_arch_library', -- model architecture table 'mst_table', -- model selection table output ARRAY[1,2], -- model ids from model architecture table $$ {'loss': ['my_custom_loss'], 'optimizer_params_list': [ {'optimizer': ['Adam', 'SGD'], 'lr': [0.001, 0.01]} ], 'metrics': ['my_custom_accuracy']} $$, -- compile_param_grid $$ { 'batch_size': [64, 128], 'epochs': [10] } $$, -- fit_param_grid 'grid', -- search_type NULL, -- num_configs NULL, -- random_state 'custom_function_table' -- table with custom functions ); SELECT * FROM mst_table ORDER BY mst_key;
mst_key | model_id | compile_params | fit_params ---------+----------+---------------------------------------------------------------------------------+-------------------------- 1 | 1 | optimizer='Adam(lr=0.001)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=64 2 | 1 | optimizer='Adam(lr=0.001)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=128 3 | 1 | optimizer='SGD(lr=0.001)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=64 4 | 1 | optimizer='SGD(lr=0.001)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=128 5 | 1 | optimizer='Adam(lr=0.01)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=64 6 | 1 | optimizer='Adam(lr=0.01)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=128 7 | 1 | optimizer='SGD(lr=0.01)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=64 8 | 1 | optimizer='SGD(lr=0.01)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=128 9 | 2 | optimizer='Adam(lr=0.001)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=64 10 | 2 | optimizer='Adam(lr=0.001)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=128 11 | 2 | optimizer='SGD(lr=0.001)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=64 12 | 2 | optimizer='SGD(lr=0.001)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=128 13 | 2 | optimizer='Adam(lr=0.01)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=64 14 | 2 | optimizer='Adam(lr=0.01)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=128 15 | 2 | optimizer='SGD(lr=0.01)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=64 16 | 2 | optimizer='SGD(lr=0.01)',metrics=['my_custom_accuracy'],loss='my_custom_loss' | epochs=10,batch_size=128 (16 rows)Similarly, if you created a custom top k categorical accuracy function 'top_3_accuracy' in Define Custom Functions you can generate the model configurations as:
DROP TABLE IF EXISTS mst_table, mst_table_summary; SELECT madlib.generate_model_configs( 'model_arch_library', -- model architecture table 'mst_table', -- model selection table output ARRAY[1,2], -- model ids from model architecture table $$ {'loss': ['categorical_crossentropy'], 'optimizer_params_list': [ {'optimizer': ['Adam', 'SGD'], 'lr': [0.001, 0.01]} ], 'metrics': ['top_3_accuracy']} $$, -- compile_param_grid $$ { 'batch_size': [64, 128], 'epochs': [10] } $$, -- fit_param_grid 'grid', -- search_type NULL, -- num_configs NULL, -- random_state 'custom_function_table' -- table with custom functions ); SELECT * FROM mst_table ORDER BY mst_key;
mst_key | model_id | compile_params | fit_params ---------+----------+---------------------------------------------------------------------------------+-------------------------- 1 | 1 | optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 2 | 1 | optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 3 | 1 | optimizer='SGD(lr=0.001)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 4 | 1 | optimizer='SGD(lr=0.001)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 5 | 1 | optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 6 | 1 | optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 7 | 1 | optimizer='SGD(lr=0.01)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 8 | 1 | optimizer='SGD(lr=0.01)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 9 | 2 | optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 10 | 2 | optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 11 | 2 | optimizer='SGD(lr=0.001)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 12 | 2 | optimizer='SGD(lr=0.001)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 13 | 2 | optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 14 | 2 | optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 15 | 2 | optimizer='SGD(lr=0.01)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=64 16 | 2 | optimizer='SGD(lr=0.01)',metrics=['top_3_accuracy'],loss='categorical_crossentropy' | epochs=10,batch_size=128 (16 rows)
DROP TABLE IF EXISTS mst_table, mst_table_summary; SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table 'mst_table', -- model selection table output ARRAY[1,2], -- model ids from model architecture table ARRAY[ -- compile params $$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['categorical_accuracy']$$, $$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy']$$, $$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy']$$ ], ARRAY[ -- fit params $$batch_size=4,epochs=1$$, $$batch_size=8,epochs=1$$ ] ); SELECT * FROM mst_table ORDER BY mst_key;
mst_key | model_id | compile_params | fit_params ---------+---------------+---------------------------------------------------------------------------------+----------------------- 1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['categorical_accuracy'] | batch_size=4,epochs=1 2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['categorical_accuracy'] | batch_size=8,epochs=1 3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy'] | batch_size=4,epochs=1 4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy'] | batch_size=8,epochs=1 5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy'] | batch_size=4,epochs=1 6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy'] | batch_size=8,epochs=1 7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['categorical_accuracy'] | batch_size=4,epochs=1 8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['categorical_accuracy'] | batch_size=8,epochs=1 9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy'] | batch_size=4,epochs=1 10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['categorical_accuracy'] | batch_size=8,epochs=1 11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy'] | batch_size=4,epochs=1 12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['categorical_accuracy'] | batch_size=8,epochs=1 (12 rows)The name of the model architecture table is stored in the summary table:
SELECT * FROM mst_table_summary;
model_arch_table --------------------+ model_arch_library (1 row)