10 Automated Machine Learning for Supervised Learning (Part 2)

This article was published as a part of the Data Science Blogathon

Introduction

This post will discuss 10 Automated Machine Learning (autoML) packages that we can run in Python. If you are tired of running lots of Machine Learning algorithms just to find the best one, this post might be what you are looking for. This post is the second part of this first post. The first part explains the general concept of Machine Learning from defining the objective, pre-processing, model creation and selection, hyperparameter-tuning, and model evaluation. At the end of that post, Auto-Sklearn is introduced as an autoML. If you are already familiar with Machine Learning, you can skip that part 1.

The main point of the first part is that we require a relatively long time and many lines of code to run all of the classification or regression algorithms before finally selecting or ensembling the best models. Instead, we can run AutoML to automatically search for the best model with a much shorter time and, of course, less code. Please find my notebooks for conventional Machine Learning algorithms for regression (predicting house prices) and classification (predicting poverty level classes) tasks in the table below. My AutoML notebooks are also in the table below. Note that this post will focus only on regression and classification AutoML while AutoML also can be applied for image, NLP or text, and time series forecasting.

Now, we will start discussing the 10 AutoML packages that can replace those long notebooks.

 AutoSklearn

This autoML, as mentioned above, has been discussed before. Let’s do a quick recap. Below is the code for searching for the best model in 3 minutes. More details are explained in the first part.

Regression

!apt install -y build-essential swig curl
!curl https://raw.githubusercontent.com/automl/auto-sklearn/master/requirements.txt | xargs -n 1 -L 1 pip install
!pip install auto-sklearn
!pip install scipy==1.7.0
!pip install -U scikit-learn
from autosklearn.regression import AutoSklearnRegressor
from sklearn.metrics import mean_squared_error, mean_absolute_error

Running AutoSklearn

# Create the model
sklearn = AutoSklearnRegressor(time_left_for_this_task=3*60, per_run_time_limit=30, n_jobs=-1)
# Fit the training data
sklearn.fit(X_train, y_train)
# Sprint Statistics
print(sklearn.sprint_statistics())
# Predict the validation data
pred_sklearn = sklearn.predict(X_val)
# Compute the RMSE
rmse_sklearn=MSE(y_val, pred_sklearn)**0.5
print('RMSE: ' + str(rmse_sklearn))

Output:

auto-sklearn results:
   Dataset name: c40b5794-fa4a-11eb-8116-0242ac130202
   Metric: r2
   Best validation score: 0.888788
   Number of target algorithm runs: 37
   Number of successful target algorithm runs: 23
   Number of crashed target algorithm runs: 0
   Number of target algorithms that exceeded the time limit: 8
   Number of target algorithms that exceeded the memory limit: 6
 RMSE: 27437.715258009852

Result:

# Scatter plot true and
predicted values
plt.scatter(pred_sklearn, y_val, alpha=0.2)
plt.xlabel('predicted')
plt.ylabel('true value')
plt.text(100000, 400000, 'RMSE: ' + str(round(rmse_sklearn)))
plt.text(100000, 350000, 'MAE: ' + str(round(mean_absolute_error(y_val,
pred_sklearn))))
plt.text(100000, 300000, 'R: ' + str(round(np.corrcoef(y_val,
pred_sklearn)[0,1],4)))
plt.show()

Classification

Applying AutoML

!apt install -y build-essential swig curl
!curl https://raw.githubusercontent.com/automl/auto-sklearn/master/requirements.txt | xargs -n 1 -L 1 pip install 
from autosklearn.classification import AutoSklearnClassifier
# Create the model
sklearn = AutoSklearnClassifier(time_left_for_this_task=3*60, per_run_time_limit=15, n_jobs=-1)
# Fit the training data
sklearn.fit(X_train, y_train)
# Sprint Statistics
print(sklearn.sprint_statistics())
# Predict the validation data
pred_sklearn = sklearn.predict(X_val)
# Compute the accuracy
print('Accuracy: ' + str(accuracy_score(y_val, pred_sklearn)))

Output:

auto-sklearn results:
   Dataset name: 576d4f50-c85b-11eb-802c-0242ac130202
   Metric: accuracy
   Best validation score: 0.917922
   Number of target algorithm runs: 40
   Number of successful target algorithm runs: 8
   Number of crashed target algorithm runs: 0
   Number of target algorithms that exceeded the time limit: 28
   Number of target algorithms that exceeded the memory limit: 4
 Accuracy: 0.923600209314495

Result:

# Prediction results
print('Confusion Matrix')
print(pd.DataFrame(confusion_matrix(y_val, pred_sklearn), index=[1,2,3,4], columns=[1,2,3,4]))
print('')
print('Classification Report')
print(classification_report(y_val, pred_sklearn))

Output:

Confusion Matrix
     1    2    3     4
1  123   14    3    11
2   17  273   11    18
3    3   17  195    27
4    5   15    5  1174
Classification Report
              precision    recall  f1-score   support
           1       0.83      0.81      0.82       151
           2       0.86      0.86      0.86       319
           3       0.91      0.81      0.86       242
           4       0.95      0.98      0.97      1199
    accuracy                           0.92      1911
   macro avg       0.89      0.86      0.88      1911
weighted avg       0.92      0.92      0.92      1911

Tree-based Pipeline Optimization Tool (TPOT)

TPOT is built on top of scikit-learn. TPOT uses a genetic algorithm to search for the best model according to the “generations” and “population size”. The higher the two parameters are set, the longer will it take time. Unlike AutoSklearn, we do not set the specific running time for TPOT. As its name suggests, after the TPOT is run, it exports lines of code containing a pipeline from importing packages, splitting the dataset, creating the tuned model, fitting the model, and finally predicting the validation dataset. The pipeline is exported in .py format.

In the code below, I set the generation and population_size to be 5. The output gives 5 generations with increasing “scoring”. I set the scoring to be “neg_mean_absolute_error” and” accuracy” for regression and classification tasks respectively. Neg_mean_absolute_error means Mean Absolute Error (MAE) in negative form. The algorithm chooses the highest scoring value. Making the MAE negative will make the algorithm selecting the MAE closest to zero.

Regression

from tpot import TPOTRegressor
# Create model
cv = RepeatedStratifiedKFold(n_splits=3, n_repeats=3, random_state=123)
tpot = TPOTRegressor(generations=5, population_size=5, cv=cv, scoring='neg_mean_absolute_error', verbosity=2, random_state=123, n_jobs=-1)
# Fit the training data
tpot.fit(X_train, y_train)
# Export the result
tpot.export('tpot_model.py')

Output:

Generation 1 - Current best internal CV score: -20390.588131563232
Generation 2 - Current best internal CV score: -19654.82630417806
Generation 3 - Current best internal CV score: -19312.09139004322
Generation 4 - Current best internal CV score: -19312.09139004322
Generation 5 - Current best internal CV score: -18752.921100941825
Best pipeline: RandomForestRegressor(input_matrix, bootstrap=True, max_features=0.25, min_samples_leaf=3, min_samples_split=2, n_estimators=100)

Classification

from tpot import TPOTClassifier
# TPOT that are stopped earlier. It still gives temporary best pipeline.
# Create the model
cv = RepeatedStratifiedKFold(n_splits=3, n_repeats=3, random_state=123) 
tpot = TPOTClassifier(generations=5, population_size=5, cv=cv, scoring='accuracy', verbosity=2, random_state=123, n_jobs=-1)
# Fit the training data
tpot.fit(X_train, y_train)
# Export the result
tpot.export('tpot_model.py')

Output:

Generation 1 - Current best internal CV score: 0.7432273262661955
Generation 2 - Current best internal CV score: 0.843824979278454
Generation 3 - Current best internal CV score: 0.8545565589146273
Generation 4 - Current best internal CV score: 0.8545565589146273
Generation 5 - Current best internal CV score: 0.859616978580465
Best pipeline: RandomForestClassifier(GradientBoostingClassifier(input_matrix, learning_rate=0.001, max_depth=2, max_features=0.7000000000000001, min_samples_leaf=1, min_samples_split=19, n_estimators=100, subsample=0.15000000000000002), bootstrap=True, criterion=gini, max_features=0.8500000000000001, min_samples_leaf=4, min_samples_split=12, n_estimators=100)

AutoSklearn gives the result of RandomForestRegressor for the regression task. As for the classification, it gives the stacking of GradientBoostingClassifier and RandomForestClassifier. All algorithms already have their hyperparameters tuned.

Here is to see the validation data scoring metrics.

Regression

pred_tpot = results
# Scatter plot true and predicted values
plt.scatter(pred_tpot, y_val, alpha=0.2)
plt.xlabel('predicted')
plt.ylabel('true value')
plt.text(100000, 400000, 'RMSE: ' + str(round(MSE(y_val, pred_tpot)**0.5)))
plt.text(100000, 350000, 'MAE: ' + str(round(mean_absolute_error(y_val, pred_tpot))))
plt.text(100000, 300000, 'R: ' + str(round(np.corrcoef(y_val, pred_tpot)[0,1],4)))
plt.show()

Output:

Fig. 2 TPOT regression result.

Classification

pred_tpot = results
# Compute the accuracy
print('Accuracy: ' + str(accuracy_score(y_val, pred_tpot)))
print('')
# Prediction results
print('Confusion Matrix')
print(pd.DataFrame(confusion_matrix(y_val, pred_tpot), index=[1,2,3,4], columns=[1,2,3,4]))
print('')
print('Classification Report')
print(classification_report(y_val, pred_tpot))

Output:

Accuracy: 0.9246467817896389

Confusion Matrix
     1    2    3     4
1  117   11    7    16
2    6  288   10    15
3    2   18  186    36
4    5   12    6  1176

Classification Report
              precision    recall  f1-score   support

           1       0.90      0.77      0.83       151
           2       0.88      0.90      0.89       319
           3       0.89      0.77      0.82       242
           4       0.95      0.98      0.96      1199

    accuracy                           0.92      1911
   macro avg       0.90      0.86      0.88      1911
weighted avg       0.92      0.92      0.92      1911

Distributed Asynchronous Hyper-parameter Optimization (Hyperopt)

Hyperopt is also usually used to optimize the hyperparameters of one model that has been specified. For example, we decide to apply Random Forest and then run hyperopt to find the optimal hyperparameters for the Random Forest. My previous post discussed it. But, this post is different in that it uses hyperopt to search for the best Machine Learning model automatically, not just tuning the hyperparameters. The code is similar but different.

The code below shows how to use hyperopt to run AutoML. Max evaluation of 50 and trial timeout of 20 seconds is set. These will determine how long the AutoML will work. Like in TPOT, we do not set the time limit in hyperopt.

Regression

!pip install git+https://github.com/hyperopt/hyperopt-sklearn.git
from hpsklearn import HyperoptEstimator
from hpsklearn import any_regressor
from hpsklearn import any_preprocessing
from hyperopt import tpe
from sklearn.metrics import mean_squared_error
# Create the model
hyperopt = HyperoptEstimator(regressor=any_regressor('reg'), preprocessing=any_preprocessing('pre'),
                             loss_fn=mean_squared_error, algo=tpe.suggest, max_evals=50,
                             trial_timeout=20)
# Fit the data
hyperopt.fit(X_train, y_train)

Classification

!pip install git+https://github.com/hyperopt/hyperopt-sklearn.git
from hpsklearn import HyperoptEstimator
from hpsklearn import any_classifier
from hpsklearn import any_preprocessing
from hyperopt import tpe
# Create the model
hyperopt = HyperoptEstimator(classifier=any_classifier('cla'), preprocessing=any_preprocessing('pre'),
                             algo=tpe.suggest, max_evals=50, trial_timeout=30)
# Fit the training data
hyperopt.fit(X_train_ar, y_train_ar)

In the Kaggle notebook (in the table above), every time I finished performing the fitting and predicting, I will show the prediction of validation data results in scatter plot, confusion matrix, and classification report. The code is always almost the same with a little adjustment. So, from this point onwards, I am not going to put them in this post. But, the Kaggle notebook provides them.

To see the algorithm from the AutoML search result. Use the below code. The results are ExtraTreeClassfier and XGBRegressor. Observe that it also searches for the preprocessing technics, such as standard scaler and normalizer.

# Show the models
print(hyperopt.best_model())

Regression

{'learner': XGBRegressor(base_score=0.5, booster='gbtree',
            colsample_bylevel=0.6209369845565308, colsample_bynode=1,
            colsample_bytree=0.6350745975782562, gamma=0.07330922089021298,
            gpu_id=-1, importance_type='gain', interaction_constraints='',
            learning_rate=0.0040826994703554555, max_delta_step=0,
            max_depth=10, min_child_weight=1, missing=nan,
            monotone_constraints='()', n_estimators=2600, n_jobs=1,
            num_parallel_tree=1, objective='reg:linear', random_state=3,
            reg_alpha=0.4669165283261672, reg_lambda=2.2280355282357056,
            scale_pos_weight=1, seed=3, subsample=0.7295609371405459,
            tree_method='exact', validate_parameters=1, verbosity=None), 'preprocs': (Normalizer(norm='l1'),), 'ex_preprocs': ()}

Classification

{'learner': ExtraTreesClassifier(bootstrap=True, max_features='sqrt', n_estimators=308,
                    n_jobs=1, random_state=1, verbose=False), 'preprocs': (StandardScaler(with_std=False),), 'ex_preprocs': ()}

AutoKeras

AutoKeras, as you might guess, is an autoML specializing in Deep Learning or Neural networks. The “Keras” in the name gives the clue. AutoKeras helps in finding the best neural network architecture and hyperparameters for the prediction model. Unlike the other AutoML, AutoKeras does not consider tree-based, distance-based, or other Machine Learning algorithms.

Deep Learning is challenging not only for the hyperparameter-tuning but also for the architecture setting. Many ask about how many neurons or layers are the best to use. There is no clear answer to that. Conventionally, users must try to run and evaluate their Deep Learning architectures one by one before finally decide which one is the best. It takes a long time and resources to accomplish. I wrote a post describing this here. However, AutoKeras can solve this problem.

To apply the AutoKeras, I set the max_trials to be 8 and it will try to find the best deep learning architecture for a maximum of 8 trials. Set the epoch while fitting the training dataset and it will determine the accuracy of the model.

Regression

!pip install autokeras
import autokeras
# Create the model
keras = autokeras.StructuredDataRegressor(max_trials=8)
# Fit the training dataset
keras.fit(X_train, y_train, epochs=100)
# Predict the validation data
pred_keras = keras.predict(X_val)

Classification

!pip install autokeras 
import autokeras
# Create the model
keras = autokeras.StructuredDataClassifier(max_trials=8)
# Fit the training dataset
keras.fit(X_train, y_train, epochs=100)
# Predict the validation data
pred_keras = keras.predict(X_val)
# Compute the accuracy
print('Accuracy: ' + str(accuracy_score(y_val, pred_keras)))

To find the architecture of the AutoKeras search, use the following code.

# Show the built models
keras_export = keras.export_model()
keras_export.summary()

Regression

Model: "model"
 _________________________________________________________________
 Layer (type)                 Output Shape              Param #   
 =================================================================
 input_1 (InputLayer)         [(None, 20)]              0         
 _________________________________________________________________
 multi_category_encoding (Mul (None, 20)                0         
 _________________________________________________________________
 dense (Dense)                (None, 512)               10752     
 _________________________________________________________________
 re_lu (ReLU)                 (None, 512)               0         
 _________________________________________________________________
 dense_1 (Dense)              (None, 32)                16416     
 _________________________________________________________________
 re_lu_1 (ReLU)               (None, 32)                0         
 _________________________________________________________________
 regression_head_1 (Dense)    (None, 1)                 33        
 =================================================================
 Total params: 27,201
 Trainable params: 27,201
 Non-trainable params: 0
 _________________________________________________________________

Classification

Model: "model"
 _________________________________________________________________
 Layer (type)                 Output Shape              Param #   
 =================================================================
 input_1 (InputLayer)         [(None, 71)]              0         
 _________________________________________________________________
 multi_category_encoding (Mul (None, 71)                0         
 _________________________________________________________________
 normalization (Normalization (None, 71)                143       
 _________________________________________________________________
 dense (Dense)                (None, 512)               36864     
 _________________________________________________________________
 re_lu (ReLU)                 (None, 512)               0         
 _________________________________________________________________
 dense_1 (Dense)              (None, 32)                16416     
 _________________________________________________________________
 re_lu_1 (ReLU)               (None, 32)                0         
 _________________________________________________________________
 dropout (Dropout)            (None, 32)                0         
 _________________________________________________________________
 dense_2 (Dense)              (None, 4)                 132       
 _________________________________________________________________
 classification_head_1 (Softm (None, 4)                 0         
 =================================================================
 Total params: 53,555
 Trainable params: 53,412
 Non-trainable params: 143
 _________________________________________________________________

 MLJAR

MLJAR is another great AutoML. And, you will find why soon. To run MLJAR, I assign the arguments of mode, eval_metric, total_time_limit, and feature_selection. This AutoML will understand whether it is a regression or classification task from the eval_metric. The total_time_limit is the duration of how long we allow MLJAR to run in seconds. In this case, it will take 300 seconds or 5 minutes to find the possible best model. We can also specify whether to allow feature selection. The output then will report the used algorithms and how long they take to finish.

Regression

!pip install -q -U git+https://github.com/mljar/mljar-supervised.git@master
from supervised.automl import AutoML
# Create the model
mljar = AutoML(mode="Compete",  eval_metric="rmse", total_time_limit=300,
               features_selection=True)
# Fit the training data
mljar.fit(X_train, y_train)
# Predict the training data
mljar_pred = mljar.predict(X_val)

Classification

!pip install -q -U git+https://github.com/mljar/mljar-supervised.git@master 
from supervised.automl import AutoML
# Create the model
mljar = AutoML(mode="Compete",  eval_metric="accuracy", total_time_limit=300,
               features_selection=True)
# Fit the training data
mljar.fit(X_train, y_train)
# Predict the training data
mljar_pred = mljar.predict(X_val)

The argument “mode” lets us decide what the MLJAR is expected for. There are 4 types of modes to define the purpose of running the MLJAR. In the example code above, the mode of “compete” is used for winning a competition by finding the best model by tuning and ensembling methods. The mode of “optuna” is used to find the best-tuned model with unlimited computation time. The mode of “perform” builds a Machine Learning pipeline for production. The mode of “explain” is used for data explanation.

The result of MLJAR is automatically reported and visualized. Unfortunately, Kaggle does not display the report result after saving. So, below is how it should look. The report compares the MLJAR results for every algorithm. We can see the ensemble methods have the lowest MSE for the regression task and the highest accuracy for the classification task. The increasing number of iteration lowers the MSE for the regression task and improves the accuracy of the classification task. (The leaderboard tables below actually have more rows, but they were cut.)

Regression

 

Fig. 3 MLJAR Report for regression

Fig. 4 MLJAR for regession

 AutoGluon

AutoGluon requires users to format the training dataset using TabularDataset to recognize it. Users can then specify the time_limit allocation for AutoGluon to work. In the example code below, I set it to be 120 seconds or 2 minutes.

Regression

!pip install -U pip
!pip install -U setuptools wheel
!pip install -U "mxnet<2.0.0"
!pip install autogluon
from autogluon.tabular import TabularDataset, TabularPredictor
# Prepare the data
Xy_train = X_train.reset_index(drop=True)
Xy_train['Target'] = y_train
Xy_val = X_val.reset_index(drop=True)
Xy_val['Target'] = y_val
X_train_gluon = TabularDataset(Xy_train)
X_val_gluon = TabularDataset(Xy_val)
# Fit the training data
gluon = TabularPredictor(label='Target').fit(X_train_gluon, time_limit=120)
# Predict the training data
gluon_pred = gluon.predict(X_val)

Classification

!pip install -U pip
!pip install -U setuptools wheel
!pip install -U "mxnet<2.0.0"
!pip install autogluon 
from autogluon.tabular import TabularDataset, TabularPredictor
# Prepare the data
Xy_train = X_train.reset_index(drop=True)
Xy_train['Target'] = y_train
Xy_val = X_val.reset_index(drop=True)
Xy_val['Target'] = y_val
X_train_gluon = TabularDataset(Xy_train)
X_val_gluon = TabularDataset(Xy_val)
# Fit the training data
gluon = TabularPredictor(label='Target').fit(X_train_gluon, time_limit=120)
# Predict the training data
gluon_pred = gluon.predict(X_val)

After finishing the task, AutoGluon can report the accuracy of each Machine Learning algorithm it has tried. The report is called leaderboard. The columns below are actually longer, but I cut them for this post.

# Show the models
leaderboard = gluon.leaderboard(X_train_gluon)
leaderboard

Regression

Classification

H2O

Similar to AutoGluon, H2O requires the training dataset in a certain format, called H2OFrame. To decide how long H2O will work, either max_runtime_secs or max_models must be specified. The names explain what they mean.

Regression

import h2o
from h2o.automl import H2OAutoML
h2o.init()
# Prepare the data
Xy_train = X_train.reset_index(drop=True)
Xy_train['SalePrice'] = y_train.reset_index(drop=True)
Xy_val = X_val.reset_index(drop=True)
Xy_val['SalePrice'] = y_val.reset_index(drop=True)
# Convert H2O Frame
Xy_train_h2o = h2o.H2OFrame(Xy_train)
X_val_h2o = h2o.H2OFrame(X_val)
# Create the model
h2o_model = H2OAutoML(max_runtime_secs=120, seed=123)
# Fit the model
h2o_model.train(x=Xy_train_h2o.columns, y='SalePrice', training_frame=Xy_train_h2o)
# Predict the training data
h2o_pred = h2o_model.predict(X_val_h2o)

Classification

import h2o
from h2o.automl import H2OAutoML
h2o.init()
# Convert H2O Frame
Xy_train_h2o = h2o.H2OFrame(Xy_train)
X_val_h2o = h2o.H2OFrame(X_val)
Xy_train_h2o['Target'] = Xy_train_h2o['Target'].asfactor()
# Create the model
h2o_model = H2OAutoML(max_runtime_secs=120, seed=123)
# Fit the model
h2o_model.train(x=Xy_train_h2o.columns, y='Target', training_frame=Xy_train_h2o)
# Predict the training data
h2o_pred = h2o_model.predict(X_val_h2o)
h2o_pred

For the classification task, the prediction result is a multilabel classification result. It gives the probability value for each class. Below is an example of the classification.

H2O reports its result by a simple table showing various scoring metrics of each Machine Learning algorithm.

# Show the model results
leaderboard_h2o = h2o.automl.get_leaderboard(h2o_model, extra_columns = 'ALL')
leaderboard_h2o

Regression output:

Classification

PyCaret

This is the longest AutoML code that this post is exploring. PyCaret does not need splitting the features (X_train) and label (y_train). So, the below code will only randomly split the training dataset into another training dataset and a validation dataset. Preprocessing, such as filling missing data or feature selection, is also not required. Then, we set up the PyCaret by assigning the data, target variable or label, numeric imputation method, categorical imputation method, whether to use normalization, whether to remove multicollinearity, etc.

Regression

!pip install pycaret
from pycaret.regression import *
# Generate random numbers
val_index = np.random.choice(range(trainSet.shape[0]), round(trainSet.shape[0]*0.2), replace=False)
# Split trainSet
trainSet1 = trainSet.drop(val_index)
trainSet2 = trainSet.iloc[val_index,:]
# Create the model
caret = setup(data = trainSet1, target='SalePrice', session_id=111,
              numeric_imputation='mean',  categorical_imputation='constant',
              normalize = True, combine_rare_levels = True, rare_level_threshold = 0.05,
              remove_multicollinearity = True, multicollinearity_threshold = 0.95)

Classification

!pip install pycaret
from pycaret.classification import *
# Generate random numbers
val_index = np.random.choice(range(trainSet.shape[0]), round(trainSet.shape[0]*0.2), replace=False)
# Split trainSet
trainSet1 = trainSet.drop(val_index)
trainSet2 = trainSet.iloc[val_index,:]
# Create the model
caret = setup(data = trainSet1, target='Target', session_id=123,
              numeric_imputation='mean',  categorical_imputation='constant',
              normalize = True, combine_rare_levels = True, rare_level_threshold = 0.05,
              remove_multicollinearity = True, multicollinearity_threshold = 0.95)

After that, we can run the PyCaret by specifying how many cross-validations folds we want. The PyCaret for regression gives several models sorting from the best scoring metrics. The top models are Bayesian Ridge, Huber Regressor, Orthogonal Matching Pursuit, Ridge Regression, and Passive-Aggressive Regressor. The scoring metrics are MAE, MSE, RMSE, R2, RMSLE, and MAPE. PyCaret for classification also gives several models. The top models are the Extra Trees Classifier, Random Forest Classifier, Decision Tree Classifier, Extreme Gradient Boosting, and Light Gradient Boosting Machine. The below tables are limited for their rows and columns. Find the complete tables in the Kaggle notebook.

# Show the models
caret_models = compare_models(fold=5)

Regression

Classification

To create the top 5 models, run the following code.

Regression

# Create the top 5 models
br = create_model('br', fold=5)
huber = create_model('huber', fold=5)
omp = create_model('omp', fold=5)
ridge = create_model('ridge', fold=5)
par = create_model('par', fold=5)

Classification

# Create the top 5 models
et = create_model('et', fold=5)
rf = create_model('rf', fold=5)
dt = create_model('dt', fold=5)
xgboost = create_model('xgboost', fold=5)
lightgbm = create_model('lightgbm', fold=5)

To tune the selected model, run the following code.

Regression

# Tune the models, BR: Regression
br_tune = tune_model(br, fold=5)
# Show the tuned hyperparameters, for example for BR: Regression
plot_model(br_tune, plot='parameter')

Classification

# Tune the models, LightGBM: Classification
lightgbm_tune = tune_model(lightgbm, fold=5)
# Show the tuned hyperparameters, for example for LightGBM: Classification
plot_model(lightgbm_tune, plot='parameter')

PyCaret lets the users manually perform ensemble methods, like Bagging, Boosting, Stacking, or Blending. The below code performs each ensemble method.

Regression

# Bagging BR
br_bagging = ensemble_model(br_tune, fold=5)
# Boosting BR
br_boost = ensemble_model(br_tune, method='Boosting', fold=5)
# Stacking with Huber as the meta-model
stack = stack_models(caret_models_5, meta_model=huber, fold=5)
# Blending top models
caret_blend = blend_models(estimator_list=[br_tune,huber_tune,omp_tune,ridge_tune,par_tune])

Classification

# Bagging LightGBM
lightgbm_bagging = ensemble_model(lightgbm_tune, fold=5)
# Boosting LightGBM
lightgbm_boost = ensemble_model(lightgbm_tune, method='Boosting', fold=5)
# Stacking with ET as the meta-model
stack = stack_models(caret_models_5, meta_model=et, fold=5)
# Blending top models
caret_blend = blend_models(estimator_list=[lightgbm_tune,rf,dt])

Now, let’s choose blending models as the predictive models. The following code uses the blending models to predict the validation datasets.

Regression

# Predict the validation data 
caret_pred = predict_model(caret_blend, data = trainSet2.drop(columns=['SalePrice']))

Classification

# Predict the validation data 
pred_caret = predict_model(caret_blend, data = trainSet2.drop(columns=['Target']))

AutoViML

I run AutoViML in the notebook by assigning many arguments. Just like PyCaret, AutoViML does not need splitting the features (X_train) and label (y_train). Users only need to split the training dataset (as trainSet1) and the validation dataset (as trainSet2). Users can specify other parameters, like scoring parameter, hyperparameter, feature reduction, boosting, binning, and so on.

Regression

!pip install autoviml
!pip install shap
from autoviml.Auto_ViML import Auto_ViML
# Create the model
viml, features, train_v, test_v = Auto_ViML(trainSet1, 'SalePrice', trainSet2.drop(columns=['SalePrice']),
                                            scoring_parameter='', hyper_param='RS',
                                            feature_reduction=True, Boosting_Flag=True,
                                            Binning_Flag=False,Add_Poly=0, Stacking_Flag=False, 
                                            Imbalanced_Flag=True, verbose=1)

Classification

!pip install autoviml
!pip install shap
from autoviml.Auto_ViML import Auto_ViML
# Create the model
viml, features, train_v, test_v = Auto_ViML(trainSet1, 'Target', trainSet2.drop(columns=['Target']),
                                            scoring_parameter='balanced_accuracy', hyper_param='RS',
                                            feature_reduction=True, Boosting_Flag=True,
                                            Binning_Flag=False,Add_Poly=0, Stacking_Flag=False, 
                                            Imbalanced_Flag=True, verbose=1)

After fitting the training data, we can examine what has been done from the output. For example, we can see that AutoViML was doing preprocessing by filling the missing data. Similar to MLJAR, AutoViML also gives the visualized report. For the regression task, it visualizes the scatter plot of true and predicted values using the XGBoost model. It also plots the prediction residual error in a histogram. From the two plots, we can observe that the model has relatively good accuracy and the residual error is a normal distribution.

The next report is ensemble method trials. We can find this report for both regression and classification tasks. The last graph displays the feature importance. We can see that the most important feature to predict house prices is exterior material quality, followed by overall material and finish quality, and size of the garage respectively.

As for the classification task, it does not visualize the scatter plot, but the ROC curve for each of the four prediction classes, micro-average, and macro-average as well as the AUC value. We can see that all of the classes have AUC values above 0.98. The next chart reports the iso-f1 curves as the accuracy metric. It also later gives the classification report.

Average precision score, micro-averaged over all classes: 0.97
Macro F1 score, averaged over all classes: 0.83

#####################################################
              precision    recall  f1-score   support
           0       0.84      0.97      0.90       963
           1       0.51      0.59      0.54       258
           2       0.18      0.11      0.14       191
           3       0.00      0.00      0.00       118
    accuracy                           0.73      1530
   macro avg       0.38      0.42      0.40      1530
weighted avg       0.64      0.73      0.68      1530
[[938  19   6   0]
 [106 151   1   0]
 [ 53 117  21   0]
 [ 16  12  90   0]]

The following code prints out the result of AutoViML search. We can see that the best model is XGBRegressor and XGBClassifier for each task.

viml

Regression

XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,
             colsample_bynode=1, colsample_bytree=0.7, gamma=1, gpu_id=0,
             grow_policy='depthwise', importance_type='gain',
             interaction_constraints='', learning_rate=0.1, max_delta_step=0,
             max_depth=8, max_leaves=0, min_child_weight=1, missing=nan,
             monotone_constraints='()', n_estimators=100, n_jobs=-1, nthread=-1,
             num_parallel_tree=1, objective='reg:squarederror',
             predictor='cpu_predictor', random_state=1, reg_alpha=0.5,
             reg_lambda=0.5, scale_pos_weight=1, seed=1, subsample=0.7,
             tree_method='hist', ...)

Classification

CalibratedClassifierCV(base_estimator=OneVsRestClassifier(estimator=XGBClassifier(base_score=None,
                                                                                  booster='gbtree',
                                                                                  colsample_bylevel=None,
                                                                                  colsample_bynode=None,
                                                                                  colsample_bytree=None,
                                                                                  gamma=None,
                                                                                  gpu_id=None,
                                                                                  importance_type='gain',
                                                                                  interaction_constraints=None,
                                                                                  learning_rate=None,
                                                                                  max_delta_step=None,
                                                                                  max_depth=None,
                                                                                  min_child_weight=None,
                                                                                  missing=nan,
                                                                                  monotone_constraints=None,
                                                                                  n_estimators=200,
                                                                                  n_jobs=-1,
                                                                                  nthread=-1,
                                                                                  num_parallel_tree=None,
                                                                                  objective='binary:logistic',
                                                                                  random_state=99,
                                                                                  reg_alpha=None,
                                                                                  reg_lambda=None,
                                                                                  scale_pos_weight=None,
                                                                                  subsample=None,
                                                                                  tree_method=None,
                                                                                  use_label_encoder=True,
                                                                                  validate_parameters=None,
                                                                                  verbosity=None),
                                                          n_jobs=None),
                       cv=5, method='isotonic')

LightAutoML

Now, we have reached the 10th AutoML. LightAutoML is expected to be light as its name explains. Here, we set the task to be ‘reg’ for regression, ‘multiclass’ for multiclass classification, or ’binary’ for binary classification. We can also set the metrics and losses in the task. I set the timeout to be 3 minutes to let it find the best model. After simple .fit and .predict, we can already get the result.

Regression

!pip install -U https://github.com/sberbank-ai-lab/LightAutoML/raw/fix/logging/LightAutoML-0.2.16.2-py3-none-any.whl
!pip install openpyxl
from lightautoml.automl.presets.tabular_presets import TabularAutoML
from lightautoml.tasks import Task
# Create the model
light = TabularAutoML(task=Task('reg',), timeout=60*3, cpu_limit=4)
train_data = pd.concat([X_train, y_train], axis=1)
# Fit the training data
train_light = light.fit_predict(train_data, roles = {'target': 'SalePrice', 'drop':[]})
# Predict the validation data
pred_light = light.predict(X_val)

Classification

!pip install -U https://github.com/sberbank-ai-lab/LightAutoML/raw/fix/logging/LightAutoML-0.2.16.2-py3-none-any.whl
!pip install openpyxl
from lightautoml.automl.presets.tabular_presets import TabularAutoML
from lightautoml.tasks import Task
train_data = pd.concat([X_train, y_train], axis=1)
# Create the model
light = TabularAutoML(task=Task('multiclass',), timeout=60*3, cpu_limit=4)
# Fit the training data
train_light = light.fit_predict(train_data, roles = {'target': 'Target'})
# Predict the validation data
pred_light = light.predict(X_val)

The results for the classification task are the predicted class and the probability of each of the classes. In other words, the result fulfills multiclass and multilabel classification expectations.

# Convert the prediction result into dataframe
pred_light2 = pred_light.data
pred_light2 = pd.DataFrame(pred_light2, columns=['4','2','3','1'])
pred_light2 = pred_light2[['1','2','3','4']]
pred_light2['Pred'] = pred_light2.idxmax(axis=1)
pred_light2['Pred'] = pred_light2['Pred'].astype(int)
pred_light2.head()

Conclusion

We have discussed 10 AutoML packages now. Please notice that there are still more AutoML not yet discussed in this post. Not only in Python notebook but AutoML also can be found in cloud computing or software. Which AutoML is your favourite? Or, do you have other AutoML in mind?

Are you still going to run 10 conventional Machine Learning or prefer the 10 AutoML packages?

About Author

References

1. Fig1 – https://www.kaggle.com/rendyk/automl-regression-part1
2. Fig2- https://www.kaggle.com/rendyk/automl-regression-part1
3. Fig3- Image by Author
4. Fig4-Image by Author
5. Fig5-Image by Author
6. Fig6-Image by Author
7. Fig7- https://www.kaggle.com/rendyk/automl-regression-part2
8. Fig8- https://www.kaggle.com/rendyk/automl-regression-part2
9. Fig9- https://www.kaggle.com/rendyk/automl-regression-part2
10. Fig10- sklearn, topot, hyperopt, autokeras, mljar, autogluon, h2o, pycaret, autoviml

Connect with me here.

Rendyk 05 Sep 2021