This article was published as a part of the Data Science Blogathon.

AdaBoost stands for Adaptive Boosting. It is a statistical classification algorithm. It is an algorithm that forms a committee of weak classifiers. It boosts the performance of machine learning algorithms. It helps you form a committee of weak classifiers by combining them into a single strong classifier. It can be used to solve a wide range of problems. The main idea of AdaBoost classifier is to increase the weight of the unclassified points and also to decrease the weight of the classified points. In the case of the Adaboost classifier, the trees are not fully grown, and the trees are just one root and two leaves, called stumps. The algorithm is as follows:

• It is less prone to overfitting
• Less parameters tweaking
• Helps in reducing bias and variance
• Accuracy of weak classifiers can be improved using this method
• Easy to use

• It needs a quality dataset
• Very sensitive to outliers and noise
• Slower than XGBoost
• Hyperparameter optimization is much more difficult

## Implementation

Creating a dataframe

First we will be creating a dataframe for whic we have to perform classification

```df = pd.DataFrame()
df['feature1'] = [9,9,7,6,6,5,4,3,2,1]
df['feature2'] = [2,9,8,5,9,1,8,6,3,5]
df['target'] =[-1,-1,1,-1,1,-1,1,-1,1,1]
df['w'] = 1/df.shape
df```

Separate the features and target variable

```#features and target values
X = df.iloc[:,0:2].values
y = df.iloc[:,2].values```

Plot the points in 2D Space

```from typing import Optional
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
#visualizing our datapoints
y: np.ndarray,
clf=None,
sample_weights: Optional[np.ndarray] = None,
annotate: bool = False,
ax: Optional[mpl.axes.Axes] = None) -> None:
assert set(y) == {-1, 1}, 'Expecting response labels to be ±1'
if not ax:
fig, ax = plt.subplots(figsize=(5, 5), dpi=100)
fig.set_facecolor('white')
if sample_weights is not None:
sizes = np.array(sample_weights) * X.shape * 100
else:
sizes = np.ones(shape=X.shape) * 100
X_pos = X[y == 1]
sizes_pos = sizes[y == 1]
ax.scatter(*X_pos.T, s=sizes_pos, marker='+', color='green')
X_neg = X[y == -1]
sizes_neg = sizes[y == -1]
ax.scatter(*X_neg.T, s=sizes_neg, marker='.', c='purple')
if clf:
plot_step = 0.01
xx, yy = np.meshgrid(np.arange(x_min, x_max, plot_step),
np.arange(y_min, y_max, plot_step))
Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

if list(np.unique(Z)) == :
fill_colors = ['g']
else:
fill_colors = ['purple', 'g']
ax.contourf(xx, yy, Z, colors=fill_colors, alpha=0.2)
if annotate:
for i, (x, y) in enumerate(X):
offset = 0.05
ax.annotate(f'\$x_{i + 1}\$', (x + offset, y - offset))
ax.set_xlim(x_min+0.5, x_max-0.5)
ax.set_ylim(y_min+0.5, y_max-0.5)
ax.set_xlabel('\$x_1\$')
ax.set_ylabel('\$x_2\$')

Create a class AdaBoost with variables like stumps, errors etc.

```class AdaBoost:
def __init__(self):
self.stumps = None
self.stump_weights = None
self.errors = None
self.sample_weights = None

def _check_X_y(self, X, y):
assert set(y) == {-1, 1}, 'Response variable must be ±1'
return X, y```

Here we used Decision tree with max depth=1 as
our base classifier. AdaBoost repeatedly calls this weak learner with each time feeding it a different distribution of the training data. Every time we call, it generates a weak classifier and combine all of them into a single classifier which will give better performance.

```from sklearn.tree import DecisionTreeClassifier
def fit(self, X: np.ndarray, y: np.ndarray, iters: int):
X, y = self._check_X_y(X, y)
n = X.shape
self.sample_weights = np.zeros(shape=(iters, n))
self.stumps = np.zeros(shape=iters, dtype=object)
self.stump_weights = np.zeros(shape=iters)
self.errors = np.zeros(shape=iters)
self.sample_weights = np.ones(shape=n) / n
for t in range(iters):
curr_sample_weights = self.sample_weights[t]
stump = DecisionTreeClassifier(max_depth=1, max_leaf_nodes=2)
stump = stump.fit(X, y, sample_weight=curr_sample_weights)
stump_pred = stump.predict(X)
err = curr_sample_weights[(stump_pred != y)].sum()# / n
stump_weight = np.log((1 - err) / err) / 2
new_sample_weights = (curr_sample_weights * np.exp(-stump_weight * y * stump_pred))
new_sample_weights /= new_sample_weights.sum()
if t+1 < iters:
self.sample_weights[t+1] = new_sample_weights
self.stumps[t] = stump
self.stump_weights[t] = stump_weight
self.errors[t] = err
return self
def predict(self, X):
stump_preds = np.array([stump.predict(X) for stump in self.stumps])
return np.sign(np.dot(self.stump_weights, stump_preds))

Visualizing our learner step by step

Here, the left column depicts the weak learner selected and the right column depicts the cumulative strong learner so far obtained in each iteration. Accordinh to the Adaboost classifier, the points which are misclassified in their previous iteration will be now more heavily weighted and therefore appear larger in the next iteration.

```def truncate_adaboost(clf, t: int):
assert t > 0, 't must be a positive integer'
from copy import deepcopy
new_clf = deepcopy(clf)
new_clf.stumps = clf.stumps[:t]
new_clf.stump_weights = clf.stump_weights[:t]
return new_clf

fig, axes = plt.subplots(figsize=(8, iters*3),
nrows=iters,
ncols=2,
sharex=True,
dpi=100)
fig.set_facecolor('white')
_ = fig.suptitle('Decision boundaries by iteration')
for i in range(iters):
ax1, ax2 = axes[i]
_ = ax1.set_title(f'Weak learner at t={i + 1}')
sample_weights=clf.sample_weights[i],
annotate=False, ax=ax1)
trunc_clf = truncate_adaboost(clf, t=i + 1)
_ = ax2.set_title(f'Strong learner at t={i + 1}')
sample_weights=clf.sample_weights[i],
annotate=False, ax=ax2)
plt.tight_layout()
plt.show()

Plot for error rate is as follows-

The error rate is:

Comparison with sklearn’s inbuilt adaboost classifier

The n_estimators used are 1, 5, 10 trees to help notice a difference in performance and understand the working of boosting models.

Split the dataset into 70:30

```from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)```

```from sklearn.ensemble import AdaBoostClassifier
from sklearn import metrics
model = abc.fit(X_train, y_train)
y_pred = model.predict(X_test)

```from sklearn.ensemble import AdaBoostClassifier
from sklearn import metrics
model = abc.fit(X_train, y_train)
y_pred = model.predict(X_test)

```from sklearn.ensemble import AdaBoostClassifier
from sklearn import metrics
model = abc.fit(X_train, y_train)
y_pred = model.predict(X_test)

## Conclusion

• Decisions trees are used as base classifiers
• Even though we have used weak classifiers still, all the instances are correctly classified.
• The main principle in the classifier is to increase the weight of misclassified datapoints and to decrease the weight value of correctly classified points. This updating of weights in the classifier allows the next iteration’s weak learner to have a meaningful decision boundary.
• When the trees are increased, the model results in better and more accurate classification. 