Ever since its introduction in 2014, XGBoost has been lauded as the holy grail of machine learning hackathons and competitions. From predicting ad click-through rates to classifying high energy physics events, XGBoost has proved its mettle in terms of performance â€“ and speed. XGBoost is the first algorithm of choice for many in ML hackathon. The accuracy it consistently gives, and the time it saves, demonstrates how useful it is. But how does it actually work? What kind of mathematics power XGBoost? We’ll figure out the answers to these questions soon.

In this article, we will first look at the power of XGBoost, and then deep dive into the inner workings of this popular and powerful technique. It’s good to be able to implement it in Python or R, but understanding the nitty-gritties of the algorithm will help you become a better data scientist.

XGBoost is a machine learning algorithm that belongs to the ensemble learning category, specifically the gradient boosting framework. It utilizes decision trees as base learners and employs regularization techniques to enhance model generalization. Known for its computational efficiency, feature importance analysis, and handling of missing values, XGBoost is widely used for tasks such as regression, classification, and ranking.

It’s no wonder then that CERN recognized it as the best approach to classify signals from the Large Hadron Collider. This particular challenge posed by CERN required a solution that would be scalable to process data being generated at the rate of 3 petabytes per year and effectively distinguish an extremely rare signal from background noises in a complex physical process. XGBoost emerged as the most useful, straightforward and robust solution.

We recommend going through the below article as well to fully understand the various terms and concepts mentioned in this article:

XGBoost, or eXtreme Gradient Boosting, is a machine learning algorithm under ensemble learning. It is trendy for supervised learning tasks, such as regression and classification. XGBoost builds a predictive model by combining the predictions of multiple individual models, often decision trees, in an iterative manner.

The algorithm works by sequentially adding weak learners to the ensemble, with each new learner focusing on correcting the errors made by the existing ones. It uses a gradient descent optimization technique to minimize a predefined loss function during training.

Key features of XGBoost Algorithm include its ability to handle complex relationships in data, regularization techniques to prevent overfitting and incorporation of parallel processing for efficient computation. XGBoost is widely used in various domains due to its high predictive performance and versatility across different datasets

XGBoost is an ensemble learning method. Sometimes, it may not be sufficient to rely upon the results of just one machine learning model. Ensemble learning offers a systematic solution to combine the predictive power of multiple learners. The resultant is a single model which gives the aggregated output from several models.

The models that form the ensemble, also known as base learners, could be either from the same learning algorithm or different learning algorithms. Bagging and boosting are two widely used ensemble learners. Though these two techniques can be used with several statistical models, the most predominant usage has been with decision trees.

Let’s briefly discuss bagging before taking a more detailed look at the concept of gradient boosting.

While decision trees are one of the most easily interpretable models, they exhibit highly variable behavior. Consider a single training dataset that we randomly split into two parts. Now, letâ€™s use each part to train a decision tree in order to obtain two models.

When we fit both these models, they would yield different results. Decision trees are said to be associated with high variance due to this behavior. Bagging or boosting aggregation helps to reduce the variance in any learner. Several decision trees which are generated in parallel, form the base learners of bagging technique. Data sampled with replacement is fed to these learners for training. The final prediction is the averaged output from all the learners.

In boosting, the trees are built sequentially such that each subsequent tree aims to reduce the errors of the previous tree. Each tree learns from its predecessors and updates the residual errors. Hence, the tree that grows next in the sequence will learn from an updated version of the residuals.

The base learners in boosting are weak learners in which the bias is high, and the predictive power is just a tad better than random guessing. Each of these weak learners contributes some vital information for prediction, enabling the boosting technique to produce a strong learner by effectively combining these weak learners. The final strong learner brings down both the bias and the variance.

**In contrast to bagging techniques like Random Forest, in which trees are grown to their maximum extent, boosting makes use of trees with fewer splits**. Such small trees, which are not very deep, are highly interpretable. Parameters like the number of trees or iterations, the rate at which the gradient boosting learns, and the depth of the tree, could be optimally selected through validation techniques like k-fold cross validation. **Having a large number of trees might lead to overfitting. So, it is necessary to carefully choose the stopping criteria for boosting.**

The gradient boosting ensemble technique consists of three simple steps:

- An initial model F0 is defined to predict the target variable y. This model will be associated with a residual (y â€“ F0)
- A new model h1 is fit to the residuals from the previous step
- Now, F0 and h1 are combined to give F1, the boosted version of F0. The mean squared error from F1 will be lower than that from F0:

To improve the performance of F1, we could model after the residuals of F1 and create a new model F2:

This can be done for *â€˜mâ€™ *iterations, until residuals have been minimized as much as possible:

Here, the additive learners do not disturb the functions created in the previous steps. Instead, they impart information of their own to bring down the errors.

Consider the following data where the years of experience is predictor variable and salary (in thousand dollars) is the target. Using regression trees as base learners, we can create an ensemble model to predict the salary. For the sake of simplicity, we can choose square loss as our loss function and our objective would be to minimize the square error.

As the first step, the model should be initialized with a function F0(x). F0(x) should be a function which minimizes the loss function or MSE (mean squared error), in this case:

Taking the first differential of the above equation with respect to Î³, it is seen that the function minimizes at the mean i=1nyin. So, the boosting model could be initiated with:

F0(x) gives the predictions from the first stage of our model. Now, the residual error for each instance is (yi â€“ F0(x)).

We can use the residuals from F0(x) to create h1(x). h1(x) will be a regression tree which will try and reduce the residuals from the previous step. The output of h1(x) wonâ€™t be a prediction of y; instead, it will help in predicting the successive function F1(x) which will bring down the residuals.

The additive model h1(x) computes the mean of the residuals (y â€“ F0) at each leaf of the tree. The boosted function F1(x) is obtained by summing F0(x) and h1(x). This way h1(x) learns from the residuals of F0(x) and suppresses it in F1(x).

This can be repeated for 2 more iterations to compute h2(x) and h3(x). Each of these additive learners, hm(x), will make use of the residuals from the preceding function, Fm-1(x).

The MSEs for F0(x), F1(x) and F2(x) are 875, 692 and 540. It’s amazing how these simple weak learners can bring about a huge reduction in error!

Note that each learner, hm(x), is trained on the residuals. All the additive learners in boosting are modeled after the residual errors at each step. Intuitively, it could be observed that the boosting learners make use of the patterns in residual errors. At the stage where maximum accuracy is reached by boosting, the residuals appear to be randomly distributed without any pattern.

**Plots of F****n**** and h****n**

In the case discussed above, MSE was the loss function. The mean minimized the error here. When MAE (mean absolute error) is the loss function, the median would be used as F0(x) to initialize the model. A unit change in y would cause a unit change in MAE as well.

For MSE, the change observed would be roughly exponential. Instead of fitting hm(x) on the residuals, fitting it on the gradient of loss function, or the step along which loss occurs, would make this process generic and applicable across all loss functions.

Gradient descent helps us minimize any differentiable function. Earlier, the regression tree for hm(x) predicted the mean residual at each terminal node of the tree. In gradient boosting, the average gradient component would be computed.

For each node, there is a factor Î³ with which hm(x) is multiplied. This accounts for the difference in impact of each branch of the split. Gradient boosting helps in predicting the optimal gradient for the additive model, unlike classical gradient descent techniques which reduce error in the output at each iteration.

The following steps are involved in gradient boosting:

- F0(x) â€“ with which we initialize the boosting algorithm â€“ is to be defined:

- The gradient of the loss function is computed iteratively:

- Each hm(x) is fit on the gradient obtained at each step
- The multiplicative factor Î³m for each terminal node is derived and the boosted model Fm(x) is defined:

XGBoost is a popular implementation of gradient boosting. Letâ€™s discuss some features of XGBoost that make it so interesting:

**Regularization:**XGBoost has an option to penalize complex models through both L1 and L2 regularization. Regularization helps in preventing overfitting**Handling sparse data:**Missing values or data processing steps like one-hot encoding make data sparse. XGBoost incorporates a sparsity-aware split finding algorithm to handle different types of sparsity patterns in the data**Weighted quantile sketch:**Most existing tree based algorithms can find the split points when the data points are of equal weights (using quantile sketch algorithm). However, they are not equipped to handle weighted data. XGBoost has a distributed weighted quantile sketch algorithm to effectively handle weighted data**Block structure for parallel learning:**For faster computing, XGBoost can make use of multiple cores on the CPU. This is possible because of a block structure in its system design. Data is sorted and stored in in-memory units called blocks. Unlike other algorithms, this enables the data layout to be reused by subsequent iterations, instead of computing it again. This feature also serves useful for steps like split finding and column sub-sampling**Cache awareness:**In XGBoost, non-continuous memory access is required to get the gradient statistics by row index. Hence, XGBoost has been designed to make optimal use of hardware. This is done by allocating internal buffers in each thread, where the gradient statistics can be stored**Out-of-core computing:**This feature optimizes the available disk space and maximizes its usage when handling huge datasets that do not fit into memory

Here’s a live coding window to see how XGBoost works and play around with the code without leaving this article!

**High accuracy:**XGBoost is known for its accuracy and has been shown to outperform other machine learning algorithms in many predictive modeling tasks.**Scalability:**It is highly scalable and can handle large datasets with millions of rows and columns.**Efficiency:**It is designed to be computationally efficient and can quickly train models on large datasets.**Flexibility:**It supports a variety of data types and objectives, including regression, classification, and ranking problems.**Regularization:**It incorporates regularization techniques to avoid overfitting and improve generalization performance.**Interpretability:**It provides feature importance scores that can help users understand which features are most important for making predictions.**Open-source:**XGBoost is an open-source library that is widely used and supported by the data science community.

So that was all about the mathematics that power the popular XGBoost algorithm. If your basics are solid, this article must have been a breeze for you. It’s such a powerful algorithm and while there are other techniques that have spawned from it (like CATBoost), XGBoost remains a game changer in the machine learning community. We highly recommend you to take up this course to sharpen your skills in machine learning and learn all the state-of-the-art techniques used in the field with our Applied Machine Learning – Beginner to Professional course.

A. The performance of XGBoost and random forest depends on the data and problem being solved. XGBoost tends to perform better on structured data, while random forest can be more effective on unstructured data.

A. XGBoost Python is a Python package that enables building and training models using the XGBoost algorithm in Python. It includes many functions for tuning and optimizing model performance.

A. XGBoost is a versatile algorithm that can be used for both classification and regression problems. It can handle different types of data and can be customized for specific needs.

XGBoost and Random Forest are both ensemble methods. Random Forest builds independent trees in parallel, while XGBoost builds trees sequentially to correct errors. XGBoost often outperforms due to its regularization, sequential error correction, and better predictive performance, especially for structured data. Random Forest is robust but may not achieve the same predictive accuracy. The choice depends on data characteristics and the problem at hand.

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Become a full stack data scientist##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

Understanding Cost Function
Understanding Gradient Descent
Math Behind Gradient Descent
Assumptions of Linear Regression
Implement Linear Regression from Scratch
Train Linear Regression in Python
Implementing Linear Regression in R
Diagnosing Residual Plots in Linear Regression Models
Generalized Linear Models
Introduction to Logistic Regression
Odds Ratio
Implementing Logistic Regression from Scratch
Introduction to Scikit-learn in Python
Train Logistic Regression in python
Multiclass using Logistic Regression
How to use Multinomial and Ordinal Logistic Regression in R ?
Challenges with Linear Regression
Introduction to Regularisation
Implementing Regularisation
Ridge Regression
Lasso Regression

Introduction to Stacking
Implementing Stacking
Variants of Stacking
Implementing Variants of Stacking
Introduction to Blending
Bootstrap Sampling
Introduction to Random Sampling
Hyper-parameters of Random Forest
Implementing Random Forest
Out-of-Bag (OOB) Score in the Random Forest
IPL Team Win Prediction Project Using Machine Learning
Introduction to Boosting
Gradient Boosting Algorithm
Math behind GBM
Implementing GBM in python
Regularized Greedy Forests
Extreme Gradient Boosting
Implementing XGBM in python
Tuning Hyperparameters of XGBoost in Python
Implement XGBM in R/H2O
Adaptive Boosting
Implementing Adaptive Boosing
LightGBM
Implementing LightGBM in Python
Catboost
Implementing Catboost in Python

Introduction to Clustering
Applications of Clustering
Evaluation Metrics for Clustering
Understanding K-Means
Implementation of K-Means in Python
Implementation of K-Means in R
Choosing Right Value for K
Profiling Market Segments using K-Means Clustering
Hierarchical Clustering
Implementation of Hierarchial Clustering
DBSCAN
Defining Similarity between clusters
Build Better and Accurate Clusters with Gaussian Mixture Models

Introduction to Machine Learning Interpretability
Framework and Interpretable Models
model Agnostic Methods for Interpretability
Implementing Interpretable Model
Understanding SHAP
Out-of-Core ML
Introduction to Interpretable Machine Learning Models
Model Agnostic Methods for Interpretability
Game Theory & Shapley Values

Deploying Machine Learning Model using Streamlit
Deploying ML Models in Docker
Deploy Using Streamlit
Deploy on Heroku
Deploy Using Netlify
Introduction to Amazon Sagemaker
Setting up Amazon SageMaker
Using SageMaker Endpoint to Generate Inference
Deploy on Microsoft Azure Cloud
Introduction to Flask for Model
Deploying ML model using Flask

Nice explanation !

Hi.Nice article. Thanks for sharing. Couple of clarification 1. what's the formula for calculating the h1(X) 2. How did the split happen x23.

Hi Srinivas,The split was decided based on a simple approach. A tree with a split at x = 23 returned the least SSE during prediction. Hope this answers your question.Thanks & Regards, Ramya Bhaskar

Do you have your app for iOS?

yes

Great article.Can you just give a brief in the terms of regularization. What parameters get regularized? How the regularization happens in the case of multiple trees?

Very enlightening about the concept and interesting read. Can you brief me about loss functions?

Hi, I wish to propose to you a project using Xgboost with R. How can I contact you ? [email protected]

Thanks for sharing this great ariticle! Just have one clarification:h1 is calculated by some criterion(>23) on y-f0. In the resulted table, why there are some h1 with value 25.5 when y-f0 is negative (<23)?

Grate post! How this method treats outliers?

Its a great article. I have few clarifications:1. How MSE is calculated. 2. Could you please explain in detail about the graphs.

Understandable explanation of boosting. However a small question For one of the question above you said, the split is made at x<=23 because it provides least SSE. In your excel did you went thru each of the x value as split to find the min SSE? While using XGBoost will it take its own the value of x (for min SSE) or we need to provide?

What is 1nyin?

I guess the summation symbol is missing there. I took a while to understand what it must have been. So as the line says, that's the expression for mean, i= (Î£

^{1}_{n}y_{i})/nWow... You are awsome.. Thanks a lot for explaining in details...

A good interpretable explanation of the XGBoost algorithm! One thing I found confusing about the article is why we use a decision tree of depth 1 for making the first round of predictions and decision trees of depth 2 for making predictions on the residuals.

Thank you... It was very good article

Very clear. I am much grateful. There is one thing I could not wrap my head around. When devising h1(x) where the residuals are sorted according to x23, where does the criterion 23 come from?

Hello. May I clarify two points please: a) On the first step you calculate F0(x) as the argument that minimises the loss function. But isn't this the end? Once you have minimised the loss function why do you need to do anything else? b) On the 3rd step each hm(x) is fitted on the gradient obtained at each step. So you model the gradient? Would you please explain how exactly it is fitted? Thank you

Thank you for sharing quality content

Thank you for sharing quality content. keep sharing

Really clear explanation, thanks!