guest_blog — Published On January 7, 2021 and Last Modified On June 6th, 2023
Intermediate Machine Learning
Explainable AI

Can you explain how your model works?

Meet XAI!

Artificial intelligence techniques are used to solve real-world problems. We get the data, perform some operations to make it clean & ready for the following processes.

We basically pick things from this world and take them into the world of machines, represent it with numbers, and then feed it to a bunch of models. Try to improve them and eventually ”the winner model” gets the test data. A vital question comes to the minds :

” How do we take this result back to real world ? ”

What is Explainable AI? (with a cooler name: XAI)

A formal definition: According to Wikipedia, Explainable AI refers to methods and techniques in the application of artificial intelligence technology such that the results of the solution can be understood by humans. [1]

In the early phases of AI adoption, it was okay to not understand what the model predicts in a certain way, as long as it gives the correct outputs. Explaining how they work was not the first priority. Now, the focus is turning to build human interpretable models.

Three important aspects of model interpretation are:
1. Transparency
2. The ability to question
3. The ease of understanding .[2]

Model interpretability can be examined in two levels:

  • Global Interpretation: Examines the model from a broader perspective. For example, let’s say we are working on a house price dataset and we implemented a neural network. The global interpretation might say “Your model uses # of squared feet as an important feature to derive predictions”
  • Local Interpretation: As the name suggests, this approach is focused on a certain observation/data point. Let’s continue moving forward with our example. Prediction for a really small house turned out large. Local interpretation looks at the other features and it might say “Your model predicted this way because the location of the house is very close to the city center.”
Interpretation scope explainable AI
           Source: Sri Ambati, Get Hands on MLI

The Trade-off Between Accuracy and Interpretability

In the industry, you will often hear that business stakeholders tend to prefer models that are more interpretable like linear models (linear\logistic regression) and trees which are intuitive, easy to validate, and explain to a non-expert in data science. [2]

In contrast, when we look at the complex structure of real-life data, in the model building & selection phase, the interest is mostly shifted towards more advanced models. That way, we are more likely to obtain improved predictions.

Models like these (ensembles, neural networks, etc.) are called black-box models. As the model gets more advanced, it becomes harder to explain how it works. Inputs magically go into a box and voila! We get amazing results.

But, HOW?

When we suggest this model to stakeholders, will they completely trust it and immediately start using it? NOThey will ask questions and we should be ready to answer them.

Why should I trust your model?

Why did the model take a certain decision?

What drives model predictions?

We should consider both improving our model accuracy and not get lost in the explanation. There should be a balance between both.

Resource: DPhi Advanced ML Bootcamp — Explainable AI [2]

Source: DPhi Advanced ML Bootcamp — Explainable AI [2]

Here, I would like to share a sentence from Dipanjan Sarkar’s medium post about explainable AI:

Any machine learning model at its heart has a response function which tries to map and explain relationships and patterns between the independent (input) variables and the dependent (target or response) variable(s). [3]

So, models take inputs and process them to get outputs. What if our data is biased? It will also make our model biased and therefore untrustworthy. It is important to understand & be able to explain to our models so that we can also trust their predictions and maybe even detect issues and fix them before presenting them to others.

To improve the interpretability of our models, there are various techniques some of which we already know and implement. Traditional techniques are exploratory data analysis, visualizations, and model evaluation metrics. With the help of them, we can get an idea of the model’s strategy. However, they have some limitations. To learn more about traditional ways and their limitations, check out this amazing article by Dipanjan Sarkar.[4]

Other model interpretation techniques and libraries have been developed to overcome limitations. Some of these are :

  • LIME ( Local Interpretable Model-Agnostic Explanations)
  • SHAP (Shapley Additive Explanations)
  • ELI5 (Explain Like I’m 5)

These libraries use feature importance, partial dependence plots, individual conditional expectation plots to explain less complex models such as linear regression, logistic regression, decision trees, etc.

Feature importance shows how a feature is important for the model. In other words, when we delete the feature from the model, how our error changes? If the error increases a lot, this means that a feature is important for our model to predict the target variable.

feature importance

Source: Machine Learning Mastery, XGBoost Feature Importance Bar Chart

Partial dependence plots visualize the effect of the change for a certain feature when everything else is held constant (with a cooler phrase: ceteris paribus). With the help of these, we can see a possible limit value, where this value is exceeded, it directs the model predictions the other way. When we are visualizing partial dependence plots, we are examining the model globally.

Source: Dipanjan (DJ) Sarkar, Model Interpretation Strategies

Individual conditional expectation plots show the effect of changes for a certain feature, just like partial dependency plots. But this time, the point of view is local. We are interested to see the effect of changes for a certain feature for all instances in our data. A partial dependence plot is the average of the lines of an ICE plot.[5]

individual conditional expectation plots

Source: Christoph Molnar, Interpretable Machine Learning- A Guide for Making Black Box Models Explainable

When it comes to explaining more advanced models, model-agnostic (does not depend on the model) techniques are used.

Global surrogate models take the original inputs and your black-box machine learning predictions. When this new dataset is used to train and test the appropriate global surrogate model (more interpretable models such as linear model, decision tree, etc.), it basically tries to mimic your black-box model’s predictions. By interpreting and visualizing this “easier” model, we get a better understanding of how our actual model predicts in a certain way.

Other interpretability tools are LIME, SHAP, ELI5, and SKATER libraries. We will talk about them in the next post, over a guided implementation. Until then, I am sharing some amazing resources I used to form this post along with some extra links. Stay tuned for the next post, see you there!

Happy learning!

Frequently Asked Questions

Q1. What are the 4 principles of explainable AI?

A. The field of explainable AI aims to provide transparency and understanding of how AI systems make decisions. While there are various principles and frameworks proposed, here are four commonly discussed principles of explainable AI:
1. Transparency: The system’s decision-making process should be transparent and understandable to users. This involves providing explanations in a human-interpretable form, such as highlighting important features or providing rule-based explanations.
2. Interpretability: The system should provide insights into the internal workings and logic behind its decisions. This can involve revealing the model’s structure, feature importance, or showcasing the relationship between input variables and output predictions.
3. Accountability: The AI system should be designed to take responsibility for its decisions and actions. This includes tracking and recording decision-making processes, ensuring proper governance, and potentially allowing for recourse or redress in case of erroneous or biased outcomes.
4. Fairness and Bias Mitigation: AI systems should strive to mitigate bias and ensure fairness in decision-making. This involves identifying and addressing biases in training data, monitoring and evaluating the system’s behavior for discriminatory patterns, and taking measures to ensure equitable outcomes for different groups.
These principles aim to promote transparency, trust, and ethical use of AI by enabling humans to understand and evaluate the decisions made by AI systems.

Q2. How does explainable AI work?

A. Explainable AI employs various techniques to provide insights into the decision-making process of AI systems. One approach is to use model-agnostic methods, such as feature importance analysis, rule extraction, or surrogate models, which can be applied to any black-box model. Alternatively, specific models, like decision trees or rule-based systems, inherently provide interpretability. Recent advancements include generating textual or visual explanations based on attention mechanisms or saliency maps. These explanations highlight the important features or factors that influenced the model’s output. Overall, explainable AI combines algorithmic approaches, interpretability techniques, and human-centered design to enhance transparency and understanding of AI systems.


[1] Wikipedia, Explainable AI,

[2] DPhi Tech, Explainable AI Course,

[3] Dipanjan (DJ) Sarkar, The Importance of Human Interpretable Machine Learning,

[4] Dipanjan (DJ) Sarkar, Model Interpretation Strategies,

[5] Christoph Molnar, Interpretable Machine Learning- A Guide for Making Black Box Models Explainable, 2019,

Additional Resources

  1. DPhi Tech, Importance of Human Interpretable models & Explainable A.I,
  2. Sci-kit Learn Documentation, Partial Dependence Plots,
  3. Yellowbrick Documentation, Feature Importances,

About the Author

Semanur Kapusızoğlu

Recent graduate Industrial engineer aiming for a career in Data Science. Fascinated by how much one can do with data and determined to make an impact using this. Current research area: NLP – Deep Learning



The media shown in this article are not owned by Analytics Vidhya and is used at the Author’s discretion.

Leave a Reply Your email address will not be published. Required fields are marked *