AI has the power to improve efficiency in various sectors such as business and healthcare. However, the lack of explainability hinders our reliance on using it for decision-making.
Should we trust the judgment of an algorithm?
It is important for decision-makers in any industry to understand the limitations and potential biases of machine learning models. To ensure that these models are behaving as intended, the output of any AI system should be explainable to a human.
In this article, we will go over the importance of explainability in AI. We will provide a brief overview of the types of methods used to derive explanations from machine learning models.
What is Explainable AI?
Explainable artificial intelligence or XAI refers to the techniques and methods used to allow humans to understand how machine learning models arrive at a certain output.
Many popular machine learning algorithms work as if it were a “black box”. In machine learning, black box algorithms refer to ML models where it is impossible to verify how a certain input leads to a particular output. Even the developer of the AI will not be able to fully explain how the algorithm works.
For example, deep learning algorithms use neural networks to identify patterns from a ton of data. Even though AI researchers and developers understand how neural networks work from a technical point of view, not even they can fully explain how a neural network came up with a particular result.
Some neural networks handle millions of parameters that all work in unison to return the final result.
In situations where decisions matter, the lack of explainability may become problematic.
Why Does Explainability Matter
Explainability provides insight into how models make decisions. Businesses that plan on adapting AI to make decisions will have to determine whether the AI has used the right input to arrive at the best decision.
Models that are unexplainable are an issue in several industries. For example, if a company were to use an algorithm to make hiring decisions, it would be to everyone’s best interest to have transparency in how the algorithm decides to reject an applicant.
Another field where deep learning algorithms are being used more frequently is in healthcare. In cases where algorithms try to detect possible signs of cancer, it is important for doctors to understand how the model arrived at a particular diagnosis. Some level of explainability is required for experts to take full advantage of AI and not blindly follow it
Overview of Explainable AI Algorithms
Explainable AI algorithms fall into two broad categories: self-interpretable models and post-hoc explanations.
Self-interpretable models are algorithms that a human can directly read and interpret. In this case, the model itself is the explanation.
Some of the most common self-interpretable models include decision trees and regression models.
For example, let’s consider a linear regression model that predicts house prices. A linear regression means that with some value x, we will be able to predict our target value y by applying a particular linear function f.
Suppose that our model uses lot size as the main input to determine house price. Using linear regression, we were able to come up with the function y = 5000 * x where x is the amount of square feet or lot size.
This model is human-readable and is completely transparent.
Post-hoc explanations are a group of algorithms and techniques that can be used to add explainability to other algorithms.
Most post-hoc explanation techniques do not need to understand how the algorithm works. The user only needs to specify the input and resulting output of the target algorithm.
These explanations are further divided into two kinds: local explanations and global explanations.
Local explanations aim to explain a subset of inputs. For example, given a particular output, a local explanation will be able to pinpoint which parameters contributed to making that decision.
Global explanations aim to produce post-hoc explanations of the entire algorithm. This type of explanation is typically more difficult to do. Algorithms are complex and there may be countless parameters that are significant in achieving the final result.
Examples of Local Explanation Algorithms
Among the many techniques used to achieve XAI, the algorithms used for local explanations are what most researchers focus on.
In this section, we’ll take a look at some popular local explanation algorithms and how each of them work.
LIME (Local Interpretable Model-Agnostic Explainer) is an algorithm that can explain the predictions of any machine learning algorithm.
As the name implies, LIME is model-agnostic. This means LIME could work for any type of model. The model is also locally interpretable, meaning we can explain the model using local results rather than explaining the entire model.
Even if the model being explained is a black box, LIME creates a local linear model around points nearby a certain position.
LIMe provides a linear model that approximates the model in the vicinity of a prediction but not necessarily globally.
You can learn more about this algorithm by visiting this open-source repository.
Shapley Additive Explanations (SHAP) is a method to explain individual predictions. To understand how SHAP works, we’ll have to explain what Shapley values are.
The Shapley value is a concept in game theory that involves assigning a “value” to each player in the game. This is distributed such that the value assigned to each player is based on the player’s contribution to the game.
How do we apply game theory to machine learning models?
Suppose that each feature in our model is a “player” and that the “game” is the function that outputs the prediction.
The SHAP method creates a weighted linear model that assigns Shapley values to various features. Features with high Shapley values have a greater influence on the outcome of the model while features with low Shapley values have less of an impact.
AI explainability is important not only for ensuring the fairness and accountability of AI systems, but also for building trust in AI technology in general.
There is still plenty of research to be done in the area of AI explainability, but there are some promising approaches that can help us understand the complex black box AI systems that are already widely used today.
With further research and development, we can hope to build AI systems that are more transparent and easier to understand. In the meantime, businesses and experts in fields such as healthcare should be aware of the limitations of AI explainability.