Explainability on XAI Today

Revisiting the Rashomon Set Argument

Wed, 26 Feb 2025 00:00:00 +0000

About eighteen months ago, I posted about this paper discussing the Accuracy-Interpretability Trade-Off (AITO) or Performance-Explainability Trade-Off (PET). This paper revisited the sometimes overlooked debate over the validity of this trade-off. That is to say, is it even necessary to accept that such a trade-off or dichotomy exists? Are we really forced to choose between an accurate model and an interpretable one, or must we always compromise our target metrics? You can read my previous blog here

One of the arguments against accepting the Trade-Off is the so-called Rashomon Set (RS) argument. The RS argument suggests that, for many real-world tasks, multiple models from a single function class can achieve nearly the same level of performance. Within this set of models, some will be inherently interpretable. This idea, named after Breiman’s Rashomon Effect, has been discussed extensively but never finally settled. The reason stems from the fact that finding an optimal model is an NP-hard problem and this is at the foundation of machine learning. We approximate a near optimal solution through risk minimization, a paradigm that encourages the thinking that our single, finally selected model is the best we can do. Decades of research into ensemble models hasn’t changed that, because the ensemble takes the seat of a single, risk-minimized model. Rashomon Set theoretical research throws out this limiting paradigm in favour of an exploration of many near optimal models in the space of all possible models in a single function class.

In their paper Exploring the Whole Rashomon Set of Sparse Decision Trees, Xin et al. develop a dynamic programming-based method to generate and sample from the RS of sparse decision trees derived from several benchmark datasets. Three novel applications are presented, including a fascinating take on RS-derived variable importance. Most importantly, there is proof that traditional tree ensemble methods generate only a fraction of the RS several orders of magnitude smaller than its theoretical maximum size.

The question of RS-derived feature importance is explored in fine detail in Partial Order in Chaos: Consensus on Feature Attributions in the Rashomon Set. The authors address the variability in feature attribution explanations. Those of us who have worked with foundational models, such as Random Forests and Boosting methods are familiar with their capability to provide feature importance measures. We are, however, inevitably frustrated by the inconsistency of feature importance across model classes that differ in only trivial ways, and even within multiple runs of the same model while merely adjusting the random seed. In this paper, Laberge et al. show that the same is true even for theoretically stable methods, such as SHAP (Lundberg and Lee, 2017) in a trivial example with a simulated data set for which the ground truth explanation is known. They go on to propose a framework for achieving much more consistent measures of variable importance that is based on consensus within the Rashomon Set.

In the paper On the Existence of Simpler Machine Learning Models, Semanova et al. proposed the Rashomon Ratio as a means to estimate the opportunity of finding a highly interpretable model for any given problem. However, these methods are limited to the Sparse Decision Tree model class and cannot be adapted to incorporate other foundational models that aren’t based on recursive partitioning of binary features. A linear model, for example, with continuous features cannot have its RS enumerated by the application of combinatorics.

The Rashomon Set argument is compelling. So far, however, the strongest evidence remains empirical rather than axiomatic. A formal theoretical proof remains elusive. Nevertheless, ongoing research continues to explore the conditions and methodologies that facilitate the identification of interpretable models within the Rashomon set and, as such, the RS argument remains a fascinating question of theoretical machine learning research.

Evaluating the Influences of Explanation Style on Human-AI Reliance

Fri, 01 Nov 2024 00:00:00 +0000

The reccent paper “Evaluating the Influences of Explanation Style on Human-AI Reliance” investigates how different types of explanations affect human reliance on AI systems. The research focused on three explanation styles: feature-based, example-based, and a combined approach, with each style hypothesized to influence human-AI reliance in unique ways. A two-part experiment with 274 participants explored how these explanation styles impact reliance and interpretability in a human-AI collaboration task, specifically using a bird identification task. The study sought to address mixed evidence from previous literature on whether certain explanation styles reduce over-reliance on AI or improve human decision-making accuracy.

To study human responses to various AI explanations, the researchers used a quantitative methodology, measuring reliance through initial and final decision accuracy shifts. The study employed the Judge-Advisor System (JAS) model to capture differences in human reliance before and after AI assistance. Key measures included the Appropriateness of Reliance (AoR) framework, developed by Schemmer et al., which introduced two metrics: Relative AI Reliance (RAIR) and Relative Self-Reliance (RSR). These metrics quantified reliance by assessing how often humans appropriately switched to AI-supported decisions or maintained their initial, correct judgments. The researchers noted individual participant performance variations, revealing that higher-performing individuals demonstrated different reliance patterns compared to lower-performing ones, particularly when interacting with high-complexity tasks.

The quantitative approach included metrics from the AoR model and the JAS framework. Both RAIR and RSR metrics from AoR provided a structured comparison across explanation styles by evaluating the effect of explanations on reliance in human-AI interactions. While the reliance metrics were based on existing literature, this study extended their use by separating participants based on individual performance, creating a novel analysis approach. Additionally, accuracy shift measures captured how reliance on AI suggestions varied with task complexity and participant ability. This nuanced view highlighted reliance discrepancies based on cognitive engagement, suggesting that explanation styles should be tailored to user expertise and task requirements.

The paper emphasizes the importance of Explainable AI (XAI) for human-in-the-loop tasks, where humans need to understand and trust AI recommendations effectively. Such explanations can calibrate user trust, ideally preventing over-reliance on incorrect AI outputs. XAI’s value is underscored in collaborative tasks, but this study’s focus on bird classification, although useful for understanding complex identification tasks, may not directly apply to more general, real-world applications. The research reveals challenges in establishing broad applicability for explanation methods due to the inherent limitations in specific experimental tasks that may not fully capture the varied decision-making contexts encountered in real-life scenarios.

Despite these limitations, the study makes a significant contribution to the ongoing discourse on XAI by providing evidence that certain explanation forms (example-based, feature-based, or combined) affect reliance differently. This research highlights that example-based explanations, though beneficial for identifying incorrect AI suggestions, can also foster over-reliance, particularly when high-quality explanations reinforce trust. The paper suggests that balancing clarity and trust calibration remains an open question, crucial for advancing reliable human-AI collaboration frameworks.

In summary, this work advances the conversation on XAI by demonstrating that different explanation styles yield complex, context-dependent effects on human reliance. Although the field has matured in recent years, debates persist about the ideal form and substance of explanations. This study contributes to understanding the nuanced roles of example- and feature-based explanations, highlighting that while explanations enhance interpretability, they do not uniformly improve reliance, particularly across varied user expertise levels and decision contexts. This reinforces the need for adaptable XAI methods that align with diverse human-AI collaboration needs.

Explainable AI for Improved Heart Disease Prediction

Tue, 09 Jul 2024 00:00:00 +0000

The paper “Optimized Ensemble Learning Approach with Explainable AI for Improved Heart Disease Prediction” focuses on explaining machine learning models in healthcare, similar to my original work in “Ada-WHIPS: explaining AdaBoost classification with applications in the health sciences”. The newer paper combines a novel Bayesian method to optimally tune the hyper-paremeters of ensemble models such as AdaBoost, XGBoost and Random Forest and then applies the now well established SHAP method to assign Shapley values to each feature. The authors use their method to analyse three heart disease prediction datasets, included the well-known Cleveland set used as a benchmark in many ML research papers.

SHAP (Lundberg and Lee) came hot on the heels of the revolutionary LIME method (Ribeiro, Singh and Guestrin), which together delivered a paradigm shift in the usefulness and feasibility of eXplainable Artificial Intelligence (XAI). In fact, LIME was published at exactly the time I was becoming interested in the topic of XAI and served as inspiration for my own Ph.D journey. Both methods fall into the category of Additive Feature Attribution Methods (AFAM) and work by assign a unitless value to each level of the set of input features. The main benefits of AFAM become clear when viewing a beeswarm plot of their responses across a larger dataset, such as the whole training data. Patterns emerge showing which input variables affect the response variable most strongly, and in which direction. This usage is much more sophisticated than classic variable importance plots, which lack direction and mathematical guarantees offered by SHAP.

In the clinical setting, these mathematical guarentees mean that the resulting variable sensitivity information could be used to create a broader diagnostic tool. However, while this approach can provide a general understanding of which variables drive a model’s predictions, it lacks the fine-grained, instance-specific clarity offered by perfect fidelity, decompositional methods.

On the other hand, my original method Ada-WHIPS (firmly within the decompositional methods category) enhances interpretability in clinical settings by providing direct, case-specific explanations, making it a powerful tool for clinicians needing detailed transparency for patient-specific decision-making. Given the choice of an AdaBoost model (or a Gradient Boosted Model, or a Random Forest), it makes sense to use an XAI method that is highly targeted to these decomposable ensembles. Ada-WHIPS digs deep into the internal structure of AdaBoost models, redistributing the adaptive classifier weights generated during model training (and therefore a function of the training data distribution) to extract interpretable rules at the decision node level.

One area where Ada-WHIPS could benefit from the techniques in the new paper is the use of Bayesian methods to tune hyperparameters. Their approach potentially leads to improved model accuracy, a crucial factor in high-stakes environments like healthcare and “juicing up” the model internals for greater accuracy in the generated decision nodes. However, the paper appears to omit any detail about how this approach is deployed. This omission is indeed a great pity because, from what I understood, the Bayesian parameter selection was actually the authors’ novel contribution (the use of ensembles and SHAP on these particular datasets being nothing particularly new).

In conclusion, the SHAP-based approach offers valuable insights at a macro level, the new paper boasts improvements in model accuracy through Bayesian tuning, and my Ada-WHIPS method’s per-instance clarity and actionable insights should prove practical in scenarios where clinicians require detailed explanations of specific cases. I would be delighted to see some confluence of the three ideas, so that the benefits from each can combine and reinforce the use of highly targeted explainability in clinical applications.

Revisiting the Performance-Explainability Trade-Off

Fri, 01 Sep 2023 00:00:00 +0000

I was very excited to read and review the paper Revisiting the Performance-Explainability Trade-Off in Explainable Artificial Intelligence (XAI) last month. I wrote an extensive section on this topic for my Ph.D. thesis (although I coined the name Accuracy-Interpretability Trade-Off or AITO). I have always felt that the subject is too rarely discussed, and never in enough depth and scientific rigour. This Performance-Explanability Trade-off (PET) in the notion that improving model performance (by this, they must mean accuracy or related measures such as true positive rate or AUC/ROC) comes at the cost of explainability.

The authors of this paper state their goal as wanting to refine the discussion of PET in the field of Requirements Engineering for AI systems. Frankly, the paper is quite generic with respect to this self-stated niche once the text gets going, although that in no way detracts from their position on the topic itself. For the most part, the paper focuses on Cynthia Rudin’s influential critique of the performance-explainability trade-off. Rudin is described by the authors as being particularly critical of post-hoc explainability techniques, arguing that they can produce misleading or incomplete explanations that fail to remain faithful to the model’s decision-making process. Again, this was also a foundational point in my thesis on XAI; what good is an explanation of an output other than what the model gave? Proxy (simplified) explanatory models are particularly prone to this behaviour.

Rudin also contends that interpretable models can often match the performance of black-box models, provided that sufficient effort is invested in knowledge discovery and feature engineering. This phenomenon is known as the Rashomon Set argument, which posits that for many real-world tasks, there exist multiple high-performing models, including some that are inherently explainable. The authors argue that while this is an intriguing theoretical claim, it lacks strong empirical backing and does not guarantee that such explainable models will be easily identifiable or practical to develop in all domains. On this point, I find myself in total agreement with the authors. The Rashomon Set argument is merely conjecture from what I can tell and it’s something I would like to revisit in a future blog post.

The authors’ strongest arguments lie in the fact that analyst/researcher-led feature engineering has been vastly superseded and overpowered by the capabilities of deep learning, which hinges on a feature self-learning paradigm built into the model. It’s just so much faster to build a deep neural layer, that researcher/analyst time a resource can be freed up and expended on making post-hoc explainability much for feasible. The authors argue that the real issue is not just whether performance and explainability are in tension, but how much effort is required to achieve both. They suggest that model development should be viewed as a multi-objective optimization problem, where teams must balance the trade-offs between performance, explainability, and available resources, while also considering domain-specific risks such as ethical concerns or financial constraints. From this more nuanced position, they are able to derive an extended framework called PET+ (Performance-Explainability-Time trade-off), which incorporates time and resource constraints into the equation.

I appreciate and commend the reflection on this chronically overlooked and misunderstood topic and hope that their paper contributes towards frameworks for evaluating modeling approaches in the future.

Algebraic Aggregation of Random Forests

Thu, 10 Aug 2023 00:00:00 +0000

In my paper, “CHIRPS: Explaining random forest classification”, I took an empirical approach to addressing model transparency by extracting rules that make Random Forest (RF) models more interpretable. Importantly, this was done without sacrificing the high levels of accuracy achieved by the high-performing RF models.

The recently published “Algebraic aggregation of random forests: towards explainability and rapid evaluation” by Gossen and Steffen provides a theoretical counterpart, offering essential proofs and a mathematical framework for achieving explainability with RF models.

While my paper focused on simplifying complex models by rule extraction on a per instance basis, this subsequent work introduces Algebraic Decision Diagrams (ADDs) to aggregate Random Forests, optimizing their structure and enhancing interpretability at the model level. Both papers aim to improve model transparency, though by different means: my approach is empirical, leveraging rule extraction to clarify black-box models, whereas the latter introduces algebraic methods to combine decision trees into efficient, understandable diagrams.

The mathematical concepts in Gossen and Steffen’s paper, such as path reduction and algebraic operations, support model simplification. Importantly, the authors provide formal proofs that this aggregation retains the original model’s accuracy. This complements the practical focus in my paper, where the goal was also to maintain accuracy while increasing explainability.

Ultimately, the two papers reach the same destination—improving transparency of RF models—but by different routes. While my paper uses rule extraction to bring clarity to complex models, the subsequent work constructs a theoretical basis using algebraic tools, providing formal assurances to the outcomes I demonstrated empirically. Together, they offer complementary perspectives on making RF models more understandable and efficient.

Explaining Random Forests with Representative Trees

Thu, 15 Jun 2023 00:00:00 +0000

The paper “Can’t see the forest for the trees: Analyzing groves to explain random forests” explores a novel take on model-specific explanations, as outlined in my own research (e.g. you can look at “CHIRPS: Explaining random forest classification” as a reference). This new paper by Szepannek and von Holt seeks to make Random Forests (RF) more interpretable. RF are notoriously hard to explain due to their complexity and these novel methods works well for both classification and regression, which is a very useful extension to the field.

The authors introduce most representative trees (MRT) and surrogate trees, essentially distilling a simpler model to run side by side with the black box RF. MRTs focus on highlighting individual trees within a random forest that best explain the overall model behavior, while surrogate trees mimic the forest with simpler, more digestible versions. I have some reservations about the latter approach, because my own research showed that any surrogate model comes with a failure rate, which is the number of examples that the surrogate classifies differently than the black box model under scrutiny. I also question the assertion that a model of 10 or 24 decision trees really is so interpretable. Even a model of this reduced size still likely contains far too many components for a human-in-the-loop to consider and understand.

In any case, to give the authors their due credit, they navigate the trade-offs between accuracy and interpretability of both MRT and surrogate tree methods, and propose a novel concept called groves: small collections of decision trees that balance the need for interpretability with predictive accuracy. Groves provide a middle ground by combining the benefits of MRTs and surrogate models, reducing the overall complexity while still offering meaningful insights into how the model operates. This approach aligns with the goal of making models more transparent and trustworthy.

Through various case studies, the paper shows how groves and surrogate trees can be effectively applied to real-world datasets. The trade-off between model accuracy and explainability remains a central challenge. Yet, in these studies, groves provide a workable compromise by making it easier for humans to understand what is driving the model’s predictions without overwhelming them with unnecessary detail.

The discussion also highlights a key challenge in using groves: deciding on the right number of trees to use for explanation. Using too many trees risks overwhelming the user with information (as I have already pointed out), while too few might fail to capture the complexity of the underlying model and run with an untenable failure rate. I dicuss ways to achieve a zero failure rate in my thesis. Keeping explanations concise and accessible is just a part of the complete picture.

In conclusion, this paper underscores the crucial need for enhancing the interpretability of machine learning models, particularly in high-stakes fields like healthcare and finance, where decision transparency is essential. By extending the work in interpretability through methods like groves and surrogate trees, it addresses the challenge of making powerful models like random forests more understandable.

Explaining Random Forests with Boolean Satisfiability

Mon, 21 Jun 2021 00:00:00 +0000

The paper “On Explaining Random Forests with SAT” uses Boolean satisfiability (SAT) methods to provide a formal framework for generating explanations of Random Forest (RF) predictions. A key result in the paper is that abductive explanations (AXp) and contrastive explanations (CXp) can be derived by encoding the RF’s decision paths into propositional logic.

Encoding a decision path as propositional logic, is an entirely reasoned approach and quite straightforward, as I showed in my paper CHIRPS: Explaining random forest classification. The decision paths of an RF model can be transformed into a Boolean formula in Conjunctive Normal Form (CNF). For example, each decision tree in the forest is represented as a set of clauses. Following the paths for a single example prediction essentially carves out a region of the feature space with a set of step functions, resulting in a sub-region that must return the target response. When the clauses of this step functions set correspond to a subset of the features, a change in the remaining feature inputs has no effect on the model prediction. This subset is a prime implicant (PI) explanation.

A PI-explanation is a minimal subset of features that are sufficient to guarantee a particular prediction made by a machine learning model. It represents the smallest set of conditions that, if held constant, would lead to the same classification result. Essentially, it’s the most concise explanation of why the model arrived at its decision, highlighting the critical features responsible for that prediction. In fact, my own research centred finding soft PI-explanations, and revealing the limits where they no longer hold true for extreme outliers and unusual examples.

The authors of this paper show that finding AXp and CXp by this PI-explanation method reduces to solving a SAT problem and is therefore NP-hard but can be polynomial under specific conditions. This insight into the problem complexity is significant because it establishes that generating explanations is feasible when those assumptions are met and opens up the method to practical applications with real-world data.

Overall, the SAT-based methodology enables a structured, efficient way to uncover the decision-making process of Random Forests, ensuring that their predictions are not just accurate but also explainable, which is crucial for domains requiring transparency like healthcare and finance.

Counterfactual Explanations Help Identify Sources of Bias

Sun, 08 Mar 2020 00:00:00 +0000

By the end of 2020, the topic of eXplainable Artificial Intelligence (XAI) has become quite mainstream. One important developlment is counterfactual explanations, which (among other benefits) can to identify and reduce bias in machine learning models. Counterfactual explanations provide insights by showing how minimal changes in input features can alter model predictions. This approach has been crucial in exposing biased behavior, especially in sensitive applications like credit scoring or hiring. By identifying how protected attributes (e.g., gender or race) affect outcomes, practitioners could better address and mitigate unfair biases in AI systems (Verma et al., 2020).

Reference: Verma, S., & Rubin, J. (2020). Fairness Definitions Explained. Proceedings of the 2020 ACM/IEEE International Workshop on Software Fairness.

International Women's Day 2020

Sun, 08 Mar 2020 00:00:00 +0000

Profile: Cynthia Rudin #

Today, for International Women’s Day, I wanted to share my huge respect for Cynthia Rudin. She is a leading academic in the research field in which I am currently involved for my PhD - interpretability in machine learning. Her work is very widely cited and comes up in all searches related to solving the “black box” problem of machine learning. She is a true thought leader.

Here is excellent podcast where Cynthia succinctly and eruditely explains the current state, problems and solutions of ML interpretability. It’s well worth a listen and, I believe, entirely understandable by non-expert listeners.

Here are a couple of profile sites where you can find out more about this brilliant and accomplished woman who is a true thought leader in Machine Learning and Data Science.

https://en.wikipedia.org/wiki/Cynthia_Rudin

https://users.cs.duke.edu/~cynthia/

Whenever I can - and not only on IWD - I try to raise the profile of prominent women in STEM (science, technology, engineering and mathematics). Under-representation and a lack of visible role-models is one barrier to increasing diversity and equality in any field. Certainly, there is no lack of talented women in STEM but if young women and girls at school aren’t given examples of successful women, this can limit aspirations at the grass roots level.