Home on XAI Today

Revisiting the Rashomon Set Argument

Wed, 26 Feb 2025 00:00:00 +0000

About eighteen months ago, I posted about this paper discussing the Accuracy-Interpretability Trade-Off (AITO) or Performance-Explainability Trade-Off (PET). This paper revisited the sometimes overlooked debate over the validity of this trade-off. That is to say, is it even necessary to accept that such a trade-off or dichotomy exists? Are we really forced to choose between an accurate model and an interpretable one, or must we always compromise our target metrics? You can read my previous blog here

One of the arguments against accepting the Trade-Off is the so-called Rashomon Set (RS) argument. The RS argument suggests that, for many real-world tasks, multiple models from a single function class can achieve nearly the same level of performance. Within this set of models, some will be inherently interpretable. This idea, named after Breiman’s Rashomon Effect, has been discussed extensively but never finally settled. The reason stems from the fact that finding an optimal model is an NP-hard problem and this is at the foundation of machine learning. We approximate a near optimal solution through risk minimization, a paradigm that encourages the thinking that our single, finally selected model is the best we can do. Decades of research into ensemble models hasn’t changed that, because the ensemble takes the seat of a single, risk-minimized model. Rashomon Set theoretical research throws out this limiting paradigm in favour of an exploration of many near optimal models in the space of all possible models in a single function class.

In their paper Exploring the Whole Rashomon Set of Sparse Decision Trees, Xin et al. develop a dynamic programming-based method to generate and sample from the RS of sparse decision trees derived from several benchmark datasets. Three novel applications are presented, including a fascinating take on RS-derived variable importance. Most importantly, there is proof that traditional tree ensemble methods generate only a fraction of the RS several orders of magnitude smaller than its theoretical maximum size.

The question of RS-derived feature importance is explored in fine detail in Partial Order in Chaos: Consensus on Feature Attributions in the Rashomon Set. The authors address the variability in feature attribution explanations. Those of us who have worked with foundational models, such as Random Forests and Boosting methods are familiar with their capability to provide feature importance measures. We are, however, inevitably frustrated by the inconsistency of feature importance across model classes that differ in only trivial ways, and even within multiple runs of the same model while merely adjusting the random seed. In this paper, Laberge et al. show that the same is true even for theoretically stable methods, such as SHAP (Lundberg and Lee, 2017) in a trivial example with a simulated data set for which the ground truth explanation is known. They go on to propose a framework for achieving much more consistent measures of variable importance that is based on consensus within the Rashomon Set.

In the paper On the Existence of Simpler Machine Learning Models, Semanova et al. proposed the Rashomon Ratio as a means to estimate the opportunity of finding a highly interpretable model for any given problem. However, these methods are limited to the Sparse Decision Tree model class and cannot be adapted to incorporate other foundational models that aren’t based on recursive partitioning of binary features. A linear model, for example, with continuous features cannot have its RS enumerated by the application of combinatorics.

The Rashomon Set argument is compelling. So far, however, the strongest evidence remains empirical rather than axiomatic. A formal theoretical proof remains elusive. Nevertheless, ongoing research continues to explore the conditions and methodologies that facilitate the identification of interpretable models within the Rashomon set and, as such, the RS argument remains a fascinating question of theoretical machine learning research.

Evaluating the Influences of Explanation Style on Human-AI Reliance

Fri, 01 Nov 2024 00:00:00 +0000

The reccent paper “Evaluating the Influences of Explanation Style on Human-AI Reliance” investigates how different types of explanations affect human reliance on AI systems. The research focused on three explanation styles: feature-based, example-based, and a combined approach, with each style hypothesized to influence human-AI reliance in unique ways. A two-part experiment with 274 participants explored how these explanation styles impact reliance and interpretability in a human-AI collaboration task, specifically using a bird identification task. The study sought to address mixed evidence from previous literature on whether certain explanation styles reduce over-reliance on AI or improve human decision-making accuracy.

To study human responses to various AI explanations, the researchers used a quantitative methodology, measuring reliance through initial and final decision accuracy shifts. The study employed the Judge-Advisor System (JAS) model to capture differences in human reliance before and after AI assistance. Key measures included the Appropriateness of Reliance (AoR) framework, developed by Schemmer et al., which introduced two metrics: Relative AI Reliance (RAIR) and Relative Self-Reliance (RSR). These metrics quantified reliance by assessing how often humans appropriately switched to AI-supported decisions or maintained their initial, correct judgments. The researchers noted individual participant performance variations, revealing that higher-performing individuals demonstrated different reliance patterns compared to lower-performing ones, particularly when interacting with high-complexity tasks.

The quantitative approach included metrics from the AoR model and the JAS framework. Both RAIR and RSR metrics from AoR provided a structured comparison across explanation styles by evaluating the effect of explanations on reliance in human-AI interactions. While the reliance metrics were based on existing literature, this study extended their use by separating participants based on individual performance, creating a novel analysis approach. Additionally, accuracy shift measures captured how reliance on AI suggestions varied with task complexity and participant ability. This nuanced view highlighted reliance discrepancies based on cognitive engagement, suggesting that explanation styles should be tailored to user expertise and task requirements.

The paper emphasizes the importance of Explainable AI (XAI) for human-in-the-loop tasks, where humans need to understand and trust AI recommendations effectively. Such explanations can calibrate user trust, ideally preventing over-reliance on incorrect AI outputs. XAI’s value is underscored in collaborative tasks, but this study’s focus on bird classification, although useful for understanding complex identification tasks, may not directly apply to more general, real-world applications. The research reveals challenges in establishing broad applicability for explanation methods due to the inherent limitations in specific experimental tasks that may not fully capture the varied decision-making contexts encountered in real-life scenarios.

Despite these limitations, the study makes a significant contribution to the ongoing discourse on XAI by providing evidence that certain explanation forms (example-based, feature-based, or combined) affect reliance differently. This research highlights that example-based explanations, though beneficial for identifying incorrect AI suggestions, can also foster over-reliance, particularly when high-quality explanations reinforce trust. The paper suggests that balancing clarity and trust calibration remains an open question, crucial for advancing reliable human-AI collaboration frameworks.

In summary, this work advances the conversation on XAI by demonstrating that different explanation styles yield complex, context-dependent effects on human reliance. Although the field has matured in recent years, debates persist about the ideal form and substance of explanations. This study contributes to understanding the nuanced roles of example- and feature-based explanations, highlighting that while explanations enhance interpretability, they do not uniformly improve reliance, particularly across varied user expertise levels and decision contexts. This reinforces the need for adaptable XAI methods that align with diverse human-AI collaboration needs.

Explainable AI for Improved Heart Disease Prediction

Tue, 09 Jul 2024 00:00:00 +0000

The paper “Optimized Ensemble Learning Approach with Explainable AI for Improved Heart Disease Prediction” focuses on explaining machine learning models in healthcare, similar to my original work in “Ada-WHIPS: explaining AdaBoost classification with applications in the health sciences”. The newer paper combines a novel Bayesian method to optimally tune the hyper-paremeters of ensemble models such as AdaBoost, XGBoost and Random Forest and then applies the now well established SHAP method to assign Shapley values to each feature. The authors use their method to analyse three heart disease prediction datasets, included the well-known Cleveland set used as a benchmark in many ML research papers.

SHAP (Lundberg and Lee) came hot on the heels of the revolutionary LIME method (Ribeiro, Singh and Guestrin), which together delivered a paradigm shift in the usefulness and feasibility of eXplainable Artificial Intelligence (XAI). In fact, LIME was published at exactly the time I was becoming interested in the topic of XAI and served as inspiration for my own Ph.D journey. Both methods fall into the category of Additive Feature Attribution Methods (AFAM) and work by assign a unitless value to each level of the set of input features. The main benefits of AFAM become clear when viewing a beeswarm plot of their responses across a larger dataset, such as the whole training data. Patterns emerge showing which input variables affect the response variable most strongly, and in which direction. This usage is much more sophisticated than classic variable importance plots, which lack direction and mathematical guarantees offered by SHAP.

In the clinical setting, these mathematical guarentees mean that the resulting variable sensitivity information could be used to create a broader diagnostic tool. However, while this approach can provide a general understanding of which variables drive a model’s predictions, it lacks the fine-grained, instance-specific clarity offered by perfect fidelity, decompositional methods.

On the other hand, my original method Ada-WHIPS (firmly within the decompositional methods category) enhances interpretability in clinical settings by providing direct, case-specific explanations, making it a powerful tool for clinicians needing detailed transparency for patient-specific decision-making. Given the choice of an AdaBoost model (or a Gradient Boosted Model, or a Random Forest), it makes sense to use an XAI method that is highly targeted to these decomposable ensembles. Ada-WHIPS digs deep into the internal structure of AdaBoost models, redistributing the adaptive classifier weights generated during model training (and therefore a function of the training data distribution) to extract interpretable rules at the decision node level.

One area where Ada-WHIPS could benefit from the techniques in the new paper is the use of Bayesian methods to tune hyperparameters. Their approach potentially leads to improved model accuracy, a crucial factor in high-stakes environments like healthcare and “juicing up” the model internals for greater accuracy in the generated decision nodes. However, the paper appears to omit any detail about how this approach is deployed. This omission is indeed a great pity because, from what I understood, the Bayesian parameter selection was actually the authors’ novel contribution (the use of ensembles and SHAP on these particular datasets being nothing particularly new).

In conclusion, the SHAP-based approach offers valuable insights at a macro level, the new paper boasts improvements in model accuracy through Bayesian tuning, and my Ada-WHIPS method’s per-instance clarity and actionable insights should prove practical in scenarios where clinicians require detailed explanations of specific cases. I would be delighted to see some confluence of the three ideas, so that the benefits from each can combine and reinforce the use of highly targeted explainability in clinical applications.

Gender Controlled Data Sets for XAI Research

Sun, 30 Jun 2024 00:00:00 +0000

The paper “GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations” introduces a novel dataset, GECO, to evaluate biases in AI explanations, specifically focusing on gender. The authors constructed the dataset with sentence pairs that differ only in gendered pronouns or names, enabling a controlled analysis of gender biases in AI-generated text. GECOBench, an accompanying benchmark, assesses different explainable AI (XAI) methods by measuring their ability to detect and mitigate biases within this context.

The study investigates biases in language models, emphasizing that traditional AI systems often produce biased explanations due to their training on unbalanced datasets. By employing GECO, the researchers show how these biases manifest and affect AI explanations. They demonstrate that existing XAI methods, which aim to make AI decisions more transparent, also carry biases, potentially reinforcing stereotypes or presenting skewed explanations.

Moreover, the authors evaluate several fine-tuning and debiasing strategies to reduce bias in AI models. Their findings suggest that certain fine-tuning approaches can significantly decrease gender bias in explanations without compromising the model’s overall performance. This highlights the importance of combining XAI methods with robust debiasing techniques to create fairer and more trustworthy AI systems.

The paper also provides a comprehensive framework for evaluating bias in XAI methods by using GECOBench. This benchmark allows for a standardized comparison across different methods, providing insights into their strengths and limitations concerning gender bias. It helps identify which methods are more susceptible to biases and under what conditions, promoting the development of better XAI techniques.

Overall, the paper underscores the critical need for datasets like GECO and benchmarks like GECOBench in understanding and mitigating biases in AI explanations. It calls for further research and development in the field of fair and explainable AI, providing resources and guidelines for future studies to build upon. The dataset and code are made publicly available, fostering community efforts toward more equitable AI systems. The paper’s findings have broad implications for the design of AI systems, particularly those deployed in sensitive or high-stakes environments.

Revisiting the Performance-Explainability Trade-Off

Fri, 01 Sep 2023 00:00:00 +0000

I was very excited to read and review the paper Revisiting the Performance-Explainability Trade-Off in Explainable Artificial Intelligence (XAI) last month. I wrote an extensive section on this topic for my Ph.D. thesis (although I coined the name Accuracy-Interpretability Trade-Off or AITO). I have always felt that the subject is too rarely discussed, and never in enough depth and scientific rigour. This Performance-Explanability Trade-off (PET) in the notion that improving model performance (by this, they must mean accuracy or related measures such as true positive rate or AUC/ROC) comes at the cost of explainability.

The authors of this paper state their goal as wanting to refine the discussion of PET in the field of Requirements Engineering for AI systems. Frankly, the paper is quite generic with respect to this self-stated niche once the text gets going, although that in no way detracts from their position on the topic itself. For the most part, the paper focuses on Cynthia Rudin’s influential critique of the performance-explainability trade-off. Rudin is described by the authors as being particularly critical of post-hoc explainability techniques, arguing that they can produce misleading or incomplete explanations that fail to remain faithful to the model’s decision-making process. Again, this was also a foundational point in my thesis on XAI; what good is an explanation of an output other than what the model gave? Proxy (simplified) explanatory models are particularly prone to this behaviour.

Rudin also contends that interpretable models can often match the performance of black-box models, provided that sufficient effort is invested in knowledge discovery and feature engineering. This phenomenon is known as the Rashomon Set argument, which posits that for many real-world tasks, there exist multiple high-performing models, including some that are inherently explainable. The authors argue that while this is an intriguing theoretical claim, it lacks strong empirical backing and does not guarantee that such explainable models will be easily identifiable or practical to develop in all domains. On this point, I find myself in total agreement with the authors. The Rashomon Set argument is merely conjecture from what I can tell and it’s something I would like to revisit in a future blog post.

The authors’ strongest arguments lie in the fact that analyst/researcher-led feature engineering has been vastly superseded and overpowered by the capabilities of deep learning, which hinges on a feature self-learning paradigm built into the model. It’s just so much faster to build a deep neural layer, that researcher/analyst time a resource can be freed up and expended on making post-hoc explainability much for feasible. The authors argue that the real issue is not just whether performance and explainability are in tension, but how much effort is required to achieve both. They suggest that model development should be viewed as a multi-objective optimization problem, where teams must balance the trade-offs between performance, explainability, and available resources, while also considering domain-specific risks such as ethical concerns or financial constraints. From this more nuanced position, they are able to derive an extended framework called PET+ (Performance-Explainability-Time trade-off), which incorporates time and resource constraints into the equation.

I appreciate and commend the reflection on this chronically overlooked and misunderstood topic and hope that their paper contributes towards frameworks for evaluating modeling approaches in the future.

Algebraic Aggregation of Random Forests

Thu, 10 Aug 2023 00:00:00 +0000

In my paper, “CHIRPS: Explaining random forest classification”, I took an empirical approach to addressing model transparency by extracting rules that make Random Forest (RF) models more interpretable. Importantly, this was done without sacrificing the high levels of accuracy achieved by the high-performing RF models.

The recently published “Algebraic aggregation of random forests: towards explainability and rapid evaluation” by Gossen and Steffen provides a theoretical counterpart, offering essential proofs and a mathematical framework for achieving explainability with RF models.

While my paper focused on simplifying complex models by rule extraction on a per instance basis, this subsequent work introduces Algebraic Decision Diagrams (ADDs) to aggregate Random Forests, optimizing their structure and enhancing interpretability at the model level. Both papers aim to improve model transparency, though by different means: my approach is empirical, leveraging rule extraction to clarify black-box models, whereas the latter introduces algebraic methods to combine decision trees into efficient, understandable diagrams.

The mathematical concepts in Gossen and Steffen’s paper, such as path reduction and algebraic operations, support model simplification. Importantly, the authors provide formal proofs that this aggregation retains the original model’s accuracy. This complements the practical focus in my paper, where the goal was also to maintain accuracy while increasing explainability.

Ultimately, the two papers reach the same destination—improving transparency of RF models—but by different routes. While my paper uses rule extraction to bring clarity to complex models, the subsequent work constructs a theoretical basis using algebraic tools, providing formal assurances to the outcomes I demonstrated empirically. Together, they offer complementary perspectives on making RF models more understandable and efficient.

Explaining Random Forests with Representative Trees

Thu, 15 Jun 2023 00:00:00 +0000

The paper “Can’t see the forest for the trees: Analyzing groves to explain random forests” explores a novel take on model-specific explanations, as outlined in my own research (e.g. you can look at “CHIRPS: Explaining random forest classification” as a reference). This new paper by Szepannek and von Holt seeks to make Random Forests (RF) more interpretable. RF are notoriously hard to explain due to their complexity and these novel methods works well for both classification and regression, which is a very useful extension to the field.

The authors introduce most representative trees (MRT) and surrogate trees, essentially distilling a simpler model to run side by side with the black box RF. MRTs focus on highlighting individual trees within a random forest that best explain the overall model behavior, while surrogate trees mimic the forest with simpler, more digestible versions. I have some reservations about the latter approach, because my own research showed that any surrogate model comes with a failure rate, which is the number of examples that the surrogate classifies differently than the black box model under scrutiny. I also question the assertion that a model of 10 or 24 decision trees really is so interpretable. Even a model of this reduced size still likely contains far too many components for a human-in-the-loop to consider and understand.

In any case, to give the authors their due credit, they navigate the trade-offs between accuracy and interpretability of both MRT and surrogate tree methods, and propose a novel concept called groves: small collections of decision trees that balance the need for interpretability with predictive accuracy. Groves provide a middle ground by combining the benefits of MRTs and surrogate models, reducing the overall complexity while still offering meaningful insights into how the model operates. This approach aligns with the goal of making models more transparent and trustworthy.

Through various case studies, the paper shows how groves and surrogate trees can be effectively applied to real-world datasets. The trade-off between model accuracy and explainability remains a central challenge. Yet, in these studies, groves provide a workable compromise by making it easier for humans to understand what is driving the model’s predictions without overwhelming them with unnecessary detail.

The discussion also highlights a key challenge in using groves: deciding on the right number of trees to use for explanation. Using too many trees risks overwhelming the user with information (as I have already pointed out), while too few might fail to capture the complexity of the underlying model and run with an untenable failure rate. I dicuss ways to achieve a zero failure rate in my thesis. Keeping explanations concise and accessible is just a part of the complete picture.

In conclusion, this paper underscores the crucial need for enhancing the interpretability of machine learning models, particularly in high-stakes fields like healthcare and finance, where decision transparency is essential. By extending the work in interpretability through methods like groves and surrogate trees, it addresses the challenge of making powerful models like random forests more understandable.

Combatting Fake News With XAI

Mon, 13 Mar 2023 00:00:00 +0000

With all the unhinged hype over ChatGPT stealing everyone’s jobs and AI taking over the world, it’s great to see postiive use cases for Machine Learning (ML) technologies. As usual, eXplainable Artificial Intelligence (XAI) has something to contribute to the ethical landscape of fairness and transparency. In this recent news article we see a concerted attempt to combat fake news with XAI and a pretty sophisticated tech stack.

With the rise of social media and other online platforms, the spread of fake news has become a major problem. Fake news is defined as news stories that are intentionally false and designed to mislead readers. It is often spread through social media and can have serious consequences, such as influencing public opinion and even swaying elections.

XAI is a field of AI that focuses on making machine learning models transparent and explainable. By using XAI, it is possible to detect and filter out fake news, while also providing a clear explanation of how the model came to its decision. One way XAI can help in combatting fake new is through the generation of counterfactual explanations. Counterfactual explanations are used to explain how a model would have made a different decision if the input data had been different. In the context of fake news detection, counterfactual explanations can be used highlight words and phrases that were critical in the classification as fake or not fake. The counterfactual interpretation is that when those phrases are substituted, the news article would flip its classification. In the article, you can see a good example of the SHAP Force Plot highlighting specific text elements that add to or detract from the suspect nature of text extract (the figure labelled “Explainability module developed for multiclass classification”).

If we are going to have any chance of finding solutions to the technology driven problems of today, then we need to embrace positive applications of ML and not get carried along with all the negative hype. XAI provides us with many such positive examples.

Explaining Random Forests with Boolean Satisfiability

Mon, 21 Jun 2021 00:00:00 +0000

The paper “On Explaining Random Forests with SAT” uses Boolean satisfiability (SAT) methods to provide a formal framework for generating explanations of Random Forest (RF) predictions. A key result in the paper is that abductive explanations (AXp) and contrastive explanations (CXp) can be derived by encoding the RF’s decision paths into propositional logic.

Encoding a decision path as propositional logic, is an entirely reasoned approach and quite straightforward, as I showed in my paper CHIRPS: Explaining random forest classification. The decision paths of an RF model can be transformed into a Boolean formula in Conjunctive Normal Form (CNF). For example, each decision tree in the forest is represented as a set of clauses. Following the paths for a single example prediction essentially carves out a region of the feature space with a set of step functions, resulting in a sub-region that must return the target response. When the clauses of this step functions set correspond to a subset of the features, a change in the remaining feature inputs has no effect on the model prediction. This subset is a prime implicant (PI) explanation.

A PI-explanation is a minimal subset of features that are sufficient to guarantee a particular prediction made by a machine learning model. It represents the smallest set of conditions that, if held constant, would lead to the same classification result. Essentially, it’s the most concise explanation of why the model arrived at its decision, highlighting the critical features responsible for that prediction. In fact, my own research centred finding soft PI-explanations, and revealing the limits where they no longer hold true for extreme outliers and unusual examples.

The authors of this paper show that finding AXp and CXp by this PI-explanation method reduces to solving a SAT problem and is therefore NP-hard but can be polynomial under specific conditions. This insight into the problem complexity is significant because it establishes that generating explanations is feasible when those assumptions are met and opens up the method to practical applications with real-world data.

Overall, the SAT-based methodology enables a structured, efficient way to uncover the decision-making process of Random Forests, ensuring that their predictions are not just accurate but also explainable, which is crucial for domains requiring transparency like healthcare and finance.

How Subsets of the Training Data Aﬀect a Prediction

Sun, 20 Dec 2020 00:00:00 +0000

I was quite excited by the title of a new paper, on pre-publication this month. “Explainable Artificial Intelligence: How Subsets of the Training Data Affect a Prediction” by Andreas Brandsæter and Ingrid K. Glad, at first glance, appeared to have some close alignment to my own work CHIRPS: Explaining random forest classification, published earlier this year in June. It’s generally highly desirable to connect with other researchers with which you share common ground, working contemporaneously. Often, fruitful collaborations are born.

As it turns out, the authors have taken a fairly different approach to mine. The CHIRPS method discovers a large, high precision subset of neighbours in the training data, using a minimal number of constraints, that share the same classification from the model, and returns robust statistics that proxy for precision and coverage. Brandsæter and Glad’s method is a novel approach that works with regression and time series problems, and pre-supposes that there are subsets in the data (that may or may not be adjacent) that can be set up in advance to reveal regions of influence on the final prediction of a given data point. We share a recognition of the importance of interpretability in AI and machine learning, especially in critical applications.

Tthe authors propose a methodology that uses Shapley values to measure the importance of different training data subsets in shaping model predictions. Shapley values, originating from coalitional game theory, are adapted here to quantify the contribution of each subset of training data as if each subset were a “player” influencing the outcome of the model’s prediction. This approach offers a fresh perspective by directly associating predictions with specific training data subsets, which can reveal patterns or biases that feature-based explanations might miss.

The paper delves into the theoretical framework of Shapley values in a coalitional game context and extends this to analyze subset importance. The authors describe how their methodology can pinpoint the impact of specific subsets on predictions, facilitating insights into model behavior, training data errors, and potential biases. By using subsets rather than individual data points or features, this approach is particularly well-suited to models that rely on large, high-dimensional datasets where feature importance alone may not fully capture influential patterns. This method is demonstrated to be useful in understanding how similar predictions may stem from different subsets of data, emphasizing the complex interactions within training data that influence predictions.

Through several case studies, the paper demonstrates how Shapley values for subset importance can be applied in real-world scenarios. For example, in time series data and autonomous vehicle predictions, subsets of training data based on chronological segmentation reveal how specific periods contribute to model outputs. This approach is shown to be valuable for identifying anomalies or segment-specific patterns that could affect model accuracy or introduce biases. Additionally, by explaining the squared error for predictions, the authors illustrate how this methodology can also diagnose errors in training data, which could improve overall model reliability.

The authors discuss limitations and challenges, particularly around the computational complexity of retraining models on multiple subsets to calculate Shapley values. They suggest that, while computationally intensive, this process can be optimized with parallel processing and may not need to be repeated for each new test instance. They also propose potential applications of this methodology in tailoring training data acquisition strategies, such as for cases where predictions are most critical, which can improve model performance by selectively sampling from influential subsets.

In conclusion, Brandsæter and Glad’s paper represents a significant advancement in explainable AI by emphasizing the training data’s impact on model predictions. By shifting focus to data-centric explanations, their approach highlights how subsets within the data contribute directly to individual predictions, expanding the interpretative toolkit beyond traditional feature importance. This approach aligns with my own work on CHIRPS, underscoring the notion that providing contextual information from training data strengthens model transparency and interpretability. Using training data as a reference framework enables explainable AI methods to draw on established statistical theory, which ultimately lends robustness to explanations, even in black-box models. Together, these methods suggest a promising direction for explainable AI, wherein training data subsets serve as crucial elements to understand and elucidate model behavior effectively.

About

Sat, 27 Jun 2020 23:14:15 +0800

Welcome to XAI Today! A blog dedicated to Explainable AI (XAI)—an essential field that better aligns the complex workings of artificial intelligence and machine learning algorithms with human understanding.

As AI continues to influence more aspects of our lives, the need for trust, freedom from bias, and accountability in these systems is critical, particularly in high-stakes areas like healthcare, finance, and law. Any domain or sector where personal information is used and people are affected can make no compromises when it comes to fairness and transparency. XAI can help to reveal whether automated decision making is ethical and compliant.

XAI and interpretable ML are equally valuable in business intelligence (BI), predictive, and prescriptive analytics, as they provide transparency and clarity around data-driven decisions. In fields such as marketing, customer experience, and user journeys, decision-makers want to understand how actionable insights have been generated. Only by sense-checking for real-world causality and intuition about human behaviour, can decision makers trust and leverage these insights effectively. XAI ensures that predictions and recommendations, such as forecasting trends or optimizing strategies, are comprehensible and aligned with business objectives. This transparency fosters more informed decision-making, improving outcomes by connecting insights directly to actionable business strategies.

This blog explores the latest developments, research, and practical applications of XAI. Whether you’re a data scientist, developer, or simply curious about how AI works under the hood, our goal is to make AI/ML explainability and interpretability accessible, insightful, and interesting for everyone.

Counterfactual Explanations Help Identify Sources of Bias

Sun, 08 Mar 2020 00:00:00 +0000

By the end of 2020, the topic of eXplainable Artificial Intelligence (XAI) has become quite mainstream. One important developlment is counterfactual explanations, which (among other benefits) can to identify and reduce bias in machine learning models. Counterfactual explanations provide insights by showing how minimal changes in input features can alter model predictions. This approach has been crucial in exposing biased behavior, especially in sensitive applications like credit scoring or hiring. By identifying how protected attributes (e.g., gender or race) affect outcomes, practitioners could better address and mitigate unfair biases in AI systems (Verma et al., 2020).

Reference: Verma, S., & Rubin, J. (2020). Fairness Definitions Explained. Proceedings of the 2020 ACM/IEEE International Workshop on Software Fairness.

International Women's Day 2020

Sun, 08 Mar 2020 00:00:00 +0000

Profile: Cynthia Rudin #

Today, for International Women’s Day, I wanted to share my huge respect for Cynthia Rudin. She is a leading academic in the research field in which I am currently involved for my PhD - interpretability in machine learning. Her work is very widely cited and comes up in all searches related to solving the “black box” problem of machine learning. She is a true thought leader.

Here is excellent podcast where Cynthia succinctly and eruditely explains the current state, problems and solutions of ML interpretability. It’s well worth a listen and, I believe, entirely understandable by non-expert listeners.

Here are a couple of profile sites where you can find out more about this brilliant and accomplished woman who is a true thought leader in Machine Learning and Data Science.

https://en.wikipedia.org/wiki/Cynthia_Rudin

https://users.cs.duke.edu/~cynthia/

Whenever I can - and not only on IWD - I try to raise the profile of prominent women in STEM (science, technology, engineering and mathematics). Under-representation and a lack of visible role-models is one barrier to increasing diversity and equality in any field. Certainly, there is no lack of talented women in STEM but if young women and girls at school aren’t given examples of successful women, this can limit aspirations at the grass roots level.

Faithful and Customizable Explanations of Black Box Models

Sun, 05 Jan 2020 00:00:00 +0000

The authors of “Faithful and Customizable Explanations of Black Box Models” (MUSE) share a common goal with my own research: addressing the challenge of making machine learning models interpretable. Both emphasize the importance of transparency in decision-making, particularly in scenarios where human trust and understanding are critical, such as healthcare, judicial decisions, and financial assessments. Both they and I see decision rule structures as the ideal explanation format to explain model behaviour.

MUSE uses a two-level decision set framework, which combines subspace descriptors and decision logic to generate explanations for different regions of the feature space. This is useful for zooming in to specific features and observation subsets of interest. Just like my own research, this is a highly user centric approach, emphasising a Human-in-the-Loop process of expert review of model decisions. My method differs in that it facilitates an individual detail review, potentially allowing the expert user to respond to individuals seeking some kind of review or redress over an automated decision. In essence, this is a response to the “computer says no” problem. The explanations are tailored to specific needs or contexts.

This focus on end-user interaction reflects a broader effort in both frameworks to build trust in machine learning outputs by providing meaningful insights. Despite these similarities, the research ideas diverge in significant ways. MUSE has a broader scope, offering global explanations as well as targeted insights into specific subspaces of the model’s behaviour. It is designed to be model-agnostic, meaning it can work with any type of predictive system. My research has a specific focus on Decision Tree ensembles (Random Forest and Boosting methods), explaining how such a classifier reached a decision for a particular data point, emphasising precision and counterfactual reasoning.

The methodologies also differ. MUSE employs optimization techniques to create compact and interpretable decision sets that balance fidelity, unambiguity, and interpretability. My approach, in contrast, extracts decision paths from random forests using frequent pattern mining, constructing rules that highlight the most influential attributes in a model’s classification. These distinct methods reflect their differing objectives: MUSE aims to provide a comprehensive view of a model’s behaviour, while I seek to zero in on individual classifications with a high degree of local accuracy.

Together, these research approaches represent two sides of the same coin: one offering a high-level overview and the other delivering precise, localised explanations. There is a lot of scope for combining the two methods in a collaborative framework for holistic explanations.