Explaining AI Decisions: SHAP and LIME

Your model predicts a loan will default. Why? “The neural network said so” isn’t acceptable to regulators, customers, or debugging engineers.

Model interpretability matters. SHAP and LIME are the tools to achieve it.

Why Explainability?

Regulatory Compliance

GDPR Article 22: Right to explanation for automated decisions affecting individuals.

Debugging

Understanding why a model fails helps fix it.

Trust

Users trust predictions they understand.

Fairness

Explanations reveal bias that metrics might miss.

Local vs Global Explanations

Global: How does the model work overall?

• Feature importance rankings
• Partial dependence plots

Local: Why this specific prediction?

• Why was this loan denied?
• Why was this email flagged as spam?

SHAP and LIME focus on local explanations.

LIME: Local Interpretable Model-agnostic Explanations

LIME explains individual predictions by:

1. Creating variations of the input
2. Getting predictions for variations
3. Fitting a simple (interpretable) model to those predictions
4. Using the simple model to explain

How LIME Works

Original Input: Image of a cat
                     │
                     ▼
┌────────────────────────────────────────┐
│ Create perturbed samples               │
│ (hide different parts of image)        │
└────────────────────────────────────────┘
                     │
                     ▼
┌────────────────────────────────────────┐
│ Get predictions for all samples        │
│ from black-box model                   │
└────────────────────────────────────────┘
                     │
                     ▼
┌────────────────────────────────────────┐
│ Fit linear model to predictions        │
│ weighted by similarity to original     │
└────────────────────────────────────────┘
                     │
                     ▼
Explanation: "Ears and whiskers → cat"

LIME in Python

import lime
import lime.lime_tabular
from sklearn.ensemble import RandomForestClassifier

# Train a model
model = RandomForestClassifier()
model.fit(X_train, y_train)

# Create LIME explainer
explainer = lime.lime_tabular.LimeTabularExplainer(
    X_train,
    feature_names=feature_names,
    class_names=['rejected', 'approved'],
    mode='classification'
)

# Explain a prediction
exp = explainer.explain_instance(
    X_test[0],
    model.predict_proba,
    num_features=5
)

exp.show_in_notebook()

LIME for Images

from lime import lime_image
from skimage.segmentation import mark_boundaries

explainer = lime_image.LimeImageExplainer()

explanation = explainer.explain_instance(
    image,
    model.predict,
    top_labels=5,
    hide_color=0,
    num_samples=1000
)

# Show which regions support the prediction
temp, mask = explanation.get_image_and_mask(
    explanation.top_labels[0],
    positive_only=True
)
plt.imshow(mark_boundaries(temp, mask))

SHAP: SHapley Additive exPlanations

SHAP is grounded in game theory. It calculates the contribution of each feature using Shapley values—fair allocation of “credit” among features.

Key Properties

• Local accuracy: Explanation predictions sum to actual prediction
• Missingness: Missing features have zero attribution
• Consistency: If a feature’s contribution increases, its SHAP value increases

SHAP in Python

import shap

# Train model
model = xgboost.XGBClassifier()
model.fit(X_train, y_train)

# Create explainer
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# Force plot for single prediction
shap.force_plot(
    explainer.expected_value,
    shap_values[0],
    X_test[0]
)

# Summary plot for global understanding
shap.summary_plot(shap_values, X_test)

SHAP for Deep Learning

import shap

# For neural networks
explainer = shap.DeepExplainer(model, X_train[:100])
shap_values = explainer.shap_values(X_test[:10])

shap.image_plot(shap_values, X_test[:10])

LIME vs SHAP

Use LIME when: Speed matters, quick explanations needed Use SHAP when: Consistency matters, regulatory requirements

Practical Tips

For Tabular Data

# SHAP with tree models
explainer = shap.TreeExplainer(model)  # Fast, exact

# SHAP with any model
explainer = shap.KernelExplainer(model.predict, X_train[:100])  # Slower

For Text

import shap

# Text classification
explainer = shap.Explainer(model, tokenizer)
shap_values = explainer(texts[:5])
shap.plots.text(shap_values)

Debugging with Explanations

# Find misclassified samples
wrong = X_test[y_pred != y_test]

# Explain why
for sample in wrong[:5]:
    exp = explainer.explain_instance(sample, model.predict_proba)
    print(exp.as_list())

Limitations

Explanations Aren’t Justifications

An explanation says “income was important” not “the decision was correct.”

Instability

LIME explanations can vary between runs. Use higher num_samples.

Simplification

Both methods approximate complex models with simpler ones. The explanation is a simplification.

Adversarial Attacks

Explanations can be manipulated. Don’t rely on them for security.

Best Practices

1. Explain failures: Focus on wrong predictions first
2. Validate explanations: Do they match domain knowledge?
3. Combine with global: Use local + global for full picture
4. Document limitations: Users should understand approximations
5. Update regularly: Models change, regenerate explanations

Final Thoughts

Explainability isn’t a checkbox—it’s a capability. SHAP and LIME make black-box models more transparent.

Start with SHAP for tree-based models (fast and consistent). Use LIME for quick exploration. Invest in explanations wherever decisions affect people.

If you can’t explain it, you don’t understand it.