Content
# Model Explainability Skills
A Claude Code MCP skill that explains any trained predictive model. Point it at a model file and a feature CSV, say **"analyse this model"**, and get back a full 9-section explainability report with embedded plots — no code required.
Works with scikit-learn, PyTorch, XGBoost, LightGBM, random forests, and any other sklearn-compatible estimator.
---
## Supported model types
| Model | File format | SHAP method |
| --- | --- | --- |
| scikit-learn estimators / Pipelines | `.pkl` | TreeExplainer or KernelExplainer |
| PyTorch `nn.Module` | `.pt` / `.pth` | KernelExplainer |
| XGBoost (native Booster) | `.json` / `.ubj` | TreeExplainer |
| XGBoost (sklearn API) | `.pkl` | TreeExplainer |
| LightGBM (native Booster) | `.txt` / `.bin` | TreeExplainer |
| LightGBM (sklearn API) | `.pkl` | TreeExplainer |
| Any joblib/pickle model | `.pkl` | KernelExplainer fallback |
---
## What it produces
| Method | What you learn |
| --- | --- |
| **Sobol / Functional ANOVA** | Which features drive output variance, and how much comes from interactions vs main effects (S1, ST, S2) |
| **Permutation Importance** | How much model score drops when each feature is shuffled |
| **SHAP** | Per-observation attribution — global beeswarm, importance bar chart, interaction heatmap |
| **SHAP Dependence Plots** | How the effect of feature A shifts with feature B (top Sobol S2 pairs) |
| **ALE / PDP + ICE** | Effect-shape plots — ALE selected automatically when features are correlated |
| **Cross-method consensus** | Features ranked highly by all three methods are robustly important |
The skill automatically selects the right method:
- Correlated features → ALE instead of PDP (avoids extrapolation bias)
- Tree model → TreeExplainer (fast, exact SHAP + interaction values)
- Other models → KernelExplainer (model-agnostic) + SHAP value correlation heatmap as interaction proxy
- All analysis runs once (`precompute_all`), then cached — follow-up questions answer in milliseconds
---
## Quick start
### Step 1 — Register the MCP server
```bash
claude mcp add model-explainability \
-e EXPLAIN_OUTPUT_DIR="$HOME/Desktop/nn-explainability-skill/output" \
-- python3 "$HOME/Desktop/nn-explainability-skill/mcp_server/server.py"
```
### Step 2 — Install dependencies (say this in Claude Code)
```
set up
```
### Step 3 — Run the analysis
```
analyse this model:
model: examples/credit_risk_artifacts/model_state_dict.pt
data: examples/credit_risk_artifacts/X_test_processed.csv
features: examples/credit_risk_artifacts/feature_names.json
```
Then ask plain-language questions:
```
What drives this model?
Are there any feature interactions?
What is the effect of loan_int_rate?
Why was row 42 scored high?
Write a summary report.
```
---
## Example output
The `examples/credit_risk_artifacts/explain_output/` directory contains a pre-generated report for a PyTorch loan grade model:
- [`explainability_report.md`](examples/credit_risk_artifacts/explain_output/explainability_report.md) — full 9-section report
- ALE effect-shape plots for top 5 features
- SHAP beeswarm, importance bar chart, and interaction heatmap
- SHAP dependence plots for top Sobol S2 interaction pairs
Key finding: `loan_int_rate` dominates with Sobol ST = 0.97 and SHAP mean |φ| = 0.78. `cb_person_default_on_file_num` and `loan_status` are consensus drivers across all three methods.
---
## Project structure
```
nn-explainability-skill/
├── SKILL.md # Claude Code skill definition
├── INSTRUCTIONS.md # Installation and usage guide
├── setup.sh # Automated dependency installer
├── regen_report.py # Regenerate report from cached results
│
├── mcp_server/
│ ├── server.py # 9 MCP tools
│ ├── requirements.txt
│ └── explainability/
│ ├── utils.py # Model loading (PyTorch/sklearn/XGBoost/LightGBM),
│ │ # prediction wrappers, correlation diagnostics
│ ├── sobol.py # Sobol / Functional ANOVA (SALib)
│ ├── pdp_ice.py # PDP + ICE (scikit-learn)
│ ├── ale.py # ALE (PyALE)
│ ├── shap_explain.py # SHAP global + local + interaction plots
│ ├── report.py # 9-section markdown report assembler
│ └── cache.py # Precompute cache (cache.json + shap_values.npy)
│
├── examples/
│ └── credit_risk_artifacts/ # Worked example: PyTorch MLP, credit risk
│ ├── README.md
│ ├── nn_model_generator.ipynb
│ ├── model_state_dict.pt
│ ├── X_test_processed.csv
│ ├── feature_names.json
│ └── explain_output/ # Pre-generated analysis results and report
│
└── references/
├── sobol_sensitivity_notes.md
├── pdp_ale_ice_notes.md
├── neural_network_anova_notes.md
└── model_risk_explainability_template.md
```
---
## MCP tools
| Tool | Description |
| --- | --- |
| `precompute_all` | Run full pipeline once and cache everything to disk |
| `ask` | Answer any plain-language question from cache (no recompute) |
| `run_full_analysis` | End-to-end analysis, returns markdown report directly |
| `run_sobol_analysis` | Sobol S1 / ST (+ S2 if `second_order=True`) |
| `run_pdp_ice` | PDP + ICE for a named feature |
| `run_ale` | ALE for a named feature |
| `run_shap_importance` | Global SHAP importance + beeswarm |
| `run_shap_interactions` | Top pairwise SHAP interactions (tree models) |
| `explain_single_observation` | Local SHAP waterfall for one row |
---
## Dependencies
```
torch
numpy
pandas
matplotlib
scikit-learn
shap
PyALE
SALib
joblib
mcp[cli]
```
Install automatically via `setup.sh`, or manually:
```bash
pip install -r mcp_server/requirements.txt
```
---
## References
- **Credit risk dataset & end-to-end pipeline:** [orbenh/credit-risk-scoring-end-to-end](https://github.com/orbenh/credit-risk-scoring-end-to-end) — source of the worked example model
- **Sobol sensitivity analysis:** Saltelli, A. et al. (2010). *Variance based sensitivity analysis of model output.* Computer Physics Communications.
- **SHAP:** Lundberg, S. & Lee, S.-I. (2017). *A unified approach to interpreting model predictions.* NeurIPS.
- **ALE:** Apley, D.W. & Zhu, J. (2020). *Visualizing the effects of predictor variables in black box supervised learning models.* JRSS-B.
- **Interpretable ML Book:** Molnar, C. (2022). [christophm.github.io/interpretable-ml-book](https://christophm.github.io/interpretable-ml-book)
