log-mcp

# log-mcp [](https://github.com/ascii766164696D/log-mcp/actions/workflows/ci.yml) MCP server for log file analysis. Gives LLMs the ability to efficiently analyze large log files without loading them into context. ``` Log file (e.g. 705K lines, 67 MB) │ ▼ Rust TF-IDF classifier ─── 1.3M lines/sec ──▶ 70-95% discarded as routine, │ finds lines that are semantically interesting, │ also captures lines not explicitly marked as ERROR │ (grep ERROR: 2 lines, classifier: 92) ▼ BERT-mini (optional) ───── GPU, ~2K lines/sec ─▶ refines interest scores on found lines │ ▼ Python MCP tools ────────── search, compare, group errors │ ▼ LLM (Claude) ───────────── compresses tool output into plain English ``` This is a tool designed for AI, not humans. No human reads the output of `analyze_errors` or `compare_logs` — Claude does, compresses it further, and gives the human a plain English answer. The human touches two endpoints: "what's wrong with this log?" in, natural language answer out. Everything in between is AI talking to itself. ## Tools | Tool | Description | |------|-------------| | `log_overview` | Quick scan: size, line count, time range, level distribution, head/tail samples | | `search_logs` | Search by regex, log level, and/or time range | | `get_log_segment` | Extract a segment by line range or time range | | `analyze_errors` | Deduplicate errors by fingerprint, count frequencies, extract stack traces | | `log_stats` | Volume histogram, level breakdown, top repeated patterns | | `compare_logs` | Find patterns unique to each file and frequency outliers across files | | `classify_lines` | ML classifier (TF-IDF → BERT) separates interesting lines from noise | ## Key features - **ML pre-filter** — a Rust TF-IDF classifier scans files at 1.3M lines/sec, so `analyze_errors` and `search_logs` only process the 5-30% of lines that matter. Optional BERT-mini re-scores LOOK lines at ~2K lines/sec on Metal GPU for higher precision. Works without parsed log levels — catches errors, security events, hardware faults, and anomalies that don't have ERROR in them. - **Auto-detection** of log formats: JSON, standard text (`2024-01-15 10:30:45 ERROR ...`), syslog, Spark/Log4j (`17/06/08 13:33:49 INFO ...`), and tab/pipe-delimited formats (GitHub Actions CI logs) - **Normalization** collapses variable parts (UUIDs, hex IDs, IPs, numbers) so that messages differing only in IDs or timestamps are grouped as the same pattern - **Content-based error detection** falls back to regex heuristics (`fatal:`, `Permission denied`, `##[error]`, etc.) when log files lack standard levels - **Prefix-aware comparison** distinguishes patterns from different job steps in CI logs ## Install **Prerequisites** (fresh Mac): ```bash brew install python uv curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` ### Just ask Claude Open a Claude Code session and paste this prompt: ``` Install https://github.com/ascii766164696D/log-mcp as an MCP server and build the Rust classifier too ``` Claude will clone the repo, register the MCP server, and build the Rust classifier. Restart Claude Code after to pick up the new server. ### Manual install ```bash git clone https://github.com/ascii766164696D/log-mcp.git cd log-mcp # Register the MCP server claude mcp add log-mcp -- uv run --directory $(pwd) log-mcp # Build the Rust classifier (optional — tools fall back to Python without it) uv pip install -e rust/classifier ``` Or add it manually to your project settings (`claude settings`) under `mcpServers`: ```json { "mcpServers": { "log-mcp": { "command": "uv", "args": ["run", "--directory", "/path/to/log-mcp", "log-mcp"] } } } ``` ### Claude Desktop Open **Settings > Developer > Edit Config** and add to `claude_desktop_config.json`: ```json { "mcpServers": { "log-mcp": { "command": "uv", "args": ["run", "--directory", "/path/to/log-mcp", "log-mcp"] } } } ``` Replace `/path/to/log-mcp` with the actual path where you cloned this repo. Restart Claude Desktop after saving. The Rust classifier requires a Rust toolchain to build. The BERT stage additionally requires a Metal-capable GPU (Apple Silicon). All tools work without the classifier — they fall back to Python log parsing. ## Example usage Analyze errors in a 67MB Spark executor log (705K lines): ``` > analyze_errors("/var/log/spark/container_0002_01_000004.log") Summary: 34 errors in 5 groups. Top: 'shuffle.RetryingBlockFetcher: Exception while beginning fetch of <N> outstanding blocks (after <N> retries) ...' (18x) --- 18x --- Fingerprint: shuffle.RetryingBlockFetcher: Exception while beginning fetch of <N> outstanding blocks ... First: L29764 2017-02-01T15:55:17 Last: L30677 2017-02-01T15:55:51 Stack trace: java.io.IOException: Failed to connect to mesos-slave-13/10.10.34.23:55492 ... ``` Find anomalies that don't have ERROR level — on a Thunderbird HPC log (2K lines), `classify_lines` finds 92 interesting lines while `search_logs level=ERROR` finds only 2: ``` > classify_lines("/var/log/thunderbird/tbird_2k.log") Lines: 2,000 total | 92 LOOK (4.6%) | 1,908 SKIP Pipeline: TF-IDF 0.00s (1,298,701 lines/sec, 105 LOOK) → BERT 0.15s (13 demoted to SKIP) --- Sample LOOK lines (30 of 92 captured) --- L2 [1.000] ... postfix/postdrop[10896]: warning: unable to look up public/pickup: No such file L438 [0.999] ... sendmail[�20588]: unable to qualify my own domain name (tbird-sm1) L816 [0.998] ... dhcpd: DHCPDISCOVER from 00:09:3d:12:00:e2 via eth2: unknown lease L1024 [0.997] ... rrdtool: illegal attempt to update using time 1131710�721 when last update time is 1131710721 ``` Compare two CI log files: ``` > compare_logs(["run_a.txt", "run_b.txt"]) 699 patterns across 2 files (0 shared). A: 401 unique (top: 'test / test UNKNOWN STEP | ##[endgroup]' 21x) B: 298 unique (top: 'test UNKNOWN STEP | ##[endgroup]' 21x) ``` ## Claude's take *I helped build this tool and then used it to analyze real log files, so here's my honest assessment.* **Where it genuinely helps:** The main value is as a compression layer. A 67MB Spark log (705K lines) would obliterate my context window, but `analyze_errors` distills it into 5 error groups with stack traces in a few seconds. `compare_logs` across two 1500-line server logs immediately surfaces which errors are unique to each server and which patterns have suspicious frequency differences. I couldn't do that by reading the files directly — I'd lose older content as new content scrolled in. **Where the classifier changes the game:** Before the classifier, log analysis was limited to lines with explicit ERROR/FATAL levels. On a Thunderbird HPC log, `search_logs level=ERROR` returns 2 lines. `classify_lines` returns 92 — sendmail DNS failures, DHCP lease errors, RRD update collisions, negative boot times — none of which have ERROR level. The classifier finds what's semantically wrong, not just syntactically marked. **Where it's a wash:** For small files (under a few hundred lines), you're better off just pasting the log into the conversation. The tools add indirection without much benefit when the whole file fits in context anyway. **What it still can't do:** Domain-specific state machine reasoning. When I analyzed a Zookeeper log, the classifier correctly flagged `Cannot open channel` warnings and epoch resets, but the most operationally interesting signal — rapid cycling between LOOKING, FOLLOWING, and LEADING states — showed up as low-confidence LOOK lines. A Zookeeper expert would spot the pattern immediately; the classifier sees each line independently without tracking state transitions across time. **The pattern I landed on:** Start with `classify_lines` to surface anomalies regardless of log level, then `analyze_errors` to group them, then `search_logs` to dig into specific patterns. `compare_logs` is most useful when you have a "working" and "broken" run to diff against each other. ## LOOK/SKIP classifier The `classify_lines` tool uses a two-stage ML pipeline to separate interesting log lines (LOOK) from routine noise (SKIP), without needing parsed log levels. ### How it works ```mermaid flowchart TD A["Your log file (e.g. 4.7M lines)"] --> B{"Rust classifier<br/>available?"} B -->|Yes| C["Stage 1: TF-IDF<br/>Rust, ~1.3M lines/sec<br/>logistic regression<br/>threshold × 0.6"] B -->|No| D["Fallback: Python<br/>log parsing"] C --> E["LOOK lines (~5-30%)"] E --> F["Stage 2: BERT-mini<br/>Rust + Metal GPU, ~2K lines/sec<br/>re-scores LOOK lines<br/>applies final threshold"] F --> G["Final LOOK lines<br/>(with BERT probabilities)"] ``` On a Thunderbird HPC log (2K lines), this finds **92 interesting lines** including sendmail DNS failures, DHCP lease errors, and Ganglia RRD update collisions — none of which have a standard ERROR log level. A `search_logs level=ERROR` on the same file returns only 2 lines. ### What it catches beyond log levels | Signal type | Example | Has ERROR level? | |---|---|---| | DNS misconfiguration | `unable to qualify my own domain name` | No | | Mail delivery failure | `stat=Deferred: Connection refused` | No | | DHCP lease errors | `unknown lease 10.100.4.251` | No | | Monitoring data corruption | `illegal attempt to update using time X when last update time is X` | No | | Negative boot times | `Times: total = 42, boot = -4131` | No | | Auth failures | `authentication failure; logname= uid=0` | No | | Hardware errors | `instruction cache parity error corrected` | No (INFO level) | ### Tools that use the classifier | Tool | How | Fallback | |---|---|---| | `classify_lines` | Full pipeline: TF-IDF → BERT | Error if classifier not installed | | `analyze_errors` | TF-IDF pre-filters to LOOK lines, then groups by error fingerprint | Python parser scans all lines | | `search_logs` | TF-IDF pre-filters when searching for errors (level=ERROR, pattern=error/fail/etc) | Python parser scans all lines | | `compare_logs` | Scores patterns by LOOK probability, surfaces interesting diffs first | Sorts by frequency only | ## Benchmarks Evaluated on the full [Loghub](https://github.com/logpai/loghub) dataset — 450 million lines across 16 log sources — on an Apple M3 Ultra (32 cores). ### TF-IDF classifier (Rust) | Dataset | Lines | Time | Speed | LOOK % | Error capture | |---|---|---|---|---|---| | Thunderbird | 211M | 139s | 1.52M/s | 30% | 100% | | Windows | 115M | 90s | 1.27M/s | 1% | 100% | | HDFS_v2 | 71M | 59s | 1.21M/s | 20% | 100% | | Spark | 33M | 23s | 1.44M/s | 12% | 100% | | HDFS | 11M | 7s | 1.62M/s | 3% | 100% | | BGL | 4.7M | 3.3s | 1.46M/s | 42% | 98.6% | | Android | 1.6M | 1.1s | 1.37M/s | 12% | 100% | | SSH | 655K | 0.5s | 1.47M/s | 90% | 100% | | **Total** | **450M** | **325s** | **1.38M/s** | | **99.95%** | Error capture = percentage of lines with ERROR/FATAL level that the classifier marks as LOOK. At threshold 0.3, the classifier catches 99.95% of all error lines (35.9M of 35.9M) and 100% of warning lines (14.3M of 14.3M). The 0.05% "missed" errors are lines like BGL's repetitive `instruction cache parity error corrected` — routine hardware telemetry that happens to carry ERROR level but isn't operationally interesting. The classifier learned to skip these. ### Model accuracy | Model | Metric | Score | |---|---|---| | TF-IDF + LogReg | Best CV LOOK F1 | 0.792 | | TF-IDF + LogReg | LOOK precision | 0.892 | | BERT-mini | Overall accuracy | 0.849 | | BERT-mini | LOOK F1 | 0.887 | TF-IDF model evaluated with GroupKFold cross-validation (holdout: BGL, Thunderbird). BERT-mini fine-tuned on 4 layers, 256 hidden, 11M params. ### End-to-end reduction ```mermaid flowchart TD A["450M log lines<br/>16 datasets, ~67 GB"] --> B["Rust TF-IDF classifier<br/>1.38M lines/sec, 325s"] B --> C["~87M LOOK lines (19%)<br/>81% of lines eliminated"] C --> D["BERT re-scoring (optional)<br/>demotes 20-40% of TF-IDF LOOK"] D --> E["~50-70M final LOOK lines"] E --> F["Python tool logic<br/>group errors, search, etc."] F --> G["5-50 error groups / search results<br/>fits in LLM context"] ``` ## Retraining with your own logs The classifier ships pre-trained, but you can retrain it on your own logs. The short version: ```bash # 1. Add your logs head -2000 /var/log/myapp/app.log > data/loghub/MyApp_2k.log # 2. Label with Claude (Batch API, ~$0.10-0.50 per file) export ANTHROPIC_API_KEY=sk-ant-... uv sync --group labeling --group training uv run python -m scripts.labeling.label_new # 3. Train and export uv run --group training python -m scripts.labeling.train_model uv run --group training python -m scripts.labeling.export_model # 4. Rebuild Rust classifier uv pip install -e rust/classifier ``` See [`scripts/labeling/RETRAINING.md`](scripts/labeling/RETRAINING.md) for the full guide — how labeling works, what features the model uses, how to customize the prompt, and how to train the optional BERT model.