内容
# log-mcp
[](https://github.com/ascii766164696D/log-mcp/actions/workflows/ci.yml)
MCP server for log file analysis. Gives LLMs the ability to efficiently analyze large log files without loading them into context.
```
Log file (e.g. 705K lines, 67 MB)
│
▼
Rust TF-IDF classifier ─── 1.3M lines/sec ──▶ 70-95% discarded as routine,
│ finds lines that are semantically interesting,
│ also captures lines not explicitly marked as ERROR
│ (grep ERROR: 2 lines, classifier: 92)
▼
BERT-mini (optional) ───── GPU, ~2K lines/sec ─▶ refines interest scores on found lines
│
▼
Python MCP tools ────────── search, compare, group errors
│
▼
LLM (Claude) ───────────── compresses tool output into plain English
```
This is a tool designed for AI, not humans. No human reads the output of `analyze_errors` or `compare_logs` — Claude does, compresses it further, and gives the human a plain English answer. The human touches two endpoints: "what's wrong with this log?" in, natural language answer out. Everything in between is AI talking to itself.
## Tools
| Tool | Description |
|------|-------------|
| `log_overview` | Quick scan: size, line count, time range, level distribution, head/tail samples |
| `search_logs` | Search by regex, log level, and/or time range |
| `get_log_segment` | Extract a segment by line range or time range |
| `analyze_errors` | Deduplicate errors by fingerprint, count frequencies, extract stack traces |
| `log_stats` | Volume histogram, level breakdown, top repeated patterns |
| `compare_logs` | Find patterns unique to each file and frequency outliers across files |
| `classify_lines` | ML classifier (TF-IDF → BERT) separates interesting lines from noise |
## Key features
- **ML pre-filter** — a Rust TF-IDF classifier scans files at 1.3M lines/sec, so `analyze_errors` and `search_logs` only process the 5-30% of lines that matter. Optional BERT-mini re-scores LOOK lines at ~2K lines/sec on Metal GPU for higher precision. Works without parsed log levels — catches errors, security events, hardware faults, and anomalies that don't have ERROR in them.
- **Auto-detection** of log formats: JSON, standard text (`2024-01-15 10:30:45 ERROR ...`), syslog, Spark/Log4j (`17/06/08 13:33:49 INFO ...`), and tab/pipe-delimited formats (GitHub Actions CI logs)
- **Normalization** collapses variable parts (UUIDs, hex IDs, IPs, numbers) so that messages differing only in IDs or timestamps are grouped as the same pattern
- **Content-based error detection** falls back to regex heuristics (`fatal:`, `Permission denied`, `##[error]`, etc.) when log files lack standard levels
- **Prefix-aware comparison** distinguishes patterns from different job steps in CI logs
## Install
**Prerequisites** (fresh Mac):
```bash
brew install python uv
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
### Just ask Claude
Open a Claude Code session and paste this prompt:
```
Install https://github.com/ascii766164696D/log-mcp as an MCP server and build the Rust classifier too
```
Claude will clone the repo, register the MCP server, and build the Rust classifier. Restart Claude Code after to pick up the new server.
### Manual install
```bash
git clone https://github.com/ascii766164696D/log-mcp.git
cd log-mcp
# Register the MCP server
claude mcp add log-mcp -- uv run --directory $(pwd) log-mcp
# Build the Rust classifier (optional — tools fall back to Python without it)
uv pip install -e rust/classifier
```
Or add it manually to your project settings (`claude settings`) under `mcpServers`:
```json
{
"mcpServers": {
"log-mcp": {
"command": "uv",
"args": ["run", "--directory", "/path/to/log-mcp", "log-mcp"]
}
}
}
```
### Claude Desktop
Open **Settings > Developer > Edit Config** and add to `claude_desktop_config.json`:
```json
{
"mcpServers": {
"log-mcp": {
"command": "uv",
"args": ["run", "--directory", "/path/to/log-mcp", "log-mcp"]
}
}
}
```
Replace `/path/to/log-mcp` with the actual path where you cloned this repo. Restart Claude Desktop after saving.
The Rust classifier requires a Rust toolchain to build. The BERT stage additionally requires a Metal-capable GPU (Apple Silicon). All tools work without the classifier — they fall back to Python log parsing.
## Example usage
Analyze errors in a 67MB Spark executor log (705K lines):
```
> analyze_errors("/var/log/spark/container_0002_01_000004.log")
Summary: 34 errors in 5 groups.
Top: 'shuffle.RetryingBlockFetcher: Exception while beginning fetch of <N>
outstanding blocks (after <N> retries) ...' (18x)
--- 18x ---
Fingerprint: shuffle.RetryingBlockFetcher: Exception while beginning fetch of <N> outstanding blocks ...
First: L29764 2017-02-01T15:55:17
Last: L30677 2017-02-01T15:55:51
Stack trace:
java.io.IOException: Failed to connect to mesos-slave-13/10.10.34.23:55492
...
```
Find anomalies that don't have ERROR level — on a Thunderbird HPC log (2K lines), `classify_lines` finds 92 interesting lines while `search_logs level=ERROR` finds only 2:
```
> classify_lines("/var/log/thunderbird/tbird_2k.log")
Lines: 2,000 total | 92 LOOK (4.6%) | 1,908 SKIP
Pipeline: TF-IDF 0.00s (1,298,701 lines/sec, 105 LOOK) → BERT 0.15s (13 demoted to SKIP)
--- Sample LOOK lines (30 of 92 captured) ---
L2 [1.000] ... postfix/postdrop[10896]: warning: unable to look up public/pickup: No such file
L438 [0.999] ... sendmail[�20588]: unable to qualify my own domain name (tbird-sm1)
L816 [0.998] ... dhcpd: DHCPDISCOVER from 00:09:3d:12:00:e2 via eth2: unknown lease
L1024 [0.997] ... rrdtool: illegal attempt to update using time 1131710�721 when last update time is 1131710721
```
Compare two CI log files:
```
> compare_logs(["run_a.txt", "run_b.txt"])
699 patterns across 2 files (0 shared).
A: 401 unique (top: 'test / test UNKNOWN STEP | ##[endgroup]' 21x)
B: 298 unique (top: 'test UNKNOWN STEP | ##[endgroup]' 21x)
```
## Claude's take
*I helped build this tool and then used it to analyze real log files, so here's my honest assessment.*
**Where it genuinely helps:** The main value is as a compression layer. A 67MB Spark log (705K lines) would obliterate my context window, but `analyze_errors` distills it into 5 error groups with stack traces in a few seconds. `compare_logs` across two 1500-line server logs immediately surfaces which errors are unique to each server and which patterns have suspicious frequency differences. I couldn't do that by reading the files directly — I'd lose older content as new content scrolled in.
**Where the classifier changes the game:** Before the classifier, log analysis was limited to lines with explicit ERROR/FATAL levels. On a Thunderbird HPC log, `search_logs level=ERROR` returns 2 lines. `classify_lines` returns 92 — sendmail DNS failures, DHCP lease errors, RRD update collisions, negative boot times — none of which have ERROR level. The classifier finds what's semantically wrong, not just syntactically marked.
**Where it's a wash:** For small files (under a few hundred lines), you're better off just pasting the log into the conversation. The tools add indirection without much benefit when the whole file fits in context anyway.
**What it still can't do:** Domain-specific state machine reasoning. When I analyzed a Zookeeper log, the classifier correctly flagged `Cannot open channel` warnings and epoch resets, but the most operationally interesting signal — rapid cycling between LOOKING, FOLLOWING, and LEADING states — showed up as low-confidence LOOK lines. A Zookeeper expert would spot the pattern immediately; the classifier sees each line independently without tracking state transitions across time.
**The pattern I landed on:** Start with `classify_lines` to surface anomalies regardless of log level, then `analyze_errors` to group them, then `search_logs` to dig into specific patterns. `compare_logs` is most useful when you have a "working" and "broken" run to diff against each other.
## LOOK/SKIP classifier
The `classify_lines` tool uses a two-stage ML pipeline to separate interesting log lines (LOOK) from routine noise (SKIP), without needing parsed log levels.
### How it works
```mermaid
flowchart TD
A["Your log file (e.g. 4.7M lines)"] --> B{"Rust classifier<br/>available?"}
B -->|Yes| C["Stage 1: TF-IDF<br/>Rust, ~1.3M lines/sec<br/>logistic regression<br/>threshold × 0.6"]
B -->|No| D["Fallback: Python<br/>log parsing"]
C --> E["LOOK lines (~5-30%)"]
E --> F["Stage 2: BERT-mini<br/>Rust + Metal GPU, ~2K lines/sec<br/>re-scores LOOK lines<br/>applies final threshold"]
F --> G["Final LOOK lines<br/>(with BERT probabilities)"]
```
On a Thunderbird HPC log (2K lines), this finds **92 interesting lines** including sendmail DNS failures, DHCP lease errors, and Ganglia RRD update collisions — none of which have a standard ERROR log level. A `search_logs level=ERROR` on the same file returns only 2 lines.
### What it catches beyond log levels
| Signal type | Example | Has ERROR level? |
|---|---|---|
| DNS misconfiguration | `unable to qualify my own domain name` | No |
| Mail delivery failure | `stat=Deferred: Connection refused` | No |
| DHCP lease errors | `unknown lease 10.100.4.251` | No |
| Monitoring data corruption | `illegal attempt to update using time X when last update time is X` | No |
| Negative boot times | `Times: total = 42, boot = -4131` | No |
| Auth failures | `authentication failure; logname= uid=0` | No |
| Hardware errors | `instruction cache parity error corrected` | No (INFO level) |
### Tools that use the classifier
| Tool | How | Fallback |
|---|---|---|
| `classify_lines` | Full pipeline: TF-IDF → BERT | Error if classifier not installed |
| `analyze_errors` | TF-IDF pre-filters to LOOK lines, then groups by error fingerprint | Python parser scans all lines |
| `search_logs` | TF-IDF pre-filters when searching for errors (level=ERROR, pattern=error/fail/etc) | Python parser scans all lines |
| `compare_logs` | Scores patterns by LOOK probability, surfaces interesting diffs first | Sorts by frequency only |
## Benchmarks
Evaluated on the full [Loghub](https://github.com/logpai/loghub) dataset — 450 million lines across 16 log sources — on an Apple M3 Ultra (32 cores).
### TF-IDF classifier (Rust)
| Dataset | Lines | Time | Speed | LOOK % | Error capture |
|---|---|---|---|---|---|
| Thunderbird | 211M | 139s | 1.52M/s | 30% | 100% |
| Windows | 115M | 90s | 1.27M/s | 1% | 100% |
| HDFS_v2 | 71M | 59s | 1.21M/s | 20% | 100% |
| Spark | 33M | 23s | 1.44M/s | 12% | 100% |
| HDFS | 11M | 7s | 1.62M/s | 3% | 100% |
| BGL | 4.7M | 3.3s | 1.46M/s | 42% | 98.6% |
| Android | 1.6M | 1.1s | 1.37M/s | 12% | 100% |
| SSH | 655K | 0.5s | 1.47M/s | 90% | 100% |
| **Total** | **450M** | **325s** | **1.38M/s** | | **99.95%** |
Error capture = percentage of lines with ERROR/FATAL level that the classifier marks as LOOK. At threshold 0.3, the classifier catches 99.95% of all error lines (35.9M of 35.9M) and 100% of warning lines (14.3M of 14.3M). The 0.05% "missed" errors are lines like BGL's repetitive `instruction cache parity error corrected` — routine hardware telemetry that happens to carry ERROR level but isn't operationally interesting. The classifier learned to skip these.
### Model accuracy
| Model | Metric | Score |
|---|---|---|
| TF-IDF + LogReg | Best CV LOOK F1 | 0.792 |
| TF-IDF + LogReg | LOOK precision | 0.892 |
| BERT-mini | Overall accuracy | 0.849 |
| BERT-mini | LOOK F1 | 0.887 |
TF-IDF model evaluated with GroupKFold cross-validation (holdout: BGL, Thunderbird). BERT-mini fine-tuned on 4 layers, 256 hidden, 11M params.
### End-to-end reduction
```mermaid
flowchart TD
A["450M log lines<br/>16 datasets, ~67 GB"] --> B["Rust TF-IDF classifier<br/>1.38M lines/sec, 325s"]
B --> C["~87M LOOK lines (19%)<br/>81% of lines eliminated"]
C --> D["BERT re-scoring (optional)<br/>demotes 20-40% of TF-IDF LOOK"]
D --> E["~50-70M final LOOK lines"]
E --> F["Python tool logic<br/>group errors, search, etc."]
F --> G["5-50 error groups / search results<br/>fits in LLM context"]
```
## Retraining with your own logs
The classifier ships pre-trained, but you can retrain it on your own logs. The short version:
```bash
# 1. Add your logs
head -2000 /var/log/myapp/app.log > data/loghub/MyApp_2k.log
# 2. Label with Claude (Batch API, ~$0.10-0.50 per file)
export ANTHROPIC_API_KEY=sk-ant-...
uv sync --group labeling --group training
uv run python -m scripts.labeling.label_new
# 3. Train and export
uv run --group training python -m scripts.labeling.train_model
uv run --group training python -m scripts.labeling.export_model
# 4. Rebuild Rust classifier
uv pip install -e rust/classifier
```
See [`scripts/labeling/RETRAINING.md`](scripts/labeling/RETRAINING.md) for the full guide — how labeling works, what features the model uses, how to customize the prompt, and how to train the optional BERT model.
