内容
# log-essence
Extract the essence of your logs for LLM analysis.
Analyzes log files using template extraction (Drain3) and semantic clustering (FastEmbed) to produce token-efficient summaries for LLM consumption. Includes automatic secret/PII redaction for safe external analysis.
https://github.com/user-attachments/assets/97d66d89-1745-4d0f-ac10-38744f98b636
<p align="center">
<em>Raw logs → Token-efficient summary with automatic secret redaction</em>
</p>
## Features
- **Auto-detection**: JSON, syslog, Apache, nginx, Docker, Kubernetes log formats
- **Template extraction**: Drain3 algorithm identifies log patterns and groups similar messages
- **Semantic clustering**: Groups related patterns using FastEmbed embeddings
- **Token budget**: Respects LLM context limits with intelligent summarization
- **Secret redaction**: Correlation-preserving redaction of emails, IPs, API keys, credit cards
- **Error chain analysis**: Traces root causes through related log entries
- **Time filtering**: Filter logs by duration (1h, 30m, 2d) or datetime
- **Multi-source**: Files, directories, glob patterns, Docker containers, journald
- **Web UI**: Paste-and-copy interface with real-time processing metrics
## Installation
```bash
# Using uv (recommended)
uvx log-essence
# Using pip
pip install log-essence
```
## CLI Usage
```bash
# Analyze a log file
log-essence /var/log/app.log
# Analyze with glob pattern
log-essence "/var/log/*.log"
# Read from stdin — pipe any command's output
docker logs my-app 2>&1 | log-essence -
journalctl --since "1h ago" --no-pager | log-essence -
# Filter by severity
log-essence /var/log/app.log --severity ERROR WARNING
# Filter by time
log-essence /var/log/app.log --since 1h
# Strict redaction mode
log-essence /var/log/app.log --redact strict
# Disable redaction (for internal logs only)
log-essence /var/log/app.log --no-redact
# JSON output for programmatic use
log-essence /var/log/app.log -o json
# Watch mode for live log monitoring
log-essence /var/log/app.log --watch --interval 5
# Run as MCP server
log-essence --serve
```
### CLI Options
| Option | Description |
|--------|-------------|
| `--token-budget N` | Maximum tokens in output (default: 8000) |
| `--clusters N` | Number of semantic clusters (default: 10) |
| `--severity LEVEL...` | Filter by severity (ERROR, WARNING, INFO, DEBUG) |
| `--since TIME` | Only logs since TIME (1h, 30m, 2d, 2025-01-01) |
| `--redact MODE` | Redaction: strict, moderate (default), minimal, disabled |
| `--no-redact` | Disable redaction |
| `-o, --output FORMAT` | Output format: markdown (default) or json |
| `-w, --watch` | Watch log file for changes (live updates) |
| `--interval SECONDS` | Update interval for watch mode (default: 3.0) |
| `--config FILE` | Path to config file |
| `--profile NAME` | Use named configuration profile |
| `--serve` | Run as MCP server |
| `--version` | Show version number |
## Web UI
A browser-based interface for quick log analysis without command-line setup.
```bash
# Using uvx (recommended - no installation needed)
uvx --with log-essence[ui] log-essence ui
# Or specify a custom port
uvx --with log-essence[ui] log-essence ui --port 8080
# Using pip (if you prefer to install)
pip install log-essence[ui]
log-essence ui
```
### Features
- **Browser-based analysis**: Paste logs, get LLM-ready output with real-time metrics
- **Configurable settings**: Token budget, cluster count, redaction mode, severity filter
- **Processing metrics**: Real-time stats displayed after analysis
- **Time**: Processing duration
- **Redactions**: Number of secrets/PII items redacted
- **Tokens**: Original → Output token count
- **Savings**: Compression percentage achieved
- **Download**: Export analysis as markdown file
### UI Options
| Option | Description |
|--------|-------------|
| `--port N` | Port to run on (default: 8501) |
| `--no-browser` | Don't auto-open browser |
## MCP Server Usage
log-essence can run as an [MCP (Model Context Protocol)](https://modelcontextprotocol.io/) server, allowing Claude Desktop, Claude Code, Cursor, and other MCP-compatible clients to directly analyze logs from your system. This enables natural language interactions like:
> "Check the logs from the last hour for any database errors"
> "What's causing the slow response times in my API?"
> "Analyze the docker logs and find the root cause of the crash"
### Setup
Add to your Claude Desktop config (`~/Library/Application Support/Claude/claude_desktop_config.json`):
```json
{
"mcpServers": {
"log-essence": {
"command": "uvx",
"args": ["log-essence", "--serve"]
}
}
}
```
Restart Claude Desktop. You'll see log-essence listed in the MCP servers (🔌 icon).
### How It Works
1. **You ask Claude** about logs in natural language
2. **Claude calls log-essence** with the appropriate tool and parameters
3. **log-essence analyzes** the logs, redacts secrets, and returns a compressed summary
4. **Claude interprets** the results and provides actionable insights
The logs never leave your machine unredacted - log-essence strips sensitive data before Claude sees it, and the analysis runs locally using Drain3 and FastEmbed.
### Available Tools
#### `discover_log_sources`
Find log sources for a code project — project-scoped, so it returns the project's own log
files (vendored/VCS/build dirs pruned), the containers and Compose file belonging to that
project's Docker Compose stack, and — since most code projects log to stdout rather than
files — the **run commands** (from `package.json` scripts) you can pipe in, plus any standard
agent-instruction files (`CLAUDE.md`, `AGENTS.md`, `GEMINI.md`, …) that document how to run
and where logs go. It does not surface machine-wide system logs or other projects'
containers. Each result is annotated with the tool or command to use. Call this first when
you don't know what's available.
```python
discover_log_sources() # scan the current directory
discover_log_sources(path="/repo") # scan a specific project
```
#### `get_logs`
Analyze and consolidate log files.
```python
get_logs(
path="/var/log/app.log",
token_budget=8000,
num_clusters=10,
severity_filter=["ERROR", "WARNING"],
since="1h",
redact=True # or "strict", "minimal", False
)
```
#### `get_container_logs`
Analyze Docker container logs.
```python
get_container_logs(
container="my-app",
since="1h",
token_budget=8000
)
```
#### `get_docker_logs`
Analyze logs from Docker Compose services.
```python
get_docker_logs(
path="/path/to/project",
services=["api", "worker"],
since="30m"
)
```
#### `get_error_chain`
Trace error root causes through related log entries.
```python
get_error_chain(
path="/var/log/app.log",
error_pattern="database",
time_window=60
)
```
#### `search_logs`
Semantic search through log entries.
```python
search_logs(
path="/var/log/app.log",
query="connection timeout",
top_k=10
)
```
#### `get_journald_logs`
Analyze systemd journal logs.
```python
get_journald_logs(
unit="nginx.service",
since="1h",
priority="err"
)
```
#### `list_containers`
List running Docker containers.
```python
list_containers()
```
#### `list_docker_services`
List Docker Compose services in a project.
```python
list_docker_services(path="/path/to/project")
```
### Per-cluster retrieval
Every analysis tool (`get_logs`, `get_docker_logs`, `get_container_logs`,
`get_journald_logs`) ends its summary with an `analysis_id`. Pass it back to
`get_raw_logs` to pull the full, **redacted** lines on demand:
- `get_raw_logs(analysis_id)` — all lines (paginate with `start_line`/`max_lines`).
- `get_raw_logs(analysis_id, cluster_id=N)` — only the lines behind "Cluster N"
from the summary, so an agent can expand exactly the cluster under
investigation without pulling the whole log back.
Retrieved lines carry the same redaction as the summary (redacted unless that
analysis was run with `redact=False`).
## Secret Redaction
Logs are automatically redacted before analysis to prevent leaking sensitive data to external LLMs.
### Redaction Modes
| Mode | Description |
|------|-------------|
| `moderate` | Default. Emails, IPs, credit cards, SSNs, phones, API keys |
| `strict` | All moderate patterns + high-entropy strings in key=value |
| `minimal` | Only obvious secrets (bearer tokens, API keys) |
| `disabled` | No redaction (use only for internal logs) |
### Output Format
Redacted values use the format `[TYPE:length?:hash4]`:
```
# Input
user@acme.com logged in from 192.168.1.50
Error processing payment for user@acme.com card 4111111111111111
# Output (same entity → same hash for correlation)
[EMAIL:a7f2] logged in from [IPV4:3bc1]
Error processing payment for [EMAIL:a7f2] card [CC:16:d4e8]
```
### Detected Patterns
**PII:**
- Email addresses
- IPv4 and IPv6 addresses
- Credit card numbers (Luhn-validated)
- Social Security Numbers (xxx-xx-xxxx)
- Phone numbers
**Secrets:**
- AWS access keys and secret keys
- GitHub tokens (ghp_, ghs_)
- Stripe API keys (sk_live_, sk_test_)
- JWT tokens
- Bearer tokens
- Private key headers
- Connection strings (postgres://, mongodb://, redis://)
## Example Output
```markdown
# Log Analysis Summary
**Format detected:** docker
**Total lines:** 15,432
**Unique patterns:** 47
**Semantic clusters:** 10
---
## Log Patterns by Severity
### Cluster 1: Database Operations
**Occurrences:** 5,234 | **Patterns:** 8
- `Query executed in <*>ms` (2,341x)
- `Connection pool size: <*>` (1,892x)
- `Transaction committed` (1,001x)
**Example:**
```
2025-01-01T10:00:00Z INFO Query executed in 45ms
```
### Cluster 2: HTTP Requests
**Occurrences:** 4,123 | **Patterns:** 5
- `[IPV4:3bc1] - GET /api/<*> <*>` (3,456x)
- `Response time: <*>ms` (667x)
```
## Development
```bash
# Clone and install
git clone https://github.com/petebytes/log-essence
cd log-essence
uv sync --all-groups
# Run tests
uv run pytest
# Lint
uv run ruff check src/ tests/
# Format
uv run ruff format src/ tests/
```
## License
MIT
