Content
<h2 align="center">Kloakt</h2>
<p align="center">
<strong>Cloaked headless browser for AI agents.</strong><br>
Lightweight, stealthy, built in Rust. Based on <a href="https://github.com/h4ckf0r0day/obscura">Obscura</a>.
</p>
---
<!-- mcp-name: io.github.KultMember6Banger/kloakt -->
Kloakt is a headless browser built for AI agents. It runs JavaScript via V8, extracts clean markdown from any page (including SPAs), and exposes tools via MCP for Claude Code and other AI systems.
Beyond one-shot extraction it can drive **persistent, stateful sessions** (click, type, navigate — cookies *and* page/JS state persist across calls), emit an **accessibility/structure snapshot** as an agent-vision substitute, and capture **real screenshots** via system Chrome — 12 MCP tools in all.
### Why Kloakt?
| Metric | Kloakt | Headless Chrome |
|--------------|--------------|------------------|
| Memory | **30 MB** | 200+ MB |
| Binary size | **70 MB** | 300+ MB |
| Anti-detect | **Built-in** | None |
| Page load | **85 ms** | ~500 ms |
| Startup | **Instant** | ~2s |
| SPA extract | **Yes** | Manual |
## Install
### Prebuilt binary (recommended)
One-line install (Linux & macOS). Downloads the right binary for your OS/arch from the latest GitHub Release and installs it to `~/.local/bin` (or `/usr/local/bin` when run as root):
```bash
curl -fsSL https://raw.githubusercontent.com/KultMember6Banger/kloakt/main/install.sh | sh
```
You can pin a version or override the install dir:
```bash
KLOAKT_VERSION=v0.1.2 INSTALL_DIR=/usr/local/bin \
sh -c "$(curl -fsSL https://raw.githubusercontent.com/KultMember6Banger/kloakt/main/install.sh)"
```
Windows: download `kloakt-x86_64-windows.zip` from the [Releases page](https://github.com/KultMember6Banger/kloakt/releases) and extract `kloakt.exe` onto your `PATH`.
### Homebrew (macOS)
```bash
brew install KultMember6Banger/kloakt/kloakt
# or, from a local checkout:
brew install --formula ./Formula/kloakt.rb
```
(Until a dedicated tap exists, `brew tap KultMember6Banger/kloakt https://github.com/KultMember6Banger/kloakt` then `brew install kloakt`.)
### cargo install
Builds the CLI from crates.io (requires Rust toolchain; first build compiles V8, ~5 min):
```bash
cargo install obscura-cli
```
This installs the `kloakt` binary. To build with stealth mode, add `--features stealth`.
### Build from source
```bash
git clone https://github.com/KultMember6Banger/kloakt.git
cd kloakt
cargo build --release
# With stealth mode (anti-detection + tracker blocking)
cargo build --release --features stealth
```
Requires Rust 1.75+ ([rustup.rs](https://rustup.rs)). First build takes ~5 min (V8 compiles from source, cached after).
## Quick Start
### Extract content (AI agent use)
```bash
# Clean markdown from any page
kloakt extract https://example.com --main
# Structured JSON with metadata
kloakt extract https://example.com --main --json
# Cap output for agent context windows
kloakt extract https://en.wikipedia.org/wiki/Rust --main --json --max-chars 3000
# Wait for SPA hydration
kloakt extract https://example.com --delay 2000 --json
```
### Fetch a page
```bash
# Get the page title
kloakt fetch https://example.com --eval "document.title"
# Extract all links
kloakt fetch https://example.com --dump links
# Render JavaScript and dump markdown
kloakt fetch https://news.ycombinator.com --dump markdown
# Wait for dynamic content
kloakt fetch https://example.com --wait-until networkidle0
```
### Start the CDP server
```bash
kloakt serve --port 9222
# With stealth mode
kloakt serve --port 9222 --stealth
```
### Scrape in parallel
```bash
kloakt scrape url1 url2 url3 ... \
--concurrency 25 \
--eval "document.querySelector('h1').textContent" \
--format json
```
### Snapshot page structure (agent vision)
```bash
# Indexed accessibility/structure tree — tags, text, roles, what's clickable, visibility
kloakt snapshot https://example.com
# Only the actionable elements (links, buttons, inputs), with id/name/label for targeting
kloakt snapshot https://example.com --interactive
```
kloakt has no rasterizer, so this is the lightweight "what's on the page and what can I act on" view for agents that work from structure rather than pixels.
### Screenshot (via system Chrome)
```bash
# Real PNG — delegates to a locally-installed Chrome/Chromium/Edge
kloakt screenshot https://example.com --output shot.png --width 1280 --height 800
```
### Persistent sessions
Drive a named session whose cookies **and** page/JS state survive across separate
invocations, backed by a running `kloakt serve` daemon:
```bash
kloakt serve --port 9222 & # start the daemon once
kloakt session open shop --url https://example.com
kloakt session snapshot shop --interactive # see the page structure
kloakt session type shop 'input[name=q]' 'hello' # fill a field
kloakt session click shop 'button[type=submit]' # click an element
kloakt session text shop # read the body text
kloakt session eval shop 'document.title' # run JS, get the value back
kloakt session close shop # tear down (drops cookies + page)
```
## Smart Extraction
The `extract` command uses a multi-phase pipeline optimized for AI agents:
1. **Noise removal** — strips cookie banners, ads, popups, nav, social widgets
2. **Content scoring** — text-density algorithm (Readability-like) finds the main content block
3. **Markdown conversion** — DOM-to-markdown with absolute URL resolution
4. **SPA fallback** — when JS rendering fails, extracts from meta tags, Open Graph, JSON-LD, and noscript content
Works on static HTML, server-rendered pages, and pure client-side SPAs (React, Vue, etc.).
## Python API
```python
from kloakt import (
extract, extract_fields, fetch, scrape, search, crawl,
snapshot, screenshot, session_open, session_close,
)
# Extract clean markdown
page = extract("https://example.com")
print(page.title, page.content, page.meta)
# Cap output length
page = extract("https://example.com", max_chars=3000)
# Wait for SPA content
page = extract("https://example.com", delay=2000)
# Structured field extraction via CSS selectors
data = extract_fields("https://news.ycombinator.com", {
"title": "title",
"stories": ".titleline > a[]", # [] => list of all matches
"links": ".titleline > a[]@href" # @href => an attribute
})
print(data["data"]["stories"])
# Raw fetch
html = fetch("https://example.com", dump="html")
title = fetch("https://example.com", eval_js="document.title")
# Parallel scrape
results = scrape(["https://a.com", "https://b.com"], concurrency=5)
# Discover links, or breadth-first crawl a small section of a site
links = search("https://news.ycombinator.com", same_domain=True)
pages = crawl("https://example.com", max_pages=5, max_depth=1)
# Structure snapshot (agent vision) and a real screenshot via system Chrome
snap = snapshot("https://example.com", interactive=True)
screenshot("https://example.com", output="shot.png")
# Persistent session — auto-starts a daemon if one isn't running; cookies + page
# state persist across calls. (session_nav / _click / _type / _eval / _text / _snapshot)
session_open("shop", url="https://example.com")
# ... drive the page across calls ...
session_close("shop")
```
## MCP Server (Claude Code)
Kloakt includes an MCP server for use as a Claude Code tool:
```json
{
"mcpServers": {
"kloakt": {
"command": "python3",
"args": ["/path/to/kloakt/mcp_server.py"]
}
}
}
```
Exposes 12 native tools:
| Tool | What it does |
|------|--------------|
| `kloakt_extract` | Clean markdown, or structured fields via `schema` |
| `kloakt_fetch` | Low-level fetch (html/text/links/markdown, or JS eval) |
| `kloakt_scrape` | Many URLs in parallel |
| `kloakt_search` | Discover outbound links on a page |
| `kloakt_crawl` | Budget/depth-limited breadth-first crawl |
| `kloakt_snapshot` | Accessibility/structure tree (agent vision) |
| `kloakt_screenshot` | Real PNG via system Chrome |
| `kloakt_session_open` | Open a persistent named session |
| `kloakt_session_act` | navigate / click / type / eval within a session |
| `kloakt_session_read` | Read a session's page as text or snapshot |
| `kloakt_session_list` | List open sessions |
| `kloakt_session_close` | Close a session (drops its cookies + page) |
## Puppeteer / Playwright
### Puppeteer
The CDP server embeds a per-session token in the WebSocket path (like Chrome). Connect via
`browserURL` so the client discovers the token from `/json/version` automatically — don't
hardcode the `ws://.../devtools/browser` path.
```javascript
import puppeteer from 'puppeteer-core';
const browser = await puppeteer.connect({
browserURL: 'http://127.0.0.1:9222', // discovers the tokenized ws endpoint
});
const page = await browser.newPage();
await page.goto('https://news.ycombinator.com');
const stories = await page.evaluate(() =>
Array.from(document.querySelectorAll('.titleline > a'))
.map(a => ({ title: a.textContent, url: a.href }))
);
await browser.disconnect();
```
### Playwright
```javascript
import { chromium } from 'playwright-core';
const browser = await chromium.connectOverCDP({
endpointURL: 'http://127.0.0.1:9222', // discovers the tokenized ws endpoint
});
const page = await browser.newContext().then(ctx => ctx.newPage());
await page.goto('https://en.wikipedia.org/wiki/Web_scraping');
console.log(await page.title());
await browser.close();
```
## Stealth Mode
Enable with `--features stealth`.
- Per-session fingerprint randomization (GPU, screen, canvas, audio, battery)
- Realistic `navigator.userAgentData` (Chrome 145, high-entropy values)
- `event.isTrusted = true` for dispatched events
- Native function masking (`Function.prototype.toString()` → `[native code]`)
- `navigator.webdriver = undefined`
- Realistic `Accept-Language` + Client Hints (`Sec-CH-UA`) request headers
- Per-session randomized `navigator.languages`
- TLS fingerprint (JA3) rotation across Chrome 145 Linux / Windows / macOS profiles
- 3,520 tracker domains blocked
## CLI Reference
### `kloakt extract <URL>`
| Flag | Default | Description |
|------|---------|-------------|
| `--format` | `markdown` | Output: `markdown`, `text`, or `links` |
| `--main` | off | Strip nav, header, footer, sidebar |
| `--json` | off | Structured JSON: title, URL, content, meta |
| `--max-chars` | unlimited | Truncate content to N characters |
| `--delay` | `0` | Extra ms to wait after load |
| `--stealth` | off | Anti-detection mode |
| `--selector` | — | Wait for CSS selector |
| `--wait-until` | `load` | `load`, `domcontentloaded`, `networkidle0` (bounded by `--wait`) |
| `--schema` | — | Extract structured fields as JSON (see below) |
| `--har` | — | Write captured network activity to a HAR file |
| `--cache-ttl` | `0` | Cache the result on disk and reuse it for N seconds |
#### Structured extraction with `--schema`
Pass a JSON object mapping field names to CSS selectors. Suffix a selector with `[]` to
return **all** matches as a list, and with `@attr` to return an **attribute** instead of text:
```bash
kloakt extract https://news.ycombinator.com \
--schema '{"title":"title","stories":".titleline > a[]","first_link":".titleline > a@href"}'
# => { "url": ..., "data": { "title": "...", "stories": [...], "first_link": "..." }, "elapsed_ms": ... }
```
This is also exposed through the MCP `kloakt_extract` tool via an optional `schema` argument.
### `kloakt fetch <URL>`
| Flag | Default | Description |
|------|---------|-------------|
| `--dump` | `html` | Output: `html`, `text`, `links`, `markdown` |
| `--eval` | — | JavaScript expression to evaluate |
| `--wait-until` | `load` | Wait condition |
| `--selector` | — | Wait for CSS selector |
| `--stealth` | off | Anti-detection mode |
| `--quiet` | off | Suppress banner |
### `kloakt serve`
| Flag | Default | Description |
|------|---------|-------------|
| `--port` | `9222` | WebSocket port |
| `--proxy` | — | HTTP/SOCKS5 proxy URL |
| `--stealth` | off | Anti-detection + tracker blocking |
| `--workers` | `1` | Parallel workers |
### `kloakt scrape <URL...>`
| Flag | Default | Description |
|------|---------|-------------|
| `--concurrency` | `10` | Parallel workers |
| `--eval` | — | JS expression per page |
| `--format` | `json` | Output: `json` or `text` |
### `kloakt snapshot <URL>`
Emit an indexed accessibility/structure tree (always JSON) — an agent-vision substitute.
Each node is compact: `i` (index), `tag`, `depth`, `vis` (visible), and when present `click`,
`role`, `text`, `type`/`value`, `href`, `id`, `name`, `label`.
| Flag | Default | Description |
|------|---------|-------------|
| `--interactive` | off | Only actionable elements (links, buttons, inputs) |
| `--max-nodes` | `1500` | Cap on nodes emitted |
| `--stealth` | off | Anti-detection mode |
| `--delay` | `0` | Extra ms to wait after load |
| `--wait-until` | `load` | Wait condition |
### `kloakt screenshot <URL>`
Capture a real PNG by delegating to a locally-installed Chrome/Chromium/Edge (kloakt has no
rasterizer). Errors clearly if none is found; override detection with `--chrome <path>` or the
`KLOAKT_CHROME` env var.
| Flag | Default | Description |
|------|---------|-------------|
| `--output` | `screenshot.png` | Output PNG path |
| `--width` | `1280` | Viewport width |
| `--height` | `800` | Viewport height |
| `--chrome` | auto-detect | Path to a Chrome/Chromium/Edge binary |
### `kloakt session <COMMAND>`
Drive a persistent, named session against a running `kloakt serve` daemon. Cookies **and**
page/JS state survive across separate invocations (client state in `~/.kloakt/sessions/<name>.json`).
| Command | Description |
|---------|-------------|
| `open <name> [--url URL] [--port]` | Open (or reattach to) a session and create a page |
| `nav <name> <url>` | Navigate the session's page |
| `eval <name> <expr>` | Evaluate a JS expression, print the JSON value |
| `text <name>` | Print `document.body.innerText` |
| `snapshot <name> [--interactive]` | Structure snapshot of the session's page |
| `click <name> <selector>` | Click the first matching element |
| `type <name> <selector> <text>` | Focus an element, set its value, fire input/change |
| `list [--port]` | List open sessions on the daemon |
| `close <name>` | Close the session (drops its pages + cookies) |
> Multi-statement JS passed to `eval` returns `null` (the daemon evaluates a single
> expression); wrap it in an IIFE — `(function(){ ...; return v })()` — to get a value back.
### `kloakt benchmark <URL...>`
Measure load performance per URL — average/min/max load time, request count, bytes, and DOM
node count — as a table or `--json`.
```bash
kloakt benchmark https://example.com https://news.ycombinator.com --runs 3
```
| Flag | Default | Description |
|------|---------|-------------|
| `--runs` | `1` | Runs per URL (reports the average) |
| `--json` | off | Emit JSON instead of a table |
| `--wait-until` | `load` | `load`, `domcontentloaded`, or `networkidle0` |
### Challenge / bot-wall detection
`kloakt extract --json` includes a `"challenge"` field reporting a detected captcha or bot
wall (`recaptcha`, `hcaptcha`, `turnstile`, `cloudflare`, `datadome`, `perimeterx`) or
`null`. This is **detection only** — kloakt tells you a page is gated so an agent can stop
and back off; it does not attempt to solve or evade challenges. (Also surfaced via the MCP
`kloakt_extract` output and the Python `Page.challenge` field.)
### Global flags
| Flag | Default | Description |
|------|---------|-------------|
| `--obey-robots` | off | Respect `robots.txt` — refuse to fetch disallowed paths |
| `--allow-private` | off | Allow private/internal/loopback hosts (disables the SSRF guard) |
> **Security note:** by default kloakt refuses to fetch private, loopback, link-local, and
> cloud-metadata addresses (SSRF protection), and rejects `file://` URLs. Use `--allow-private`
> only when you intentionally need to reach internal services. The CDP server binds to
> `127.0.0.1` and validates the `Host` header to block DNS-rebinding.
## CDP API
Full Chrome DevTools Protocol support for Puppeteer/Playwright compatibility.
| Domain | Methods |
|--------|---------|
| **Session** | open, close, list — named, persistent browser contexts that keep cookies alive |
| **Target** | createTarget (`browserContextName` binds a page to a session), closeTarget, attachToTarget, createBrowserContext, disposeBrowserContext |
| **Page** | navigate, getFrameTree, addScriptToEvaluateOnNewDocument, lifecycleEvents |
| **Runtime** | evaluate, callFunctionOn, getProperties, addBinding |
| **DOM** | getDocument, querySelector, querySelectorAll, getOuterHTML, resolveNode |
| **Network** | enable, setCookies, getCookies, setExtraHTTPHeaders, setUserAgentOverride |
| **Fetch** | enable, continueRequest, fulfillRequest, failRequest |
| **Storage** | getCookies, setCookies, deleteCookies |
| **Input** | dispatchMouseEvent, dispatchKeyEvent |
## License
Apache 2.0 — Based on [Obscura](https://github.com/h4ckf0r0day/obscura) by h4ckf0r0day.
