Osprey

# Osprey Osprey is a penetration-testing harness with an evidence-driven, non-LLM investigation loop, an optional LLM judgment layer, a FastAPI capability backend, an MCP gateway, and an interactive Python CLI. ## Architecture - `backend/` — FastAPI control plane, API endpoints, database models, workflow logic - `cli/` — Osprey CLI tool (prompt_toolkit + Rich) for interacting with the platform - `mcp-servers/` — Capability-based tool adapters for recon, osint, network, web, vuln, … - `platform-mcp/` — MCP gateway server connecting AI clients to the platform - `docker-compose.yml` — Postgres + (optional) Kali Linux tools + FastAPI backend **How tools run.** The backend never runs scanners itself — it `docker exec`s them into a tools container (`KALI_CONTAINER`), or, when no such container is running, executes them **natively on the host**. So you can run the full Kali image, bring your own tools, or run entirely without Docker. ## Setup — pick the path that fits you **Fastest way to start: the guided wizard.** It asks the same questions as the manual paths below, writes `.env` for you, and — for the fully-local path — detects and offers to install missing tools. Nothing it does is hidden magic; every command it runs is also spelled out in the manual sections below. ```bash bash scripts/install.sh # Linux / macOS / WSL / Kali powershell -ExecutionPolicy Bypass -File scripts\install.ps1 # Windows ``` Prefer to do it by hand, or just want to know what's happening under the hood? Quick guide: - **Want it to just work, no local tools needed?** → **Path A** (full Docker + Kali, all tools included — pulls a prebuilt image). - **Have your own tools *container*?** → **Path B** (lean Docker, point `KALI_CONTAINER` at it). - **Want to use tools already on your machine, or no Docker at all?** → **Path C** (fully local). This is the right choice for host tools — a containerized backend (Path B) cannot reach them. All three paths start the same way: ```bash cp .env.example .env # Edit .env: set LLM_MODEL + LLM_API_KEY (any LiteLLM provider — OpenAI, Anthropic, # Gemini, Groq, DeepSeek, Ollama, …). See the comments in the file. ``` ### Path A — Full Docker stack (easiest; no local tools needed) Builds Postgres, a Kali container with 90+ tools, and the backend. ```bash docker compose --profile kali up -d ``` This **pulls the prebuilt `ospreytools/osprey-kali` image from Docker Hub** instead of building it. Without it the backend runs tools natively or against your own `KALI_CONTAINER` (see Path B/C). > No WSL / VirtualBox needed on Windows — the tools run inside the container, > reached over the Docker socket. Postgres is on host port 5433, backend on 9000. > First pull is a few GB over the network — much less than a from-source build, but > still worth doing on a good connection. **Want to build the image yourself instead** (you modified `kali-tools/Dockerfile`, need a platform the prebuilt image doesn't cover, or don't want to trust a third-party image)? Add the build override — this takes the original ~40-60 min / ~16GB disk: ```bash docker compose --profile kali -f docker-compose.yml -f docker-compose.build.yml up -d --build ``` Pin or redirect the pulled image with `OSPREY_KALI_IMAGE` in `.env` (e.g. to a specific tag/sha, or your own mirror) — see [`.env.example`](.env.example). The image is rebuilt and republished automatically on every change to `kali-tools/` (see [`.github/workflows/build-kali-image.yml`](.github/workflows/build-kali-image.yml)). ### Path B — Lean Docker + your own tools container (skip the Kali build) Already have a tools container (a native Kali image, or your own) and don't want to build ours? Start just Postgres + backend, then point the backend at that container: ```bash docker compose up --build # note: no --profile kali ``` ``` # in .env — the name of your already-running tools container: KALI_CONTAINER=<your-tools-container-name> ``` The backend `docker exec`s each tool into that container (it mounts the Docker socket for this), so the container must be running on the same Docker daemon. > **Important — Path B needs a tools *container*, not host tools.** The backend > itself runs inside a container here, so it can only reach tools in another > container it can `docker exec` into — it **cannot** run binaries installed on your > host machine, and there is no automatic "fall back to host tools". If > `KALI_CONTAINER` is unset or its container isn't running, only the built-in Python > tools work and every compiled scanner returns *"not available in your execution > environment"* (with the fix). **To use the tools already installed on your host, > use Path C** — a native backend, which runs them directly. `/tools` and > `/health` show your real execution mode and what's runnable. ### Path C — Fully local, no Docker (uses your host's own tools) The path for running Osprey against tools already on your machine, with no Docker at all. Virtualenv at the repo root, SQLite instead of Postgres, tools on your host PATH: ```bash # creates .venv, installs backend[dev] + cli + native tool deps bash scripts/setup.sh # Linux / macOS / WSL / Kali # or on Windows: powershell -ExecutionPolicy Bypass -File scripts\setup.ps1 ``` In `.env` set a keyless DB and native execution: ``` DATABASE_URL=sqlite:///./pentest.db KALI_CONTAINER= ``` Then run the backend (the SQLite schema is created on startup): ```bash uvicorn osprey.main:app --host 0.0.0.0 --port 9000 ``` Compiled scanners (nmap, subfinder, httpx, nuclei, …) must be on your PATH; the keyless custom Python tools (domain_hunter, js_recon, subdomain_takeover, contact harvest, …) work from `mcp-servers/requirements.txt` alone. **`/tools` shows exactly what's present** (see below) and the agent only uses tools it actually has. ## Optional API keys Most tools are keyless. A few tap external intel services — set their keys in `.env` (see [`.env.example`](.env.example)) to unlock them; leave blank to skip. Keys are forwarded into the tools container automatically. | Tool(s) | Env var | Get a key | | ------- | ------- | --------- | | `shodan_search`, `shodan_host_info` | `SHODAN_API_KEY` | https://account.shodan.io | | `intelx_scan` — email/subdomain harvest | `INTELX_API_KEY` | https://intelx.io/account?tab=developer | | `intelx_scan` — **leaked credentials** (Identity API) | `INTELX_IDENTITY_API_KEY` | IntelX Identity Portal licence | | `resecurity_scan` — leaked credentials | `RESECURITY_API_KEY` (+ optional `RESECURITY_API_BASE`/`RESECURITY_ENDPOINT`) | https://resecurity.com | | `domain_hunter`'s `ai_hunt` module — LLM-hypothesized sister domains | `dom_hunter_gemini_key` / `dom_hunter_anthropic_key` / `dom_hunter_openai_key` (lowercase — read directly by the tool, first one set wins) | whichever provider you already use | ## Dependencies (pip) `pyproject.toml` is the source of truth for each package's declared dependencies: - `backend/pyproject.toml` — backend runtime + `[dev]` extras (pytest, ruff, httpx) - `cli/pyproject.toml` — CLI runtime - `mcp-servers/requirements.txt` — Python deps for native tool wrappers The `requirements.txt` files are generated pinned locks (regenerate with `pip-compile pyproject.toml`). ## Run Osprey CLI Prerequisites: the backend is up, and `LLM_MODEL`/`LLM_API_KEY` are set in the CLI's own `.env` (any LiteLLM-supported provider). The CLI runs its own local agent loop — the same tool surface OpenCode/Claude drive via MCP, with your own key — it does not depend on any backend-side agent config. If you used `scripts/setup.*`, the CLI is already installed. Otherwise: ```bash pip install -e ./cli # from the repo root (Windows + Linux) ``` Run it (connects to `http://localhost:9000`; override with `API_BASE_URL`): ```bash osprey # interactive CLI, from any working directory osprey scan example.com # scriptable scan entry point osprey run --target example.com "show me the current attack surface" ``` `python -m cli` and the legacy `pentest` console script still work. > **Engine mode (`/scan <target> --engine`) needs a working tool backend.** It runs > scanners with no LLM, so the Kali tools container must be up: > `docker compose --profile kali up -d` (or point `KALI_CONTAINER` at your own tools > container / set it empty for native host tools). ### CLI commands | Command | Description | | ----------------------------- | ------------------------------------------------- | | `/help` | Show available commands | | `/health` | Check backend service health | | `/tools [--all]` | Missing tools + install hints (default), or every tool with `--all` | | `/models` | List configured LLM models | | `/model` | Show active model + key status | | `/scan <target> [phase]` | Bind engagement + scan (`recon`/`network`/`full`) | | `/engage list` | List all engagements | | `/engage new <target>` | Create a fresh engagement for a target | | `/engage set <id>` | Bind this session to an existing engagement | | `/findings` | Show findings for the active engagement | | `/tool [id]` | List recent tool calls or expand one transcript | | `/output [id]` | Alias for `/tool` | | `/chat [count]` | Show this CLI session's model conversation | | `/details [mode]` | Set live output detail: `compact`, `preview`, or `verbose` | | `/status` | Backend / model / active engagement status | | `/config` · `/config reload` | Show config · force backend to re-read `.env` | | `/reconnect` | Re-read `.env` and reconnect to `API_BASE_URL` | | `/reset` · `/clear` · `/exit` | Clear conversation · clear screen · exit | `/scan <target> [phase]` binds an engagement so `/findings` and prompts know the target. You can also just **type natural-language prompts** — these go to the CLI harness. A full scan follows the same visible sense → decide → execute → evaluate → replan lifecycle with or without an LLM. The deterministic driver is the baseline intelligence; a configured model may reorder current bounded opportunities and add analysis, but cannot invent capability parameters. During a run the CLI streams decisions, individual tool progress, evidence, and replans. Configure the CLI's optional model with `LLM_MODEL`, `LLM_API_KEY`, and `LLM_API_BASE` in `.env`; `/model reload` refreshes it without moving the engagement. The CLI owns the conversation. Jobs, observations, findings, graph state, and investigation decisions remain durable in the backend engagement. ### Seeing which tools you have Tool availability is environment-aware: the backend probes each binary in whatever execution mode is active (Kali container or host). Surface it with: - **CLI:** `/tools` — missing tools + how to install each; `/tools --all` for every tool. - **MCP / harness:** the `platform_tools` tool, or `GET /api/v1/tools/catalog` (returns per-tool `installed` plus a summary count). The agent plans around what's present, so a partial toolset still works — you just get fewer techniques where a tool is missing. ## Adding your own skills (drop-in, zero code) Skills are markdown files under `skills/<phase>/` that steer the agent's methodology for a phase. Both executors index them by their YAML frontmatter **description** and surface the descriptions for the *active* phase every turn, pulling full text on demand — so a well-described skill can't be missed, and adding more never bloats the prompt. To add one, drop a file into the right phase folder with frontmatter: ```markdown --- name: my-graphql-idor-playbook description: How I hunt IDOR in GraphQL mutations on this org's APIs — the checks and payloads that work here. phase: web tags: [web, graphql, idor] --- # My GraphQL IDOR playbook ...your methodology... ``` - **`phase`** is which phase it appears in: `recon`, `network`, `web`, `vuln`, `exploit`, `osint`, `commander`, or `shared` (always-relevant). The folder you drop it in usually matches, but the `phase:` field is authoritative — so a file under `skills/custom/` with `phase: web` shows up during the web phase too. - **`description`** is the single most important field: it's what the LLM reads to decide whether to open the skill. Make it concrete (when to use it + what it does). - No restart or code change needed — the registry re-reads `skills/` on each request. **Learned skills (runtime).** You don't have to hand-place files. The agent can capture a reusable technique it discovers with `platform_propose_skill` (you approve it with `/skill approve <id>` in the CLI), and you can author one yourself in one step with `/skill add <file.md>`. These land in a git-ignored, user-local `skills/learned/` (never committed or shipped), indexed like any other skill. The tier is on by default (`enable_learned_skills`); LLM proposals always wait for your approval, and a proposal is rejected if it just restates a skill you already have. Verify it's indexed: `GET /api/v1/capabilities/skills-index?phase=web` (or `platform_skills(phase="web")` from an MCP harness) — your skill's `name` and `description` should appear. ## MCP Client Configuration (Alternate Approach) Connect your AI client directly to the platform via MCP. All clients use the same MCP server at `platform-mcp/server.py`. Below are setup guides for some harnesses, setup in others is similar. ### OpenCode CLI recommended instead of GUI for detailed commands outputs Edit `~/.config/opencode/opencode.json` (global config — works in any project): ```jsonc { "$schema": "https://opencode.ai/config.json", "mcp": { "osprey": { "type": "local", "command": ["python", "C:\\path\\to\\osprey\\platform-mcp\\server.py"], "env": { "PENTEST_API_BASE": "http://localhost:9000", "PENTEST_QUICK_TIMEOUT": "30", "PENTEST_HTTP_TIMEOUT": "900" }, "enabled": true, "timeout": 600000 } } } ``` Replace `C:\\path\\to` with the absolute path to the project. The global config makes `osprey` tools available in any repository you open with OpenCode. Verify with `/tools` in the chat — you should see all MCP tools listed. ### Claude Desktop (native MCP support) In Claude Desktop app, go to Settings -> Developer -> Edit Config -> Edit `claude_desktop_config.json`: ```json { "mcpServers": { "osprey": { "command": "python", "args": [ "C:\\path\\to\\osprey\\platform-mcp\\server.py" ], "cwd": "C:\\path\\to\\osprey", "env": { "PENTEST_API_BASE": "http://localhost:9000", "PENTEST_QUICK_TIMEOUT": "30", "PENTEST_HTTP_TIMEOUT": "900" } } } } ``` Replace `C:\\path\\to` with your actual project path. You should see `osprey` listed as a running MCP server in Claude's settings. ### ChatGPT Desktop You can configure your MCP server either from the graphical interface or directly through `config.toml`. #### Option 1 (Recommended): GUI Configuration Open Settings → Plugins → MCPs → Add Server Configure the server as follows. ##### Name: osprey ##### Type: STDIO ##### Command System Python: ``` python ``` or preferably your virtual environment: ``` C:\path\to\osprey\.venv\Scripts\python.exe ``` ##### Arguments ``` C:\path\to\osprey\platform-mcp\server.py ``` ##### Environment Variables | Variable | Value | |----------|-------| | PENTEST_API_BASE | http://localhost:9000 | | PENTEST_QUICK_TIMEOUT | 30 | | PENTEST_HTTP_TIMEOUT | 900 | ##### Working Directory ``` C:\path\to\osprey ``` Click **Save**. If the configuration is valid, ChatGPT Desktop will automatically launch the MCP server whenever it is needed. --- #### Option 2: config.toml Instead of using the GUI, you can edit ChatGPT Desktop's `config.toml`. Add: ```toml [mcp_servers.osprey] command = "python" args = [ "C:\\path\\to\\osprey\\platform-mcp\\server.py" ] cwd = "C:\\path\\to\\osprey" [mcp_servers.osprey.env] PENTEST_API_BASE = "http://localhost:9000" PENTEST_QUICK_TIMEOUT = "30" PENTEST_HTTP_TIMEOUT = "900" ``` If using a virtual environment, replace the command with: ```toml command = "C:\\path\\to\\osprey\\.venv\\Scripts\\python.exe" ``` Restart ChatGPT Desktop after editing `config.toml`. For the best integrated experience, use **Hermes** or **OpenCode**. **Claude Code** if you have completed Cyber Verification Program or **ChatGPT** if approved OpenAI's Trusted Access for Cyber (TAC) access. ## Backend API endpoints - `GET /health` — Service health check - `GET /api/v1/health/` — API health check - `POST /api/v1/agent/chat` — Send a prompt to the agent - `GET /api/v1/tools` — List available tools - `GET /api/v1/models` — List configured models - `GET /api/v1/engagements` — List engagements - `POST /api/v1/engagements` — Create an engagement - `GET /api/v1/findings` — List findings ## Backend entry points - `backend/src/osprey/main.py` — FastAPI app - `backend/src/osprey/core/config.py` — Runtime settings - `backend/src/osprey/api/v1/router.py` — API route registration - `backend/tests/` — Health check tests ## Running tests ```bash cd backend python -m pytest tests/ -v ``` ## Documentation Full docs live in [`docs/`](docs/README.md). Quick links: - [Architecture & workflow](docs/ARCHITECTURE.md) — how it works and why - [Operator guide](docs/PLATFORM_GUIDE.md) — running and driving the platform - [Developer guide](docs/DEVELOPER_GUIDE.md) — code map and pipeline trace - [Capability reference](docs/CAPABILITY_REFERENCE.md) — per-tool + memory model - [Integration contract](docs/INTEGRATION_CONTRACT.md) — MCP + HTTP APIs - [Status & roadmap](docs/STATUS_AND_ROADMAP.md) — what's built vs planned ## Next steps See [`docs/STATUS_AND_ROADMAP.md`](docs/STATUS_AND_ROADMAP.md) for the current status and the active reliability plan. Near-term focus: real scope/ROE governance, durable jobs + attempt history, graph/evidence provenance hardening, and the killchain engine. ## Contributing Contributions are welcome! Please see [`CONTRIBUTING.md`](CONTRIBUTING.md) for guidelines on reporting issues, suggesting features, and submitting pull requests. ## License This project is licensed under the **GNU Affero General Public License v3.0 (AGPL-3.0)**. See the [`LICENSE`](LICENSE) file for details.