ByteCrawl

# ByteCrawl [](https://bytecrawl.vercel.app/mcp) [](https://pypi.org/project/bytecrawl/) [](https://pypi.org/project/bytecrawl/) [](LICENSE) [](https://github.com/abrahamperz/ByteCrawl/actions/workflows/ci.yml) *Léelo en [español](README.es.md).* **Give your AI agent focused web crawling.** ByteCrawl is an MCP server (and a small Python library) that doesn't just scrape a page — it crawls a whole site and returns the pages *most relevant* to your topic first, using Shark-Search and OPIC in pure Python. - **Webpage**: https://bytecrawl.vercel.app/ - **Hosted MCP endpoint**: https://bytecrawl.vercel.app/mcp - **Agent skill**: https://bytecrawl.vercel.app/agent-onboarding/SKILL.md - **Latest release**: **1.4.0** — clearer errors when a site's bot protection (Cloudflare, DataDome, …) blocks a fetch ([changelog](https://github.com/abrahamperz/ByteCrawl/blob/main/CHANGELOG.md)) ## Why focused crawling? Most crawlers visit pages in whatever order they find them. Under a request budget, order is everything. From `en.wikipedia.org/wiki/Silicon_Valley` with query `san francisco`, 20 pages each: | Strategy | Pages over 0.1 relevance | Best page | |---|---|---| | `shark` | **20** | **0.7774** | | `opic` | 4 | 0.2799 | | `bfs` | 1 | 0.1068 | Same twenty requests; only the order changed. Reproduce it with `result.relevant(0.1)`. **That is a token argument as much as a quality one.** Those pages end up in a context window. BFS spends nineteen of its twenty on whatever happened to be linked first, and you pay for all nineteen. And each page arrives as Markdown rather than raw HTML — around **6.5× fewer tokens** for the same content. `fetch_markdown` reports both counts, so you can state the saving rather than assume it. **How it works.** Every link gets a score before it is visited. Shark-Search takes how much the anchor text and the parent page look like your query, decayed with depth so a branch that stops matching fades out on its own; OPIC scores by importance instead, cash flowing along links. The frontier is a priority queue, so the crawler always spends its next request on the best link it knows about — which is why the budget lands on relevant pages instead of nearby ones. ```python from bytecrawl import SharkSearch result = SharkSearch(query="vector databases").crawl(url, max_pages=100) result.top(10) # [{url, title, relevance, depth, order}, ...] ``` - **Shark-Search** (Hersovici et al., 1998) — topical best-first; links inherit their parent's relevance with decay, so barren branches die out on their own. - **OPIC** (Abiteboul et al., 2003) — live PageRank via "cash" flowing along links, no full graph needed. `pagerank()` is included to compare against. - **BFS** — level by level. The honest baseline. All three share one loop — pop, fetch, score links, push — so an equal page budget is a fair comparison, and every surface takes the same three names. Scrapy is a framework you wire up yourself and Firecrawl is a paid SaaS. ByteCrawl is a library with a three-package core and these strategies built in. ## Quick start One command, nothing installed: ```bash claude mcp add --scope user --transport http bytecrawl https://bytecrawl.vercel.app/mcp ``` Your agent now has six tools — the same six the playground gives a human: | Tool | What it does | |---|---| | `focused_crawl` | Crawl a site, rank pages by relevance to a query (Shark-Search / OPIC / BFS) | | `compare_strategies` | Run all three on the same budget and see which one wins | | `fetch_markdown` | One page → clean Markdown, around 6.5× fewer tokens than raw HTML | | `extract` | Structured records via CSS selectors — or call it with just a URL and it tells you what the page offers | | `list_links` | Every outbound link, absolute and deduplicated | | `fetch_json_api` | Hit a hidden JSON API | Then ask in plain language — you never call the tools yourself: > Use **bytecrawl** to find everything on python.org about the packaging ecosystem Say the name. Most agents ship their own single-page fetcher and reach for it by default; **bytecrawl** is what gets you a crawl ranked by your query instead of one page read in isolation. The hosted server is static-only, rate-limited, capped at 10 pages per crawl, and refuses non-public URLs. For JS-rendered pages or bigger crawls, run it yourself: ```bash pipx install "bytecrawl[mcp]" # a CLI app, hence pipx claude mcp add --scope user bytecrawl -- bytecrawl-mcp ``` ## `/bytecrawl` — the agent skill The server gives your agent the tools; the skill gives it the judgment about using them, and you a slash command: ```bash mkdir -p ~/.claude/skills/bytecrawl && \ curl -so ~/.claude/skills/bytecrawl/SKILL.md \ https://bytecrawl.vercel.app/agent-onboarding/SKILL.md ``` The same command updates it. Installing copies the file, so later fixes never reach your copy — the skill compares its hash against [`/agent-onboarding/skill.json`](https://bytecrawl.vercel.app/agent-onboarding/skill.json) once a session and tells you when yours is behind. It never overwrites anything on its own. No skills directory? Hand the file to any agent instead — it works for that conversation, installs nothing: ``` Read and follow https://bytecrawl.vercel.app/agent-onboarding/SKILL.md ``` ## Library API ```python from bytecrawl import Scraper bot = Scraper() page = bot.fetch("https://books.toscrape.com") # auto: static, browser fallback books = page.extract("article.product_pod", {"title": "h3 a::attr(title)", "price": "p.price_color::text"}) page.markdown() # clean Markdown for LLMs · page.tokens() # token estimate ``` ```python bot.static(url) # plain HTML bot.api(url, params={...}) # hidden JSON API bot.browser(url, wait="div.results") # JS via Playwright bot.crawl(url, item="article", fields={...}, next_page="li.next a::attr(href)") # pagination bot.session().login(url, data, csrf_field="csrf_token") # authenticated ``` ## Install ```bash pip install bytecrawl # slim core (requests + beautifulsoup4 + lxml) pip install bytecrawl[llm] # + Markdown for LLMs pip install bytecrawl[browser] # + Playwright (then: playwright install chromium) pip install bytecrawl[mcp] # + local MCP server (Python 3.10+) pip install bytecrawl[all] ``` A missing extra never fails silently — each one raises an ImportError naming the exact command to run. ## HTTP API (no install) The same engine behind one hosted GET endpoint. Everything except `url` is optional; no key, no account. ```bash curl "https://bytecrawl.vercel.app/api?url=books.toscrape.com" curl "https://bytecrawl.vercel.app/api?url=en.wikipedia.org/wiki/Silicon_Valley&method=crawl&query=san+francisco&strategy=shark" ``` `method` is one of `markdown` (default), `text`, `html`, `links`, `json`, `extract`, `crawl`. Static HTML only, crawls capped at 10 pages, rate-limited, responses cached 10 minutes, private and loopback addresses refused. Full reference: https://bytecrawl.vercel.app/docs ## Learn each scraping technique A guided walkthrough (in Spanish) with a runnable example against a practice site: [static HTML](docs/01-html-estatico.md) · [dynamic JS](docs/02-js-dinamico.md) · [hidden APIs](docs/03-api-oculta.md) · [pagination](docs/04-crawling-paginacion.md) · [login](docs/05-login-sesion.md) · [graph crawling](docs/06-crawling-grafos.md) · [Markdown for LLMs](docs/markdown-llms.md) · [ethics](docs/nota-etica.md) ## Contributing ```bash pip install -e ".[llm,dev,mcp]" pytest # 120 tests, no network required pytest -m live # + live browser tests (needs the browser extra) ruff check bytecrawl tests ``` Scrape responsibly: respect `robots.txt`, terms of service and rate limits. ByteCrawl ships with a configurable delay between requests. ## License MIT — see [LICENSE](LICENSE).