OpenCrab

<p align="center"> <img src="logo.png" alt="OpenCrab Logo" width="260"/> </p> # OpenCrab **LocalCrab builds. OpenCrab SaaS distributes.** OpenCrab is the public integration repository for the LocalCrab ontology factory and the OpenCrab hosted ecosystem at [opencrab.sh](https://opencrab.sh). This repository contains the local engine: MetaOntology OS grammar, MCP tools, CrabHarness evidence collection, local stores, promotion lifecycle, and pack export contracts. It does **not** contain the private implementation of the hosted `opencrab.sh` SaaS product. Any sample app or API code in this repository is local/demo infrastructure for developer testing. It is not the production `opencrab.sh` SaaS code. ## What This Repo Is For | Layer | Role | Lives here? | | --- | --- | --- | | LocalCrab | Local ontology factory for crawling, parsing, evidence indexing, Neo4j validation, and ZIP pack export. | Yes | | CrabHarness | Mission-first control plane for crawler planning, worker delegation, evidence validation, and promotion packages. | Yes | | MetaOntology OS | Canonical grammar, schemas, ReBAC, identity/canonicalization, promotion lifecycle, and MCP server tools. | Yes | | OpenCrab SaaS | Hosted ingestion, marketplace, profiles, MCP access, community, and paid/free pack circulation. | No, linked via [opencrab.sh](https://opencrab.sh) | The intended flow: ```text source material or crawl target | v CrabHarness mission planning | v evidence collection + OCR/CLIP indexing | v MetaOntology grammar extraction | v Neo4j/Cypher validation | v OpenCrab Pack v1 ZIP | v opencrab.sh ingest + marketplace + ecosystem distribution ``` ## LocalCrab and OpenCrab SaaS LocalCrab is quality-first. It exists to produce ontology packs with strong evidence coverage, traceable parsing, OCR/CLIP context, graph validation, and promotion receipts. OpenCrab SaaS is ecosystem-first. It exists to ingest packs, make them useful to users and agents, distribute them through marketplace/community surfaces, and expose hosted MCP access. Read the full relationship model: - [LocalCrab and OpenCrab SaaS relationship](./docs/localcrab-opencrab-relationship.md) - [LocalCrab factory workflow](./docs/localcrab-factory-workflow.md) - [OpenCrab Pack v1 ZIP format](./docs/opencrab-pack-v1.md) ## Quick Start ### 1. Install LocalCrab ```bash pip install -e ".[dev]" ``` ### 2. Run LocalCrab ```bash opencrab serve ``` LocalCrab runs locally by default. It uses SQLite, JSON files, and a local Chroma persistent store under `./opencrab_data`. ### 3. Verify the grammar and query path ```bash opencrab status opencrab manifest opencrab query "system performance and error rates" ``` ### 3-1. Ingest documents (Kordoc 4.x) PDF, HWP/HWPX, DOCX, XLSX and PPTX are parsed with [Kordoc](https://www.npmjs.com/package/kordoc) 4.x; Markdown and other text files are read directly. ```bash npm install --global kordoc@latest # 4.x is required for HWPX, DOCX and XLSX opencrab ingest ./reports -r # parses documents, then indexes the text opencrab extract ./reports -r --dry-run # LLM extraction over the same parsed text ``` `KORDOC_BIN` (or `--kordoc-bin`) selects another Kordoc command, for example `npx kordoc`. Files Kordoc cannot parse are reported and skipped, never indexed empty. The MCP tool `ontology_ingest` also accepts `path` to read a local file the same way. ### 4. Add LocalCrab as an MCP server ```bash claude mcp add opencrab -- opencrab serve ``` Or add it manually: ```json { "mcpServers": { "opencrab": { "command": "opencrab", "args": ["serve"] } } } ``` ## CrabHarness [`crabharness/`](./crabharness/) is the mission-first control plane for evidence collection. It plans what to crawl, delegates heavy work to plugin workers, validates the collected bundle, and emits OpenCrab-ready promotion packages. Core responsibilities: - Decide crawl target, scope, depth, volume, rate limits, and success criteria. - Store every collected page, document, file, image, and log as evidence. - Preserve hashes, source URLs or paths, crawl timestamps, parser status, and missing-context candidates. - Promote only after completeness, semantic relevance, and autoresearch gates pass. See the [CrabHarness README](./crabharness/README.md). ## MetaOntology OS LocalCrab keeps the existing MetaOntology OS grammar and MCP surface as the canonical ontology contract. ### 9 Spaces | Space | Role | | --- | --- | | subject | Actors with identity, agency, roles, and permissions. | | resource | Documents, datasets, tools, APIs, files, and projects. | | evidence | Raw observations, logs, text units, parser/OCR outputs, and empirical records. | | concept | Entities, concepts, topics, classes, and domain abstractions. | | claim | Derived assertions grounded by evidence. | | community | Clusters and summaries of related concepts or actors. | | outcome | KPIs, risks, impacts, and measurable results. | | lever | Tunable controls that affect outcomes or concepts. | | policy | Access, sensitivity, approval, and governance rules. | ### Core MCP Tools - `ontology_manifest`: return the full grammar. - `ontology_add_node`: add or update a grammar-validated node. - `ontology_add_edge`: add a grammar-validated edge. - `ontology_query`: hybrid vector + BM25 + graph query. - `ontology_impact`: I1-I7 impact analysis. - `ontology_rebac_check`: relationship-based access check. - `ontology_ingest`: ingest text into the local ontology stores. - `harness_promotion_apply`: apply a CrabHarness promotion package. ## OpenCrab Pack v1 LocalCrab exports ontology deliveries as an OpenCrab Pack v1 ZIP. The pack is designed to be recognized by OpenCrab SaaS while remaining reproducible in a local Neo4j environment. Required high-level layout: ```text manifest.json graph/nodes.jsonl graph/edges.jsonl evidence/index.jsonl quality/report.json neo4j/import.cypher neo4j/opencrab_ingest.jsonl neo4j/export_status.json README.md sample_queries.json community_reports.json ``` The packaging pipeline is: ```text validate -> Neo4j import/check -> Neo4j graph export -> normalized SaaS export -> ZIP package ``` See [OpenCrab Pack v1 ZIP format](./docs/opencrab-pack-v1.md). ## Development ```bash make dev-install make seed make test make status ``` Run integration tests: ```bash OPENCRAB_INTEGRATION=1 pytest tests/ -v ``` ## Project Structure ```text opencrab/ grammar/ MetaOntology grammar, validator, glossary schemas/ YAML type schemas, schema packs, action schemas ontology/ builder, query, identity, canonicalization, promotion, ReBAC execution/ workflow and approval runtime billing/ local usage hooks stores/ Neo4j, Chroma, Mongo, SQL, and local adapters mcp/ MCP server and tool registry crabharness/ crabharness/ mission planner, runtime, validation, promotion package builder codex_workers/ plugin workers for crawlers and collectors missions/ example missions docs/ public integration and pack delivery contracts ``` ## Korean Summary 이 리포지토리는 LocalCrab과 OpenCrab SaaS를 하나의 제품처럼 설명하는 공개 통합 리포지토리입니다. LocalCrab은 온톨로지 공장입니다. 크롤링, 파싱, OCR, CLIP 이미지 컨텍스트, evidence 풀 인덱싱, Neo4j 검증, ZIP 팩 생성을 담당합니다. OpenCrab SaaS는 [opencrab.sh](https://opencrab.sh)의 생태계 허브입니다. 완성된 팩을 인제스트하고, 마켓플레이스와 커뮤니티에서 배포하며, hosted MCP 접근을 제공합니다. 단, `opencrab.sh`의 내부 SaaS 코드는 이 공개 리포지토리에 포함하지 않습니다. ## License MIT.