mirobody-rare

<div align="center"> # Mirobody Rare **A rare-disease work-up as structured, cited data an AI agent can reason over.** Narratives, trio VCFs, pedigrees, imaging and EEG in; HPO-coded phenotypes, annotated variants with inheritance, signal indexes and ranked Orphanet candidates out — each answer pointing back to the sentence, the variant call or the file it came from. English | [简体中文](README.zh-CN.md) </div> Mirobody Rare is [Mirobody](https://github.com/thetahealth/mirobody) 1.5.2 with the **`mirobody-rare`** plugin (`plugins/mirobody-rare/`). Mirobody supplies the server, the chat agent, the bundled web client and the storage; the plugin adds everything rare-disease. The main package changes only where a plugin needs a door: entry-point groups (`mirobody.file_handlers / text_hooks / delete_hooks / routers / schemas`), an upload admission hook, spooled WebSocket uploads, a write-permission check on chat attachments, and one CORS fix. Everything Mirobody does on its own is described in [its README](https://github.com/thetahealth/mirobody#readme) and in `docs/`. > Synthetic data, research use only. Not a medical device and not medical advice. ## What it reads Files arrive as attachments on the Ask page (or through the upload API). What each one becomes: | Input | Output | Stored in | | --- | --- | --- | | Clinical narrative, Chinese or English (Markdown, text, or the text layer of a PDF / Word file) | One assertion per finding: HPO term, polarity (present / absent / uncertain), experiencer (the patient, or which relative), the source sentence and its span. Terms the dictionary cannot settle go to a review queue rather than being guessed | `th_phenotype`, `th_phenotype_review` | | Trio VCF (GRCh38) + PED | Pathogenic / likely-pathogenic calls annotated from ClinVar, population frequency from gnomAD v4.1 (read locally), genotype, and the inheritance call across the pedigree. GRCh37 or build-less VCFs are refused with a reason | `th_sequencing_sample`, `th_variant`, `th_variant_annotation`, `th_pedigree*` | | DICOM series (zip), EDF / EDF+ / BDF | A de-identified index: modality and series, or channels, sample rate, duration and seizure / sleep-stage annotations. Identifying header fields never reach the index; an EDF+ recording whose patient field names a person or a birth date is hidden | `th_signal_object` | | All of the above | Orphanet candidates ranked from the coded phenotypes, refreshed when variant evidence lands; the gene for a disorder with several causal genes is left open, with the candidates listed | `th_disease_code` | FASTQ, BAM and CRAM are refused before any byte is read, as are files above the per-kind limits (`upload.limits` in the plugin config: VCF 2 GB, DICOM zip 5 GB, EDF 2 GB, PED 10 MB). The chat agent reads all of it through MCP tools: `resolve_hpo`, `code_phenotypes`, `rank_rare_diseases`, `query_phenotype`, `query_variant`, `query_signal_index`, `query_pedigree`, `query_family_history`, `record_consent` and `resolve_phenotype_review`. A relative's data is read only through Mirobody's care circle: the relative shares with the caller, or the tool refuses; a pedigree member marked analysis-only contributes to the computation but never gets a result returned (`consent/gate.py`). Deleting an uploaded file deletes everything derived from it (`erase.py`). ## How the narrative is read 1. **Sections and sentences.** The text hook splits the document by headings (history, examination, investigations, family history, diagnosis) and filters teaching-question blocks and page debris. 2. **Rules first.** A ConText / NegEx engine (`assertion/context.py`) decides polarity and experiencer for every HPO term found in the sentence. Its cue words are data, not code: medspaCy's published English rules vendored verbatim, plus `res/cues/en.yaml` and `zh.yaml`, where every added entry names the held-out sentence that motivated it and its known cost. 3. **Dictionary.** `hpo/adapter.py` matches HPO labels and synonyms (Chinese labels from the official babelon translation), exactly first and then by Snowball stem with at most one intervening word; a loose match only replaces an exact one when it is an HPO descendant of it. 4. **LLM fallback.** Sentences the rules leave uncoded go to an LLM (OpenRouter, default `google/gemini-3.8-flash`, one cached call per document). The model names terms; the same dictionary codes them, so the output is still an HPO ID from the ontology. Rows it adds carry `source = 'nlp+llm'`. If the call fails the rule result stands. ## Quick start ```bash git clone https://github.com/thetahealth/mirobody-rare.git && cd mirobody-rare uv venv && uv pip install -e '.[app]' -e plugins/mirobody-rare # reference data, not shipped: download from the sources under their own terms export MIROBODY_RARE_ONTOLOGY_DIR=/path/to/ontology # hp.obo, hp-zh.babelon.tsv, phenotype.hpoa, genes_to_phenotype.txt, # hgnc_complete_set.txt, Orphanet en_product1.xml / en_product4.xml / en_product6.xml export MIROBODY_RARE_CLINVAR_VCF=/path/to/clinvar.vcf.gz # GRCh38 export MIROBODY_RARE_GNOMAD_DIR=/path/to/gnomad_v4.1_exomes # tabix-indexed, per chromosome # optional: keep the reference tables in Postgres instead of memory export MIROBODY_RARE_REFERENCE_BACKEND=pg mirobody-rare-load-reference all # Postgres with pgvector, e.g. docker run -d -p 5432:5432 -e POSTGRES_USER=user -e POSTGRES_PASSWORD=pw \ # -e POSTGRES_DB=mirobody pgvector/pgvector:pg17 mirobody dev --pg-url postgres://user:pw@localhost:5432/mirobody --port 28085 # API + bundled web client ``` The plugin's tables are created by the server's schema bootstrap on first start. Open `http://localhost:28085`, sign in, and attach files to a question on the Ask page. After syncing a new build of the web client into `frontend/`, rerun `tools/patch_frontend_rare.py`: the stock client refuses VCF, PED and zip files in the browser. The LLM fallback needs `OPENROUTER_API_KEY`; without it the rules run alone. Every setting lives in `plugins/mirobody-rare/mirobody_rare/config.yaml`, and each path has an environment override. For benchmarking there is an OpenAI-compatible shim, `mirobody-rare-serve --port 8765`, which answers the rare-disease tasks of the HAEnv benchmark (`haenv_rare`) with the same pipeline. ## How well it works Everything below is measured on synthetic cases. There is no real-patient gold standard yet, so none of these numbers says how the system does on real clinical records. | Set | What it tests | Reading | | --- | --- | --- | | Templated benchmark cases (120, zh + en) | every layer, on narratives generated from the gold | variant hit, genotype and inheritance 1.000 · DICOM and EDF index 1.000 · EDF identity hidden 1.000 · narrative recall 0.991–1.000, polarity 1.000 · Orphanet top-1 0.95–1.00 | | Held-out rewrite, test half (30 cases, 219 gold sentences) | the same gold, narratives freely rewritten by another model family, scored once | recall: rules 0.868, with the LLM fallback 0.941 · polarity 0.97 · experiencer 1.000 | | Harder held-out rewrite (40 cases) | descriptive paraphrase of HPO names | recall with the fallback 0.64–0.70: the remaining misses are descriptions the dictionary cannot reach | | Round trip (40 cases, 1,969 items; 5 cases through the Ask page, 286 items) | what is stored is what is read back, across 11 layers | 1 true difference (a coding error, stored faithfully); 0 on the Ask page | Method, per-batch readings and known gaps: `docs/rare-mvp-impl.md` and `docs/rare-mvp-testing.md` (Chinese). Per-sentence reports: `reports/assertion_arms/` and `reports/roundtrip/`. The LLM models compared for the fallback are in `reports/assertion_arms/models-20260929/`. ## Repository layout | Path | What | | --- | --- | | `plugins/mirobody-rare/mirobody_rare/` | the plugin: `handlers.py` (uploads), `text_hook.py` → `assertion/` → `coding.py` (narratives), `variant/`, `pedigree/`, `signal/`, `hpo/`, `disease/`, `gene/`, `tools.py` (MCP), `web.py` (`/api/rare/*`), `consent/`, `erase.py`, `schema/` (DDL) | | `plugins/mirobody-rare/tests/` | unit tests, plus Postgres, web-client and end-to-end suites that run when their environment is set | | `tools/` | evaluation and operations: `assertion_arms.py`, `roundtrip_check.py`, `roundtrip_bundled.py`, `delete_cascade_check.py`, `patch_frontend_rare.py` | | `.claude/skills/case-preprocess/`, `scripts/ccprep` | case preprocessing stage (IO spec v0.2): inventories a directory of vendor files, indexes them, joins samples and measures QC, writing `inventory.jsonl`, `identity/`, `qc/`, `PROVENANCE.jsonl` and `RUN.json`; `ccprep` runs it through Claude Code, `engine/preprocess.py` runs the engine alone, `checks/selftest.py` checks it | | `docs/rare-*.md`, `docs/ingest-plan.md` | plan, implementation record, test checklist | | everything else | Mirobody 1.5.2 | ## Tests ```bash .venv/bin/python3 -m pytest plugins/mirobody-rare/tests -q # no database, no network # with a database (MIROBODY_RARE_PG_DSN, MIROBODY_RARE_PG_SCHEMA) the Postgres suites run too; # with MIROBODY_RARE_E2E_BASE=http://127.0.0.1:28085 set, run the web-client suite (-m web) and the end-to-end suite (-m e2e) # each in its own process; a plain pytest skips both ``` ## License Apache-2.0, as Mirobody (`LICENSE`). Vendored resources keep their own licenses: medspaCy's ConText rules (MIT, `res/cues/LICENSE.medspacy`) and Snowball's English stop list (BSD-3-Clause, see `res/cues/README.md`). HPO, Orphanet, ClinVar, gnomAD and HGNC are not distributed here and are used under their own terms.