Content
# prompt-engineering-best-practices
> **The prompt-engineering reference your agent can call directly — held to an eval bar most reference repos never set.**
## What this isn't
Not another multi-provider LLM wrapper. That layer is solved:
- [LiteLLM](https://github.com/BerriAI/litellm) (~46k★) — call any vendor with one API
- [AISuite](https://github.com/andrewyng/aisuite) (~13k★) — same thesis, lighter
You don't need a sixth wrapper. The wrappers ship the *transport*. They don't carry the *prompts*.
## What it is
The layer **above** the wrappers — the prompts, patterns, and anti-patterns the wrappers carry, with three properties no existing reference combines:
1. **Canonical** — one schema, one source-of-truth file (`patterns.json`), one auto-rendered handbook. CI fails if the handbook drifts from the data.
2. **Agent-pingable** — exposed over MCP so any LLM client can call `get_pattern("role-and-rules")` mid-conversation. Stdio transport in v0.1, HTTP in v0.2.
3. **Eval-disciplined** — every promoted Showcase entry clears a 3-vendor judge ensemble with a Kendall-τ agreement gate. Most "best prompts" lists are someone's vibes. This one isn't.
## What's missing in existing repos
These three capabilities exist separately. Their intersection was empty across the top 20+ candidates audited on 2026-05-11:
| Project | Canonical reference? | MCP-callable? | Eval-graded entries? |
|---|:---:|:---:|:---:|
| [`dair-ai/Prompt-Engineering-Guide`](https://github.com/dair-ai/Prompt-Engineering-Guide) (74k★) | ✅ | ❌ | ❌ |
| [`microsoft/generative-ai-for-beginners`](https://github.com/microsoft/generative-ai-for-beginners) (110k★) | ✅ | ❌ | ❌ |
| [`f/prompts.chat`](https://github.com/f/awesome-chatgpt-prompts) (162k★) | curated | ❌ | ❌ |
| [`anthropics/anthropic-cookbook`](https://github.com/anthropics/anthropic-cookbook) | ✅ recipes | ❌ | ❌ |
| [`gdli6177/mcp-prompt-server`](https://github.com/gdli6177/mcp-prompt-server) | ❌ | ✅ | ❌ |
| [`nashsu/llm_wiki`](https://github.com/nashsu/llm_wiki) | ❌ | partial | ✅ |
| **`prompt-engineering-best-practices`** | **✅** | **✅** | **✅** |
Existing repos cover the transport layer. This is the reference layer above it.
## How the eval bar works
Showcase entries are capped at six and judged before they ship.
Three judges, **three different model families** — Claude Opus, GPT, Gemini Pro — score each candidate 1–10 on five dimensions: universality, pivotal-ness, reproducibility, outcome legibility, safety. An entry is rejected unless:
| Gate | Threshold |
|---|---|
| Mean across all dimensions | ≥ 8.0 |
| Lowest single dimension | ≥ 6 |
| **Kendall-τ across the 3 judges** | **≥ 0.5** |
| τ between any two judges | ≤ 0.95 (flag for vendor-correlated bias review — manual review queue is v0.2)\* |
The Kendall-τ gate is the most novel piece — it forces agreement on the *relative* ranking of a prompt's strengths and weaknesses, not just on absolute scores. Same-vendor judges produce false agreement on numbers; cross-vendor + rank-correlation catches a specific failure mode where judges agree on scores but disagree on shape. It does meaningful work as a sanity check, but on a 5-dimension ranking it's a coarse instrument; the mean-≥-8.0 gate and the per-dimension floor (≥ 6) carry most of the rejection weight. See [`docs/eval-rubric.md`](docs/eval-rubric.md) for limitations.
\* The vendor-bias flag is wired into [`scoring.py`](scoring.py) but the manual-review queue it would surface to is v0.2 (see [SPEC](docs/SPEC.md)).
Implementation: [`judges.py`](judges.py) (async cross-vendor ensemble) + [`scoring.py`](scoring.py) (Kendall-τ minimum across all judge pairs) + [`tests/eval_judge_prompt.py`](tests/eval_judge_prompt.py) (5-prompt calibration set that gates any change to the judge prompt template). Full rubric in [`docs/eval-rubric.md`](docs/eval-rubric.md).
**v0.1 ships with the Showcase slot empty.** The bar held first.
## What's inside (v0.1)
- **10 principles** — foundational rules with failure-mode rationale (e.g. *make the objective explicit*, *iterate with evals not vibes*)
- **10 patterns** — reusable scaffolds with `when_to_use` + edge cases (Role & Rules, Chain-of-Thought, Extractor + Validator Pair, Judge Prompt, …)
- **5 anti-patterns** — production failure modes (kitchen-sink instructions, vague success criteria, untested prompt changes, …)
- **Showcase slot** — capped at 6, currently 0/6 (eval bar held first)
- **Update agent** — fetches a curated feed → classifies → dedups → scores via the 3-vendor ensemble → emits proposals for human review (manual `peb-review --accept` in v0.1; auto-PR in v0.2)
- **MCP stdio server** — five tools (`list_patterns`, `get_pattern`, `search`, `showcase`, `get_schema`) + `peb://` resources
Browse the rendered handbook at [`docs/handbook.md`](docs/handbook.md). Interactive view at [`docs/index.html`](docs/index.html).
## 30-second use (MCP)
Add to `claude_desktop_config.json` (or any MCP-compatible client config):
```json
{
"mcpServers": {
"peb": {
"command": "peb-server",
"args": []
}
}
}
```
### HTTP transport (v0.2)
For HTTP-mode MCP clients, run the server with `--transport http`:
```bash
export PEB_MCP_TOKEN="$(openssl rand -hex 32)"
peb-server --transport http --bind 127.0.0.1 --port 8765
```
Loopback binds (`127.0.0.1`, `::1`) may run without a token for local dev; any non-loopback bind (`0.0.0.0`, an external IP) refuses to start without `PEB_MCP_TOKEN` set. Clients send `Authorization: Bearer <token>` per request. Token rotation: set `PEB_MCP_TOKEN_PREVIOUS=<old>` alongside the new `PEB_MCP_TOKEN`, then unset `_PREVIOUS` after clients have rotated. See [`docs/SPEC-v0.2-to-v1.0-orchestration.md`](docs/SPEC-v0.2-to-v1.0-orchestration.md) §4.1.
Verified client recipes: [`docs/integrations/claude-desktop.md`](docs/integrations/claude-desktop.md), [`docs/integrations/cursor.md`](docs/integrations/cursor.md), [`docs/integrations/mcp-python-sdk.md`](docs/integrations/mcp-python-sdk.md).
Restart your client. Then any tool call:
| Tool | What it returns |
|---|---|
| `list_patterns()` | summary of all entries (or one category) |
| `get_pattern("<id>")` | full entry by id |
| `search("<query>")` | free-text across title / summary / scaffold |
| `showcase()` | top entries from the Showcase slot, ranked by mean eval score |
| `get_schema()` | the JSON Schema `patterns.json` validates against |
## Install
```bash
git clone https://github.com/Mike-E-Log/prompt-engineering-best-practices.git
cd prompt-engineering-best-practices
pip install -e .[dev]
pytest # full suite, no API keys required
peb-server # MCP stdio server
peb-update --dry-run # discovery agent without spending budget
peb-update --list-feeds # show the 5 curated feeds
peb-update --feed anthropic-blog --dry-run # run one feed only
peb-review --list # pending proposals
```
If `pip install` warns that scripts went to a directory not on PATH (common on Windows), invoke the modules directly instead:
```bash
python -m server # equivalent to peb-server
python -m update_agent --dry-run
python -m peb_review --list
```
For the integration calibration test (live judge ensemble), set `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, and `GOOGLE_API_KEY` and run `pytest tests/eval_judge_prompt.py`.
## Auto-PR setup (v0.2)
Accepted proposals can be opened as draft PRs against `main` automatically. **This requires explicit operator setup; the agent never merges its own PRs.**
**Branch protection prerequisite:** Before enabling auto-PR — and especially before flipping the repo public — enable required-review on `main` in repo Settings → Branches. The agent opens drafts only; the maintainer is the sole entity that can mark ready-for-review and merge. Without branch protection, a misconfigured CI or a stolen `PEB_AUTO_PR_TOKEN` could result in unreviewed content landing on `main`.
**Credential scoping:** `PEB_AUTO_PR_TOKEN` MUST be a fine-grained GitHub PAT (or GitHub App installation token) scoped to **this single repository** with **only** `pull_requests:write` and `contents:write`. It MUST be distinct from `PEB_MCP_TOKEN`; `auto_pr.py` refuses to start at runtime if both env vars are non-empty and equal. See [`SECURITY.md`](SECURITY.md) §2 and [`docs/SPEC-v0.2-to-v1.0-orchestration.md`](docs/SPEC-v0.2-to-v1.0-orchestration.md) §4.4 for the full threat model.
Run:
```bash
export PEB_AUTO_PR_TOKEN="<fine-grained PAT scoped per above>"
peb-review pr open <proposal-id> # opens a draft PR; maintainer reviews + merges
peb-review pr status # list open agent-opened draft PRs
```
## Docs
- [`docs/SPEC.md`](docs/SPEC.md) — design rationale, data model, D1–D16 review decisions
- [`docs/handbook.md`](docs/handbook.md) — rendered from `patterns.json` (CI fails on drift)
- [`docs/eval-rubric.md`](docs/eval-rubric.md) — scoring dimensions + thresholds
- [`docs/index.html`](docs/index.html) — interactive showcase view
- [`CONTRIBUTING.md`](CONTRIBUTING.md) — how to add an entry
- [`CHANGELOG.md`](CHANGELOG.md) — release notes
## Roadmap
- **v0.1** (shipped 2026-05-11) — Core spec implemented; stdio MCP; manual proposal review; one curated feed
- **v0.2** (this release) — HTTP MCP transport; 5-feed update agent with per-feed `trust_tier`; manual-review queue; auto-PR creation from accepted proposals
- **v0.3** — 5-agent discovery pipeline (Intake → Insight → Editor → Reviewer → Supervisor)
- **v0.4** — Multi-source feeds (arXiv, GitHub releases, HN); semantic dedup; per-run dollar caps
- **v1.0** — API freeze; PyPI publication; full docs site
## License
MIT. See [`LICENSE`](LICENSE).
