Content
# Universal Tool Calling Protocol (UTCP)
[](https://github.com/universal-tool-calling-protocol)
[](https://pepy.tech/projects/utcp)
[](https://github.com/universal-tool-calling-protocol/python-utcp/blob/main/LICENSE)
[](https://cdtm.com/)
## Introduction
The Universal Tool Calling Protocol (UTCP) is a secure, scalable standard for defining and interacting with tools across a wide variety of communication protocols. UTCP 1.0.0 introduces a modular core with a plugin-based architecture, making it more extensible, testable, and easier to package.
In contrast to other protocols, UTCP places a strong emphasis on:
* **Scalability**: UTCP is designed to handle a large number of tools and providers without compromising performance.
* **Extensibility**: A pluggable architecture allows developers to easily add new communication protocols, tool storage mechanisms, and search strategies without modifying the core library.
* **Interoperability**: With a growing ecosystem of protocol plugins (including HTTP, SSE, CLI, and more), UTCP can integrate with almost any existing service or infrastructure.
* **Ease of Use**: The protocol is built on simple, well-defined Pydantic models, making it easy for developers to implement and use.
## Repository Structure
This repository contains the complete UTCP Python implementation:
- **[`core/`](core/)** - Core `utcp` package with foundational components ([README](core/README.md))
- **[`plugins/communication_protocols/`](plugins/communication_protocols/)** - Protocol-specific plugins:
- [`http/`](plugins/communication_protocols/http/) - HTTP/REST, SSE, streaming, OpenAPI ([README](plugins/communication_protocols/http/README.md))
- [`cli/`](plugins/communication_protocols/cli/) - Command-line tools ([README](plugins/communication_protocols/cli/README.md))
- [`mcp/`](plugins/communication_protocols/mcp/) - Model Context Protocol ([README](plugins/communication_protocols/mcp/README.md))
- [`text/`](plugins/communication_protocols/text/) - File-based tools ([README](plugins/communication_protocols/text/README.md))
- [`socket/`](plugins/communication_protocols/socket/) - TCP/UDP (🚧 In Progress)
- [`gql/`](plugins/communication_protocols/gql/) - GraphQL (🚧 In Progress)
## Architecture Overview
UTCP uses a modular architecture with a core library and protocol plugins:
### Core Package (`utcp`)
The [`core/`](core/) directory contains the foundational components:
- **Data Models**: Pydantic models for `Tool`, `CallTemplate`, `UtcpManual`, and `Auth`
- **Client Interface**: Main `UtcpClient` for tool interaction
- **Plugin System**: Extensible interfaces for protocols, repositories, and search
- **Default Implementations**: Built-in tool storage and search strategies
## Quick Start
### Installation
Install the core library and any required protocol plugins:
```bash
# Install core + HTTP plugin (most common)
pip install utcp utcp-http
# Install additional plugins as needed
pip install utcp-cli utcp-mcp utcp-text
```
### Basic Usage
```python
from utcp.utcp_client import UtcpClient
# Create client with HTTP API
client = await UtcpClient.create(config={
"manual_call_templates": [{
"name": "my_api",
"call_template_type": "http",
"url": "https://api.example.com/utcp"
}]
})
# Call a tool
result = await client.call_tool("my_api.get_data", {"id": "123"})
```
## Protocol Plugins
UTCP supports multiple communication protocols through dedicated plugins:
| Plugin | Description | Status | Documentation |
|--------|-------------|--------|---------------|
| [`utcp-http`](plugins/communication_protocols/http/) | HTTP/REST APIs, SSE, streaming | ✅ Stable | [HTTP Plugin README](plugins/communication_protocols/http/README.md) |
| [`utcp-cli`](plugins/communication_protocols/cli/) | Command-line tools | ✅ Stable | [CLI Plugin README](plugins/communication_protocols/cli/README.md) |
| [`utcp-mcp`](plugins/communication_protocols/mcp/) | Model Context Protocol | ✅ Stable | [MCP Plugin README](plugins/communication_protocols/mcp/README.md) |
| [`utcp-text`](plugins/communication_protocols/text/) | Local file-based tools | ✅ Stable | [Text Plugin README](plugins/communication_protocols/text/README.md) |
| [`utcp-websocket`](plugins/communication_protocols/websocket/) | WebSocket real-time bidirectional communication | ✅ Stable | [WebSocket Plugin README](plugins/communication_protocols/websocket/README.md) |
| [`utcp-socket`](plugins/communication_protocols/socket/) | TCP/UDP protocols | 🚧 In Progress | [Socket Plugin README](plugins/communication_protocols/socket/README.md) |
| [`utcp-gql`](plugins/communication_protocols/gql/) | GraphQL APIs | 🚧 In Progress | [GraphQL Plugin README](plugins/communication_protocols/gql/README.md) |
For development, you can install the packages in editable mode from the cloned repository:
```bash
# Clone the repository
git clone https://github.com/universal-tool-calling-protocol/python-utcp.git
cd python-utcp
# Install the core package in editable mode with dev dependencies
pip install -e "core[dev]"
# Install a specific protocol plugin in editable mode
pip install -e plugins/communication_protocols/http
```
## Migration Guide from 0.x to 1.0.0
Version 1.0.0 introduces several breaking changes. Follow these steps to migrate your project.
1. **Update Dependencies**: Install the new `utcp` core package and the specific protocol plugins you use (e.g., `utcp-http`, `utcp-cli`).
2. **Configuration**:
* **Configuration Object**: `UtcpClient` is initialized with a `UtcpClientConfig` object, dict or a path to a JSON file containing the configuration.
* **Manual Call Templates**: The `providers_file_path` option is removed. Instead of a file path, you now provide a list of `manual_call_templates` directly within the `UtcpClientConfig`.
* **Terminology**: The term `provider` has been replaced with `call_template`, and `provider_type` is now `call_template_type`.
* **Streamable HTTP**: The `call_template_type` `http_stream` has been renamed to `streamable_http`.
3. **Update Imports**: Change your imports to reflect the new modular structure. For example, `from utcp.client.transport_interfaces.http_transport import HttpProvider` becomes `from utcp_http.http_call_template import HttpCallTemplate`.
4. **Tool Search**: If you were using the default search, the new strategy is `TagAndDescriptionWordMatchStrategy`. This is the new default and requires no changes unless you were implementing a custom strategy.
5. **Tool Naming**: Tool names are now namespaced as `manual_name.tool_name`. The client handles this automatically.
6. **Variable Substitution Namespacing**: Variables that are substituted in different `call_templates`, are first namespaced with the name of the manual with the `_` duplicated. So a key in a tool call template called `API_KEY` from the manual `manual_1` would be converted to `manual__1_API_KEY`.
## Usage Examples
### 1. Using the UTCP Client
**`config.json`** (Optional)
You can define a comprehensive client configuration in a JSON file. All of these fields are optional.
```json
{
"variables": {
"openlibrary_URL": "https://openlibrary.org/static/openapi.json"
},
"load_variables_from": [
{
"variable_loader_type": "dotenv",
"env_file_path": ".env"
}
],
"tool_repository": {
"tool_repository_type": "in_memory"
},
"tool_search_strategy": {
"tool_search_strategy_type": "tag_and_description_word_match"
},
"manual_call_templates": [
{
"name": "openlibrary",
"call_template_type": "http",
"http_method": "GET",
"url": "${URL}",
"content_type": "application/json"
},
],
"post_processing": [
{
"tool_post_processor_type": "filter_dict",
"only_include_keys": ["name", "key"],
"only_include_tools": ["openlibrary.read_search_authors_json_search_authors_json_get"]
}
]
}
```
**`client.py`**
```python
import asyncio
from utcp.utcp_client import UtcpClient
from utcp.data.utcp_client_config import UtcpClientConfig
async def main():
# The UtcpClient can be created with a config file path, a dict, or a UtcpClientConfig object.
# Option 1: Initialize from a config file path
# client_from_file = await UtcpClient.create(config="./config.json")
# Option 2: Initialize from a dictionary
client_from_dict = await UtcpClient.create(config={
"variables": {
"openlibrary_URL": "https://openlibrary.org/static/openapi.json"
},
"load_variables_from": [
{
"variable_loader_type": "dotenv",
"env_file_path": ".env"
}
],
"tool_repository": {
"tool_repository_type": "in_memory"
},
"tool_search_strategy": {
"tool_search_strategy_type": "tag_and_description_word_match"
},
"manual_call_templates": [
{
"name": "openlibrary",
"call_template_type": "http",
"http_method": "GET",
"url": "${URL}",
"content_type": "application/json"
}
],
"post_processing": [
{
"tool_post_processor_type": "filter_dict",
"only_include_keys": ["name", "key"],
"only_include_tools": ["openlibrary.read_search_authors_json_search_authors_json_get"]
}
]
})
# Option 3: Initialize with a full-featured UtcpClientConfig object
from utcp_http.http_call_template import HttpCallTemplate
from utcp.data.variable_loader import VariableLoaderSerializer
from utcp.interfaces.tool_post_processor import ToolPostProcessorConfigSerializer
config_obj = UtcpClientConfig(
variables={"openlibrary_URL": "https://openlibrary.org/static/openapi.json"},
load_variables_from=[
VariableLoaderSerializer().validate_dict({
"variable_loader_type": "dotenv", "env_file_path": ".env"
})
],
manual_call_templates=[
HttpCallTemplate(
name="openlibrary",
call_template_type="http",
http_method="GET",
url="${URL}",
content_type="application/json"
)
],
post_processing=[
ToolPostProcessorConfigSerializer().validate_dict({
"tool_post_processor_type": "filter_dict",
"only_include_keys": ["name", "key"],
"only_include_tools": ["openlibrary.read_search_authors_json_search_authors_json_get"]
})
]
)
client = await UtcpClient.create(config=config_obj)
# Call a tool. The name is namespaced: `manual_name.tool_name`
result = await client.call_tool(
tool_name="openlibrary.read_search_authors_json_search_authors_json_get",
tool_args={"q": "J. K. Rowling"}
)
print(result)
if __name__ == "__main__":
asyncio.run(main())
```
### 2. Providing a UTCP Manual
A `UTCPManual` describes the tools you offer. The key change is replacing `tool_provider` with `tool_call_template`.
**`server.py`**
UTCP decorator version:
```python
from fastapi import FastAPI
from utcp_http.http_call_template import HttpCallTemplate
from utcp.data.utcp_manual import UtcpManual
from utcp.python_specific_tooling.tool_decorator import utcp_tool
app = FastAPI()
# The discovery endpoint returns the tool manual
@app.get("/utcp")
def utcp_discovery():
return UtcpManual.create_from_decorators(manual_version="1.0.0")
# The actual tool endpoint
@utcp_tool(tool_call_template=HttpCallTemplate(
name="get_weather",
url=f"https://example.com/api/weather",
http_method="GET"
), tags=["weather"])
@app.get("/api/weather")
def get_weather(location: str):
return {"temperature": 22.5, "conditions": "Sunny"}
```
No UTCP dependencies server version:
```python
from fastapi import FastAPI
app = FastAPI()
# The discovery endpoint returns the tool manual
@app.get("/utcp")
def utcp_discovery():
return {
"manual_version": "1.0.0",
"utcp_version": "1.0.2",
"tools": [
{
"name": "get_weather",
"description": "Get current weather for a location",
"tags": ["weather"],
"inputs": {
"type": "object",
"properties": {
"location": {"type": "string"}
}
},
"outputs": {
"type": "object",
"properties": {
"temperature": {"type": "number"},
"conditions": {"type": "string"}
}
},
"tool_call_template": {
"call_template_type": "http",
"url": "https://example.com/api/weather",
"http_method": "GET"
}
}
]
}
# The actual tool endpoint
@app.get("/api/weather")
def get_weather(location: str):
return {"temperature": 22.5, "conditions": "Sunny"}
```
### 3. Full examples
You can find full examples in the [examples repository](https://github.com/universal-tool-calling-protocol/utcp-examples).
## Protocol Specification
### `UtcpManual` and `Tool` Models
The `tool_provider` object inside a `Tool` has been replaced by `tool_call_template`.
```json
{
"manual_version": "string",
"utcp_version": "string",
"tools": [
{
"name": "string",
"description": "string",
"inputs": { ... },
"outputs": { ... },
"tags": ["string"],
"tool_call_template": {
"call_template_type": "http",
"url": "https://...",
"http_method": "GET"
}
}
]
}
```
## Call Template Configuration Examples
Configuration examples for each protocol. Remember to replace `provider_type` with `call_template_type`.
### HTTP Call Template
```json
{
"name": "my_rest_api",
"call_template_type": "http", // Required
"url": "https://api.example.com/users/{user_id}", // Required
"http_method": "POST", // Required, default: "GET"
"content_type": "application/json", // Optional, default: "application/json"
"allowed_communication_protocols": ["http"], // Optional, defaults to [call_template_type]. Restricts which protocols tools can use.
"auth": { // Optional, authentication for the HTTP request (example using ApiKeyAuth for Bearer token)
"auth_type": "api_key",
"api_key": "Bearer $API_KEY", // Required
"var_name": "Authorization", // Optional, default: "X-Api-Key"
"location": "header" // Optional, default: "header"
},
"auth_tools": { // Optional, authentication for converted tools, if this call template points to an openapi spec that should be automatically converted to a utcp manual (applied only to endpoints requiring auth per OpenAPI spec)
"auth_type": "api_key",
"api_key": "Bearer $TOOL_API_KEY", // Required
"var_name": "Authorization", // Optional, default: "X-Api-Key"
"location": "header" // Optional, default: "header"
},
"headers": { // Optional
"X-Custom-Header": "value"
},
"body_field": "body", // Optional, default: "body"
"header_fields": ["user_id"] // Optional
}
```
### SSE (Server-Sent Events) Call Template
```json
{
"name": "my_sse_stream",
"call_template_type": "sse", // Required
"url": "https://api.example.com/events", // Required
"event_type": "message", // Optional
"reconnect": true, // Optional, default: true
"retry_timeout": 30000, // Optional, default: 30000 (ms)
"auth": { // Optional, example using BasicAuth
"auth_type": "basic",
"username": "${USERNAME}", // Required
"password": "${PASSWORD}" // Required
},
"headers": { // Optional
"X-Client-ID": "12345"
},
"body_field": null, // Optional
"header_fields": [] // Optional
}
```
### Streamable HTTP Call Template
Note the name change from `http_stream` to `streamable_http`.
```json
{
"name": "streaming_data_source",
"call_template_type": "streamable_http", // Required
"url": "https://api.example.com/stream", // Required
"http_method": "POST", // Optional, default: "GET"
"content_type": "application/octet-stream", // Optional, default: "application/octet-stream"
"chunk_size": 4096, // Optional, default: 4096
"timeout": 60000, // Optional, default: 60000 (ms)
"auth": null, // Optional
"headers": {}, // Optional
"body_field": "data", // Optional
"header_fields": [] // Optional
}
```
### CLI Call Template
```json
{
"name": "multi_step_cli_tool",
"call_template_type": "cli", // Required
"commands": [ // Required - sequential command execution
{
"command": "git clone UTCP_ARG_repo_url_UTCP_END temp_repo",
"append_to_final_output": false
},
{
"command": "cd temp_repo && find . -name '*.py' | wc -l"
// Last command output returned by default
}
],
"env_vars": { // Optional
"GIT_AUTHOR_NAME": "UTCP Bot",
"API_KEY": "${MY_API_KEY}"
},
"working_dir": "/tmp", // Optional
"auth": null // Optional (always null for CLI)
}
```
**CLI Protocol Features:**
- **Multi-command execution**: Commands run sequentially in single subprocess
- **Cross-platform**: PowerShell on Windows, Bash on Unix/Linux/macOS
- **State preservation**: Directory changes (`cd`) persist between commands
- **Argument placeholders**: `UTCP_ARG_argname_UTCP_END` format
- **Output referencing**: Access previous outputs with `$CMD_0_OUTPUT`, `$CMD_1_OUTPUT`
- **Flexible output control**: Choose which command outputs to include in final result
### Text Call Template
```json
{
"name": "my_text_manual",
"call_template_type": "text", // Required
"file_path": "./manuals/my_manual.json", // Required
"auth": null, // Optional (always null for Text)
"auth_tools": { // Optional, authentication for generated tools from OpenAPI specs
"auth_type": "api_key",
"api_key": "Bearer ${API_TOKEN}",
"var_name": "Authorization",
"location": "header"
}
}
```
### MCP (Model Context Protocol) Call Template
```json
{
"name": "my_mcp_server",
"call_template_type": "mcp", // Required
"config": { // Required
"mcpServers": {
"server_name": {
"transport": "stdio",
"command": ["python", "-m", "my_mcp_server"]
}
}
},
"auth": { // Optional, example using OAuth2
"auth_type": "oauth2",
"token_url": "https://auth.example.com/token", // Required
"client_id": "${CLIENT_ID}", // Required
"client_secret": "${CLIENT_SECRET}", // Required
"scope": "read:tools" // Optional
}
}
```
## Security: Protocol Restrictions
UTCP provides fine-grained control over which communication protocols each manual can use through the `allowed_communication_protocols` field. This prevents potentially dangerous protocol escalation (e.g., an HTTP-based manual accidentally calling CLI tools).
### Default Behavior (Secure by Default)
When `allowed_communication_protocols` is not set or is empty, a manual can only register and call tools that use the **same protocol type** as the manual itself:
```python
from utcp_http.http_call_template import HttpCallTemplate
# This manual can ONLY register/call HTTP tools (default restriction)
http_manual = HttpCallTemplate(
name="my_api",
call_template_type="http",
url="https://api.example.com/utcp"
# allowed_communication_protocols not set → defaults to ["http"]
)
```
### Allowing Multiple Protocols
To allow a manual to work with tools from multiple protocols, explicitly set `allowed_communication_protocols`:
```python
from utcp_http.http_call_template import HttpCallTemplate
# This manual can register/call both HTTP and CLI tools
multi_protocol_manual = HttpCallTemplate(
name="flexible_manual",
call_template_type="http",
url="https://api.example.com/utcp",
allowed_communication_protocols=["http", "cli"] # Explicitly allow both
)
```
### JSON Configuration
```json
{
"name": "my_api",
"call_template_type": "http",
"url": "https://api.example.com/utcp",
"allowed_communication_protocols": ["http", "cli", "mcp"]
}
```
### Behavior Summary
| `allowed_communication_protocols` | Manual Type | Allowed Tool Protocols |
|----------------------------------|-------------|------------------------|
| Not set / `null` | `"http"` | Only `"http"` |
| `[]` (empty) | `"http"` | Only `"http"` |
| `["http", "cli"]` | `"http"` | `"http"` and `"cli"` |
| `["http", "cli", "mcp"]` | `"cli"` | `"http"`, `"cli"`, and `"mcp"` |
### Registration Filtering
During `register_manual()`, tools that don't match the allowed protocols are automatically filtered out with a warning:
```
WARNING - Tool 'dangerous_tool' uses communication protocol 'cli' which is not in
allowed protocols ['http'] for manual 'my_api'. Tool will not be registered.
```
### Call-Time Validation
Even if a tool somehow exists in the repository, calling it will fail if its protocol is not allowed:
```python
# Raises ValueError: Tool 'my_api.some_cli_tool' uses communication protocol 'cli'
# which is not allowed by manual 'my_api'. Allowed protocols: ['http']
await client.call_tool("my_api.some_cli_tool", {"arg": "value"})
```
## Testing
The testing structure has been updated to reflect the new core/plugin split.
### Running Tests
To run all tests for the core library and all plugins:
```bash
# Ensure you have installed all dev dependencies
python -m pytest
```
To run tests for a specific package (e.g., the core library):
```bash
python -m pytest core/tests/
```
To run tests for a specific plugin (e.g., HTTP):
```bash
python -m pytest plugins/communication_protocols/http/tests/ -v
```
To run tests with coverage:
```bash
python -m pytest --cov=utcp --cov-report=xml
```
## Build
The build process now involves building each package (`core` and `plugins`) separately if needed, though they are published to PyPI independently.
1. Create and activate a virtual environment.
2. Install build dependencies: `pip install build`.
3. Navigate to the package directory (e.g., `cd core`).
4. Run the build: `python -m build`.
5. The distributable files (`.whl` and `.tar.gz`) will be in the `dist/` directory.
## OpenAPI Ingestion - Zero Infrastructure Tool Integration
🚀 **Transform any existing REST API into UTCP tools without server modifications!**
UTCP's OpenAPI ingestion feature automatically converts OpenAPI 2.0/3.0 specifications into UTCP tools, enabling AI agents to interact with existing APIs directly - no wrapper servers, no API changes, no additional infrastructure required.
### Quick Start with OpenAPI
```python
from utcp_http.openapi_converter import OpenApiConverter
import aiohttp
# Convert any OpenAPI spec to UTCP tools
async def convert_api():
async with aiohttp.ClientSession() as session:
async with session.get("https://api.github.com/openapi.json") as response:
openapi_spec = await response.json()
converter = OpenApiConverter(openapi_spec)
manual = converter.convert()
print(f"Generated {len(manual.tools)} tools from GitHub API!")
return manual
# Or use UTCP Client configuration for automatic detection
from utcp.utcp_client import UtcpClient
client = await UtcpClient.create(config={
"manual_call_templates": [{
"name": "github",
"call_template_type": "http",
"url": "https://api.github.com/openapi.json",
"auth_tools": { # Authentication for generated tools requiring auth
"auth_type": "api_key",
"api_key": "Bearer ${GITHUB_TOKEN}",
"var_name": "Authorization",
"location": "header"
}
}]
})
```
### Key Benefits
- ✅ **Zero Infrastructure**: No servers to deploy or maintain
- ✅ **Direct API Calls**: Native performance, no proxy overhead
- ✅ **Automatic Conversion**: OpenAPI schemas → UTCP tools
- ✅ **Selective Authentication**: Only protected endpoints get auth, public endpoints remain accessible
- ✅ **Authentication Preserved**: API keys, OAuth2, Basic auth supported
- ✅ **Multi-format Support**: JSON, YAML, OpenAPI 2.0/3.0
- ✅ **Batch Processing**: Convert multiple APIs simultaneously
### Multiple Ingestion Methods
1. **Direct Converter**: `OpenApiConverter` class for full control
2. **Remote URLs**: Fetch and convert specs from any URL
3. **Client Configuration**: Include specs directly in UTCP config
4. **Batch Processing**: Process multiple specs programmatically
5. **File-based**: Convert local JSON/YAML specifications
📖 **[Complete OpenAPI Ingestion Guide](docs/openapi-ingestion.md)** - Detailed examples and advanced usage
---
## [Contributors](https://www.utcp.io/about)
