Content
# Commercial LLM and MCP-based In-vehicle Intelligent AI Assistant (Demo Version)
<p align="center"> <img src="asset\flow.png" width="80%"> </p>
- Work Content: Based on commercial large model APIs, fine-tuned small semantic recognition models, and MCP technology, a in-vehicle intelligent AI assistant (demo) with 400+ skills was constructed. By developing unrelated semantic rejection technology and double-layer intent recognition technology, accurate and efficient dialogue and control in vehicle scenarios are achieved.
- Optimization Focus: By using double-layer intent recognition, dynamic Function Calling is achieved. First, the one-level fine-tuned model is used to recall Top-k (5) candidate intents, and then the commercial LLM is used for second-level precise intent matching and parameter slot extraction. This approach avoids performance degradation and increased consumption caused by large models blindly searching in a massive tool library.
- Task Output: On the three-layer Bert-tiny fine-tuned rejection model, the rejection rate of invalid semantics reaches 90%+, and the QPS reaches 400+. On the Bert-large fine-tuned one-level intent recall model, the Top1 accuracy reaches 85%, and the Top5 accuracy reaches 98%. When combined with the commercial large model's second-level intent recall, the accuracy reaches 87%. Third-party MCP services such as navigation and music are introduced.
## Environment Installation and Variable Import
```bash
conda create -n agent_nlu python=3.12
conda activate agent_nlu
pip install -r requirements.txt
```
Modify preset environment variables (config/config.ini):
```
# pass
export API_KEY="Bearer xxxx" # API key for commercial LLM
export BASE_URL="https://ark.cn-beijing.volces.com/api/v3/chat/completions"
export BOT_URL="https://ark.cn-beijing.volces.com/api/v3/bots/chat/completions"
export AMAP_MAPS_API_KEY="xxxx" # API key for AMAP
# Microservices
export REJECT_URL="http://127.0.0.1:8007/reject-server/v1"
export INTENT_URL="http://127.0.0.1:8008/intent-server/v1"
export NLU_URL="http://127.0.0.1:8009/chatnlu-server/v1"
export ENTRY_URL="http://127.0.0.1:8080/request_nlu"
```
Load preset environment variables:
```
source config/config.ini
```
## Model Training or Model
### Directly Use Trained Models
Download [trained models](https://drive.google.com/drive/folders/1SDmqF-Nzf_zmj2F4Y-8VxvCGVxdntb5X?usp=drive_link)
bert_tiny.ckpt is a 6-layer bert rejection model, place it in saved\reject
bert.ckpt is an intent recognition model, place it in saved\intent
### Train Your Own Models
Train rejection/intent recognition models (train\run.py):
```bash
python train\run.py --model bert_tiny --data reject
```
## Start Service
The main program defaults to **http://127.0.0.1:8080**
The rejection model defaults to **http://0.0.0.0:8007**
The intent recognition model defaults to **http://0.0.0.0:8008**
The nlu model defaults to **http://0.0.0.0:8009**
Redis defaults to **standalone** mode, port=6379:
```bash
bash server.sh
```
### Local Dialogue System
```bash
python dialog.py
>> connected to server
>> enter query:
>> Open the left window
>> Response: { 'query': 'Open the left window', 'tarce_id': '9n4k316g7', 'intent': 'Open_Window', 'intent_id': '35', 'function': 'Open_Window', 'slots': {'location': 'left'}, 'cost': 0.9744772911071777 }
```
## Module Accuracy and Stress Testing
### Module Accuracy Testing
Test the accuracy of the rejection model:
```bash
python test\reject_client.py
>> 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1332/1332 [00:09<00:00, 136.24it/s]
>> test avg acc: 0.9114114114114115
```
Test the accuracy of the intent recognition model (Top1):
```bash
python test\intent_client.py
>> 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7400/7400 [01:05<00:00, 112.54it/s]
>> test avg acc@1: 0.8572972972972973
```
Test the accuracy of the nlu module:
```bash
python test\nlu_client.py
>> 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2212/2212 [36:42<00:00, 1.00it/s]
>> test intent acc: 0.8652802893309223 slots acc: 0.9159132007233273
```
### Module Stress Testing
The specific stress testing results for the three modules are as follows:
- Rejection model QPS: around 400
- Intent recognition model QPS: around 200
- nlu module QPS: around 1-2 (using a 4090D GPU)
### Full-Process Testing
```bash
python test.py
```
## Code Understanding
### Main Inference Function ``start.py:inference``
```python
@socketio.on("request_nlu")
def inference(req):
"""
Core function to handle client-sent natural language requests
Responsible for coordinating multiple NLP services (NLU, arbitration, rejection, chat, etc.) and returning processing results according to business rules
Triggered when the client sends a "request_nlu" event via WebSocket
Parameters:
req: JSON string sent by the client, containing user query, user identifier, etc.
"""
# ... (rest of the code remains the same)
```
### NLU Processing
<p align="center"> <img src="asset\nlu.png" width="80%"> </p>
`function_call\chatnlu_infer.py:inference,predict`
```python
@app.post("/chatnlu-server/v1") # Define POST interface path
async def inference(request: Request): # Asynchronous interface function, receiving Request object
json_info = await request.json() # Parse JSON data in the request body
begin = time.time() # Record the start time of interface processing
query = json_info.get("query") # Get user query text
enable_dm = json_info.get("enable_dm", True) # Get whether to enable dialogue management flag (default True)
trace_id = json_info.get("trace_id", "1") # Get tracking ID (default "1")
# Call predict function to extract intent and slots
nlu = predict(query, trace_id)
# Parse NLU result (format like "intent-slot1:value1,slot2:value2")
nlu_items = nlu.split("-") # Split into intent and slots parts
intent = nlu_items[0] # Extract intent (like "weather query")
# Process slots part (if slots contain "-" symbol, need special processing)
if len(nlu_items) > 2:
slots_str = "-".join(nlu_items[1:]) # Merge slots string with "-"
else:
slots_str = nlu_items[1] # Ordinary slots string
# Parse slots into dictionary format
if slots_str != "无": # If slots exist
slots = {}
for item in slots_str.split(","): # Split multiple slots by comma
if ":" in item: # Verify slot format (must contain ":")
if len(item.split(":")) != 2: # Skip format error slots
continue
k, v = item.split(":") # Split slot name and value
slots[k] = v # Add to slots dictionary
else:
slots = {} # Return empty dictionary if no slots
# Get intent ID and corresponding function name
intent_id = name2id.get(intent) # Query intent ID by name
func_name = id2func.get(intent_id) # Query function name by intent ID
# Construct basic response body
response = {
"query": query, # User original query
"trace_id": trace_id, # Tracking ID (note: there is a spelling error, should be "trace_id")
"intent": intent, # Intent name
"intent_id": intent_id, # Intent ID
"function": func_name, # Corresponding processing function name
"slots": slots, # Slots dictionary
}
# If enable dialogue management, call corresponding domain's dialogue manager
if enable_dm:
# Traverse domains to be processed (weather, music, maps)
for name in ["weather", "music", "maps"]:
# Call domain's dialogue manager to process
dm_result = await DMFactory.get(name)(func_name, query, slots)
if dm_result: # If processing successful
tool_response, nlg = dm_result # Unpack tool response and natural language generation result
response["tool"] = tool_response # Add tool response to result
response["nlg"] = nlg # Add natural language reply to result
# Calculate total time consumption and add to response
cost = time.time() - begin
response["cost"] = cost
return response # Return final processing result
```
### Slot Extraction
function_call\slot_process.py
```python
def value_process(key, value):
# Define position mapping table: convert Chinese position descriptions to standardized English identifiers
position_map = {
"主驾": "MAIN",
"副驾": "VICE",
"左侧": "LEFT",
"右侧": "RIGHT",
"前排": "FRONT",
"后排": "REAR",
# More position mappings...
}
# Process number/proportion type slots (like percentage, numerical value)
if key in ["NUMBER", "RATIO"]:
if "%" in value: # If contains percentage sign, convert to decimal (like "50%" → 0.5)
value = float(eval(value.replace("%", "")) / 100)
else: # Pure number, directly convert to float (like "25" → 25.0)
value = float(eval(value))
# Process position type slots: unify to English identifiers
elif key in ["POSITION"]:
value = position_map.get(value, value) # If no match, keep original value
# Process "dialogue duration" slot: remove "second" unit (like "30 seconds" → "30")
elif key == "对话时长":
value = value.replace("秒", "")
# Process "Extreme" slot: unify to "max" or "min"
elif key == "Extreme":
if value in ["最大", "最高", "最强", "最亮", "最热"]:
value = "最大" # Unify positive extreme descriptions
if value in ["最小", "最低", "最弱", "最暗", "最冷"]:
value = "最小" # Unify negative extreme descriptions
return value # Return converted standardized value
def intent_slot(function, map_intent, slot_map):
try:
# 1. Extract model-predicted function name (original intent identifier)
predict_e = function[0].get("function", {}).get("name", "NULL")
# 2. Map function name to user-readable intent name (like "set_temp" → "set temperature")
predict_z = map_intent.get(predict_e, predict_e) # If mapping fails, keep original value
# 3. Extract slot parameters and parse into dictionary (model returns JSON string)
slots_predict = function[0].get("function", {}).get("arguments", "{}")
slots_predict = json.loads(slots_predict) # Convert to Python dictionary
# 4. Construct result string (format: "intent-slot1:value1,slot2:value2")
result = predict_z + "-" # Concatenate intent name and separator
# 5. Convert slots according to slot mapping table (slot_map) and standardize slot values
dict_slot = slot_map.get(predict_e) # Get current intent's corresponding slot mapping rules
if slots_predict: # If slot parameters exist
for key, value in slots_predict.items():
# Filter invalid values (empty values, "不限", or no mapping rules)
if value and isinstance(dict_slot, dict) and value != "不限":
# Convert slot name (like "pos" → "POSITION")
key = dict_slot.get(key, key)
# Call value_process to standardize slot value
value = value_process(key, value)
# Concatenate slot information to result string (like "POSITION:MAIN,")
result = result + f"{key}:{str(value)}" + ","
else:
continue # Skip invalid slots
# Remove extra comma at the end (like "intent-slot1:value1,slot2:value2," → "intent-slot1:value1,slot2:value2")
result = result.rsplit(",", 1)[0]
# 6. If no valid slots, supplement "无" (like "intent-无")
if ":" not in result:
result = result + "无"
# If any exception occurs, return default "未知-无"
except Exception as e:
return "未知-无"
return result # Return final "intent-slot" string
```
### Reject Model, Intent Recognition Model Service Startup
train\intent_infer.py or train\reject_infer.py
```python
def predict(query):
with torch.no_grad(): # Disable gradient calculation, save memory and accelerate inference
# 1. Tokenize input text
token = config.tokenizer.tokenize(query) # Split query text into tokens (like "open air conditioner" → ["open", "air conditioner"])
# 2. Add context marker and process sequence length
token = [CLS] + token # Add CLS marker at the beginning (required by BERT model)
seq_len = len(token) # Record current sequence length
# 3. Process mask and token IDs
mask = [] # Used to mark valid tokens (1) and padding tokens (0)
token_ids = config.tokenizer.convert_tokens_to_ids(token) # Convert tokens to model-recognizable IDs
# 4. Pad or truncate sequence to fixed length (model-required input length)
if len(token) < config.pad_size: # If sequence length is less than specified length, pad
mask = [1] * len(token_ids) + [0] * (config.pad_size - len(token)) # Valid token part is 1, padding part is 0
token_ids += [0] * (config.pad_size - len(token)) # Pad token IDs with 0
else: # If sequence length exceeds specified length, truncate
mask = [1] * config.pad_size # Truncated all positions are valid tokens
token_ids = token_ids[: config.pad_size] # Truncate token IDs to specified length
seq_len = config.pad_size # Update sequence length to specified length
# 5. Convert to PyTorch tensor and move to specified device (CPU/GPU)
x = torch.LongTensor([token_ids]).to(config.device) # Token ID tensor
seq_len = torch.LongTensor([seq_len]).to(config.device) # Sequence length tensor
mask = torch.LongTensor([mask]).to(config.device) # Mask tensor
# 6. Model inference
texts = (x, seq_len, mask) # Package input data
output = model(texts) # Model output (raw logits without softmax)
# 7. Calculate probability and get TopK results
prob = F.softmax(output, dim=-1).cpu().numpy()[0] # Apply softmax to output, get probability distribution, and move to CPU as numpy array
index = np.argsort(-prob)[:TOPK] # Sort by probability in descending order, take top TOPK intent indices
return index, prob[index] # Return TopK intent indices and corresponding probabilities
@app.post("/intent-server/v1") # Define POST request interface path
async def inference(request: Request): # Asynchronous interface function, receiving Request object
json_info = await request.json() # Parse JSON data in the request body
query = json_info.get("query") # Get user input query text
trace_id = json_info.get("trace_id") # Get unique ID for tracking
result = {} # Initialize return result dictionary
try:
# Call predict function to get TopK intent indices and probabilities
response, score = predict(query)
except:
# If inference process errors, return default result (intent 3, probability 1.0, TOPK items)
response, score = [3] * TOPK, [1.0] * TOPK
# Format result: convert indices and probabilities to comma-separated strings
result["data"] = ",".join([str(k) for k in response]) # Intent index string (like "0,1,2,3,4")
result["score"] = ",".join([str(k) for k in score]) # Probability string (like "0.9,0.05,0.03,0.01,0.01")
# Log: include tracking ID, query text, return result, and confidence
logger.info(
"Trace ID: {}, Request: {}, response: {}, confidence: {}".format(
trace_id, query, result["data"], result["score"]
)
)
return result # Return processing result
```
## Todo
- Identify multiple intents in one dialogue and associate execution (like "close the left window and increase volume")
- Integrate RAG in encyclopedia casual conversation flow, including user personal database and professional database (also possible through commercial APIs)
- Develop ask conversion system (currently only manual input query)
- Pressure test on more computing devices
