xiaozhi-esp32-server-java

<h1 align="center">Xiaozhi ESP32 Server Java</h1> <p align="center"> A Java version server developed based on the <a href="https://github.com/78/xiaozhi-esp32">Xiaozhi ESP32</a> project, featuring a complete front-end and back-end management platform.<br/> Provides robust back-end support and an intuitive management interface for smart hardware devices. </p> <p align="center"> <a href="https://github.com/joey-zhou/xiaozhi-esp32-server-java/issues">Report Issues</a> · <a href="#deployment">Deployment Documentation</a> · <a href="https://github.com/joey-zhou/xiaozhi-esp32-server-java/blob/main/CHANGELOG.md">Changelog</a> </p> <p align="center"> <a href="https://github.com/joey-zhou/xiaozhi-esp32-server-java/graphs/contributors"> <img alt="GitHub Contributors" src="https://img.shields.io/github/contributors/joey-zhou/xiaozhi-esp32-server-java?logo=github" /> </a> <a href="https://github.com/joey-zhou/xiaozhi-esp32-server-java/issues"> <img alt="Issues" src="https://img.shields.io/github/issues/joey-zhou/xiaozhi-esp32-server-java?color=0088ff" /> </a> <a href="https://github.com/joey-zhou/xiaozhi-esp32-server-java/pulls"> <img alt="GitHub pull requests" src="https://img.shields.io/github/issues-pr/joey-zhou/xiaozhi-esp32-server-java?color=0088ff" /> </a> <a href="https://github.com/joey-zhou/xiaozhi-esp32-server-java/blob/main/LICENSE"> <img alt="License" src="https://img.shields.io/badge/license-MIT-white?labelColor=black" /> </a> <a href="https://github.com/joey-zhou/xiaozhi-esp32-server-java"> <img alt="stars" src="https://img.shields.io/github/stars/joey-zhou/xiaozhi-esp32-server-java?color=ffcb47&labelColor=black" /> </a> </p> <p align="center"> <b>If this project is helpful to you, please consider giving it a ⭐ Star!</b><br/> Your support is our motivation for continuous improvement! </p> --- ## Project Introduction 📝 Xiaozhi ESP32 Server Java is a Java version server developed based on the [Xiaozhi ESP32](https://github.com/78/xiaozhi-esp32) project, featuring a complete front-end and back-end management platform. This project aims to provide users with a feature-rich and user-friendly management interface to help them better manage devices, configurations, and more. Considering the needs of enterprise-level application scenarios, Java, as a mature enterprise-level development language, offers a more complete ecosystem support and stronger concurrency handling capabilities. Therefore, we chose to develop this Java version server to provide more possibilities and expansion space for the project. - **Back-end Framework**: Spring Boot + Spring MVC - **Front-end Framework**: Vue.js + Ant Design - **Data Storage**: MySQL + Redis - **Global Responsiveness**: Adapts to various devices and resolutions --- ## Target Audience 👥 If you have purchased ESP32-related hardware and wish to control and manage your devices through a feature-complete and user-friendly management platform, this project is perfect for you. Especially suitable for: - Users requiring enterprise-level stability - Individual developers looking for a quick setup - Users wanting a complete front-end management interface - Users needing stronger data management and analysis capabilities - Users with high requirements for system scalability - Scenarios requiring support for a large number of concurrent device connections - Applications with high demands for real-time data processing --- ## Feature Modules ✨ (Some content is not open-sourced; please contact us through the contact information below for inquiries) ### Completed Features ✅ | Feature Module | Status | Description | |----------------|--------|-------------| | **Device Management** | ✅ | View the list of all connected devices, real-time monitoring of device status, add/edit/delete device information, automatic application of default settings for device binding | | **Tone Selection** | ✅ | Provides various tone templates, previews tone effects, assigns different tone configurations for different devices | | **Tone Cloning** | ✅ | Supports tone cloning with Volcano Engine and Alibaba Cloud, enabling personalized voice customization | | **Chat History** | ✅ | View historical chat records, search chat content by date/keywords, delete messages, clear memory function | | **Intelligent Agents** | ✅ | Integrates with platforms like Coze and Dify to enable complex scenario dialogue capabilities | | **Role Switching** | ✅ | Pre-set role switching (AI teacher, male/female friend, smart home assistant, etc.) supports voice role switching | | **Persistent Dialogue** | ✅ | Supports persistent dialogue records for easy viewing of historical dialogue content | | **LLM Multi-Platform Support** | ✅ | Supports various large language models such as OpenAI, Zhiyuan AI, iFLYTEK Spark, Ollama, etc. | | **Default Configuration Management** | ✅ | Supports setting default configurations, automatically applying default settings to newly bound devices | | **IoT Device Control** | ✅ | Supports managing IoT devices through voice commands, enabling smart home control | | **Intelligent Function Invocation** | ✅ | Supports intelligent invocation of functions such as music playback (music service provided by third parties for personal entertainment use; this project does not assume any copyright responsibility), role switching, etc. | | **Multi-Voice Recognition Services** | ✅ | Supports various voice recognition services such as Funasr, Alibaba, Tencent, Vosk, etc. | | **Bidirectional Streaming Interaction** | ✅ | Supports real-time voice input and real-time reply output, enhancing dialogue fluency | | **Multimodal Interaction** | ✅ | Supports image recognition and processing, enabling richer interaction methods | | **Function Call** | ✅ | Supports LLM function calling capabilities for complex task processing and intelligent decision-making | | **Real-Time Interruption** | ✅ | Supports real-time interruption functionality, improving dialogue fluency | | **Local Offline Recognition** | ✅ | Supports Vosk local offline voice recognition, usable without internet connection | | **WebSocket Communication** | ✅ | High-performance WebSocket communication, supporting real-time status updates and control of devices | | **MQTT Communication** | ✅ | Supports MQTT communication protocol, long connections, and server-initiated wake-up | | **Automatic Voice Wake-Up** | ✅ | Supports custom wake words for activation without pressing buttons | | **Multiple Devices Simultaneous Access** | ✅ | Supports simultaneous access of multiple devices, achieving voice coverage throughout the house | | **TTS Multi-Engine Support** | ✅ | Supports various TTS engines such as Microsoft, Alibaba, Volcano, etc. | | **Multi-User Support** | ✅ | Supports multi-user configuration to meet the needs of multiple family members | | **User End** | ✅ | Native card-style user end device management page for easy configuration by users | ### Features Under Development 🚧 | Feature Module | Status | Description | |----------------|--------|-------------| | **Chat Data Visualization** | 🚧 | Data visualization features such as chat frequency statistics | | **Mixed Mode Roles** | 🚧 | Supports mixed mode for multiple roles, waking different roles with different wake words (automatic switching) | | **Memory Management** | 🚧 | Customizable memory dialogue count, historical dialogue summary/abstract function, manual operation of dialogue records | | **Voiceprint Recognition** | 🚧 | Supports voiceprint recognition functionality for personalized voice assistants | | **Multilingual Support** | 🚧 | Supports multilingual interfaces to meet the needs of users from different regions | | **Home Assistant** | 🚧 | Supports smart home device control, managing Home Assistant devices through voice commands | | **Sentiment Analysis** | 🚧 | Provides more humanized responses through voice sentiment analysis | | **Multi-Device Collaboration** | 🚧 | Supports collaborative work among multiple devices, achieving a voice assistant system that covers the entire house | | **Custom Plugin System** | 🚧 | Supports custom plugin development to extend system functionality | | **Knowledge Base Integration** | 🚧 | Supports integration with external knowledge bases to enhance Q&A capabilities | | **Voice Reminders and Alarms** | 🚧 | Supports setting voice reminders and alarm functions | | **Remote Control** | 🚧 | Supports remote control of devices for management while away | --- ## UI Showcase 🎨 <div align="center"> <img src="docs/images/device.jpg" alt="Device Management" width="600" style="margin: 10px;" /> <p><strong>Device Management</strong> - Comprehensive management and monitoring of all connected devices</p> </div> <details> <summary style="cursor: pointer; font-size: 1.2em; color: #0366d6; text-align: center; display: block; margin: 20px 0; padding: 10px; background-color:rgb(48, 48, 48); border-radius: 5px;"> <strong>👉 Click to see more interface screenshots 👈</strong> </summary> <div align="center"> <img src="docs/images/login.jpg" alt="Login Interface" width="600" style="margin: 10px;" /> <p><strong>Login Interface</strong> - Secure access point to the system</p> <img src="docs/images/dashboard.jpg" alt="Dashboard" width="600" style="margin: 10px;" /> <p><strong>Dashboard</strong> - Overview of the system and display of key data</p> <img src="docs/images/user.jpg" alt="User Management" width="600" style="margin: 10px;" /> <p><strong>User Management</strong> - Manage user information and permissions</p> <img src="docs/images/message.jpg" alt="Message Records" width="600" style="margin: 10px;" /> <p><strong>Message Records</strong> - View and search historical dialogue content</p> <img src="docs/images/model.jpg" alt="Model Management" width="600" style="margin: 10px;" /> <p><strong>Model Management</strong> - Configure and manage AI models</p> <img src="docs/images/agent.jpg" alt="Agent Management" width="600" style="margin: 10px;" /> <p><strong>Agent Management</strong> - Set and switch agents, Coze/Dify</p> <img src="docs/images/role.jpg" alt="Role Management" width="600" style="margin: 10px;" /> <p><strong>Role Management</strong> - Set and switch AI roles</p> <img src="docs/images/voiceClone.jpg" alt="Tone Cloning" width="600" style="margin: 10px;" /> <p><strong>Tone Cloning</strong> - Clone your own voice for a personalized voice assistant</p> </div> </details> --- <a id="deployment"></a> ## Deployment Documentation 📚 We provide various deployment methods to meet different user needs: ### 1. Local Source Run (This documentation is based on the old version JDK8 and has not been updated; the current project depends on JDK21) - [Windows Deployment Documentation](./docs/WINDOWS_DEVELOPMENT.md) - Suitable for development and testing in Windows environments - Provided by community member "汇合" - [CentOS Deployment Documentation](./docs/CENTOS_DEVELOPMENT.md) - Suitable for deployment in Linux server environments - Provided by community member "汇合" After successful operation, the console will output the OTA and WebSocket connection addresses. Refer to the firmware compilation documentation to connect the device to the service. ### 2. Docker Deployment - [Docker Deployment Documentation](./docs/DOCKER.md) - Quick containerized deployment solution - Provided by community member "💍Mr_li" After successful startup, the WebSocket connection needs to communicate through the host IP, for example: `ws://192.168.31.100:8091/ws/xiaozhi/v1/` ### 3. Video Tutorial - [Video Deployment Tutorial](https://doc.sivitacraft.com/article/xiaozhiai-javaserver/) - Recorded by community member "苦瓜" ### 4. Firmware Compilation - [Firmware Compilation Documentation](./docs/FIRMWARE-BUILD.md) - Detailed firmware compilation and flashing process After successful flashing and network connection, wake up Xiaozhi with the wake word and pay attention to the console information output from the server. --- ## Development Roadmap 🗺️ Based on our [Project Development Requirement List](https://github.com/users/joey-zhou/projects/1), we plan to implement the following features in the future: ### Short-Term Plans (2025 Q2) - Improve Function Call functionality to support more complex task processing - Implement mixed mode for multiple roles, supporting different wake words for different roles - Optimize the memory management system to provide more flexible historical dialogue management - Implement chat data visualization features to provide data analysis capabilities ### Mid-Term Plans (2025 Q3-Q4) - Implement voiceprint recognition functionality to support personalized voice assistants - Enhance Home Assistant integration to provide more comprehensive smart home control capabilities - Develop multimodal interaction features to support image recognition and processing - Implement a custom plugin system to support functional expansion ### Long-Term Plans (2026+) - Develop a multi-device collaborative working mechanism to achieve a voice assistant system that covers the entire house - Implement sentiment analysis functionality to provide a more humanized interaction experience - Develop knowledge base integration functionality to enhance Q&A capabilities - Implement multi-user support to meet the needs of multiple family members We will continuously adjust our development plans based on community feedback and technological advancements to ensure the project continues to meet user needs. --- ## Contribution Guidelines 👐 We welcome contributions in any form! If you have good ideas or find issues, please contact us through the following methods: ### WeChat Due to the WeChat group exceeding 200 members, you cannot scan to join. You can add me on WeChat with the note **小智**, and I will add you to the WeChat group. <img src="./web/static/img/wechat.jpg" alt="WeChat" width="200" /> ### QQ Feel free to join our QQ group for discussions. QQ group number: 790820705 <img src="./web/static/img/qq.jpg" alt="QQ Group" width="200" /> ### Custom Development We accept various customized development projects. If you have specific needs, please contact us via WeChat for discussions. --- ## Disclaimer ⚠️ This project only provides the technical implementation code for music and picture book playback and does not provide any media content. Users should ensure they have legal usage rights or copyright licenses when using related features and comply with copyright laws and regulations in their regions. Any example content or resources that may be involved in the project are sourced from the internet or provided by users, and are only for functional demonstration and technical testing. If any content infringes on your rights, please contact us immediately, and we will take appropriate measures such as deletion after verification. The developers of this project do not assume legal responsibility for any content obtained or played by users using this project's code. Using this project indicates that you agree to assume all legal risks and responsibilities during the usage process. --- ## Star History 📈 <a href="https://www.star-history.com/#joey-zhou/xiaozhi-esp32-server-java&Date"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=joey-zhou/xiaozhi-esp32-server-java&type=Date&theme=dark" /> <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=joey-zhou/xiaozhi-esp32-server-java&type=Date" /> <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=joey-zhou/xiaozhi-esp32-server-java&type=Date" /> </picture> </a>