ESP32-Car-with-Visual-Recognition

3D Printed parts 

Qualcomm Dragonwing™ QCS8550

Ubuntu22.04

Personal Computer

ROS 2 Humble

<html><body> <figure><iframe src="https://youtu.be/VW9GyyRaVP8"></iframe></figure> <h3>Hardware Requirements</h3>Sandbox Environment : Road system with lane lines, zebra crossings, stop lines, and traffic lights <figure><img src="https://projects.arduinocontent.cc/e254ce23-cbb6-4cb5-a5a6-f09cc8b280ec.png"/><figcaption></figcaption></figure> Smart Car : Integrated with ESP32 microprocessor, camera, motor drive module, and Wi-Fi capability <figure><img src="https://projects.arduinocontent.cc/f5b2b797-acbb-40b3-a5bf-c7b87e204595.jpg"/><figcaption></figcaption></figure> Control System : Rhino Pi-X1(QCS8550) with the following specifications: <ol><li>AI Performance: ~48 TOPS INT8 </li><li>CPU: 1× Kryo Prime @ 3.2GHz, 2× Kryo Gold @ 2.8GHz, 3× Kryo Silver @ 2.0GHz </li><li>GPU: Adreno 740, GPU performance (GFLOPS) 3000 GFLOPS, 3 tflops </li><li>Memory: 16GB LPDDR5x (default), 24GB LPDDR5x (optional) </li><li>Connectivity: Wi-Fi 6, Bluetooth 5.1, LAN </li><li>Power: DC 12V/24V 3A </li></ol><h3>Software Requirements</h3>Operating System: RhinoPi X1 Android Image（Android 14 + Ubuntu 22.04） or Ubuntu 22.04 Development Tools: <ol><li>ROS2 (Robot Operating System 2) </li><li>OpenCV for computer vision </li><li>C++ and Python programming languages </li><li>ADB tools for device communication </li><li>QtScrcpy for screen mirroring </li></ol><h3>System Architecture</h3><ol><li>The system adopts a layered distributed architecture with three main layers: </li><li>Perception Layer (car end): Collects environmental data in real-time via camera </li><li>Decision Layer (central control system): Processes sensor data and generates control commands </li><li>Execution Layer (car end): Receives commands and controls vehicle movement </li></ol><h3>Core Modules</h3><ol><li>Communication Module: Enables data exchange between car and central system via Wi-Fi </li><li>Sensor Module: Captures RGB images and transmits video stream to central system </li><li>Motion Control Module: Parses speed commands and drives motors </li><li>Visual Perception Module: Detects lane lines, zebra crossings, traffic lights, and stop lines </li><li>Decision Planning Module: Implements finite state machine (FSM) for control behavior </li><li>Control Command Generation Module: Converts control behavior to speed commands </li></ol><h3>Installation and Setup</h3><figure><iframe src="https://youtu.be/0nSeEg7gBkg?si=xeZhPvH5AlC2K3i_"></iframe></figure>Hardware Setup : <ol><li>Assemble the smart car with ESP32, camera, and motor drive module</li></ol><figure><img src="https://projects.arduinocontent.cc/169135a2-b52f-4046-9c77-ac63b9203c41.png"/><figcaption></figcaption></figure><ol><li>Set up the sandbox environment with road markings and traffic elements </li><li>Connect the Rhino Pi-X1 to the network </li></ol>Software Setup : <ol><li>Install RhinoPi X1 Android Image or RhinoPi X1 Ubuntu22.04 on the Rhino Pi-X1 </li><li>Set up ROS2 environment </li><li>Install required dependencies (OpenCV, etc.) </li><li>Configure network connection between car and central system </li></ol>ROS2 Package Creation : <ol><li>Create interfaces for motion control (tank_interfaces) </li><li>Implement motion control nodes in C++ and Python </li><li>Create video stream publisher nodes </li><li>Implement computer vision nodes for environmental perception </li></ol><h3>Usage</h3>Start the System : <ol><li>Power on the Rhino Pi-X1 and smart car </li><li>Establish Wi-Fi connection between car and central system </li></ol>Run the Nodes : <ol><li>Start video stream publisher to receive camera feed </li><li>Run computer vision node for environmental perception </li><li>Start motion control node to receive and execute commands </li></ol>Testing : <ol><li>Use ROS2 topic publishing to send speed commands </li><li>Monitor the car's behavior in the sandbox environment </li><li>Adjust parameters for optimal performance </li></ol><h3>Communication Protocol</h3>Video Stream : HTTP/MJPEG stream via http://192.168.4.1:81/stream Control Commands : UDP protocol to 192.168.4.1:9000 with binary data format <h3>Code Structure</h3><pre>src/ </pre><pre>├── tank_interfaces/ # Custom ROS2 interfaces </pre><pre>│ └── msg/ </pre><pre>│ └── TankCommand.msg # Motion control message definition </pre><pre>├── tank_control/ # C++ motion control implementation </pre><pre>│ └── src/ </pre><pre>│ └── tank_control.cpp # UDP command sender </pre><pre>├── tank_control_py/ # Python motion control implementation </pre><pre>│ └── tank_control_py/ </pre><pre>│ └── tank_control.py # UDP command sender (Python) </pre><pre>├── video_stream_publisher/ # C++ video stream publisher </pre><pre>│ └── src/ </pre><pre>│ └── video_stream_publisher.cpp </pre><pre>├── video_stream_publisher_py/ # Python video stream publisher </pre><pre>│ └── video_stream_publisher_py/ </pre><pre>│ └── video_stream_publisher.py </pre><pre>├── computer_vision/ # C++ computer vision implementation </pre><pre>│ └── src/ </pre><pre>│ └── computer_vision.cpp </pre><pre>└── computer_vision_py/ # Python computer vision implementation </pre><pre>	 └── computer_vision_py/ </pre><pre>	 └── computer_vision.py </pre><h3>Key Features</h3><ol><li>Real-time Environmental Perception : Detects lane lines, zebra crossings, traffic lights, and stop lines </li><li>Distributed Control Architecture : Separates perception, decision, and execution layers </li><li>Dual-language Implementation : Provides both C++ and Python code for flexibility </li><li>ROS2 Integration : Leverages ROS2 for efficient communication and modular design </li><li>Custom Communication Protocol : Optimized for reliable control of the ESP32-based smart car </li></ol><h3>Troubleshooting</h3><ol><li>Connection Issues : Ensure both car and central system are on the same network </li><li>Video Stream Problems : Check camera connections and network stability </li><li>Motion Control Issues : Verify UDP port settings and command format </li><li>Detection Accuracy : Adjust OpenCV parameters for different lighting conditions </li></ol> </body></html>