Robotic Arm with Vision Grasping

Qualcomm Dragonwing™ QCS6490

Screwdriver set

ROS 2 Humble

Ubuntu22.04

OpenCV (cv2)

MG996R servo motors (Towerpro)

<html><body><h3></h3><figure><iframe src="https://www.youtube.com/watch?v=mnWCWt-nCNQ"></iframe></figure> <h3>Project Introduction</h3>This project is a robotic arm control system based on ROS2 Humble, integrating functional modules including low-level motion control, visual recognition (QR code/color), and MoveIt2 motion planning. Main Features: <ol><li>Low-Level Robotic Arm Control: Controls the arm servos via serial communication, and realizes the subscription and execution of the joint_states interface. </li><li>Visual Recognition System: Supports visual positioning functions such as camera calibration, ArUco marker detection, and RGB color recognition (red, green, and blue circular objects). </li><li>Motion Planning System: Implements obstacle-avoidance trajectory planning and execution for the robotic arm based on the MoveIt2 framework. </li><li>Eye-in-Hand Calibration: Supports both Eye-in-Hand and Eye-on-Base calibration modes. </li><li>Automated Process Control: Provides a complete task flow control node that implements automated grasping and placement based on a state machine. </li></ol>Application Scenarios: Robotic applications such as object grasping, visual positioning, and automated sorting. <h3>Environment Setup and Compilation</h3>Before running the program, please ensure that the ROS2 Humble environment has been properly configured and the workspace has been compiled. <pre># Navigate to the workspace </pre><pre>cd ~/01-program/05-mbot-marm/marm-aplux/marm_aplux_ws </pre><pre># Build the project </pre><pre>colcon build </pre><pre># Refresh environment variables (run this every time you open a new terminal, or add it to ~/.bashrc) </pre><pre>source install/setup.bash </pre><h3>Preparation: Calibration Procedures</h3>[!tip] Calibration Order It is recommended to complete calibration in the following order: Arm calibration → Camera intrinsic calibration → Hand-eye calibration. Each step affects the accuracy of subsequent steps. Robotic Arm Zero-Position Calibration Due to assembly tolerances, the robotic arm may have angular errors at joints. Zero-position calibration is required to avoid grasping deviations caused by inconsistencies between actual and commanded poses. Steps: 1. Open a terminal and launch the arm SDK and MoveIt control program respectively: <pre>ros2 run arm_controller arm_controller </pre><pre>ros2 launch A1_moveit_config demo.launch.py </pre>2. Observe the initial posture of the arm after power-on. Under normal conditions, it should be perfectly vertical: <figure><img src="https://projects.arduinocontent.cc/cc9c007c-cdb1-4576-9296-3be42bebabe1.png"/><figcaption></figcaption></figure>3. If the arm has angular deviation, open the configuration file and modify the servo offset parameters: <figure><img src="https://projects.arduinocontent.cc/cdf8b5e5-a5e1-4af1-b2f9-c12396ec20e2.png"/><figcaption></figcaption></figure>4. Modify the offset values for the corresponding joints in servo_offsets (adjust incrementally), then save and recompile: <figure><img src="https://projects.arduinocontent.cc/7c29241a-bf3c-445e-a842-33e6536792a7.png"/><figcaption></figcaption></figure><pre>servo_offsets: </pre><pre>servo_0: 0.0 # joint1 </pre><pre>servo_1: 1.0 # joint2 </pre><pre>servo_2: 0.0 # joint3 </pre><pre>servo_3: 0.0 # joint4 </pre><pre>servo_4: 0.0 # joint5 </pre><pre>servo_5: 0.0 # joint6 </pre>5. Recompile and launch arm_controller , then check whether the arm is vertical: <pre>cd ~/marm_ws &amp;&amp; colcon build </pre><pre>ros2 run arm_controller arm_controller </pre><h3>Camera Intrinsic Calibration</h3>[!warning] Important Notice Intrinsic parameters vary significantly across different USB cameras. Recalibration is required every time the camera is replaced, otherwise 3D localization accuracy will be severely degraded. <h3>Steps:</h3>1. Launch the camera driver and calibration program: <pre>ros2 launch capture_image capture_image.launch.py </pre><pre>ros2 run camera_calibration cameracalibrator \ </pre><pre>--size 10x7 --square 0.021 \ </pre><pre>image:=/camera/image_raw camera:=/camera \ </pre><pre>--no-service-check </pre>2. A calibration window will pop up after launch: <figure><img src="https://projects.arduinocontent.cc/049bf2dc-55f5-47f6-aecf-d1c72c26f956.png"/><figcaption></figcaption></figure>3. Move the calibration board in front of the camera at various angles and orientations until all indicators (X, Y, Size, Skew) in the right panel turn green: <figure><img src="https://projects.arduinocontent.cc/38f4aee1-a3a2-406d-80b1-65a471b8fc89.png"/><figcaption></figcaption></figure>4. Click the Calibrate button, wait for the calibration computation to complete, then click Save to save the results. Calibration data is saved at /tmp/calibrationdata.tar.gz : <figure><img src="https://projects.arduinocontent.cc/3fd69ab8-e068-4873-9736-5aa06448e52c.png"/><figcaption></figcaption></figure>5. Copy the ost.yaml file from the archive to the capture_image package directory, then recompile: <figure><img src="https://projects.arduinocontent.cc/ae61e5c5-b101-48d2-9c0c-55a31fb838b7.png"/><figcaption></figcaption></figure><pre>cd ~/marm_ws &amp;&amp; colcon build </pre><h3>Startup Procedure (Execute in Order)</h3>Please open multiple terminals sequentially. In each terminal, run source install/setup.bash first before executing the following commands. <pre>ros2 run arm_controller arm_controller </pre>Function Description: Starts the low-level control node of the robotic arm, subscribes to the /joint_states topic, converts joint angles into servo serial commands, and sends them to the robotic arm. <figure><img src="https://projects.arduinocontent.cc/56f9b1e6-e864-4b31-b517-9bc3924ebc70.png"/><figcaption></figcaption></figure>Clamp control test (optional): <pre># 打开夹⽖ </pre><pre>ros2 topic pub /gripper_control std_msgs/msg/Bool "data: true" --once </pre><pre># 关闭夹⽖ </pre><pre>ros2 topic pub /gripper_control std_msgs/msg/Bool "data: false" --once </pre><h3>Start camera driver</h3><pre>ros2 launch capture_image capture_image.launch.py </pre>Function Description: Starts the USB camera node and publishes the image data stream. Note: The program runs successfully if no errors or the "open camera failed" message appears. If the camera fails to open, the USB camera may be loose or you may have insufficient permissions. Please reconnect the camera and restart the program. <figure><img src="https://projects.arduinocontent.cc/75e81d64-5ca2-48ca-9345-39fc4a8e6535.png"/><figcaption></figcaption></figure><h3>Launch Visual Recognition Node</h3>Select either ArUco Marker Detection or Color Detection to launch based on your task requirements (the two cannot be run simultaneously). Option A: ArUco Marker Detection <pre>ros2 run capture_image aruco_object_pose_node </pre>Function Description Detects ArUco markers and publishes the pose (TF) of the object in the camera coordinate system. You can view the detection results using rqt_image_view: <pre>ros2 run rqt_image_view rqt_image_view </pre> <figure><img src="https://projects.arduinocontent.cc/3ca15a81-9b9e-465b-afca-9551d5ffc8f6.png"/><figcaption></figcaption></figure>Option B: RGB color recognition <pre>ros2 run capture_image rgb_detect_pose_node </pre>Function Description: Detects circular objects in red, green, and blue, calculates their 3D positions, and publishes TF transforms.After the program starts, three windows will pop up: <ol><li>Color threshold adjustment window </li><li>Threshold visualization window (Mask Preview) </li><li>Detection result window (RGB Detection) </li></ol> <figure><img src="https://projects.arduinocontent.cc/22ef13fe-b96e-4203-8458-c587e0acf052.png"/><figcaption></figcaption></figure> <figure><img src="https://projects.arduinocontent.cc/40d99d0d-0ebb-490c-bebc-7c56a7df6fbf.png"/><figcaption></figcaption></figure>Color Threshold Adjustment Steps: <ol><li>1.Select the color channel: In the adjustment window, the Color Select slider corresponds to three colors (0 = Red, 1 = Green, 2 = Blue). </li><li>2.Adjust the red threshold (0): Set Color Select to 0. Observe the Mask Preview window and adjust the HSV min–max values until only one clear, bright circular area remains. The RGB Detection window will then display the detection bounding box and position information. </li><li>3.Adjust green/blue thresholds: Set the Color Select slider to 1 (Green) or 2 (Blue), and repeat the HSV adjustment steps above until a complete circular area appears. </li></ol> <figure><img src="https://projects.arduinocontent.cc/0f54de3d-ff72-49a5-80b9-0a9675928b6e.png"/><figcaption></figcaption></figure> <figure><img src="https://projects.arduinocontent.cc/223fc07e-6edf-4e58-ab34-510b13ef9e72.png"/><figcaption></figcaption></figure> <figure><img src="https://projects.arduinocontent.cc/c3f20976-32b0-4d5f-a1ce-b1ef31e83bc6.png"/><figcaption></figcaption></figure> <h3>Activate MoveIt2 motion planning</h3><pre>ros2 launch A1_moveit_config demo.launch.py </pre>Function description: Launch the MoveIt2 core framework, load the robot URDF model, semantic configuration (SRDF), and open the RViz visualization interface. <h3>Launch Automated Process Control Node</h3><pre>ros2 run moveit_controller_server marm_demo </pre>Function Description： <ol><li>Executes a predefined automated grasping process to implement complete task control logic. </li></ol>Grasping State Machine Flow: <ol><li>Initial Detection: The robot moves to the stand observation position and checks for target objects (ArUco markers or specific colors) within the field of view. </li></ol>Grasping and Rechecking： <ol><li>After detecting the object, the robotic arm moves above the target for grasping, then lifts to the stand position for a second recognition. If the object is still detected in the field of view, it is judged as a failed grasp (empty grab). The system will automatically adjust the pose and retry grasping. </li></ol>Placing and Resetting： <ol><li>After successful grasping, the arm moves the object to the preset put position and releases it. Once completed, the arm returns to the home position, then moves to the stand position to prepare for the next grasping cycle. </li></ol> <figure><img src="https://projects.arduinocontent.cc/484b8d31-fdf9-4725-9111-587977de25b1.png"/><figcaption></figcaption></figure> <figure><img src="https://projects.arduinocontent.cc/6a00ec51-4602-492b-9632-2b505385853e.png"/><figcaption></figcaption></figure> <figure><img src="https://projects.arduinocontent.cc/fa7385fd-7af8-4954-9637-ea48e11bb42b.png"/><figcaption></figcaption></figure><h3>Workspace Directory Structure</h3><pre>marm_aplux_ws/src/ </pre><pre>├── A1_description/ # Robotic arm URDF model package (models, meshes, RViz configurations) </pre><pre>├── A1_moveit_config/ # MoveIt2 configuration package (SRDF, kinematics, controller config, launch) </pre><pre>├── arm_controller/ # Low-level robotic arm control package (serial communication, servo control, gripper control) </pre><pre>├── capture_image/ # Vision recognition package (camera driver, ArUco detection, RGB color detection) </pre><pre>├── custom_interface/ # Custom message and service interface package (grasp actions, pose services) </pre><pre>├── easy_handeye2/ # Hand-eye calibration toolkit (supports Eye-in-Hand / Eye-on-Base) </pre><pre>├── easy_handeye2_msgs/ # Custom messages for hand-eye calibration </pre><pre>└── moveit_controller_server/# Motion control &amp; business logic package (MoveIt service encapsulation, state machine demo) </pre><h3>Tutorial</h3><figure><iframe src="https://www.youtube.com/watch?v=WdoS1cCuBN8"></iframe></figure> </body></html>