[AfA2026-PhysicalAI] - Semantic Manipulator using UNO-Q

VS Code

 USB C to USB C Cable 100W Fast Charging

Metal cutter, a machine or tool that can cut aluminium (Or some way to cut the V-shaped Curved aluminium Cover for the leds)

Arduino® UNO™ Q 2GB

Aluminum Profile 2020, 400 mm, T-Slot Type

RoArm - M2

USB C Hub Multiport Adapter for MacBook Pro/Air, 10 in 1 USB C Dongle Mac HDMI Adapter, USBC Dock Compatible for iPad Pro, iMac, Dell, HP, Lenovo (4K@60Hz HDMI,SD/TF Card Reader,100W PD, 6 USB Ports) 

Intel RealSense Camera

Arduino App Lab

printer paper

<html><body><h3>Description</h3><p>Industrial robotic arms are traditionally complex to program, requiring specialized knowledge of coordinate systems, kinematics, and pendant programming. This creates a massive barrier to entry for non-technical users who want to utilize robotics for dynamic tasks like sorting, organization, or lab automation. </p><p>The <strong>Semantic Manipulator</strong> is an Embodied AI Agent that bridges the gap between human language and physical actuation. By integrating a Large Language Model (LLM) with the Arduino UNO Q kit, this project allows users to control a 4-DOF robotic arm using natural language commands (e.g., "Pick up the red block and move it to the right"). </p><p> </p><figure><iframe src="https://youtu.be/FhsI4L-pOcw"></iframe></figure><p> </p><h3>How does it work?</h3><ol><li>First, we run a calibration script which aligns the coordinate frame of the RoArm - M2 and the camera so that when the camera sees a block, the robotic arm knows where to grab it from. </li><li>After calibration is done, the camera first sees the table and figures out where the block are and their colors. </li><li>When you give a command, uno-Q first understands what you said by making your instruction go through a text-parser. A text parser basically figures out which function to call based on your instruction. </li><li>Once that is done, the uno-Q instructs the arm to move accordingly, carrying out your requests. </li></ol><h3>Setup</h3><p>We have a chart paper setup on the table with markings for our own reference. We have placed the RoArm - M2 as show in the picture and have the camera mounted on top, facing down at the table. We have secured the camera with a rig made of aluminum profiles. </p><p> </p><figure><img src="https://projects.arduinocontent.cc/f2f35943-0768-4e43-a420-7d0d059179d8.jpg"/><figcaption></figcaption></figure><p> </p><p> </p><h3>User Interface</h3><p>The primary interface is a Natural Language Chat Console made using <a href="https://www.gradio.app/">Gradio </a></p><p> </p><p> </p><figure><img src="https://projects.arduinocontent.cc/468bbde2-cd1b-4ed4-a011-83524aa8327a.png"/><figcaption></figcaption></figure><p> </p><p><strong>Input </strong>: The user types a command (e.g., "Sort the green blocks"). </p><p><strong>Feedback </strong>: The system responds with text confirmation (e.g., "Identified 2 green blocks. Executing sequence...") and visual feedback via the Arduino Uno Q's LEDs (Green for 'Active', Red for 'Error/Busy'). </p><p><strong>Safety </strong>: If a user inputs an impossible command (e.g., "Drop block" when the gripper is empty), the interface returns a "No Operation" warning, preventing hardware damage. </p><p> </p><h3>AI Model</h3><p>This project implements a Hybrid Neuro-Symbolic Architecture on UNO-Q that emphasizes local data processing: </p><p><strong>Semantic Perception (Generative AI) </strong>: We utilize an <strong>LLM</strong> to perform semantic parsing. It decodes vague human intent into a deterministic JSON format, extracting Action, Target_Color, and Coordinate_Modifiers. </p><p><strong>Visual Spatial Intelligence (Computer Vision) </strong>: A local <strong>vision pipeline</strong> processes the RGB feed from the webcam. It generates a <strong>dynamic scene</strong> of the table, creating a dictionary of objects and their real-world coordinates relative to the robot's base frame. </p><p><strong>Deterministic Logic (The Arduino Core) </strong>: The Arduino Uno Q runs a lightweight, <strong>low-latency AI logic model</strong>. It acts as the "Grounding Layer," cross-referencing the LLM's intent with the Vision System's reality. This ensures that the AI's "hallucinations" never result in unsafe physical actions. </p><p> </p><h3>Mechanical design</h3><p>The <a href="https://www.waveshare.com/wiki/RoArm-M2-S?srsltid=AfmBOorC5C_zdrNgfwGIXpWhtkNhBEC25YuppkjZukkAWBEfGckbQhTs">RoArm - M2 </a>is a commercially available arm which comes with lots of handy features for projects. It was mounted to the table with a screw clamp. We have created a simple rig with aluminum profiles to mount the camera at the appropriate height(120 cm worked for us). Do secure your chart paper to avoid movement. Every time your setup moves(i.e. the camera, chart paper, or the arm) you will have to calibrate it again. </p><h3>Electronics</h3><p>Connect your uno-Q with a monitor, mouse, keyboard, camera and power. We used a Intel RealSense camera. You can use any camera(webcam etc) that you have. The RoArm - M2 just requires power and is controlled via JSON commands through WiFi(hosted by the onboard ESP). For more details on controlling the arm, visit <a href="https://www.waveshare.com/wiki/RoArm-M2-S?srsltid=AfmBOorC5C_zdrNgfwGIXpWhtkNhBEC25YuppkjZukkAWBEfGckbQhTs">this page </a>. </p><h3>Software</h3><p>We use openCV library for all camera related tasks. We have created an webpage as an interface for instructions using <a href="https://www.gradio.app/">Gradio </a>. </p><p>RoArm - M2 is controlled using JSON commands. </p><p>To run the setup, run homepage.py after setting up the dependencies. Detailed instructions are available in the <a href="https://github.com/szyfrowac/Qualcomm-AI-Challenge.git">github repository </a>. </p><h3>Software Architecture</h3><p>The system is built on a modular "Sense-Think-Act" loop: </p><p><strong>Vision Node </strong>(Python/OpenCV) </p><ol><li>Captures frames from the webcam. </li><li>Detects ArUco markers to establish a World Coordinate Frame. </li><li>Detects colored blocks and broadcasts their coordinates (x, y, z) to the controller. </li></ol><p> </p><p><strong>Intelligence Node </strong>(LLM + Parser): </p><ol><li>Receives user text input. </li><li>Outputs a command packet: {"cmd": "PICK", "target": "RED", "id": 1}. </li></ol><p> </p><p><strong>Control Node </strong>(Arduino UNO Q): </p><ol><li>State Machine: IDLE -&gt; VALIDATE -&gt; SOLVE_IK -&gt; EXECUTE -&gt; RETURN. </li><li>Receives the command packet via Serial. </li><li>Validates the target exists in the Vision Node's data. </li><li>Delegates the motion control to the robotic arm's ESP32 driver. </li></ol><h3>Calibration</h3><p>Setup your overhead camera and robotic arm on a table. </p><p>Place Aruco markers the corners of a rectangle of size 45cmx30cm. Place marker '0' at the furthest left corner from the arm and the rest of them in clockwise order. </p><p>Connect to RoArm - M2 on WiFi. Switch off torque to the motors. </p><p>Place the arm on each Aruco marker in ascending order and press enter. </p><p>Now, move the arm away from the frame such that markers are clearly seen from the camera. Press enter. </p><p>Calibration is now done. </p><h3></h3><h3></h3><p> </p></body></html>

Github Link

[AfA2026-PhysicalAI] - Semantic Manipulator using UNO-Q

Controlling RoArm - M2 to 'pick' and 'place' colored blocks via instructions in natural language