Autonomous pantry assistant for low-vision users

Arduino® UNO™ Q 2GB

Edge Im pulse

3D printer 

<html><body>This is an automated system to recognize pantry items. It runs on the Arduino UNO Q dual-brain architecture. It uses a Time-of-Flight hardware sensor on the microcontroller to measure distance, combined with Edge Impulse AI and RapidOCR text reading on Debian Linux. Developed for Hackestiu 2026 (Càtedra CHIP UPC &amp; Qualcomm). <h3>Authors</h3><ol><li>Lluis Vidal </li><li>Umar Mohammad </li></ol><h3>Quick Start (One-Liner Installation)</h3>Run this single command in your Arduino UNO Q terminal. It will clone the repository, install all dependencies, and flash the STM32 microcontroller: <pre>git clone https://github.com/mr-umar/pantry-assistant.git &amp;&amp; cd pantry-assistant &amp;&amp; chmod +x install.sh run.sh src/model.eim &amp;&amp; ./install.sh </pre>Once installed, start the system: <pre>./run.sh </pre>Open a web browser on any device connected to the same local network: <code class="code">http://&lt;ARDUINO_UNO_Q_IP&gt;:5000</code><h3>Demo: </h3><figure><iframe src="https://www.youtube.com/watch?v=OMCTbtRUUI4"></iframe></figure><h3>System Architecture</h3>This assistant helps people with low vision identify grocery packages (like canned chickpeas or tomato sauce). It works entirely on the device without needing the internet. This guarantees zero delay and keeps your data completely private. <pre>+--------------------------------------------------------------------------+ </pre><pre>| Arduino UNO Q | </pre><pre>| | </pre><pre>| +---------------------------+ +--------------------------------+ | </pre><pre>| | STM32U585 (MCU) | | Qualcomm Dragonwing QRB2210 | | </pre><pre>| | | | (Debian Linux) | | </pre><pre>| | - Modulino Distance | | | | </pre><pre>| | | RPC | - UNIX Socket Bridge Client | | </pre><pre>| | - Arduino_RouterBridge | ----&gt; | - Stability Filter (25-50 cm) | | </pre><pre>| | service provider | | - Edge Impulse Model (.eim) | | </pre><pre>| +---------------------------+ | - Fallback RapidOCR Engine | | </pre><pre>| | - Flask Telemetry Streamer | | </pre><pre>| +--------------------------------+ | </pre><pre>+--------------------------------------------------------------------------+ </pre> 1. Dual-Core Hardware Integration Microcontroller (STM32U585): It constantly checks an Arduino Modulino distance sensor connected to the physical Qwiic port (Wire1 / I2C4 bus). It sets up an RPC method (get_distance) using the official Arduino_RouterBridge library. Microprocessor (Qualcomm Dragonwing): A light Python client asks for measurements directly through /var/run/arduino-router.sock using standard MessagePack-RPC, avoiding heavy graphical interfaces. 2. Distance &amp; Stability Filter To avoid blurry pictures and save computer power, the AI only works when you hold a product steady between 25 and 50 cm (250 mm to 500 mm) from the sensor for a few seconds. 3. AI Pipeline Primary Stage (Edge Impulse): It checks high-quality camera images using an Edge Impulse model (src/model.eim) saved on the device. This model finds the product and identifies it. Fallback Stage (RapidOCR): If the system cannot find the product, or if it is not very sure (less than 80% confidence), it sends the image to RapidOCR. This tool reads the text and brand names on the packaging. <h3>Edge Impulse Machine Learning Pipeline</h3>Edge Impulse Project: <a href="https://studio.edgeimpulse.com/public/1106136/live">https://studio.edgeimpulse.com/public/1106136/live </a>Model Architecture: YOLO-Pro (Float32 unoptimized for ARM Cortex-A) Dataset: 220+ images <a href="https://github.com/mr-umar/pantry-assistant/tree/main/Labelling">automatically</a>labeled with Gemini 3.5 Flash Lite Deployment: Compiled standalone .eim binary targeting Arduino UNO Q <h3>Hardware Bill of Materials</h3><ol><li>Arduino UNO Q (2GB or 4GB)* </li><li>Arduino Modulino Distance (ABX00102) </li><li>10cm Qwiic cable </li><li>Any USB Webcam (1080p compatible) ** </li><li>USB Type-C Power Supply - USB-C Hub for power and camera </li></ol>* The 2GB version of the UNO Q should be enough for this project. We used the provided Logitech Brio 105 with manual focus set for our needs. **You can adjust the focus of your webcam by turning the lens ring, as shown in the following image: <figure><img src="https://raw.githubusercontent.com/mr-umar/pantry-assistant/main/media/lens_adjust.gif"/><figcaption>Logitech adds glue to the ring, but you can remove it easily with isopropyl alcohol :)</figcaption></figure> <h3>Manual Step-by-Step Setup</h3>If you prefer to run each step yourself instead of using install.sh: 1. Install Debian packages <pre>sudo apt update </pre><pre>sudo apt install -y python3-pip python3-opencv python3-pyaudio portaudio19-dev python3-msgpack </pre>2. Install Python dependencies. (The following command installs them for the whole system. We recommend using a virtual environment instead). <pre>pip install -r src/requirements.txt --break-system-packages </pre>3. Flash the STM32 firmware. <pre>arduino-cli lib update-index </pre><pre>arduino-cli lib install "Arduino_Modulino" "Arduino_RouterBridge" </pre><pre>cd firmware </pre><pre>arduino-cli compile --fqbn arduino:zephyr:unoq -u . </pre><pre>cd .. </pre>4. Set permissions and run <pre>chmod +x src/model.eim run.sh </pre><pre>./run.sh </pre>5. (Optional) Remote Access with Tailscale If you want to connect to the assistant from a device outside your local network: <pre>curl -fsSL https://tailscale.com/install.sh | sh </pre><pre>sudo tailscale up </pre><h3>3D Case</h3>Check the 3D folder for the STL files if you need them. <h3>GITHUB REPO WITH CODE:https://github.com/mr-umar/pantry-assistant</h3> </body></html>

Autonomous pantry assistant for low-vision users

An autonomous edge-AI assistant that identifies pantry products and reads their labels aloud for people with low vision.