Segment anything

Flask (WSGI)

Type-A Usb Camera

Type-C to Type-A Usb Hub

OpenCV (cv2)

Ubuntu24.04

 Qualcomm Dragonwing™ IQ-9075 EVK  

Python  3.12

QNN  2.36

<html><body><h3>Summary</h3>This project utilizes the powerful NPU computing power of the Qualcomm Dragonwing™ IQ-9075 development board, combines YOLOv8 object detection with SAM (Segment Anything Model) fine-grained segmentation technology, and implements an automatic patrol scheme for robots in warehouse scenarios. The system can identify defects such as damage and stains on the surface of cardboard boxes in real-time, and visualize and store the data through a Web-based Dashboard, providing end-to-end AI visual detection support for smart warehousing. <figure><iframe src="https://www.youtube.com/watch?v=-R9vmSkpaew"></iframe></figure> <h3>The Story</h3>In modern smart logistics centers, the integrity of paper box packaging directly affects the safety of goods. Traditional manual inspections are inefficient and prone to fatigue. <figure><img src="https://projects.arduinocontent.cc/8cb7fdc9-88b6-409c-abf5-39bb1fbba682.jpg"/><figcaption></figcaption></figure> This project aims to deploy complex deep learning models (such as SAM) on edge-end devices. By using YOLOv8 to quickly locate targets, SAM then performs pixel-level defect segmentation, achieving high-precision real-time quality inspection tasks on low-power edge-end devices. <figure><img src="https://projects.arduinocontent.cc/a99ce702-0274-41e3-b9ef-d9035850cc38.gif"/><figcaption></figcaption></figure> <h3>Scene Description</h3>Simulate a warehouse scenario, prepare several brown paper boxes without printed characters on their surfaces, some with surface marks (damaged, dirty), and stack them on shelves. Move the IQ-9 camera to record the front of the boxes (with damage and dirt on the front), and transmit the recorded content in real time to the core program for surface mark detection. Then, the segmentation effect of the algorithm detecting surface marks can be viewed in real time on the web page. Paper box surface instance and expected detection results (damaged) <figure><img src="https://projects.arduinocontent.cc/8957716e-89e5-4ef2-baa4-31b4d41e9292.png"/><figcaption></figcaption></figure> Paper box surface instance and expected detection result (soiled) <figure><img src="https://projects.arduinocontent.cc/f9306bd3-1c34-4bec-b639-687513c4f484.png"/><figcaption></figcaption></figure> <h3>Main FunctionDescription</h3>IQ-9 connects via a USB Type-C converter to a USB Type-A camera device. <figure><img src="https://projects.arduinocontent.cc/bc0e6704-c024-40e4-b0d9-927b0b798fb3.png"/><figcaption></figcaption></figure> The detection program and web service program start up automatically. <figure><img src="https://projects.arduinocontent.cc/36fbae57-a745-4e08-85cd-047916f95cf9.png"/><figcaption></figcaption></figure> Users can access the specific port of the development board through a browser to view the real-time camera feed and check the detection results records of the camera. The web interface has the following functions: real-time camera image, detection result display box, start detection/stop detection button, detection frequency setting box (can set the detection interval frequency, for example: 1s interval), result reset button. When the user clicks the "&lt;Start Inspection&gt;" button, the camera is slowly moved on the cloud platform. The camera detects defects in real-time based on the image and transmits the inspection image to the camera frame in real-time, and reports the relevant results to the web page inspection result box. <figure><img src="https://projects.arduinocontent.cc/8c2082a1-cab5-4e7e-a7db-71e8eeb5cd4b.png"/><figcaption></figcaption></figure> When the shelf inspection is completed, click the "&lt;End Inspection&gt;" button. At this time, all defect reports during this cloud platform movement are displayed on the web page, and corresponding defect images can be viewed by selecting the corresponding records. <figure><img src="https://projects.arduinocontent.cc/d5a4e5a9-998c-41f6-81fe-d9dc51b6085a.png"/><figcaption></figcaption></figure> <h3>Core Workflow</h3>Object Detection: YOLOv8 identifies and generates a mask for the target object (e.g., a cardboard box). Region Extraction (Crop): The system utilizes the detection mask/bounding box to crop the target object from the original high-resolution frame. Defect Segmentation: The cropped image is passed to the SAM (Segment Anything Model), which focuses exclusively on the localized area to segment surface defects with high granularity. Visualization &amp; Storage: Final results are overlaid on the original feed, displayed on the Dashboard, and archived to the --save-dir. <h3>System Features</h3>Auto-start: The detection program and web service automatically start with the device. <div id="gallery"><img id="gallery" src="https://projects.arduinocontent.cc/d888d634-42cb-468c-a056-1d67e1bf571c.gif"/><figcaption></figcaption>,<img id="gallery" src="https://projects.arduinocontent.cc/b0c2e332-90db-4810-a3c4-2faa8806a564.gif"/><figcaption></figcaption></div> Remote Web Dashboard: Users can access the board's IP port via a browser. With the following functions: Real-time video stream preview. Detection frequency adjustment: Customize the detection interval (e.g., 1s per detection). <figure><img src="https://projects.arduinocontent.cc/b308efdd-8496-466b-ab5f-23c147951369.gif"/><figcaption></figcaption></figure> History query: Click the list on the right to view historical photos of detected defects. <figure><img src="https://projects.arduinocontent.cc/86d7af3a-d55b-4712-bab2-81eb2d9c930c.gif"/><figcaption></figcaption></figure>Control interaction: One-click start/stop inspection tasks. <h3>Troubleshooting</h3>1.QNN Environment Errors (Initialization Failure) Issue: The application fails to initialize models or throws errors related to missing <code class="code">.so </code>libraries. Cause: This is typically caused by missing or incorrect environment variables pointing to the QNN SDK dynamic libraries. Solution: Ensure that <code class="code">ADSP_LIBRARY_PATH </code>and <code class="code">LD_LIBRARY_PATH </code>are exported to the directory containing the QNN libraries before starting the application: <pre># Get the absolute path of the project directoryexport PROJ_ROOT=$(pwd)export LD_LIBRARY_PATH=$PROJ_ROOT/resources/qnn236:$LD_LIBRARY_PATHexport ADSP_LIBRARY_PATH=$PROJ_ROOT/resources/qnn236 </pre> 2. Camera Stream Warnings (JPEG Corruption) Issue: The console is flooded with warnings like Corrupt JPEG data: 2 extraneous bytes before marker 0xd5. Cause: This is a common issue with certain UVC cameras or low-level V4L2 drivers when the JPEG stream contains non-standard markers or padding bytes. Solution: While these warnings are often non-fatal and can be ignored if the feed looks normal, you can try changing the pixel format to YUYV in the source code if your camera supports it. Locate lines 54-55 in python/run.py and swap the commented lines as shown below: Original Code (Default): <pre>self.cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*'MJPG')) </pre><pre># self.cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*'YUYV')) </pre>Updated Code (YUYV): <pre># self.cap.set(cv2.CAP_PROP_FOURCC,cv2.VideoWriter_fourcc(*'MJPG')) </pre><pre>self.cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*'YUYV')) </pre> </body></html>

segmentanything

SegmentAnything

Segment anything

This project is featured in the Robotics Hub. Leverages the NPU capability of the IQ-9075 to implement robotic inspection for packaging box defects in warehouse scenarios.