Digital twin-enabled Smart Shipping Workstation w/ Omniverse

PrusaSlicer

KiCad

Raspberry Pi 5

Tiny (Embedded) Thermal Printer

Soldering iron (generic)

NVIDIA Omniverse

SparkFun Logic Level Converter - Bi-Directional

Edge Impulse Studio

ELECROW Custom PCB

OpenCV

MIT App Inventor 2

5 mm Common Anode RGB LED

SSD1306 OLED Display

Autodesk Fusion 360

USB Webcam (PK-910H)

Button (6x6)

Hot glue gun (generic)

VS Code

Thonny IDE

A4988 Driver Module

Infrared (IR) Break-beam Sensor (300 mm)

Arduino Nano Matter

Nema 17 (17HS3401) Stepper Motor

Micro Switch with Pulley (KW10-Z5P)

Arduino IDE

Raspberry Pi Raspbian

<html><body>At the pinnacle of industrial artificial intelligence and machine learning applications, a digital twin represents a virtual construction of real-world physical products, mechanisms, or mechanical procedures. Since the simulation of a real-world product or industrial technique provides the flexibility of exerting countless examination scenarios, even to the point of cycling arduous or dangerous tasks, without causing any tangible ramifications and safety risks, digital twins hold immeasurable importance in developing adaptive product manufacturing procedures and building secure, cost-effective, and efficient industrial facilities. After inspecting recent research papers about the applications of digital twins in industrial operations, I noticed the focal point of employing a virtual representation is to improve the safety and efficiency of an already existing industrial facility or mechanical process. Even though forestalling acute physical hazards due to dangerous work safety risks and advancing the precision of ongoing industrial operations are the prominent digital twin use cases, I wanted to explore the innovative opportunities of reversing the digital twin implementation and starting with a virtual industrial construction, consisting of individual machinery and sample product components, to develop a safe, practical, cost-effective, and efficient real-world mechanism from scratch. By reversing the digital twin application process, I wanted to investigate whether having a virtual construction before building the real-world counterpart could help to forfend concomitant risks of assembling an industrial manufacturing system, reduce exorbitant overhaul costs due to the lack of internal design blueprints, and test device components to obtain the optimum performance for multifaceted operations. As I was conceptualizing this proof-of-concept project, I inspected various industrial settings with which I could show the benefits of reversing the digital twin implementation. Since product transportation and shipping operations require complex industrial mechanisms to achieve accuracy and reliability while maintaining a time-sensitive workflow, I decided to apply my reverse digital twin approach to design a virtual shipping workstation, construct a synthetic data set of customized sample products, and train a precise object detection model to accomplish building a production-ready product transportation mechanism. In accordance with my approach, I designed all sample products from scratch to emphasize the strengths of a full-fledged digital twin, providing the opportunity to train an object detection model for products waiting to be manufactured. Since I needed to know the exact electronic components employed by the shipping workstation to create compatible 3D parts, I decided to prototype the mechanical device structure and design a unique PCB (inspired by Wall-E) based on Arduino Nano Matter as the workstation control panel. I designed the Wall-E PCB outline and encasement on Autodesk Fusion 360 to place the electronic components in relation to the PCB easily while designing the virtual shipping workstation. After testing electronic components and completing the PCB layout, I designed a plethora of 3D parts on Autodesk Fusion 360, including but not limited to custom bearings optimized for 5 mm steel balls, planetary gear mechanisms, and separated rotating platforms. After finalizing the required mechanical 3D parts, I exported the virtual shipping workstation as a single file in the OBJ format to produce an accurate virtual representation of the shipping workstation. Then, I imported the virtual shipping workstation into the NVIDIA Omniverse USD Composer, which allows users to assemble, light, simulate, and render large-scale scenes for world-building. To generate a realistic scenery for shipping operations, I utilized some free 3D models provided by Omniverse and designed some additional assets. After completing my shipping warehouse scenery, I experimented with camera, material, and lighting configurations to render the virtual shipping workstation with exceptional quality and produce a precise digital twin. After employing built-in NVIDIA Omniverse features to construct my synthetic data set of customized sample products as instantiated by the shipping workstation digital twin, I uploaded the collected samples to Edge Impulse to train an advanced object detection model (FOMO) with synthetic product images. After validating and testing my FOMO model, I deployed it as a Linux (AARCH64) application (.eim) compatible with Raspberry Pi 5. After building my object detection model successfully and completing my assignments with the shipping workstation digital twin on NVIDIA Omniverse, I started to print all workstation 3D parts to assemble the real-world counterpart. To create a fully functioning smart shipping workstation with state-of-the-art features, I developed a web application from scratch to manage the MariaDB database server hosted by Raspberry Pi 5, run the Edge Impulse FOMO object detection model, and transfer the detection results with the modified model resulting images. I also developed a mobile application (Android) operating as the workstation interface and the proxy between the workstation control panel (based on Arduino Nano Matter) and the web application. So, this is my project in a nutshell 😃 🤖 To review code and design files with thorough instructions, you can check the project's GitHub page or the project tutorial on Hackster: <a href="https://github.com/KutluhanAktar/Digital-twin-enabled-Smart-Shipping-Workstation-w-Omniverse">https://github.com/KutluhanAktar/Digital-twin-enabled-Smart-Shipping-Workstation-w-Omniverse </a><a href="https://github.com/KutluhanAktar/Digital-twin-enabled-Smart-Shipping-Workstation-w-Omniverse"> </a><a href="https://www.hackster.io/kutluhan-aktar/digital-twin-enabled-smart-shipping-workstation-w-omniverse-049792">https://www.hackster.io/kutluhan-aktar/digital-twin-enabled-smart-shipping-workstation-w-omniverse-049792 </a> <figure><iframe src="https://youtu.be/ewXsc-yktX4"></iframe></figure> <figure><iframe src="https://youtu.be/hGNoAyI7vjU"></iframe></figure> <figure><img src="https://projects.arduinocontent.cc/f14667a6-06cd-473b-9a9b-a79acbd3f4da.png"/><figcaption></figcaption></figure><figure><img src="https://projects.arduinocontent.cc/6f13afd7-4b91-4aad-8391-299a8593c382.png"/><figcaption></figcaption></figure><figure><img src="https://projects.arduinocontent.cc/0a7a08f6-061f-4876-87ba-dcbcc78d13f5.jpg"/><figcaption></figcaption></figure><figure><img src="https://projects.arduinocontent.cc/63e7ab25-c523-49cf-a2ed-a43610841d69.jpg"/><figcaption></figcaption></figure><figure><img src="https://projects.arduinocontent.cc/aa1e98d2-9a54-42b7-9c83-f60275bad0f2.jpg"/><figcaption></figcaption></figure><figure><img src="https://projects.arduinocontent.cc/a029b398-221a-4d08-b1bf-a8a1e39ec005.jpg"/><figcaption></figcaption></figure><figure><img src="https://projects.arduinocontent.cc/d43b1a38-4fec-4d78-814b-2c7dd6994136.jpg"/><figcaption></figcaption></figure><figure><img src="https://projects.arduinocontent.cc/5e8572c7-6add-49a4-abe1-112452fadc04.jpg"/><figcaption></figcaption></figure><figure><img src="https://projects.arduinocontent.cc/bee5db0b-b9d4-43aa-a30d-f8deb804c84b.jpg"/><figcaption></figcaption></figure><figure><img src="https://projects.arduinocontent.cc/61fc7118-d7e6-4425-9ace-e521ee5ddeb1.jpg"/><figcaption></figcaption></figure><figure><img src="https://projects.arduinocontent.cc/859e6128-28f6-4bb2-8bb4-8831ef67ca25.jpg"/><figcaption></figcaption></figure><figure><img src="https://projects.arduinocontent.cc/e7c04a20-9f91-42be-84bc-abc7e65a726a.jpg"/><figcaption></figcaption></figure> </body></html>

Code Files

Project Tutorial

Digital twin-enabled Smart Shipping Workstation w/ Omniverse

Exploring the digital twin synthetic data generation and AI-oriented advancements on real-world shipping operations w/ NVIDIA Omniverse.