Talk to Your Arduino: Build an On-Device AI Agent in Arduino App Lab

Modulino™ Buzzer

Arduino App Lab

Modulino™ Distance

Modulino™ Pixels

Arduino® UNO™ Q 4GB

Modulino™ Knob

Modulino™ Movement

Modulino™ Thermo

Modulino™ Buttons

<html><body><h3>Talk to Your Arduino: Build an On-Device AI Agent in Arduino App Lab</h3>We're going to build a tiny AI agent that runs entirely on an Arduino UNO Q. You type plain English, a language model living on the board itself works out what you mean, and the hardware responds. No internet. No API keys. And once it's running, new behaviour comes from a sentence, not a re-flash.We build the whole thing from a blank New App in Arduino App Lab, one piece at a time, so you understand every part, and I've signposted every pothole we hit so you sail past them.<blockquote>Just want to run it? Import the pre-built hatchagent-app.zip from the repo (see the shortcut at the end) and skip the building.</blockquote><h3>Why you'll want to finish this one</h3>Most "smart" projects ship a fixed set of buttons. Someone decided in advance what each one does, and the day you want something new, you go back to the code. Fun the first time, tedious the tenth.This flips that. We turn an app into an agent, something you talk to. Tell it "if I press A, turn the lights green," and it just does it. Tell it something else a minute later, and it does that too. Same board, no re-flashing. And the model runs on the board, on a chip you can close your fist around: no cloud, no bill, works on a mountaintop with zero signal.<h3>How it works (the 30-second version)</h3>The UNO Q is two computers on one credit-card-sized board:<ol><li>MPU: a full Linux machine (Qualcomm Dragonwing). Runs the language model and the "brain."</li><li>MCU: a real-time microcontroller. Drives the lights and reads the sensors, instantly.</li></ol>They talk over a built-in Bridge. The whole pipeline:The LLM never writes C++. It emits a small, safe "program", a mini language with a few verbs (turn this colour, blink, beep, when this sensor crosses that value). A rule engine on the Linux side runs it. The firmware stays a dumb-but-reliable "effects + sensors" device that never changes; all the cleverness lives in software that updates instantly. That's why new commands need zero re-flashing.<figure><img src="https://projects.arduinocontent.cc/0779ba39-5aa4-4763-b619-31f8cbe12260.png"/><figcaption></figcaption></figure><h3>Step 1 - Create a blank app in App Lab</h3><ol><li>Open Arduino App Lab and connect to your UNO Q (you'll see it bottom-left, e.g. myUNOQ (Arduino UNO Q)).</li><figure><img src="https://projects.arduinocontent.cc/45b960fa-45e3-427d-9f78-1c4ebbf7c8e9.png"/><figcaption></figcaption></figure><li>Click My Apps, then New App. Give it a name (we'll refer to it as &lt;AppName&gt; below) and pick an icon.</li><figure><img src="https://projects.arduinocontent.cc/b1512b6a-94f5-4459-a85d-19b1421d6033.png"/><figcaption></figcaption></figure><li>App Lab scaffolds a starter app: sketch/sketch.ino and sketch/sketch.yaml (the MCU side), python/main.py (the MPU side), app.yaml (the manifest), plus README.md and .gitignore.</li><figure><img src="https://projects.arduinocontent.cc/d37e2f44-d896-455d-a657-4830d2d1a9df.png"/><figcaption></figcaption></figure><li>On the left you'll also see three panels: Bricks, Sketch Libraries, and Files. We'll use all three.</li></ol><h3>Step 2 - The firmware (the MCU side)</h3><ol>Open sketch/sketch.ino in the editor. This is the "effects + sensors" device, deliberately simple so it's rock-solid. It plays one pixel effect at a time (solid / blink / pulse / off), fires sounds, and reads the sensors. Everything is exposed to the MPU as remote calls:</ol><pre>Bridge.provide("set_effect", rpc_set_effect); // mode 0=off 1=solid 2=blink 3=pulse + colour + rate </pre><pre>Bridge.provide("tone", rpc_tone); </pre><pre>Bridge.provide("melody", rpc_melody); </pre><pre>Bridge.provide("get_knob", rpc_get_knob); </pre><pre>Bridge.provide("get_distance", rpc_get_distance); </pre><pre>Bridge.provide("get_temp", rpc_get_temp); </pre><pre>Bridge.provide("get_buttons", rpc_get_buttons); // latched: never misses a tap </pre><pre>Bridge.provide("caps", rpc_caps); // which Modulinos are present </pre>Paste the full sketch.ino from the Code section below. Two design choices in there matter:<ol><li>Every sensor is read once per loop and cached. The Linux side reads those cached values instead of triggering live I2C, which keeps the sensor bus calm so fast events (a quick tap) aren't missed.</li><li>Buttons are latched and debounced (a press must show on two samples in a row), so noise can't fire phantom presses.</li></ol><figure><img src="https://projects.arduinocontent.cc/5fcd996c-922f-46f2-917a-74c81e1448ab.png"/><figcaption></figcaption></figure>We now need to update the sketch.yaml file which is read-only in the App Lab editor. So, you edit it from the terminal (the terminal icon, bottom-left):<code class="code">nano ~/ArduinoApps/&lt;AppName&gt;/sketch/sketch.yaml</code><figure><img src="https://projects.arduinocontent.cc/5d5133b2-0f18-4090-bb90-fd7563cfcfd2.png"/><figcaption></figcaption></figure>Paste the full library profile from the Code section, then save (Ctrl+O, Enter, Ctrl+X). (Or add each library from the Sketch Libraries panel.) When you Run later, the build log confirms each one.<figure><img src="https://projects.arduinocontent.cc/27b6e03e-36f3-48e6-8f05-9e897db26e0a.png"/><figcaption></figcaption></figure>The Distance Modulino's library starts its time-of-flight sensor with an inter-measurement period of 0, which makes it take one reading and freeze. You wave your hand and the number won't budge.One number fixes it. In the App Lab terminal, edit the library the build actually uses:<code class="code">nano ~/.arduino15/internal/Arduino_Modulino_0.8.0_*/Arduino_Modulino/src/Modulino.h</code>Find (around line 915) api-&gt;setRangeTiming(20, 0); and change it to:<code class="code">api-&gt;setRangeTiming(20, 100); // 100 ms = smooth 10 Hz streaming; 0 = stalls after one shot</code>Save. Now the sensor streams ~10x/second. Don't skip this thinking you'll deal with it later, later-you will be very confused.<figure><img src="https://projects.arduinocontent.cc/5d3d951f-7803-44c7-97c6-a33fdd26c2f0.png"/><figcaption></figcaption></figure><h3>Step 3 - The web UI</h3>App Lab serves the page through a ready-made brick.<ol><li>Add the web_ui brick. In the Bricks panel, add arduino:web_ui.</li></ol><figure><img src="https://projects.arduinocontent.cc/a6374ab9-4bb9-4f3e-b4bf-b9b9a7687995.png"/><figcaption></figcaption></figure><ol><li>Set the port. app.yaml is read-only in the editor, so use the terminal:</li></ol><code class="code">nano ~/ArduinoApps/&lt;AppName&gt;/app.yaml</code><code class="code"></code><figure><img src="https://projects.arduinocontent.cc/664d9423-381a-45db-83c3-e2d4d685f35a.png"/><figcaption></figcaption></figure>Make sure it has:<pre><code class="code">ports:</code> </pre><pre><code class="code"> - 7000</code> </pre><pre><code class="code">bricks:</code> </pre><pre><code class="code"> - arduino:web_ui</code> </pre><code class="code"></code><figure><img src="https://projects.arduinocontent.cc/648e8008-060c-4885-8570-584f2c474f85.png"/><figcaption></figcaption></figure><ol><li>Add the page. The web UI is three small text files plus one library. Create the `assets/` folder and the three files, then paste each from the Code section. The quickest reliable way is the terminal (paste, then save with `Ctrl+O`, `Enter`, `Ctrl+X`):</li></ol><pre>bash </pre><pre>mkdir -p ~/ArduinoApps/&lt;AppName&gt;/assets/libs </pre><pre>nano ~/ArduinoApps/&lt;AppName&gt;/assets/index.html # paste index.html </pre><pre>nano ~/ArduinoApps/&lt;AppName&gt;/assets/app.js # paste app.js </pre><pre>nano ~/ArduinoApps/&lt;AppName&gt;/assets/style.css # paste style.css </pre><figure><img src="https://projects.arduinocontent.cc/db6be46d-a38d-46f1-91a0-b0e61a7f731a.png"/><figcaption>index.html</figcaption></figure><figure><img src="https://projects.arduinocontent.cc/ca0a93b2-ccf9-46e3-94af-8b9597129ba9.png"/><figcaption>style.css</figcaption></figure><figure><img src="https://projects.arduinocontent.cc/90593ac0-a3b7-44a2-8c3c-f86207365db6.png"/><figcaption>app.js</figcaption></figure><ol><li>Then download the one file you can't type:</li></ol><pre>mkdir -p ~/ArduinoApps/&lt;AppName&gt;/assets/libs </pre><pre>curl -L -o ~/ArduinoApps/&lt;AppName&gt;/assets/libs/socket.io.min.js \ </pre><pre>	 https://raw.githubusercontent.com/hardikchadda/hatchagent/master/assets/libs/socket.io.min.js </pre><figure><img src="https://projects.arduinocontent.cc/d93aec1e-2ef6-4a65-b6de-671bf8e4e00b.png"/><figcaption></figcaption></figure><figure><img src="https://projects.arduinocontent.cc/6875920d-058c-4646-abf2-258fcf3dbbac.png"/><figcaption></figcaption></figure><h3>Step 4 - The agent (the MPU side)</h3>Open python/main.py and paste the full agent from the Code section. It serves the page, sends your text to the local model, and runs a rule engine that watches the sensors and fires effects.<figure><img src="https://projects.arduinocontent.cc/16ec0721-d7aa-434e-82aa-0b296908ba50.png"/><figcaption>main.py</figcaption></figure><pre>LLM_HOST = os.environ.get("HOST_IP", "127.0.0.1") # the model runs on the host, not in the app </pre><pre>LLM_URL = f"http://{LLM_HOST}:8080/v1/chat/completions" </pre><blockquote>Sneaky gotcha: some models are "thinking" models and reply with nothing unless you turn that off. The agent sends chat_template_kwargs: {enable_thinking: false} in every request. Leave it in.</blockquote><blockquote>Why HOST_IP and not localhost? App Lab apps run sandboxed in Docker and can't see the board's host, which is where the model lives. The app reaches it over the network via HOST_IP.</blockquote>The rule engine loops in the background: say "if the knob goes above 60, turn red," and the model returns a rule; the engine polls the cached sensor values and fires set_effect over the Bridge the moment it's true. New rules need no firmware change.<h3>Step 5 - Give it a brain (the language model), and keep it running</h3>App Lab runs your app; the model runs beside it on the board's Linux host. Set it up once, in the App Lab terminal.The easy way, one script:<code class="code">bash setup.sh</code><code class="code"></code><figure><img src="https://projects.arduinocontent.cc/d329556f-c8e2-4bf4-8e34-84770cfeab4a.png"/><figcaption></figcaption></figure><figure><img src="https://projects.arduinocontent.cc/2b999fa6-11cf-4167-af59-2c3d9187a972.png"/><figcaption></figcaption></figure>It installs llama.cpp, downloads the model, and serves it on port 8080. First run ~10-15 min; after that it's instant.Or do it by hand, so you see exactly what happens:<pre># 1. install llama.cpp (build tools + clone + build) - see setup.sh for the full commands </pre><pre> </pre><pre># 2. DOWNLOAD THE MODEL - a fine-tuned Qwen2.5-1.5B (GGUF, ~0.9 GB) from Hugging Face </pre><pre>mkdir -p ~/hatch-llm/models </pre><pre>curl -L -o ~/hatch-llm/models/hatch-q25-1.5b-Q4_K_M.gguf \ </pre><pre>	 https://huggingface.co/hardikchadda/hatch-agent-qwen2.5-1.5b/resolve/main/hatch-q25-1.5b-Q4_K_M.gguf </pre><pre> </pre><pre># 3. SERVE IT on port 8080 (the app reaches it via HOST_IP) </pre><pre>llama-server -m ~/hatch-llm/models/hatch-q25-1.5b-Q4_K_M.gguf --host 0.0.0.0 --port 8080 -c 2048 </pre><blockquote>Prefer the stock model? Swap the URL for https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF/resolve/main/qwen2.5-1.5b-instruct-q4_k_m.gguf. setup.sh falls back to this automatically if the fine-tuned model can't be reached.</blockquote>Important: the model server is a separate process from your app. The app does NOT start it.<ol><li>It must be running before you talk to the app. If you see Connection refused or urlopen error [Errno 111], the model server is down.</li><li>It does NOT survive a reboot. After a restart, the app comes back on its own, but the model server does not. Start it again with bash setup.sh (it skips the build and download, and just relaunches in seconds).</li></ol>Confirm it's up any time:<code class="code">curl -s -o /dev/null -w "model: %{http_code}\n" http://127.0.0.1:8080/health # 200 = ready</code><h3>Step 6 - Run it and talk to your board</h3>Hit Run (top-right). App Lab compiles the sketch, flashes the MCU, and starts your app. Watch the Python tab in the bottom panel for the Network URL (http://&lt;your-board-ip&gt;:7000).<figure><img src="https://projects.arduinocontent.cc/64924757-e282-4d11-b92a-56c230415cd2.png"/><figcaption></figcaption></figure><figure><img src="https://projects.arduinocontent.cc/b1c901b7-c572-4c6f-9405-bacf5fe09cd4.png"/><figcaption></figcaption></figure>Sanity-check the hardware in the terminal:<code class="code">python3 -c "from arduino.app_utils import *; print('caps =', Bridge.call('caps'))"</code>caps is a bitmask: 1=Buttons, 2=Buzzer, 4=Pixels, 8=Knob, 16=Distance, 32=Movement, 64=Thermo. All seven = 127. If it's low, a Qwiic cable is loose, reseat it and Run again.Open http://&lt;your-board-ip&gt;:7000, make sure the model server is running, and wait for the badge to read on-device, ready. Then try:<figure><img src="https://projects.arduinocontent.cc/3bdc8bae-bc19-4894-a179-b77248aa9dd5.png"/><figcaption></figcaption></figure><h3>How we taught it: the dataset</h3>A stock model trips over real phrasing. So we showed it thousands of examples, a synthetic dataset pairing everyday instructions with the exact "program" the board should run:<pre>{ "instruction": "if the knob is between 40 and 60, glow yellow", </pre><pre>	 "program": { "rule": { "src": "knob", "op": "between", "lo": 40, "hi": 60 }, </pre><pre>	 "effect": { "type": "solid", "color": "yellow" } } } </pre>The generator (gen_dataset.py) makes these by the thousand. We then LoRA-fine-tuned a small Qwen model on them; on held-out instructions it had never seen (new paraphrases, ranges, AND/OR logic, scenes), it scored 19/19. The trained model is published at huggingface.co/hardikchadda/hatch-agent-qwen2.5-1.5b, and setup.sh pulls it automatically. Fine-tuning is optional; the stock model works for the basics.<figure><img src="https://projects.arduinocontent.cc/435d657f-1f36-4157-9913-1e637df2baf3.png"/><figcaption></figcaption></figure><h3>Why this matters</h3>For years we assumed this kind of intelligence had to live in a data centre far away. Here it fits in your palm and runs on almost nothing, fully offline. An app hands you fixed buttons. An agent hands you a board you can talk to. You just built a tiny, working piece of that future.</body></html>

HatchAgent

hatchagent-app

Talk to Your Arduino: Build an On-Device AI Agent in Arduino App Lab

Type plain English, and a language model running on the board itself controls real hardware. No cloud, no API keys, fully offline.