The Protector - a life saving wearable

Arduino IDE 2.0 (beta)

 K-Way LE VRAI 3.0 CLAUDE jacket

PlatformIO

Nicla Sense ME

Edge Impulse Studio

The Protector

Visual Studio Code Extension for Arduino

<h3>Story</h3><p>Many of us like to go for a walk, whether it is to do some exercise, to shop at the market, or for a simple walk in the park, whatever our case may be, we are exposed to the weather conditions of the moment, if we walk in a prairie we probably don't have to worry about the quality of the air we breathe, but this is not the case in town, stores or even our office.</p><p>To take care of ourselves in these spaces that can have an impact on our health, I have created "The Protector", a device that is always with us, going almost unnoticed, but keeping a constant record of certain parameters that will allow us to know if we are in a safe place or If we are fine, these include temperature, humidity, atmospheric pressure, air quality, CO2 concentration, and even what we are doing at the moment, if we are standing still, walking, running, in a car, bicycle and last but not least, detecting if we have suffered a fall.</p><p>The project is accompanied by a custom-designed mobile application in which we will be able to see all the mentioned variables in real time, and also see the records of important moments such as exposure to highly polluted air, having been in very high places or the fact that we fell.</p><p>All this will be recorded with the exact time, the conditions and the place where it occurred.</p><h3>The Hardware</h3><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="PXL_20230108_004408653.jpg" data-file-size="2528570" data-index="16" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="2268,4032" src="https://projects.arduinocontent.cc/86bfc90e-98a3-441a-b3d4-7144d5ded29f.jpg" style=""/><figcaption spellcheck="false"><div>Close-up of the Nicla Sense ME on the K-Way jacket</div></figcaption><grammarly-extension class="cGcvT" data-grammarly-shadow-root="true" style="position: absolute; top: 0px; left: 0px; pointer-events: none; z-index: 2;"></grammarly-extension><grammarly-extension class="cGcvT" data-grammarly-shadow-root="true" style="mix-blend-mode: darken; position: absolute; top: 0px; left: 0px; pointer-events: none; z-index: 2;"></grammarly-extension></figure></div><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="PXL_20230127_201246641.jpg" data-file-size="244021" data-index="17" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="2175,1467" src="https://projects.arduinocontent.cc/fd942e0a-6e21-45ac-9fae-7aa618545a4f.jpg" style=""/><figcaption spellcheck="false">The mighty Nicla Sense ME</figcaption><grammarly-extension class="cGcvT" data-grammarly-shadow-root="true" style="position: absolute; top: 0px; left: 0px; pointer-events: none; z-index: 2;"></grammarly-extension><grammarly-extension class="cGcvT" data-grammarly-shadow-root="true" style="mix-blend-mode: darken; position: absolute; top: 0px; left: 0px; pointer-events: none; z-index: 2;"></grammarly-extension></figure></div><h3>How does it work?</h3><p>Using the integrated motion sensor unit (BHI260AP) in the Nicla Sense ME, the system identify automatically known activities:</p><ul><li>Standing still.</li><li>Walking.<br/></li><li>Running.<br/></li><li>Riding a bicycle.<br/></li><li>Being in a car.</li></ul><p>For custom activity detection as <strong>falling down</strong>, a motion recognition model on Edge Impulse was created for this specific case.</p><p>With the integrated pressure sensor (BMP390) the system is able to measure absolute atmospheric pressure and so, high altitude conditions.</p><p>Using the BME688 sensor, it can measure:</p><ul><li>Temperature.<br/></li><li>Humidity.<br/></li><li>Index of Air Quality.<br/></li><li>CO2 concentration.<br/></li></ul><p>All this data is sent to a smartphone through BLE and is shown on a dashboard, and if a fall detection or a low pressure condition is sent, the app will record the situation including other variables status and the exact location to then be reviewed. (I am trying to send an SMS or email after a fall is detected).</p><h3>Training the Edge Impulse model (*if you will create your own model)</h3><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="image.png" data-file-size="24673" data-index="0" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="1334,268" src="https://projects.arduinocontent.cc/c12d03c5-3ddd-4aee-bd57-001b633457d6.png" style=""/><figcaption><div>ESP32 device connected through USB waiting for Nicla sensor data</div></figcaption></figure></div><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="image.png" data-file-size="108050" data-index="1" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="790,397" src="https://projects.arduinocontent.cc/6656ea3f-8320-4526-8a98-8da4e86eb754.png" style=""/><figcaption><div>Falling event recorded</div></figcaption></figure></div><p><em><strong>​<br/></strong></em></p><p><em><strong>Remember that this model is trained mainly to detect falls.</strong></em></p><p>I developed a method to train my model with accelerometer data (x, y, z) from the Nicla Sense ME wirelessly, taking advantage of its BLE capability, for this I created a code that sends the data to an ESP32 board connected to the PC and using the Edge Impulse data forwarder, upload the samples directly to my model training dataset.</p><li>Clone <a class="se_link" href="https://github.com/mcmchris/NiclaSenseME-EI-Trainer" rel="noopener noreferrer" target="_blank">the code</a> for uploading your data to Edge Impulse</li><p>This was very important because that way I could record my movements without any cable connected to the Nicla board that could ruin my readings, and also allow me to record hard and sudden movements such as falls.</p><p>I recorded 10 seconds of samples falling and split them into 1-second windows.</p><p>My dataset consists on +2 minutes of each event:</p><ul><li>Walking.<br/></li><li>Standing still.<br/></li><li>Falling.<br/></li></ul><p><em><strong>This project is public on Edge Impulse so you can clone it, modify it and test it!</strong></em></p><h3>Deploy the model back to the microcontroller (*if you will modify mine or create your own)</h3><li>In the Deployment section, create an Arduino library, enabling the EON compiler (8 bits) and clicking on "<strong>Build</strong>":</li><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="image.png" data-file-size="24403" data-index="2" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="892,222" src="https://projects.arduinocontent.cc/07052159-855d-4311-b0c7-c5612de144f7.png" style=""/><figcaption><div>Arduino library with your trained model</div></figcaption></figure></div><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="image.png" data-file-size="66338" data-index="3" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="899,556" src="https://projects.arduinocontent.cc/8c7681a7-d489-46bf-8a84-0755a263194b.png" style=""/><figcaption><div>EON runner, will optimize the space and velocity of your model</div></figcaption></figure></div><ul><li>Unzip the downloaded library.</li></ul><p>💡 <em>Note: The Nicla Sense ME has limited (64 kB) SRAM, this will cause the board to crash due to the size of stack reserved memory by the ArduinoBLE and ArduinoBHY2 libraries (on Arduino IDE), after a week of trying to fix this problem, I suddenly thought in compiling the code with PlatformIO and this solved the problem (apparently PlatformIO compilation is more memory friendly).</em></p><ul><li>Set the <strong>PlatformIO</strong> environment to program the <a class="se_link" href="https://docs.platformio.org/en/latest/boards/nordicnrf52/nicla_sense_me.html" rel="noopener noreferrer" target="_blank">Nicla Sense ME board</a>.<br/></li><li>Clone the PlatformIO project attached from the <a class="se_link" href="https://github.com/mcmchris/TheProtector-PlatformIO" rel="noopener noreferrer" target="_blank">GitHub repository</a>.<br/></li><li>Copy the model library to the /lib folder of the project:<br/><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="image.png" data-file-size="17556" data-index="4" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="373,392" src="https://projects.arduinocontent.cc/f5e3046f-a7a2-4785-be73-0186c2b05081.png" style=""/><figcaption><div>PlatformIO project folder with the exported Arduino library added</div></figcaption></figure></div><br/></li><li>Build the code and flash the Arduino board:<br/><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="image.png" data-file-size="17784" data-index="5" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="753,157" src="https://projects.arduinocontent.cc/bced9e59-ebbb-47a5-9ad0-1bb7959a050b.png" style=""/><figcaption><div>Successful code compilation</div></figcaption></figure></div><br/></li><li>Let the battery be fully charged.<br/><br/></li></ul><h3>Mobile app download and setup</h3><li>Go to Play Store and download “<a class="se_link" href="https://play.google.com/store/apps/details?id=io.mcmchris4.starter" rel="noopener noreferrer" target="_blank">The Protector</a>” app.</li><li>Open it once downloaded.</li><p>As you can see, the dashboard and graphs are empty, to fill them with some data, let’s pair the project with the app.</p><li>In the right upper corner, touch on the BT icon, and select the device called “The Protector”.</li><li>After a few seconds, data will show up.</li><li>Hook the Nicla Sense ME to your K-Way jacket.</li><p>We are almost ready to go.</p><h3>Understanding the APP functions</h3><p><strong>Sensor tab:</strong></p><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="image.png" data-file-size="51068" data-index="6" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="305,633" src="https://projects.arduinocontent.cc/187b5936-3d67-45fd-b852-c01b24ddc547.png" style=""/><figcaption><div>First tab of the mobile app</div></figcaption></figure></div><ul><li>Let us pair the app with the hardware.<br/></li><li>Here we can see a live stream of the environment variables being measured by the Nicla Sense ME onboard sensors.<br/></li><li>Also, we can enable the graphs of each variable to compare them in time.</li></ul><p><strong>History tab:</strong></p><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="image.png" data-file-size="92629" data-index="7" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="303,627" src="https://projects.arduinocontent.cc/7bc09549-f1d8-4cd0-9cc1-b790d3579e47.png" style=""/><figcaption><div>Second tab of the mobile app</div></figcaption></figure></div><li>Here we can see our real-time location using the smartphone GPS.</li><li>We can see the Alert history.</li><li>If we select an Alert event, it will show the location where it occurred and the sensor data recorded in that moment.</li><p><strong>Settings tab:</strong></p><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="image.png" data-file-size="43250" data-index="8" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="303,626" src="https://projects.arduinocontent.cc/aaba0a37-4df6-4694-9784-dfbce372c4a8.png" style=""/><figcaption><div>Third tap of the mobile app</div></figcaption></figure></div><p>Here you can enable the notifications of harmful events like a bad IAQ, low pressure because of high altitude, or after falling.</p><p>You can enable the LED of the Nicla Sense ME so it can give you feedback of the air quality with a very simple color code.</p><ul><li>🟩Green - Good IAQ<br/></li><li>🟨Yellow - Slightly polluted air<br/></li><li>🟥Red - Highly polluted air.<br/></li></ul><p>We can delete the history logs of alerts.</p><h3>Real use case and demonstration</h3><p>This is me wearing "The Protector", let's go for a walk.</p><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="Yo.png" data-file-size="4048772" data-index="9" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="1920,1080" src="https://projects.arduinocontent.cc/abbf5ca0-cb6b-4293-8abb-53bf6ffa17ed.png" style=""/><figcaption><div>This is me, Christopher!</div></figcaption></figure></div><p>These are the readings on an open and surrounded by nature place.</p><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="NatureCel.png" data-file-size="3418543" data-index="10" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="1920,1081" src="https://projects.arduinocontent.cc/0d3ebb9f-f36b-40d0-9bfd-8289dcb27c63.png" style=""/><figcaption><div>Live view of the data (also look the green LED of the board)</div></figcaption></figure></div><p>These are the readings on a very polluted place (waste room).</p><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="WasteRoom.png" data-file-size="3837193" data-index="11" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="1920,1080" src="https://projects.arduinocontent.cc/423fe70d-4820-4d33-a573-28927b5d9d98.png" style=""/><figcaption>Very polluted air (look the red LED on the board)</figcaption></figure></div><p>Look how the app notifies me about the bad air quality and records an alert specifying the location, activity, and conditions.</p><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="AppWasteRoomRecor.png" data-file-size="246329" data-index="12" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="1185,1080" src="https://projects.arduinocontent.cc/11830fa9-db03-47d0-b1dd-28e89fd331b7.png" style=""/><figcaption>Alert recorded and notification showed</figcaption></figure></div><p>If I suddenly fall, the app will prepare a WhatsApp message for my emergency contact.</p><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="Falling.png" data-file-size="3413213" data-index="13" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="1080,1404" src="https://projects.arduinocontent.cc/60f331b0-9e26-4683-8461-459de4426710.png" style=""/><figcaption><div>After a fall, WhatsApp will automatically open and you just need to tap on send</div></figcaption></figure></div><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="FallingApp.png" data-file-size="1301147" data-index="14" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="935,791" src="https://projects.arduinocontent.cc/307fdab4-ec75-4270-b15c-38ecf2ab14e6.png" style=""/><figcaption><div>After a fall, I will be able to see where, the conditions and activity.</div></figcaption></figure></div><p><br/></p><p>It can detect normal activities like walking, running, riding a bicycle and being in a car.</p><div class="se-component se-image-container __se__float-none" contenteditable="false"><figure style="margin: 0px;"><img alt="" data-align="none" data-file-name="ActivityDetect.png" data-file-size="4568333" data-index="15" data-origin="," data-percentage="auto,auto" data-rotate="" data-rotatex="" data-rotatey="" data-size="," origin-size="3040,3040" src="https://projects.arduinocontent.cc/98008fc3-0a8e-4c82-82f1-a444050122dd.png" style=""/><figcaption><div>Internal IMU sensor inferences about the activity</div></figcaption></figure></div><h3>Video demo</h3>

Nicla Sense ME code

Edge Impulse Wireless Data Uploading

The Protector - a life saving wearable

Air quality monitoring, activity tracker, configurable alerts, AI-powered fall detection, long-lasting battery, and a great app to track and visualize everything through BLE.