Smart Jacket for Fall Detection: A Human Activity Recognition (HAR) Application for Healthcare

Nicla Sense ME

Edge Impulse Studio

Arduino IDE 2.0 (beta)

<html><body><h3>Introduction</h3>Human Activity Recognition (HAR) refers to the identification of the physical activities performed by individuals based on sensor data collected from devices such as smartphones, smartwatches, or other wearable devices. These activities can include simple movements like walking, sitting, or running, as well as more complex activities like playing sports or using tools. The importance of Human Activity Recognition lies in its potential to assist in a variety of applications and fields, such as: Healthcare: HAR is used to monitor the health and physical activity of patients, particularly elderly people or those with physical disabilities, and to assess their physical and mental well-being. Sports and fitness: In this field, HAR is used to track and analyze the performance of athletes, providing valuable insights into their training and physical fitness. Security and surveillance: HAR is used in security and surveillance systems to detect and respond to security threats, such as identifying and tracking individuals who may pose a risk to public safety. Marketing and advertising: By analyzing the activities and behaviors of individuals, HAR can help companies to better understand consumer habits and make informed decisions about product development and marketing strategies. Transportation: HAR can be used in transportation systems to monitor the activity levels of passengers, for example, to optimize the use of resources, improve safety, and reduce the risk of accidents. Human-Computer Interaction (HCI): In HCI, HAR is used to develop systems that can recognize and respond to the gestures and movements of users, making it easier to interact with computers and other digital devices. Robotics: HAR can be used in robotics to help robots understand and respond to human actions and movements. Overall, Human Activity Recognition has the potential to revolutionize the way we monitor and respond to human behavior, providing valuable insights and enabling new applications across a wide range of industries and domains. <h3>Human Activity Recognition (HAR) Applications in Healthcare</h3>HAR has a number of important applications in the healthcare field. Some of the most significant applications include: Monitoring and assessment of physical activity: By monitoring the activity levels of patients, healthcare providers can assess their physical and mental well-being and develop personalized treatment plans. Fall detection and prevention: HAR can be used to detect and prevent falls in elderly patients, helping to reduce the risk of injury and improve their quality of life. Rehabilitation: By tracking the progress of patients undergoing physical rehabilitation, healthcare providers can evaluate their progress and adjust their treatment plans accordingly. Chronic disease management: For patients with chronic diseases such as diabetes or heart disease, HAR can be used to monitor their activity levels and provide early warning signs of potential health problems. Behavioral and mental health: By analyzing the activity patterns of patients, healthcare providers can gain valuable insights into their behavioral and mental health, helping to diagnose and treat conditions such as depression or anxiety. Elderly care: By monitoring the activity levels of elderly patients, healthcare providers can assess their physical and mental well-being and provide timely support and care when needed. Overall, Human Activity Recognition has the potential to revolutionize the way we monitor and respond to the health and well-being of patients, providing valuable insights and enabling new approaches to healthcare. <h3>How HAR can be used in Fall Detection and Prevention?</h3>Fall detection and prevention is one of the key applications of Human Activity Recognition (HAR) in the healthcare field. Falls are a common and serious problem for elderly people, and can result in serious injury or death. HAR can be used to detect and prevent falls in several ways: Wearable devices: By using wearable devices such as smartwatches or fitness trackers, elderly individuals can continuously monitor their activity levels and be alerted if they experience a sudden change in movement that suggests a fall has taken place. Ambient sensors: Ambient sensors such as accelerometers and gyroscopes can be placed in the home to monitor the movements and activity levels of elderly individuals. These sensors can be used to detect falls and alert care providers or family members in real-time. Machine learning algorithms: By analyzing data collected from wearable devices or ambient sensors, machine learning algorithms can be used to detect falls and distinguish them from other movements and activities. Predictive models: Predictive models can be developed to identify individuals at high risk of falling and provide targeted interventions to reduce the risk of falls. By using these and other techniques, Human Activity Recognition has the potential to significantly improve the ability of healthcare providers to detect and prevent falls in elderly individuals, helping to improve their quality of life and reduce the risk of serious injury. <h3>About This Project</h3>This project is submitted for Arduino x Kway Project competition. The hardware for this project, the Nicla Sense ME, is provided by Arduino. The jacket is provided by <a href="https://www.k-way.com">K-Way </a>. In this project Nicla Sense ME along with K-Way Jacket is used to create smart jacket to detect the fall detection. To built the system, the sensor node is embedded in the jacket. Nicla Sense ME is used for the sensor node, which allows inference to be done on device and results to be sent through BLE. Nicla Sense ME has an accelerometer that detects changes in velocity and hence detects falls. The fall is detected with the help of TinyML model built using <a href="https://edgeimpulse.com/">Edge Impulse </a>Studio. The following image show the jacket setup for data collection and testing. <figure><img src="https://projects.arduinocontent.cc/87e246cf-10a8-49e4-97b0-73633a462529.jpg"/><figcaption></figcaption></figure><h3>Methodology</h3>A built in accelerometer is used to measure acceleration, or changes in velocity, in one or more directions. The accelerometer is used to detect and analyze the movements and activity levels of individuals, in order to identify and respond to falls. When a fall occurs, the accelerometer measures a sudden change in velocity, which can be used to identify the event as a fall. This data can then be analyzed using embedded machine learning algorithm built with the Edge Impulse Platform to determine if the fall has occured and if medical attention is needed. By continuously monitoring the activity levels of individuals, accelerometers can provide real-time information about falls and enable early intervention to prevent serious injury. Overall, this application may play a critical role in fall detection and prevention, providing valuable insights into the movements and activity levels of individuals and enabling healthcare providers to respond quickly and effectively to falls. The graphic below depicts the entire system architecture. When a fall is detected using the TinyML model, the data is transferred to the gateway device over BLE. The gateway then sends the data to an IoT cloud server via the MQTT protocol, and the healthcare professional receives the alarm through a web app and takes necessary action. <figure><img src="https://projects.arduinocontent.cc/67da1bd7-e2c6-4456-89ec-d1bb4c76cbf2.png"/><figcaption></figcaption></figure>The accelerometer dataset for this project is collected for the following classes: <ol><li>idle </li><li>walking </li><li>falling </li></ol>The TinyML model is built with 99% accuracy. <figure><img src="https://projects.arduinocontent.cc/86832a05-f0cc-4f34-8972-4dcc96b2ae67.png"/><figcaption></figcaption></figure>The model is then deployed as Arduino library and uploaded on the Nicla. In the following video demo, the working of TinyML model for fall detection is shown. The model is deployed on device and tested with device connected via wired connection. <figure><iframe src="https://youtu.be/rPjd4o8PK3U"></iframe></figure> <figure><iframe src="https://youtu.be/yRhpk_RCUn0"></iframe></figure>From the videos above it is seen that the tinyml model is working fine. Now, its time to deploy the full application. The sensor node, as seen in the following figure, uses battery power for operation and bluetooth for data transmission. <figure><img src="https://projects.arduinocontent.cc/eb0ab700-3078-42ce-b348-ecc5f58d249b.jpg"/><figcaption></figcaption></figure>Visual Studio Code IDE and PlatformIO extension is used to build the firmware for the Nicla Sense ME sensor node. <figure><img src="https://projects.arduinocontent.cc/81af6289-75c9-4420-a589-ed2178c698b5.png"/><figcaption></figcaption></figure>The firmware code is in the main.cpp file as shown below.<figure><img src="https://projects.arduinocontent.cc/e3e7e97b-bc0f-4b9e-8817-834f209ef49f.png"/><figcaption></figcaption></figure>The data from sensor node is published via bluetooth to the gateway node. The gateway node is implemented using Raspberry Pi Model B 3 (RPi3). The RPi3 uses built-in bluetooth modeule to receive data and WiFi module to transmit data further. The python script executing on RPi3 does this task as it detects and connect to sensor node, fetch inference data and transmit to MQTT broker <a href="http://www.mqtt-dashboard.com/">broker.hivemq.com </a>via WiFi signal. The <a href="https://pypi.org/project/paho-mqtt/">Paho-MQTT library for python </a>is used in python script. The web app in JavaScript is built which receives the data from MQTT broker and generates alert/display activity status. In JavaScript code the <a href="https://www.eclipse.org/paho/index.php?page=clients/js/index.php">Paho-MQTT library for JavaScript </a>is used. The following image show python script executing on gateway device. <figure><img src="https://projects.arduinocontent.cc/caef6daf-3afb-46eb-8dea-cb32002d21f3.png"/><figcaption></figcaption></figure>The status of the activity is displayed on web app as shown in following figure. <figure><img src="https://projects.arduinocontent.cc/c5fa57d9-a91f-4f8f-aac6-82db42fa45cb.png"/><figcaption></figcaption></figure>The final video of the project show how the Fall is detected and reported on web application. <figure><iframe src="https://youtu.be/lXSnFvxj4oM"></iframe></figure>The public Edge Impulse project is available <a href="https://studio.edgeimpulse.com/public/184891/latest">here </a>. *Declaration: Some content in this project are edited using ChatGPT. </body></html>

Smart Jacket for Fall Detection: A Human Activity Recognition (HAR) Application for Healthcare

This project is used to monitor patient's activity and alerts if fall is detected. This project is submitted for Arduino x Kway Project competition.