Embedded Machine Learning for Person Detection

Jumper wires (generic)

Arducam Mini 2MP Plus – OV2640

Arduino Nano 33 BLE Sense Rev2 with headers

Arduino IDE

TensorFlow

<h3 >Deep Learning Workflow</h3><p >In this project, the steps followed to implement our person detection application are the book referenced [1]:</p><ul ><li >Decide the goal –person detection using Arduino Nano BLE 33 and Arducam</li></ul><ul ><li >Collect a dataset – VisualWake Words (derived from COCO dataset)</li></ul><ul ><li >Design a ModelArchitecture – Convolutional Neural Networks – MobileNet v1</li></ul><ul ><li >Train the Model – Training, validation and test</li></ul><ul ><li >Convert the Model –using TensorFlow Lite Converter</li></ul><ul ><li >Run Inference – Output data – Person Detected and Not detected scores, green/red light</li></ul><ul ><li >Evaluate andTroubleshoot – Real world performance </li></ul><h3 >Convert the Model</h3><p >In a normal environment, TensorFlow is used to build and train large Machine Learning models. A TensorFlow model is a set of instructions that tell an interpreter how to transform data in order to produce an output. To use our model, we load into the memory and execute it using the TensorFlow interpreter. </p><p >TensorFlow interpreter is supposed to be run in powerful desktop computers and servers. To run models on microcontrollers, we need a different interpreter for this case.TensorFlow has an interpreter and tools to run models on small-low system powered devices and it’s called “TensorFlow Lite”.</p><h3 >Run Inference</h3><p >After the model is converted, it can be deployed using the library “TensorFlow Lite forMicrocontrollers C++”. The code takes raw input data from the sensors and transforms it into the same form that the model was trained. This transformed data is passed to the model and run inference. </p><p >This result is an output containing predictions. In case of our “person detection” model, the output will be a score for each of our classes “person” and “no person”. These models that classify data, typically the scores for all classes will sum to 1, and the class with the highest score will be the prediction. The higher the difference between the scores, the higher the confidence in the prediction. In our application we are using a score from -100 to +100 </p><figure><img src=https://projects.arduinocontent.cc/ce49b30e-c7e1-4561-aa38-b932343f3857.png /><figcaption> </figcaption></figure><p >Each individual inference takes into account only a snapshot of the data, it tells us the probability of a person being detected within the last 1 second, based on the camera input image capture. </p><figure><img src=https://projects.arduinocontent.cc/4e668bd5-c030-4c42-a81d-0a945a743964.png /><figcaption>Higher "no person" score</figcaption></figure><h3 >Fundamentals - Person Detection</h3><p >When a person is detected, the hardware will light a green LED, when there is no person, a red LED will stay on. This can be extended for any other type of control. A person/not-person use-case is checking whether a person is present or not. Sensing person/not-person can be used in many applications including smart homes, retail, and smart buildings [2]. The inference result could be used in various IoT devices enabling other devices to start or trigger alerts. Generalizing, low-cost vision sensors can be deployed to sense the present of specific objects like pets in a home or cars in a garage</p><h3 >Visual Wake Words Dataset</h3><p >The Visual Wake WordsData set was obtained by re-labeling the images of the COCO dataset with labels to whether the object-of-interest is present or not [2]. COCO dataset is publicly available with natural images of everyday scenes that contain multiple objects [3]. The new labels were created based on the following criteria:</p><ul ><li >Label 1 – at least one bounding box corresponding to the object (e.g. person)</li></ul><ul ><li >Label 0 – not containing any objects from the class (e.g. person)</li></ul><p >The TensorFlow command used to generate new annotations can be modified to classify the presence of any object of interest from the COCO categories.</p><h3 >Application Architecture</h3><p >Our embedded machine learning application for person detection follow the sequence [1]:</p><ul ><li >- Obtain an input (camera)</li></ul><ul ><li >- Preprocess the input to extract features suitable to feed into the model</li></ul><ul ><li >- Run inference of the processed input</li></ul><ul ><li >- Post process the models’s output</li></ul><ul ><li >- Use the resulting information to act(light up different LEDs)</li></ul><p >Image data is represented as an array of pixel values. We obtain our image data from the embedded camera module (Arducam Mini 2MP Plus)that also provide data in this format so not so much preprocessing is required.It will require the conversion from colored image to grayscale. The application takes a snapshot of data from the camera, feeds into the model and finds out which output class was detected, then displays the results in the serial monitor and output LED.</p><p >The figure below shows the structure of the person detection application:</p><figure><img src=https://projects.arduinocontent.cc/1c2082d2-7f13-4bac-b564-c031b2aded19.png /><figcaption>Architecture of the person detection application</figcaption></figure><p >This person detection model uses the MobileNet architecture trained on the VisualWake Words dataset. The model takes a 96x96 pixel grayscale image as input, each image is a 3D tensor with shape (96, 96, 1). The final dimension (1)  is the value that represents a single pixel raging from 0 (black) and 255 (white). </p><p >The models outputs two probabilities ranging from -100 to +100: one indicating that a person was present in the input, and another that there was no person present. </p><p >The program is divided in five main parts:</p><ul ><li ><strong >Main Loop</strong><span >– embedded applications run in a continous loop</span></li></ul><ul ><li ><strong >Image provider</strong><span >– captures the image from the camera and writes in the input tensor</span></li></ul><ul ><li ><strong >TensorFlow Lite interpreter</strong><span >– runs the TensorFlow Lite model, trainforming the input image into the set ofprobabilities</span></li></ul><ul ><li ><strong >Model</strong><span > – is a data arrayand run by the interpreter </span></li></ul><ul ><li ><strong >Detection responder</strong><span >– takes the probabilities output and uses to display them</span></li></ul><h3 >Deploying to Microcontrollers</h3><p >In this session we will show how to deploy the code into the selected device Arduino Nano 33 BLESense. Arduino has a great variety of compatible third-party hardware and libraries and we used the Arducam Mini 2MPPlus [6] camera in this project. We connect the camera module to the Arduino board via the following pins:</p><figure><img src=https://projects.arduinocontent.cc/0726c326-531c-447c-a9a0-647088576f2c.png /><figcaption> </figcaption></figure><p ></p><figure><img src=https://projects.arduinocontent.cc/ffe1fe16-69ee-4e77-856b-5bad8350feff.png /><figcaption>Green LED - person detected</figcaption></figure><p ></p><figure><img src=https://projects.arduinocontent.cc/ae0bbe23-0442-4c70-be7e-b3766a1ed6f1.png /><figcaption>Red LED - no person detected</figcaption></figure><h3 >Acknowledgments</h3><p ><span > This project was presented for the course AIE-604 Deep Learning Applications by </span><a href="https://www.mangoud.com/">Dr. Mohab Mangoud</a><span >.</span></p><h3 >References</h3><p >[1] P. Warden and D. Situnayake, TinyML - Machine Learning withTensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers, O'Reilly, 2019.</p><p >[2] A. Chowdhery, P. Warden, J. Shlens, A. Howard and R. Rhodes, &#34;Visual Wake Words Dataset, &#34; Google Research, 2019.</p><p ><span >[3]    &#34;COCO - Common Objects in Context, &#34; [Online].  Available: </span><a href="https://cocodataset.org/#home.">https://cocodataset.org/#home.</a><span > [Accessed May 2022].  </span></p>

Embedded Machine Learning for Person Detection

Is easy to run large Machine Learning models in the cloud with large computational and memory resources. What about in the Arduino Nano?