Anti-stalking device with thermal camera using computer vision (CEC 2022)

Spring '22

technical skills: Electronics Design, Machine Learning, Mechanical Engineering

Canadian Engineer Competition (CEC) is an annual competition held by Canadian Federation of Engineer Student. It brings together over 200 of the most innovative and creative engineering undergraduate students from across Canada to compete against each other in eight categories (Junior Design, Senior Design, Innovative Design, Engineering Consulting, Parliamentary Debate, Re-engineering, Programming, and Engineering Communication).

The concept for the design is simple: we capture sensors inputs from the thermal camera, then perform object detection on the captured imagery with machine learning, and if suspicious activities are detected, alert the user via the app.For the sensor inputs, the device uses a thermal-camera and an ultrasonic sensor. Object detection is performed using a TinyML algorithm, and the forms of notifications for suspicious activities include phone vibrations, ringtones, and popup windows. The picture below also showed our prototype that use for gathering our own data as well. One thing is that we use an Arduino Portenta for the prototype instead of Arduino 33BLE in our actual model due to out of stock at that time.

I worked on designing the electrical system of the project as well as developing the workflow for the firmware. This involved sourcing required electronics (camera, MCU, other sensors,) to make sure we got optimal parts at minimum budget (we only receive $300 after winning 1st prize in WEC – regional competion before CEC).

Beside the electrical aspect, I also work on building the ML system. Our project utilized TinyML from Edge Impulse which is a ML framework for microcontroller-based system. For the camera picture, the camera gave an matrix of 24x32 pixels but we translate from 2D to 1D array. This provided less computationally intensive than convetional option, which require less-preprocessing for our system. This feature would save us from the cost as we do not have to upgrade our controller to train the ML algorithm effectively.

Our ML model use a Neural Network as follow. This model architecture was chosen because it performed best amongst other model variations at predicting the test set. The input is the 1D thermal array then goes through three different dense layers 80, 40, then 20 neurons respectively. Dense layers mean that all the different neurons are fully connected. Training accuracy is 75.7% which represents the fraction that the predictions of the model got right. These are also the training model’s on-device performance.

Upon testing it on the test dataset, we received 70% accuracy in general with the confusion matrix as follow. “Human” means that our model “see” that there is a person approaching where as “Background” means no human.

Finally, here are the demos for our project. The first video shows the "confidence level" whether the object is human or not. As the person is approaching closer, the confidence increases significantly. The second video demonstrate the visualization of our thermal camera as a person is approaching.