According to statistics from the World Health Organization, there are 2.2 billion people with visual impairment in the world, including 285 million visually impaired people and 39 million completely blind people. Moreover, this number will continue to increase as the aging population intensifies.
Although the visually impaired people face many inconveniences, the assistive devices for them have problems such as high cost, difficult maintenance, and complex operation, making it difficult to meet their life needs.
To this end, researchers from Guangdong Polytechnic Normal University and Wuhan University of Science and Technology jointly developed a set of visually impaired assistance devices based on the smart Internet of Things, bringing convenience to the lives of the visually impaired through AI algorithms and sensors.

Author | Xuecai

Editor | Three Sheep, Iron Tower

This article was first published on HyperAI WeChat public platform~

In 2019, the World Health Organization (WHO) released the first "World Vision Report". The report stated,There are at least 2.2 billion visually impaired people in the world, of whom 285 million are visually impaired and 39 million are completely blind.Due to global population growth and aging, the number of visually impaired people may triple by 2040.

Figure 1:Estimated number of visually impaired people worldwide

Visually impaired people cannot receive information from the outside world through their visual system and face many inconveniences in their daily lives. It is difficult for them to avoid nearby obstacles and they cannot get information from books or screens.

However, long-term health care for the visually impaired is very costly.However, existing assistive devices for the visually impaired are expensive, bulky and have poor interactivity., and it is difficult to meet their daily needs.

To solve the above problems,Researchers from Guangdong Polytechnic Normal University and Wuhan University of Science and Technology have developed a set of visually impaired assistive devices based on the Artificial Intelligence of Things (AIoT)..

This set of equipment mainly includes smart glasses and a smart cane, with a total cost of about 480 yuan.It can monitor the status of users and the environment through a variety of sensors, help users interact with the environment and issue alarms in dangerous situations. This result has been published in "Electronics".

This result has been published in "Electronics"

Paper link:

https://www.mdpi.com/2079-9292/12/18/3760

Equipment Design

AIoT Architecture: Perception and Interaction

The AIoT architecture of visually impaired assistive devices consists of 3 layers:

1.Perception Layer, i.e. sensors and modules that collect external data;

2.Grid Layer, uses Narrowband Internet of Things (NB-IoT) for data connection and HTTP and MQTT protocols for data transmission;

3.Application Layer, mainly interacting with users through APP and Bluetooth audio devices.

Figure 2: AIoT architecture diagram

Smart glasses:YOLO v5 + OCR

Visually impaired assistive devices should be both convenient and reliable to meet the daily needs of the visually impairedTo do this, the researchers collected images using a stereo camera, and then analyzed objects and their distances using the lightweight grid model YOLO v5, or read text using optical character recognition (OCR).

Figure 3:Schematic diagram of the structure of smart glasses

During object recognition,The YOLO v5 algorithm divides the image into a grid, then predicts the bounding box for each grid cell and identifies different objects in the image while creating the bounding box..

The YOLO v5 grid is mainly divided into 4 layers, including:

1. input: Used to initially define the input image;

2. backbone：Use convolutional neural network (CNN) to extract image features;

3. neck: Integrate image features and connect the backbone layer and output layer;

4. output: Prediction of objects in an image using bounding box input.

Figure 4: YOLO v5 algorithm architecture in this study

In addition to image recognition, the YOLO v5 algorithm can also calculate the distance between the user and obstacles by comparing images from the binocular camera.

Figure 5: YOLO v5’s method for calculating obstacle distance

Figure 6: YOLO v5 formula for calculating obstacle distance

In the formula, B is the distance between the left and right lenses of the camera, f is the focal length of the camera, and d is the parallax between the corresponding left and right points.

The OCR function of smart glasses is realized by calling Baidu OCR API. The OCR process includes image acquisition, noise reduction, binarization, text area extraction, character segmentation, character recognition, optimization and audio output.

Figure 7: Schematic diagram of the OCR process

Smart Cane:Health Monitoring and Environmental Sensing

One of the main functions of smart canes is health monitoring.Therefore, sensors such as heart rate and body temperature and data collection modules are integrated into the cane.To achieve contactless measurement, the heart rate sensor uses photoplethysmography (PPG) for monitoring, and the body temperature sensor uses infrared radiation for monitoring.

Figure 8: Schematic diagram of the structure of the smart cane

In addition, to enhance the visually impaired's perception of the environment,The cane also integrates temperature, humidity and posture sensorsThe posture sensor consists of an accelerometer and a gyroscope, and monitors whether the user is at risk of falling by calculating the user's combined acceleration (ACLR).

If the user is too close to an obstacle or the posture sensor detects that the user may fall, the cane will sound an alarm through vibration and buzzer.

Figure 9: 3D coordinate system for fall detection

Experimental Results

Image Recognition:Objects and distance

The researchers conducted 20 tests on the device’s object recognition capabilities in common indoor and outdoor environments. The results showed thatThe YOLO v5 algorithm can accurately identify objects in the environment and judge the distance between the objects and the user in both indoor and outdoor environments.

Figure 10: Test results in indoor environment

Figure 11: Test results in outdoor environment

However, the error rate increases as the number of objects in the image increases, especially for objects with similar features, such as a TV and a monitor, or a bicycle and a motorcycle.Finally, the recognition accuracy of the smart glasses was 92.16%.

Even if the YOLO v5 algorithm misidentifies an object, it can still make a judgment about the distance.The deviation rate of the YOLO v5 algorithm in judging the distance is between 0.28-6.32%, increases with the distance.

Based on this judgment, when the distance between the user and the object is less than 0.7 m, the smart cane will sound an alarm and take over the microcontroller to avoid obstacles.

OCR:The accuracy is close to 100%

At the same time, the researchers tested the OCR performance of the smart glasses with 10 paragraphs of text. Under the test, the OCR accuracy of the smart glasses was 100%. Although the movement of the head during reading can cause some recognition errors,But the overall OCR accuracy is still 99.91%.

Figure 12: OCR results from smart glasses

Health Monitoring: Heart rate and body temperature

The heart rate monitoring module records the user's heart rate in real time through PPG. During the test, the smart cane recorded the heart rates of subjects of different ages when they were stationary and walking.The deviation rate between the results and commercial wristbands is between 0.72-3.52%.

Table 1: Heart rate monitoring results of different experimental groups

In temperature testing, compared with medical forehead thermometers,The maximum deviation rate of the test module is 0.19% This result shows that even on a smart cane, the infrared body temperature sensor can accurately monitor the user's body temperature without being disturbed by the environment.

Table 2: Body temperature monitoring results of different experimental groups

Posture monitoring:Walking, going up and down stairs

Smart canes can monitor the user's posture through accelerometers and gyroscopes and issue an alarm if a fall is likely.The experiment tested the monitoring results of the cane when the user was walking, going up and down stairs.The results are divided into 3 categories:

1. Correct: The monitoring result of the cane is consistent with the user's status;

2. Incorrect: The cane detected a risk of falling, but the user was still in a normal state;

3. No response: The cane displays normal status, but the user is at risk of falling.

Table 3: Fall detection accuracy under different experimental conditions

In different scenarios,The average monitoring accuracy of the smart cane is 87.33%, indicating that this device can make correct judgments about the user's status in most cases.

The Internet of Everything:APP + Bluetooth

Finally, all data will be transferred to the mobile phone.And integrated through APP.at the same time,Bluetooth audio will broadcast this data to the user, helping them understand their health status and external environment.

Figure 13: Integrated APP interface

Focus on general eye health

Although barrier-free facilities are becoming more and more popular in many scenarios, these devices are more targeted at people with limited mobility. For the visually impaired people in the dark, they need more barrier-free facilities.

But becauseThe population is small and the cost of facilities is high, so these facilities are difficult to popularize. Blind paths, Braille and other facilities that facilitate the visually impaired.Lack of maintenance and management, cannot be effective.

However,Many individuals and companies have participated in helping the visually impaired.Currently, most mainstream mobile phone brands already have operating methods designed specifically for the visually impaired.

In cities such as Beijing and Kunming, "cinemas in the mind" have been opened to tell movies for the visually impaired, helping them watch blockbuster films in theaters.

In game communities such as "Minecraft" and "Hearthstone", there are also enthusiastic players who have developed barrier-free modules (Mods) to help improve the gaming experience for the visually impaired.

Figure 14: Blind audience in the “cinema in the mind”

June 6, 2023 is the 28th National Eye Care Day, with the theme of "Focus on general eye healthToday is the 23rd World Sight Day. I hope everyone can pay attention to vision health and contribute to the visually impaired people around them.Work with AI to build an accessible world.

Reference Links:

[1]https://www.who.int/zh/news/item/08-10-2019-who-launches-first-world-report-on-vision

[2]https://bjo.bmj.com/content/96/5/614.short

[3]https://www.bilibili.com/video/BV1fs4y1K7Q7