Today, our understanding and assessment of the physical characteristics of the world around us relies less and less on human intelligence and more and more on artificial intelligence.
Remote sensing (RS) technologies have become critical tools for government intelligence, environmental monitoring, autonomous transportation, urban planning and disaster management. But the vast number of images produced by remote cameras must be processed and interpreted, and these tasks are being delegated more and more to deep learning (DL) models.
DL models can process and interpret images much more quickly than humans can, and recent advances in AI have only improved this capacity over time. Despite this increase in efficiency, no study has ever attempted to assess the overall robustness and potential vulnerabilities of deep-neural-network (DNN)-based models used for object detection and image classification in RS images.
To address this issue, a team of scientists from Northwestern Polytechnical University and The Hong Kong Polytechnic University performed a review of all the existing research studies on the robustness of DL models used for object detection and classification using RS images and developed a benchmark to assess the performance of various DL model detectors ( e.g. , YOLO versions, RetinaNet, FreeAnchor). Critically, their analysis revealed several vulnerabilities of DL algorithms for object detection, which could be leveraged by attackers.
The team published their review in the September 11 issue of the Journal of Remote Sensing .
“We sought to address the lack of comprehensive studies on the robustness of deep learning models used in remote sensing tasks, particularly focusing on image classification and object detection. Our aim was to understand the vulnerabilities of these models to various types of noise, especially adversarial noise, and to systematically evaluate their natural and adversarial robustness,” said Shaohui Mei, professor in the School of Electronic Information at Northwestern Polytechnical University in Xi’an, China and lead author of the review paper.
More specifically, the team investigated the effects of natural noise and various attacks on model performance. The scientists used natural noise sources, including salt-pepper and random noise and rain, snow and fog, at different intensities to test the robustness of object detection and identification using DL models. The team also tested model performance using various digital attacks to exploit vulnerabilities in models, including the Fast Gradient Sign Method (FGSM), AutoAttack, Projected Gradient Descent, Carlini & Wagner and Momentum Iterative FGSM. They also determined the effects of potential physical attacks, where a patch could be physically painted or attached to an object or object’s background to impair the DL model.
The researchers found many vulnerabilities in DL models that could be exploited by potential adversaries. Deep learning models, despite their powerful capabilities in remote sensing applications, are susceptible to different kinds of disturbances, including adversarial attacks. It is crucial for developers and users of these technologies to be aware of these vulnerabilities and to work towards improving model robustness to ensure reliable performance in real-world conditions,” said Jiawei Lian, graduate student at the School of Electronic Information at Northwestern Polytechnical University and an author on the paper.
To help other researchers improve DL model robustness in these applications, the authors summarized the results of their analysis across various models, noise types and attacks:
The research team acknowledges that their analysis provides only a blueprint for improving RS DL model robustness. “[Our] next step[s] involve further refining our benchmarking framework and conducting more extensive tests with a wider range of models and noise types. Our ultimate goal is to contribute to the development of more robust and secure DL models for RS, thereby enhancing the reliability and effectiveness of these technologies in critical applications such as environmental monitoring, disaster response, and urban planning,” said Mei.
Xiaofei Wang, Yuru Su and Mingyang Ma from the School of Electronic Information at the Northwestern Polytechnical University in Xi’an, China and Lap-Pui Chau from Department of Electrical and Electronic Engineering at The Hong Kong Polytechnic University also contributed to this research.
Shaohui Mei is a professor at the School of Electronic Information at Northwestern Polytechnical University and can be contacted at meish@nwpu.edu.cn . Jiawei Lian is a Ph.D. candidate at the School of Electronic Information at Northwestern Polytechnical University and can be contacted at lianjiawei@mail.nwpu.edu.cn . Follow research in the Mei Lab at https://sciprofiles.com/profile/280878.
Journal of Remote Sensing
Literature review
Not applicable
A Comprehensive Study on the Robustness of Deep Learning-Based Image Classification and Object Detection in Remote Sensing: Surveying and Benchmarking
3-Oct-2024
The authors declare that they have no competing interests.