A Robust and Secure Edge-Based AI System Against Adversarial Attacks

Abstract

Ensuring the safe deployment and use of artificial intelligence (AI) in safety-critical systems is crucial in the reality of effective adversarial attacks (AAs). AAs involve manipulating the data inputs to AI models to make them behave abnormally and make mistakes. Such attacks may lead AI systems to perform destructive behaviors, leading to unintended outcomes. Therefore, the threat of AAs must be considered in the design process of such systems. The main objectives of this thesis were to understand the status quo of AAs and defenses in image and video object detection (OD) that needs attention in Kongsberg Defense & Aerospace's (KDA's) context (Goal-01), and use that information to derive a system architecture and requirements for the safe deployment of AI in an unmanned military setting (Goal-02). A systematic literature review of AAs and defenses in OD was done to uncover the state of the art in the field. The results showed that most existing research focused on AAs in the digital domain with white box knowledge. However, for AAs to become a real threat to unmanned military systems, the research field must focus more on creating physically realizable black box AAs - a challenging task yet to be properly solved and a somewhat premature research field with debatable real-world threats. A system for the safe deployment of AI in an unmanned military setting was designed based on the systematic literature review results and a requirements engineering process with KDA. Supporting manned vehicles and a back-end server were included to cover the entire AI lifecycle and cope with the limitations of military systems. The use of GPUs for accelerated AI was an essential enabling technology. The threat of most AAs against the system was considered negligible due to the strict security requirements of military systems. Adversarial defenses, like adversarial- training and detection, were recommended to further reduce the threat of AAs, especially the most prominent threat of physically realizable black box AAs. This thesis delivered a comprehensive review of the state-of-the-art AAs and defenses in OD in the context of unmanned military vehicles – the first of its kind and a valuable resource for the research field and the defense industry. KDA gained valuable information on how AI can safely be deployed in unmanned military systems and how to maintain a cycle of ever-improving AI models throughout the lifetime of the system.