1. Home
  2. Vol 5, No 4 (2024)
  3. Editya

Visual Instruction Tuning for Drone Accident Forensics

Arda Surya Editya, Tohari Ahmad, Hudan Studiawan

Abstract


The increasing use of drones in both commercial and personal use has led to a growing demand for effective forensic analysis following drone-related accidents. This research focuses on improving forensic analysis through the development of LLaVAFor, a fine-tuned version of the Large Language and Vision Assistant (LLaVA) model. The objective of this study is to enhance the interpretability of visual instruction tuning for drone accident forensics. LLaVAFor was developed by fine-tuning LLaVA via a specialized dataset of drone accident scenarios. The model's performance was evaluated via the BLEU score, a metric commonly used to assess machine translation and natural language processing models. The results demonstrated that LLaVAFor achieved superior BLEU scores compared with baseline models such as LLaVA, Google Gemini, and ChatGPT. It demonstrates its ability to provide more accurate and contextually relevant analyses. The key innovation in LLaVAFor is its ability to explain forensic findings in the context of drone accidents, making it a valuable tool for investigators. The results show that the model's fine-tuning process on drone-specific datasets enables it to offer detailed, domain-specific insights, improving the accuracy and reliability of forensic analyses in this field. Through these advancements, LLaVAFor represents a step forward in the integration of AI into drone accident investigations.

 

Doi: 10.28991/HIJ-2024-05-04-01

Full Text: PDF


Keywords


Forensic Analysis; Drone Forensics; LLaVA; Drone Accident.

References


Ajakwe, S.O., Ihekoronye, V.U., Akter, R., Kim, D.S., Lee, J.M. (2022). Adaptive drone identification and neutralization scheme for real-time military tactical operations. 2022 International Conference on Information Networking (ICOIN), 380–384. doi:10.1109/ICOIN53446.2022.9687268.

Famula, J., Pittman, D.E., Haring, K.S. (2022). Building trust with a mobile application for last-mile commercial drone delivery. 2022 International Conference on Unmanned Aircraft Systems (ICUAS), 462–467. doi:10.1109/ICUAS54217.2022.9836198

Bengiamin, N.N. (2018). Quadcopter drones - beyond the hobby. 2018 IEEE Frontiers in Education Conference (FIE), 1–5. doi:10.1109/FIE.2018.8659124.

Jain, U., Rogers, M., Matson, E.T. (2017). Drone forensic framework: Sensor and data identification and verification. 2017 IEEE Sensors Applications Symposium (SAS), 1–6. doi:10.1109/SAS.2017.7894059.

Minaeian, S., Liu, J., Son, Y.J. (2018). Effective and efficient detection of moving targets from a UAV’s camera. IEEE Transactions on Intelligent Transportation Systems 19(2):497–506. doi:10.1109/TITS.2017.2782790.

Editya, A.S., Ahmad, T., Studiawan, H. (2023). Forensic analysis of drone malfunction based on location data. IEEE International Conference on Communication, Networks and Satellite (COMNETSAT), 658–663. doi:10.1109/COMNETSAT59769.2023.10420805.

Renduchintala, A.L.P.S., Albehadili, A., Javaid, A.Y. (2017). Drone forensics: Digital flight log examination framework for micro drones. International Conference on Computational Science and Computational Intelligence (CSCI-IEEE), 91-96. doi:10.1109/CSCI.2017.15.

Horsman, G. (2016). Unmanned aerial vehicles: A preliminary analysis of forensic challenges. Digital Investigation, 16, 1-11. doi:10.1016/j.diin.2015.11.002.

Mora, G.O.D., Zamudio, B.Z. (2018). Real-time drone (UAV) trajectory generation and tracking by optical flow. 2018 International Conference on Mechatronics, Electronics and Automotive Engineering (ICMEAE). Cuernavaca, Mexico. doi:10.1109/ICMEAE.2018.00014.

Editya, A.S., Ahmad, T., Studiawan, H. (2023). Forensic analysis of drone collision with transfer learning. Jordanian Journal of Computers and Information Technology (JJCIT), 9(2), 175 - 186, doi:10.5455/jjcit.71-1673581703.

Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2024). Visual instruction tuning. Advances in Neural Information Processing Systems, 36. doi:10.5555/3666122.3667638.

Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J. (2023). LLaVA-Med: Training a large language-and-vision assistant for biomedicine in one day. Proceedings of the 37th International Conference on Neural Information Processing Systems, 24-29. doi: 10.5555/3666122.3667362.

Al-Dhaqm, A., Ikuesan, R. A., Kebande, V. R., Razak, S., & Ghabban, F. M. (2021). Research challenges and opportunities in drone forensics models. Electronics, 10(13), 1519.. doi:10.3390/electronics10131519.

Barton, T.E.A., Hannan Bin Azhar, M.A. (2017). Forensic analysis of popular UAV systems. 2017 Seventh International Conference on Emerging Security Technologies (EST), 91–96. doi:10.1109/EST.2017.8090405.

Mantas, E., Patsakis, C. (2019). GRYPHON: Drone forensics in dataflash and telemetry logs. Advances in Information and Computer Security, 377–390. doi:10.1007/978-3-030-26834-3_22.

Scanlon, M., Breitinger, F., Hargreaves, C., Hilgert, J. N., & Sheppard, J. (2023). ChatGPT for digital forensic investigation: The good, the bad, and the unknown. Forensic Science International: Digital Investigation, 46, 301609. doi:10.1016/j.fsidi.2023.301609.

Michelet, G., Breitinger, F. (2024). ChatGPT, Llama, can you write my report? An experiment on assisted digital forensics reports written using (local) large language models. Forensic Science International: Digital Investigation, 48, 301683. doi:10.1016/j.fsidi.2023.301683.

Piggott, B., Patil, S., Feng, G., Odat, I., Mukherjee, R., Dharmalingam, B. (2023). Net-GPT: A LLM-Empowered Man-in-the-Middle chatbot for unmanned aerial vehicle. 2023 IEEE/ACM Symposium on Edge Computing (SEC), 287–293. doi:10.1145/3583740.3626809.

Wickramasekara, A., Breitinger, F., Scanlon, M. (2024). SoK: Exploring the potential of large language models for improving digital forensic investigation efficiency. ARES '20: Proceedings of the 15th International Conference on Availability, Reliability and Security, 1-10. doi: 10.1145/3407023.3407068.

Pedro, D., Matos-Carvalho, J.P., Fonseca, J.M., Mora, A. (2021). Collision avoidance on unmanned aerial vehicles using neural network pipelines and flow clustering techniques. Remote Sensing, 13(13), 2643. doi: 10.3390/rs13132643.

Cloutier, N.A., Japkowicz, N. (2023). Fine-tuned generative LLM oversampling can improve performance over traditional techniques on multiclass imbalanced text classification. 2023 IEEE International Conference on Big Data (BigData), 5181–5186. doi: 10.1109/BigData59044.2023.10386772.

Li, H., Shan, L. (2023). LLM-based vulnerability detection. 2023 International Conference on Human-Centered Cognitive Systems (HCCS), 1–4. doi: 10.1145/3639476.3639762.

Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311-318. doi:10.3115/1073083.1073135.

He, W., Li, Z., Wang, H., Xu, T., Wang, Z., Huai, B. (2024). Multimodal dialogue systems via capturing context-aware dependencies and ordinal information of semantic elements. ACM Transactions on Intelligent Systems and Technology, 15(3), 1–25. doi:10.1145/3394171.3413679.


Full Text: PDF

DOI: 10.28991/HIJ-2024-05-04-01

Refbacks

  • There are currently no refbacks.


Copyright (c) 2024 Arda Surya Editya, Tohari Ahmad, Hudan Studiawan