1. Home
  2. Vol 4, No 1 (2023)
  3. Alexandrov

Development of an Algorithm for Multicriteria Optimization of Deep Learning Neural Networks

Islam A. Alexandrov, Andrey V. Kirichek, Vladimir Z. Kuklin, Leonid M. Chervyakov

Abstract


Nowadays, machine learning methods are actively used to process big data. A promising direction is neural networks, in which structure optimization occurs on the principles of self-configuration. Genetic algorithms are applied to solve this nontrivial problem. Most multicriteria evolutionary algorithms use a procedure known as non-dominant sorting to rank decisions. However, the efficiency of procedures for adding points and updating rank values in non-dominated sorting (incremental non-dominated sorting) remains low. In this regard, this research improves the performance of these algorithms, including the condition of an asynchronous calculation of the fitness of individuals. The relevance of the research is determined by the fact that although many scholars and specialists have studied the self-tuning of neural networks, they have not yet proposed a comprehensive solution to this problem. In particular, algorithms for efficient non-dominated sorting under conditions of incremental and asynchronous updates when using evolutionary methods of multicriteria optimization have not been fully developed to date. To achieve this goal, a hybrid co-evolutionary algorithm was developed that significantly outperforms all algorithms included in it, including error-back propagation and genetic algorithms that operate separately. The novelty of the obtained results lies in the fact that the developed algorithms have minimal asymptotic complexity. The practical value of the developed algorithms is associated with the fact that they make it possible to solve applied problems of increased complexity in a practically acceptable time.

 

Doi: 10.28991/HIJ-2023-04-01-011

Full Text: PDF


Keywords


Neural Networks; Genetic Algorithms; Hybrid Co-Evolutionary Algorithm; Feature Selection; Multicriteria Optimization.

References


Zhang, J. Z., Srivastava, P. R., Sharma, D., & Eachempati, P. (2021). Big data analytics and machine learning: A retrospective overview and bibliometric analysis. Expert Systems with Applications, 184, 115561. doi:10.1016/j.eswa.2021.115561.

Kuklin, V., Alexandrov, I., Polezhaev, D., & Tatarkanov, A. (2023). Prospects for developing digital telecommunication complexes for storing and analyzing media data. Bulletin of Electrical Engineering and Informatics, 12(3), 1536–1549. doi:10.11591/eei.v12i3.4840.

Kuklin, V. Z., Alexandrov, I. A., Umyskov, A. A., & Lampezhev, A. K. (2022). Analysis of the Prospects for Developing Storage and Processing Complexes for Multiformat Media Data. Journal of Computer Science, 18(12), 1159–1169. doi:10.3844/jcssp.2022.1159.1169

Xu, Y., Wang, Z., Gao, H., Jiang, Z., Yin, Y., & Li, R. (2023). Towards machine-learning-driven effective mashup recommendations from big data in mobile networks and the Internet-of-Things. Digital Communications and Networks, 9(1), 138–145. doi:10.1016/j.dcan.2022.12.009.

Antonakoudis, A., Barbosa, R., Kotidis, P., & Kontoravdi, C. (2020). The era of big data: Genome-scale modelling meets machine learning. Computational and Structural Biotechnology Journal, 18, 3287–3300. doi:10.1016/j.csbj.2020.10.011.

Párizs, R. D., Török, D., Ageyeva, T., & Kovács, J. G. (2022). Machine Learning in Injection Molding: An Industry 4.0 Method of Quality Prediction. Sensors, 22(7), 2704. doi:10.3390/s22072704.

Bachute, M. R., & Subhedar, J. M. (2021). Autonomous Driving Architectures: Insights of Machine Learning and Deep Learning Algorithms. Machine Learning with Applications, 6, 100164. doi:10.1016/j.mlwa.2021.100164.

Yakimovich, A., Beaugnon, A., Huang, Y., & Ozkirimli, E. (2021). Labels in a haystack: Approaches beyond supervised learning in biomedical applications. Patterns, 2(12), 100383. doi:10.1016/j.patter.2021.100383.

Alexandrov, I. A., Kuklin, V. Z., Muranov, A. N., & Tatarkanov, A. A. (2022). Theoretical Foundations of an Algorithm of Visualization of a Set of Points of a Multidimensional Space for Use in Anthropotechnical Decision Support Systems. Proceedings of the Institute for System Programming of the RAS, 34(4), 201–210. doi:10.15514/ispras-2022-34(4)-14.

Ige, A. O., & Mohd Noor, M. H. (2022). A survey on unsupervised learning for wearable sensor-based activity recognition. Applied Soft Computing, 127, 109363. doi:10.1016/j.asoc.2022.109363.

Tatarkanov, A., Alexandrov, I., Muranov, A., & Lampezhev, A. (2022). Development of a Technique for the Spectral Description of Curves of Complex Shape for Problems of Object Classification. Emerging Science Journal, 6(6), 1455–1475. doi:10.28991/esj-2022-06-06-015.

Sawant, S. S., & Prabukumar, M. (2020). A review on graph-based semi-supervised learning methods for hyperspectral image classification. The Egyptian Journal of Remote Sensing and Space Science, 23(2), 243–248. doi:10.1016/j.ejrs.2018.11.001.

Rožanec, J. M., Trajkova, E., Dam, P., Fortuna, B., & Mladenic, D. (2022). Streaming Machine Learning and Online Active Learning for Automated Visual Inspection. IFAC-PapersOnLine, 55(2), 277–282. doi:10.1016/j.ifacol.2022.04.206.

Zhao, T., Zheng, Y., & Wu, Z. (2022). Improving computational efficiency of machine learning modeling of nonlinear processes using sensitivity analysis and active learning. Digital Chemical Engineering, 3, 100027. doi:10.1016/j.dche.2022.100027.

Khayrutdinov, M. M., Golik, V. I., Aleksakhin, A. V., Trushina, E. V., Lazareva, N. V., & Aleksakhina, Y. V. (2022). Proposal of an algorithm for choice of a development system for operational and environmental safety in mining. Resources, 11(10), 88. doi:10.3390/resources11100088.

Zheng, X., Li, P., & Wu, X. (2022). Data stream classification based on extreme learning machine: a review. Big Data Research, 30, 100356. doi:10.1016/j.bdr.2022.100356.

Binkhonain, M., & Zhao, L. (2019). A review of machine learning algorithms for identification and classification of non-functional requirements. Expert Systems with Applications: X, 1, 100001. doi:10.1016/j.eswax.2019.100001.

Singh, M. P., & Saraswat, V. K. (2017). Multilayer feed forward neural networks for non-linear continuous bidirectional associative memory. Applied Soft Computing, 61, 700–713. doi:10.1016/j.asoc.2017.08.026.

Laddach, K., Łangowski, R., Rutkowski, T. A., & Puchalski, B. (2022). An automatic selection of optimal recurrent neural network architecture for processes dynamics modelling purposes. Applied Soft Computing, 116, 108375. doi:10.1016/j.asoc.2021.108375.

Stewart, R. H., Palmer, T. S., & DuPont, B. (2021). A survey of multi-objective optimization methods and their applications for nuclear scientists and engineers. Progress in Nuclear Energy, 138, 103830. doi:10.1016/j.pnucene.2021.103830.

Maier, H. R., Razavi, S., Kapelan, Z., Matott, L. S., Kasprzyk, J., & Tolson, B. A. (2019). Introductory overview: Optimization using evolutionary algorithms and other metaheuristics. Environmental Modelling & Software, 114, 195–213. doi:10.1016/j.envsoft.2018.11.018.

Grygar, D., & Fabricius, R. (2019). An efficient adjustment of genetic algorithm for Pareto front determination. Transportation Research Procedia, 40, 1335–1342. doi:10.1016/j.trpro.2019.07.185.

Lessmann, S., Stahlbock, R., & Crone, S. F. (2006). Genetic Algorithms for Support Vector Machine Model Selection. The 2006 IEEE International Joint Conference on Neural Network Proceedings. doi:10.1109/ijcnn.2006.1716515.

Gavrilescu, M., Floria, S. A., Leon, F., & Curteanu, S. (2022). A Hybrid Competitive Evolutionary Neural Network Optimization Algorithm for a Regression Problem in Chemical Engineering. Mathematics, 10(19), 3581. doi:10.3390/math10193581.

Akhmedova, S. A., & Semenkin, E. S. (2016). Collective bionic algorithm with biogeography based migration operator for binary optimization. Journal of Siberian Federal University - Mathematics & Physics, 9(1), 3–10. doi:10.17516/1997-1397-2016-9-1-3-10.

Tan, K. C., Chew, Y. H., Lee, T. H., & Yang, Y. J. (n.d.). A cooperative coevolutionary algorithm for multiobjective optimization. SMC’03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483). doi:10.1109/icsmc.2003.1243847.

Fan, Z., Zi-xuan, X., & Ming-hu, W. (2023). State of health estimation for Li-ion battery using characteristic voltage intervals and genetic algorithm optimized back propagation neural network. Journal of Energy Storage, 57, 106277. doi:10.1016/j.est.2022.106277.

Zangeneh, M., Aghajari, E., & Forouzanfar, M. (2020). A Review on Optimization of Fuzzy Controller Parameters in Robotic Applications. IETE Journal of Research, 68(6), 4150–4159. doi:10.1080/03772063.2020.1787878.

Sopov, E. (2015). A Self-configuring Metaheuristic for Control of Multi-Strategy Evolutionary Search. Advances in Swarm and Computational Intelligence. ICSI 2015. Lecture Notes in Computer Science, 9142. Springer, Cham, Switzerland. doi:10.1007/978-3-319-20469-7_4.

Semeraro, F., Griffiths, A., & Cangelosi, A. (2023). Human–robot collaboration and machine learning: A systematic review of recent research. Robotics and Computer-Integrated Manufacturing, 79, 102432. doi:10.1016/j.rcim.2022.102432.

Hamdi, Y., Boubaker, H., Rabhi, B., Qahtani, A. M., Alharithi, F. S., Almutiry, O., Dhahri, H., & Alimi, A. M. (2022). Deep learned BLSTM for online handwriting modeling simulating the Beta-Elliptic approach. Engineering Science and Technology, an International Journal, 35, 101215. doi:10.1016/j.jestch.2022.101215.

Xu, X., Li, D., Zhou, Y., & Wang, Z. (2022). Multi-type features separating fusion learning for Speech Emotion Recognition. Applied Soft Computing, 130, 109648. doi:10.1016/j.asoc.2022.109648.

Peer, D., Stabinger, S., & Rodriguez-Sanchez, A. (2021). Auto-tuning of Deep Neural Networks by Conflicting Layer Removal. Neural Networks. doi:10.48550/arXiv.2103.04331.

Alkatheiri, M. S. (2022). Artificial intelligence assisted improved human-computer interactions for computer systems. Computers and Electrical Engineering, 101, 107950. doi:10.1016/j.compeleceng.2022.107950.

Ualiyeva, R. (2023). Functional role of vitelline glands and mehlis gland in the process of resistant egg shell formation in trematodes. OnLine Journal of Biological Sciences, 23(2), 124–132. doi:10.3844/ojbsci.2023.124.132.

Fan, X., & Zhong, X. (2022). Artificial intelligence-based creative thinking skill analysis model using human–computer interaction in art design teaching. Computers and Electrical Engineering, 100, 107957. doi:10.1016/j.compeleceng.2022.107957.

Abiodun, O. I., Jantan, A., Omolara, A. E., Dada, K. V., Mohamed, N. A. E., & Arshad, H. (2018). State-of-the-art in artificial neural network applications: A survey. Heliyon, 4(11), 938. doi:10.1016/j.heliyon.2018.e00938.

Coulibaly, S., Kamsu-Foguem, B., Kamissoko, D., & Traore, D. (2022). Deep Convolution Neural Network sharing for the multi-label images classification. Machine Learning with Applications, 10, 100422. doi:10.1016/j.mlwa.2022.100422.

Costa-Carrapiço, I., Raslan, R., & González, J. N. (2020). A systematic review of genetic algorithm-based multi-objective optimisation for building retrofitting strategies towards energy efficiency. Energy and Buildings, 210, 109690. doi:10.1016/j.enbuild.2019.109690.

Schaffer, J. D. (1985). Multiple objective optimization with vector evaluated genetic algorithms. Proceedings of an international conference on genetic algorithms and their applications, 24-26 July, 1985, Carnegie-Mellon University, Pittsburgh, United States.

Srinivas, N., & Deb, K. (1994). Muiltiobjective Optimization Using Nondominated Sorting in Genetic Algorithms. Evolutionary Computation, 2(3), 221–248. doi:10.1162/evco.1994.2.3.221.

Deb, K., Pratap, A., Agarwal, S., & Meyarivan, T. (2002). A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6(2), 182–197. doi:10.1109/4235.996017.

Zitzler, E., Laumanns, M., & Thiele, L. (2001). SPEA2: Improving the strength 418 Pareto evolutionary algorithm. Tech. Rep., Eidgenössische Technische 419 Hochschule Zürich (ETH), Institut für Technische Informatik und Kom-420 munikationsnetze (TIK), 1-22. doi:10.3929/ethz-a-004284029.

Corne, D. W., Jerram, N. R., Knowles, J. D., & Oates, M. J. (2001). PESA-II: Region-based selection in evolutionary multiobjective optimization. Proceedings of the 3rd annual conference on genetic and evolutionary computation, 7-11 July, 2001, San Francisco, United States.

Zitzler, E., & Künzli, S. (2004). Indicator-Based Selection in Multiobjective Search. Parallel Problem Solving from Nature - PPSN VIII, PPSN 2004, Lecture Notes in Computer Science, 3242, Springer, Berlin, Germany. doi:10.1007/978-3-540-30217-9_84.

Bader, J., & Zitzler, E. (2011). HypE: An algorithm for fast hypervolume-based many-objective optimization. Evolutionary Computation, 19(1), 45–76. doi:10.1162/EVCO_a_00009.

Zhang, Q., & Li, H. (2007). MOEA/D: A multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on Evolutionary Computation, 11(6), 712–731. doi:10.1109/TEVC.2007.892759.

Deb, K., & Jain, H. (2014). An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, Part I: Solving problems with box constraints. IEEE Transactions on Evolutionary Computation, 18(4), 577–601. doi:10.1109/TEVC.2013.2281535.

Zheng, W., Liu, Y., & Doerr, B. (2022). A first mathematical runtime analysis of the Non-Dominated Sorting Genetic Algorithm II (NSGA-II). In Proceedings of the AAAI Conference on Artificial Intelligence, 36(9), 10408-10416. doi:10.1609/aaai.v36i9.21283.

Roy, P. C., Islam, Md. M., & Deb, K. (2016). Best Order Sort. Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion. doi:10.1145/2908961.2931684.

Jensen, M. T. (2003). Reducing the Run-Time Complexity of Multiobjective EAs: The NSGA-II and Other Algorithms. IEEE Transactions on Evolutionary Computation, 7(5), 503–515. doi:10.1109/TEVC.2003.817234.

Fortin, F.-A., Grenier, S., & Parizeau, M. (2013). Generalizing the improved run-time complexity algorithm for non-dominated sorting. Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation. doi:10.1145/2463372.2463454.

Nebro, A. J., & Durillo, J. J. (2009). On the effect of applying a steady-state selection scheme in the multi-objective genetic algorithm NSGA-II. Studies in Computational Intelligence, 193, 435–456. doi:10.1007/978-3-642-00267-0_16.

Li, K., Deb, K., Zhang, Q., & Zhang, Q. (2017). Efficient Nondomination Level Update Method for Steady-State Evolutionary Multiobjective Optimization. IEEE Transactions on Cybernetics, 47(9), 2838–2849. doi:10.1109/TCYB.2016.2621008.

Mishra, S., Mondal, S., & Saha, S. (2017). Improved solution to the non-domination level update problem. Applied Soft Computing, 60, 336–362. doi:10.1016/j.asoc.2017.06.038.

Canayaz, M. (2022). Classification of diabetic retinopathy with feature selection over deep features using nature-inspired wrapper methods. Applied Soft Computing, 128, 109462. doi:10.1016/j.asoc.2022.109462.

Greenberg, C. S., Mason, L. P., Sadjadi, S. O., & Reynolds, D. A. (2020). Two decades of speaker recognition evaluation at the national institute of standards and technology. Computer Speech & Language, 60, 101032. doi:10.1016/j.csl.2019.101032.


Full Text: PDF

DOI: 10.28991/HIJ-2023-04-01-011

Refbacks

  • There are currently no refbacks.


Copyright (c) 2023 Islam Alexandrovich Alexandrov, Andrey Victorovich Kirichek, Vladimir Zhanovich Kuklin, Leonid Mikhajlovich Chervyakov