Multidisciplinary Journal of Science and Technology

MATHEMATICAL MODEL BASED ON MULTI-AGENT REINFORCEMENT LEARNING (MARL) AND PARTIALLY OBSERVABLE MARKOV DECISION PROCESS (POMDP) FOR MODELING CARGO MOVEMENT FOR A MOBILE ROBOTS GROUP

Olena Chala

Department of Computer-Integrated Technologies, Automation and Robotics, Kharkiv National University of Radio Electronics, Ukraine

Vladyslav Yevsieiev

Department of Computer-Integrated Technologies, Automation and Robotics, Kharkiv National University of Radio Electronics, Ukraine

Svitlana Maksymova

Department of Computer-Integrated Technologies, Automation and Robotics, Kharkiv National University of Radio Electronics, Ukraine

Amer Abu-Jassar

Department of Computer Science, College of Information Technology, Amman Arab University, Amman, Jordan

Keywords: Multi-Agent Reinforcement Learning, POMDP

---

Abstract

This paper presents a mathematical model that combines multi-agent reinforcement learning (MARL) and partially observable Markov decision process (POMDP) to model the task of moving cargo by a mobile robots group. The use of such methods allows agents to effectively interact under conditions of incomplete information, optimizing task performance in real, dynamic environments. This model has significant potential for process automation in industrial and logistics systems.

---

References

1. Kuzmenko, O., Yevsieiev, V., Maksymova, S., & Abu-Jassar, A. (2024). Robot Model for Mines Searching Development. Multidisciplinary Journal of Science and Technology, 4(6), 347-355.

2. Maksymova, S., Yevsieiev, V., Nevliudov, I., & Bahlai, O. (2024). Balancing System For A Zoomorphic Spot Type Mobile Robot Development Using An Accelerometer MPU 6050 (GY-521). In 2024 IEEE 19th International Conference on the Perspective Technologies and Methods in MEMS Design (MEMSTECH), IEEE, 39-42.

3. Nevliudov, I., Yevsieiev, V., Maksymova, S., & Chala, O. (2023). A Small-Sized Robot Prototype Development Using 3D Printing. CAD In Machinery Design Implementation and Educational Issues (CADMD'2023) : proceedings of the XXXI International Conference, Suprasl, 12.

4. Borysov, H., & Maksymova, S. (2023). Parameters for Mobile Robot Kinematic Model Development Determination. Multidisciplinary Journal of Science and Technology, 3(4), 85-91.

5. Maksymova, S., Abu-Jassar, A., Gurin, D., & Yevsieiev, V. (2024). Comparative Analysis of methods for Predicting the Trajectory of Object Movement in a Collaborative Robot-Manipulator Working Area. Multidisciplinary Journal of Science and Technology, 4(10), 38-48.

6. Yevsieiev, V., Alkhalaileh, A., Maksymova, S., & Gurin, D. (2024). Research of Existing Methods of Representing a Collaborative Robot-Manipulator Environment within the Framework of Cyber-Physical Production Systems. Multidisciplinary Journal of Science and Technology, 4(9), 112-120.

7. Basiuk, V., Maksymova, S., Chala, O., & Miliutina, O. (2023). Mobile Robot Position Determining Using Odometry Method. Multidisciplinary Journal of Science and Technology, 3(3), 227-234.

8. Maksymova, S., Yevsieiev, V., & Abu-Jassar, A. (2024). The Bipedal Robot a Kinematic Diagram Development. Journal of Universal Science Research, 2(1), 6–17.

9. Al-Sharo, Y. M., Abu-Jassar, A. T., Sotnik, S., & Lyashenko, V. (2021). Neural networks as a tool for pattern recognition of fasteners. International Journal of Engineering Trends and Technology, 69(10), 151-160.

10. Abu-Jassar, A. T., Al-Sharo, Y. M., Lyashenko, V., & Sotnik, S. (2021). Some Features of Classifiers Implementation for Object Recognition in Specialized Computer systems. TEM Journal: Technology, Education, Management, Informatics, 10(4), 1645-1654.

11. Sotnik, S., Mustafa, S. K., Ahmad, M. A., Lyashenko, V., & Zeleniy, O. (2020). Some features of route planning as the basis in a mobile robot. International Journal of Emerging Trends in Engineering Research, 8(5), 2074-2079.

12. Baker, J. H., Laariedh, F., Ahmad, M. A., Lyashenko, V., Sotnik, S., & Mustafa, S. K. (2021). Some interesting features of semantic model in Robotic Science. SSRG International Journal of Engineering Trends and Technology, 69(7), 38-44.

13. Lyashenko, V., Abu-Jassar, A. T., Yevsieiev, V., & Maksymova, S. (2023). Automated Monitoring and Visualization System in Production. International Research Journal of Multidisciplinary Technovation, 5(6), 9-18.

14. Ahmad, M. A., Sinelnikova, T., Lyashenko, V., & Mustafa, S. K. (2020). Features of the construction and control of the navigation system of a mobile robot. International Journal of Emerging Trends in Engineering Research, 8(4), 1445-1449.

15. Lyashenko, V., Laariedh, F., Ayaz, A. M., & Sotnik, S. (2021). Recognition of Voice Commands Based on Neural Network. TEM Journal: Technology, Education, Management, Informatics, 10(2), 583-591.

16. Ahmad, M. A., Baker, J. H., Tvoroshenko, I., & Lyashenko, V. (2019). Computational complexity of the accessory function setting mechanism in fuzzy intellectual systems. International Journal of Advanced Trends in Computer Science and Engineering, 8(5), 2370-2377.

17. Sotnik S., & et al.. (2022). Key Directions for Development of Modern Expert Systems. International Journal of Engineering and Information Systems (IJEAIS), 6(5), 4-10.

18. Abu-Jassar, A. T., Attar, H., Amer, A., Lyashenko, V., Yevsieiev, V., & Solyman, A. (2025). Development and Investigation of Vision System for a Small-Sized Mobile Humanoid Robot in a Smart Environment. International Journal of Crowd Science, 9(1), 29-43.

19. Abu-Jassar, A. T., Attar, H., Amer, A., Lyashenko, V., Yevsieiev, V., & Solyman, A. (2024). Remote Monitoring System of Patient Status in Social IoT Environments Using Amazon Web Services (AWS) Technologies and Smart Health Care. International Journal of Crowd Science, 8.

20. Samoilenko, H., Yevsieiev, V., Maksymova, S., & Amer, A. J. (2024). Review for Collective Problem-Solving by a Group of Robots. Journal of Universal Science Research, 2(6), 7-16.

21. Maksymova, S., Yevsieiev, V., & Alkhalaileh, A. (2024). The Lucas-Kanade method implementation for estimating the objects movement in the mobile robot’s workspace. Journal of Universal Science Research, 2(3), 187-197.

22. Yevsieiev, V., Abu-Jassar, A., Maksymova, S., & Gurin, D. (2024). Human Operator Identification in a Collaborative Robot Workspace within the Industry 5.0 Concept. Multidisciplinary Journal of Science and Technology, 4(9), 95-105.

23. Гиренко, А. В., Ляшенко, В. В., Машталир, В. П., & Путятин, Е. П. (1996). Методы корреляционного обнаружения объектов. Харьков: АО “БизнесИнформ, 112.

24. Lyashenko, V., Kobylin, O., & Ahmad, M. A. (2014). General methodology for implementation of image normalization procedure using its wavelet transform. International Journal of Science and Research (IJSR), 3(11), 2870-2877.

25. Lyashenko, V. V., Babker, A. M. A. A., & Kobylin, O. A. (2016). The methodology of wavelet analysis as a tool for cytology preparations image processing. Cukurova Medical Journal, 41(3), 453-463.

26. Lyashenko, V. V., Matarneh, R., & Deineko, Z. V. (2016). Using the Properties of Wavelet Coefficients of Time Series for Image Analysis and Processing. Journal of Computer Sciences and Applications, 4(2), 27-34.

27. Babker, A., Suliman, R. S., Ghazwani, A. A. M., & Al Harbi, W. (2025). Combined use of Edge Selection Operators in Individual Color Channels for the Analysis of Cytological Images Presented in RGB Format. Biomedical and Pharmacology Journal, 18(Spl.), 121-138.

28. Ababneh, J., Abu-Jassar, A., Abuowaida, S., Liubchenko, V., & Lyashenko, V. (2024, December). Evaluation of Three Different Operators for Object Highlighting in Medical RGB Images: Canny, Roberts, and LoG in Independent Color Spaces. In 2024 25th International Arab Conference on Information Technology (ACIT) (pp. 1-7). IEEE.

29. Babker, A., Abbas, A., Shalabi, M., & Mohamedahmed, K. A. (2024). Edge Detection and Contrast Enhancement in the Examination of Megaloblastic Anemia Cells in Medical Images with Comparative Analysis of Different Approaches. Biomedical and Pharmacology Journal, 17(4), 2273-2286.

30. Tahseen A. J. A., & et al.. (2023). Binarization Methods in Multimedia Systems when Recognizing License Plates of Cars. International Journal of Academic Engineering Research (IJAER), 7(2), 1-9.

31. Orobinskyi, P., Deineko, Z., & Lyashenko, V. (2020). Comparative Characteristics of Filtration Methods in the Processing of Medical Images. American Journal of Engineering Research, 9(4), 20-25.

32. Mousavi, S. M. H., Lyashenko, V., & Prasath, V. B. S. (2019). Analysis of a robust edge detection system in different color spaces using color and depth images. Computer Optics, 43(4), 632-646.

33. Lyubchenko, V., Veretelnyk, K., Kots, P., & Lyashenko, V. (2024). Digital image segmentation procedure as an example of an NP-problem. Multidisciplinary Journal of Science and Technology, 4(4), 170-177.

34. Orobinskyi, P., Petrenko, D., & Lyashenko, V. (2019, February). Novel approach to computer-aided detection of lung nodules of difficult location with use of multifactorial models and deep neural networks. In 2019 IEEE 15th International Conference on the Experience of Designing and Application of CAD Systems (CADSM) (pp. 1-5). IEEE.

35. Mousavi, S. M. H., MiriNezhad, S. Y., & Lyashenko, V. An Evolutionary-Based Adaptive Neuro-Fuzzy Expert System as a Family Counselor before Marriage with the Aim of Divorce Rate Reduction. Education, 1, 5.

36. Sotnik, S. Overview: PHP and MySQL Features for Creating Modern Web Projects / S Sotnik, V. Manakov, V. Lyashenko //International Journal of Academic Information Systems Research (IJAISR). – 2023. – Vol. 7, Issue 1. – P. 11-17.

37. Lyashenko, V. V., Matarneh, R., Baranova, V., & Deineko, Z. V. (2016). Hurst Exponent as a Part of Wavelet Decomposition Coefficients to Measure Long-term Memory Time Series Based on Multiresolution Analysis. American Journal of Systems and Software, 4(2), 51-56.

38. Lyashenko, V. V., Matarneh, R., & Deineko, Z. V. (2016). Using the Properties of Wavelet Coefficients of Time Series for Image Analysis and Processing. Journal of Computer Sciences and Applications, 4(2), 27-34.

39. Tvoroshenko, I., Lyashenko, V., Ayaz, A. M., Mustafa, S. K., & Alharbi, A. R. (2020). Modification of models intensive development ontologies by fuzzy logic. International Journal of Emerging Trends in Engineering Research, 8(3), 939-944.

40. Matarneh, R., Tvoroshenko, I., & Lyashenko, V. (2019). Improving Fuzzy Network Models For the Analysis of Dynamic Interacting Processes in the State Space. International Journal of Recent Technology and Engineering, 8(4), 1687-1693.

41. Lyashenko, V., & et al.. (2016). The Methodology of Image Processing in the Study of the Properties of Fiber as a Reinforcing Agent in Polymer Compositions. International Journal of Advanced Research in Computer Science, 7(1), 15-18.

42. Kuzemin, A., Lуashenko, V., Bulavina, E., & Torojev, A. (2005). Analysis of movement of financial flows of economical agents as the basis for designing the system of economical security (general conception). In Third international conference «Information research, applications, and education (pp. 27-30).

43. Deineko, Zh., & et al.. (2021). Features of Database Types. International Journal of Engineering and Information Systems (IJEAIS), 5(10), 73-80.

44. Sotnik, S., & Lyashenko, V. (2022). Prospects for Introduction of Robotics in Service. Prospects, 6(5), 4-9.

45. Kobylin, O., & Lyashenko, V. (2020). Time Series Clustering Based on the K-Means Algorithm. Journal La Multiapp, 1(3), 1-7.

46. Vasiurenko, O., Baranova, V., & Lyashenko, V. (2024). Probability distributions of interest rates on loans and deposits in a study of banking activities. Multidisciplinary Journal of Science and Technology, 4(1), 49-56.

47. Omarov, M., Tykha, T., & Lyashenko, V. (2019). Use of Wavelet Techniques in the Study of Internet Marketing Metrics. Eskişehir Technical University Journal of Science and Technology A-Applied Sciences and Engineering, 20, 157-163.

48. Gu, S., Kuba, J. G., Chen, Y., Du, Y., Yang, L., Knoll, A., & Yang, Y. (2023). Safe multi-agent reinforcement learning for multi-robot control. Artificial Intelligence, 319, 103905.

49. Hu, J., Bhowmick, P., & Lanzon, A. (2021). Group coordinated control of networked mobile robots with applications to object transportation. IEEE Transactions on Vehicular Technology, 70(8), 8269-8274.

50. Sarcinelli-Filho, M., & Carelli, R. (2023). Control of multi-robot systems. In Control of Ground and Aerial Robots, Cham: Springer International Publishing, 99-144.

51. Chen, S., O'Brien, M. J., Talbot, F., Williams, J., Tidd, B., Pitt, A., & Arkin, R. C. (2022, October). Multi-modal user interface for multi-robot control in underground environments. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 9995-10002.

52. Guo, S., Cao, S., & Guo, J. (2021). Study on decentralization of spherical amphibious multi-robot control system based on smart contract and blockchain. Journal of Bionic Engineering, 18(6), 1317-1330.

53. Orr, J., & Dutta, A. (2023). Multi-agent deep reinforcement learning for multi-robot applications: A survey. Sensors, 23(7), 3625.

54. Yevsieiev, V., & et al. (2024). Building a traffic route taking into account obstacles based on the A-star algorithm using the python language. echnical Science Research In Uzbekistan, 2(3), 103-112.

55. Gurin, D., & et al. (2024). Web site reliability analysis using the python parsing method. Journal of Universal Science Research, 2(5), 113-126.

56. Yevsieiev, V., & et al. (2024). A Program for Analyzing the Structure of a Web site Development Using the Parsing Method Based on the Python. Journal of Universal Science Research, 2(4, 172-183.