Machine Learning Approaches for Automated User Interface Component Recognition: A Comprehensive Review

This paper explores the role of machine learning algorithms in the automated recognition of graphical user interface (GUI) components to enhance UI/UX testing processes. By reviewing methods based on computer vision and deep learning, including Convolutional Neural Networks (CNNs) and YOLO models, it is demonstrated that these techniques offer higher accuracy and speed compared to traditional approaches. Challenges such as platform fragmentation, dynamic elements, and computational resource limitations are discussed, with proposed solutions including hybrid approaches and AI-based testing. The findings confirm the positive impact of machine learning on improving the quality and efficiency of UI testing.

Keywords:

Machine Learning , UI Testing , Computer Vision , YOLO Model , Convolutional Neural Networks (CNNs)