Claims:We claim:
1. A system for automated testing of graphical user interface (100) comprising:
a first computing device, the first computing device having a first processing unit coupled to a first visual display unit, a first input unit, and a first memory unit being configured with an application module having a graphical user interface (GUI) under test, wherein the processing unit functions based on the instructions configured within the memory unit,
an image capturing module configured within the first computing device, the image capturing module being adapted to capture screenshots of the first computing device at regular intervals upon activating the application module; and
a second computing device operably connected to the first computing device, the second computing device having a second processing unit coupled to a second visual display unit, a second input unit, and a second memory unit being configured with an image processing module having a set of instructions for processing the screenshot to locate the user interface (UI) elements on the screenshot, and a test script execution module having a test script database with a plurality of test scripts therein, wherein the processing unit functions based on the instructions configured within the memory unit;
wherein the screenshots of the first computing device taken by the image capturing module is analyzed by the image processing module to locate the position of a UI element on the GUI and execute the test scripts on the respective UI element on the application module in reference to the UI element position identified on the screenshot.
2. The system (100) as claimed in claim 1, wherein the first computing device is any one of the devices selected from a smartphone and a personal computer.
3. The system (100) as claimed in claim 1, wherein the second computing device is a personal computer.
4. The system (100) as claimed in claim 1, wherein the image processing module having OpenCV library for locating the UI element in the screenshot.
5. The system (100) as claimed in claim 1, wherein the first computing device and the second computing device are connected through a medium supporting full duplex communication.
6. The system (100) as claimed in claim 1, wherein the first computing device and the second computing device are connected through a medium selected from wired communication link and wireless communication link.
7. The system (100) as claimed in claim 1, wherein the first and second input unit includes alphanumeric keyboard, mouse and like.
8. A method for automated testing of graphical user interface (50) having a first computing device in communication with a second computing device, an image capturing module configured within the first computing device, the first computing device having a first processing unit coupled to a first visual display unit, a first input unit, and a first memory unit being configured with an application module having a graphical user interface (GUI) under test, second computing device having a second processing unit coupled to a second visual display unit, a second input unit, and a second memory unit being configured with an image processing module having a set of instructions for processing the screenshot to locate the user interface (UI) elements on the screenshot, and a test script execution module having a test script database stored with a plurality of test scripts, the method comprising steps of:
opening the GUI of the application module on the first computing device;
capturing screenshot of the display of the first computing device at predefined intervals by the image capturing module;
processing the screenshot after every iteration by the second computing device;
identifying texts and text screen position on the screenshot by the second computing device;
identifying the index of the center of a word/ sequence of word representing text of a UI element on the screenshot by the second computing device;
locating the position of the UI element in reference to the identified texts on the screenshot and thereby the position of a corresponding UI element on the GUI of the application module on the first computing device, by the second computing device;
loading the test scripts within the test script database configured within the second computing device, wherein the test script instructions are associated with testing of GUI; and
executing test scripts on the respective UI element on the application module in reference to the UI element position identified on the screenshot by the second computing device.
9. The method (50) as claimed in claim 5, wherein the text identification is performed by using optical character recognition techniques.
Dated this on 8th day of July 2020

Ragitha K
Agent for Applicant
(IN/PA/2832)

, Description:Field of the invention:
The present invention relates to information technology and more particularly relates to automated testing of graphical user interface of computing devices by utilising the possibilities of image processing.
Background of the invention:
The graphical user interface (GUI) is an interface having interactive visual components through which user can interact with the electronic devices like computers, smart phones etc. The elements of GUI include pull-down menus, buttons, scroll bars, iconic images, wizards, text box, check box, drop down lists and so on. Normally, the GUI of the applications are adjusted according to various standards based on the objectives of the application, market requirements, technology and amendments made to the application. Hence testing of these GUI elements are also equally important to identify the defects and ensure whether the application meets all the specifications it claims.
GUI testing refers to the systematic process of generating test cases in order to evaluate the functionality of the application and its control elements. Graphical user interface testing procedures are either manual or automated. The manual testing of computer systems is generally conducted by a test engineer who identifies defects by running a plurality test scripts through a defined series of steps and observing a result after each step. Such testing process are time consuming and extremely expensive because, series of steps that are intended to be thoroughly exercised and the product functions as well as the scenarios that have failed has to be re-executed.
The automated test tools are normally available under a variety of licenses and are supported by a variety of platforms. Popular test automation tools are selenium webserver, QTP, Unified Functional Testing (UFT), Squish GUI Tester etc. Such test tools require human resources for locating the UI elements on the application screen and executing the test scripts. Also, to develop automation scripts, the test engineer need understanding of programming languages such as Java, Python, .Net etc. Additionally, the issues associated with dynamic contents like pop up notifications, amendments in web applications and network latency may introduce element variability and unpredictability on GUI. Such unpredictable software changes may go against the actual concept of using the test scripts of one computer towards another automatically.
An image based automation testing technique currently available is provided a node for automatically testing a computer program by using a predefined GUI. The method of testing involves determining whether an image object in the screen shot matches an image object retrieved from the data and object image database; calculating a target position on the screen of a matched image object based on data retrieved from the data and image object database; and activating a control function for controlling the predefined GUI based on the program script instructions and the calculated target position. The system captures images of UI control elements during recording and while executing it finds the region of screen which has similar image pattern. The method found to be time consuming in capturing the object image for each interaction and it may fail for different resolution of machine color values.
Accordingly, there exists need to provide an automated testing of GUI of computing devices that would eliminate all the above mentioned issues associated with testing of UI elements.
Object of the invention:
An object of the present invention is to test a graphical user interface of a computing device automatically.
Another object of the present invention is to eliminate the dependency on human resources in testing of the graphical user interface of a computing device and reduce the risk of human errors.
Yet another object of the present invention is to allow a test engineer to create test scripts without having any knowledge on technical properties of the GUI elements.
Summary of the Invention
The present invention a system for automated testing of graphical user interface comprising a first computing device, an image capturing module configured within the first computing device and a second computing device. The first computing device having a first processing unit coupled to a first visual display unit, a first input unit, and a first memory unit. The first computing device is configured with an application module having a graphical user interface (GUI) under test. The image capturing module is adapted to capture screenshots of the first computing device at regular intervals upon running the application module. The second computing device having a second processing unit coupled to a second visual display unit, a second input unit, and a second memory unit. The second memory unit is configured with an image processing module and a test script execution module having a test script database stored with a plurality of test scripts. The screenshots of the first computing device taken by the image capturing module is analyzed by the image processing module to locate the position of a UI element on the GUI and execute the test scripts on the respective UI element on the application module in reference to the UI element position identified on the screenshot.
Brief description of the drawings:
The objects and advantages of the present invention will become apparent when the disclosure is read in conjunction with the following figures, wherein
Figure 1 shows a block representation of a system for automated testing of graphical user interface in accordance with the present invention,
Figure 2 shows a flow diagram of automated testing of graphical user interface in operation in accordance with the present invention.

Detailed Description of the invention:
The foregoing objects of the invention are accomplished, and the problems and shortcomings associated with prior art techniques and approaches are overcome by the present invention described in the present embodiments.
The present invention provides a system and method for testing a graphical user interface (GUI) of a computing device. The method executes a plurality of test scripts on the UI elements based on the information related to the position and location of the UI elements extracted from the screenshots of GUI. The screenshots of GUI are taken at predefined time intervals and are analyzed by using a plurality of image processing algorithms.
The present invention is illustrated with reference to the accompanying drawings, throughout which reference numbers indicate corresponding parts in the various figures. These reference numbers are shown in bracket in the following description.
Referring to the figures from 1 & 2, there is shown a system for automated testing of graphical user interface (hereinafter ‘the system (100)’), in accordance with the present invention. The system (100) comprising a first computing device, a second computing device and an image capturing module.
The first computing device comprising a first processing unit coupled to a first memory unit through a first memory interface. The processing unit is also coupled to a first network interface unit, a first visual display unit, a first input device and other peripheral devices. The first input devices include an alphanumeric keyboard, mouse and like. The first processing unit includes a plurality of integrated circuits, or chips, that are designed to work in accordance with a set of instructions configured within an internal memory.
The second computing device comprising a second processing unit coupled to a second memory unit through a second memory interface. The second processing unit is also coupled to a second network interface unit, a second visual display unit, a second input device and other peripheral devices. The second input devices include an alphanumeric keyboard, mouse and like. The second processing unit include a plurality of integrated circuits, or chips, that are designed to work with a set of instructions configured within an internal memory.
Preferably, the first computing device is a smart phone and the second computing device is a personal computer. The first memory unit is configured with an application module for which the GUI elements are under test. The image capturing module is configured within the first computing device and is capable of taking screenshots (hereinafter referred to as ‘the image’) of the display of the first computing device while the application module is active. Here the screenshot indicates that a digital image which represents the contents of display of a computing device.
The memory device of the second computing device includes a testing module having a collection of testing procedures that are required for examining the functionality of the GUI of the application stored in the first computing device. The testing module includes an image processing module comprising a plurality of algorithms for analyzing the images and test script execution module ideal for GUI testing. The test script execution module having database of a plurality of test scripts prepared in accordance with the test scenario.
The first computing device is operably connected to the second computing device through a medium. The connection between the first and second computing devices allows full duplex communication between the devices. The connection between the first computing device and the second computing device is made through a mode selected from wired communication and wireless communication. In the embodiment, first and second computing devices are connected through USB (Universal Serial Bus) cable.
In a preferred embodiment, the image processing module employs OpenCV library for image analysis. The test script execution module includes open source test automation tools like Appium for testing smart phone devices based applications and PyAutoGui library for testing personal computer based applications.
In an alternative embodiment, the first and second computing devices are the personal computers. In another alternative embodiment, both the testing module and application module for which the GUI under test are configured within a single computing device. The image capturing module configured within the computing device takes the screen shots of GUI of the application module at regular intervals and feed the same to the testing module.
Again, referring to the figures 1and 2, a method of automated testing of graphical user interface (50) hereinafter referred to as ‘the method (50)’ is described in accordance with the present invention. The method (50) is described in conjunction with the system (100).
In the first step, after activating the application module in the first computing device, the image capturing module captures the screenshots (herein after referred to as ‘the images’) of the display of the first computing device at predefined intervals. Further the images are transported to the second computing device, such that the images can be analyzed by the image processing module therein.
The captured images are processed by the image processing module having a plurality of algorithms to process the color scheme, fonts and application layout in the images to identify the text. The images are subject to a plurality of transformations for obtaining a best result for each screenshot. The transformation procedures carried out in specific, are listed below.
o converting the color image into grayscale image,
o generating gray scale images before subject to threshold conversion,
o applying bilateral filters to the original images,
o converting the original image into canny edges,
o smoothening the images by applying median filter and then applying higher threshold value,
o applying mean adaptive threshold on the images,
o applying Gaussian adaptive threshold on the images,
o after smoothening the image by median filter, the images are subject to morphological transformation.
The best transformation reference is retained in an image database for next execution so that images need not be go through all the above mentioned transformations for each iteration.
The image processing module further extracts all the required information that may leads to the detection of presence of texts and thereby UI elements. In a preferred method, optical character recognition (OCR) techniques are used for text identification.
After the text identification, text screen position is calculated and accordingly, the index of the center of the word or sequence of word representing text of the element. Further the coordinates of the center of the word are determined to locate the position of UI element in reference to the identified texts. For example, UI elements such as text field/checkbox/radio button will be either on the right, below or above the text for that element. After identifying the text position, the image processing module locates a UI element in a close proximity to the text. The pixel distance and direction of the UI element from center of the text are provided through a script. This distance is interpolated for different screen resolution to calculate the position of the UI element.
The information related to the position of UI element in reference to the image is further passed to the test script execution module in the second computing device. The test script execution module takes this information to locate the position of UI element on the GUI of the application module in the first computing device. Further, the test script within the test script database are loaded and starts executing the test scripts upon the UI element. The test scripts include a set of instructions to execute a variety of actions on the UI element, and that implies performing tapping, dragging, scrolling, entering alphanumeric keywords, and so on.
The images of the display of the first computing device are taken after regular intervals while executing the successive stages of the application module. The process of operation continues till, identifying the last UI element in an image representing the final stage of the GUI of the application module and executing the predesigned test scripts thereon.
Advantages
1. The system (100) reduces complexity in testing of a variety of application modules having different layout, color schemes, fonts; by adopting a plurality of image processing algorithms for the identification of a UI element.
2. The method of automated testing of graphical user interface (50) allows the user in creating test scripts without having any knowledge on the technical properties of the elements.
3. The method of automated testing of graphical user interface (50) offload the task of execution of test cases to computing devices reduces the dependency on human resources and also reduce the risk of human errors.

The foregoing descriptions of specific embodiments of the present invention have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the present invention to the precise forms disclosed, and obviously many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the present invention and its practical application, and to thereby enable others skilled in the art to best utilize the present invention and various embodiments with various modifications as are suited to the particular use contemplated. It is understood that various omissions and substitutions of equivalents are contemplated as circumstances may suggest or render expedient, but such omissions and substitutions are intended to cover the application or implementation without departing from the scope of the present invention.