AI in UI automation
Artificial intelligence is the buzzword in industry which has been utilized in many domains. AI is used to learn and predict the changes in stock market, sales and what not. QA has adapted to test these AI applications and also started to automate it. Automating an AI application is highly challenging and QA in the current world has started exploring and implementing AI in automating UI tests. AI is changing the way a tester normally tests the application. There will be ‘n’ number of possibilities, but we are mainly concentrating on AI in UI automation.
Benefits of AI in automations
- Reduction in maintenance costs
- Easy and quick scripting
- Remarkable increase in automation coverage due to points above
- Anyone without programming background can also contribute in automation
We can utilize Artificial Intelligence mainly in 2 ways to automate scenarios
Interaction with objects in DOM
- To interact with objects in DOM
- To script test cases
To interact with objects in DOM
- Smart object identification
One of the most challenging tasks for an automation engineer is to maintain the scripts. Developers can change the class name or other attributes used to identify the objects, this will eventually lead to failure of test cases. Once the tests are failed testers should check if it is a script issue or an application issue and if it is a script issue need to debug and fix it. These activities consume time and all these actions are due to a small change in class name or attribute value. What if the tool automatically identifies these small changes and continues execution. Testers don’t have to spend too much time maintaining the objects. If there is any change in attribute value, the tool will automatically identify and proceed the test, later asks confirmation to update the values.
- Identifying objects as the humans does
This is similar to the above mentioned point. Humans identify an object based on some intelligence. If the color or position is changed, still can identify it. But for a tool it was specific. If a position is changed i.e. an extra html tag is added in the xpath used[even if it is relative xpath], tools fail to identify the object. AI helps the tools to identify based on the related objects, parent or neighbour as specified in the config. This will help smooth execution of scripts.
- Intelligent recommendation of objects.
AI helps to recommend objects if it is not identified. It will also automatically acknowledge when application changes have been made, and prompt the tester to make the necessary updates, or to delete unused application components.
Script Test cases [Spidering the application]
This is another important area where AI plays a vital role in UI automation, to write test scripts automatically using ML Techniques. For example there are some tools available in the market which allow you to write test scripts by crawling through applications after the url to be automated is given. As the tools spiders through the application, it will collect data, screenshots, download html of all pages it navigated through. And during execution, it will compare with these data, screenshots and html pages and come up with a status report. It can also capture the load times of each page on specific network speed and compare it with the next run on the same network speed. Some of the deviations noticed while running might be valid, if we flag it as valid and the machine will learn and execute the subsequent runs accordingly.
This approach is still at an infancy stage, as it won’t cover the exact business flows of an application, with the current pace we are advancing that too would be possible in near future.
How Implement AI in UI automation
- First of all understand the difference between AI and ML
- Only once this difference is understood, we could select the technology to use
- Define the UI automation needs.
- From the above mentioned points finalize where to implement the AI support for UI automation.
- Prioritize the requirements
- First of all we should analyze potential financial benefit on implementing AI in UI automation. To implement AI in conventional automation is a tedious task, after implementing it if the ROI is not worth we should not give it a try. This may be popular today and by the time we implement it, if new tools with better features comes into the market, our implementation would not be sufficient
- Evaluate our Internal resource capabilities to implement AI in UI automation
- There is always a huge difference between what we want to achieve and what we can do within the given time frame and given resources.
- We need to check if we want to
- Build the software from the scratch or to integrate features from different open source tools to meet the need
- We can also collaborate with another vendor who are already working on similar project
- If we really want to build up a product and do not have the internal capabilities, outsource it to a company which is working on similar projects
- It is always to good approach a domain expert and get the expert advice on the implementation we have selected
- Implementing AI in UI automation is not same as developing an automation framework, it is good to get an expert advice on plan of implementation we had selected
- Select algorithm that will help us to achieve the requirement
- There are n number of algorithms available in the industry, we need to shortlist two or three algorithms that can serve the purpose
- Prepare the data to train the model
- To get a fair result from AI we need to have fair data or example for algorithm to learn and achieve what we are expecting
- Getting data is as hard as implementing AI projects as the data
- Should be sufficient to train the AI algorithm
- Should be valid, [should not be incorrect or unclear ]
- Should also have the attribute which is needed for the algorithm to learn behavior.
- And most importantly, this data should be frequently updated
- Once Algorithms, data and plans are ready, start implementing. But we should always start small
- Work on the shortlisted algorithm with a smaller data set and see what all refinements are needed.
- By this way we can compare the algorithms result and select the one that gives most accurate result