Select Page
Blog - 3 Ways Machine Learning Can Improve Test Automation

3 Ways Machine Learning Can Improve Test Automation

Scaling test automation and managing it over time remains a challenge for DevOps teams. Development teams can utilize machine learning (ML) both in the platform’s test automation authoring and execution phases, as well as in the post-execution test analysis that includes looking at trends, patterns and impact on the business.

Before diving deeper into how ML can help during both of these phases of the test automation process, it is important to understand the root causes of why test automation is so unstable when not utilizing ML technologies:

  • The testing stability of both mobile and web apps are often impacted by elements within them that are either dynamic by definition (e.g., react native apps), or that were changed by the developers.
  • Testing stability can also be impacted when changes are made to the data that the test is dependent on, or more commonly, changes are made directly to the app (i.e. new screens, buttons, user flows or user inputs are added).
  • Non-ML test scripts are static, so they cannot automatically adapt and overcome the above changes. This inability to adapt results in test failures, flaky/brittle tests, build failures, inconsistent test data and more.

Let’s dig into a few specific ways that machine learning can be valuable for DevOps teams:

Make sense of extremely high quantities of test data

Organizations that implement continuous testing within Agile and DevOps execute a large variety of testing types multiple times a day. This includes unit, API, functional, accessibility, integration and other testing types.

With each test execution, the amount of test data that’s being created grows significantly, making the decision-making process harder. From understanding where the key issues in the product are, through visualizing the most unstable test cases and other areas to focus on, ML in test reporting and analysis makes life easier for executives.

With AI/ML systems, executives should be able to better slice and dice test data, understand trends and patterns, quantify business risks, and make decisions faster and continuously. For example, learning which CI jobs are more valuable or lengthy, or which platforms under test (mobile, web, desktop) are faultier than others.

With AI/ML systems, executives should be able to better slice and dice test data, understand trends and patterns, quantify business risks, and make decisions faster and continuously.

Without the help of AI or machine learning, the work is error prone, manual and sometimes impossible. With AI/ML, practitioners of test data analysis have the opportunity to add features around:

  • Test impact analysis
  • Security holes
  • Platform-specific defects
  • Test environment instabilities
  • Recurring patterns in test failures
  • Application element locators’ brittleness

Make actionable decisions around quality for specific releases

With DevOps, feature teams or squads are delivering new pieces of code and value to customers almost on a daily basis. Understanding the level of quality, usability and other aspects of code quality on each feature is a huge benefit to the developers.

By utilizing AI/ML to automatically scan the new code, analyze security issues and identify test coverage gaps, teams can advance their maturity and deliver better code faster. As an example, code-climate can review any code changes upon a pull request and spot quality issues, and optimize the entire pipeline. In addition, many DevOps teams today leverage the feature flags technique to gradually expose new features, and hide them in cases of issues.

By utilizing AI/ML to automatically scan the new code, analyze security issues and identify test coverage gaps, teams can advance their maturity and deliver better code faster.

With AI/ML algorithms, such decision making could be made easier by automatically validating and comparing between specific releases based on predefined datasets and acceptance criteria.

Enhance test stability over time through self-healing and other test impact analysis (TIA) abilities

In traditional test automation projects, the test engineers often struggle to continuously maintain the scripts each time a new build is being delivered for testing, or new functionality is added to the app under test.

In most cases, these events break the test automation scripts — either due to a new element ID that was introduced or changed since the previous app, or a new platform-specific capability or popup was added that interferes with the test execution flow. In the mobile landscape specifically, new OS versions typically change the UI and add new alerts or security popups on top of the app. These kinds of unexpected events would break a standard test automation script.

With AI/ML and self-healing abilities, a test automation framework can automatically identify the change made to an element locator (ID), or a screen/flow that was added between predefined test automation steps, and either quickly fix them on the fly, or alert and suggest the quick fix to the developers. Obviously, with such capabilities, test scripts that are embedded into CI/CD schedulers will run much smoother and require less intervention by developers.

An additional benefit would also be the reduction of “noise” within the pipeline. Most of the above mentioned brittleness in testing are not real defects, but interruptions to automation scripts. By eliminating them proactively through AI, teams will get more time back to focus on real issues.

Conclusion

When thinking about ML within the DevOps pipeline, it is also critical to consider how ML is able to analyze and monitor ongoing CI builds, and point out trends within build-acceptance testing, unit or API testing, and other testing areas. An ML algorithm can look into the entire CI pipeline and highlight builds that are consistently broken, lengthy or inefficient. In today’s reality, CI builds are often flaky, repeatedly failing without proper attention. With ML entering this process, the immediate value is a shorter cycle and more stable builds, which translates into faster feedback to developers and cost savings to the business.

There is no doubt that ML will shape the next generation of software defects with new categories and classification of issues. But most importantly, it will increase the quality and efficiency of releases.

Apps targeting young people

Naturally, banks and online brokers are also increasingly offering mobile solutions for stock trading. However, this new group of fintech startups has a different structure than traditional providers. As international apps with social media appeal, they are aimed at a particularly young target group of 25- to 35-year-olds who want not only access to stock trading but also a new kind of user experience. It has become clear that accessibility and user-friendliness are key selling points for these new investment apps. For example, according to Bitkom’s Digital Finance Report 2020, 40% of respondents expressed the expectation that “smartphone apps’ ease of use for stock and securities transactions will enable more people to benefit from companies’ performances.”

In a nutshell, the easy access via smartphones makes these “neobrokers” so appealing. Clear design, community integration, and ease of entry has turned UI/UX into an actual product.

Special opportunities – special risks?

Many apps have little to no limit on how small a trade can be, making it possible to buy fractional shares. As mentioned, they charge very low fees — or none at all — and are available outside of regular trading hours. The apps clearly aim to lower the entry threshold for stock trading, and sometimes lure new users with free shares. On the flip side, the apps offer no or minimal investing advice, unlike traditional brokers. Consequently, purchasers must do their own research outside of the app, using articles, forums and social media. This aspect has raised suspicions in the German market. In the survey undertaken for the Bitkom Digital Finance Report referenced above, 69% of respondents stated that “an advisor’s input is absolutely key to making good investment decisions.” As a result, the separation of professional advisory services and the gamification of trading stocks carries certain risks, especially for inexperienced users.

Too much power?

The potential dynamics unleashed by direct market access were demonstrated in an interesting case study in January. Small investors coordinated a purchase of GameStop stock via Reddit to prevent a decline in the company’s value, on which hedge funds had speculated. In fact, the Reddit community’s actions were so successful that U.S. authorities are now investigating the possibility of market manipulation. Outrage erupted, however, when Robinhood simply suspended trading in GameStop shares at the height of the buying frenzy.

Ultimately, the neobroker did have a good reason for halting trading. The security it had deposited with clearinghouse DTCC was insufficient to match increased trading volume. However, this episode illustrates that some luster has fallen from the new market power of small investors: Even trading apps do not eliminate the intermediary function; they only replace it, sometimes with even more opaque conditions than before.

The outlook is promising

And yet, neobrokers are attracting young investors by reinventing the process of investing and stock trading. With pleasing designs and customer experiences geared toward millennials, these apps will be able to gain many users in the next few years. At that point, they will have to show that they can keep up with the momentum that they created. Users expect apps, acting as financial service providers and managers of highly sensitive data, to be error-free at all times and in all places – and rightly so. User trust and compliance with financial rules will play a crucial role in determining whether neobrokers will remain competitive as market penetration continues.

However, the new investment apps’ penetration of the DACH market is still at an early stage. Established providers, especially banking apps, may leverage the trend by incorporating a more attractive UX and simplified investment features into their existing apps. For example, a whitepaper from the Sparkassen Innovation Hub on the topic of changing values recommends “opening up products to small investment amounts” as well as “using a clear, appealing interface (UI), playful elements for data entry and maintenance, [and] the use of status and progress indicators to guide users through processes” to attract a new group of potential investors.

One thing is certain: The phenomenal growth of investment and trading apps, especially in Germany, could be a precursor to interesting developments in the coming years.

Want to see more like this?
AI Training & TestingAutomated Functional
Dan Cagen
Dan Cagen
Former Product Marketing Manager
Published On: October 12, 2020
Reading Time: 6 min
AI Training & Testing

Avoiding The Hidden Cost of Payment Failures

Learn how you can improve your payment experience and protect revenue across markets, methods, and moments.
AI Training & Testing

Claude Code Crash Course: How to Master Claude Code for QA Engineering

Discover how you can use Claude Code to safely implement AI tools and skills to accelerate your testing lifecycle.
AI Training & Testing

Why Test Plans Fail in the Real World

Your current test strategy might be the reason products fail. Find out why test plans must be adaptable.
AI Training & Testing

How to Conduct AI Evals: Best Practices for Building AI Confidence

Discover why AI evals are crucial for releasing with confidence and get best practices for improving AI system performance.
AI Training & Testing

Crowdtesting vs. System Integrators

Compare system integrator testing with managed crowdtesting services to find the right QA approach for real-world digital quality.
AI Training & Testing

EU AI Act: A Practical Guide for QA Leaders

See how the EU AI Act affects QA and product leaders — and how to adapt testing workflows ahead of compliance deadlines.
No results found.
This site is registered on wpml.org as a development site. Switch to a production site key to remove this banner.