Select Page

How Can You Run More Mobile A/B Testing Experiments?

Imagine you and your team just a released a redesign of the login section of your software product. Shortly after the release you notice that the numbers of newly registered users dropped almost to zero. But why? Is it because the change of the register button to a register link? Or is it the new naming of the register element?

Maybe this kind of change should have been tested before with an A/B test.

What is A/B Testing?

A/B testing in the context of digital software products gives a team the power to test and compare ideas like in a science experiment with real users. Basically everything can be tested: the color of buttons, new text for navigation elements or a complete different version of the product. A/B testing will help teams understand the impact of the changes on a subset of real users to get real insights.

Every A/B test starts with the creation of a hypothesis. A strong hypothesis is needed to describe the A/B test upfront and to explain the expected outcome. You can use the pattern:

Changing__________ from__________to__________will__________.

An example:

“Changing the register button from button to link will increase the sign-up rate.”

This is just a theoretical assumption but it will help you to solve the problem.

Once the hypothesis is created, it’s important to define a test user group. Group A is the control group, which still gets the existing product and group B gets the modified version. If one group is performing better according to the hypothesis, then it should be applied.

How Can You Run More A/B Testing Experiments

So how can you start with A/B testing in your mobile project? First, you need to find the A/B testing vendor you want to use in your apps.

All vendors provide an A/B testing SDK (Software Development Kit) that needs to be added to your application. The integration is a one-time task and should not take too much time.

Once the SDK is installed, you can start to create an A/B test on the mobile app using the dashboard of the A/B testing vendor. Most of them offer a web based dashboard, that can be used to configure the A/B test, as well as select the users for the control and testing groups. From the dashboard, you can see the progress of the A/B test as well as the results once the test has been finished.

However, there are two things you should keep in mind when adding an A/B testing SDK.

  1. Users need to update the mobile app with the included SDK in order to be part of an A/B test. If users don’t have the right version installed, make sure to target the test group based on the app version.
  2. Adding the SDK will increase the size of the mobile app. Make sure to update the app to the latest version in case the user doesn’t have enough memory left on the device.

Benefits of A/B Testing

By adding an A/B testing SDK to your mobile app, you gain a number of benefits that are worth increasing the app size as well as the one-time integration effort. The following list includes an excerpt of the benefits you gain:

  • You can run multiple A/B tests at the same time
  • You can extend the A/B to an A/B/C../Z test
  • The findings of an A/B test can result in higher conversion rates
  • It will be easier to make decisions based on real customer usage data and statistics
  • You gain more insights about customer behavior

Apps targeting young people

Naturally, banks and online brokers are also increasingly offering mobile solutions for stock trading. However, this new group of fintech startups has a different structure than traditional providers. As international apps with social media appeal, they are aimed at a particularly young target group of 25- to 35-year-olds who want not only access to stock trading but also a new kind of user experience. It has become clear that accessibility and user-friendliness are key selling points for these new investment apps. For example, according to Bitkom’s Digital Finance Report 2020, 40% of respondents expressed the expectation that “smartphone apps’ ease of use for stock and securities transactions will enable more people to benefit from companies’ performances.”

In a nutshell, the easy access via smartphones makes these “neobrokers” so appealing. Clear design, community integration, and ease of entry has turned UI/UX into an actual product.

Special opportunities – special risks?

Many apps have little to no limit on how small a trade can be, making it possible to buy fractional shares. As mentioned, they charge very low fees — or none at all — and are available outside of regular trading hours. The apps clearly aim to lower the entry threshold for stock trading, and sometimes lure new users with free shares. On the flip side, the apps offer no or minimal investing advice, unlike traditional brokers. Consequently, purchasers must do their own research outside of the app, using articles, forums and social media. This aspect has raised suspicions in the German market. In the survey undertaken for the Bitkom Digital Finance Report referenced above, 69% of respondents stated that “an advisor’s input is absolutely key to making good investment decisions.” As a result, the separation of professional advisory services and the gamification of trading stocks carries certain risks, especially for inexperienced users.

Too much power?

The potential dynamics unleashed by direct market access were demonstrated in an interesting case study in January. Small investors coordinated a purchase of GameStop stock via Reddit to prevent a decline in the company’s value, on which hedge funds had speculated. In fact, the Reddit community’s actions were so successful that U.S. authorities are now investigating the possibility of market manipulation. Outrage erupted, however, when Robinhood simply suspended trading in GameStop shares at the height of the buying frenzy.

Ultimately, the neobroker did have a good reason for halting trading. The security it had deposited with clearinghouse DTCC was insufficient to match increased trading volume. However, this episode illustrates that some luster has fallen from the new market power of small investors: Even trading apps do not eliminate the intermediary function; they only replace it, sometimes with even more opaque conditions than before.

The outlook is promising

And yet, neobrokers are attracting young investors by reinventing the process of investing and stock trading. With pleasing designs and customer experiences geared toward millennials, these apps will be able to gain many users in the next few years. At that point, they will have to show that they can keep up with the momentum that they created. Users expect apps, acting as financial service providers and managers of highly sensitive data, to be error-free at all times and in all places – and rightly so. User trust and compliance with financial rules will play a crucial role in determining whether neobrokers will remain competitive as market penetration continues.

However, the new investment apps’ penetration of the DACH market is still at an early stage. Established providers, especially banking apps, may leverage the trend by incorporating a more attractive UX and simplified investment features into their existing apps. For example, a whitepaper from the Sparkassen Innovation Hub on the topic of changing values recommends “opening up products to small investment amounts” as well as “using a clear, appealing interface (UI), playful elements for data entry and maintenance, [and] the use of status and progress indicators to guide users through processes” to attract a new group of potential investors.

One thing is certain: The phenomenal growth of investment and trading apps, especially in Germany, could be a precursor to interesting developments in the coming years.

Daniel Knott
Daniel Knott
Mobile Testing Expert
Published On: February 22, 2019
Reading Time: 4 min

Avoiding The Hidden Cost of Payment Failures

Learn how you can improve your payment experience and protect revenue across markets, methods, and moments.

Claude Code Crash Course: How to Master Claude Code for QA Engineering

Discover how you can use Claude Code to safely implement AI tools and skills to accelerate your testing lifecycle.

Why Test Plans Fail in the Real World

Your current test strategy might be the reason products fail. Find out why test plans must be adaptable.

How to Conduct AI Evals: Best Practices for Building AI Confidence

Discover why AI evals are crucial for releasing with confidence and get best practices for improving AI system performance.

Crowdtesting vs. System Integrators

Compare system integrator testing with managed crowdtesting services to find the right QA approach for real-world digital quality.

EU AI Act: A Practical Guide for QA Leaders

See how the EU AI Act affects QA and product leaders — and how to adapt testing workflows ahead of compliance deadlines.
No results found.
This site is registered on wpml.org as a development site. Switch to a production site key to remove this banner.