Payment Testing Pilot: A Logical First Step

At the end of it all – the customer journey and choosing a product or service – lies the ability for customers to pay and for merchants to be paid. It’s why companies are in business.

The payment space is packed with complexity. Beyond the variety of payment instruments (PIs), there are market trends – things like buy now pay later, invisible payments, biometric payments and cryptocurrency payments. These are all vying for attention, and contribute to the possibility of creating friction, or worse, immobility in the payments process.

This blog looks at how Applause helps customers address their individual payment concerns in short-term pilots and ongoing engagements.

Why is payment testing important?

It may seem like a simple question with a simple answer, but some firms do not test their end-to-end payments flow. They may rely on customer service reps to gather information and feed issues back to engineering. Some companies use their own cards for testing, dummy cards or do testing in the staging phase of the software development lifecycle. Happy-path testing like this, however, does not generally connect to the final payment process or portal. It’s everything right up to the purchase, and it’s simply not the same as testing in production with real, valid PIs.

Without testing in the real world, merchants, global card companies, POS vendors and payment providers run the very real risk of having transactions fail. When this happens, companies lose revenue and customer trust. Failed transactions, so often bearing no fault of the customer, feel personal. Why was I declined? Is there something wrong with my bank or financial health? There is substantial risk involved when organizations opt to not test, or not thoroughly test by using real PIs in the hands of real users.

Examples of significant payment issues

In an engagement with one of the world’s leading software firms, Applause audited certain countries for payment issues. Applause suggested these audits as a best practice for ensuring market acceptance. In Brazil, a cash/debit-heavy country, we found that 100% of the debit card attempts were being declined. The software firm was losing out on all debit purchases in Brazil due to an inadvertent mistake by its payment processor.

In another example, a major card company came to us for help in Europe. Their card data for U.S. transactions was pristine. They could look up any transaction or failure and easily track down anything they needed. However, when U.S. card holders traveled to Europe, the company was only getting back about 50% of the data they needed to execute. They needed in-market users who were using out-of-market cards, as if they were traveling at various points of service — hotels, restaurants, gift shops, etc. Applause gave the company a wealth of data on each test — pictures of receipts, authorization codes, terminal IDs — everything they needed to track down a problem. This enabled the card issuer to speed up their quality assurance with partnerships in other countries, and help them to onboard other international acquirers on their networks.

What is typically done in a payment testing engagement?

Well-conceived and -built payment testing services are designed to test payments (including transfers, refunds and more) in the real-world, using tester-owned PIs to ensure a smooth transfer of funds. Payment testing can be performed around the world using credit cards, digital wallets, loyalty programs, bank accounts, payment platforms and many other payment methods.

When we work with organizations in pilots and ongoing engagements, Applause testers provide relevant information regarding the payment transaction tests. This includes but is not limited to:

• bugs/technical blockers

• transaction details

• screenshots of receipts and billing or account statements

• payment instrument or account information

• POS terminal information

• device/OS/browser versions

• transaction videos

• adherence to payment standards like PSD2, taxation laws and other compliance points

For pilots, we typically suggest a range of 10 to 50 transactions. These may be broken into subgroups: an initial test of 10 transactions, then another 15 or so transactions. From these initial batches, we decide what areas to retest. For payments testing, we tend to focus on the spread of payment instruments. As an example, we’d test three different payment instruments in four different cities in Spain for purchases under five Euros. In addition, we typically include a handful of feedback questions that are important to the company conducting the pilot. Globalization and localization feedback is often critical to understanding an end-user in a potentially unfamiliar market, providing cultural and language feedback on things that might impede the payment flow. Examples include when a company should use the word “proceed” instead of “check out,” or when a date/number format is incorrect for the country.

Most of the organizations that engage with us in payment testing pilots have custom needs. A leading travel technology company came to Applause because it had a problem with customer support issues. Customers were not seeing an identifiable charge from the company on their statements; instead, they would see the payment provider’s name, unfamiliar to the customer. The travel tech company had no visibility into what happened on the consumer end and were completely unaware of the issue or how to remedy it. They asked us to find out what various customer statements contained – different types of banks, various types of bookings such as flights, hotels, rental cars, etc. We provided a wide assortment of data points to enable the company to work backwards to the source of the statement issues.

Three primary types of payment testing

In our pilot or regular engagements, Applause executes the following types of testing:

Active testing - any payment test we conduct on a company’s behalf based on its testing requirements and criteria. Applause sources testers with the PIs required in local markets, and we execute digital and/or physical transactions.

Passive testing - employed when a company's engineering team requires a tester's full PI details to execute the test using its internal system. We provide an encrypted file containing a tester's credit card number, CVV, expiration, address, etc.

Split testing - used when a PI requires the involvement of the PI owner. We provide a tester's PI details and companies execute the test with the involvement of the tester. This is most common with alternative PIs and/or in pre-production environments.

Payment testing beyond pass/fail

We’re often asked to help test/launch a new payment service for an organization that is not familiar with the service. Organizations frequently look to understand new payment options, how they work, and even when they work, if there anything about the experience that could be better from a user perspective.

In one such case, an organization wanted to know everything about the payment service from the ground up, such as how it is typically used and any failures occurring before the service was launched. Although the payments were successful, could anything about the experience be improved? How did their user experience compare to the competition? Applause collects this in-market feedback and provides dashboards that list many data points beyond this more subjective user feedback.

Though we can do intensive UX studies, for payment pilots, we typically give general user feedback, such as first capturing answers to about five questions that the client has around the payment flow experience. At times, this feedback will lead to a full usability study.

Reporting output for payments testing

We collect many utilization and tracking data points in payment engagements, typically illustrated in easy-to-read dashboards. Here is an example of some of the output:

• total transactions completed

• transit transactions completed

• pending transactions

• number of cities tested

• number of unique merchants tested

• number of unique merchant locations tested

• number of different credit card types tested

• number of unique card numbers tested

• number of successful contactless transactions

• number of failed contactless transactions

• number of invalid contactless transactions

• number of blocked transactions

• number of chip transactions

• overall transaction success rate

Payments engagements depend on the client

This blog gives a quick view of what we do in pilots as well as on larger, ongoing payment testing engagements, but each engagement is different and custom crafted to meet a company’s needs. A major component of scoping a payments pilot revolves around fully understanding what specific customer needs exist, and then validating that testing for those needs is feasible and/or the best approach for an initial engagement.

Only through thorough project scoping and planning can the best pilot test be designed. And, of course, only through testing with real, live payment instruments can you confidently know your customers — regardless of payment method — can transact successfully.