DevOps May Encourage Speed, But Don’t Neglect Quality

The quality of digital products now offers a key competitive advantage across industries that traditionally relied on brick and mortar. To sharpen their focus on digitalization, many brands from industries like finance, retail and telco are rebranding as technology companies and embracing Agile. This trend took off around five years ago, when media like Reuters and Inc. started publishing pieces like “All companies are tech companies now” and “Why even a salad chain wants to call itself a tech company.”

Yet digital transformation is not easy. It starts with changing mindset and redefining ways of working, which for many means adopting or improving their approach to DevOps.

However, many companies working in DevOps struggle to deliver technology at the right quality that provides a genuine competitive advantage in the digital world. Many dare not admit it for fear of being dismissed as old-school, but DevOps as it is being practised today in many companies is not realising its potential. Too many organisations focus on release velocity at the expense of quality, with the result that most companies today are simply pushing poorer products to market faster.

This article urges companies to reconsider their current DevOps practices to put quality and stability front and centre. Here’s why — and how to get started.

A misplaced focus on speed jeopardises quality

Teams that are undergoing digital transformation are caught in a rat race in which they pride themselves on their ability to release software multiple times a day, or even after every code commit. While it is true that keeping up with the pace of technological development means releasing new features at increasingly higher frequency, it should not come at the cost of quality.

Yet, quality assurance is often seen as a bottleneck and is compromised in the quest to achieve greater speed. A dedicated focus on quality is often lost, as responsibility for the quality assurance function moves to developers and operations in the name of resource optimization. An Applause survey found that over 52% of teams that rely on developers for testing agreed that shifting left is impacting their developers’ productivity. This results in declining quality over time.

Driving DevOps maturity with in-sprint testing

In order to achieve quality while maintaining speed, the recommended DevOps approach is to automate as much as possible. This includes both functional tests written by developers as part of development, as well as regression tests written by QA.

Another way to close the quality gap is in-sprint testing, which is usually achieved with QA team members embedded in the sprint teams. By testing individual features as they are developed, companies can both increase development velocity and improve quality. This is because testing early and often avoids stumbling upon major issues far along the development process, when the financial and time cost of remediation can be crippling. In-sprint testing also ensures that no feature is released without having undergone a set amount of testing, which improves product quality.

To ensure no tests are overlooked or rushed prior to the end of a sprint, companies must automate in-sprint tests, or at least automate the majority. This is especially true of companies that run continuous integration, continuous delivery (CICD) cycles that push updates multiple times per day.

While many companies recognise the need for in-sprint automation, far fewer have implemented it successfully. Capgemini’s World Quality Report 2022/3 found that DevOps teams only achieve the benefits of automation around half of the time and integration of CI/CD and automation is below expectations. The teams that did best tended to have more mature Agile processes and saw better results when they automated early on in the test cycle.

Going one step further with customer journey automation

Companies that have reached the point of carrying out thorough, effective in-sprint automation in DevOps are already advanced compared to their peers. However, in-sprint automation alone is not sufficient to hit the right quality benchmarks.

A key deficiency of DevOps is that many developers only test features in isolation, with no consideration for the end-to-end customer journey. Even if developers execute best-in-class in-sprint automation, excellent individual features mean little if they do not perform well with each other in the context of a wider product. The key to top-notch overall product quality is to have both a detail-oriented and holistic view inside the delivery pipeline.

The best way to achieve this is with customer journey automation. In addition to automating in-sprint tests on individual features, companies need to automate tests that validate entire customer journeys involving multiple different systems. However, this can be easier said than done, especially in industries like telco, where customer journeys are very complex. To onboard a new customer, a telco may need to allocate them a free account number, process their registration, issue a SIM card, organise a landline connection, handle customer requests and set up payments. In other words, a single customer journey may involve five to ten major applications and numerous touchpoints.

How can companies execute these end-to-end test cases? Not all teams have end-to-end environments that allow for complex processes such as this to be automated. Developers may find themselves waiting several minutes for a system to return a required output before they can continue testing the rest of a customer journey, at which point the journey is simply not a good candidate for automation. To succeed at customer journey automation, companies must design for testability and consider the need to test full customer journeys, which could involve virtualized systems to help simulate end-to-end-scenarios.

Companies that set themselves apart as leading digital innovators understand the value of the right quality at the right speed, which they achieve by testing in-sprint, automating in-sprint and automating testing across entire customer journeys.

Quantifying success using the right metrics

One way companies can measure their DevOps performance is to observe the DevOps Research and Assessment (DORA) metrics, of which there are four. They are divided into metrics that measure velocity (deployment frequency and lead time for changes) and those that measure stability (change failure rate and time to restore service). Typically, companies focus on the former, especially deployment frequency, as a measure of their performance. In the process, they don’t think about how many changes are failing. In practice, the lower the lead time for a change, the higher the probability of failure — if proper quality strategy is not followed.

Close the feedback loop with crowdtesting

One of the fundamental principles of Agile and DevOps is to develop fast, collect user feedback on products and reiterate product development based on the feedback received. However, many DevOps teams miss out these final two steps, usually because they are unsure how to best go about it. How do you get the right feedback? How do you ensure that there is no friction in your end-to-end user journey?

Crowdtesting complements DevOps because it provides companies with an efficient way to get customer feedback. With canary deployments and feature flags, each feature can be sent to a specialised tester team for feedback that can be directly implemented in the very same sprint. Crowdtesting’s scalable, flexible model also meets the pace of Agile development, as customised, scalable tester teams can be drafted in as and when needed, day or night.