As the software development industry has seen unprecedented levels of digital transformation, the demand for automated testing in the CI/CD pipeline has taken on greater urgency, especially at the early stages.
Also, new advancements in AI are helping developers with some of the biggest challenges in testing: test creation, maintenance, and many of the manual tasks. Many companies have noticed and are spending more on their automated testing initiatives.
Strong testing practices have gotten to be so crucial that they’ve become “the main differentiator between companies that are successful and those that aren’t,” according to Guy Arieli, the QA CTO at Digital.ai.
When companies are looking for an automated testing solution, they’re primarily looking for one that will increase the quality of their releases, increase the speed at which they can be done, and be the most cost-efficient. A common approach among enterprise customers is to seek a vendor that satisfies most of their needs while integrating into their CI/CD pipelines.
“We see customers want a unified solution. You don’t want to be using disparate tools for end-to-end testing of different types of clients,” said Dan Belcher, the co-founder of Mabel. “Increasingly, they’re pushing us to add value to those end-to-end tests with insight around things like performance and visual correctness and other kinds of attributes of quality, because they’re trying to move from pure quality assurance like ‘did I break this core feature?’ to quality engineering: ‘Is the feature better than it was before? Is it faster? Is it more accessible? Is it visually appealing?'”
Chris Haggan, the product management lead at HCL OneTest, said it’s more than just getting the solution with the most features. It’s also about supporting users with the tools they already have and seeing if it fits appropriately with the development organization’s overall approach. Another issue is whether the organization has enough resources to add testing solutions to the mix since they can add complexity.
Where to start?
To start with their automated initiatives, organizations need to build quality into the application earlier, as quality has become a core functional necessity, and testing early on is a vital part of that.
“We see people all the time that want to fully automate everything in weeks. Yes, of course, that is technically possible. Still, it takes time to evaluate what’s important to test, how solutions fit into your CI/CD chain, which generates test data, and so on, according to Kevin Surace, the CEO and co-founder of Appvance. “While no one wants to hear it, the best automation strategy is one that is laid out over a year,” Surace added.
Building in quality comes down to both the culture of the organization and in executing deep code analysis, as well as deep reliability and security at the earliest stages. “If you put it at the end, you really can’t kind of accelerate your delivery. You’re always kind of running into a bottleneck at the end of the process,” said Mark Lambert, the vice president of Strategic Initiatives at Parasoft.
This has led to continuous quality and compliance as aspects that need to be tested in the CI/CD pipeline. Mabel’s Belcher said that as more extensive automated testing becomes available, his one concern is that it will create a test sprawl in which it’s so easy to make end-to-end tests and get the coverage you want in a place that perhaps teams will become more complacent about testing.
“Just because it’s easy doesn’t mean it’s right,” Belcher said. “They have to put more thought into, you know, are these tests accomplishing the objectives that I set out? Are we doing only what is necessary? Do we think about the data? Do we have the right environments? And there’s a lot, a lot more than just the capability to add lots of requests. We keep score by the quality of what’s in production.”
Organizations also need to prioritize those tests that need to be automated first to avoid getting overwhelmed. “What I want to achieve is not more and more tests. What I actually want is a few tests as I possibly can because that will minimize the maintenance effort and still get the kind of risk coverage that I’m looking for,” said Gartner senior director Joachim Herschmann, who is on the App Design and Development team.
In the past, what used to happen is that the organization used to say, “we’ll recruit the developer, and we’ll do the R&D and then when we need to test it, we’ll send it to India, and then it will be tested there.” Still, organizations realize that this has to be at the core of your R&D organization, Digital.ai’s Arieli explained.
Now, as developers are starting to get more and more involved in quality, the notion of building quality as part of the application began to take hold.
“So developers have to think about how do I engender unit testing and more and more of it and when you reach the total extreme of it and you’re totally mature, they start thinking of automation,” said Anand Sundaram, the SVP of Products, UI, Device Cloud, and Performance Testing at SmartBear Software.
Security and performance testing
After quality, there are other aspects of the application for which automated testing can be leveraged: security and performance testing of your APIs and microservices at the developer level before everything comes in for integration testing or the entire application comes together.
“We’ve accepted that test automation is valuable, deep code analysis is valuable, and now we’re actually starting to say the same thing around security; how can we embed security in each stage so that we can build security into the pipeline,” Parasoft’s Lambert said.
Now, testers are trying to apply the same methods that they used for testing quality to security. So that means deep code analysis to identify potential runtime exceptions that could go uncaught. As they’re moving up the stack, they’re looking to leverage unit testing for fuzzing the underlying code and seeing how they can utilize API tests for API security testing. Developers can start to build quality and security by taking advantage of those earliest-stage validation techniques, Lambert explained.
While automated testing has received widespread recognition as a must for today’s software development environments, organizations face many challenges when setting up effective testing strategies in their CI/CD pipelines.
At the bottom of the testing pyramid, the struggle with unit testing is that there isn’t a lot of visibility, and it’s difficult to understand how much it actually covers. On top of that is the service component testing usually driven by an API. The system and UI testing are at the top of the pyramid, which can be the most challenging.
Implementing all of these levels of testing can be a challenge, especially for legacy systems, since these aspects of testing were not initially accounted for when the applications were created, Digital.ai’s Arieli added.
Another challenge in implementing automated testing is finding the staff with the appropriate skill set. Testing complex enterprise applications requires business domain expertise. Also, maintaining test scripts makes it difficult to achieve continuous test automation — as automation requires teams to ensure that testing doesn’t become a bottleneck. Therefore, tests must be designed in a way that minimizes disruption to the continuous testing process. The goal is for test automation teams to build robust and reusable test scripts that don’t require constant attention and maintenance, according to Clinton Sprauve, the director of Product Marketing at Tricentis.
Organizations also need to find a way to manage and track test automation efforts across multiple tools through observability and analytics.
“There is a challenge to testing in the sense that we need to do it more frequently, we need to do it for more complex applications, and we need to do it at a larger scale. This is not feasible without automation, so test automation is a must,” Gartner’s Herschmann said.
AI and observability in automated testing
With value being a core tenet of DevOps, managers must see how each decision impacts the user experience, revenue, and business performance. This is why testing providers are looking to create more intelligent means of testing that can provide analytics.
Intelligent testing can combine data analytics, clever heuristics and algorithms, machine learning, and anything that analyzes data in real-time and makes decisions or recommendations that then help solve the problem. Developers can then use that need to have instant feedback of where precisely the problem occurred and move much more quickly.
Observability is needed in the pipeline because it gives testers a clue as to where exactly the problem is when the problem occurred and then alerts the tester. In addition to observability, automated testing solutions have also created ways to help developers with many of the pain points around testing and speed up the process.
“At the beginning of Agile, when you start talking about quarterly releases, you could still kind of fake it, right? You could still handle quality. You would have minimal time to do all of your regression testings and so forth, but you could build that into a schedule and make it work. When you move to CI/CD where change is continuous and disruptive, you need to find new solutions,” Mabel’s Belcher said. “And so for a few years, as an industry, we turned to, well, let’s just make us another thing that the developers have to worry about and have them write tests that do end-to-end validation. But the problem with that is that those tests relied on the stability of the very thing that was constantly changing.”