AI, the Future of QE

In our latest World Quality Report (WQR), we have a deep dive into organizational attitudes to AI within QE. The WQR is our annual global survey in which we reach out to 1,750 organizations in more than 30 countries to assess trends and attitudes in and around QE.

AI and testing

Broadly speaking, our survey reveals a huge interest in AI as 70% to 79% of organizations acknowledge that AI will be used in a variety of testing use cases. When asked about actively using AI to optimize testing processes, 74% responded that they were. Furthermore, we found that organizations are using AI to build and improve test scope as well as using it within the general test ecosystem overall.

These responses show a phenomenal take-up of a relatively new technology which has gained considerable traction very quickly.

Higher productivity

While AI has been around for a while and has been applied to improving QE outcomes like self-healing automation, test case optimization etc., the use of GenAI appears to be primarily driving interest in higher productivity with 65% of those surveyed seeking this outcome. While the emphasis on higher productivity is unsurprising, it is curious to note that defect prediction (29%) and driving reliability (33%) were at the lower end of the scale.

The drive for higher productivity could be due to the need for QE to keep pace with the incredible productivity boosts that are expected in the Software Development Life Cycle (SDLC) through GenAI, in particular, Large Language Models (LLMs). Additionally, concepts like ‘perpetual beta’ and ‘continuous testing’ have become more prevalent whereby end users demand the faster release of products knowing that they aren’t perfect but workable. This has increased end users’ tolerance levels to failure and stands in stark contrast to the traditional idea of releasing a ‘perfect’ product.

Drilling down to the frequency of AI in use cases, organizations that stated that AI was ‘used a lot’ showed the most interest in the use case around autonomous testing (55%), whereas test case design (37%) was the lowest. Our data implies that more than 80% of organizations have ‘used a lot’ or ‘used on some occasions’ AI for a variety of test cases.

AI challenges

The data also suggests that there are multiple challenges for organizations when it comes to GenAI, such as concerns about privacy (35%) to a lack of resilience (33%) and dealing with hallucinations (24%).

Surprisingly, 31% of companies do not see any value whatsoever in adopting AI which is probably down to the view that costs and risks outweigh the benefits.

While there is no doubt that there has been a lot of investment in AI, successfully testing and deploying solutions is critical for driving future success. Those surveyed plan to ‘focus on requirements to explainability, transparency, robustness, privacy’ (75%) along with an ‘investment on data science skills’ (54%) and in additional ‘skillsets in QA teams’ (23%).

Some considerations as you embark on your AI for QE journey…

• As higher productivity driven by AI will rapidly increase the development speed, organizations need to continue to invest and scale the QE process, skills, and bandwidth.
• To use AI in QE effectively, we will need to analyze carefully how to prioritize the use cases and have clear KPIs that measure concrete outcomes to achieve a successful transformation with AI.
• We need to use an iterative method for the selected/prioritized use cases and an MVP strategy, as there will be many trials with AI solutions in the near future, and many uncertainties will emerge too.

 

Author