Leveraging GenAI for quality engineering | CQI | IRCA Skip to main content
Dave Colwell, VP of Artificial Intelligence and Machine Learning

Leveraging GenAI for quality engineering

Progress indicator

Dave Colwell, VP of Artificial Intelligence and Machine Learning
Published: 9 Aug 2024

Dave Colwell, Vice President of Artificial Intelligence and Machine Learning at Tricentis, explores how generative artificial intelligence can be used to support quality in software development.

In an intensely competitive market, where customer expectations for digital experiences are growing constantly, digital capability is now integral to business. With more than four million apps available across iOS and Android platforms alone, brands must ensure the timely delivery of high-quality digital products and services to stand out, capture market share and generate revenue.

Empowering software teams to deliver flawless digital experiences, quickly and at scale, is strategically vital. Testing to guarantee application functionality in a real-world setting is central to that objective.

Quality engineering (QE) ensures quality is built into every step of the software development lifecycle, to create applications that meet customer useability and functionality demands. However, development and testing teams face time, accessibility and productivity challenges, which makes achieving application quality slow and difficult. Generative artificial intelligence (GenAI) can help simplify and automate profoundly complex software development and testing challenges.

Indeed, the International Data Corporation estimates that enterprises will drive US$1tn in productivity gains by 2026 using GenAI and automation, with software development anticipating significant benefits to help deliver high-quality applications, fast.

With concerns growing around compliance, data use and intellectual property (IP) protection when using GenAI in large language models (LLMs), QE leaders need new tools to ensure these gains without compromising on responsible usage, safety, or accessibility.

GenAI benefits for enterprise

With global businesses estimating that poor mobile app quality alone costs up to US$2.49m in lost revenue, the fast development and ongoing maintenance of high-quality applications has become mission-critical.

The problem is that modern enterprises are complex mazes of integrated systems, custom software and third-party apps, where defects can go unnoticed. The volume of information that needs to be tracked to find and fix software defects to improve application quality is simply overwhelming for humans in development teams.

"By maintaining data hygiene, rigorous testing and governance, businesses can harness GenAI’s power to drive innovation and maintain customer trust, ensuring the quality of their digital applications and a strong market position."

Dave Colwell, Vice President of Artificial Intelligence and Machine Learning, Tricentis

The process is all the more challenging because software bugs do not generally make themselves readily apparent. Instead, they stay hidden in the intersections and logic gates of how data interacts with itself, posing a constant threat to the delivery of seamless digital experiences. Meanwhile, properly reporting on the work required to fix these bugs for knowledge sharing can make developer workloads unmanageable.

Providing QE teams with GenAI tools is like giving them a second pair of hands, as these tools can analyse large datasets to learn patterns, process data at runtime to give discrete answers, and draw connections between disparate concepts.

AI can scan software documentation, resources or other large text bodies for information, which can be requested and given on command, cutting out significant portions of tedious manual work. GenAI can even help to write code with natural language processing, enabling users to enter simple text descriptions to generate and explain required code or identify and recommend fixes for potential issues.

By automating manual processes, developers can spend time on innovation and future-proofing their development and delivery cycles.

AI can also be harnessed to protect company reputation by reducing the potentially catastrophic risks of getting things wrong – whether that is the failure of high-stakes applications, compliance needs not being met, or customers losing out because of buggy apps. When managed effectively and responsibly, AI holds the key to maximising value for customers and stakeholders.

Balancing GenAI’s benefits with imperfections

As LLM models evolve, ensuring responsible and trustworthy AI will become paramount to harnessing the benefits. AI tools are not perfect or immune to mistakes. They hallucinate, and display bias, discrimination, incuriosity, and more.

Robust governance and clear policies are therefore crucial to avoid problems, from data ownership to biased outcomes or security threats that can harm an organisation’s reputation, customer trust and profitability. Ensuring human oversight, data privacy, security and transparency are critical cornerstones of AI adoption, as highlighted by the terms of the EU AI Act.

Understanding the origins of data used to train AI is an important first step for practical and safe application. Public LLMs, for instance, may incorporate data from user prompts into their training, posing a risk of exposing sensitive information. Adhering to strict data hygiene practices, such as using separate AI models for each customer, can eliminate this risk.

Testing GenAI

Rigorous verification and testing of GenAI systems to assess reliability, accuracy and performance, and to protect against hallucination and bias, are also crucial. Although AI-driven solutions offer the potential to automate repetitive tasks and enhance code generation, they can also introduce complexities and technical debt when not managed carefully.

Recognising GenAI’s limitations, developing robust output validation procedures, and establishing comprehensive testing frameworks will all help uncover potential issues. The ‘gold standard’ must be a robust testing method that does not automatically trust the AI. The beauty of generative AI is that stakeholders can weigh in before taking its answers verbatim. It is these interactions that will improve the AI model over time, as well as the quality of its answers.

Integrating GenAI tools allows QE teams to streamline processes, automate test creation, and optimise code, resulting in faster and more cost-effective application development. Successfully leveraging GenAI, however, means balancing these advantages with responsible usage to manage its limitations with robust testing frameworks and human oversight.

By maintaining data hygiene, rigorous testing and governance, businesses can harness GenAI’s power to drive innovation and maintain customer trust, ensuring the quality of their digital applications and a strong market position.

Learn more about regulating AI

An international BSI poll reveals that 62% of the public want tighter controls around AI, but governments around the world are working to differing standards and timescales. Can the quality sector step in to provide some much-needed confidence?

Quality Live 2024 recordings now available

CQI Quality Careers Hub Frequently Asked Questions

Access recordings from all twelve Quality Live 2024 sessions on Digital Transformation, Culture, and Skills.

Start your career in quality

Fantastic career opportunities await you in the quality profession.

Quality World

Get the latest news, interviews and features on quality in our industry leading magazine.