Those of us with years of experience in IRT / RTSM may have experienced different hands writing scripts during UAT. Whether it is a Contract Research Organization (CRO), Sponsor, IRT vendor, or a 3rd party, there is no real consensus across the industry on who should be driving User Acceptance Testing (UAT) activities.

In early 2023, the Europeans Medicine Agency (EMA) republished the paper “Reflection paper on the use of interactive response technologies (Interactive voice / web response systems) in clinical trials, with particular emphasis on the handling of expiry dates.” The handling of IRT systems was becoming an increasing interest to national competent authorities (NCAs) ever since the technology expanded to greater and higher risk functionalities, highlighting “if not handled appropriately, posed an increased risk to the patient.” While the paper largely touches on expiry dates and labeling within the IRT, there is repeated mention of sponsor ownership of UAT.

Section 2.2.1 states, “This is an opportunity for the sponsor to test the system and this should be undertaken, preferably with test scripts written by the sponsor.” Sponsor ownership is again reiterated in section 2.2.4, “The sponsor should assure themselves through UAT of the suitability of each system. It is expected that the sponsor will write their own test scripts for testing the system.”

Based on the name – “user acceptance testing,” it makes the most sense for the user of the system, (i.e., sponsors) to author test scripts. However, authoring test scripts is simply not a reality for organizations that don’t have the in-house expertise. Often this is the case with small to mid-sized companies.

Turning UAT over to your IRT vendor can be problematic as there is a clear conflict of interest with the software provider both building and doing the user testing of the product. The same could be said in a layman’s example – “would you let your car salesman or manufacturer test drive your car?”

Beyond the conflict of interest, the vendor’s interests may also be misaligned in several other ways. It is in the best interest of the IRT vendor’s validation team to have as few issues as possible. The more issues they find the more conflict that will be created within their organization. This could lead to undue pressure to “not be so picky” since any error will result in rework by the development / operations part of the organization.

To gain further insights, we turned to Craig Mooney, an IRT / RTSM veteran who has worked in the space for over 30 years. With Sponsor (Bristol Meyers Squibb), CRO (Covance), and IRT (ICTI, and Calyx) experience under his belt, we’ve asked for his perspectives on UAT ownership, shedding light on how UAT has been managed on both sides of the fence:

1. What makes User Acceptance Testing (UAT) so complex?

Part of UAT is to determine not only does the system reflect the spec but is it fit for purpose. You can have a system that is 100% compliant with the spec and still not built for purpose. In the specification process the sponsor carries a knowledge of the protocol the system provider does not. While not frequent, you can certainly have a situation where the system is not acting how it was intended to even if it is per specifications. Independent organizations are better positioned to identify these misunderstandings.

2. Can you describe the conflict of interest that occurs when sponsors outsource their UAT to an IRT company?

UAT findings reflect on the quality of the system and as such are typically part of KPIs/metrics used in vendor and sponsor governance. The vendor has a vested interest in minimizing findings to reflect quality delivery. There could develop a culture where findings are passed back to the dev/ops group for a quick fix in order to minimize official findings.

Sponsors or 3rd party partners don’t have this vested interest. In fact, you could make the argument the opposite is true when these groups conduct UAT. These groups (sponsors & 3rd party partners) UAT reports are more likely to reflect the true state of the system.

3. Why do sponsors so often outsource their UAT services to their IRT provider?

I understand why a sponsor that lacks the resources or expertise to perform a formal UAT may reflexively ask the system provider to do so, but it is in the interest of the study and ultimately patients that an independent group perform this task.

Even an IRT provider you trust and has provided good service may have an unconscious bias or conflict. I always have, and always will, recommend to sponsors they do this UAT work themselves or outsource it to a 3rd party expert. My opinion in this matter has not changed regardless of which side of the sponsor/provider equation I have been on.

In summary, while it’s ideal for sponsors to write their own test scripts and drive UAT, practical constraints often lead to outsourcing. If you choose to outsource, consider some basic questions to your UAT services provider on their experience and expertise:

At Poxy, we’ve done it right hundreds of times. For over 16 years we’ve applied our longitudinal knowledge into a testing methodology that advocates for sponsors, ensuring their system is fit for purpose. The result is an expedited go-live so your clinical trial can run without encountering any patient safety issues.

Want to safeguard your IRT from dosing, unblinding, and randomization errors? Reach out to Poxy Clinical to see how we can support you.

Written by:

Susan Vong
President, Poxy Clinical

Guest Insights by:

Craig Mooney
Founder, Craig Mooney Consulting Group