Throughout the procurement, implementation and testing, the organization will gain a stronger basis for conducting the cost-benefit analysis. This analysis forms the foundation of the benefits realization plan, including specific and measurable outcomes.
The introduction of an AI system often leads to changes in how work is organized. The organization should monitor whether the AI system is being used as intended and whether the benefits of new workflows are being realized.
- An example of a new workflow: At Vestre Viken Hospital Trust, patients who receive a "no fracture" result from an AI analysis are sent home immediately, instead of waiting at the emergency department for a radiologist to review the images. As a result, radiologists no longer speak with patients where no fracture is detected, unlike previous practice.
During the governance phase, the organization can systematically measure and document the quality and outcomes of the AI system, including through statistics and reports. This information serves to track progress toward the expected benefits. If the anticipated benefits are not achieved, the organization should assess the underlying causes, identify corrective actions, and implement the most appropriate and effective measures. In many cases, the principles of benefits realization for AI systems are similar to those applied to other systems [161].
The organization's experience with implementing an AI system can provide valuable insight for others—both nationally and internationally. Results from the implementation and any accompanying research may benefit other institutions and should be published and shared. Sharing quantitative data on realized benefits can support others in conducting cost-benefit assessments of AI systems. An effective way of communicating qualitative benefits is through interviews with healthcare professionals who use the AI system, or testimonials from citizens, patients, or service recipients who have received faster or improved care. Likewise, negative lessons learned should also be shared to help others avoid making similar investments without achieving the desired outcomes. Organizations that collect data such as patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs) can also contribute valuable knowledge [162].
As quality assurance of AI-based systems becomes increasingly relevant across the health and care sector, this is an area where experience-sharing has great potential to strengthen future efforts.
[161] The Agency for Public Management and Financial Control has produced a general guide to benefit realisation (dfo.no) that may be useful.
[162] PROM; Patient-Reported Outcome Measure and PREM; Patient-Reported Experience