The mapping conducted so far has provided insight into which AI systems are available, as well as the human and technical factors the organization must consider when procuring a system. This knowledge helps update the risk assessments and benefit analyses, ensuring that the organization has a solid basis for decision-making regarding whether to proceed with the procurement of an AI system.

Update benefit analyses

Where possible, the organisation should evaluate the potential benefits of the AI system using an approach similar to a health technology assessment, based on the available evidence. A health economic evaluation, comparing costs with expected health outcomes, is a key component in the decision-making process for whether to invest in an AI system. The assessment should include the following elements:

• The benefits provided by the AI system, whether clinical, administrative, research-related, or other
• The resources required to implement and use the AI system
• The risks associated with the introduction of an AI system

The estimated use of the solution, such as the number of users or number of examinations, may affect the cost. The introduction of AI systems can also lead to increased use of, for example, diagnostic methods.  Therefore, an assessment of potential increases in total healthcare costs due to increased service utilization should be conducted.

By conducting and documenting relevant baseline measurements, for instance, the amount of time healthcare personnel currently spend on specific tasks, these measurements can serve as a reference point in the ongoing assessment of benefits.

Updating risk assessments and planning security measures

Information security and data protection relate to the entire system and its use, not just the AI model itself. Within the organization, the AI system will be part of a larger whole, operated, managed, and used within an environment that includes multiple other systems, established workflows, organizational culture, and existing competencies. The Code of Conduct for information security and data protection and the NIST framework for AI risk management can be useful tools in assessing whether requirements for information security and data protection are adequately addressed [102]. Based on the results of the risk assessment, the organization should plan appropriate measures needed to be implemented to ensure sound risk management for safe service delivery, both during the implementation phase and throughout the AI system's regular operation [103].

Planning clinical validation

The organization should assess whether additional clinical validation is necessary before or in connection with the implementation of an AI system it plans to procure. The market mapping conducted in Phase 2 can serve as a basis for evaluating the extent and scope of validation that the organization itself must carry out. For further details on validation, see AI Factsheet 2 [104]

In some cases, it may be sufficient to test the solution on a small sample of the organization's own data before putting the product into operation, followed by quality control measures to detect any errors. In other cases, it may be appropriate to conduct prospective studies, such as randomized controlled trials, to validate the AI system within the intended workflow.

It may also be advisable to plan a pilot [105] phase before the AI system is deployed in other parts of the organization. Conducting a pilot can help reduce risks by allowing corrections to be made before the consequences become significant, prior to any potential scaling. It is important that healthcare institutions ensure that such validation does not constitute clinical investigation without seeking approval in accordance with the MDR.