Ambient AI constraints policy

1. Purpose and scope

1.1 This policy sets out the constraints CAREFUL places on the artificial intelligence used in its Ambient Voice feature, the controls by which those constraints are enforced, and the limits of those controls.

1.2 It applies to the Ambient Voice feature of the CAREFUL platform: the recording of a consultation, the drafting of a clinical note, and the capture of clinical actions.

1.3 It supports CAREFUL's Statement of Intended Purpose and its clinical risk management under DCB0129.

2. Why we publish this

2.1 Ambient AI in healthcare is usually a black box. Buyers, clinical safety officers and regulators are asked to trust a description of what a tool does, with no way to hold anyone to it. We think that is the wrong way round.

2.2 This policy is published so that it can be relied on and referred to. It is binding on CAREFUL.

2.3 The instructions given to the AI model, and the controls implemented in CAREFUL's software, are subordinate to this policy. They are the means by which CAREFUL complies with it. They may change; this policy is what they must satisfy.

3. The role of the AI

3.1 CAREFUL's ambient AI is a recorder. Its role is to write down what was said and what was decided, accurately and in the format requested.

3.2 It is not a clinician, an adviser, or an assistant to clinical thinking. The clinical judgement is made by the responsible clinician before the AI does anything with it.

3.3 The AI does not add to a clinician's judgement, complete it, improve it, or comment on it.

4. Prohibited behaviours

4.1 CAREFUL's ambient AI does not undertake clinical reasoning or inference in any part of its output. This applies without exception to the summary, the clinical note, the actions, and to any internal or diagnostic output.

4.2 Specifically, the AI does not:

4.2.1 create or infer any fact or information that is not present in the material provided;
4.2.2 assess, interpret or comment on the patient's clinical condition;
4.2.3 offer, imply or complete a diagnosis, differential diagnosis, prognosis or risk assessment;
4.2.4 suggest, recommend or propose any investigation, referral, treatment, medication or follow-up;
4.2.5 identify a clinical risk or concern, or flag anything as urgent, abnormal or dangerous;
4.2.6 comment on the adequacy, completeness or safety of the care recorded, or on anything a clinician did not do;
4.2.7 add any step, action or precaution on the grounds that it is clinically indicated;
4.2.8 infer clinical urgency, or set a deadline on any basis other than one a clinician stated.

4.3 Where a requested document contains a section that is inherently clinical — an assessment, an impression, a diagnosis, a differential, a plan — that section is populated only with what a clinician stated. The AI does not supply one of its own and does not complete a partial one. An empty section is a correct outcome where nothing was stated.

5. Where actions come from

5.1 An action is something a clinician explicitly stated should happen during the consultation.

5.2 Actions are captured only from the transcript, and only from words spoken by a clinician. They are never derived from demographic information, from supplemental clinical information, or from anything other than what a clinician said.

5.3 The following are not actions and are not captured:

5.3.1 anything no clinician stated, however clearly it follows from what was said, and however clinically indicated it may appear;
5.3.2 a further step implied by a stated action, unless a clinician stated that step as well;
5.3.3 anything raised and then declined, deferred, ruled out or hedged;
5.3.4 anything a clinician said is already done or already in progress;
5.3.5 anything asked for by the patient, a relative, a carer or an advocate that a clinician did not then adopt and state.

5.4 Capturing no actions is a valid and expected outcome. CAREFUL does not produce an action in order to have something to show.

6. Controls

6.1 Traceability. Every action carries a verbatim quotation of the words in which the clinician stated it. CAREFUL's software checks that quotation against the transcript and rejects any action it cannot find. An action that no clinician stated does not reach the clinician.

6.2 Clinician confirmation. No note and no action is saved until the responsible clinician has reviewed it, edited it if needed, and confirmed it.

6.3 Transcript retention. The transcript is retained so that any item can be checked against what was actually said.

6.4 Evaluation. CAREFUL evaluates how often the AI adds an action that was not stated, omits one that was, or alters the meaning of one. These are managed as clinical safety risks under DCB0129.

6.5 Model change control. The AI model and its version are recorded and controlled. Where the model provider issues an update, that update is reviewed and re-evaluated against this policy before CAREFUL adopts it.

7. The limits of these controls

7.1 Instructions to an AI model reduce unwanted behaviour. They do not guarantee its absence. CAREFUL does not rely on instruction alone, and does not claim that the model is incapable of error.

7.2 The controls in section 6 are layered for that reason. Traceability establishes that an action came from the clinician's words; it does not establish that the action faithfully represents what those words meant. Clinician confirmation is therefore a required control and not a formality, and evaluation under 6.4 is how the residual is measured rather than assumed.

7.3 Where the controls in section 6 identify a failure, it is recorded and managed through CAREFUL's clinical risk management process.

8. Governance

8.1 This policy is owned by the person named above and is reviewed at least annually.

8.2 It is also reviewed on any material change to the AI model, to the instructions given to it, or to the behaviour of the Ambient Voice feature.

8.3 Changes to this policy require the owner's approval.

8.4 [MODEL / PROVIDER / PROCESSING REGION — DECISION REQUIRED. Adopting NHS organisations require this under DPIA and DSPT in any event; the question is only whether it is stated here or provided on request. Either is legitimate. Do not leave it ambiguous.]

8.5 The full instruction set given to the AI model is available to adopting organisations and other customers on request, as part of clinical safety and information-governance assurance.

Statement of Intended Purpose
Software Qualification Rationale — internal, available on request
Clinical Safety Case Report (DCB0129) — available to adopting organisations