Legal teams are being asked to evaluate AI tools faster than their checklists are being rewritten. Most procurement questionnaires still open with features and close with price, and the questions that actually determine risk — where inference runs, what is retained, what happens when a model is unavailable — appear late or not at all.

This is the checklist we would hand a friend. It is also, openly, the checklist we hold ourselves to; links to our own answers are at the end.

The ten questions

Where does inference run — on our premises, on a host the vendor operates, or with a third-party provider?
What exactly leaves our control during a task — files, prompts, retrieved passages, metadata? Ask for the list, not the adjectives.
What is retained, where, and for how long — and does deletion on our side actually delete on yours?
Is any part of our material used to train or improve models? Including "product improvement", which is training under a softer name.
What happens when the model or endpoint is unavailable — does the system stop, or silently fall back to a different provider?
Who at the vendor can see prompts or documents, under what process, and is that access logged?
Can the tool's conclusions be traced to their sources after the fact — by us, not by the vendor on request?
Which claims on the vendor's website carry evidence we can inspect, and which are pending?
What are the exit terms — what do we leave with, in what format, and what is destroyed?
If the vendor's answer to any of the above changes, how will we find out?

The two that get dodged

In our experience, questions four and five — training on your material, and silent fallback to another provider — are the two most often answered with reassurance instead of facts. Both deserve a sentence in writing that names the behaviour, not the intention. A vendor that has decided the answer in advance can state it in one line. Vaultr's one-liners: no training on your material, ever; and no silent fallback — when a model or endpoint is unavailable, the task stops with an explicit state.

Our own answers

The security brief walks the mode matrix, retention by data class and failure behaviour, with the boundary test published so you can verify the local mode yourself rather than take our word for it. The trust register lists each public claim with its evidence — including the rows that are still pending, which is precisely why the completed rows are worth reading.

A checklist can't make a vendor trustworthy. It can make evasion expensive.