# Evaluating AI Tools and Vendors: A Board-Level Checklist

Date: 2026-07-08

AI vendor discussions are full of confident claims (“secure,” “enterprise-ready,” “no training on your data”). Boards should treat AI procurement like any other critical dependency: **verify what matters, contract it, and monitor it.**

<!--more-->

## 1) Data Handling (The Non-Negotiables)

Ask the vendor:

- What data is processed (inputs, outputs, logs, files, tool results)?
- Where is it processed and stored (regions, subprocessors)?
- What is the retention policy by plan/tier?
- Is customer content used for training by default, opt-in, or never?
- Can you enforce data minimization and redaction controls?

If your AI workflows touch personal data, your organization still bears GDPR accountability—even if the model is a vendor service ([GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng)).

## 2) Security Controls (Not “Trust Me,” Show Me)

Ask for evidence of:

- access controls, MFA, admin separation
- vulnerability management and disclosure process
- audit logs (what happened, who did it, when)
- tenant isolation (for multi-tenant offerings)
- incident notification timelines

For LLM products, include LLM-specific risks like prompt injection, system prompt leakage, and excessive agency ([OWASP LLM Top 10 2025](https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/)).

## 3) Governance and Accountability

Boards should ask management to ensure each vendor relationship has:

- a named internal accountable owner
- an approved scope of use (“allowed use”)
- periodic risk review and re-approval process
- a documented exit plan (data export, deletion, replacement)

NIST AI RMF gives a practical way to structure this governance across the lifecycle ([NIST AI RMF](https://www.nist.gov/itl/ai-risk-management-framework)).

## 4) Transparency: What Will You Be Able to Prove?

Ask:

- Can we get logs and metrics suitable for audits?
- Can we reconstruct an incident (inputs, outputs, actions, sources) without guesswork?
- What changes without notice (models, safety policies, routing, retention)?
- How do you communicate changes and breaking behavior?

## 5) Cost and “Runaway Spend” Controls

For LLM and agentic systems, cost is a risk dimension:

- rate limits and quotas
- budget caps per team/system
- alerts on spikes and anomalous usage
- safeguards against “unbounded consumption” (an OWASP risk category)

## 6) Regulatory Alignment (Practical, Not Performative)

Ask:

- Do you support AI governance standards (e.g., ISO/IEC 42001 alignment)?
- If operating in the EU, how do you support EU AI Act obligations relevant to our role?
- Do you support customer AI literacy programs with role-based guidance (Article 4 expectations)?

## Red Flags in AI Marketing

Boards should treat these as “slow down and verify” signals:

- “We’re compliant” with no scope definition or evidence
- “No training on your data” while retention/logging is unclear
- vague answers on subprocessors, regions, or incident response
- “It’s just a chatbot” while it has tool access or can trigger actions
- refusal to define accountability or provide auditability

---

**I consult with boards on technology vendor risk and deliver board-level courses on cyber, AI, and regulations—including practical vendor evaluation patterns and templates.** [Contact me](https://goldmanmalka.com/about).

---

### Relevant Sources

1. **AI Risk Management Framework (overview)** — NIST — [https://www.nist.gov/itl/ai-risk-management-framework](https://www.nist.gov/itl/ai-risk-management-framework)
2. **AI RMF 1.0 (NIST AI 100-1)** — NIST — [https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-ai-rmf-10](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-ai-rmf-10)
3. **OWASP Top 10 for LLM Applications 2025** — OWASP — [https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/](https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/)
4. **ISO/IEC 42001 explained (AI management systems)** — ISO — [https://www.iso.org/cms/%20render/live/en/sites/isoorg/home/insights-news/resources/iso-42001-explained-what-it-is.html](https://www.iso.org/cms/%20render/live/en/sites/isoorg/home/insights-news/resources/iso-42001-explained-what-it-is.html)
5. **GDPR (Regulation (EU) 2016/679)** — EUR-Lex — [https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng](https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng)
6. **EU AI Act Article 4: AI literacy** — EU AI Act Service Desk — [https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-4](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-4)