An effective AI audit checklist verifies ten audit dimensions and produces a single evidence pack you can defend to auditors and regulators. Those dimensions are inventory and scope, risk classification, data governance, model performance testing, bias and fairness, explainability, security and privacy, documentation, human oversight, and monitoring with change control. Run the audit before deployment, after any material change to the model or its data, and on a fixed schedule tied to risk tier. Two standards anchor almost every item on this list: the NIST AI Risk Management Framework and the EU AI Act's Annex IV documentation requirements. If you can only remember one thing, remember this:
- Inventory every AI system with a named owner and version
- Classify risk and set audit depth and cadence accordingly
- Trace data provenance and run disaggregated performance tests
- Test for bias, log explainability outputs, and probe for security gaps
- Assemble one evidence pack that maps findings to owners and deadlines
Key Takeaways
A defensible AI audit produces one evidence pack mapping every finding to a named owner, a threshold, and a remediation date, verified against NIST and EU AI Act artifacts.
| Point | Details |
|---|---|
| Inventory first | Build a model card and system map with a named owner before testing anything. |
| Test on your own data | Build a representative test set and disaggregate metrics rather than relying on vendor benchmarks. |
| Undocumented equals noncompliant | Score every control as documented, undocumented, or unknown, and fix unknowns first. |
| Cadence follows risk tier | Schedule re-audits after material changes and on a fixed cycle based on risk classification. |
| Start small on prototypes | Hanad Kubat's fixed-price Prototype Audit delivers a model card, a representative test set, and a prioritized remediation list in three to five days. |
Table of Contents
- AI Audit Checklist: Items to Verify and Evidence to Gather
- How Do You Run an AI Audit Step by Step?
- What Metrics Should an AI Audit Actually Test?
- Governance and Policy: What Auditors Look for on Ownership
- Operational Controls: Monitoring, Logging, and Change Management
- Mapping the Checklist to NIST, the EU AI Act, and ISO
- Practitioner Notes on Auditing a Prototype
- What Prototype Audits Get Wrong Most Often
- Fixed-Price Prototype Audit: What I Deliver
- Sources
- FAQ
AI Audit Checklist: Items to Verify and Evidence to Gather
A checklist item without evidence is not a control, it is a hope. Every item below needs a named owner, a piece of documentation, and a way to reproduce the result.
- Inventory and scope. Build a model card and a system map for each AI feature: what it does, what data it touches, who owns it, and which version is live. The EDPB's AI auditing checklist treats the model card and system map as the starting artifacts for any audit, not optional extras.
- Risk classification. Write down which tier the system falls into and why. A resume screener and an internal chatbot summarizer do not get the same scrutiny.
- Data governance. Document where training and fine-tuning data came from, the legal basis for using it, how it flows through your pipeline, and what quality checks ran before it touched a model.
- Model testing. Build a test set that looks like your real users, not a vendor benchmark. Record disaggregated metrics and the acceptance threshold you're testing against.
- Fairness. Name the fairness metrics you chose, the sample size behind them, and any disparities found, along with the mitigation applied.
- Security and privacy. Verify access controls, encryption at rest and in transit, and run adversarial and prompt-injection probes against anything LLM-based.
- Documentation. Index every artifact above into one evidence pack, so a regulator, investor, or acquirer can find the answer to any question in minutes, not weeks.
Scoring each item as documented, undocumented, or unknown gives you an immediate priority list. Items flagged unknown under the EU AI Act get fixed first, followed by accountability gaps, then fairness, then security.
Pro Tip: Don't wait for a perfect evidence pack before you start. A partial pack with clear "not yet documented" flags is more credible to an auditor than a polished document that quietly hides gaps.
How Do You Run an AI Audit Step by Step?
The audit sequence that shows up across practitioner guides, including Redwerk's step-by-step framework, follows five practical phases.
- Prepare. Inventory the systems in scope, assign owners, write the audit objective, and list the evidence you already have versus what you'll need to produce.
- Assess. Check data lineage end to end and confirm the model's baseline performance against its original claims.
- Test. Run bias tests, explainability checks, adversarial probes, and security scans, all logged with timestamps.
- Operate. Review monitoring dashboards, service-level agreements, and human override procedures to confirm someone can actually stop the system if it misbehaves.
- Report. Write the internal report with full detail and, where required, a public-facing summary. Assign remediation owners and a verification date for every finding.
An audit is not a one-time pass or fail. Treat it as an iterative cycle that repeats after material changes and on a schedule set by risk tier.
What Metrics Should an AI Audit Actually Test?
Vendor benchmarks tell you how a model performs on someone else's data. They tell you almost nothing about how it performs on yours. Build a representative test set from your own users, your own edge cases, and your own failure modes, then disaggregate every result by the groups that matter for your use case.
- Calibration to check whether confidence scores match real-world accuracy
- Precision and recall broken out by subgroup, not just averaged across the whole set
- Disparate impact ratios to catch disparities that a single aggregate score hides
- Explainability outputs, recorded and stored, from whatever tool you use (SHAP, LIME, or a model's native attribution method)
Testing on representative, out-of-sample data and disaggregating by demographic group is the step most teams skip under deadline pressure, and it's usually the one that surfaces the finding nobody wanted.
Adversarial checks matter just as much: prompt injection attempts, malformed input handling, and basic red-team probing on anything that takes free-text input from a user.
Governance and Policy: What Auditors Look for on Ownership
Auditors don't accept "the team knows about it" as an answer. They want a name attached to a decision.
- A named accountable executive and a named business owner for every AI system in scope
- Written acceptable-use policies and data classification rules that staff can actually find
- A clear definition of what "control complete" means: owner, evidence, threshold, and review trigger, not just a checked box
- A review cadence tied to risk tier, with dates and triggers logged somewhere auditable
An undocumented control counts as a nonexistent control for compliance purposes. If it's not written down, it didn't happen.
Operational Controls: Monitoring, Logging, and Change Management
A model that passed its audit in January can drift by June. Ongoing operational checks are what catch that.
- Drift monitoring with defined alert thresholds and a clear escalation path
- Decision logs and prompt version history, retained and retrievable when someone asks "why did it do that?"
- Incident definitions and reporting timelines that align with local regulatory obligations for high-risk systems
- Change control triggers, retraining, a new data source, or an expanded user population, that automatically force a re-audit
Mapping the Checklist to NIST, the EU AI Act, and ISO
Every checklist item above maps to something a regulator or enterprise buyer will recognize by name.
- NIST AI RMF: inventory and risk classification map to Govern; data lineage and system mapping map to Map; bias and performance testing map to Measure; monitoring and incident response map to Manage
- EU AI Act: model cards, system maps, and test results map directly to the Annex IV technical documentation dossier for high-risk systems, and incident logs feed the Act's reporting obligations
- ISO/IEC 42001 and DPIAs: reference these in your evidence pack when a customer or regulator asks for a recognized management-system standard behind your process, alongside a data protection impact assessment for anything touching personal data
Practitioner Notes on Auditing a Prototype
On a prototype audit, I ask for three things first: system access, a model card if one exists, and a 100 to 500 row representative test set. That's usually enough to find the real gaps fast.
Pro Tip: A named owner, timestamped logs, and one passing acceptance test will get you further with an auditor than a beautifully formatted policy document nobody has followed.
Most prototypes fail the same three checks: no named owner, no test set beyond a demo script, and zero incident logging. I fix those in that order, because everything else depends on them.
What Prototype Audits Get Wrong Most Often
Most advice on AI audits assumes a company already has a compliance team, a data governance function, and a security review process. Founders running a prototype have none of that, and treating the full enterprise checklist as a pass or fail test on day one is a waste of time.
The bigger gap I see is not technical. It's that nobody owns the system. A model can be tested, documented, and monitored, but if no single person is accountable for its outputs, none of that documentation means anything when something breaks. Assign an owner before you assign a metric.
The conventional advice also overweights exotic bias metrics before a team has even built a representative test set. Get the test set right first. A disaggregated metric run on a biased or tiny sample tells you less than no metric at all.
If you're starting from a prototype, prioritize evidence you can produce this week: a model card, a few hundred rows of real test data, and one documented incident response step. Everything else can wait for the next audit cycle.
— Hanad Kubat
Fixed-Price Prototype Audit: What I Deliver
Most AI audit checklists assume you already have a security team, a data governance function, and months to spend. Hanad Kubat is the alternative: one accountable engineer runs your prototype audit in three to five days at a fixed price on a transparent fee basis, credited against a future build if you continue.
The audit covers system inventory, the core checklist items above, adversarial and access-control checks, and a prioritized remediation list ranked by what actually breaks first under real users. If your prototype came out of Lovable, Bolt, Replit, Cursor, or a similar AI-assisted builder, that prototype becomes the specification for what gets fixed. You get one evidence pack, one name on the contract, and no surprise invoices along the way. If your AI features also need production-grade RAG, agentic patterns, or EU AI Act compliant architecture, that scope gets appropriately scoped and estimated before any code changes.
To start, request a Prototype Audit and include your current access details, any existing model card or documentation, and a short description of where the system feels shakiest.
Sources
- NIST AI Risk Management Framework (AI RMF)
- AI auditing checklist (EDPB project)
- NIST AI RMF Playbook (AIRC)
- AI Audit Checklist: 8 Steps to Audit an AI System (Redwerk)
FAQ
What Are AI Auditing Tools?
AI auditing tools are software and frameworks that help teams inventory AI systems, run bias and performance tests, log decisions, and generate documentation, ranging from open-source fairness libraries to platforms like the website crawlability audit tool for discovering embedded AI activity across a site.
Can AI Create a Checklist?
AI tools can draft a checklist template quickly, but the resulting document still needs a human to verify evidence, assign owners, and confirm the checklist matches your actual system and jurisdiction's rules.
How Often Should You Run an AI Audit?
Run a full audit before deployment, after any material change to the model or its data, and on a recurring schedule tied to risk tier, with high-risk systems typically reviewed more frequently than low-risk ones.
What Belongs in an AI Audit Evidence Pack?
An evidence pack should contain the model card, system map, risk classification rationale, test results with disaggregated metrics, bias findings and mitigations, security test results, and monitoring logs, all indexed so an outside reviewer can find any item quickly.
Is a Prototype Too Early for an AI Audit?
A prototype is actually the ideal time for a focused audit, since fixing an unnamed owner, a missing test set, or a security gap costs far less before real users and real data are involved than after.