AI governance for software systems: controls that fit delivery

AI governance becomes useful when it changes how a system is designed, evaluated, released, and monitored. A policy that sits outside delivery creates paperwork; an engineering control creates evidence at the same point a team makes a decision.

This guide focuses on applied AI and language-model systems. It complements the foundational notes on artificial intelligence, machine learning, and deep learning.

Establish the system boundary

Begin with an inventory entry that describes the complete system, not only the model. Record the user, intended decision, model and data providers, retrieval sources, tools, human checkpoints, outputs, and downstream actions. Assign accountable technical and business owners.

The boundary should make dependencies visible. A retrieval index, prompt template, policy filter, and external API can each change system behaviour even when the underlying model does not change.

Classify consequences before selecting controls

Risk depends on how output is used. A drafting assistant with mandatory human review has a different consequence profile from a system that can change access, move money, or communicate externally without review.

Classify the system according to impact, reversibility, affected users, data sensitivity, autonomy, and exposure. Use that classification to determine evaluation depth, approval requirements, monitoring, and fallback behaviour.

Turn requirements into testable controls

Translate principles into checks a delivery pipeline or reviewer can verify. Examples include:

• approved model and data-provider versions;
• documented data lineage and retention;
• prompt-injection and data-exfiltration tests;
• quality thresholds for defined task sets;
• access controls for tools and retrieval sources;
• human confirmation before consequential actions;
• immutable records of configuration, evaluations, and approvals.

Controls should identify their evidence. A statement such as “the system is fair” is not a control. A defined evaluation dataset, metric, threshold, owner, and review cadence is.

Evaluate the system, not just the model

Test representative end-to-end tasks, including refusal, uncertainty, retrieval failure, malformed inputs, conflicting instructions, and unavailable dependencies. Include regression cases drawn from production failures and near misses.

For generative systems, separate factuality, relevance, instruction following, safety, latency, and cost. A single aggregate score can hide a failure that matters to users.

Limit authority by default

Use the least privilege required for tools, data, and actions. Prefer read-only access, scoped credentials, explicit allowlists, bounded execution, and user confirmation for irreversible operations. Treat model output as untrusted input at every integration boundary.

Monitor change over time

Models, prompts, retrieval content, and external services change. Record the deployed configuration, watch for quality and safety drift, and define when a change requires re-evaluation. Provide a clear shutdown or fallback path when the system moves outside approved limits.

Sources and further reading

• NIST AI Risk Management Framework
• NIST Generative AI Profile
• OWASP Top 10 for LLM applications