---
title: "AI governance for software systems: controls that fit delivery | Dhruv Doshi"
description: "A delivery-focused guide to AI inventories, risk boundaries, evaluation, human oversight, and operational evidence."
canonical: https://doshidhruv.com/guides/ai-governance/
author: Dhruv Doshi
---

[Skip to content](#main-content)

[Dhruv Doshi](/)

[Work](/projects)[Experience](/resume)[Notes](/notes)[Guides](/guides)[Research](/research)[About](/about)[Contact](/contact)[Search /](/search)dark mode

[All guides](/guides)

AI governance · Staff engineering

# AI governance for software systems: controls that fit delivery

A delivery-focused guide to AI inventories, risk boundaries, evaluation, human oversight, and operational evidence.

Published 2026-07-28 · Last reviewed 2026-07-28 · 8 minute read

AI governance becomes useful when it changes how a system is designed, evaluated, released, and monitored. A policy that sits outside delivery creates paperwork; an engineering control creates evidence at the same point a team makes a decision.

This guide focuses on applied AI and language-model systems. It complements the foundational notes on [artificial intelligence](/notes/introduction-to-artificial-intelligence-history-and-evolution/), [machine learning](/notes/the-fundamentals-of-machine-learning/), and [deep learning](/notes/deep-learning-explained-from-basics-to-advanced/).

## Establish the system boundary

Begin with an inventory entry that describes the complete system, not only the model. Record the user, intended decision, model and data providers, retrieval sources, tools, human checkpoints, outputs, and downstream actions. Assign accountable technical and business owners.

The boundary should make dependencies visible. A retrieval index, prompt template, policy filter, and external API can each change system behaviour even when the underlying model does not change.

## Classify consequences before selecting controls

Risk depends on how output is used. A drafting assistant with mandatory human review has a different consequence profile from a system that can change access, move money, or communicate externally without review.

Classify the system according to impact, reversibility, affected users, data sensitivity, autonomy, and exposure. Use that classification to determine evaluation depth, approval requirements, monitoring, and fallback behaviour.

## Turn requirements into testable controls

Translate principles into checks a delivery pipeline or reviewer can verify. Examples include:

- approved model and data-provider versions;
- documented data lineage and retention;
- prompt-injection and data-exfiltration tests;
- quality thresholds for defined task sets;
- access controls for tools and retrieval sources;
- human confirmation before consequential actions;
- immutable records of configuration, evaluations, and approvals.

Controls should identify their evidence. A statement such as “the system is fair” is not a control. A defined evaluation dataset, metric, threshold, owner, and review cadence is.

## Evaluate the system, not just the model

Test representative end-to-end tasks, including refusal, uncertainty, retrieval failure, malformed inputs, conflicting instructions, and unavailable dependencies. Include regression cases drawn from production failures and near misses.

For generative systems, separate factuality, relevance, instruction following, safety, latency, and cost. A single aggregate score can hide a failure that matters to users.

## Limit authority by default

Use the least privilege required for tools, data, and actions. Prefer read-only access, scoped credentials, explicit allowlists, bounded execution, and user confirmation for irreversible operations. Treat model output as untrusted input at every integration boundary.

## Monitor change over time

Models, prompts, retrieval content, and external services change. Record the deployed configuration, watch for quality and safety drift, and define when a change requires re-evaluation. Provide a clear shutdown or fallback path when the system moves outside approved limits.

## Sources and further reading

- [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework)
- [NIST Generative AI Profile](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence)
- [OWASP Top 10 for LLM applications](https://genai.owasp.org/llm-top-10/)

## Notes behind this guide

- [Production RAG requires retrieval evidence and control](/notes/production-rag-requires-retrieval-evidence-and-control)
- [Introduction to Artificial Intelligence - History and Evolution](/notes/introduction-to-artificial-intelligence-history-and-evolution)
- [The Fundamentals of Machine Learning](/notes/the-fundamentals-of-machine-learning)
- [Deep Learning Explained - From Basics to Advanced](/notes/deep-learning-explained-from-basics-to-advanced)

**Dhruv Doshi** · Toronto, Canada · [work@doshidhruv.com](mailto:work@doshidhruv.com)

[Resume](/resume)[Notes](/notes)[Guides](/guides)[Topics](/topics)[Search](/search)[Research](/research)[LinkedIn](https://www.linkedin.com/in/dhruvdoshi25071999)[GitHub](https://github.com/DhruvDoshi)

[Sitemap](/sitemap.xml)[RSS](/feed.xml)[LLMs](/llms.txt)

© 2026 Dhruv Doshi