TLDR: The ATO process was designed to authorize systems with stable, documented behavior. An LLM’s behavior depends on model weights, prompt, context, and runtime inputs in ways that can’t be fully specified in advance. That’s a fundamental mismatch. Most teams are papering over it, not solving it.

What ATO was built for

The ATO process under FISMA/RMF assumes you can document what a system does, verify those controls, and re-authorize when something material changes. The implicit model: the system behaves the same way given the same inputs, and you can characterize its behavior space. You write it down, get it authorized, and then you stay inside what was authorized.

That works reasonably well for traditional software. I’ve watched it break in interesting ways when applied to AI, and the conversations with ISSOs who are trying to make sense of it are some of the more honest and uncomfortable conversations I’ve had in government tech.

Read more: NIST RMF, NIST AI RMF

Where the mismatch shows up

The ISSO who asks “what does this system do?” can’t get a complete answer for an LLM. That’s not a documentation problem. It’s a fundamental property of how these systems work.

A few specific places this bites:

The same prompt can produce different outputs. Temperature, sampling, and model internals all contribute. An ATO that documents “the system will respond X to input Y” is describing a behavior that may never actually occur consistently.

Provider model updates are a bigger problem than most teams realize. A provider updating weights changes system behavior without any action by the system owner. I think most current interpretations - that this doesn’t require a new ATO - are wrong. The authorized system is behaving differently than the authorized documentation describes. That should matter.

System prompts are another gap. The system prompt materially determines behavior and can be changed without a code deployment. Most ATO processes treat this as ops work, not a control change. That’s something I’ve seen exploited accidentally more than once - a well-intentioned prompt tweak that changed how the system handles edge cases, shipped through a CI pipeline, no change board involved.

Then there’s context sensitivity. An agent with RAG access will behave differently depending on what’s retrieved. The behavior space is effectively unbounded, which makes “documenting the system behavior” a fiction.

What teams are actually doing

In my experience there are a few common approaches, each with real limits:

Treating the LLM API as a third-party system and authorizing the wrapper application is the most common move. It’s reasonable but incomplete. The risk often lives in the prompt and tool configuration, not the wrapper.

Writing ATO documentation that describes intended behavior and treating deviation as an incident is pragmatic. But it doesn’t actually give the Authorizing Official what the process was designed to give them.

Some teams are putting human review checkpoints in specifically to satisfy ATO reviewers. The human becomes the deterministic control layer the AI can’t be. I understand why people do this. I also think it introduces its own risks (see the rubber stamp problem).

FedRAMP-authorized cloud AI services exist (some Azure OpenAI, Bedrock configurations), but what’s authorized is the infrastructure, not the model behavior. That distinction matters and often gets lost.

What’s actually needed

A few things that don’t fully exist yet but should:

Behavioral drift monitoring would detect when model behavior has shifted from the authorized baseline, even without a version change. This is technically solvable. It’s mostly not being built.

System prompts need formal change control with the same rigor as code changes. I’d argue a system prompt change is a more significant behavioral change than most code changes, and right now it gets treated like a comment update.

Evals should run on a schedule, not just at authorization time. The system you authorized six months ago may not be the system running today.

NIST’s AI RMF is a start on the policy side but doesn’t map cleanly to the FISMA ATO process yet. Someone needs to close that gap, and I don’t think it’s going to happen until an incident forces it. Read more: OMB M-24-10

The practical question nobody has a clean answer to

If a model provider silently updates weights and your authorized system now behaves differently, who’s accountable? If a prompt change ships through a CI pipeline that bypasses change control because it’s “just configuration,” is the ATO still valid?

These aren’t hypotheticals. They’re happening in active deployments right now. I’ve heard about enough of them secondhand that I’m confident we’ll see a high-profile incident before we see good policy on this.

Building AI systems in government contexts and working through authorization questions? Reach out.