Why most AI failures are permission problems, not model problems
What you let AI do matters more than what it can do.
What the AI is allowed to do matters more than what it can do
I do >80% of my work at Eagle Rock through AI. People hear that number and ask what model I’m using. That's the wrong question. The right one — what is the system allowed to do, and who decides when the answer changes — never makes the sales call.
It's not a capability problem. It's a permission problem. Most of the AI failures in the news aren't "the AI did something it couldn't do." Most are "the AI did something we didn't authorize it to do." Same capability. Different boundary.
The permission layer nobody is buying
A mediocre AI with good rules beats a brilliant AI with no rules. The constrained system fails inside the box; the unconstrained system fails outside it — and that's where the news stories come from. It produces work that looks brilliant untill the morning it sends the wrong email to the wrong customer with the operator's name on it.
A junior analyst told what they can and cannot do is more useful than a senior analyst told nothing. Same math for AI. (I keep wanting to soften this. I shouldn't.)
Where teams forget to draw the line
Five places the line gets missed.
Draft vs. send. The AI that drafts does what the operator would have, only faster; the AI that sends does the same thing with the operator's name attached. Most teams wire "human in the loop" to mean a human approved the original prompt, not this specific output.
Read vs. send out. Reading is internal; sending data out isn't. A team grants folder access because the system needs it. Six months later, some of it shows up where it shouldn't.
Reversible vs. irreversible. Updating a database record can be undone; emailing the customer about it cannot. For a finance team on cash flow and audit trails, the irreversible side of the line is where the disasters live.
One action vs. a chain. Each step is reasonable in isolation; the sequence isn't.
Scope at design vs. in practice. A hundred small variations later, the team has built a different system than the one they signed off on. Nobody went back to update the permission.
The five permission patterns AI teams forget to draw
- Every AI permission document answers these five questions in order. If any answer is missing, the system is operating outside its boundary.
- Draft vs. send — Can the system send to a third party, or only draft for review? If send, who approves each output?
- Read vs. send out — Can the system move data out of the system, or only work with internal summaries?
- Reversible vs. irreversible — Can the action be undone? If not, a human approves the specific output before it executes.
- One action vs. a chain — Does the system take one bounded action per request, or can it iterate without human check-in?
- Scope at design vs. in practice — When did someone last read the boundary document and compare it to what the system is actually doing?
The fix is one page, written before deployment
The fix is definitional, not technical. The team writes, in plain language, what the system can do without a human on the specific output, what it isn't, and what happens when the line is crossed. One page. The discipline is the document, not a safety review six months later.
Can it send or only draft. Can it read sensitive data, or only work with summaries. Can it take irreversible actions, or only reversible ones. Can it chain actions, or only do one at a time. Has the scope drifted. If any has no answer, the system is outside the line. (Last Tuesday is how you discover your permissions. It's not how you define them.)
The person whose name is on the deployment signs the document. A document that has to be signed has to be specific — same way controller oversight works only when someone is on the hook for the close.
Trust in AI is the team's job, not the AI's
The capability question is fun. The permission question is boring. The boring question is the one that determines whether the AI you deployed last quarter is the one you wanted. Capability is what sells the AI tool. Permission is what keeps the customer.