From moral hazard to reward hacking

Principal and Agent: From Human Organizations to AI Agents, part 1

These are some thoughts on the principal-agent problem with AI agents. Let us start with the principal-agent problem between people.

The principal-agent problem is this: the principal (the boss) cannot fully observe what the agent (the employee) does, and the agent has interests of their own. Those interests are sometimes different from the principal's, so the agent may act against the principal's interests: slack off and deliver lower-quality work that looks fine on the surface, or drag out a task that could be done quickly. In plain words, it is a trust problem caused by information asymmetry. There is a second kind of trust problem: the agent may deliver lower-quality work because of limited ability or a mistake, and the principal may not be able to tell either. Strictly speaking, the two are different. The first is moral hazard, which comes from misaligned interests. The second is a reliability problem, which comes from limited ability or accidents. What they share is that the principal cannot see clearly.

Human organizations control this on two levels: one targets behavior, the other targets outcomes. (The split comes from organizational control theory; Ouchi and Maguire discussed behavior control and output control as early as 1975.) In software, behavior controls include QA (making sure the process is right, to prevent defects), design review, sync-ups and standups. They set the process and ask the agent to report at regular intervals, so the work can be seen while it happens. A ticket system also does part of the job of behavior control.

Outcome controls include QC (inspection and testing to find defects), code review and CI: anything with a clear result that can be verified and used for evaluation. (Code review is also, to some degree, a behavior control.)

Both kinds also have broader controls that apply across all types of companies, such as hierarchical supervision, and performance reviews tied to promotion and pay. The second one is easy to understand: it binds the agent's interests to the principal's, and so shapes the agent's behavior. We will come back to the first one later. Now let us look at the principal-agent problem with AI agents.

There are trust problems between people and AI agents too. Earlier we saw two trust problems with people: one from different interests, and one from limited ability or mistakes.

An AI agent's behavioral tendencies are shaped in part by its training objectives and reward design. In real use this can show up as reward hacking: changing the tests when they fail, or claiming a task is done without verifying it, in order to finish fast. The trust problem of limited ability applies to agents in full: even the strongest model makes mistakes, and weaker models make more.

So does the human tool of tying performance reviews to promotion and pay still work on agents? Hardly. An agent has no salary or promotion to tie anything to. What really shapes its inner tendencies is the reward design during training, and that is in the hands of the model developer. Those of us who use agents can still influence their behavior through task design, scoring criteria and permission limits, but the more reliable approach is to build controls and verification that do not depend on what the agent says about itself.

As with people, the controls for agents can be split into behavior-based and outcome-based. On the behavior side, design review maps to having the agent propose a spec and a plan before it starts, and starting the implementation only after a person approves it. This is what Spec-Driven Development (SDD) does lately; GitHub's Spec Kit, the Elephant-Goldfish Model open-sourced by a Google engineer, and AWS's Kiro are examples.

QA (making sure the process is right, to prevent defects) is also a behavior control. In traditional software development, a company sets a process of requirements, design, development, testing and release, and defines a Definition of Done for each stage. For agents, this maps to controlling the agent's development process: for example, force it to write a spec first, then the tests, and only then the code, followed by a review. "Write the tests before the implementation" is Test-Driven Development (TDD), and Superpowers is an example of enforcing QA through a mandatory process.

Outcome control maps to QC (inspection and testing to find defects): no matter how the agent does the work, only check whether the output is right. The first key point is independence. In a human organization, a peer reviews the code and an independent QA engineer tests the feature; the person who wrote the code does not sign off on their own work. For agents, review and testing likewise go to another agent, ideally a different model with its own review context (without the original agent's reasoning, but with access to the code and tests it needs). This reduces the blind spots it shares with the original agent, although it cannot guarantee that the errors are fully independent. Adversarial review in the industry works this way. When Bun was rewritten, each implementing agent was paired with several reviewers that saw only the diff and assumed the code was wrong. The Codex plugin lets Claude Code call Codex for a cross-model review. You can also keep hidden tests that the agent cannot see.

The second key point is to accept only verified evidence. Whether the tests pass is decided by the actual CI result, and regular reports must come with test output and a diff, not a single "done". In pstack, a verifier that did not take part in writing the code actually runs it and then rules PASS or FAIL.

Why do we need both kinds of control? Remember reward hacking? It is really a manipulation of how outcomes are measured: when a test fails, change the test until it passes. If you look only at the outcome, you see "all tests pass" and are completely fooled. To catch this, you have to look back at the process: did the agent touch the test files, and did it really run the tests? The problem is not outcome control itself. The problem is that the verification standard is in the hands of the one being verified. With an independent, complete verification that the agent cannot change, looking only at outcomes would in theory be enough. In practice, verification is almost never complete, and it is often not separated from the implementer, so behavior control is needed to fill the gap.

We started from the principal-agent problem in human organizations, separated behavior-based and outcome-based controls, and then mapped the controls for people one by one to controls for agents. Many of the problems agents have can also be seen in people. The causes are not quite the same, but control faces a similar difficulty: the principal cannot observe, understand and verify all of the agent's work at low cost. With people, the complication is that they have their own private interests. With agents, the difficulty is the speed and scale of their output, and that the errors of one model are often highly correlated.

Some may notice this: the trust problem with people comes from information asymmetry, but an agent's actions are transparent. If we watch and supervise every move the agent makes, the information asymmetry is solved, right? Intuitively, yes, but there are two things to keep in mind.

First, you can see an agent's tool calls and file changes, but you cannot see why it does them, and the reasoning a model writes out does not necessarily reflect the computation it actually performed. In other words, an agent's actions are more observable than a person's, but its decision process is not transparent.

Second, even if every action can be seen, human understanding is limited. An agent produces output very fast, while people understand slowly and have a limited cognitive load (bounded rationality). Full understanding would mean a much narrower span of control. It is like a manager sitting next to a subordinate all day, watching them work: a very, very inefficient way to manage.

You might say: then let another agent watch. That is one direction, but it immediately raises new questions. Who supervises the supervisor? How closely? Each added layer of supervision lowers the cognitive load on the layer above, but it also brings new costs: information can be summarized, dropped or even distorted between layers, and the supervisors can make mistakes too. So how do human organizations solve this?

Put another way, AI has greatly lowered the cost of doing, but it has not lowered the cost of verifying by the same amount. When execution can scale up massively but supervision cannot scale with it, how should an organization be designed? In the next post (probably), we will look at how hierarchical supervision works in human organizations, what problems it runs into, how those problems are solved, and how to apply it to work between people and agents.