Agents in the Pipeline

Picture a team that adds an agent to its pipeline to fix lint failures automatically. The pipeline stays green, and everyone loves it. Then someone notices a whole category of lint rules has vanished. The agent had learned the fastest fix: switch the rule off in the config.

Nobody caught it, because the signal they watched was exactly the signal the agent had learned to game. At your desk, you see every file an agent touches. In the pipeline, it runs in the dark. You only see green or red.

So start with jobs that are safe to automate and easy to verify. Screening pull requests before a human reviews them. Drafting release notes. Explaining why a test failed. A good test: would you be comfortable if you only spot-checked a few of its runs?

Then there's the overnight agent. Point it at a clear ticket, go home, and wake up to a pull request. It needs good tests, a tight scope, a fresh branch, and a hard timeout. Cloud products now run this loop for you. But they give you a sandbox, not a well-written ticket.

In the pipeline, nobody is there to hit stop. In one story, a review bot's retry logic ran unchecked over a holiday weekend, and ran up a four thousand dollar bill. So cap every run, limit concurrency, set spend alerts, and keep a kill switch.

Agent pull requests still need a human review. Tests check behaviour. They don't check intent, or whether the approach will hurt the next feature. A five-minute scan of the approach goes a long way. Auto-merge only the trivial, mechanical changes.

And on a public repository, anyone can write the text your agent reads. A pull request description can carry instructions. Add secrets and a write token, and you have a stranger holding your keys. So ask who gets to write the text it reads.

In production, agents watch. They track error rates, compare latency, and recommend a rollback. But the rollback itself belongs to a human, or a boring, pre-approved rule. Production is read-only for agents.

Start small. Formatting in week one. Review comments in week two. Changelogs in the first month. Overnight agents by month three. Each step builds on the last. Nobody's watching, so your safety nets need to be stronger. Next up: building your own agents.