You Are the CTO of Your Agents Now

Delegation floors, onboarding docs, access policies, budgets, reviews and specs: the CTO playbook, rewritten for a team that never sleeps and never says "I'm not sure".

Julio Falbo14 min read
Listen to it, read by Julio Falbo (17 min)
17:00

Hello people!

Today I want to talk about something that has been on my mind a lot lately: what our job becomes when a good part of the code is written by agents running in parallel.

As a Principal Engineer, a big part of my work has always been about people: delegating, onboarding, reviewing and checking in on risky areas before they explode. Lately I noticed that I'm doing exactly the same things with coding agents. Same problems, different tools. And most of the developers I talk to are doing it too, just without noticing.

My idea with this article is to take the classic CTO playbook and translate it, piece by piece, for a team of agents. I'll also show where the analogy breaks (because it does break), how all of this connects with Spec-Driven Development, and I'll finish with a Monday plan made of real prompts you can copy and paste. Let's dive in.

Two headlines, two CTO decisions

Two pieces of news caught my attention recently.

The first one: Garry Tan, from Y Combinator, uses AI agents so much that he spends up to $200K a year on tokens. That is not a tooling bill anymore, that is a salary line.

Then, in an experiment where AI agent swarms were put in Minecraft to build a civilization, the 50-agent teams failed. They drowned in bureaucracy, and more agents meant more coordination and less output.

One headline is about budget and the other is about team size, but both are CTO decisions. If you run more than one coding agent, you are already making them.

"But Julio, I'm not Garry Tan, my bill is tiny." Fair, so let's put it in perspective. His upper bound is about $16,667 a month. Five developers on agent plans at, say, $200 a month each would cost $1,000, which is about 6% of his bill. Small, sure, but the decisions behind it are exactly the same, and they get harder to change the bigger the bill gets.

The job changed under us

For years, a developer's unit of work was the commit: you wrote it, someone reviewed it, and it shipped. Today one developer runs several coding agents in parallel, and each one reads the repo, writes code, runs tests and reports "done".

That makes you a manager, because, as Ed Donner puts it in his AI FDE course, the agent is a tool, and you own everything that reaches production.

I want to go one level higher, though. A manager runs a team, while a CTO designs the system teams run inside: who gets which context, who can touch what, how much it costs and how quality is measured. Once you run more than one agent, you are making all four of these decisions.

Usually by accident.

Why this is harder than managing people

Good CTOs rely on things agents simply don't have, like memory across months, ambition, and the instinct to raise a hand when something smells wrong.

Agents start each session from zero, and they never get tired. That sounds great until you notice they also never get suspicious. And worse, they are very good at looking finished: the final message says "done", the tests the agent chose to run pass, and the summary is confident and well formatted (with headings, bullet points, the whole thing). None of that tells you the feature works.

And no, speed won't rescue you. A randomized controlled trial by METR found that experienced open source developers, working in their own mature repositories, took 19% longer when using AI tools, and the funny part is that they expected to be faster. In my view the lesson is simple: agents produce code fast, but they don't produce verified code fast.

A human team ships a bad pattern slowly. An agent team ships it in an afternoon.

The CTO playbook, translated

1. Delegation: pick the right floor

My favorite delegation tool is Jordan Cutler's One-Pager Delegation Framework (amazing article by the way), which I wrote about before in my article on going beyond the one-pager. Picture an elevator, where each floor is a section of a one-pager: Problem, Goal, Solution, Alternatives, Risks. Staff+ engineers get only the Problem, seniors get the Goal, mid-levels get a Solution, and juniors also get Alternatives and Risks.

For execution work, agents live on the ground floor, always.

"Ok Julio, but agents write code like seniors, why treat them like juniors?" Great question, and I expect pushback here. But writing like a senior is not framing like one: give an agent only the problem and it will solve a problem, but not necessarily yours.

Here is a classic way it fails. A PM asks for a way to update configuration values without deploying. Engineers ask AI to help, and AI happily generates thousands of lines (role-based access control, audit logs, version history, a dynamic form builder, a React frontend). The real need was five values, updated by two people, maybe four times a year. A config table and a CRUD screen, done in a day.

Nobody asked "how often does this change, and who changes it?", and that question is your job, not the agent's. Don't use a bazooka to kill a fly (I know, one more cliche).

So the brief carries everything: the Problem and the Goal, the Solution, the Alternatives you rejected and the Risks you worry about. Then you add what the framework recommends for juniors, which is showing what good looks like: an existing endpoint to copy or a test file in the house style. Examples cut misfires with juniors, and they cut misfires with agents for the same reason.

2. Onboarding: SKILL.md and AGENTS.md

A CTO doesn't explain the deploy process in every 1:1. It lives in an onboarding doc, and for agents that doc is a Markdown file.

Anthropic's Agent Skills introduced SKILL.md files, and I like to think of them as executable operational knowledge: you write down what your best engineer would do, and the agent follows the playbook. Version them next to the code. Every agent that loads the file starts from the same procedure, which removes a whole class of inconsistency. It doesn't guarantee compliance, though, and that's why section 6 still checks outcomes.

AGENTS.md, an open format for guiding coding agents, covers the always-on part: the project conventions a coding agent loads at the start of every session. Skills cover the on-demand part: the procedures the agent loads when the task matches.

One practical tip: check which file your tool actually reads, because some use their own name, such as CLAUDE.md for Claude Code. Before you create a second file, look at your tool's documentation to see whether it already reads AGENTS.md directly. If your tool only reads its own file, don't duplicate the content. Claude Code lets a CLAUDE.md import other files with the @ syntax, so a CLAUDE.md with a single line, @AGENTS.md, keeps one source of truth for every agent you use.

And one rule I follow: if an agent makes the same mistake twice, the fix goes into the file. Correcting it in the chat is a hallway conversation that nobody remembers.

3. Access policy: the gateway and permission modes

New hires don't get production credentials on day one, and agents shouldn't either.

At the organization level, I put an AI Gateway in front of everything: a centralized layer that connects to repositories, docs, tickets and data warehouses, and exposes them to agents under clearly defined rules. On your laptop, the equivalent is the permission mode of your coding agent.

And scope matters more than it looks. MCP (the Model Context Protocol, a standard way to give agents tools) is great, but a poorly scoped MCP tool can expose data the model was never meant to see.

4. Org design: fewer agents, clearer boundaries

Every CTO learns that coordination cost grows with headcount, and the Minecraft experiment, even being a single data point, says agents are not exempt: fifty agents passing text to each other is a meeting that never ends.

"But Julio, how many agents is too many?" Honestly, I don't have a magic number for you, but I have the answer I already use for services: give each agent a bounded context (a part of the domain with its own model and language, the core idea of Domain-Driven Design), and then define the contract between contexts.

An agent working inside clean boundaries produces better code than one trying to make sense of a tangled codebase. If you can't draw the boundary, you probably don't need the agent.

5. Budget: tokens are headcount

If tokens can reach a salary line, budget them like one. Each agent workflow gets an owner, a monthly ceiling and a reason to exist. If your gateway also proxies model calls, that is where the ceiling lives. If it doesn't, set the ceiling as a spend limit on the provider workspace or project each workflow uses. On a flat plan, the plan is the ceiling, so give each workflow its own seat or key if you need separate budgets.

No owner, no tokens.

6. Performance reviews: measure outcomes, not rationale

Ask an agent for a rationale and it will write a very plausible one, but that proves nothing.

Look at Claude Code, for example. One good explanation for why it improved so fast, which I picked up in Ed Donner's AI FDE course on Udemy (Forward Deployed Engineer and Agentic AI Architect), is that its results are measurable: the code runs, the linters pass, the tests pass. Presentations are harder to measure, so they drift into slop. To me that is the most useful idea in the whole agent conversation right now (and keep it in mind, because it comes back in a few sections).

So your performance review for agents is not their summary, it is tests, linters, and a check against real system state. Read the summary last.

7. 1:1s: checkpoints on risk areas

With people, I schedule check-ins on risk areas, something like "let's review memory usage after you deploy the caching module". With agents, this becomes phased work.

Start simple, work incrementally, and define a measurable success criterion for each phase (Ed Donner's advice again). Then put a hard stop before the risky phase. That stop is your 1:1.

Where the analogy breaks

With humans, I champion psychological safety: people stay silent when they fear consequences, safety removes the fear, and the problems surface.

Agents are not afraid, they are agreeable. They don't hide failure to protect themselves, they emit a completion signal because the task rewards one. So the agent version of safety is not warmth, it is skepticism. I still write an open door into every brief, "if you are unsure, say so, and list what you did not verify", and then I verify anyway.

Democratic leadership breaks too. I still believe in it for human teams, but with agents I collect proposals, never votes. While I'm still choosing a solution, two agents can give me two alternatives. Once I pick one, the agent that builds it gets the full ground-floor brief, and the decision and the accountability stay with me.

Mentorship is the last casualty: humans grow between projects, while agents only grow through your SKILL.md files and your tests.

If you didn't write it down, nothing was learned.

Spec-Driven Development: the elevator, going down

And that last line is exactly where Spec-Driven Development enters the conversation.

If you haven't met it yet, Spec-Driven Development (SDD) flips the usual order of work. Instead of jumping straight into code (or straight into a chat with an agent) and documenting later, you write the specification first, and the spec becomes the source of truth. You describe what to build and why, then a technical plan, then a list of small tasks, and only then the agent implements, task by task, checking its work against the spec. GitHub's open source Spec Kit organizes this as a sequence: a constitution with the project's non-negotiable principles, then specify, plan, tasks and implement. Kiro, from AWS, follows a similar shape with requirements, design and tasks files.

In other words, SDD treats the spec the way we used to treat code: versioned, reviewed, and the first thing you change when something is wrong.

When I put the SDD phases next to the playbook above, I noticed something.

That's when it clicked.

Spec-Driven Development is the delegation elevator, ridden from the top floor down to the ground floor, by you, in writing, before the agent starts.

Look at the floors again:

In the delegation framework, the more senior the person, the higher the floor where you leave them. SDD makes you play the Staff+ engineer who receives only the problem, and then forces you to walk every floor down yourself until the brief is ground-floor ready. The agent never has to guess what lives on the upper floors, because you already wrote them.

"But Julio, if SDD already does all this, why do I need the CTO mindset?" Amazing question. Because, in my view, SDD needs this mindset more than the mindset needs SDD. Here is why.

A spec is prose, and prose drifts into slop. Remember the idea from the performance reviews section: things that are hard to measure drift. A spec is a presentation of intent, and an agent can generate a beautiful 600-line requirements document in seconds. Ask it to spec the config problem from section 1 and it will happily spec the RBAC, the audit logs and the React frontend for you, in perfect Markdown. The bazooka just moved from the code to the documents. The CTO mindset fixes that by making every requirement end in evidence: the test that must pass, the linter that must be clean, the state you will check in the real system. A requirement without evidence is a wish.

Specs are the memory agents don't have. Agents start every session from zero. A versioned spec is the memory across months that they lack, and it turns my rule from the onboarding section into a workflow: when an agent gets something wrong twice, you don't argue in the chat, you fix the spec or the constitution and run the phase again.

One spec per bounded context. If a spec touches three contexts, it's three specs and a contract between them. That keeps the Minecraft bureaucracy out of your repository.

Plan is where you collect proposals, not votes. The plan phase is the perfect place to ask two agents for two alternatives. You pick one, you write down why the other was rejected, and the agent that implements receives the decision, not the debate.

The spec defines "done" before the agent starts. And that, to me, is the whole point of being the CTO of your agents.

Your Monday plan

Ok, enough theory. Let's make this concrete.

My favorite starting point is Matt Pocock's grill-me skill. The idea is simple and brilliant: instead of you briefing the agent, the agent interviews you about your plan, relentlessly, one question at a time, until the two of you share the same understanding. It inverts the direction of the conversation, and that alone surfaces most of the questions you forgot to ask yourself.

What I wanted was an evolution of it that walks the elevator floors, asks the CTO questions on purpose, and ends with an artifact instead of just a shared understanding: a ground-floor one-pager that doubles as your spec.

So here is your Monday, in five prompts and one checklist. Replace <task-name> and the description with your own.

Step 1: Grill the brief

Before any code, let the agent interview you through the floors.

Text
Grill me about this task before we write any code: <one sentence describing the task>.
Ask one question at a time and wait for my answer.
Walk through these floors, in order: Problem, Goal, Solution, Alternatives, Risks.
Always ask: how often does this change, who changes it, and what is the smallest thing that would solve it?
If my answers point to something smaller than what I asked for, tell me.
Do not propose an implementation until every floor has an answer.
At the end, write the result as a one-pager in docs/specs/<task-name>.md.

The "smallest thing" question is the one that would have saved the config project from section 1.

Step 2: Find what good looks like

Juniors and agents misfire less when they have an example to copy.

Text
Read docs/specs/<task-name>.md.
Find the 2 or 3 files in this repository that best show what good looks like for this task
(an existing endpoint, a test in the house style, a migration).
For each one, give the path and one line on why it is a good example.
Do not write any code. Add the list to the one-pager under "Examples of good".

Step 3: Turn "done" into evidence

This is the step that keeps the spec from drifting into slop.

Text
For each goal in docs/specs/<task-name>.md, write the evidence that would prove it is done:
the test that must pass (name and file), the lint or type check command,
and any check against real system state (a query, an API call, a log line).
Mark with UNVERIFIABLE every goal that has no concrete check, and ask me how to make it checkable.
Add the result under "Definition of done".

If you end up with a lot of UNVERIFIABLE marks, that's not a problem with the prompt. That's the prompt doing its job.

Step 4: Cut it into phases with a hard stop

This is your 1:1, scheduled before the work starts.

Text
Split the work in docs/specs/<task-name>.md into phases.
Each phase needs a measurable success criterion taken from the "Definition of done".
Identify the riskiest phase (data migrations, auth, payments, anything hard to undo)
and put a STOP before it: finish the phase before it, show me the evidence, and wait for my approval.
Add the phases under "Plan".

Step 5: Audit your agent team

This one is for you, not for the agent. For every agent session or workflow you have open, fill this in:

Text
Agent: <session or workflow name>
Bounded context: <the part of the domain it owns>
Owner: <a human name>
Monthly ceiling: <the spend limit on the gateway, workspace or key it uses, or: shared flat plan, <plan name>>

If you can't fill Bounded context or Owner, close the session. No boundary, no agent. No owner, no tokens.

Step 6: Close the loop before accepting "done"

Don't wait for the end of the task to paste this one. When you hand the spec to the implementing agent, open the brief with:

Text
Implement docs/specs/<task-name>.md phase by phase.
At every STOP, show me the evidence and wait for my approval.

Then append the block below to that same brief. If you prefer to keep it in AGENTS.md, start the block with "If this task has a spec in docs/specs/," so it does not fire on tasks without one:

Text
Before you say "done": go through every item in the "Definition of done"
and show the evidence for each one (command output or test result).
Then list what you did not verify, and why.
If you hit the same problem more than once during this task,
propose one rule for AGENTS.md that would have prevented it.

Then rerun the commands from the Definition of done yourself, or let CI run them, and do the real-state check with your own hands. The agent's evidence tells you where to look. It is not the verdict.

The last line of the prompt is how your agents "grow" between projects: through the file, not through the chat.

Bonus: turn it into a skill

Once you have used these prompts a couple of times, stop pasting them. Put steps 1 to 4 into a SKILL.md so every agent runs the same interview:

Markdown
---
name: grill-me-cto
description: Interview the user about a task before any code is written, walk the delegation floors (Problem, Goal, Solution, Alternatives, Risks), and produce a ground-floor one-pager with examples of good, a definition of done backed by evidence, and phases with a hard stop before the riskiest one. Use when the user asks to plan, brief or spec a task.
---

# Grill me like a CTO

1. Ask one question at a time. Wait for the answer before the next one.
2. Walk the floors in order: Problem, Goal, Solution, Alternatives (and why they were rejected), Risks.
3. Always ask how often this changes, who changes it, and what the smallest solution is.
   Say so when the request looks bigger than the need.
4. Find 2 or 3 files in the repository that show what good looks like. List paths only, no code.
5. Turn every goal into evidence: a test, a lint or type check command, or a check against real system state.
   Mark anything unverifiable and ask how to make it checkable.
6. Split the work into phases, each with a measurable success criterion.
   Put a STOP before the riskiest phase.
7. Propose a short kebab-case task name and confirm it with the user. Write everything to docs/specs/<task-name>.md under the headings Problem, Goal, Solution, Alternatives, Risks, Examples of good, Definition of done and Plan. Do not implement anything.

Here is the nice part: the file this skill writes is your spec. If you use Spec Kit or Kiro, it gives you a much sharper starting point for the specify and plan phases. If you don't use any SDD tool, it is still the ground-floor brief your agent needed from the beginning. Either way, you rode the elevator down yourself.

Wrapping up

Your agents will always tell you they finished. Being their CTO means deciding what finished means before they start, and Spec-Driven Development is simply that decision, written down and versioned.

I hope this playbook helps you run your agents with the same care (and a lot more skepticism) you would bring to a human team: clear briefs, knowledge written down, scoped access, a budget with an owner, specs that end in evidence, and evidence before "done". Thanks for reading it, and welcome to your new job as CTO!

References

Share on LinkedIn

Comments

No comments yet. Be the first.

We email you a 6-digit code to publish it. Your email is never shown.

← More articles · follow by RSS