When I started building agents, I thought the hard part was the prompt. Write a good one, call the model, done. Then I tried to make an agent write a full backend from one line of English, and I realized the model is actually the smallest part of the whole thing. Everything around it is what decides if it works once on your laptop, or works every time for many people.
In this post I'll explain what "agents at scale" really means to me, using a project I built : agent-that-writes-backend. You send it something like "a REST API for a todo app where users can have many todos", and it gives you back an Express + TypeScript + Prisma project that actually compiles.
The First Version Was Just a Loop
My first version was very simple. An Express endpoint received the prompt, called the model, waited for the code and sent it back. It worked for a demo. But generating a backend takes minutes, not seconds. The HTTP request would time out, and if two people used it at the same time, my machine just choked.
That's when I understood that scale for agents is not about thousands of requests per second. It's about work that is long, slow and unreliable, and still finishing it properly.
Take the Agent Out of the Request
The first real change was moving the agent out of the request cycle. Now the API does only small, fast things : it validates the body with Zod, applies a rate limit (a stricter one on the generate route, because it's expensive), puts a job in a BullMQ queue on Redis and replies with 202 and a job id. That's it.
A worker, which is a completely separate process, picks up the job, runs the agent and reports progress from 0 to 100. The client just polls GET /jobs/:id. If I need more throughput, I start more workers. The API doesn't even notice.
Two small settings here taught me a lot:
// a whole generation takes minutes, don't let BullMQ think it stalled
lockDuration: 10 * 60 * 1000,
// LLM calls already retry on their own. Re-running the whole
// job on failure just repeats minutes of work
attempts: 1,Normally you would want a queue to retry failed jobs. But here a retry means paying for every model call again. So the retries live at the smallest level - one model call - and not at the job level.
Treat the Model as an Unreliable Part
If you ask a model to write a backend one file at a time, you usually get code that looks right but doesn't compile. Imports point at files that don't exist, exports don't match, the Prisma schema has broken relations, and packages are used without being installed. Small local models do this far more often than the big ones.
“Constrain what the model writes, check every output with real tools, and feed concrete errors back until it compiles.”
That one line became the design of the whole project. The agent is not one big prompt anymore, it's a pipeline.
Plan First, and Fix the Plan in Code
The planner agent doesn't return free text. It returns a file list that has to match a Zod schema (generateObject in the Vercel AI SDK). Even then, small models make silly mistakes : backslashes in paths, duplicate files, a missing schema file. Instead of hoping the prompt fixes that, a normal function cleans the plan up, then it's checked for path traversal and a max file count.
Boilerplate like package.json, tsconfig.json, the Prisma client and the error middleware never touch the model at all. They come from templates with pinned versions. The model only writes what is actually specific to your project. Less to generate means less that can go wrong.
Generate in the Right Order
Files are written lowest layer first. The schema comes before the services, services before controllers, controllers before routes. So when the model writes a controller, the real service file is already in its prompt, with the real export names.
But you can't keep putting every file into the prompt. The context window runs out. So there is a budget of about 24,000 characters (around 6k tokens). The closest dependencies go in first, and if a file is too big, only its export names go in. Each layer also gets one small reference example, because small models copy a concrete pattern much better than they follow abstract rules.
Check Everything With Real Tools
Every file is checked the moment it's written, while its context is still fresh. The schema goes through the real prisma validate. TypeScript files are checked for syntax, for imports that point to real files, for imported names that are actually exported, and for packages that are actually installed.
If a file fails, it goes back to the model with its previous version and the exact errors, up to three times. This was one of my biggest learnings : fixing code it can see works much better than asking the model to start again from zero.
The Compiler Is the Final Judge
Pattern checks can't catch type errors. So after all files are written, the worker does what a real developer would do : npm install, prisma generate and tsc --noEmit. Compiler errors are grouped by file and sent back for repair, for up to three rounds. The final result tells you honestly what happened :
{
"success": true,
"message": "Backend generated successfully",
"compile": { "status": "passed", "repairRounds": 0 },
"duration": 146253
}The status can be passed, failed (with the errors that are left), skipped or disabled. I like this a lot. An agent that tells you it failed is far more useful than one that pretends it worked.
Concurrency Is a Resource Question
At work I deal with cost and resources every day, and this project showed me the same thing at a small scale. A local model serves one request at a time. So sending it ten calls in parallel doesn't make anything faster, the calls just wait in a line.
// A local model serves one request at a time,
// so parallel calls only queue up
const FILE_CONCURRENCY = config.llm.provider === "ollama" ? 1 : 3;A few more things I only learned by running it:
- Every model call gets its own timeout. Without it, one hung local model blocks the worker forever.
- Ollama's OpenAI-compatible API uses a 4096-token context by default and silently cuts longer prompts. The repo ships a Modelfile that builds a 16k context version of
qwen3:8b. - "Thinking" is turned off for the local model. It was several times slower and leaked
<think>blocks into the code.
On my M-series Mac with 16 GB of RAM, a two-resource CRUD backend (around 19 files) takes a few minutes with qwen3:8b. Results change a bit from run to run, and the compile-and-repair step is what makes up for that.
Know What Happened
When something runs for minutes in another process, you can't just look at the terminal. Every log line carries the request id, the job id and the component (planner, code-generator, orchestrator). So when a job fails, I can follow it from the HTTP request all the way down to the exact file and attempt that broke.
What I Learned
Making an agent work at scale is mostly normal backend engineering : queues, workers, timeouts, retries at the right level, budgets, and good logs. The model is one component inside that system, and an unreliable one. So you don't trust it, you check it. You don't restart it, you show it its mistakes.
The project is still growing - the frontend isn't built yet - but the backend is all there. If you want to see how it works, the code is on GitHub.
The model writes the code. The system decides if it's good enough.