Rate Limiting and Spend Caps for AI Routes in Next.js

The gate that was not a gate

There is a terminal widget in the corner of this site. You can type a question into it and a model answers. No login, no key, no rate-limit header from some upstream service protecting me. Just a route on the public internet that costs money every time someone pokes it.

I had already written the Laravel version of this argument, where the spend cap lives in a queued job and the budget is per run. A queue makes that easy, because the job is the natural place to hang a budget on. A streaming HTTP route gives you nowhere obvious to put it, which is how I got the ordering wrong.

I wrote the spend cap first, because that is the one that keeps you awake. Read the month's spend, compare against a ceiling, refuse if we are over. Then stream the answer, and when the stream finishes, record what it cost.

That ordering is wrong, and it took me embarrassingly long to see why.

Two requests arrive at the same moment. Both read the ledger. Both see the same number, because neither has finished streaming yet, so neither has recorded anything. Both pass the check. Now make it twenty requests. Every one of them reads a spend figure that predates all the others, and twenty calls sail through a gate that was supposed to stop the second one.

The cap was not a cap. It was a report on what had already happened.

Reserve, then reconcile

The fix is to charge yourself before you spend, not after:

// Worst case for one call: the whole system prompt billed uncached,
// plus a maxed-out response. It only has to be an over-estimate.
const MAX_COST_USD = costUsd({
  input_tokens: Math.ceil(SYSTEM_PROMPT.length / 4) + MAX_INPUT_CHARS,
  output_tokens: MAX_OUTPUT_TOKENS,
});

if (overSpendCap(SPEND_CAP_USD)) {
  return NextResponse.json({ error: "AI is resting." }, { status: 503 });
}

// Reserve BEFORE the stream opens.
recordSpend(MAX_COST_USD);

Now the twentieth caller sees nineteen reservations already on the books. The ordering does the work, not the arithmetic.

The reservation is deliberately pessimistic. It assumes a full cache miss on the system prompt and a response that runs to the token ceiling, which is almost never what actually happens. So when the stream ends, you give the difference back:

const usage = await result.usage;
recordSpend(
  costUsd({
    input_tokens: usage.inputTokens ?? 0,
    output_tokens: usage.outputTokens ?? 0,
  }) - MAX_COST_USD,
);

A negative delta is a refund. In practice this hands money back on nearly every call, which is exactly what you want from a brake: tight when it matters, loose the rest of the time.

One case I decided not to be clever about. If the stream throws halfway, there is no refund. I do not know what the provider billed for partial output, so the reservation stands and the ledger stays pessimistic. That errs toward tripping the cap early, and early is the correct direction for a spend breaker to be wrong.

The part where I tell you what it does not do

That ledger is a module-level object in one server process:

const ledger = { month: "", usd: 0 };

Parallel instances each get their own. Cold starts wipe it. So this is a per-instance brake, not a global budget, and anyone telling you a counter in process memory is a spend limit on a serverless platform is selling something.

The hard ceiling lives where it can actually be enforced, which is the AI Gateway spend limit on the Vercel project. The code in the route cannot reach that and does not pretend to. What the in-process ledger buys is narrower and still worth having: a burst of concurrent requests on one instance cannot all read zero and slip past together.

I would rather ship a small guarantee I can describe precisely than a large one I would have to hedge.

Rate limiting, and the header you should not trust

The spend cap is the money gate. The rate limit is the abuse gate, and it keys on client IP:

export function clientIp(req): string {
  return (
    req.headers.get("x-real-ip")?.trim() ||
    req.headers.get("x-forwarded-for")?.split(",")[0]?.trim() ||
    "unknown"
  );
}

Order matters here too. x-forwarded-for is a list, and behind a proxy that appends rather than overwrites, a client can prepend whatever it likes to the front. Read the leftmost entry as gospel and you have built a rate limiter with a free bypass header. x-real-ip is set by the platform on Vercel, so it goes first.

That trust assumption is Vercel-shaped. Move this behind a different edge and you need to check which header that edge actually controls before any of it holds.

The limiter itself is a Map with expiry, which grows. Pruning expired entries is not enough on its own, because rotating the key defeats expiry-only cleanup: fresh entries never expire during the window, so the map climbs while every sweep finds nothing to delete. There is a hard size cap that evicts oldest-inserted past a threshold, so an attacker cycling keys evicts their own entries instead of my memory.

Production checklist

• Reserve worst-case cost before opening the stream, reconcile after. Ordering, not arithmetic.
• Do not refund a failed stream unless you know what was billed.
• Gate on content-length before parsing the body. Cheap rejection beats parse CPU.
• Cap input characters and output tokens. Both, separately.
• Key rate limits on a header the platform controls, and write down which platform you assumed.
• Give the limiter a hard size cap, not just expiry.
• Put the real ceiling at the provider or gateway. Treat in-process counters as a burst brake and say so out loud.

Where this actually stops

Worth being precise about the ceiling, because it is not the number in the constant. The in-process ledger caps one instance at ten dollars. Run four instances and the real exposure is four times that, which is why the enforceable limit has to sit at the gateway and the code can only ever be the burst brake underneath it.

That is the honest shape of most guardrails in a serverless runtime. You get an ordering guarantee inside one process and a billing guarantee outside it, and the useful engineering is knowing which one you just wrote.

If you have a Node or Next.js app about to put a model behind a public route, that division is the first thing I look at in an AI integration audit for Node and Next.js - usually before anything about prompts.

I'm Aditya Kumar (adityakdevin) - Tech Lead & full-stack developer building AI-powered web products with Laravel, Vue, and LLM APIs. Find me at adityadev.in.