Part 3 · Chapter 3.22
You will produce: Per-environment request counters, the headers on every response, and two limiters that fail in opposite directions · about 90 minutes including the exercise
Chapter 1.3 wrote down a word the platform has never said.
rate_limited: "too many requests",Exported, typed, and emitted by nothing from that day to this one — the rest of
Part 1, the whole of Part 2, and every chapter of Part 3 before this. So is close
code 4008, "quota exhausted", and 4009, "server shutdown (drain)". So,
differently, is the fourth field of the error envelope constitution V has required
since 1.3 — request_id, above a comment promising it "joins in Part 2, when a
gateway exists to mint one". A gateway arrived in 1.4 and minted nothing.
This chapter enforces two of the four, explains why the third stays unused on purpose, and says what the fourth is waiting for.
The limiter itself is easy. Two of them, failing in opposite directions deliberately, is the chapter.
Here is FR-RTL-02, and it is worth reading twice:
Every response — 2xx as well as 429 — carries
X-RateLimit-Limit,X-RateLimit-RemainingandX-RateLimit-Reset.
The 429 half is what everybody builds. The 2xx half is the requirement. A client that only learns its allowance at the moment it runs out has learned it too late: it can back off, but it could not have paced itself. The journey map says the same thing from the developer's side, in the Test phase, as a complaint about somebody else's API:
rate limits that return
429with no headers indicating remaining quota or reset time
That is FR-RTL-02 written by the person it happens to. A limiter bolted onto a finished service sets headers where the refusal is written, because that is the branch that knows about limits. Putting them on the success path means it runs on every request and writes to responses it is not refusing — a different design, decided before the code rather than after.
The obvious alternative is a token bucket, and it is the better algorithm on almost every axis: it smooths bursts, it has no boundary, and it is what you would reach for if the only requirement were "limit the rate".
flowchart TB
subgraph w1["window N · 12:00:00 – 12:00:59"]
b1["600 requests<br/>at 12:00:59"]
end
subgraph w2["window N+1 · 12:01:00 – 12:01:59"]
b2["600 requests<br/>at 12:01:00"]
end
cost["1,200 requests in two seconds<br/>against a limit of 600 per minute"]
b1 --> cost
b2 --> cost
gain["Reset names ONE moment.<br/>A refilling bucket's honest<br/>answer is a curve, and the<br/>header has room for a number."]
cost -.->|"the price"| gain
style cost fill:#78350f,color:#fff,stroke:#d97706
style gain fill:#064e3b,color:#fff,stroke:#059669A fixed window can be spent twice at a boundary: 600 requests in the last second of one minute and 600 in the first second of the next is 1,200 in two seconds against a limit of 600 per minute. That is real and no amount of comment fixes it.
It is the price of X-RateLimit-Reset.
Because the window is fixed, the counter is two Redis commands and no Lua:
const count = await redis.incr(key);
if (count === 1) await redis.pexpire(key, windowMs);There is nothing here to make atomic. INCR is atomic and returns the new value,
so the read and the write are one operation. PEXPIRE is guarded on
count === 1 because only the request that created the key sets its lifetime. A
Lua script here would satisfy a habit rather than a race.
Three functions, no store, no clock:
// The fixed-window arithmetic (research R1).
//
// FIXED WINDOW, NOT A TOKEN BUCKET, and the SAD's own row is why the question
// arose: §6.3 lists `rl:{env}:{bucket}` as "Token buckets" with a TTL of
// "window", which are two different algorithms. The TTL column wins, for three
// reasons in order of weight.
//
// `X-RateLimit-Reset` decides it. The header names the moment an allowance
// returns, and a fixed window has exactly one. A continuously refilling bucket
// does not — the honest answer to "when do I have my full allowance back" is a
// curve, and the header is an integer. A limiter whose reset header is a lie
// fails FR-RTL-02 in the way that matters, because that requirement exists so a
// client can schedule against it.
//
// Then atomicity: `INCR` returns the new value on its own and `EXPIRE` on the
// first increment gives the window. Two commands, no Lua, no read-modify-write
// race between api instances.
//
// Then cleanup: the key expires when its window ends, so nothing accumulates.
// That matters more than it sounds — the deduplication chapter spent a baseline on four suites
// that broke because a shared store grew without bound, and this chapter's own
// baseline found a fifth.
//
// THE COST, stated rather than hidden: up to twice the limit across a boundary.
// 600 in the last instant of one window and 600 in the first instant of the next
// is 1,200 inside two minutes. The limit bounds sustained load; it does not
// smooth instantaneous rate. `bucket.test.ts` asserts it so the claim is checked
// rather than merely written down.
//
// Everything here is pure and takes the instant it should reason about. Nothing
// reads a clock, so a boundary is a test rather than a wait.
/** The window an instant belongs to, floored — and the key's own suffix.
*
* Two api instances compute this from the same wall clock and agree without
* talking to each other, which is what closes the clock-skew case by
* construction. A stored reset time would be a value they could disagree about,
* and `Retry-After` is exactly where that disagreement would surface. */
export function windowStart(nowMs: number, windowMs: number): number {
return Math.floor(nowMs / windowMs) * windowMs;
}
/** When the allowance returns: the end of the current window, in milliseconds.
*
* One moment, which is the whole argument for this algorithm over a bucket that
* refills. Never in the past — the window an instant belongs to always ends
* after it. */
export function resetAt(nowMs: number, windowMs: number): number {
return windowStart(nowMs, windowMs) + windowMs;
}
/** How many operations are left, after counting the one in hand.
*
* Clamped at zero. A limit lowered while a window is open — an operator dropping
* an environment from 600 to 2 with forty already counted — would otherwise
* produce `-38`, and a client would parse that as a number and act on it. Zero
* is both true and safe. */
export function remaining(count: number, limit: number): number {
return Math.max(0, limit - count);
}windowStart floors, which is what lets two api instances and a gateway agree on
which bucket they are incrementing without exchanging a word. Nobody coordinates;
they all divide the same clock the same way.
Being pure, a window boundary is windowStart(59_999, 60_000) === 0 asserted
rather than slept through.
-- Per-environment rate limit policy (FR-RTL-04).
--
-- NULLABLE, AND NULL IS NOT ZERO. A null column means "no override, use the
-- documented default", resolved at read time. A zero means "refuse everything",
-- which has to stay expressible — an environment can be switched off
-- deliberately — so the absent state and the refuse-everything state cannot
-- share a representation.
--
-- ON `environments` RATHER THAN IN A TABLE OF ITS OWN. FR-RTL-04's independence
-- is per environment, there is exactly one row per environment with no history
-- and no versioning, and a separate table would be a join for a value read on
-- every request.
--
-- The shape has a slot for an environment and NONE FOR A ROUTE, which forecloses
-- SRS Appendix C question 5 — whether the dev-token endpoint should be limited
-- more aggressively than the rest of its environment. That question stays open
-- and this is why (research R30).
ALTER TABLE environments
ADD COLUMN rest_limit_per_minute integer,
ADD COLUMN send_limit_per_minute integer,
ADD COLUMN connect_limit_per_minute integer;
ALTER TABLE environments
ADD CONSTRAINT environments_rest_limit_non_negative
CHECK (rest_limit_per_minute IS NULL OR rest_limit_per_minute >= 0),
ADD CONSTRAINT environments_send_limit_non_negative
CHECK (send_limit_per_minute IS NULL OR send_limit_per_minute >= 0),
ADD CONSTRAINT environments_connect_limit_non_negative
CHECK (connect_limit_per_minute IS NULL OR connect_limit_per_minute >= 0);Nullable, all three, and the nullability is the decision.
environments already had a column that looked right for this, and it was
deliberately not used:
quotaConfig: jsonb("quota_config").notNull().default({}),Declared in 2.1, named in SRS §6.1, empty for seventeen chapters — and named for quotas. A rate limit and a quota are different promises: one is ephemeral and may be lost, the other is money and must be durable. Putting one into a field named for the other would collapse in the schema exactly what this chapter spends its length drawing.
Four numbers, and each was derived rather than chosen:
| Operation | Default | Derived from |
|---|---|---|
| REST requests | 600/min | NFR-PRF-01's P95 budget at a sustainable rate |
| Message sends | 600/min | the same, and deliberately equal |
| Connections | 3,000/min | NFR-SCL-01's 10,000 per gateway instance |
| Failed authentications | 10/min/IP | slow enough to stop a sweep, fast enough not to lock out a typo |
The connect limit was originally 60/min, and wrong by a factor of fifty. NFR-SCL-01 is a P1 requirement for ten thousand connections per gateway instance; at 60 a minute that takes 167 minutes. A limit that makes a P1 capacity requirement unreachable in under three hours is a bug with a policy column.
flowchart LR
req(["request"])
rc["RequestContextMiddleware<br/>chapter 2.2"]
am["AuthenticateMiddleware<br/>chapter 3.2"]
rl["RateLimitMiddleware<br/>chapter 3.8"]
cg{"CredentialGuard"}
h["handler"]
req --> rc --> am --> rl --> cg --> h
inside[["the AUTH counter lives<br/>INSIDE this middleware:<br/>it must work when there<br/>is no principal"]]
after[["the TENANT limiter comes<br/>AFTER it: the limit belongs<br/>to an environment and only<br/>this step knows which"]]
am -.-> inside
rl -.-> after
style am fill:#1e3a8a,color:#fff,stroke:#3b82f6
style rl fill:#064e3b,color:#fff,stroke:#059669The tenant limiter runs after AuthenticateMiddleware and has no choice: the
limit belongs to an environment, and nothing knows which one until the credential
is resolved. The failed-authentication counter runs inside it, and has no
choice either: it counts the case where there is no principal, so it cannot run
anywhere that assumes one.
const address = clientAddress(req);
if (await this.authLimiter.isOverThreshold(address)) {
req[OVER_AUTH_THRESHOLD] = true;
}
const principal = await resolvePrincipal(this.db, credential);
if (principal !== null) {
req.principal = principal;
} else {
await this.authLimiter.recordFailure(address);
}The failure is observed here and refused elsewhere: this middleware has never
thrown, since the credentials chapter, so it sets a flag and CredentialGuard raises the 429
from it.
export function operationsFor(method: string, path: string): LimitedOperation[] {
if (!path.startsWith(PUBLIC_PREFIX)) return [];
if (method === "POST" && SEND_PATH.test(path)) return ["rest", "send"];
return ["rest"];
}A message send costs two budgets, one request and one message. Everything else
under /v1/ costs one request, and anything outside it costs nothing — /healthz,
the gateway's internal seam, the dispatcher's outcome reporting.
Throttling the dispatcher would turn one customer's webhook backlog into a stall for every customer, which FR-WHK-05 forbids in as many words.
Here is the chapter.
flowchart TB
out(["Redis is unreachable"])
subgraph tenant["the TENANT limiter · rl:{env}:{op}:{window}"]
t1["count unknown"]
t2["SERVE the request"]
t3["X-RateLimit-Limit only<br/>Remaining and Reset absent"]
t1 --> t2 --> t3
end
subgraph auth["the AUTH limiter · rlauth:{address}:{window}"]
a1["count unknown"]
a2["in-process fallback<br/>same threshold"]
a3["REFUSE past 10/min<br/>per instance, not per fleet"]
a1 --> a2 --> a3
end
out --> t1
out --> a1
why1["a cache outage must not<br/>refuse paid traffic<br/>SAD §6.3"]
why2["an unlimited window on<br/>failed logins is not a<br/>degradation, it is a hole"]
t3 -.-> why1
a3 -.-> why2
style t2 fill:#064e3b,color:#fff,stroke:#059669
style a3 fill:#7f1d1d,color:#fff,stroke:#dc2626Redis goes away. The tenant limiter cannot count, so it serves the request:
$ POST /v1/channels/{id}/messages # 1 of 4, limit is 2
HTTP 201
x-ratelimit-limit: 2
x-ratelimit-remaining: (absent)
x-ratelimit-reset: (absent)
$ POST /v1/channels/{id}/messages # 4 of 4, limit is 2
HTTP 201
x-ratelimit-limit: 2
x-ratelimit-remaining: (absent)
x-ratelimit-reset: (absent)
Four requests against a limit of two, all served. SAD §6.3 says Redis is not a source of truth, and refusing everything because the counter is unavailable turns a cache outage into a platform outage — a much larger failure than the one it prevents.
Notice which headers survive. Limit stays: it is policy read from Postgres and
is not degraded. Remaining and Reset vanish, because they existed only as long
as something was counting. Absent is the honest answer; -1 is a sentinel a
client that does not know the convention parses as a number and reads as "over
your limit".
Now the same outage, the same process, the same instant, against the failed-authentication limiter:
$ POST /auth/dev-token # bad credential 1 of 5, threshold is 3
HTTP 401
$ POST /auth/dev-token # bad credential 2 of 5, threshold is 3
HTTP 401
$ POST /auth/dev-token # bad credential 3 of 5, threshold is 3
HTTP 401
$ POST /auth/dev-token # bad credential 4 of 5, threshold is 3
HTTP 429
$ POST /auth/dev-token # bad credential 5 of 5, threshold is 3
HTTP 429
Refused. Run the tenant limiter's reasoning here and it gives the wrong answer: an unlimited window on failed logins is not a degradation, it is a hole, and an attacker who waits for a cache outage gets an unthrottled password sweep.
One line in the log, rate-limited to one per ten seconds, because a Redis outage under load would otherwise emit one per attempt and turn one outage into two:
{"time":"2026-08-20T03:48:30.986Z","level":"error","service":"api",
"msg":"limits.auth_degraded",
"detail":"counter store unreachable; counting failed authentications in process",
"tracked":0}No credential and no address, per NFR-SEC-06. tracked is how many addresses the
fallback holds, which is what tells an operator whether the cap is close.
The counter is keyed by the client's address, not the caller's, and getting it wrong is a hole rather than an inconvenience. A handshake authenticated through the gateway reaches the api from the gateway; key on the caller and every customer's failures land in one bucket, so one attacker exhausts a threshold that then refuses everybody — a denial of service with a rate limiter for a weapon.
The socket has an establishment limit and a send limit, and its two refusals do
not look alike. An over-limit handshake gets an HTTP 429, written onto the raw
upgrade socket before wss.handleUpgrade is ever called:
if (decision.over) {
refuseUpgrade(socket, decision); // 429 · Retry-After · the three headers
return;
}
wss.handleUpgrade(req, socket, head, (ws) => { … });An over-limit frame gets an error frame, and the connection stays open.
The gateway is exempt from the tenant limiter — its internal calls cost the customer nothing — and simultaneously not trusted to say who caused a failed login. One request, both judgements.
That reads like an inconsistency and is not. "Should this call count against a customer's budget?" is about whose work it is, and the gateway's work is the customer's own socket traffic, already counted at the frame. "Whose failure was that?" is about where a credential came from, and the gateway relays that credential rather than originating it. Trusting a service to be infrastructure is not the same as trusting it to be an origin.
The other thing the gateway cannot do is read a database. ADR-05 forbids it, and The credentials chapter already paid a round trip rather than ship every environment's signing secret to a service holding no tenant state. So the limits ride the authentication response the gateway was already making:
limits: z.strictObject({
connect: z.number().int().nonnegative(),
send: z.number().int().nonnegative(),
}),Adding one required field to that schema broke seven hand-written fixtures, all found by the compiler before a test ran. Making it optional with a default would have broken nothing — and every one of those seven would then have silently exercised the default path, including the two tests whose entire subject is a configured limit.
A socket send and a REST send spend the same budget — a client that could double its allowance by opening a WebSocket has no allowance — which is why the counter lives in Redis rather than in either process. Neither can see the other's memory.
The gateway therefore holds its own Redis client — its ninth, and the first
whose job is to count rather than to carry. Forced rather than preferred, and the
reason is the same for all eight it could have borrowed: every one of them is either
a subscriber, and a connection in subscribe mode cannot run INCR, or it belongs
to a module that is optional in the session server. A limiter riding the fan-out's
lifecycle would vanish in every configuration with no fabric; one riding presence's
would vanish in every configuration with no presence. main.ts builds it beside the
others so its close() has an owner, and main.test.ts reads that file's source to
assert the line exists — because a module built and never injected is inert code with
every line around it executing.
One integration test carries the claim, and it is the only one in the chapter that
cannot be made cheaper: a real api child process, a real gateway, a real Redis.
Five sends over REST and five message.send frames leave the shared send bucket
at 10 and rest at 5. Two separate counters read 5 and 5 and pass every
other test in the suite.
import { spawn, type ChildProcess } from "node:child_process";
import { randomUUID } from "node:crypto";
import { existsSync } from "node:fs";
import { createRequire } from "node:module";
import { dirname, join } from "node:path";
import type { Server } from "node:http";
import type { AddressInfo } from "node:net";
import { fileURLToPath } from "node:url";
import { docsUrl } from "@relay/protocol";
import { createLogger, serve, type Logger } from "@relay/service-kit";
import { Redis } from "ioredis";
import { WebSocket } from "ws";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest";
import { createApiClient } from "./api-client.js";
import { createGatewayLimits, type GatewayLimits } from "./limits.js";
import { attachSessions } from "./session.js";
// The claim no single-process test can make: that the api and the
// gateway increment ONE counter.
//
// Everything else about the limits is provable cheaper. The arithmetic is pure
// and unit-tested; the socket's two refusals are unit-tested with a stub
// counter; the api's headers are covered by its own integration suite. What
// none of those can show is the property the whole design rests on — a socket
// send and a REST send spend the same budget, because the budget lives in Redis
// and neither process can see the other's memory. A test that stubs the store
// would pass with two separate counters.
//
// docker compose up -d --wait postgres redis
// pnpm build
// RELAY_POSTGRES_PORT=… RELAY_REDIS_PORT=… \
// pnpm --filter @relay/gateway test:integration
//
// The api runs as a CHILD PROCESS for the reason session.itest.ts gives: the
// gateway is not allowed to know how the api is built (ADR-05), and importing
// it would make that dependency real in order to deny it.
const silent: Logger = createLogger("gateway", () => {});
const REDIS_URL = `redis://localhost:${process.env.RELAY_REDIS_PORT ?? "6379"}`;
const HERE = dirname(fileURLToPath(import.meta.url));
const REPO = join(HERE, "..", "..", "..");
const require_ = createRequire(import.meta.url);
interface Seeded {
environmentId: string;
credential: string;
channelId: string;
}
interface ApiUnderTest extends Seeded {
url: string;
/** Configure this environment's policy. Plain SQL through the api's pool: the
* columns are three nullable integers and there is no admin API for them yet,
* so inventing one for a test would be inventing product. */
setLimits: (limits: {
rest?: number | null;
send?: number | null;
connect?: number | null;
}) => Promise<void>;
stop: () => void;
}
async function waitForHealth(url: string): Promise<void> {
const deadline = Date.now() + 30_000;
for (;;) {
try {
if ((await fetch(url)).ok) return;
} catch {
// not up yet
}
if (Date.now() > deadline) throw new Error("api never became healthy");
await new Promise((resolve) => setTimeout(resolve, 100));
}
}
async function startApi(): Promise<ApiUnderTest> {
const dist = join(REPO, "services", "api", "dist");
if (!existsSync(join(dist, "main.js"))) {
throw new Error(
"the api is not built — run `pnpm build` before this lane " +
"(the suite talks to the real service, not a stub)",
);
}
const client = require_(join(dist, "db", "client.js")) as {
createDb: (pool: unknown) => unknown;
createPool: () => Pool;
};
const seeder = require_(join(dist, "db", "repository.js")) as {
createEnvironment: (
db: unknown,
input: { name: string },
) => Promise<{ id: string }>;
createApiKey: (
db: unknown,
input: { environmentId: string },
) => Promise<{ credential: string }>;
Repository: new (
db: unknown,
environmentId: string,
) => {
createUser: (externalId: string, name?: string) => Promise<{ id: string }>;
createChannel: (
externalId: string,
type: string,
) => Promise<{ id: string }>;
addMember: (channelId: string, userId: string) => Promise<boolean>;
upsertUser: (
externalId: string,
profile: { display_name?: string; kind?: string; description?: string },
) => Promise<{ user: { id: string } }>;
};
};
const pool = client.createPool();
const db = client.createDb(pool);
const environment = await seeder.createEnvironment(db, {
name: `limits-itest-${randomUUID().slice(0, 8)}`,
});
const repo = new seeder.Repository(db, environment.id);
const user = await repo.createUser("tuan", "Tuan");
const channel = await repo.createChannel("fleet", "public");
await repo.addMember(channel.id, user.id);
// AND A BOT FOR THE REST HALF. The socket half sends as `tuan` through a user
// token; the REST half presents an APPLICATION credential, which carries no user
// of its own — so it must name a sender and that sender must be a BOT. A person
// named by a key is 403, an unknown name 400, and neither is a 429.
//
// Seeded here rather than per send, because this file's whole subject is how many
// requests fit under one limit: an extra call per send would move every number in
// it and leave the assertions passing for the wrong reason. No channel membership
// — measured, not assumed: the suite is green without it.
await repo.upsertUser("meter", {
display_name: "Meter",
kind: "bot",
description: "spends this suite's REST budget",
});
const key = await seeder.createApiKey(db, { environmentId: environment.id });
// `PORT: "0"` AND THE PORT READ OFF THE CHILD'S OWN LOG LINE, which is what the
// three sibling suites in this directory already do. Published's version of this
// file hand-allocated `RELAY_LIMITS_ITEST_API_PORT ?? 4124`, and a hand-maintained
// port table is the failure that cannot be checked: 4124 sits inside the band this
// chapter would have claimed, and NATS runs on 4222 inside the same band. The
// failure is silent in both directions — the child cannot bind, and the health
// check gets its answer from whatever DOES hold the port.
const child: ChildProcess = spawn("node", [join(dist, "main.js")], {
env: {
...process.env,
PORT: "0",
RELAY_OUTBOX_RELAY: "off",
// Nor the notification relay: this suite counts requests, and a background
// loop sending mail is one more thing that can make a count wander.
RELAY_NOTIFICATION_RELAY: "off",
RELAY_REDIS_URL: REDIS_URL,
},
stdio: ["ignore", "pipe", "pipe"],
});
const port = await new Promise<number>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error("api never reported a port")), 30_000);
let buffered = "";
child.stdout?.on("data", (chunk: Buffer) => {
buffered += chunk.toString();
for (const line of buffered.split("\n")) {
if (!line.trim()) continue;
try {
const parsed = JSON.parse(line) as { msg?: string; port?: number };
if (parsed.msg === "listening" && typeof parsed.port === "number") {
clearTimeout(timer);
resolve(parsed.port);
return;
}
} catch {
/* a partial line; the next chunk completes it */
}
}
});
child.on("exit", (code) => {
clearTimeout(timer);
reject(new Error(`api exited before listening (code ${String(code)})`));
});
});
const url = `http://127.0.0.1:${port}`;
await waitForHealth(`${url}/healthz`);
return {
url,
environmentId: environment.id,
credential: key.credential,
channelId: channel.id,
setLimits: async ({ rest = null, send = null, connect = null }) => {
await pool.query(
"UPDATE environments SET rest_limit_per_minute = $2, " +
"send_limit_per_minute = $3, connect_limit_per_minute = $4 " +
"WHERE id = $1",
[environment.id, rest, send, connect],
);
},
stop: () => {
child.kill();
void pool.end();
},
};
}
interface Pool {
query: (text: string, values?: unknown[]) => Promise<unknown>;
end: () => Promise<void>;
}
function firstFrame(socket: WebSocket, type: string): Promise<unknown> {
return new Promise((resolve, reject) => {
socket.on("message", (raw) => {
const frame = JSON.parse(raw.toString()) as { type: string };
if (frame.type === type) resolve(frame);
});
socket.on("close", (code) => reject(new Error(`closed ${code}`)));
setTimeout(() => reject(new Error(`no ${type} within 5s`)), 5_000);
});
}
describe("one counter, two services", () => {
let api: ApiUnderTest;
let server: Server;
let limits: GatewayLimits;
let redis: Redis;
let url: string;
const sockets: WebSocket[] = [];
/** The key the api and the gateway are both supposed to be incrementing.
* Spelled out here rather than imported, because a test that computed it with
* the code under test would agree with a wrong answer. */
const key = (operation: string, atMs: number = Date.now()) =>
`rl:${api.environmentId}:${operation}:` +
`${Math.floor(atMs / 60_000) * 60_000}`;
/** THE WINDOW IS WALL-CLOCK, SO A TEST CAN SPAN TWO OF THEM.
*
* `limits.ts:115` keys on `Math.floor(now / 60_000) * 60_000`. The bucket rolls at the
* top of every minute whatever the suite is doing, so ten sends either side of the
* boundary write two keys — and reading one of them reports `7` where the test means
* `10`, which looks exactly like a limiter dropping increments.
*
* Measured, twice: run 1 of feature 043's second battery and run 2 of its third.
* Nothing in either failure named a clock.
*
* SUMMING RATHER THAN SLEEPING, and the first attempt did sleep. Waiting for the next
* boundary made the test correct and blew its 5-second timeout the moment the guard
* actually fired — a fix whose failure mode is worse than the fault, because a timeout
* says nothing about what it was waiting for. This asserts the same number without
* spending any time: ten sends are ten sends however the minute falls across them. */
const windowsSince = (sinceMs: number): number[] => {
const first = Math.floor(sinceMs / 60_000) * 60_000;
const last = Math.floor(Date.now() / 60_000) * 60_000;
const out: number[] = [];
for (let w = first; w <= last; w += 60_000) out.push(w);
return out;
};
let testStartedAt = Date.now();
const count = async (operation: string): Promise<number> => {
let total = 0;
for (const w of windowsSince(testStartedAt)) {
total += Number((await redis.get(key(operation, w))) ?? 0);
}
return total;
};
const mintToken = async (user = "tuan") => {
const res = await fetch(`${api.url}/auth/dev-token`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${api.credential}`,
},
body: JSON.stringify({ user, ttl_seconds: 3600 }),
});
if (!res.ok) throw new Error(`dev-token: ${res.status}`);
return ((await res.json()) as { token: string }).token;
};
const connect = async (token: string) => {
const socket = new WebSocket(`${url}/v1/ws?token=${token}`);
sockets.push(socket);
await firstFrame(socket, "connection.ack");
return socket;
};
const frameSend = async (socket: WebSocket, text: string) => {
socket.send(
JSON.stringify({
type: "message.send",
payload: { channel: api.channelId, text, idem_key: randomUUID() },
}),
);
};
const restSend = (text: string) =>
fetch(`${api.url}/v1/channels/${api.channelId}/messages`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${api.credential}`,
"idempotency-key": randomUUID(),
},
body: JSON.stringify({ text, user: "meter" }),
});
beforeAll(async () => {
api = await startApi();
limits = createGatewayLimits(REDIS_URL);
redis = new Redis(REDIS_URL);
server = serve({
service: "gateway",
health: () => ({}),
logger: silent,
// REQUIRED IN THIS TREE. The error-registry chapter made `serve` take the
// not-found link from `docsUrl` rather than spelling it, and it is upstream of
// this chapter here where it was downstream in the published order.
notFoundDocsUrl: docsUrl("not_found"),
});
attachSessions({
server,
api: createApiClient(api.url),
logger: silent,
limits,
});
await new Promise<void>((resolve) => server.listen(0, resolve));
url = `ws://127.0.0.1:${(server.address() as AddressInfo).port}`;
}, 60_000);
beforeEach(() => {
// Where `count()` starts summing, and where `afterEach` starts deleting.
testStartedAt = Date.now();
});
beforeEach(() => {
// Where `count()` starts summing, and where `afterEach` starts deleting.
testStartedAt = Date.now();
});
afterEach(async () => {
for (const socket of sockets.splice(0)) socket.close();
// Every test starts from an empty bucket and the documented defaults.
// Otherwise the first test's traffic is the second test's head start, and
// the window is a minute long.
await redis.del(key("rest"), key("send"), key("connect"));
await api.setLimits({});
});
afterAll(async () => {
server.close();
await limits.close();
redis.disconnect();
api.stop();
});
it("counts a socket send ONCE, against the api's own key (FR-RTL-01)", async () => {
// Not twice — the gateway's internal call to `/v1/... ` is exempt BY ROUTE,
// so the frame is counted by the gateway and not again when its HTTP hop
// lands. And not zero times, which is what an exemption applied one layer
// too broadly would produce.
//
// FR-RTL-01 is the clause: the limit is PER TENANT, and the gateway's hop is
// the platform's own traffic on the tenant's behalf. (An id sweep put
// `FR-WHK-05` here — *"webhook delivery shall be asynchronous"* — which is a
// real clause about something else entirely. A mechanical substitution that
// has no id to map to picks the nearest one it does have.)
const socket = await connect(await mintToken());
const before = await count("send");
await frameSend(socket, "one");
await firstFrame(socket, "message.ack");
expect(await count("send")).toBe(before + 1);
});
it("spends ONE budget across both transports (FR-RTL-01, research R11)", async () => {
// Five over REST and five over the socket. If the two services were
// counting separately this would read 5 and 5.
const socket = await connect(await mintToken());
for (let i = 0; i < 5; i += 1) {
const res = await restSend(`rest-${i}`);
expect(res.status).toBe(201);
}
for (let i = 0; i < 5; i += 1) {
await frameSend(socket, `frame-${i}`);
await firstFrame(socket, "message.ack");
}
expect(await count("send")).toBe(10);
// The REQUEST budget saw only the five REST calls: a frame is not an HTTP
// request, and the gateway's hop to the api does not count as one either.
// THE ASYMMETRY IS THE WHOLE TEST, and only two transports can show it: on one
// transport alone the two counters move together, and a limiter counting
// requests is indistinguishable from one counting messages (FR-RTL-01).
expect(await count("rest")).toBe(5);
});
it("reports the budget with FEWER remaining, not the one that was asked about", async () => {
// Two budgets, one set of headers. A client that saw only the request
// budget would be told it had 595 left while the send budget it is actually
// spending had 590 — an answer that is true and useless.
await api.setLimits({ rest: 100, send: 20 });
const socket = await connect(await mintToken());
for (let i = 0; i < 5; i += 1) {
await frameSend(socket, `frame-${i}`);
await firstFrame(socket, "message.ack");
}
const res = await restSend("rest-after-frames");
expect(res.status).toBe(201);
// send: 5 frames + this one = 6 of 20, so 14 left. rest: 1 of 100, 99 left.
expect(res.headers.get("x-ratelimit-limit")).toBe("20");
expect(res.headers.get("x-ratelimit-remaining")).toBe("14");
});
it("names the limit that was reached when it refuses (FR-RTL-01)", async () => {
await api.setLimits({ rest: 100, send: 2 });
await restSend("one");
await restSend("two");
const refused = await restSend("three");
expect(refused.status).toBe(429);
const body = (await refused.json()) as { code: string; message: string };
expect(body.code).toBe("rate_limited");
// "you are rate limited" leaves a developer guessing which dial to turn.
// The wording names the UNIT rather than the column: "too many messages"
// says slow down, "too many requests" says batch. This traffic was three
// sends against a send limit of two, so it has to be the first.
expect(body.message).toMatch(/too many messages/);
expect(body.message).not.toMatch(/too many requests/);
});
it("lets the gateway through an environment that is at its REST limit (FR-RTL-01)", async () => {
// The exemption cannot key off the principal: the gateway forwards the END
// USER's token, so its session and send calls resolve to `kind: "user"`
// exactly like customer traffic. A rule that exempted only the platform
// credential would refuse them — and the socket would go down for a REST
// budget it never spends (research R17).
await api.setLimits({ rest: 1 });
expect((await restSend("the one allowed request")).status).toBe(201);
expect((await restSend("over")).status).toBe(429);
// The environment is over its REST limit. The socket still opens…
const socket = await connect(await mintToken());
// …and still sends, because a frame spends the SEND budget and the
// gateway's hop to the api spends nothing.
await frameSend(socket, "through the closed door");
expect(await firstFrame(socket, "message.ack")).toMatchObject({
type: "message.ack",
});
});
});The api child takes PORT=0 and the suite reads the port off the child's own
listening line. A hand-allocated port cannot be checked: the number this chapter
would have claimed sits in the same band as the broker the lane runs, and the failure
is silent in both directions — the child cannot bind, and the health check gets its
answer from whatever does hold the port.
The ninth client also has to be passed in, and one line of main.ts is the whole
of that:
@@ -139,29 +139,57 @@ describe("every fabric createServer builds is injected", () => {
// a property of the text — every module built above the call appears inside it.
const SOURCE = readFileSync(
join(import.meta.dirname, "main.ts"),
"utf8",
);
- /** `const x = createY({` — the fabrics, derived rather than listed, so a fifth
- * arrives here without anyone remembering. */
+ /** `const x = createY(` — the fabrics, derived rather than listed, so a sixth
+ * arrives here without anyone remembering.
+ *
+ * TWO CHANGES, AND THE FIRST ONE MISSED A MODULE. The pattern required an OBJECT
+ * ARGUMENT — `create[A-Z]\w*\(\{` — and the counter store takes none:
+ * `createGatewayLimits()` reads its url from the environment. So the ninth Redis
+ * client in this file was invisible to the check written to make exactly that
+ * impossible, and it was invisible in the direction that passes. Measured: the old
+ * pattern derives five names, the new one six.
+ *
+ * SCOPED TO `createServer`'S BODY, which is what this describe's title always
+ * claimed. Dropping the `{` widens the match to `createLogger` and `createServer`
+ * in the `import.meta.main` block below the function — neither a fabric, both
+ * `const x = createY(`. Slicing to the text between the function and the call it
+ * has to appear in is the honest boundary; an exclusion list would be the thing
+ * this file exists instead of. */
function built(): string[] {
- return [...SOURCE.matchAll(/\bconst (\w+) = create[A-Z]\w*\(\{/g)].map((m) => m[1]!);
+ const start = SOURCE.indexOf("export function createServer");
+ const end = SOURCE.indexOf("attachSessions({", start);
+ return [...SOURCE.slice(start, end).matchAll(/\bconst (\w+) = create[A-Z]\w*\(/g)]
+ .map((m) => m[1]!);
}
/** The object literal `attachSessions` is called with. */
function injected(): string {
const open = SOURCE.indexOf("attachSessions({");
const close = SOURCE.indexOf("\n });", open);
return SOURCE.slice(open, close);
}
it("derives the fabrics and the call, and finds both", () => {
// THE POSITIVE CONTROL. Every assertion below is about which names are missing,
// and a derivation that found nothing satisfies all of them.
- expect(built().length, "no `const x = createY({` found in main.ts").toBeGreaterThan(1);
+ expect(built().length, "no `const x = createY(` found in main.ts").toBeGreaterThan(1);
+ // AND THE COUNT, because "more than one" was satisfied by a pattern that found
+ // five of six. A number here goes red when a fabric is added without a thought
+ // about this file, which is the moment to have it.
+ expect(built(), "the fabrics createServer builds").toEqual([
+ "fanout",
+ "presence",
+ "membership",
+ "typing",
+ "connections",
+ "limits",
+ ]);
expect(injected(), "no attachSessions call found").toContain("server,");
});
it("passes each one into attachSessions", () => {
const call = injected();
const missing = built().filter(4008 reads "quota exhausted". There is no quota yet.
Reaching for it because it was declared would collapse the distinction this chapter is built on. A rate limit says slow down, come back in forty seconds, nothing is wrong. A quota says you have used what you bought. They fail in different directions, and one belongs in Redis while the other cannot.
So there is a test asserting nothing in the gateway sends it:
for (const text of source) {
expect(text).not.toMatch(/close\(\s*400[89]/);
}
expect(source.join("")).toMatch(/close\(\s*400[12]/);The second line is the important one. A claim about absence cannot be demonstrated by any input — there is no frame you can send to observe a code that is never sent — so the check reads the source. But a regex aimed at a code nobody sends passes whether or not the regex is correct. Running the same pattern against the codes the gateway does send is what makes the negative assertion worth anything.
request_id is now on every error the platform emits — the REST envelope, the
socket's error frame, the framework's error filter. Not just the 429.
It broke twenty-three tenant-isolation tests in two files, and they were all right to break. Each compared two error bodies for equality:
expect(await foreign.json()).toEqual(await missing.json());The property is real: a credential for the wrong environment and one for a nonexistent resource must be indistinguishable, or the error becomes an oracle that enumerates what exists. A per-request id makes the bodies differ in a field carrying no information about either, so the comparison strips it and compares the rest — a more precise statement of what those tests always meant.
Twenty-two of the twenty-three are the isolation gauntlet, and they all failed
through one function: comparePair in isolation/attack.ts, which JSON.stringifys
both answers. One line, one field, twenty-two attacks green again.
@@ -9,17 +9,32 @@
* freeze today's status choices into a test, so a considered change to a status breaks
* a security suite for no security reason. It would say nothing about the body. And it
* would PASS AN ENDPOINT THAT LEAKS THROUGH ITS PROSE — an error message that echoes
* the identifier back makes the foreign answer differ from the absent one while both
* are 404, which is exactly the leak constitution I is about.
*
- * So status and whole body are compared. There is nothing to exclude from the
- * comparison yet: the error envelope is `code`, `message` and `docs_url`, all three of
- * which must match. When a per-request field joins it, the chapter that adds it owns
- * the decision to drop it here — and it will have to argue that the field reveals
- * nothing about the resource. */
+ * So status and body are compared — and ONE FIELD IS EXCLUDED, which the paragraph
+ * this replaces asked the next chapter to argue for. It said: *"There is nothing to
+ * exclude from the comparison yet … when a per-request field joins it, the chapter
+ * that adds it owns the decision to drop it here — and it will have to argue that the
+ * field reveals nothing about the resource."*
+ *
+ * THE ARGUMENT. `request_id` is minted per request, so two requests never carry the
+ * same one and a whole-body comparison of any two answers fails for a reason that has
+ * nothing to do with tenancy. It is also the one field in the envelope that is not
+ * derived from the resource at all: `code`, `message` and `docs_url` are answers about
+ * what was asked for, and the id is an answer about the asking. Dropping it removes no
+ * leak, because there is nothing in it to leak.
+ *
+ * AND THE INSTRUCTION IS WHY THIS WAS ONE LINE RATHER THAN A DEBUGGING SESSION. The
+ * field's arrival turned twenty-two attacks and one channel-surface test red at once,
+ * every one of them with two identical bodies and two different ids. The file that
+ * broke had already written down what to do about it, in the place the reader of the
+ * failure would land. */
+
+import { withoutRequestId } from "./compare";
export interface AttackRequest {
method: string;
/** Path with identifiers already substituted — `/v1/channels/<uuid>/messages`. */
path: string;
body?: unknown;
@@ -73,14 +88,17 @@ export async function send(
* `classifyRow`: an instrument that has never fired is an untested instrument. */
export function comparePair(foreign: Answer, absent: Answer): string[] {
const differences: string[] = [];
if (foreign.status !== absent.status) {
differences.push(`status ${foreign.status} (foreign) vs ${absent.status} (absent)`);
}
- const f = JSON.stringify(foreign.body);
- const a = JSON.stringify(absent.body);
+ const f = JSON.stringify(withoutRequestId(foreign.body));
+ const a = JSON.stringify(withoutRequestId(absent.body));
+ // REPORTED WITHOUT THE ID TOO, not merely compared without it. A message that
+ // printed the raw bodies would show two ids differing on every real failure and
+ // send the reader after the field this function has just decided to ignore.
if (f !== a) differences.push(`body ${f} (foreign) vs ${a} (absent)`);
return differences;
}
/** A read of another tenant's resource must answer as a read of nothing. */
/** A list's correct answer to "nothing of yours here" is an EMPTY RESULT, and thatThe one channel-surface test that does not go through comparePair gets the same
treatment, and its comment gets the same correction:
@@ -197,17 +197,19 @@ describe("the public channel surface", () => {
it("answers a foreign channel id exactly as an absent one (FR-018)", async () => {
const absent = "00000000-0000-4000-8000-000000000000";
const foreign = await addMembers(foreignChannelId, { user_ids: ["intruder"] });
const nowhere = await addMembers(absent, { user_ids: ["intruder"] });
expect(foreign.status).toBe(nowhere.status);
- // COMPARED WHOLE. The envelope is `code`, `message` and `docs_url`, and all
- // three must match for a foreign channel to be indistinguishable from an absent
- // one. Nothing here is per-request yet, so nothing is excluded.
- expect(await foreign.json()).toEqual(
- await nowhere.json(),
+ // COMPARED WHOLE EXCEPT THE ID, and this line asked for that in advance: it read
+ // *"nothing here is per-request yet, so nothing is excluded"* until the limits
+ // chapter put `request_id` in the envelope. `code`, `message` and `docs_url` are
+ // answers about the resource and all three must match; the id is an answer about
+ // the request and never matches.
+ expect(withoutRequestId(await foreign.json())).toEqual(
+ withoutRequestId(await nowhere.json()),
);
// And the other tenant's channel gained nobody. Read through ITS OWN
// repository — a repository scoped to the empty string is not a scope, it is
// a query that fails on an invalid uuid, which is how this line read first.
expect(await foreignRepo.listMembers(foreignChannelId)).toEqual([]);
});FR-014 asks something the code already did and nothing could test. A typing signal is not a send and must not spend a send's budget — a client holding a key down would otherwise empty a customer's message allowance with an indicator.
session.ts has returned from the typing.send branch before reaching anything
that could spend since the typing chapter. So the property was true by
construction and untestable: there was nothing below the early return to reach.
Adding the limiter underneath it is what makes the early return matter, and the
test that says it still comes first is what stands between an indicator and a
budget.
@@ -20,12 +20,13 @@ import { createLogger, serve, type Logger } from "@relay/service-kit";
import { Redis } from "ioredis";
import { afterEach, describe, expect, it } from "vitest";
import { WebSocket } from "ws";
import type { ApiClient } from "./api-client.js";
import { createFanout } from "./fanout.js";
+import type { Decision, GatewayLimits } from "./limits.js";
import { createMembership, type Membership } from "./membership.js";
import { createPresence } from "./presence.js";
import { attachSessions } from "./session.js";
import { createTyping, type Typing } from "./typing.js";
// The typing chapter's fabric, against a REAL Redis.
@@ -85,12 +86,16 @@ async function boot(options: {
/** Points this instance's fabric at a proxy instead of Redis, so a test can
* sever the connection without touching anything shared. */
redisUrl?: string;
/** T072 only: the other three fabrics, so one watcher can receive all four
* kinds over the same channel. */
allFabrics?: boolean;
+ /** T048b only: a RECORDING double for the send limiter, so a test can assert
+ * which operations the session layer asked it to spend. Not wired by default —
+ * every other test in this file is about delivery. */
+ limits?: GatewayLimits;
}): Promise<Instance> {
const environment = options.environment ?? "env-1";
const logger =
options.lines === undefined
? silent
: createLogger("gateway", (line: string) => {
@@ -107,12 +112,16 @@ async function boot(options: {
session: async () => ({
environment_id: environment,
user: options.user,
banned: false,
channel_ids: options.channels,
revisions: {},
+ // The limits ride this response as of the limits chapter, and this fixture is
+ // generous on purpose: T048b below asserts that a typing signal spends NO send
+ // budget, and a tight number here would make that pass for the wrong reason.
+ limits: { connect: 3_000, send: 600 },
}),
memberships: async () => options.channels,
backfill: async () => {
if (options.backfillDelayMs !== undefined) {
await new Promise((r) => setTimeout(r, options.backfillDelayMs));
}
@@ -137,12 +146,13 @@ async function boot(options: {
...(fanout === undefined ? {} : { fanout }),
...(presence === undefined ? {} : { presence }),
...(membership === undefined ? {} : { membership }),
...(options.renewalIntervalMs === undefined
? {}
: { renewalIntervalMs: options.renewalIntervalMs }),
+ ...(options.limits === undefined ? {} : { limits: options.limits }),
});
await new Promise<void>((resolve) => server.listen(0, resolve));
const { port } = server.address() as AddressInfo;
return {
url: `ws://127.0.0.1:${port}/v1/ws`,
typing,
@@ -627,12 +637,65 @@ describe("a typing signal on its way out", () => {
expect(frames).toHaveLength(afterSignal);
// And the fabric is still alive, so the silence above was a decision.
signaller.send(JSON.stringify({ type: "typing.send", payload: { channel } }));
expect(await untilTyping(frames, channel, 2)).toHaveLength(2);
}, 15_000);
+
+ /** T048b. A TYPING SIGNAL SPENDS NO MESSAGE QUOTA (FR-014).
+ *
+ * **Moved out of US3 by analysis pass 1**, and the reason is worth keeping:
+ * leaving it in a P2 story meant stopping after the MVP could ship a cosmetic
+ * feature able to exhaust a customer's message budget. A client holding a key
+ * down is not sending messages, and a limiter that thought otherwise would let
+ * an indicator empty a budget the customer pays for.
+ *
+ * **ASSERTED ON THE LIMITER, NOT ON A COUNTER.** The requirement is that the
+ * typing branch never REACHES `limits.spend` — it returns above it — and a
+ * recording double says exactly that. A counter read out of Redis would also
+ * pass against a branch that spent and then refunded, which satisfies "the
+ * count did not move" and not "the call was never made".
+ *
+ * **AND IN THIS ORDER THE PROPERTY WAS TRUE BEFORE IT WAS TESTABLE.**
+ * `session.ts` has returned from `typing.send` above everything for nine
+ * chapters; until the limiter arrived underneath it there was nothing below to
+ * reach. So this test does not merely arrive late — it is the first moment the
+ * early return can be said to matter, and it is what stands between a typing
+ * indicator and a customer's message budget from here on. */
+ it("never reaches the send limiter, however many signals arrive", async () => {
+ const channel = randomUUID();
+ const spends: string[] = [];
+ const limits: GatewayLimits = {
+ spend: async (_environmentId, operation): Promise<Decision> => {
+ spends.push(operation);
+ return {
+ over: false,
+ limit: 600,
+ remaining: 599,
+ resetSeconds: Math.floor(Date.now() / 1000) + 60,
+ retryAfterSeconds: 1,
+ };
+ },
+ close: async () => {},
+ };
+ const instance = await boot({ user: "tuan", channels: [channel], limits });
+ open.push(instance.close);
+
+ const socket = connect(instance);
+ await acked(socket);
+ // The handshake spends `connect`, and recording it here is what makes the
+ // assertion below a claim about `send` rather than about an unwired double.
+ expect(spends).toEqual(["connect"]);
+
+ for (let i = 0; i < 5; i += 1) {
+ socket.send(JSON.stringify({ type: "typing.send", payload: { channel } }));
+ }
+ await settle();
+
+ expect(spends.filter((op) => op === "send")).toEqual([]);
+ });
/** T048c. THE MID-CONNECTION JOIN (FR-004a).
*
* **The obvious test — a member who was in the channel at connect — passes
* against an implementation that never touches the membership-revocation chapter's `added`
* branch.** So this one connects first, joins second, and signals third.
*The journey map's Test phase has the developer driving a rate limit on purpose, to see what her client library does with a 429. She is the one user in the map who reaches 600 a minute deliberately.
Which is why FR-RTL-04 says a development environment's limits are meant to be raised. The columns are per environment, so raising one for load testing does not move production's ceiling — and a shared policy would mean exactly that.
The chapter captures its transcripts rather than describing them. The first capture of a 429 printed this:
HTTP 429
x-ratelimit-limit: 3
retry-after: 22
{"code":"rate_limited","message":"too many messages for this environment; …"}
Limit: 3 above "too many messages", on an environment whose send limit is 2.
Both budgets reach zero remaining on that request while only one is over: with
rest at 3 and send at 2, the third send leaves rest at 3 of 3 — spent, not
over — and send at 3 of 2, which is. The headers describe whichever has fewest
remaining, the tie goes to the first, and the first is rest. So the body named
the budget that refused and the headers named the other one. A client reads
Limit: 3, paces itself at three a minute, and is refused at two.
The fix is one line: a refusal describes the budget that refused, and "fewest remaining" governs only responses being served. Eighteen integration tests covered that middleware and none caught it, because each asserted one field and nobody had looked at a whole response.
Corrected:
$ POST /v1/channels/{id}/messages # the first send
HTTP 201
x-ratelimit-limit: 2
x-ratelimit-remaining: 1
x-ratelimit-reset: 1787197740
$ POST /v1/channels/{id}/messages # the second
HTTP 201
x-ratelimit-limit: 2
x-ratelimit-remaining: 0
x-ratelimit-reset: 1787197740
$ POST /v1/channels/{id}/messages # over the send limit
HTTP 429
x-ratelimit-limit: 2
x-ratelimit-remaining: 0
x-ratelimit-reset: 1787197740
retry-after: 51
{
"code": "rate_limited",
"message": "too many messages for this environment; retry after 51 seconds",
"docs_url": "https://relay.example/docs/errors/rate_limited",
"request_id": "67219aad-436e-434c-8da2-a6a8c9a16754"
}
x-ratelimit-reset is the same value on all three, which is the fixed window
being a fixed window. And the body has four fields, which it has not had before
this chapter.
Three things, each of which a chapter is obliged to say out loud.
EIR-API-04 documented an error body nested under an error key:
{ "error": { "code": "…", "message": "…", "docs_url": "…" } }The platform has never emitted that. Every error since chapter 1.3 has been flat, constitution V's envelope is flat, and the fence chain has been replaying flat error bodies into published chapters ever since.
Two documents disagreed and one was wrong. Wrapping every error response to match
the SRS is a breaking change to a public contract — which CON-05 makes a
versioning event — for a shape nobody has ever received. So the document was
brought to the code: docs/04-srs.md revision 1.3 records five top-level
fields and unwraps the example. A chapter that changes a source requirement says
so, which is the discipline the fence chain enforces for code applied to the
documents the code is built from.
§7.3 lists Phase 2 as FR-TEN, FR-AUT, FR-WHK and FR-RTL at P2, and FR-RTL-01…04 is the last of the four. The phase's exit criterion is a different thing and belongs to the isolation gauntlet: "an external developer integrates using only public documentation, with no assistance." Requirements complete here; the phase exits when somebody outside this repository can use what they describe.
docs_url is a placeholder in one place now, and that place still costs somethingEvery error carries a docs_url pointing at
https://relay.example/docs/errors/{code}, which does not resolve.
HALF OF THIS DEBT IS ALREADY PAID, AND BY A CHAPTER YOU HAVE READ. The error
registry made the URL a function — docsUrl(code), one line in codes.ts, with the
code as the anchor verbatim — so the host is a placeholder in exactly one place
rather than in every site that builds an error. Its own comment says what that
bought: "the host stays a placeholder until the docs site exists. What stops being
a placeholder is the NUMBER OF PLACES that have to change when it does: one."
THE OTHER HALF IS THIS CHAPTER'S PROBLEM AND IT IS THE HARDER ONE. One edit site
does not make a page appear. The link has been harmless since chapter 1.3, because
the codes it named were ones a developer meets while doing something wrong.
rate_limited is different: it is the first error a working integration receives
routinely, at the moment its author wants to look something up.
The timing is pointed. This chapter closes the requirement set for a phase that exits on integration from public documentation alone, while shipping the error code most likely to send someone looking for documentation that is not there. Constitution V requires every code to have a reachable page; none does. A docs site is not this chapter's to build, but a URL implying otherwise is worse than an absent one.
AND THE SHAPE IS ALREADY WRONG, WHICH ONLY A READER CAN NOTICE. docsUrl builds
a PATH per code — /docs/errors/rate_limited — while the reference that exists,
docs/08-error-reference.md, is one document with a heading per code. Even given a
host, the path form resolves to a page per code that nothing writes; the reference
wants a fragment. The typecheck cannot see it, the registry check cannot see it — it
compares names, not link shapes — and no test in this repository asks whether a URL
would resolve if the host did. It is one line to change on the day the site appears,
which is the whole argument for having centralised it. It is also a dead link with a
function's worth of confidence behind it until then.
Everything above, as the repository holds it.
Four files with no framework in them, and one migration. bucket.ts and the
migration are above; these are the rest.
// The limit policy: what each environment is allowed, and what each number rests
// on (research R26).
//
// R4 chose all four of these by judgement and checked none of them against a
// document stating this platform's scale. The fourteenth analysis pass read the
// SRS's NFR tables and found that one of them made a P1 requirement unreachable,
// so all four were re-derived rather than the broken one patched.
//
// WHAT WENT WRONG IS WORTH KEEPING. The connect limit was 60/min, on the
// reasoning that "sixty establishments a minute per environment is a client
// reconnecting hard, not a client working". True of a client. The limit is per
// ENVIRONMENT, and an environment is a tenant — NFR-SCL-01 puts ten thousand
// concurrent connections on one gateway instance and FR-RTM-09 allows five per
// user, so filling one instance from cold would have taken 167 minutes.
//
// Each number below names what it rests on, including the one that rests on
// nothing. That is the actual fix; the new value is a consequence of it.
/** Per environment, per minute. Overridable per environment (FR-RTL-04);
* these apply when a column is null, which means "no override" and never zero —
* refuse-everything has to stay expressible. */
export const DEFAULT_LIMITS = {
/** NO ANCHOR, and recorded as such. No SRS requirement caps a tenant's request
* rate; NFR-PRF-02's p95 under 150 ms is a latency target, not a throughput
* bound. Matched to the send limit because a REST send consumes both budgets
* (FR-RTL-01), and two different ceilings on one operation would mean a client
* hitting one while the other says it has room. */
rest: 600,
/** 1% of NFR-SCL-03's stated 1,000 messages per second aggregate — 60,000 a
* minute across the platform. So a hundred environments at their ceiling
* saturate it, and the hundred-and-first is what this protects. */
send: 600,
/** NFR-SCL-01's ten thousand connections per gateway instance, divided by
* FR-RTM-09's five per user, re-established inside one window so a deploy stays
* one reconnection cycle (NFR-REL-03).
*
* IT IS STILL A LIMIT. Its job is to stop a client reconnecting in a tight
* loop, which does thousands a minute and is refused well before a legitimate
* fleet is. It is not there to shape a tenant's capacity, and the old number
* had those two jobs confused. */
connect: 3_000,
} as const;
export type LimitedOperation = keyof typeof DEFAULT_LIMITS;
/** Failed authentications per source address per minute.
*
* NOT a per-environment column: the caller has not proved which environment they
* are, which is the point of the limiter. Configuration, and configuration the
* lane has to be able to raise — the api's own integration suites assert `401`
* twenty-six times from one loopback address inside about 110 seconds, so a
* threshold nothing could lift would refuse this project's own tests
* (research R15).
*
* THE DEFAULT ENFORCES. THE RETRY-AND-DISABLE CHAPTER'S `RELAY_DISABLE_SWEEP` states the rule: a
* flag whose default disabled a requirement would be a requirement nobody had
* built.
*
* THE COST IT CARRIES, which R4 did not name: shared egress. An office behind one
* NAT is one source address, so ten failed logins a minute is a whole building's
* budget — and the refusal is deliberately indistinguishable from a wrong
* credential (EIR-API-04), so they will experience it as a broken login. Kept anyway;
* the alternative is a threshold high enough to be worthless against the attack
* it exists for. */
export const DEFAULT_AUTH_FAILURES_PER_MINUTE = 10;
export function authFailureThreshold(): number {
const raw = process.env["RELAY_AUTH_FAILURES_PER_MINUTE"];
if (raw === undefined) return DEFAULT_AUTH_FAILURES_PER_MINUTE;
const parsed = Number.parseInt(raw, 10);
return Number.isFinite(parsed) && parsed > 0
? parsed
: DEFAULT_AUTH_FAILURES_PER_MINUTE;
}
/** The window every counter uses. One minute, because every limit above is
* expressed per minute and a second unit would be a second thing to reason
* about. */
export const WINDOW_MS = 60_000;// The in-process counter the AUTH limiter falls back to when Redis is gone
// (research R3).
//
// THE TENANT LIMITER FAILS OPEN AND THIS ONE MUST NOT, and that asymmetry is the
// chapter's whole argument. Both are the same mechanism; what differs is what is
// on the other side of the limit. The tenant limiter protects Relay's capacity
// from a customer's traffic, and over-serving a paying customer for the length of
// a cache outage costs some capacity. This one protects a customer's credentials
// from an attacker, and over-serving an attacker costs the customer their
// account.
//
// So neither of the two obvious answers is right. Failing open is unbounded — a
// hole rather than a degradation. Failing closed converts a Redis restart into an
// authentication outage, which is worse than the attack it prevents for every
// customer who is not being attacked. The third answer is this: count in memory,
// same threshold, and let the guarantee weaken from "N per window across the
// fleet" to "N per window per instance". Three api instances give an attacker
// three times the attempts for the duration of the outage — a small multiple
// rather than infinity.
//
// THE CAP IS PART OF THE DECISION, NOT A DETAIL. A map keyed by
// attacker-controlled source address is a memory-exhaustion vector if it is
// unbounded, and a fallback that closed a brute-force hole by opening a worse one
// would not be worth having.
//
// AND IT STOPS ADMITTING RATHER THAN EVICTING. An eviction policy on this map is
// a policy the attacker drives: fill it, evict the entry that was counting them,
// start again. Refusing new keys degrades to "addresses already being tracked
// stay tracked", which is the safe direction.
interface Entry {
count: number;
windowStart: number;
}
export interface FallbackCounter {
/** Count one failure against a key, returning the new count — or `null` when
* the key could not be admitted because the map is full. A caller that gets
* `null` has learned nothing about that address and must not treat it as
* "under the threshold". */
increment(key: string, nowMs: number): number | null;
/** The current count for a key WITHOUT adding to it, or `null` when the key is
* not tracked and the map is full.
*
* `null` and `0` are different answers and the caller must tell them apart:
* zero means "tracked, nothing counted", null means "we have no idea". While
* degraded, refusing an address we cannot track is the safe direction, and the
* cap makes that a bounded population rather than everybody. */
peek(key: string, nowMs: number): number | null;
/** Live keys. Exposed for the test that proves the bound holds. */
size(): number;
}
export function createFallbackCounter({
windowMs,
maxKeys,
}: {
windowMs: number;
maxKeys: number;
}): FallbackCounter {
const entries = new Map<string, Entry>();
return {
increment(key, nowMs) {
const start = Math.floor(nowMs / windowMs) * windowMs;
const existing = entries.get(key);
if (existing !== undefined) {
if (existing.windowStart === start) {
existing.count += 1;
return existing.count;
}
// Same key, new window: reuse the slot rather than counting against the
// cap twice.
existing.count = 1;
existing.windowStart = start;
return 1;
}
if (entries.size >= maxKeys) {
// Sweep what the current window has already outlived before refusing.
// The cap is on LIVE keys, not on keys ever seen — an outage lasting
// hours must not permanently refuse to count anybody new.
for (const [k, v] of entries) {
if (v.windowStart !== start) entries.delete(k);
}
}
if (entries.size >= maxKeys) return null;
entries.set(key, { count: 1, windowStart: start });
return 1;
},
peek(key, nowMs) {
const start = Math.floor(nowMs / windowMs) * windowMs;
const existing = entries.get(key);
if (existing === undefined) {
return entries.size >= maxKeys ? null : 0;
}
return existing.windowStart === start ? existing.count : 0;
},
size() {
return entries.size;
},
};
}import { Redis } from "ioredis";
import { WINDOW_MS } from "./policy";
// The counter store (research R1).
//
// THE ONLY MODULE IN THE API PERMITTED TO HOLD A REDIS CLIENT, enforced by
// `no-restricted-imports` in `eslint.config.mjs` — the same confinement the
// database driver has, for the same stated reason. The keys are per environment,
// so an unrestricted client would let any handler read or write another tenant's
// counter, and constitution I makes that a correctness property rather than a
// convention.
//
// TWO COMMANDS, NO LUA. `INCR` returns the new value atomically on its own, and
// `EXPIRE` is set only when the increment returns 1 — the first write of a
// window. A token bucket would need read-timestamp-compute-write, which across
// instances needs a script, which is a second language in the request path
// (constitution VII).
//
// The TTL does the cleanup: a key dies when its window ends and nothing
// accumulates. The deduplication chapter's baseline and this chapter's own both found suites
// broken by shared stores that grew without bound, so a counter that tidies
// itself is worth the sentence.
export interface CounterStore {
/** Count one operation against a key, returning the new count — or `null` when
* the store could not be reached.
*
* NULL IS NOT ZERO AND NOT AN ERROR. It means "we are not counting", and each
* caller decides what that is worth: the tenant limiter serves the request
* (SAD §6.3, Redis is not a source of truth), and the auth limiter falls back to
* counting in memory rather than letting an attacker through (FR-AUT-12). Same
* signal, opposite conclusions, which is the chapter's argument in one return
* type. */
increment(key: string, nowMs: number): Promise<number | null>;
/** The current count without adding to it, or `null` when the store could not
* be reached. Asking "is this address over the threshold" must not itself push
* it over — a limiter whose check is also a write refuses on its own
* questions. */
get(key: string): Promise<number | null>;
close(): Promise<void>;
}
export const DEFAULT_REDIS_URL = "redis://localhost:6379";
/** The key. `rl:` is the prefix the SAD's cache-keys table names; the operation and the
* window's start are appended so one `INCR` reaches the right counter and the key
* expires itself.
*
* That EXTENDS the SAD's three-segment `rl:{env}:{bucket}` rather than matching
* it, and the extension is what makes the TTL do the cleanup. */
export function counterKey(
scope: string,
operation: string,
windowStartMs: number,
): string {
return `rl:${scope}:${operation}:${windowStartMs}`;
}
/** The auth counter's key, keyed by source address rather than environment.
*
* A SEPARATE PREFIX, not an `operation` value on the tenant key, because it is
* keyed by something else entirely and because the two have opposite failure
* behaviour. Sharing a prefix would invite sharing a code path, and the whole
* point is that they must not.
*
* THE PREFIX IS OVERRIDABLE, and that is test isolation rather than
* configuration. The integration lane runs files in PARALLEL — only the coverage
* config sets `fileParallelism: false` — so every suite asserting a `401` from
* loopback lands in one bucket. Raising a threshold survives that; a suite that
* needs a LOW threshold needs its own key, or it compares a count filled by other
* workers against a deliberately small number and refuses requests that had
* nothing to do with it (research R21).
*
* The same pattern `attempts.itest.ts` uses for its durable name, and for the
* same reason. */
export function authKey(
address: string,
windowStartMs: number,
prefix: string = process.env["RELAY_AUTH_KEY_PREFIX"] ?? "rlauth",
): string {
return `${prefix}:${address}:${windowStartMs}`;
}
export function createCounterStore(
url: string = process.env["RELAY_REDIS_URL"] ?? DEFAULT_REDIS_URL,
): CounterStore {
// `lazyConnect` so constructing the store never blocks start-up.
//
// THE OFFLINE QUEUE STAYS ON, and the first draft had it off. With it off, the
// very first command is rejected because the lazy connection has not been
// established yet — so the first request an api instance ever serves reports no
// count, degrades, and looks like a Redis outage. The integration suite caught
// it as `expected null to be '599'` on the first test and three passes after
// it.
//
// Failing fast on a store that is genuinely down is then `maxRetriesPerRequest:
// 0` and a short `connectTimeout`: a queued command rejects as soon as the
// connection attempt fails rather than waiting out a retry schedule. A limiter
// that waits is worse than one that does not count, because the request it is
// holding is a customer's.
const redis = new Redis(url, {
lazyConnect: true,
maxRetriesPerRequest: 0,
connectTimeout: 1_000,
});
// FAILING OPEN IS NOT FREE IF IT FAILS SLOWLY, and the first version of this
// file was slow. With the store gone, every command waits out its connect
// timeout before giving up — so each request paid a second or more, twice,
// and the integration test for the degraded path timed out rather than
// asserting anything.
//
// That is worse than it looks. The tenant limiter fails open so a cache outage
// does not refuse paid traffic; an outage that instead adds seconds to every
// request has refused it in a slower way, and NFR-PRF-02 asks for a p95 under
// 150 ms.
//
// So a known-down store is not retried on the request path. The first failure
// opens a window; while it is open every call answers `null` immediately, which
// is the same signal the caller already handles. One probe per window is what
// notices the store coming back.
const DOWN_WINDOW_MS = 5_000;
let downUntil = 0;
const guard = async <T>(op: () => Promise<T>): Promise<T | null> => {
if (Date.now() < downUntil) return null;
try {
const result = await op();
downUntil = 0;
return result;
} catch {
downUntil = Date.now() + DOWN_WINDOW_MS;
return null;
}
};
// A dead store is an expected state here, not an exception. Without a listener
// ioredis emits `error` on an EventEmitter with none attached, which Node turns
// into an unhandled exception and the api dies for the thing it was designed to
// survive.
redis.on("error", () => {});
return {
async increment(key, nowMs) {
void nowMs;
return guard(async () => {
const count = await redis.incr(key);
if (count === 1) {
await redis.pexpire(key, WINDOW_MS);
}
return count;
});
},
async get(key) {
return guard(async () => {
const raw = await redis.get(key);
return raw === null ? 0 : Number.parseInt(raw, 10);
});
},
async close() {
redis.disconnect();
},
};
}import type { RequestWithPrincipal } from "../auth/principal";
// Whose failure was that? (FR-AUT-12, research R14.)
//
// THE API SEES THE GATEWAY, not the client. A WebSocket handshake is
// authenticated by the gateway forwarding the end user's token to
// `/internal/session`, so the TCP peer is the gateway for every customer at
// once. Counting the peer would put every customer's failed handshakes in one
// bucket, and one attacker would exhaust a threshold that then refused
// everybody.
//
// A FIELD ON THE INTERNAL CONTRACT, NOT A HEADER. A header the caller asserts is
// a header the caller can forge — the exact pattern the credentials chapter removed when it
// retired the two identity headers the gateway used to send. This one is
// accepted only from a caller already trusted enough to reach the internal
// routes, and it is trusted for exactly one thing: naming who was on the other
// end. The same request is trusted enough not to be throttled and not trusted to
// be the origin.
/** The field the gateway sets on its internal calls. Read from the parsed body
* rather than a header, so an ordinary customer cannot set it. */
export const CLIENT_ADDRESS_FIELD = "client_address";
export function clientAddress(
req: RequestWithPrincipal & {
socket?: { remoteAddress?: string | undefined };
body?: unknown;
},
): string {
const body = req.body;
if (typeof body === "object" && body !== null) {
const forwarded = (body as Record<string, unknown>)[CLIENT_ADDRESS_FIELD];
if (typeof forwarded === "string" && forwarded.length > 0) {
return forwarded;
}
}
return req.socket?.remoteAddress ?? "unknown";
}import { Inject, Injectable } from "@nestjs/common";
import type { Logger } from "@relay/service-kit";
import { LOGGER } from "../logger";
import { windowStart } from "./bucket";
import { createFallbackCounter } from "./fallback";
import { COUNTER_STORE } from "./limits.module";
import { authFailureThreshold, WINDOW_MS } from "./policy";
import { authKey, type CounterStore } from "./store";
// The limiter that counts FAILED AUTHENTICATIONS by source address
// (FR-AUT-12, research R3).
//
// THE ONE THAT MUST NOT FAIL OPEN, and that is the chapter's whole argument. The
// tenant limiter serves the request when Redis is gone, because Redis is not a
// source of truth and a cache outage is not a reason to refuse paid traffic. Run
// the same reasoning here and it gives the opposite answer: an unlimited window
// on failed logins is not a degradation, it is a hole.
//
// FAILING CLOSED IS NOT THE ANSWER EITHER — it turns a Redis restart into an
// authentication outage, which is worse than the attack it prevents for every
// customer who is not being attacked. So: an in-process fallback at the same
// threshold, weakening the guarantee from N per window across the fleet to N per
// window per instance. A small multiple rather than infinity.
//
// WHOSE ADDRESS. The client's, not the caller's. A handshake authenticated
// through the gateway reaches the api FROM the gateway, so counting the caller
// would put every customer's failures in one bucket and let one attacker exhaust
// a threshold that then refuses everybody (research R14).
/** Bounded, and the bound is the decision rather than a detail: this map is keyed
* by attacker-controlled input, so unbounded it would be a memory-exhaustion
* vector — a fallback that closed a brute-force hole by opening a worse one. */
const FALLBACK_MAX_KEYS = 10_000;
@Injectable()
export class AuthLimiter {
private readonly fallback = createFallbackCounter({
windowMs: WINDOW_MS,
maxKeys: FALLBACK_MAX_KEYS,
});
private lastDegradationLog = 0;
constructor(
@Inject(COUNTER_STORE) private readonly store: CounterStore,
@Inject(LOGGER) private readonly logger: Logger,
) {}
/** Count one failed authentication. */
async recordFailure(address: string): Promise<void> {
const now = Date.now();
const key = authKey(address, windowStart(now, WINDOW_MS));
if ((await this.store.increment(key, now)) === null) {
this.degradation();
this.fallback.increment(address, now);
}
}
/** Has this address already spent its allowance?
*
* READS WITHOUT COUNTING. A check that also writes would refuse on its own
* questions, and this one runs on every request that presents a credential —
* including the valid ones.
*
* When the shared store is unreachable it answers from the in-process count,
* which is the whole point: the guarantee gets weaker, not absent. A key the
* fallback could not admit answers `true` — refusing an address we cannot track
* is the safe direction while degraded, and the cap makes that a bounded
* population rather than everybody. */
async isOverThreshold(address: string): Promise<boolean> {
const now = Date.now();
const threshold = authFailureThreshold();
const shared = await this.store.get(
authKey(address, windowStart(now, WINDOW_MS)),
);
if (shared !== null) return shared >= threshold;
this.degradation();
const local = this.fallback.peek(address, now);
return local === null ? true : local >= threshold;
}
/** One line, rate limited at the logger. A Redis outage under load would
* otherwise emit one per attempt, which is how one outage becomes two. No
* credential and no address (NFR-SEC-06); the count of tracked addresses is
* what an operator actually needs. */
private degradation(): void {
const now = Date.now();
if (now - this.lastDegradationLog < 10_000) return;
this.lastDegradationLog = now;
this.logger.log("error", "limits.auth_degraded", {
detail:
"counter store unreachable; counting failed authentications in process",
tracked: this.fallback.size(),
});
}
}import type { IncomingMessage, ServerResponse } from "node:http";
import { Inject, Injectable, type NestMiddleware } from "@nestjs/common";
import type { Logger } from "@relay/service-kit";
import type { Db } from "../db/client";
import { environmentLimits } from "../db/repository";
import type { RequestWithPrincipal } from "../auth/principal";
import { LOGGER } from "../logger";
import { remaining, resetAt, windowStart } from "./bucket";
import { clientAddress } from "./client-address";
import { COUNTER_STORE, LIMITS_DB } from "./limits.module";
import { authFailureThreshold, WINDOW_MS, type LimitedOperation } from "./policy";
import { authKey, counterKey, type CounterStore } from "./store";
/** Read once per call site so a test that freezes time sees one instant. */
const now0 = (): number => Date.now();
// The tenant limiter (FR-RTL-01…04).
//
// MIDDLEWARE, NOT A GUARD, for two reasons. The credentials chapter's: Nest constructs
// request-scoped providers before the enhancer chain, so a guard cannot be the
// thing that resolves tenant scope. And one of its own: FR-RTL-02 wants the three
// headers on SUCCESSFUL responses, and a guard that returns `true` has no natural
// place to set a header on a response the handler has not produced yet.
//
// AFTER `AuthenticateMiddleware`, and the order is forced: the counters are keyed
// by environment and the environment comes from the credential.
//
// COUNT EACH OPERATION ONCE, AT THE DOOR IT ENTERED (research R17). The exemption
// cannot key off the principal, because the gateway forwards the END USER's token
// on all three of its api calls — `/internal/session`, `/internal/backfill`,
// `/internal/messages` are all `@Accepts("user")` and resolve exactly like
// customer traffic. Only the dispatcher carries the platform credential. So the
// route decides, not the caller:
//
// /v1/… counted. A message send decrements both budgets (FR-RTL-01).
// /internal/… not counted. The gateway already counted the handshake
// against `connect` and the frame against `send`; counting
// again here would charge the socket twice and make a
// reconnect storm eat a customer's request budget.
// /healthz never limited. Docker polls it every five seconds and
// `up -d --wait` depends on the answer; a limiter that can
// refuse it can stop a deployment.
const PUBLIC_PREFIX = "/v1/";
const SEND_PATH = /^\/v1\/channels\/[^/]+\/messages\/?$/;
/** Account creation, by ANALOGY with FR-AUT-12 rather than under it — that clause
* is about failed AUTHENTICATION, and a signup is not an authentication at all. Same
* key shape, same threshold, no clause of its own. Limited per SOURCE ADDRESS, because it has no
* tenant to key on — that is the point of it — and an unlimited
* account-creation route is not acceptable in a platform that limits everything
* else. It also has no guard, so T027a's refusal cannot reach it. */
const SIGNUP_PATH = /^\/auth\/[^/]+\/(start|callback)\/?$/;
interface Decision {
operation: LimitedOperation;
limit: number;
remaining: number;
resetSeconds: number;
refused: boolean;
counted: boolean;
}
/** Which budgets a path spends. Empty means the route is not counted at all. */
export function operationsFor(
method: string,
path: string,
): LimitedOperation[] {
if (!path.startsWith(PUBLIC_PREFIX)) return [];
if (method === "POST" && SEND_PATH.test(path)) return ["rest", "send"];
return ["rest"];
}
@Injectable()
export class RateLimitMiddleware implements NestMiddleware {
constructor(
@Inject(LIMITS_DB) private readonly db: Db,
@Inject(COUNTER_STORE) private readonly store: CounterStore,
@Inject(LOGGER) private readonly logger: Logger,
) {}
async use(
req: RequestWithPrincipal & IncomingMessage,
res: ServerResponse,
next: () => void,
): Promise<void> {
// `originalUrl`, NOT `url`. Express rewrites `req.url` relative to the mount
// point, and a middleware applied through `forRoutes("{*path}")` is mounted
// at the match — so `req.url` is `/` for every request and the route rules
// below would never match anything. Found by probe at implementation, and
// the same read is why the request log recorded `/` for every request from
// chapter 2.2 until this chapter fixed it.
const raw =
(req as unknown as { originalUrl?: string }).originalUrl ?? req.url ?? "/";
const path = raw.split("?")[0] ?? "/";
const operations = operationsFor(req.method ?? "GET", path);
const principal = req.principal;
// Account creation first: no tenant, no guard, so it is neither counted like
// customer traffic nor refusable by `CredentialGuard`. Same counter family
// and same threshold as failed authentication (FR-AUT-12, research R17).
if (SIGNUP_PATH.test(path)) {
const address = clientAddress(req);
const count = await this.store.increment(
authKey(address, windowStart(now0(), WINDOW_MS)) + ":signup",
now0(),
);
if (count !== null && count > authFailureThreshold()) {
res.statusCode = 429;
res.setHeader("Retry-After", "60");
res.setHeader("content-type", "application/json");
res.end(
JSON.stringify({
code: "rate_limited",
message: "too many sign-up attempts from this address; retry shortly",
docs_url: "https://relay.example/docs/errors/rate_limited",
request_id: String(res.getHeader("X-Request-Id") ?? ""),
}),
);
return;
}
next();
return;
}
// An environment to key on, or nothing to do. A platform principal has none
// by construction — the dispatcher's credential belongs to a deployment, not
// a tenant — and an absent principal means the guard is about to refuse this
// or the route is pre-credential.
const environmentId =
principal !== undefined && "environmentId" in principal
? principal.environmentId
: undefined;
if (operations.length === 0 || environmentId === undefined) {
next();
return;
}
const limits = await environmentLimits(this.db, environmentId);
if (limits === null) {
next();
return;
}
const now = Date.now();
const start = windowStart(now, WINDOW_MS);
const resetSeconds = Math.ceil(resetAt(now, WINDOW_MS) / 1000);
const decisions: Decision[] = [];
for (const operation of operations) {
const limit = limits[operation];
const count = await this.store.increment(
counterKey(environmentId, operation, start),
now,
);
decisions.push({
operation,
limit,
remaining: count === null ? limit : remaining(count, limit),
resetSeconds,
refused: count !== null && count > limit,
counted: count !== null,
});
}
const refusal = decisions.find((d) => d.refused);
// THE HEADERS DESCRIBE WHICHEVER HAS FEWER REMAINING, because that is the one
// that will refuse first and the only value a client can schedule against. A
// client with 400 request-slots and 12 send-slots needs to hear 12; reporting
// 400 would be a header that lies by omission. A tie reports the first, which
// is `rest` (research R11).
//
// EXCEPT WHEN ONE OF THEM ACTUALLY REFUSED, and that exception was found by
// capturing the transcript rather than by a test. Both budgets can reach
// zero remaining in the same request while only one of them is over: with
// `rest` at 3 and `send` at 2, the third send leaves both at zero remaining
// and only `send` refused. "Fewest remaining" then picks `rest` on the tie,
// and the response says `X-RateLimit-Limit: 3` above a body reading "too many
// messages" — two numbers describing different budgets in one refusal
// (research R41). A refusal describes the budget that refused.
const nearest =
refusal ?? decisions.reduce((a, b) => (b.remaining < a.remaining ? b : a));
const degraded = decisions.some((d) => !d.counted);
res.setHeader("X-RateLimit-Limit", String(nearest.limit));
if (degraded) {
// `Limit` only. It is policy read from Postgres and is not degraded; the
// other two exist only because something was counting, and inventing them
// is the failure FR-RTL-02 forbids. NOT a sentinel — a client that does not
// know `-1` would parse it as a number and conclude it was over its limit
// (research R6).
this.degradation(environmentId, req);
} else {
res.setHeader("X-RateLimit-Remaining", String(nearest.remaining));
res.setHeader("X-RateLimit-Reset", String(nearest.resetSeconds));
}
if (refusal !== undefined) {
const retryAfter = Math.max(1, refusal.resetSeconds - Math.floor(now / 1000));
res.setHeader("Retry-After", String(retryAfter));
res.setHeader("X-RateLimit-Remaining", "0");
// The message names WHICH limit was reached: "too many requests" and "too
// many messages" are different problems, one saying batch and the other
// saying slow down. Neither names a credential (NFR-SEC-06).
const what =
refusal.operation === "send" ? "messages" : "requests";
res.statusCode = 429;
res.setHeader("content-type", "application/json");
res.end(
JSON.stringify({
code: "rate_limited",
message: `too many ${what} for this environment; retry after ${retryAfter} seconds`,
docs_url: "https://relay.example/docs/errors/rate_limited",
request_id: String(res.getHeader("X-Request-Id") ?? ""),
}),
);
return;
}
next();
}
private lastDegradationLog = 0;
/** One line, rate limited at the logger. A Redis outage under load would
* otherwise emit one per request, which is how one outage becomes two. Carries
* the request id and the environment — NFR-OBS-01 asks for request id, tenant
* id and correlation id, and the platform mints no correlation id yet. Carries
* no credential (NFR-SEC-06). */
private degradation(environmentId: string, req: IncomingMessage): void {
const now = Date.now();
if (now - this.lastDegradationLog < 10_000) return;
this.lastDegradationLog = now;
void req;
// `error`, not `info`: the limiter is doing the right thing by serving, and
// an unreachable store is still an operational fault somebody should see.
// The logger's two levels are the service-kit's, unchanged since 1.4.
this.logger.log("error", "limits.degraded", {
environment_id: environmentId,
detail: "counter store unreachable; serving without counting",
});
}
}import { Redis } from "ioredis";
// The gateway's counter (research R12, R20).
//
// ITS OWN CLIENT, not fanout's, and that is forced rather than preferred.
// `Fanout` is a closed interface — `onDelivery`, `publish`, `subscribe`,
// `unsubscribe`, `close` — and exposes neither of the two clients it holds. One
// of them is a SUBSCRIBER, and a Redis connection in subscribe mode cannot run
// `INCR`. And `fanout` is optional in the session server, so a limiter riding its
// lifecycle would vanish in every configuration that has no fabric — which is
// every chapter-2.5 test.
//
// So: one more client, and a `close()` the session server calls. `fanout.ts`
// already sets that precedent for this service.
//
// THE SAME KEYS THE API USES. Two services increment one bucket, which is why
// the counter lives in Redis rather than in either process: neither can see the
// other's memory, and a socket send has to count against the same `send` budget a
// REST send does or a client could double its allowance by opening a socket
// (research R11).
export const DEFAULT_REDIS_URL = "redis://localhost:6379";
/** The window an instant belongs to. Floored, so two instances agree without
* coordinating — the same arithmetic as the api's, deliberately duplicated
* rather than shared: a package for two small functions would be an abstraction
* constitution VII asks to be justified, and this one could not be. */
export function windowStartFor(nowMs: number, windowMs: number): number {
return Math.floor(nowMs / windowMs) * windowMs;
}
/** Is this count past the allowance?
*
* `null` — the store could not be reached — is NOT over. Both of the gateway's
* limits are tenant limits, so they fail open like the api's: Redis is not a
* source of truth, and a cache outage is not a reason to refuse a paying
* customer's traffic. */
export function overLimit(count: number | null, limit: number): boolean {
if (count === null) return false;
return count > limit;
}
/** What one counted operation decided, and everything a refusal has to say.
*
* The api reports the same four numbers in three headers plus `Retry-After`;
* the gateway needs them for the handshake refusal, which IS an HTTP response
* and can carry headers. The frame refusal cannot — there is nowhere on an
* `error` frame to put them — which is why the socket's two refusals do not
* look alike (research R7). */
export interface Decision {
over: boolean;
limit: number;
remaining: number;
/** Unix seconds, matching `X-RateLimit-Reset`. */
resetSeconds: number;
/** Whole seconds until the window turns over, for `Retry-After`. At least 1:
* `Retry-After: 0` invites an immediate retry that is certain to fail. */
retryAfterSeconds: number;
}
/** The arithmetic, with no store in it — so a window boundary is a test rather
* than a wait. A `null` count means the store could not be reached. */
export function decide(
count: number | null,
limit: number,
nowMs: number,
windowMs: number,
): Decision {
const reset = windowStartFor(nowMs, windowMs) + windowMs;
return {
over: overLimit(count, limit),
limit,
remaining: Math.max(0, limit - (count ?? 0)),
resetSeconds: Math.ceil(reset / 1_000),
retryAfterSeconds: Math.max(1, Math.ceil((reset - nowMs) / 1_000)),
};
}
export interface GatewayLimits {
/** Count one operation and report what that decided. */
spend(
environmentId: string,
operation: "connect" | "send",
limit: number,
): Promise<Decision>;
close(): Promise<void>;
}
const WINDOW_MS = 60_000;
const DOWN_WINDOW_MS = 5_000;
export function createGatewayLimits(
url: string = process.env["RELAY_REDIS_URL"] ?? DEFAULT_REDIS_URL,
): GatewayLimits {
const redis = new Redis(url, {
lazyConnect: true,
maxRetriesPerRequest: 0,
connectTimeout: 1_000,
});
// A dead store is an expected state, not an exception. Without a listener
// ioredis emits `error` on an EventEmitter with none attached and Node turns
// that into an unhandled exception — the gateway would die for the thing it is
// designed to survive.
redis.on("error", () => {});
// A known-down store is not retried on the connect path. Waiting out a connect
// timeout per handshake would turn a cache outage into a slow one, and
// NFR-PRF-04 asks for a handshake under a second (research R34).
let downUntil = 0;
return {
async spend(environmentId, operation, limit) {
const now = Date.now();
if (now < downUntil) return decide(null, limit, now, WINDOW_MS);
const key = `rl:${environmentId}:${operation}:${windowStartFor(now, WINDOW_MS)}`;
try {
const count = await redis.incr(key);
if (count === 1) await redis.pexpire(key, WINDOW_MS);
downUntil = 0;
return decide(count, limit, now, WINDOW_MS);
} catch {
downUntil = now + DOWN_WINDOW_MS;
return decide(null, limit, now, WINDOW_MS);
}
},
async close() {
redis.disconnect();
},
};
}The store's lifecycle is a module for one reason: an ioredis client holds the
event loop open, and every long-lived resource in this api closes through
OnModuleDestroy. Without the hook the suites would hang after their assertions
pass — green tests and a lane that never returns, which is the worst shape
available.
import { Inject, Injectable, Module, type OnModuleDestroy } from "@nestjs/common";
import { createDb, createPool, type Db } from "../db/client";
import { apiLogger, LOGGER } from "../logger";
import { createCounterStore, type CounterStore } from "./store";
// The counter store's home.
//
// It is a module for one reason: the client has to be closed. Every long-lived
// resource in this api closes through `OnModuleDestroy` — the outbox relay, the
// delivery relay, the event consumer — `main.ts` enables shutdown hooks, and
// every api integration suite ends `await app.close()`.
//
// An ioredis client holds the event loop open. Without the hook the suites would
// HANG AFTER THEIR ASSERTIONS PASS, which is the worst shape available: green
// tests and a lane that never returns, in a project that has spent three chapters
// clearing one (research R20).
export const COUNTER_STORE = "COUNTER_STORE";
/** The limiter's own pool. Its own rather than the auth middleware's, because a
* middleware that had to be handed another middleware's connection would couple
* two things that only share a request. */
export const LIMITS_DB = "LIMITS_DB";
@Injectable()
export class CounterStoreLifecycle implements OnModuleDestroy {
constructor(@Inject(COUNTER_STORE) private readonly store: CounterStore) {}
async onModuleDestroy(): Promise<void> {
await this.store.close();
}
}
@Module({
providers: [
{ provide: COUNTER_STORE, useFactory: (): CounterStore => createCounterStore() },
{ provide: LIMITS_DB, useFactory: (): Db => createDb(createPool()) },
// Exported so the auth limiter can be constructed inside `AuthModule`, which
// imports this one. `AppModule` provides the same token for everything else;
// the factory is shared so both are the same kind of logger, and the DI
// bargain ADR-15 buys — a test swaps the sink by overriding one provider —
// holds in both places.
{ provide: LOGGER, useFactory: apiLogger },
CounterStoreLifecycle,
],
exports: [COUNTER_STORE, LIMITS_DB, LOGGER],
})
export class LimitsModule {}And the three pure files get pure tests: no clock, no store, no framework, so a branch they miss is a case nobody thought of rather than one nobody could reach.
import { describe, expect, it } from "vitest";
import { remaining, resetAt, windowStart } from "./bucket";
// The fixed-window arithmetic (research R1). Pure: no store, no
// clock of its own — every function takes the instant it should reason about,
// which is what lets a boundary be tested rather than waited for.
const MINUTE = 60_000;
describe("windowStart", () => {
it("floors an instant to the window it belongs to", () => {
expect(windowStart(90_000, MINUTE)).toBe(60_000);
expect(windowStart(60_000, MINUTE)).toBe(60_000);
expect(windowStart(59_999, MINUTE)).toBe(0);
});
it("is the key's own suffix, which is why nothing stores a reset time", () => {
// Two instances computing this from the same clock agree without talking.
// A stored reset time is a value they could disagree about — the clock-skew
// edge case, closed by construction rather than by agreement.
expect(windowStart(119_999, MINUTE)).toBe(windowStart(60_000, MINUTE));
});
});
describe("resetAt", () => {
it("is the end of the current window, not a refill curve", () => {
// The deciding reason for a fixed window over a token bucket (R1):
// `X-RateLimit-Reset` has to name one moment, and this is it.
expect(resetAt(90_000, MINUTE)).toBe(120_000);
});
it("never lands in the past", () => {
for (const now of [0, 1, 59_999, 60_000, 60_001]) {
expect(resetAt(now, MINUTE)).toBeGreaterThan(now - 1);
}
});
});
describe("remaining", () => {
it("counts down from the limit as the count rises", () => {
expect(remaining(1, 600)).toBe(599);
expect(remaining(600, 600)).toBe(0);
});
it("A COUNTER THAT HAS NEVER BEEN WRITTEN returns a full allowance", () => {
// The `INCR`-returns-1 path, which is also where `EXPIRE` is set. A brand-new
// environment's first request must not look like an exhausted one.
expect(remaining(1, 600)).toBe(599);
expect(remaining(0, 600)).toBe(600);
});
it("A LIMIT LOWERED MID-WINDOW yields zero, never a negative", () => {
// An operator drops an environment from 600 to 2 while 40 requests are
// already counted. `Remaining: -38` is a number a client would parse and act
// on; zero is the truth.
expect(remaining(40, 2)).toBe(0);
});
it("is zero at the limit, so a refusal reports zero rather than one", () => {
expect(remaining(600, 600)).toBe(0);
expect(remaining(601, 600)).toBe(0);
});
});
describe("the boundary burst, which is the cost of a fixed window", () => {
it("permits up to twice the limit across one window edge", () => {
// R1 accepted this rather than hiding it: 600 in the last instant of one
// window and 600 in the first instant of the next is 1,200 inside two
// minutes. Asserted so the chapter's claim is a test rather than a comment.
const limit = 600;
const firstWindow = windowStart(59_999, MINUTE);
const secondWindow = windowStart(60_000, MINUTE);
expect(firstWindow).not.toBe(secondWindow);
expect(remaining(limit, limit) + limit).toBe(limit);
// Two distinct keys, each allowing `limit`, one millisecond apart.
expect(remaining(1, limit)).toBe(limit - 1);
});
});import { afterEach, describe, expect, it } from "vitest";
import {
DEFAULT_AUTH_FAILURES_PER_MINUTE,
DEFAULT_LIMITS,
WINDOW_MS,
authFailureThreshold,
} from "./policy";
// The numbers, and what happens when someone fat-fingers the one that is
// configurable.
describe("authFailureThreshold", () => {
const previous = process.env["RELAY_AUTH_FAILURES_PER_MINUTE"];
afterEach(() => {
if (previous === undefined) delete process.env["RELAY_AUTH_FAILURES_PER_MINUTE"];
else process.env["RELAY_AUTH_FAILURES_PER_MINUTE"] = previous;
});
const withEnv = (value: string) => {
process.env["RELAY_AUTH_FAILURES_PER_MINUTE"] = value;
return authFailureThreshold();
};
it("uses the documented default when nothing is set", () => {
delete process.env["RELAY_AUTH_FAILURES_PER_MINUTE"];
expect(authFailureThreshold()).toBe(DEFAULT_AUTH_FAILURES_PER_MINUTE);
});
it("honours a configured threshold", () => {
expect(withEnv("3")).toBe(3);
});
it("FALLS BACK TO THE DEFAULT for anything that is not a positive number", () => {
// The failure mode this is protecting against is not exotic. `parseInt`
// returns NaN for "ten" and 0 for "0", and either would be silently
// catastrophic in a different direction: NaN makes every comparison false,
// so the limiter never triggers; 0 makes it trigger on the first failed
// auth, locking out an address that mistyped a password once.
//
// A misconfigured security control that fails silently is worse than one
// that is absent, because nobody looks for it.
for (const bad of ["ten", "", "0", "-1", "NaN"]) {
expect(withEnv(bad)).toBe(DEFAULT_AUTH_FAILURES_PER_MINUTE);
}
});
});
describe("the defaults themselves", () => {
it("are the numbers the chapter derived, not round ones", () => {
// Pinned so a change has to be deliberate. Each was re-derived against
// NFR-SCL and NFR-PRF (research R26); the chapter states the derivation and
// this asserts the result.
expect(DEFAULT_LIMITS).toEqual({ rest: 600, send: 600, connect: 3_000 });
expect(DEFAULT_AUTH_FAILURES_PER_MINUTE).toBe(10);
// One minute, because every limit above is stated per minute and a window
// that did not match the unit would need a second number to explain it.
expect(WINDOW_MS).toBe(60_000);
});
});import { describe, expect, it } from "vitest";
import { createFallbackCounter } from "./fallback";
// The in-process counter the auth limiter falls back to when Redis is gone
// (research R3). Pure apart from the map it owns; every call takes
// the instant it should reason about.
const MINUTE = 60_000;
describe("the fallback counter", () => {
it("counts per key within a window", () => {
const c = createFallbackCounter({ windowMs: MINUTE, maxKeys: 100 });
expect(c.increment("1.2.3.4", 0)).toBe(1);
expect(c.increment("1.2.3.4", 10)).toBe(2);
expect(c.increment("5.6.7.8", 10)).toBe(1);
});
it("starts again in the next window", () => {
const c = createFallbackCounter({ windowMs: MINUTE, maxKeys: 100 });
expect(c.increment("1.2.3.4", 0)).toBe(1);
expect(c.increment("1.2.3.4", 59_999)).toBe(2);
expect(c.increment("1.2.3.4", 60_000)).toBe(1);
});
it("STOPS ADMITTING NEW KEYS at the cap rather than evicting", () => {
// The decision research R3 spent its argument on. An eviction policy on a map
// keyed by attacker-controlled input is a policy the attacker drives: fill
// the map, evict the entry that was counting them, start again.
//
// Refusing new keys degrades to "addresses we are already tracking stay
// tracked", which is the safe direction — and the whole reason this
// structure is written down rather than being an implementation detail.
const c = createFallbackCounter({ windowMs: MINUTE, maxKeys: 2 });
expect(c.increment("a", 0)).toBe(1);
expect(c.increment("b", 0)).toBe(1);
// Third key: not admitted, and says so.
expect(c.increment("c", 0)).toBeNull();
// The two already tracked keep counting.
expect(c.increment("a", 0)).toBe(2);
expect(c.increment("b", 0)).toBe(2);
});
it("admits a new key again once a window turns over", () => {
// The cap is on live keys, not on keys ever seen. A window boundary clears
// expired entries, so an outage lasting hours does not permanently refuse to
// count anybody new.
const c = createFallbackCounter({ windowMs: MINUTE, maxKeys: 1 });
expect(c.increment("a", 0)).toBe(1);
expect(c.increment("b", 0)).toBeNull();
expect(c.increment("b", 60_000)).toBe(1);
});
it("peeks without counting, because a check must not be its own failure", () => {
// `isOverThreshold` runs on every request that presents a credential,
// including the valid ones. A check that also wrote would push an address
// over on its own questions.
const c = createFallbackCounter({ windowMs: MINUTE, maxKeys: 100 });
c.increment("1.2.3.4", 0);
c.increment("1.2.3.4", 0);
expect(c.peek("1.2.3.4", 0)).toBe(2);
expect(c.peek("1.2.3.4", 0)).toBe(2);
expect(c.peek("never-seen", 0)).toBe(0);
});
it("peeks zero once the window has turned over", () => {
const c = createFallbackCounter({ windowMs: MINUTE, maxKeys: 100 });
c.increment("1.2.3.4", 0);
expect(c.peek("1.2.3.4", 0)).toBe(1);
expect(c.peek("1.2.3.4", 60_000)).toBe(0);
});
it("never grows past the cap, however many keys arrive", () => {
// The memory bound is the point: an unbounded map keyed by source address is
// a memory-exhaustion vector, so a fallback that closed a brute-force hole
// would have opened a worse one.
const c = createFallbackCounter({ windowMs: MINUTE, maxKeys: 3 });
for (let i = 0; i < 1_000; i++) c.increment(`ip-${i}`, 0);
expect(c.size()).toBe(3);
});
});
describe("peek at the cap", () => {
it("REPORTS AN UNKNOWN KEY AS UNKNOWN, not as zero, once the map is full", () => {
// The other side of "stop admitting rather than evict". `increment` refuses
// to add a key past the cap; `peek` has to answer honestly about one it
// therefore never learned. Returning 0 would mean "this address has failed
// no auths", which for the auth limiter is the wrong direction — the whole
// point of R3 is that this counter does not fail open. `null` says "I do
// not know", and the caller decides.
const counter = createFallbackCounter({ windowMs: MINUTE, maxKeys: 2 });
counter.increment("ip-1", 0);
counter.increment("ip-2", 0);
expect(counter.peek("ip-1", 0)).toBe(1);
expect(counter.peek("ip-3", 0)).toBeNull();
});
it("reports an unknown key as zero while there is still room", () => {
// Under the cap, "I have never seen this address" and "this address has
// failed nothing" are the same statement.
const counter = createFallbackCounter({ windowMs: MINUTE, maxKeys: 2 });
counter.increment("ip-1", 0);
expect(counter.peek("ip-3", 0)).toBe(0);
});
it("forgets a count from a previous window without being swept", () => {
const counter = createFallbackCounter({ windowMs: MINUTE, maxKeys: 2 });
counter.increment("ip-1", 0);
expect(counter.peek("ip-1", MINUTE)).toBe(0);
});
});fallback.ts earns the strictest reading of constitution VI available: it is what
the AUTH limiter degrades to, and an unmeasured branch in it is a hole in the thing
this chapter is about.
rate_limited and the fourth field. Three of these four files were written in
Part 1 and have been waiting since.
@@ -214,20 +214,32 @@ export const typingSendSchema = z.strictObject({
payload: z.strictObject({
channel: z.string().min(1),
}),
});
/** Protocol-level error — EIR-API-04's error shape, reused on the socket
- * (this chapter's recorded decision). `request_id` joins in Part 2, when a
- * gateway exists to mint one. */
+ * (chapter 1.3's recorded decision).
+ *
+ * `request_id` ARRIVED IN THE RATE-LIMIT CHAPTER, not in Part 2. The comment here promised
+ * it "joins in Part 2, when a gateway exists to mint one"; Part 2 came and went,
+ * the gateway existed, and the field did not. Constitution V asks for four fields
+ * and the platform sent three for twenty-two chapters.
+ *
+ * REQUIRED, not optional, and that was a decision rather than an oversight. A
+ * server-initiated frame is arguably not a response to a request, so optional
+ * would have been defensible — and it would have been the fourth instance of the
+ * habit this chapter is about: `rate_limited`, close code 4008 and this field
+ * were all declared here and left unenforced. The gateway mints one per answered
+ * frame instead (research R13). */
export const errorFrameSchema = z.strictObject({
type: z.literal("error"),
payload: z.strictObject({
code: z.string().min(1),
message: z.string().min(1),
docs_url: z.string().min(1),
+ request_id: z.string().min(1),
field: z.string().min(1).optional(),
}),
});
/** Every frame either end may legally utter. */
export const frameSchema = z.discriminatedUnion("type", [@@ -69,12 +69,17 @@ const valid: Record<string, unknown> = {
error: {
type: "error",
payload: {
code: "invalid_frame",
message: "no",
docs_url: "https://docs.example/errors/invalid_frame",
+ // The fourth field, required rather than optional. The
+ // comment above this schema promised it "joins in Part 2, when a gateway
+ // exists to mint one" — Part 2 came and went, and constitution V has asked
+ // for four fields since 1.3.
+ request_id: "01JABCDEFGHJKMNPQRSTVWXYZ",
},
},
};
describe("every frame parses its valid specimen and round-trips", () => {
for (const [name, frame] of Object.entries(valid)) {@@ -95,12 +95,16 @@ export function serve(options: ServeOptions): Server {
} else {
status = 404;
body = {
code: "not_found",
message: `no route for ${req.method ?? "?"} ${path}`,
docs_url: notFoundDocsUrl,
+ // The fourth field constitution V has asked for since 1.3.
+ // Everywhere, not only on the rate-limit error — four fields on one
+ // status and three on the others is worse than either consistent answer.
+ request_id: requestId,
};
}
res.statusCode = status;
res.end(JSON.stringify(body));
logger.log("info", "request", {
request_id: requestId,@@ -72,16 +72,29 @@ export class ProtocolErrorFilter implements ExceptionFilter {
const message =
exception instanceof HttpException
? exception.message
: "unexpected internal error";
res.statusCode = status;
res.setHeader("content-type", "application/json");
+ // FOUR FIELDS AS OF THE RATE-LIMIT CHAPTER, and constitution V has asked for four since
+ // chapter 1.3. `request_id` was promised "in Part 2, when a gateway exists to
+ // mint one"; the gateway arrived and the field did not. It is read back off
+ // the response rather than threaded through, because `RequestContextMiddleware`
+ // has already set `X-Request-Id` by the time anything can throw — one id, in
+ // the header and the body, from one place.
+ //
+ // TOP-LEVEL, NOT NESTED. EIR-API-04's worked example wrapped these in an
+ // `error` key until this chapter checked what the platform actually sends;
+ // it never sent that shape. Wrapping every error response would be a breaking
+ // change and CON-05 makes breaking changes a URL-versioning event, so the
+ // document was brought to the code — SRS 1.3 (research R27).
res.end(
JSON.stringify({
code,
message,
docs_url: docsUrl(code),
+ request_id: String(res.getHeader("X-Request-Id") ?? ""),
...(field !== null ? { field } : {}),
}),
);
}
}The chain, the principal it carries, and the policy it reads.
schema.ts also loses three numbers. The deduplication chapter wrote a comment explaining that
a chapter number in a source comment is a reference that ages — and made its point
by listing the ordinals the cross-tenant gauntlet had already passed through. The
plan moved again while this chapter was being written, and the explanation went
stale on its own subject. It now names none.
@@ -20,17 +20,28 @@ export class RequestContextMiddleware implements NestMiddleware {
// ...and on the request, so a handler can put it in a line of its own. The
// fan-out publish logs its failure from inside the send handler, and NFR-OBS-01
// wants a request id in every structured line while NFR-OBS-06 wants five-minute
// traceability from one. Until now the id existed only here and on the response
// header, which a handler cannot read without taking over the response.
(req as { requestId?: string }).requestId = requestId;
+ // `originalUrl` first, and this line was WRONG from chapter 2.2 until this chapter.
+ // Express rewrites `req.url` relative to the mount point, and this middleware
+ // is applied through `forRoutes("{*path}")`, so `req.url` is `/` — every
+ // request this api has logged recorded `/` as its path. NFR-OBS-06 asks for
+ // one structured line per request that an operator can grep; a line whose
+ // path is always `/` is one they cannot.
+ //
+ // Found by probe while wiring the rate limiter, which reads the same value
+ // to decide which routes it counts and would have counted nothing.
+ const path =
+ (req as { originalUrl?: string }).originalUrl ?? req.url ?? "/";
res.on("finish", () => {
this.logger.log("info", "request", {
request_id: requestId,
method: req.method,
- path: req.url,
+ path,
status: res.statusCode,
});
});
next();
}
}@@ -59,19 +59,33 @@ export type Principal =
export type PrincipalKind = Principal["kind"];
/** The request as everything downstream of the middleware sees it. The
* principal is optional at the type level for one honest reason: a request that
* presented nothing has none, and pre-credential routes (signup) are reached
* exactly that way. */
+/** Set by `AuthenticateMiddleware` when this address has already
+ * spent its failed-authentication allowance, and read by `CredentialGuard`,
+ * which throws the 429.
+ *
+ * THE MIDDLEWARE NEVER THROWS, by documented design — pre-credential routes
+ * reach their handlers by having no principal — so the refusal has to be raised
+ * somewhere that already refuses. The guard owns the 401 that EIR-API-04 wants this
+ * indistinguishable from, and it already throws the object form that carries a
+ * `code` (research R18). */
+export const OVER_AUTH_THRESHOLD = Symbol.for("relay:over-auth-threshold");
+
export interface RequestWithPrincipal {
headers: Record<string, string | string[] | undefined>;
principal?: Principal;
/** The id `RequestContextMiddleware` generated for this request. A handler that
* logs on its own — the fan-out publish does — needs it, and NFR-OBS-01 requires
* it in every structured line. */
requestId?: string;
+ /** Set when this source address has spent its
+ * failed-authentication allowance. See `OVER_AUTH_THRESHOLD` above. */
+ [OVER_AUTH_THRESHOLD]?: boolean;
}
/** How a credential class is named to a human. Used by the wrong-credential
* error, which must say what was presented and what was expected — and must
* never quote the credential (NFR-SEC-06). */
export function describePrincipalKind(kind: PrincipalKind): string {@@ -3,13 +3,20 @@ import { Inject, Injectable, type NestMiddleware } from "@nestjs/common";
import type { Db } from "../db/client";
import {
authenticateApiKey,
environmentSigningSecret,
} from "../db/repository";
import { looksLikeApiKey } from "./api-key";
-import { bearerCredential, type Principal, type RequestWithPrincipal } from "./principal";
+import { AuthLimiter } from "../limits/auth-limiter";
+import { clientAddress } from "../limits/client-address";
+import {
+ bearerCredential,
+ OVER_AUTH_THRESHOLD,
+ type Principal,
+ type RequestWithPrincipal,
+} from "./principal";
import { environmentClaim, verifyUserToken } from "./user-token";
export const AUTH_DB = "AUTH_DB";
/** Credential in, principal out. The one function that decides who a caller is;
* everything else in the api reads its answer.
@@ -103,21 +110,36 @@ export async function resolvePrincipal(
* request that presents something invalid also has no principal, so "absent"
* and "does not verify" arrive at the same 401 — which is all a caller should
* be able to learn.
*/
@Injectable()
export class AuthenticateMiddleware implements NestMiddleware {
- constructor(@Inject(AUTH_DB) private readonly db: Db) {}
+ constructor(
+ @Inject(AUTH_DB) private readonly db: Db,
+ private readonly authLimiter: AuthLimiter,
+ ) {}
async use(
- req: RequestWithPrincipal,
+ req: RequestWithPrincipal & { socket?: { remoteAddress?: string } },
_res: unknown,
next: () => void,
): Promise<void> {
const credential = bearerCredential(req.headers);
if (credential !== null) {
+ // (FR-AUT-12). The failure is observable HERE — credential
+ // present, principal null — so this is where it is counted. It is not where
+ // it is refused: this middleware never throws, and `CredentialGuard` raises
+ // the 429 from the flag below (research R18).
+ const address = clientAddress(req);
+ if (await this.authLimiter.isOverThreshold(address)) {
+ req[OVER_AUTH_THRESHOLD] = true;
+ }
const principal = await resolvePrincipal(this.db, credential);
- if (principal !== null) req.principal = principal;
+ if (principal !== null) {
+ req.principal = principal;
+ } else {
+ await this.authLimiter.recordFailure(address);
+ }
}
next();
}
}@@ -1,18 +1,20 @@
import {
ForbiddenException,
+ HttpException,
Injectable,
SetMetadata,
UnauthorizedException,
type CanActivate,
type ExecutionContext,
} from "@nestjs/common";
import { Reflector } from "@nestjs/core";
import {
describePrincipalKind,
+ OVER_AUTH_THRESHOLD,
type PrincipalKind,
type RequestWithPrincipal,
} from "./principal";
const ACCEPTS = "relay:accepts";
@@ -55,12 +57,36 @@ export class CredentialGuard implements CanActivate {
context.getClass(),
]) ?? EITHER;
const req = context.switchToHttp().getRequest<RequestWithPrincipal>();
const principal = req.principal;
+ // (FR-AUT-12, FR-RTL-02, research R18). The refusal for an
+ // over-threshold address is thrown HERE and not in the middleware that
+ // counted it, because `AuthenticateMiddleware` never throws by documented
+ // design — pre-credential routes reach their handlers by having no principal.
+ //
+ // Three things fall out of putting it here. The invariant survives verbatim.
+ // Both refusals come from one place, which is what EIR-API-04 needs: a caller
+ // must not be able to tell a rate-limited refusal from a wrong-credential
+ // one, or the limiter becomes an oracle. And the guard already throws the
+ // object form that carries a `code`, which is what the envelope needs.
+ //
+ // BEFORE the principal check, so an address over its allowance is refused
+ // whether or not the credential it just presented would have worked.
+ if (req[OVER_AUTH_THRESHOLD] === true) {
+ throw new HttpException(
+ {
+ code: "rate_limited",
+ message:
+ "too many failed authentication attempts from this address; retry shortly",
+ },
+ 429,
+ );
+ }
+
if (!principal) {
throw new UnauthorizedException(
`this route requires a credential: ${expectation(accepted)}, presented as "Authorization: Bearer …"`,
);
}@@ -1,9 +1,11 @@
import { Module } from "@nestjs/common";
import { createDb, createPool, type Db } from "../db/client";
+import { LimitsModule } from "../limits/limits.module";
+import { AuthLimiter } from "../limits/auth-limiter";
import { AUTH_DB, AuthenticateMiddleware } from "./authenticate.middleware";
import { CredentialGuard } from "./credential.guard";
import { DevTokenController } from "./dev-token.controller";
// Authentication's home, beside messages/, internal/ and
// tenancy/. It lives in the api and not in a package or a new service for one
@@ -12,15 +14,20 @@ import { DevTokenController } from "./dev-token.controller";
// gateway's share of the work is one call it was already making (research R1).
//
// The DB handle is its own provider rather than MessagesModule's: this module
// runs BEFORE any tenant scope exists, and borrowing the request-scoped
// machinery 2.2 built would invert the order it needs.
@Module({
+ // The failed-authentication counter. Imported rather than built
+ // here, because the counter store is one client with one lifecycle and two
+ // consumers — this module and the tenant limiter's middleware.
+ imports: [LimitsModule],
controllers: [DevTokenController],
providers: [
{ provide: AUTH_DB, useFactory: (): Db => createDb(createPool()) },
+ AuthLimiter,
AuthenticateMiddleware,
CredentialGuard,
],
- exports: [AUTH_DB, AuthenticateMiddleware, CredentialGuard],
+ exports: [AUTH_DB, AuthenticateMiddleware, CredentialGuard, AuthLimiter],
})
export class AuthModule {}@@ -20,12 +20,14 @@ import { ConsumerModule } from "./consumer/consumer.module";
import { NotificationsModule } from "./notifications/notifications.module";
import { OutboxModule } from "./outbox/outbox.module";
import { WebhooksModule } from "./webhooks/webhooks.module";
import { TenancyModule } from "./tenancy/tenancy.module";
import { LOGGER, apiLogger } from "./logger";
import { ProtocolErrorFilter } from "./protocol-error.filter";
+import { LimitsModule } from "./limits/limits.module";
+import { RateLimitMiddleware } from "./limits/rate-limit.middleware";
import { RequestContextMiddleware } from "./request-context.middleware";
// The application described as a module graph — ADR-15's convention for the
// wide surface Phases 2-4 will grow. Registering the error filter as a
// provider (APP_FILTER) instead of wiring it in main.ts means every entry
// point — including tests — gets the same error envelope for free.
@@ -38,25 +40,30 @@ import { RequestContextMiddleware } from "./request-context.middleware";
InternalModule,
TenancyModule,
OutboxModule,
NotificationsModule,
ConsumerModule,
WebhooksModule,
+ LimitsModule,
],
controllers: [HealthController],
providers: [
{ provide: LOGGER, useFactory: apiLogger },
{ provide: APP_FILTER, useClass: ProtocolErrorFilter },
RequestContextMiddleware,
+ RateLimitMiddleware,
],
})
export class AppModule implements NestModule {
configure(consumer: MiddlewareConsumer): void {
- // Order is the chain: the request gets its id first, then its principal.
+ // Order is the chain: the request gets its id first, then its principal,
+ // then its allowance. The limiter is LAST and that is forced:
+ // it counts per environment and the environment comes from the credential,
+ // so nothing earlier in the chain knows which tenant is asking.
// The credentials chapter put authentication HERE rather than in a guard because Nest
// constructs request-scoped providers before the enhancer chain runs — the
// finding 2.6 paid for, measured again on this path in T004.
consumer
- .apply(RequestContextMiddleware, AuthenticateMiddleware)
+ .apply(RequestContextMiddleware, AuthenticateMiddleware, RateLimitMiddleware)
.forRoutes("{*path}");
}
}@@ -128,23 +128,63 @@ export const environments = pgTable(
.notNull()
.references(() => applications.id),
kind: text("kind").notNull(),
// envelope-encrypted (NFR-SEC-02)
signingSecret: text("signing_secret").notNull(),
retentionDays: integer("retention_days"),
+ // DECLARED IN 2.1 AND STILL EMPTY. Named in SRS §6.1's Environment entity
+ // and SAD §338, read by nothing in seventeen chapters. THIS chapter
+ // deliberately did NOT put rate-limit policy here: the column is named for
+ // quotas, quotas are a later chapter, and the distinction between a limit
+ // that may be lost and a quota that is money is the thing this chapter is
+ // about.
+ // (Deliberately not a chapter NUMBER: the deduplication chapter renumbered
+ // quotas once already, and a comment in a file fenced byte-exact into a
+ // published page goes stale silently. That chapter's rule — cite what a
+ // thing is, never where it will be. A grep for forward references is the
+ // gate, so this comment must not trip it either.)
+ //
+ // Putting one in a field named for the other would collapse in the schema
+ // what the prose spends a chapter drawing (research R31).
quotaConfig: jsonb("quota_config").notNull().default({}),
+ // Per-environment rate limits (FR-RTL-04).
+ //
+ // NULLABLE, AND NULL IS NOT ZERO. Null means "no override, use the
+ // documented default", resolved at read time. Zero means "refuse
+ // everything", which must stay expressible — an environment can be switched
+ // off deliberately — so the two states cannot share a representation.
+ //
+ // Three integers rather than a document, and a slot for an environment with
+ // NONE FOR A ROUTE. That forecloses SRS Appendix C question 5 — whether the
+ // dev-token endpoint should be limited more aggressively than the rest of
+ // its environment — and the question stays open because of it (R30).
+ restLimitPerMinute: integer("rest_limit_per_minute"),
+ sendLimitPerMinute: integer("send_limit_per_minute"),
+ connectLimitPerMinute: integer("connect_limit_per_minute"),
},
(t) => [
check(
"environments_kind_check",
sql`${t.kind} IN ('development','production')`,
),
// FR-TEN-04 says exactly two environments per application. With the CHECK
// above, this unique index IS that rule: two legal kinds, one row each.
// No trigger, no counting query, nothing to lose a race to.
unique("environments_application_kind_unique").on(t.applicationId, t.kind),
+ check(
+ "environments_rest_limit_non_negative",
+ sql`${t.restLimitPerMinute} IS NULL OR ${t.restLimitPerMinute} >= 0`,
+ ),
+ check(
+ "environments_send_limit_non_negative",
+ sql`${t.sendLimitPerMinute} IS NULL OR ${t.sendLimitPerMinute} >= 0`,
+ ),
+ check(
+ "environments_connect_limit_non_negative",
+ sql`${t.connectLimitPerMinute} IS NULL OR ${t.connectLimitPerMinute} >= 0`,
+ ),
],
);
// DECISION: the SRS states the requirements this table serves
// (FR-AUT-01…05, NFR-SEC-02) but no source document defines a key table —
// SAD §6.1 does not have one. Its shape is a chapter derivation, recorded here@@ -1,12 +1,16 @@
import { randomUUID } from "node:crypto";
import { and, asc, desc, eq, gt, inArray, isNull, lt, sql, type SQL } from "drizzle-orm";
import type { Attachment } from "@relay/protocol";
+import {
+ DEFAULT_LIMITS,
+ type LimitedOperation,
+} from "../limits/policy";
import type { Db } from "./client";
import {
apiKeys,
applications,
channels,
consumedEvents,
@@ -240,12 +244,43 @@ export async function environmentSigningSecret(
})
.from(environments)
.where(eq(environments.id, environmentId));
return row ?? null;
}
+/** An environment's rate limits, with nulls resolved to the documented defaults
+ * (FR-RTL-04, research R26).
+ *
+ * RESOLVED HERE RATHER THAN AT THE CALL SITE, because "null means use the
+ * default" is a property of the column and a caller that had to remember it
+ * would eventually forget. Null is NOT zero: zero means refuse everything, and an
+ * environment can be switched off deliberately.
+ *
+ * Returns null for an environment that does not exist, which the caller must tell
+ * apart from an environment with default limits — a request whose credential
+ * named a missing environment is not a request to serve generously. */
+export async function environmentLimits(
+ db: Db,
+ environmentId: string,
+): Promise<Record<LimitedOperation, number> | null> {
+ const [row] = await db
+ .select({
+ rest: environments.restLimitPerMinute,
+ send: environments.sendLimitPerMinute,
+ connect: environments.connectLimitPerMinute,
+ })
+ .from(environments)
+ .where(eq(environments.id, environmentId));
+ if (!row) return null;
+ return {
+ rest: row.rest ?? DEFAULT_LIMITS.rest,
+ send: row.send ?? DEFAULT_LIMITS.send,
+ connect: row.connect ?? DEFAULT_LIMITS.connect,
+ };
+}
+
// ---------------------------------------------------------------------------
// The outbox drain (ADR-06). Part of the ADMIN surface for the
// same reason the credential lookup is: it runs on behalf of the platform
// rather than of a tenant, and it is deliberately NOT scoped by environment —
// one relay drains every environment's events, because an outbox row is work
// the platform owes itself.The limits ride the authentication response, and the gateway caches them on the connection rather than reading them again.
@@ -193,12 +193,29 @@ export const internalSessionResponseSchema = z.strictObject({
*
* `.default(false)` so an api built before this chapter still satisfies the schema
* during a rolling deploy — the gateway then treats a missing field as "not banned",
* which is the pre-chapter behaviour and the safe direction to be wrong in for one
* deploy window. */
banned: z.boolean().default(false),
+ /** The two limits the gateway enforces, resolved from the
+ * environment's policy with nulls already turned into defaults.
+ *
+ * THEY RIDE THIS RESPONSE BECAUSE THE GATEWAY HAS NO DATABASE, and must not
+ * gain one — `registry.ts` states that as a design property: "no pg, no
+ * drizzle-orm, no repository import". The policy is three columns in Postgres
+ * and the api is the only service that reads Postgres, so the limits travel on
+ * the one call the gateway was already making at connect.
+ *
+ * The same move the credentials chapter made on this call, whose comment records it: the
+ * api "answers with the identity AND the memberships … it just asks a better
+ * question than 'what may this user hear'". This asks it for one thing more
+ * (research R12). */
+ limits: z.strictObject({
+ connect: z.number().int().nonnegative(),
+ send: z.number().int().nonnegative(),
+ }),
});
/** The deliveries stream, and its subject grammar.
*
* Here rather than in either service, for the reason the broker chapter moved the event
* grammar here: a consumer that assembles its own subject filter receives@@ -11,13 +11,14 @@ import {
import type { InternalSessionResponse } from "@relay/protocol";
import { AUTH_DB } from "../auth/authenticate.middleware";
import { Accepts, CredentialGuard } from "../auth/credential.guard";
import type { RequestWithPrincipal } from "../auth/principal";
import type { Db } from "../db/client";
-import { Repository } from "../db/repository";
+import { environmentLimits, Repository } from "../db/repository";
+import { DEFAULT_LIMITS } from "../limits/policy";
// `POST /internal/session` — the route that replaced
// `GET /internal/memberships`.
//
// It answers the gateway's only question at connect: who is this, and what may
// they hear? Both halves used to be answered in two different places — the
@@ -72,12 +73,17 @@ export class SessionController {
}
// A verified token for a user this environment has never seen is not an
// error: it is a user with no channels. The gateway's job is delivery, not
// identity forensics — 2.5's rule, and the reason a first connect from a
// brand-new user works before anything is seeded.
const channels = user ? await this.repo.channelsForUser(user.id) : [];
+ // The gateway's limits, resolved here because the gateway has no
+ // database and must not gain one (research R12). Null columns are already
+ // defaults by the time they leave the repository, so the gateway never has to
+ // know that "no override" is a state.
+ const limits = await environmentLimits(this.db, principal.environmentId);
return {
environment_id: principal.environmentId,
user: principal.userExternalId,
// FR-031. THE ROW IS ALREADY IN HAND — `getUserByExternalId` above
// reads it for the channel list — so carrying the ban costs one field and no query.
// The gateway refuses the socket; this route only reports the fact, because the
@@ -92,9 +98,13 @@ export class SessionController {
// the counter costs no extra query, because the membership join already touches
// `channels` to answer the ids.
channel_ids: channels.map((c) => c.channel_id),
revisions: Object.fromEntries(
channels.map((c) => [c.channel_id, c.revision_sequence]),
),
+ limits: {
+ connect: limits?.connect ?? DEFAULT_LIMITS.connect,
+ send: limits?.send ?? DEFAULT_LIMITS.send,
+ },
};
}
}@@ -32,12 +32,17 @@ export type Authentication =
identity: Identity;
channelIds: string[];
/** Per channel, how many revisions it has seen. Reported to the client on the
* ack and never compared here: the gateway has no opinion about staleness, and
* no database to form one with. */
revisions: Record<string, number>;
+ /** The environment's two socket allowances, read from
+ * Postgres by the api and carried on the same response — the gateway has
+ * no database client and R12 spent its whole argument on keeping it that
+ * way. */
+ limits: { connect: number; send: number };
}
| { outcome: "refused" }
| { outcome: "unavailable"; error: string }
/** FR-031. The api answered, the token is perfectly good, and the user is banned in
* this environment. Its own outcome and its own close code (4003), not a reuse of
* `refused`: 4001 means "your credential is bad", which a client acts on by
@@ -70,11 +75,12 @@ export async function authenticate(
// Carried, not trusted: the internal hop forwards this instead of
// asserting an identity the gateway invented.
token,
},
channelIds: session.channel_ids,
revisions: session.revisions,
+ limits: session.limits,
};
} catch (error) {
return { outcome: "unavailable", error: String(error) };
}
}@@ -48,12 +48,22 @@ export interface Connection {
* like the natural way to bound this and hands the duplicate straight back:
* sequences commit in order under a channel row lock but are published by
* whichever gateway instance handled each send, so a prompt 43 can beat a stalled
* 42. Bounded instead by `MAX_RESUME_CHANNELS`, which already caps the cursors
* these are scoped to. */
marks: Record<string, number> | null;
+ /** The environment's send allowance, as it stood when this socket
+ * connected — carried on the session response because the gateway has no
+ * database and must not gain one (research R12).
+ *
+ * FIXED FOR THE LIFE OF THE CONNECTION, and that is a stated property rather
+ * than an accident: a limit changed while a socket is open does not reach it
+ * until the client reconnects. The alternative is a Postgres read per frame, on
+ * the hot path of the thing the limit protects. Beside `marks` for the same
+ * reason — it describes one socket and dies with it. */
+ sendLimit: number;
}
export class Registry {
private readonly byId = new Map<string, Connection>();
add(connection: Connection): void {@@ -1,8 +1,9 @@
import { randomUUID } from "node:crypto";
import type { IncomingMessage, Server } from "node:http";
+import type { Duplex } from "node:stream";
import {
ALL_CHANNELS,
CLOSE_CODES,
docsUrl,
frameSchema,
@@ -12,23 +13,24 @@ import {
type RevisionFabric,
type TypingFabric,
isErrorCode,
type MembershipFabric,
type PresenceFabric,
} from "@relay/protocol";
-import type { Logger } from "@relay/service-kit";
+import { newRequestId, type Logger } from "@relay/service-kit";
import { WebSocketServer, type WebSocket } from "ws";
import { ApiError, type ApiClient } from "./api-client.js";
import { authenticate, type Identity } from "./auth.js";
import {
DEFAULT_HEARTBEAT_MS,
MAX_CONNECTIONS_PER_USER,
type Connections,
} from "./connections.js";
import type { Fanout } from "./fanout.js";
+import type { Decision, GatewayLimits } from "./limits.js";
import { type Membership } from "./membership.js";
import { type Presence } from "./presence.js";
import { Registry, type Connection } from "./registry.js";
import { type Typing } from "./typing.js";
import {
MAX_BUFFERED_FRAMES,
@@ -112,17 +114,64 @@ function isInboundFrame(frame: Frame): frame is Extract<Frame, { type: InboundFr
}
function send(socket: WebSocket, frame: Frame): void {
socket.send(JSON.stringify(frame));
}
-/** EIR-API-04's envelope, wearing its WebSocket clothes. */
+/** EIR-API-04's envelope, wearing its WebSocket clothes.
+ *
+ * `request_id` ARRIVED IN THE RATE-LIMIT CHAPTER, and the gateway had none to give — it
+ * minted no ids at all. The field is required on the frame rather than optional,
+ * because an optional fourth field would have been the fourth instance of the
+ * habit that chapter is about: `rate_limited`, close code 4008 and this field
+ * were all declared in 1.3 and left unenforced (research R13).
+ *
+ * WHAT THE ID IS FOR decides its shape. A developer quoting one in a support
+ * ticket needs it to find a single server-side log line, and on a socket the
+ * useful unit is the frame that failed — a client whose tenth `message.send` was
+ * refused needs to point at that refusal, not at the connection. So callers pass
+ * the id of the frame they are answering, and `sendError` mints one only for a
+ * frame nobody asked for. */
+/** The handshake refusal (FR-RTL-03). Written onto the raw upgrade
+ * socket by hand, because there is no `res` here — `server.on("upgrade")` hands
+ * over the socket and the unparsed head, and anything sent on it has to be a
+ * complete HTTP response including the blank line before the body.
+ *
+ * The same three headers the api sends on a 429, from the same numbers, plus
+ * `Retry-After` — a client should not have to learn a second dialect for the
+ * socket door. `Connection: close` because this socket is not becoming a
+ * WebSocket and is not being kept alive for a second request either. */
+function refuseUpgrade(socket: Duplex, decision: Decision): void {
+ const body = JSON.stringify({
+ code: "rate_limited",
+ message: "too many connections; retry after the window resets",
+ docs_url: "https://relay.example/docs/errors/rate_limited",
+ request_id: newRequestId(),
+ });
+ socket.write(
+ [
+ "HTTP/1.1 429 Too Many Requests",
+ "Content-Type: application/json",
+ `Content-Length: ${Buffer.byteLength(body)}`,
+ `Retry-After: ${decision.retryAfterSeconds}`,
+ `X-RateLimit-Limit: ${decision.limit}`,
+ `X-RateLimit-Remaining: ${decision.remaining}`,
+ `X-RateLimit-Reset: ${decision.resetSeconds}`,
+ "Connection: close",
+ "",
+ body,
+ ].join("\r\n"),
+ );
+ socket.destroy();
+}
+
function sendError(
socket: WebSocket,
code: ErrorCode,
message: string,
+ requestId: string = newRequestId(),
/** WHICH FIELD, on the socket door (FR-005).
*
* `errorFrameSchema` has published this key since chapter 1.3 and no gateway code path
* had ever set it — the same habit `zod-validation.pipe.ts` ended for the api at the
* channel-endpoints chapter, whose comment cites THIS schema while fixing only its own
* side.
@@ -134,12 +183,13 @@ function sendError(
send(socket, {
type: "error",
payload: {
code,
message,
docs_url: docsUrl(code),
+ request_id: requestId,
...(field !== undefined && field.length > 0 ? { field } : {}),
},
});
}
export interface SessionServerOptions {
@@ -202,12 +252,20 @@ export interface SessionServerOptions {
connections?: Connections;
/** Injectable for the reason `meterIntervalMs` and `renewalIntervalMs` are: a
* test that waits out a real minute pays it in the package that paces the lane.
* Defaults to `DEFAULT_HEARTBEAT_MS`, and the tests assert the RATIO to the
* bound rather than either value. */
heartbeatMs?: number;
+ /** The shared counter. Optional for the same reason `fanout`
+ * is: the socket chapter's tests and a single-process dev run have no Redis, and a
+ * socket server that refused to start without one would be a worse default than an
+ * uncounted one. `main.ts` always supplies it, so the optionality is a test
+ * affordance rather than a deployment mode — and the connection cap's note above
+ * says why that is a decision each module has to make for itself rather than a
+ * house style: for a counter, optional means UNCOUNTED. */
+ limits?: GatewayLimits;
}
// THE FOUR PRESENCE TIMINGS ARE NOT HERE, and an earlier draft of this chapter put
// them here. `fanout` and `presence` are INJECTED already built, and an injected thing
// carries its own configuration: a test that wants a hundred-millisecond grace period
// constructs `createPresence({ graceMs: 100, … })` and injects that, the way the
@@ -228,12 +286,13 @@ export function attachSessions({
presence,
membership,
typing,
renewalIntervalMs = DEFAULT_RENEWAL_INTERVAL_MS,
connections,
heartbeatMs = DEFAULT_HEARTBEAT_MS,
+ limits,
}: SessionServerOptions): {
registry: Registry;
/** ASYNC AS OF THIS CHAPTER, and `releaseAll` below is the reason. Freeing the
* places this instance holds is a round trip to Redis that has to COMPLETE before
* `wss.close()`, or the deploy case the method exists for is a race it can lose. */
close: () => Promise<void>;
@@ -650,12 +709,49 @@ export function attachSessions({
const token = url.searchParams.get("token");
void (async () => {
// The api verifies, and answers with the identity AND the
// memberships. This is the same one call the connect path already made —
// it just asks a better question than "what may this user hear".
const result = await authenticate(api, token);
+ // THE ESTABLISHMENT LIMIT IS SPENT HERE, before
+ // `handleUpgrade`, and that placement is the whole difference between
+ // this refusal and the one below it.
+ //
+ // A refusal needs to say WHEN to come back. `Retry-After` is an HTTP
+ // header and a close frame has nowhere to put one — a close code and a
+ // short reason string is all the protocol offers, and "4008, try later"
+ // is not an instruction a client can schedule against. So an over-limit
+ // handshake is refused with an HTTP 429 on the upgrade request, which
+ // still has a response to write headers onto (research R7).
+ //
+ // That makes it deliberately unlike the 4001 path immediately below,
+ // which COMPLETES the handshake in order to close it — because EIR-WS-05
+ // asks for a close code on a bad token, and a close code needs a socket
+ // to arrive on. Two refusals, two shapes, each because of what it has to
+ // carry.
+ //
+ // AFTER authentication, not before: the limit belongs to an environment
+ // and nothing knows which environment this is until the api has said so.
+ // The cost is that an unauthenticated flood still reaches the api — which
+ // is what the auth limiter there is for, and why that one counts by
+ // source address instead.
+ if (result.outcome === "ok" && limits !== undefined) {
+ const decision = await limits.spend(
+ result.identity.environmentId,
+ "connect",
+ result.limits.connect,
+ );
+ if (decision.over) {
+ refuseUpgrade(socket, decision);
+ logger.log("info", "connection.rejected", {
+ reason: "rate_limited",
+ environment_id: result.identity.environmentId,
+ });
+ return;
+ }
+ }
// T037. THE CAP IS CHECKED HERE, after `authenticate` and
// after the establishment limiter, and both orderings are reasons rather
// than habits. After authenticate because the environment and the user are
// not known before it — the comment above says the same of the rate limit.
// After the limiter because a client hammering the door should meet the
// cheaper check first; the limiter is one INCR and this is a walk of up to
@@ -753,21 +849,23 @@ export function attachSessions({
"this user is banned in this environment and cannot connect",
);
ws.close(4003, CLOSE_CODES[4003]);
logger.log("info", "connection.rejected", { reason: "user_banned" });
return;
}
- // NO SEND LIMIT ARGUMENT YET. `authenticate` returns the limits with the
- // session in movement VII, where the limiter is written; until then `open`
- // takes what the session answer actually carries and nothing more.
void open(
ws,
result.identity,
result.channelIds,
result.revisions,
req.url ?? "/",
+ // REQUIRED, SO IT COMES BEFORE THE TWO OPTIONAL ONES. `sendLimit` is a
+ // number and `claimedId` a string, so a wrong order here is a type error
+ // rather than a silent swap — unlike `sendError`'s two `string`s in this
+ // same file, where the compiler had nothing to say.
+ result.limits.send,
pendingId,
claimed,
);
});
})();
});
@@ -779,12 +877,13 @@ export function attachSessions({
/** BESIDE `channelIds` AND NOT AFTER `url`, because it arrives with them from one
* session answer — and because `claimedId` below is optional: gaps.md 045-18 records
* a parameter inserted ahead of an optional one silently renaming every later
* argument. A required parameter here makes the compiler name every call site. */
revisions: Record<string, number>,
url: string,
+ sendLimit: number,
/** The id the cap claimed a place with, so the connection and
* its slot agree — FR-011's "exactly one place for its lifetime". Absent when
* no `connections` module is wired, which is every fixture that does not opt
* in and the reason the cap is not enforced there. */
claimedId?: string,
claimedSlot?: number,
@@ -807,12 +906,13 @@ export function attachSessions({
phase: presented === undefined ? "live" : "buffering",
buffer: [],
overflowed: false,
// A fresh connect suppresses nothing; a resume fills this in when it
// succeeds, and leaves it null when it degrades.
marks: null,
+ sendLimit,
};
registry.add(connection);
// Subscriptions follow membership: the first local member of a channel
// makes this instance a subscriber, and the last one to leave releases
// it (reference-counted in the fabric).
@@ -1316,12 +1416,23 @@ export function attachSessions({
const frame = frameSchema.safeParse(parsed);
if (!frame.success) {
sendError(
connection.socket,
"invalid_frame",
frame.error.issues[0]?.message ?? "frame failed schema validation",
+ // AN EXPLICIT `undefined`, AND TYPESCRIPT COULD NOT ASK FOR IT. `requestId`
+ // was inserted BEFORE `field` in the signature above, and this is the only
+ // call that passed a fourth argument — so the joined path silently became the
+ // `request_id` and the `field` silently disappeared. Both parameters are
+ // `string`, so nothing went red: the typecheck passed, the lint passed, and
+ // the frame carried a zod path where a support ticket expects an id.
+ //
+ // The published order added `field` LAST and never met this. Inserting a
+ // parameter in front of an optional one is a rename of every later argument,
+ // and only reading the call sites finds it.
+ undefined,
// The joined path, which is what a developer reading their own frame sees —
// `payload.attachments.3.kind` rather than "somewhere in this frame".
frame.error.issues[0]?.path.join("."),
);
return;
}
@@ -1343,12 +1454,52 @@ export function attachSessions({
// gateway, Redis, and whoever is subscribed.
if (frame.data.type === "typing.send") {
await signalTyping(connection, frame.data.payload.channel);
return;
}
+ // THE SEND LIMIT IS SPENT ON THE FRAME, not on the api call
+ // it becomes — a socket send and a REST send count against one budget
+ // (FR-RTL-01), or a client could double its allowance by opening a socket.
+ //
+ // AND THE CONNECTION STAYS OPEN. Closing it would be the obvious move and
+ // the wrong one: a closed socket makes the client reconnect, a reconnect
+ // costs a handshake, and a handshake spends the ESTABLISHMENT allowance —
+ // a limiter that punishes the limited into hitting a second limit. The
+ // error frame says no to this frame and nothing more; the next one, after
+ // the window turns over, goes through on the connection that is still there.
+ //
+ // The limit is the one this socket was born with (`connection.sendLimit`),
+ // not one re-read per frame: the gateway has no database, and a Postgres
+ // read on the hot path of the thing the limit protects would be a strange
+ // way to protect it. A policy changed mid-connection reaches the client
+ // when it reconnects (research R12).
+ if (limits !== undefined) {
+ const decision = await limits.spend(
+ connection.identity.environmentId,
+ "send",
+ connection.sendLimit,
+ );
+ if (decision.over) {
+ // `rate_limited` — declared in chapter 1.3, emitted here for the first
+ // time. The numbers a 429 would carry in headers have nowhere to live
+ // on a frame, so the retry window goes in the message text; the code is
+ // what a client branches on.
+ sendError(
+ connection.socket,
+ "rate_limited",
+ `send rate limit exceeded; retry in ${decision.retryAfterSeconds}s`,
+ );
+ logger.log("info", "send.rate_limited", {
+ connection_id: connection.id,
+ environment_id: connection.identity.environmentId,
+ });
+ return;
+ }
+ }
+
// A NAMED DESTRUCTURE, AND THAT IS THE POINT (FR-001). Widening
// `messageSendSchema` puts `attachments` on the wire; without naming it here nothing
// carries it further, the message commits without attachments, and the client is
// acked as though it worked. There is no error anywhere in that sequence.
const { channel, text, idem_key, attachments } = frame.data.payload;
try {@@ -4,12 +4,13 @@ import { createLogger, serve, type Logger } from "@relay/service-kit";
import { createApiClient } from "./api-client.js";
import { createFanout } from "./fanout.js";
import { createMembership } from "./membership.js";
import { createPresence } from "./presence.js";
import { createConnections } from "./connections.js";
import { createTyping } from "./typing.js";
+import { createGatewayLimits } from "./limits.js";
import { attachSessions } from "./session.js";
// The gateway — SAD §4.1: terminates WebSockets and never writes to the
// database (ADR-05). Chapter 1.4 stood up the HTTP half (health, request
// ids, structured logs); chapter 2.5 gives it the job it exists for, and
// 2.6 makes that job survive a second instance. The
@@ -57,12 +58,17 @@ export function createServer(logger?: Logger) {
const typing = createTyping({ logger: log });
// The connection registry's own client. Its keys put the environment id FIRST —
// `conn:{env}:{user}:{slot}` — so a cross-tenant read would need a caller to hand
// this module another environment's id, which the session layer takes from the
// api's verified identity and never from a payload.
const connections = createConnections({ logger: log });
+ // THE NINTH REDIS CLIENT, AND THE FIRST THAT ONLY COUNTS. Not one of fanout's
+ // two: one of those is a subscriber, and a connection in subscribe mode cannot run
+ // `INCR`. Created here rather than inside `attachSessions` so the tests that call
+ // that function directly stay Redis-free, and so its close has an owner.
+ const limits = createGatewayLimits();
const sessions = attachSessions({
server,
api: createApiClient(process.env.RELAY_API_URL ?? DEFAULT_API_URL),
logger: log,
fanout,
presence,
@@ -72,23 +78,25 @@ export function createServer(logger?: Logger) {
// `connections?.claim(...)` is an optional chain on `undefined`, so every socket
// is admitted and no number moves — `**/main.ts` is excluded from the coverage
// ratchet, so no figure could show it. Registering `close()` below is the other
// half and neither substitutes for the other: without this line the cap is inert,
// without that one every gateway leaks a Redis client.
connections,
+ limits,
});
server.on("close", () => {
// `void`, LIKE ITS SIBLINGS. `sessions.close()` returns a promise as of the
// connection cap — it frees the places this instance holds before closing the
// socket server — and `server.on("close")` has nowhere to await one.
void sessions.close();
void fanout.close();
void presence.close();
void membership.close();
void typing.close();
void connections.close();
+ void limits.close();
});
return server;
}
if (import.meta.main) {
const requested = Number(process.env.PORT ?? 4001);The chapter's own suites first. The api's limiter is driven over real HTTP against the compose Postgres and Redis, and every send in it names a bot: an application credential carries no user of its own, so the sender is seeded once through the repository rather than per send — this suite counts requests, and an extra call per send moves every number in it while leaving the assertions green.
import "reflect-metadata";
import type { INestApplication } from "@nestjs/common";
import { Test } from "@nestjs/testing";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { AppModule } from "../app.module";
import { createDb, createPool, type Db } from "../db/client";
import { migrate } from "../db/migrate";
import {
createApiKey,
createEnvironment,
environmentSigningSecret,
Repository,
} from "../db/repository";
import { mintUserToken } from "../auth/user-token";
import { COUNTER_STORE } from "./limits.module";
import { createCounterStore, type CounterStore } from "./store";
import { environments } from "../db/schema";
import { eq } from "drizzle-orm";
// The limiter over real HTTP against the compose Postgres and Redis
// Each `it` mints what it needs; the suite's own environments
// keep it out of every other suite's way — 2.1's isolation property paying for
// itself again.
const HEADERS = {
limit: "x-ratelimit-limit",
remaining: "x-ratelimit-remaining",
reset: "x-ratelimit-reset",
} as const;
describe("the limiter", () => {
let app: INestApplication;
let url: string;
let db: Db;
/** An environment with a channel and a key, at whatever limits the caller
* asks for. `undefined` leaves the column null, which means "use the
* documented default" — and null is not zero. */
const seed = async (name: string, limits?: { rest?: number; send?: number }) => {
const env = await createEnvironment(db, { name });
if (limits) {
await db
.update(environments)
.set({
restLimitPerMinute: limits.rest ?? null,
sendLimitPerMinute: limits.send ?? null,
})
.where(eq(environments.id, env.id));
}
const repo = new Repository(db, env.id);
const channel = await repo.createChannel("c", "public");
// A BOT SENDER, SEEDED HERE AND NOT IN `send`. An application credential
// carries no user of its own, so FR-MSG-15 makes the body name one — and it
// must be a BOT: a person named by a key is refused 403, an unknown name 400
// (`the sender named in \`user\` is not a user of this environment`). Neither is
// a 429, so a fixture short of either turns every assertion in this file into
// "the limiter never ran".
//
// SEEDED ONCE, THROUGH THE REPOSITORY, BECAUSE THIS SUITE COUNTS REQUESTS.
// Upserting the bot inside the send helper would add an HTTP call per send,
// which moves every number below — and the assertions would still pass, for
// the wrong reason.
//
// NO `addMember`, AND THAT WAS MEASURED RATHER THAN ASSUMED. The first draft
// added one on the theory that a non-member send is refused; removing it left
// the suite green at 18 of 18, so the row was a claim this fixture does not
// need to make.
await repo.upsertUser("meter", {
display_name: "Meter",
kind: "bot",
description: "spends this suite's allowance",
});
const { credential } = await createApiKey(db, { environmentId: env.id });
return { env, channelId: channel.id, credential };
};
/** NAMES A SENDER, AND PUBLISHED'S VERSION OF THIS SUITE DID NOT HAVE TO.
* `createApiKey` mints an APPLICATION credential, which carries no user of its
* own, and FR-MSG-15 makes the send body name one — so `{ text }` alone is a 400
* here and every assertion about a 201's headers reads `expected 400 to be 201`.
* In the published order the sender chapter came LATER than the limiter; in this
* one it is upstream, and a fixture is the first thing a reorder invalidates. */
const send = (
channelId: string,
credential: string,
text = "one",
): Promise<Response> =>
fetch(`${url}/v1/channels/${channelId}/messages`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${credential}`,
},
body: JSON.stringify({ text, user: "meter" }),
});
beforeAll(async () => {
const pool = createPool();
// Migrations here, like every other suite that needs a schema newer than
// whatever the database happens to be at. The rate-limit chapter's `0008` adds the
// policy columns, and a suite that assumed they existed would pass on a
// developer's machine and fail on a fresh one.
await migrate(pool);
db = createDb(pool);
app = (
await Test.createTestingModule({ imports: [AppModule] }).compile()
).createNestApplication({ logger: false });
await app.listen(0);
url = await app.getUrl();
}, 60_000);
afterAll(async () => {
await app.close();
});
it("carries all three headers on a SUCCESSFUL response", async () => {
// FR-RTL-02, and the requirement an afterthought passes: a limiter that only
// speaks when it refuses satisfies every test written from the 429.
const { channelId, credential } = await seed("limits-headers");
const res = await send(channelId, credential);
expect(res.status).toBe(201);
expect(res.headers.get(HEADERS.limit)).toBe("600");
expect(res.headers.get(HEADERS.remaining)).toBe("599");
expect(Number(res.headers.get(HEADERS.reset))).toBeGreaterThan(
Math.floor(Date.now() / 1000),
);
});
it("counts down across successive responses", async () => {
const { channelId, credential } = await seed("limits-countdown");
const first = await send(channelId, credential);
const second = await send(channelId, credential);
expect(Number(first.headers.get(HEADERS.remaining))).toBe(599);
expect(Number(second.headers.get(HEADERS.remaining))).toBe(598);
});
it("refuses with 429, Retry-After, and a four-field body", async () => {
// Two requests allowed, the third refused. Testing a threshold means
// lowering it — driving 600 requests would measure the test runner.
const { channelId, credential } = await seed("limits-refusal", { rest: 2 });
await send(channelId, credential);
await send(channelId, credential);
const refused = await send(channelId, credential);
expect(refused.status).toBe(429);
expect(Number(refused.headers.get("retry-after"))).toBeGreaterThan(0);
expect(refused.headers.get(HEADERS.remaining)).toBe("0");
const body = (await refused.json()) as Record<string, unknown>;
// TOP-LEVEL, not nested under an `error` key. EIR-API-04's example showed it
// nested until this chapter checked what the platform actually sends; the SRS
// is amended to 1.3 and the flat shape is the documented one. This assertion
// is what stops the next reader of that requirement from "fixing" the code.
expect(body["code"]).toBe("rate_limited");
expect(typeof body["message"]).toBe("string");
expect(typeof body["docs_url"]).toBe("string");
expect(typeof body["request_id"]).toBe("string");
});
it("counts a REST send against BOTH budgets, and reports the nearer", async () => {
// FR-RTL-01. The send limit counts messages wherever they enter, so a REST send
// spends one of each. With `send` set lower than `rest`, the headers must
// follow `send` — the one that will refuse first is the only value a client
// can schedule against (research R11).
const { channelId, credential } = await seed("limits-both", {
rest: 100,
send: 3,
});
const first = await send(channelId, credential);
expect(first.headers.get(HEADERS.limit)).toBe("3");
expect(first.headers.get(HEADERS.remaining)).toBe("2");
});
it("names WHICH limit was reached, because they are different problems", async () => {
// "too many requests" says batch; "too many messages" says slow down. The
// code stays `rate_limited` — it is the protocol constant — and the message
// carries the distinction. Neither names a credential (NFR-SEC-06).
const { channelId, credential } = await seed("limits-which", {
rest: 100,
send: 1,
});
await send(channelId, credential);
const refused = await send(channelId, credential);
const body = (await refused.json()) as { message: string };
expect(refused.status).toBe(429);
expect(body.message).toContain("messages");
expect(body.message).not.toContain(credential);
});
it("describes the budget that REFUSED, not the one with fewest remaining", async () => {
// Both budgets can hit zero remaining in the same request while only one of
// them is over. Three sends against `rest: 3` and `send: 2` leave rest at
// 3/3 — spent but not over — and send at 3/2, which is. "Fewest remaining"
// ties at zero and picks rest, so the refusal used to answer
// `X-RateLimit-Limit: 3` above a body reading "too many messages".
//
// A client cannot act on that. It reads the header, sets its rate to 3 a
// minute, and is refused again at 2. Found by capturing the transcript for
// the chapter, not by a test — which is the argument for capturing them
// (research R41).
const { channelId, credential } = await seed("refusal-headers", {
rest: 3,
send: 2,
});
await send(channelId, credential, "one");
await send(channelId, credential, "two");
const refused = await send(channelId, credential, "three");
expect(refused.status).toBe(429);
expect(refused.headers.get(HEADERS.limit)).toBe("2");
expect(((await refused.json()) as { message: string }).message).toMatch(
/too many messages/,
);
});
it("never limits /healthz, whatever the environment has spent", async () => {
// Docker polls it every five seconds and `up -d --wait` depends on the
// answer. A limiter that can refuse a health check can stop a deployment.
const { channelId, credential } = await seed("limits-health", { rest: 1 });
await send(channelId, credential);
expect((await send(channelId, credential)).status).toBe(429);
const health = await fetch(`${url}/healthz`);
expect(health.status).toBe(200);
expect(health.headers.get(HEADERS.limit)).toBeNull();
});
it("does not count the gateway's internal routes as requests", async () => {
// THE CASE A PRINCIPAL-BASED EXEMPTION MISSES. The gateway forwards the END
// USER's token on `/internal/session`, `/internal/backfill` and
// `/internal/messages`, all `@Accepts("user")` — so its calls resolve exactly
// like customer traffic and only the route can tell them apart.
//
// Counting them again would charge a socket send twice and let a reconnect
// storm eat a customer's request budget (research R17).
const env = await createEnvironment(db, { name: "limits-internal" });
await db
.update(environments)
.set({ restLimitPerMinute: 1 })
.where(eq(environments.id, env.id));
const repo = new Repository(db, env.id);
const user = await repo.createUser("tuan", "Tuan");
const channel = await repo.createChannel("fleet", "public");
await repo.addMember(channel.id, user.id);
const secret = (await environmentSigningSecret(db, env.id))!.signingSecret;
const { token } = await mintUserToken(secret, {
user: "tuan",
environmentId: env.id,
ttlSeconds: 3600,
});
// Spend the environment's single request slot on the public path.
const { credential } = await createApiKey(db, { environmentId: env.id });
await send(channel.id, credential);
expect((await send(channel.id, credential)).status).toBe(429);
// The gateway's door is a different door, and it is still open.
const session = await fetch(`${url}/internal/session`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${token}`,
},
});
expect(session.status).toBe(200);
expect(session.headers.get(HEADERS.limit)).toBeNull();
});
it("does not count the dispatcher, which reaches every environment", async () => {
// A limiter that throttles the dispatcher turns one busy customer's webhook
// backlog into a stall for every customer — which is FR-WHK-05's failure
// (*"delivery shall never delay or block message delivery"*) reached from a
// direction that clause does not mention, and what the webhook dispatcher
// chapter's retry schedule was built to avoid.
//
// Unlike the gateway, this one IS recognisable by principal: the platform
// credential belongs to a deployment rather than a tenant, so it carries no
// environment to key on. Both are exempt; only one of them could have been
// exempted by looking at who was asking.
const credentialEnv = process.env["RELAY_INTERNAL_CREDENTIAL"];
expect(credentialEnv, "the lane must configure a platform credential").toBeTruthy();
const res = await fetch(`${url}/internal/dispatch/material`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${credentialEnv ?? ""}`,
},
body: JSON.stringify({ delivery_id: "00000000-0000-0000-0000-000000000000" }),
});
// Whatever it answers about a delivery that does not exist, it is not a 429
// and it carries no allowance headers — the route was never counted.
expect(res.status).not.toBe(429);
expect(res.headers.get(HEADERS.limit)).toBeNull();
});
it("an override applies to ONE environment, not to every environment", async () => {
// FR-RTL-04's configurability, and the half the journey map needed: it asks
// for "separate keys and separate quotas", so Mai can hammer her dev
// environment without moving production's ceiling. Independent counters
// would pass a weaker version of this test.
const tight = await seed("limits-tight", { rest: 2 });
const loose = await seed("limits-loose");
await send(tight.channelId, tight.credential);
await send(tight.channelId, tight.credential);
const refused = await send(tight.channelId, tight.credential);
const other = await send(loose.channelId, loose.credential);
expect(refused.status).toBe(429);
expect(other.status).toBe(201);
expect(other.headers.get(HEADERS.limit)).toBe("600");
});
it("carries the fourth field on EVERY error, not only the 429", async () => {
// Constitution V has asked for `code`, `message`, `docs_url` and
// `request_id` since chapter 1.3. The platform sent three for twenty-two
// chapters, above a comment promising the fourth would "join in Part 2, when
// a gateway exists to mint one".
//
// Four fields on one status and three on the others would be worse than
// either consistent answer, so this checks the statuses nobody was thinking
// about when the rate limiter was specified.
const notFound = await fetch(`${url}/v1/nope`);
const unauthorized = await fetch(`${url}/v1/channels/x/messages`, {
method: "POST",
headers: { "content-type": "application/json" },
body: "{}",
});
for (const res of [notFound, unauthorized]) {
const body = (await res.json()) as Record<string, unknown>;
// TOP-LEVEL, not nested under an `error` key — EIR-API-04's example showed
// it nested until this chapter checked, and the SRS is amended to 1.3.
expect(body["error"]).toBeUndefined();
expect(typeof body["code"]).toBe("string");
expect(typeof body["message"]).toBe("string");
expect(typeof body["docs_url"]).toBe("string");
expect(body["request_id"]).toBe(res.headers.get("x-request-id"));
}
});
it("refuses an address past the failed-auth threshold, and says nothing extra", async () => {
// FR-AUT-12 and EIR-API-04. Past the threshold the refusal must be
// INDISTINGUISHABLE from a wrong-credential refusal — a limiter that answers
// differently for a credential it would have accepted is an oracle.
//
// Its own key prefix, because the lane runs files in parallel and every
// suite asserting a 401 from loopback lands in one bucket. A suite that
// needs a LOW threshold needs a private key, not a private number
// (research R21).
const previousPrefix = process.env["RELAY_AUTH_KEY_PREFIX"];
const previousThreshold = process.env["RELAY_AUTH_FAILURES_PER_MINUTE"];
process.env["RELAY_AUTH_KEY_PREFIX"] = `rlauth-itest-${Date.now()}`;
process.env["RELAY_AUTH_FAILURES_PER_MINUTE"] = "3";
try {
const bad = () =>
fetch(`${url}/v1/channels/x/messages`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: "Bearer rk_live_not_a_real_key_at_all_000000",
},
body: JSON.stringify({ text: "hi" }),
});
const first = await bad();
expect(first.status).toBe(401);
await bad();
await bad();
const refused = await bad();
expect(refused.status).toBe(429);
const body = (await refused.json()) as { code: string; message: string };
expect(body.code).toBe("rate_limited");
// Never the credential (NFR-SEC-06), and no hint about whether it was
// valid.
expect(body.message).not.toContain("rk_live");
expect(body.message).not.toContain("valid");
} finally {
if (previousPrefix === undefined) delete process.env["RELAY_AUTH_KEY_PREFIX"];
else process.env["RELAY_AUTH_KEY_PREFIX"] = previousPrefix;
if (previousThreshold === undefined)
delete process.env["RELAY_AUTH_FAILURES_PER_MINUTE"];
else process.env["RELAY_AUTH_FAILURES_PER_MINUTE"] = previousThreshold;
}
});
it("counts ten client addresses as ten, not as one gateway", async () => {
// FR-AUT-12. A handshake authenticated through the gateway reaches the api FROM
// the gateway, so counting the TCP peer would put every customer's failures
// in one bucket and let one attacker exhaust a threshold that then refused
// everybody.
//
// The address rides the internal contract as a field rather than a header,
// because a header the caller asserts is a header the caller can forge —
// The credentials chapter removed exactly that pattern.
const previousPrefix = process.env["RELAY_AUTH_KEY_PREFIX"];
const previousThreshold = process.env["RELAY_AUTH_FAILURES_PER_MINUTE"];
process.env["RELAY_AUTH_KEY_PREFIX"] = `rlauth-fleet-${Date.now()}`;
process.env["RELAY_AUTH_FAILURES_PER_MINUTE"] = "3";
try {
// Ten distinct clients, one bad handshake each, all arriving from this
// process — which is what the gateway looks like to the api.
for (let i = 0; i < 10; i++) {
const res = await fetch(`${url}/internal/session`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: "Bearer not.a.real.token",
},
body: JSON.stringify({ client_address: `198.51.100.${i}` }),
});
// Every one is a 401: ten addresses, one failure each, none over three.
expect(res.status, `client ${i} should be refused for its credential`).toBe(
401,
);
}
} finally {
if (previousPrefix === undefined) delete process.env["RELAY_AUTH_KEY_PREFIX"];
else process.env["RELAY_AUTH_KEY_PREFIX"] = previousPrefix;
if (previousThreshold === undefined)
delete process.env["RELAY_AUTH_FAILURES_PER_MINUTE"];
else process.env["RELAY_AUTH_FAILURES_PER_MINUTE"] = previousThreshold;
}
});
it("limits account creation per address, which has no tenant to key on", async () => {
// FR-AUT-12. Signup has no environment — that is the point of it — and no
// guard, so neither the per-environment limiter nor `CredentialGuard`'s
// refusal reaches it. An unlimited account-creation route in a platform that
// limits everything else is a gap a reader notices.
const previousPrefix = process.env["RELAY_AUTH_KEY_PREFIX"];
const previousThreshold = process.env["RELAY_AUTH_FAILURES_PER_MINUTE"];
process.env["RELAY_AUTH_KEY_PREFIX"] = `rlauth-signup-${Date.now()}`;
process.env["RELAY_AUTH_FAILURES_PER_MINUTE"] = "2";
try {
const start = () =>
fetch(`${url}/auth/github/start`, { redirect: "manual" });
// Two allowed, whatever they answer — an unconfigured provider is a 404 and
// that is not what is under test here.
expect((await start()).status).not.toBe(429);
expect((await start()).status).not.toBe(429);
expect((await start()).status).toBe(429);
} finally {
if (previousPrefix === undefined) delete process.env["RELAY_AUTH_KEY_PREFIX"];
else process.env["RELAY_AUTH_KEY_PREFIX"] = previousPrefix;
if (previousThreshold === undefined)
delete process.env["RELAY_AUTH_FAILURES_PER_MINUTE"];
else process.env["RELAY_AUTH_FAILURES_PER_MINUTE"] = previousThreshold;
}
});
it("two environments carry DIFFERENT configured limits, each at its own number", async () => {
// Independent counters are half of what the journey map asks for. The other
// half is "separate keys and separate quotas": a developer raises her dev
// environment's ceiling and hammers it without moving production's.
//
// The previous test proves an override applies. This one proves it applies to
// ONE environment — a shared policy would pass the first and fail here.
const dev = await seed("limits-dev", { rest: 5 });
const prod = await seed("limits-prod", { rest: 2 });
const devFirst = await send(dev.channelId, dev.credential);
const prodFirst = await send(prod.channelId, prod.credential);
expect(devFirst.headers.get(HEADERS.limit)).toBe("5");
expect(prodFirst.headers.get(HEADERS.limit)).toBe("2");
// Production runs out at two; development still has room at three.
await send(prod.channelId, prod.credential);
expect((await send(prod.channelId, prod.credential)).status).toBe(429);
expect((await send(dev.channelId, dev.credential)).status).toBe(201);
});
});
// The failure direction, which is what this chapter is actually about
// (SAD §6.3, FR-AUT-12, FR-RTL-02, research R3, R6).
//
// A REAL ioredis client against a dead port, not a mock that throws. The
// question is what the platform does when a store it depends on is gone, and a
// stub that rejects on command would answer a different question — it would skip
// connection handling, which is where the first draft of `store.ts` got it wrong.
//
// A dead port rather than stopping the container, because the lane runs files in
// PARALLEL and stopping Redis would break every other suite mid-run.
describe("when the counter store is gone", () => {
let app: INestApplication;
let url: string;
let db: Db;
let channelId: string;
let credential: string;
let deadStore: CounterStore;
beforeAll(async () => {
const pool = createPool();
await migrate(pool);
db = createDb(pool);
const env = await createEnvironment(db, { name: "limits-degraded" });
const repo = new Repository(db, env.id);
channelId = (await repo.createChannel("c", "public")).id;
// Its own bot, because this block seeds its own environment rather than
// calling `seed()`. Same name, so the sends below read the same as the
// others, and the reason is the note on that fixture: the test under this one
// asserts a 201, and a 4xx for a missing or non-bot sender would turn "the
// limiter failed OPEN" into "the request was refused for something else".
await repo.upsertUser("meter", {
display_name: "Meter",
kind: "bot",
description: "spends this suite's allowance while the store is gone",
});
credential = (await createApiKey(db, { environmentId: env.id })).credential;
// Port 1 is reserved and nothing listens on it.
deadStore = createCounterStore("redis://127.0.0.1:1");
app = (
await Test.createTestingModule({ imports: [AppModule] })
.overrideProvider(COUNTER_STORE)
.useValue(deadStore)
.compile()
).createNestApplication({ logger: false });
await app.listen(0);
url = await app.getUrl();
}, 60_000);
afterAll(async () => {
await app.close();
await deadStore.close();
});
it("SERVES the request rather than refusing it", async () => {
// Redis is not a source of truth (SAD §6.3), and a cache outage is not a
// reason to refuse a paying customer's traffic. This is the direction the
// tenant limiter fails in, and it is a decision rather than an accident.
const res = await fetch(`${url}/v1/channels/${channelId}/messages`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${credential}`,
},
body: JSON.stringify({ text: "served anyway", user: "meter" }),
});
expect(res.status).toBe(201);
});
it("keeps Limit and DROPS Remaining and Reset, rather than inventing them", async () => {
// FR-RTL-02. `Limit` is policy read from Postgres and is not degraded. The other
// two exist only because something was counting, and a client must be able to
// tell "you have N left" from "we are not counting".
//
// NOT a sentinel: a client that does not know `-1` would parse it as a number
// and conclude it was over its limit (research R6).
const res = await fetch(`${url}/v1/channels/${channelId}/messages`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${credential}`,
},
body: JSON.stringify({ text: "no counts", user: "meter" }),
});
expect(res.headers.get(HEADERS.limit)).toBe("600");
expect(res.headers.get(HEADERS.remaining)).toBeNull();
expect(res.headers.get(HEADERS.reset)).toBeNull();
});
it("does NOT let an address spend an unlimited number of failed logins", async () => {
// The other direction, in the same outage, and the reason the chapter exists.
// Failing open here is not a degradation — it is a brute-force window. The
// in-process fallback holds the same threshold per instance, so the guarantee
// weakens from N per window per fleet to N per window per instance rather
// than disappearing.
const previous = process.env["RELAY_AUTH_FAILURES_PER_MINUTE"];
process.env["RELAY_AUTH_FAILURES_PER_MINUTE"] = "3";
try {
const bad = () =>
fetch(`${url}/v1/channels/${channelId}/messages`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: "Bearer rk_live_still_not_a_real_key_0000000",
},
body: JSON.stringify({ text: "hi" }),
});
expect((await bad()).status).toBe(401);
await bad();
await bad();
expect((await bad()).status).toBe(429);
} finally {
if (previous === undefined)
delete process.env["RELAY_AUTH_FAILURES_PER_MINUTE"];
else process.env["RELAY_AUTH_FAILURES_PER_MINUTE"] = previous;
}
});
});import { readFile } from "node:fs/promises";
import { describe, expect, it } from "vitest";
import { createGatewayLimits, decide, overLimit, windowStartFor } from "./limits.js";
// The gateway's share of the arithmetic. Pure, so a window
// boundary is a test rather than a wait.
const MINUTE = 60_000;
describe("windowStartFor", () => {
it("floors to the window, so two instances agree without talking", () => {
expect(windowStartFor(90_000, MINUTE)).toBe(60_000);
expect(windowStartFor(59_999, MINUTE)).toBe(0);
});
});
describe("overLimit", () => {
it("is false below the limit and true at it", () => {
expect(overLimit(599, 600)).toBe(false);
expect(overLimit(600, 600)).toBe(false);
expect(overLimit(601, 600)).toBe(true);
});
it("TREATS AN UNKNOWN COUNT AS UNDER THE LIMIT", () => {
// `null` means the store could not be reached. The gateway's two limits are
// TENANT limits — they protect Relay's capacity from a customer's traffic —
// so they fail open, exactly as the api's do. This is the direction that is
// right here and wrong for the auth limiter, which is the chapter's whole
// argument (research R3).
expect(overLimit(null, 600)).toBe(false);
});
it("a limit of zero refuses everything, and that is expressible on purpose", () => {
// Null in the policy column means "use the default"; zero means "refuse
// everything", and an environment can be switched off deliberately. The two
// states cannot share a representation, so zero has to behave.
expect(overLimit(1, 0)).toBe(true);
});
});
describe("decide", () => {
const at = (ms: number, count: number | null, limit = 600) =>
decide(count, limit, ms, MINUTE);
it("reports the reset as one moment, not a curve", () => {
// The whole reason this is a fixed window and not a token bucket: a
// refilling bucket's honest answer to "when may I retry" is a slope, and
// `X-RateLimit-Reset` has room for one number.
expect(at(90_000, 601).resetSeconds).toBe(120);
expect(at(119_999, 601).resetSeconds).toBe(120);
});
it("never asks a client to retry in zero seconds", () => {
// 1ms before the boundary the honest answer rounds to 0, and a client that
// obeys it retries into the same refusal.
expect(at(119_999, 601).retryAfterSeconds).toBe(1);
expect(at(60_000, 601).retryAfterSeconds).toBe(60);
});
it("floors remaining at zero rather than going negative", () => {
expect(at(0, 605).remaining).toBe(0);
expect(at(0, 599).remaining).toBe(1);
});
it("reports a full allowance when the store is unreachable", () => {
// Fail-open all the way through: not over, and the headers say so rather
// than reporting a count the gateway does not have.
expect(at(0, null)).toMatchObject({ over: false, remaining: 600 });
});
});
describe("the gateway's dependencies (ADR-05)", () => {
it("gains no database client, which is the property R12 exists to protect", async () => {
// ADR-05: the gateway never touches Postgres. The rate-limit chapter needed the
// environment's limits, which live in Postgres, and the tempting fix was a
// read-only pool "just for this". R12 spent its whole argument on why not —
// and then nothing checked it, which is how a design statement becomes a
// comment. The limits ride the authentication response the gateway was
// already making, so this file's manifest can stay clean.
const manifest = JSON.parse(
await readFile(new URL("../package.json", import.meta.url), "utf8"),
) as { dependencies: Record<string, string> };
const runtime = Object.keys(manifest.dependencies);
for (const forbidden of ["pg", "postgres", "drizzle-orm", "@relay/db"]) {
expect(runtime).not.toContain(forbidden);
}
// `ioredis` IS here, and that is a different claim: Redis is a cache the
// gateway may hold, Postgres is the source of truth it may not (SAD §6.3).
expect(runtime).toContain("ioredis");
// `@relay/api` sits in devDependencies for the integration harness, and it
// does bring Postgres with it — into the test process, never into `dist`.
expect(runtime).not.toContain("@relay/api");
});
});
describe("a store that is not there", () => {
it("FAILS OPEN, and stops paying the connect timeout on every call", async () => {
// Port 1 answers nothing. The first `spend` waits out the connect timeout
// and fails open; the second must NOT — a cache outage that turns every
// handshake into a one-second wait has converted an unavailable limiter
// into an unavailable gateway, and NFR-PRF-04 asks for a handshake under a
// second (research R34).
const limits = createGatewayLimits("redis://127.0.0.1:1");
try {
const first = await limits.spend("env-1", "connect", 600);
expect(first.over).toBe(false);
expect(first.remaining).toBe(600);
const started = Date.now();
const second = await limits.spend("env-1", "connect", 600);
expect(second.over).toBe(false);
// Not "fast enough to feel nice" — fast enough that it cannot have made
// a connection attempt, whose timeout is a full second.
expect(Date.now() - started).toBeLessThan(100);
} finally {
await limits.close();
}
}, 15_000);
});Two earlier suites raise the failed-authentication threshold on purpose, and both
now take a counter bucket of their own. Raising the threshold is private to a vitest
worker; the Redis key is not. A suite that raises its ceiling and keeps the
default prefix pushes a shared count up while being personally immune to it — 8 from
one and 13 from the other against a threshold of 10, refused by nothing, and only
because the suites that spawn a child reach the api over ::ffff:127.0.0.1 while
the in-process one reaches it over ::1. Two spellings of localhost were the whole
of the isolation.
@@ -15,12 +15,13 @@ import {
provisionOrganisation,
Repository,
revokeApiKey,
} from "../db/repository";
import { parseApiKeyCredential } from "./api-key";
import { MAX_TOKEN_LIFETIME_SECONDS } from "./user-token";
+import { withoutRequestId } from "../isolation/compare";
// The refusals, over real HTTP against the compose Postgres.
// Invariants 1-7, 9 and 11 of contracts/credentials.md live here; 8 and 12 are
// pure and live in the unit lane; 10 needs a socket and lives in the gateway's
// session.itest.ts.
//
@@ -84,12 +85,37 @@ describe("credentials", () => {
.setIssuedAt(over.iat ?? now)
.setExpirationTime(over.exp ?? now + 3600)
.sign(new TextEncoder().encode(secret));
};
beforeAll(async () => {
+ // This suite submits bad credentials ON PURPOSE — that is what
+ // it is for — and the failed-authentication limiter counts them all against
+ // one loopback address. The default is ten a minute.
+ //
+ // RAISING WORKS HOWEVER POLLUTED THE SHARED COUNT, which is why this is a
+ // threshold and not a private key: the integration lane runs files in
+ // parallel, every suite asserting a `401` lands in the same bucket, and a
+ // high ceiling never refuses. A suite needing a LOW threshold needs its own
+ // key instead — see `limits.itest.ts` (research R21).
+ //
+ // Explicit and visible, rather than the default being chosen to suit the
+ // tests. The retry-and-disable chapter's `RELAY_DISABLE_SWEEP` states the rule: a flag whose
+ // default disabled a requirement would be a requirement nobody had built.
+ process.env["RELAY_AUTH_FAILURES_PER_MINUTE"] = "10000";
+ // AND ITS OWN BUCKET. Raising the threshold is private to this worker —
+ // vitest gives each file its own process — but the Redis key is not, so a
+ // suite that raises its ceiling and keeps the default prefix pushes a SHARED
+ // count up while being personally immune to it. T004a measured this file's
+ // contribution to the default bucket at 8 and signup's at 13, against a
+ // threshold of 10: nothing was refused, and only because the suites that
+ // spawn a child reach the api over `::ffff:127.0.0.1` while this one reaches
+ // it in-process over `::1`. Two address formats were the whole of the
+ // isolation. Now it is a prefix, which is a decision rather than an accident.
+ process.env["RELAY_AUTH_KEY_PREFIX"] =
+ `rlauth-credentials-${Date.now()}`;
db = createDb(createPool());
env = await createEnvironment(db, { name: "credentials-itest" });
key = await createApiKey(db, { environmentId: env.id });
const repo = new Repository(db, env.id);
channelId = (await repo.createChannel("general", "public")).id;
@@ -214,13 +240,15 @@ describe("credentials", () => {
{ text: "nowhere", user: "cred-bot" },
key.credential,
"00000000-0000-0000-0000-000000000000",
);
expect(foreignAnswer.status).toBe(404);
expect(absentAnswer.status).toBe(404);
- expect(await foreignAnswer.json()).toEqual(await absentAnswer.json());
+ expect(withoutRequestId(await foreignAnswer.json())).toEqual(
+ withoutRequestId(await absentAnswer.json()),
+ );
// And the reverse direction, so the test cannot pass by both being broken.
expect(
(await post({ text: "mine", user: "cred-bot" }, foreignKey.credential, foreignChannelId))
.status,
).toBe(201);@@ -46,12 +46,32 @@ describe("signup", () => {
let app: INestApplication;
let url: string;
let db: ReturnType<typeof createDb>;
let provider: Awaited<ReturnType<typeof standInProvider>>;
beforeAll(async () => {
+ // The rate-limit chapter limited account creation per source address (FR-AUT-12), and this
+ // suite drives the signup routes repeatedly from one loopback address — which
+ // is what a suite about signup does.
+ //
+ // Raised explicitly and visibly, rather than the default being chosen to suit
+ // the tests. The same move `credentials.itest.ts` makes for the
+ // failed-authentication threshold, and for the same reason: raising survives a
+ // shared count, lowering does not (research R21).
+ process.env["RELAY_AUTH_FAILURES_PER_MINUTE"] = "10000";
+ // AND ITS OWN BUCKET. Raising the threshold is private to this worker —
+ // vitest gives each file its own process — but the Redis key is not, so a
+ // suite that raises its ceiling and keeps the default prefix pushes a SHARED
+ // count up while being personally immune to it. T004a measured the credentials
+ // suite's contribution to the default bucket at 8 and THIS file's at 13, against
+ // a threshold of 10: nothing was refused, and only because the suites that
+ // spawn a child reach the api over `::ffff:127.0.0.1` while this one reaches
+ // it in-process over `::1`. Two address formats were the whole of the
+ // isolation. Now it is a prefix, which is a decision rather than an accident.
+ process.env["RELAY_AUTH_KEY_PREFIX"] =
+ `rlauth-signup-${Date.now()}`;
db = createDb(createPool());
provider = await standInProvider({
id: 90210,
login: "tuan",
name: "Tuan",
});Two gateway suites forge one frame per union member to prove the seam refuses an
outbound type by DIRECTION, and both carried a note explaining why the error
sample omits request_id. Every clause of it was true and the conclusion inverted
the moment the field became required: the same strictObject that refused an extra
field now refuses its absence, and nine direction assertions came back
invalid_frame. Which fields make a sample valid is a fact about the schema on the
day, not a rule to be stated once.
@@ -826,19 +826,28 @@ function sample(type: string, channel: string, user: string): unknown {
return { type, payload: { channel, user, change: "added" } };
case "presence.changed":
return { type, payload: { user, state: "online" } };
case "typing":
return { type, payload: { channel, user } };
case "error":
- // NO `request_id`. The payload is a `strictObject`, so an extra field is refused
- // as `invalid_frame` — and this loop asserts `unknown_frame_type`, which is a
- // claim about DIRECTION. A sample that fails validation tests the validator
- // instead, and the assertion then passes or fails for the wrong reason.
+ // A `request_id`, AND THIS COMMENT USED TO SAY *"NO `request_id`"* — for the
+ // right reason, until the field existed. The payload is a `strictObject`, so it
+ // refused an extra field then and refuses a MISSING one now that the limits
+ // chapter made `request_id` required. Either way the sample must be exactly
+ // what the schema of the day accepts, because this loop asserts
+ // `unknown_frame_type`, a claim about DIRECTION: a sample that fails validation
+ // tests the validator instead, and the assertion then passes or fails for the
+ // wrong reason. `session.itest.ts`'s builder carries the same correction.
return {
type,
- payload: { code: "forged", message: "forged", docs_url: "/x" },
+ payload: {
+ code: "forged",
+ message: "forged",
+ docs_url: "/x",
+ request_id: randomUUID(),
+ },
};
default:
return { type, payload: { idem_key: randomUUID(), channel, text: "forged" } };
}
}@@ -1249,22 +1249,31 @@ describe("the socket's delivery, with a fan-out attached", () => {
return { type, payload: { channel, user: "tuan", change: "added" } };
case "presence.changed":
return { type, payload: { user: "tuan", state: "online" } };
case "typing":
return { type, payload: { channel, user: "tuan" } };
default:
- // NO `request_id` IN THE ERROR SAMPLE. The payload is a `strictObject`, so an
- // extra field is refused as `invalid_frame` — and this loop asserts
- // `unknown_frame_type`, which is a claim about DIRECTION. A sample that fails
- // validation tests the validator instead.
+ // A `request_id` IN THE ERROR SAMPLE, AND THIS COMMENT USED TO SAY THE OPPOSITE.
+ // It read *"NO `request_id` IN THE ERROR SAMPLE — the payload is a `strictObject`,
+ // so an extra field is refused as `invalid_frame`"*, and it was right for exactly
+ // as long as the field did not exist. The limits chapter makes it REQUIRED, so
+ // the same `strictObject` now refuses the sample for its ABSENCE, and the loop's
+ // nine direction assertions came back `invalid_frame` — the failure this
+ // builder's own header warns about, arriving from the other side.
+ //
+ // The unchanged half is the reason: this loop asserts `unknown_frame_type`, a
+ // claim about DIRECTION, and a sample that fails validation tests the validator
+ // instead. Which fields make a sample valid is a fact about the schema on the
+ // day, not a rule to be stated once.
return {
type,
payload: {
code: "forged",
message: "forged",
docs_url: "/x",
+ request_id: randomUUID(),
},
};
}
};
/** T034 — T009 INVERTED, and the same shape on purpose.And the dispatcher's drain gets a batch large enough to reach its own delivery, which is the chapter's first commit and has nothing to do with limits: a batch sized below what the lane leaves claimable asserts on whatever else was lying about.
@@ -264,12 +264,28 @@ describe("the dispatcher", () => {
const r = relay.createDeliveryRelay({
db,
publisher: publisherMod.createJetStreamPublisher({
ensure: relay.ensureDeliveriesStream,
}),
logger: kit.createLogger("itest-relay"),
+ // A BATCH BIG ENOUGH TO REACH THIS TEST'S OWN DELIVERY. `drainOnce` is
+ // global: it takes the fifty oldest due deliveries in the platform,
+ // oldest first, and this suite's is the newest. Every earlier suite in the
+ // run leaves due deliveries behind, so once more than fifty of them
+ // accumulate the batch fills before reaching ours, the poll times out at
+ // eight seconds, and `expected 0 to be greater than 0` is what a reader
+ // sees.
+ //
+ // It only bites in the COVERAGE lane, where `fileParallelism: false` puts
+ // every suite in one process against one database. The failing run drains
+ // the backlog itself, so the next run passes — which is why it reads as a
+ // flake rather than as the threshold it is.
+ //
+ // Found at the rate-limit chapter's baseline. The deduplication chapter fixed the same global drain
+ // in `deliveries.itest.ts` twice and never looked at this door.
+ batchSize: 10_000,
});
return r.drainOnce();
};
/** As `deliverEvent`, but with a tenant's message text in the payload — so
* invariant 15 has something that must NOT appear in a log line. */Three more are earlier chapters' suites, changed only because request_id
made two error bodies that must be indistinguishable differ in a field that says
nothing about either. test-event.itest.ts loses one more line: the retry-and-disable chapter
labelled it with a range of that feature's own working numbers, which resolve
nowhere a reader can follow. FR-WHK-09 is what they meant — and a traceability
sweep that its own explanation trips is a sweep somebody starts ignoring, so the
numbers are not repeated here.
@@ -3,18 +3,19 @@ import { afterEach, describe, expect, it } from "vitest";
import { readFile } from "node:fs/promises";
import type { Server } from "node:http";
import type { AddressInfo } from "node:net";
import { createLogger, type Logger } from "@relay/service-kit";
import { serve } from "@relay/service-kit";
-import type { Frame, RevisionFabric } from "@relay/protocol";
+import { CLOSE_CODES, type Frame, type RevisionFabric } from "@relay/protocol";
import type { InternalSendResponse, Message } from "@relay/protocol";
import type { ApiClient } from "./api-client.js";
import type { Fanout } from "./fanout.js";
+import { decide, type GatewayLimits } from "./limits.js";
import { attachSessions, INBOUND_FRAME_TYPES } from "./session.js";
import { docsUrl } from "@relay/protocol";
// The door, the frames, and the liveness clock — all provable without a
// database, because the gateway has no database (ADR-05). The api is a
// stub here for exactly that reason: if these tests needed Postgres, the
@@ -50,12 +51,17 @@ function stubApi(overrides: Partial<ApiClient> = {}): ApiClient {
user: "tuan",
// The api now reports whether the user is banned, and a stub
// that does not say is a stub that has not thought about it.
banned: false,
channel_ids: [CHANNEL],
revisions: {},
+ // The limits ride the session response because the
+ // gateway has no database to read them from — so the stub supplies
+ // them, exactly as the api would. Generous by default: every test
+ // in this file is about something else.
+ limits: { connect: 3_000, send: 600 },
}
: null,
backfill: async () => ({}),
sendMessage: async () => committed(42),
// The backstop reads this. The default answers what the session above says,
// so a stub that never overrides it is a stub whose re-read agrees with its
@@ -159,17 +165,40 @@ function stubFanout(): Fanout & {
},
unsubscribe: async () => {},
close: async () => {},
};
}
+/** A counter with no Redis in it. The arithmetic is unit-tested
+ * in `limits.test.ts`; what these tests need is control over the ANSWER, so a
+ * refusal is a line of code instead of three thousand sockets. */
+function stubLimits(
+ allowances: { connect?: number; send?: number } = {},
+): GatewayLimits & { spent: { connect: number; send: number } } {
+ const spent = { connect: 0, send: 0 };
+ return {
+ spent,
+ spend: async (_environmentId, operation, limit) => {
+ spent[operation] += 1;
+ // The stub honours whichever allowance the test set, falling back to the
+ // limit the session response carried — which is what makes T034a's
+ // distinction visible: an allowance the test names here is the store's
+ // view, `limit` is the socket's cached one.
+ const allowed = allowances[operation] ?? limit;
+ return decide(spent[operation], allowed, 0, 60_000);
+ },
+ close: async () => {},
+ };
+}
+
async function boot(
api: ApiClient = stubApi(),
pingIntervalMs?: number,
fanout?: Fanout,
resumeDeadlineMs?: number,
+ limits?: GatewayLimits,
): Promise<Harness> {
const server: Server = serve({
service: "gateway",
notFoundDocsUrl: docsUrl("not_found"),
health: () => ({}),
logger: silent,
@@ -178,12 +207,13 @@ async function boot(
server,
api,
logger: silent,
...(fanout !== undefined && { fanout }),
...(pingIntervalMs !== undefined && { pingIntervalMs }),
...(resumeDeadlineMs !== undefined && { resumeDeadlineMs }),
+ ...(limits !== undefined && { limits }),
});
await new Promise<void>((resolve) => server.listen(0, resolve));
const { port } = server.address() as AddressInfo;
return {
url: `ws://127.0.0.1:${port}/v1/ws`,
close: async () => {
@@ -883,6 +913,257 @@ describe("INBOUND_FRAME_TYPES", () => {
// use to type as somebody else.
for (const forgeable of ["message.ack", "message.created", "typing"]) {
expect(INBOUND_FRAME_TYPES.has(forgeable as never)).toBe(false);
}
});
});
+
+// The socket's two limits — one at the door, one on every frame —
+// and the two shapes a refusal takes, which are different because a handshake
+// has an HTTP response to write headers onto and a frame does not.
+describe("the socket's limits", () => {
+ let harness: Harness | undefined;
+ afterEach(async () => {
+ await harness?.close();
+ harness = undefined;
+ });
+
+ /** The upgrade's HTTP answer, for the case where there is no WebSocket to
+ * ask. `ws` surfaces a non-101 as `unexpected-response`, which hands back the
+ * request and the raw `IncomingMessage` — status and headers included. */
+ function unexpectedResponse(
+ socket: WebSocket,
+ ): Promise<{ status: number; headers: Record<string, string | undefined> }> {
+ return new Promise((resolve, reject) => {
+ const timer = setTimeout(() => reject(new Error("no response")), 2000);
+ socket.on("unexpected-response", (_req, res) => {
+ clearTimeout(timer);
+ res.resume();
+ resolve({
+ status: res.statusCode ?? 0,
+ headers: res.headers as Record<string, string | undefined>,
+ });
+ });
+ socket.on("open", () => {
+ clearTimeout(timer);
+ reject(new Error("the handshake completed"));
+ });
+ socket.on("error", () => {});
+ });
+ }
+
+ it("refuses an over-limit handshake with an HTTP 429, before the handshake (FR-RTL-03)", async () => {
+ // An allowance of one, so the second connect is the refused one.
+ harness = await boot(
+ stubApi(),
+ undefined,
+ undefined,
+ undefined,
+ stubLimits({ connect: 1 }),
+ );
+ const first = new WebSocket(`${harness.url}?token=${await token()}`);
+ await nextFrame(first, "connection.ack");
+
+ const second = new WebSocket(`${harness.url}?token=${await token()}`);
+ const { status, headers } = await unexpectedResponse(second);
+ expect(status).toBe(429);
+ // The instruction, not just the refusal. This is the reason the limiter is
+ // a fixed window: `Retry-After` and `X-RateLimit-Reset` both name one
+ // moment, and a refilling bucket's honest answer would be a curve.
+ expect(Number(headers["retry-after"])).toBeGreaterThan(0);
+ expect(headers["x-ratelimit-limit"]).toBe("1");
+ expect(headers["x-ratelimit-remaining"]).toBe("0");
+ expect(headers["x-ratelimit-reset"]).toBeDefined();
+
+ first.close();
+ });
+
+ it("leaves already-open sockets alone when the door is shut (FR-RTL-03)", async () => {
+ // The refusal is about establishing connections, not about the ones that
+ // exist. A limiter that killed live sockets to enforce an establishment
+ // limit would be enforcing a concurrency limit, which is a different
+ // promise and one Relay has not made.
+ harness = await boot(
+ stubApi(),
+ undefined,
+ undefined,
+ undefined,
+ stubLimits({ connect: 1 }),
+ );
+ const open = new WebSocket(`${harness.url}?token=${await token()}`);
+ await nextFrame(open, "connection.ack");
+
+ const refused = new WebSocket(`${harness.url}?token=${await token()}`);
+ expect((await unexpectedResponse(refused)).status).toBe(429);
+
+ // Still there, and still working — a round trip rather than a readyState
+ // check, because "the socket object says OPEN" is not the same claim.
+ open.send(
+ JSON.stringify({
+ type: "message.send",
+ payload: { channel: CHANNEL, text: "hello", idem_key: "k1" },
+ }),
+ );
+ expect(await nextFrame(open, "message.ack")).toMatchObject({
+ payload: { seq: 42 },
+ });
+ open.close();
+ });
+
+ it("answers an over-limit frame with rate_limited and KEEPS THE CONNECTION OPEN", async () => {
+ harness = await boot(
+ stubApi(),
+ undefined,
+ undefined,
+ undefined,
+ stubLimits({ send: 1 }),
+ );
+ const socket = new WebSocket(`${harness.url}?token=${await token()}`);
+ await nextFrame(socket, "connection.ack");
+ const send = () =>
+ socket.send(
+ JSON.stringify({
+ type: "message.send",
+ payload: { channel: CHANNEL, text: "hello", idem_key: "k1" },
+ }),
+ );
+
+ send();
+ await nextFrame(socket, "message.ack");
+ send();
+ const error = await nextFrame(socket, "error");
+ // `rate_limited` was declared in chapter 1.3 and emitted by nothing until
+ // now. This is the first line of the codebase that sends it.
+ expect(error).toMatchObject({ payload: { code: "rate_limited" } });
+ // And every error frame carries an id now, which is the other contract
+ // chapter 1.3 wrote down and never wired.
+ expect((error as { payload: { request_id: string } }).payload.request_id)
+ .toBeTruthy();
+
+ // THE POINT: the socket is still up. Closing it would make the client
+ // reconnect, and a reconnect spends the ESTABLISHMENT allowance — a
+ // limiter that pushes the limited into a second limit.
+ expect(socket.readyState).toBe(WebSocket.OPEN);
+ socket.close();
+ });
+
+ it("enforces a CONFIGURED connect limit, not just the default (ADR-05, FR-RTL-04)", async () => {
+ // The limit arrives on the authentication response, because the gateway has
+ // no database to read it from. A test that only exercised the default would
+ // pass with the plumbing missing entirely.
+ harness = await boot(
+ stubApi({
+ session: async () => ({
+ environment_id: "env-1",
+ user: "tuan",
+ // The ban flag, which is upstream of this chapter in this order — a stub
+ // that does not say is a stub that has not thought about it.
+ banned: false,
+ channel_ids: [CHANNEL],
+ revisions: {},
+ limits: { connect: 2, send: 600 },
+ }),
+ }),
+ undefined,
+ undefined,
+ undefined,
+ // No allowance override: the stub honours the limit the session response
+ // carried, so the number under test is the CONFIGURED one.
+ stubLimits(),
+ );
+ const first = new WebSocket(`${harness.url}?token=${await token()}`);
+ await nextFrame(first, "connection.ack");
+ const second = new WebSocket(`${harness.url}?token=${await token()}`);
+ await nextFrame(second, "connection.ack");
+
+ const third = new WebSocket(`${harness.url}?token=${await token()}`);
+ expect((await unexpectedResponse(third)).status).toBe(429);
+ first.close();
+ second.close();
+ });
+
+ it("does not apply a limit changed mid-connection until the client reconnects (research R12)", async () => {
+ // The consequence R12 accepted, asserted so it is a property rather than a
+ // surprise. The alternative is a Postgres read per frame, from a service
+ // that holds no database client, on the hot path of the thing the limit
+ // protects.
+ let configured = 600;
+ harness = await boot(
+ stubApi({
+ session: async () => ({
+ environment_id: "env-1",
+ user: "tuan",
+ // The ban flag, which is upstream of this chapter in this order — a stub
+ // that does not say is a stub that has not thought about it.
+ banned: false,
+ channel_ids: [CHANNEL],
+ revisions: {},
+ limits: { connect: 3_000, send: configured },
+ }),
+ }),
+ undefined,
+ undefined,
+ undefined,
+ stubLimits(),
+ );
+ const socket = new WebSocket(`${harness.url}?token=${await token()}`);
+ await nextFrame(socket, "connection.ack");
+
+ // The policy changes to "refuse everything" while the socket is open.
+ configured = 0;
+
+ socket.send(
+ JSON.stringify({
+ type: "message.send",
+ payload: { channel: CHANNEL, text: "hello", idem_key: "k1" },
+ }),
+ );
+ // Still allowed: this connection is spending the allowance it was born
+ // with. A new one would not be.
+ expect(await nextFrame(socket, "message.ack")).toMatchObject({
+ payload: { seq: 42 },
+ });
+
+ const reconnected = new WebSocket(`${harness.url}?token=${await token()}`);
+ await nextFrame(reconnected, "connection.ack");
+ reconnected.send(
+ JSON.stringify({
+ type: "message.send",
+ payload: { channel: CHANNEL, text: "hello", idem_key: "k1" },
+ }),
+ );
+ expect(await nextFrame(reconnected, "error")).toMatchObject({
+ payload: { code: "rate_limited" },
+ });
+
+ socket.close();
+ reconnected.close();
+ });
+
+ it("STILL emits close code 4008 from nowhere (quickstart V7)", async () => {
+ // 4008 reads "quota exhausted". There is no quota yet — quotas are a later
+ // chapter — and reaching for the code because it was declared would collapse
+ // the distinction this chapter is built on: a rate limit is a smoothing
+ // instruction, a quota is a commercial one, and they do not deserve the same
+ // signal. So does 4009, "server shutdown (drain)", for the same kind of
+ // reason (NFR-REL-03).
+ //
+ // Grep rather than behaviour, because the claim is about absence: no input
+ // makes the gateway send it, and the only way to check "no input" is to read
+ // what the source can send.
+ const source = await Promise.all(
+ ["session.ts", "limits.ts", "resume.ts", "main.ts"].map((file) =>
+ readFile(new URL(file, import.meta.url), "utf8"),
+ ),
+ );
+ for (const text of source) {
+ expect(text).not.toMatch(/close\(\s*400[89]/);
+ }
+ // A grep that can only pass is not a check. The SAME pattern, aimed at the
+ // codes this file does emit, has to match — otherwise "nothing sends 4008"
+ // would also be true of a typo in the regex.
+ expect(source.join("")).toMatch(/close\(\s*400[12]/);
+ // And the vocabulary still declares them, so this is "unused", not "gone".
+ expect(CLOSE_CODES[4008]).toBeDefined();
+ expect(CLOSE_CODES[4009]).toBeDefined();
+ });
+});@@ -116,20 +116,23 @@ describe("resume across a real fabric", () => {
// window. Neither side coordinates; only the buffer saves this.
harness = await boot({
session: async () => ({
environment_id: "env-1",
user: "tuan",
// The api now reports whether the user is banned, and a stub
// that does not say is a stub that has not thought about it.
banned: false,
channel_ids: [CHANNEL],
revisions: {},
+ // The limits ride the session response now. Generous, and
+ // beside the point of every test in this file.
+ limits: { connect: 3_000, send: 600 },
}),
backfill: async () => {
await publishFromElsewhere(frame(43));
await settle(150); // give Redis time to actually deliver it
return {
[CHANNEL]: { messages: [frame(42), frame(43)], truncated: false },
};
},
sendMessage: async () => {
throw new Error("not used");
@@ -155,20 +158,23 @@ describe("resume across a real fabric", () => {
// and the flush is the only reason the client ever sees it.
harness = await boot({
session: async () => ({
environment_id: "env-1",
user: "tuan",
// The api now reports whether the user is banned, and a stub
// that does not say is a stub that has not thought about it.
banned: false,
channel_ids: [CHANNEL],
revisions: {},
+ // The limits ride the session response now. Generous, and
+ // beside the point of every test in this file.
+ limits: { connect: 3_000, send: 600 },
}),
backfill: async () => {
await publishFromElsewhere(frame(43));
await settle(150);
return { [CHANNEL]: { messages: [frame(42)], truncated: false } };
},
sendMessage: async () => {
throw new Error("not used");
},
// Agrees with `session` above: this file is about the resume,
@@ -188,20 +194,23 @@ describe("resume across a real fabric", () => {
it("goes live after the flush, with no buffering left behind", async () => {
harness = await boot({
session: async () => ({
environment_id: "env-1",
user: "tuan",
// The api now reports whether the user is banned, and a stub
// that does not say is a stub that has not thought about it.
banned: false,
channel_ids: [CHANNEL],
revisions: {},
+ // The limits ride the session response now. Generous, and
+ // beside the point of every test in this file.
+ limits: { connect: 3_000, send: 600 },
}),
backfill: async () => ({
[CHANNEL]: { messages: [frame(42)], truncated: false },
}),
sendMessage: async () => {
throw new Error("not used");
},
// Agrees with `session` above: this file is about the resume,
// and a backstop that disagreed with the connect would be a second subject
// under test.
@@ -239,20 +248,23 @@ describe("resume across a real fabric", () => {
// One number different from the test above it. That is the whole bug.
harness = await boot({
session: async () => ({
environment_id: "env-1",
user: "tuan",
// The api now reports whether the user is banned, and a stub
// that does not say is a stub that has not thought about it.
banned: false,
channel_ids: [CHANNEL],
revisions: {},
+ // The limits ride the session response now. Generous, and
+ // beside the point of every test in this file.
+ limits: { connect: 3_000, send: 600 },
}),
backfill: async () => ({
[CHANNEL]: { messages: [frame(42)], truncated: false },
}),
sendMessage: async () => {
throw new Error("not used");
},
// Agrees with `session` above: this file is about the resume,
// and a backstop that disagreed with the connect would be a second subject
// under test.
@@ -283,20 +295,23 @@ describe("resume across a real fabric", () => {
// drop the mark, and then deliver the 42 (research R3).
harness = await boot({
session: async () => ({
environment_id: "env-1",
user: "tuan",
// The api now reports whether the user is banned, and a stub
// that does not say is a stub that has not thought about it.
banned: false,
channel_ids: [CHANNEL],
revisions: {},
+ // The limits ride the session response now. Generous, and
+ // beside the point of every test in this file.
+ limits: { connect: 3_000, send: 600 },
}),
backfill: async () => ({
[CHANNEL]: { messages: [frame(42)], truncated: false },
}),
sendMessage: async () => {
throw new Error("not used");
},
// Agrees with `session` above: this file is about the resume,
// and a backstop that disagreed with the connect would be a second subject
// under test.
@@ -387,20 +402,23 @@ describe("resume across a real fabric", () => {
// duplicate into a gap, which constitution II ranks worse.
harness = await boot({
session: async () => ({
environment_id: "env-1",
user: "tuan",
// The api now reports whether the user is banned, and a stub
// that does not say is a stub that has not thought about it.
banned: false,
channel_ids: [CHANNEL],
revisions: {},
+ // The limits ride the session response now. Generous, and
+ // beside the point of every test in this file.
+ limits: { connect: 3_000, send: 600 },
}),
backfill: async () => {
throw new Error("backfill unavailable");
},
sendMessage: async () => {
throw new Error("not used");
},
// Agrees with `session` above: this file is about the resume,
// and a backstop that disagreed with the connect would be a second subject
// under test.@@ -399,22 +399,26 @@ describe("POST /v1/channels/:channelId/messages", () => {
const missing = await fetch(
`${url}/v1/channels/${crypto.randomUUID()}/messages?limit=10`,
{ headers: { authorization: `Bearer ${credential}` } },
);
expect(foreign.status).toBe(404);
expect(missing.status).toBe(404);
- expect(await foreign.json()).toEqual(await missing.json());
+ expect(withoutRequestId(await foreign.json())).toEqual(
+ withoutRequestId(await missing.json()),
+ );
});
it("answers a FOREIGN channel id with the same 404 as a missing one", async () => {
const foreign = await send({ text: "not for you", user: "courier" }, foreignChannelId);
const missing = await send({ text: "nobody home", user: "courier" }, crypto.randomUUID());
expect(foreign.status).toBe(404);
expect(missing.status).toBe(404);
// Indistinguishable — no data, and no reveal that the id exists.
- expect(await foreign.json()).toEqual(await missing.json());
+ expect(withoutRequestId(await foreign.json())).toEqual(
+ withoutRequestId(await missing.json()),
+ );
});
// ── THE ROUTE A CUSTOMER'S CLIENT ACTUALLY CALLS (FR-001) ─────────────────────
//
// The membership check lives in `repository.sendMessage` and is gated on `userId`
// being present. `repository.itest.ts` proves the check EXISTS by driving that@@ -18,12 +18,13 @@ import {
expandEventToDeliveries,
recordAttemptOutcome,
Repository,
} from "../db/repository";
import { encryptSecret, mintSigningSecret } from "./secret";
import { MAX_ATTEMPTS } from "./schedule";
+import { withoutRequestId } from "../isolation/compare";
// The attempt record, against a real broker and a real api.
//
// Invariants 1, 2, 3 and 5 of contracts/attempts.md live here. Invariant 4 is the
// swallowed publish failure and is pure, so it lives in analytics.test.ts.
//
@@ -385,13 +386,15 @@ describe("the attempt record", () => {
latency_ms: 30,
};
const first = await report(body);
const second = await report(body);
// The dispatcher is told the same thing both times — that is what idempotent
// means here — so the repeat is invisible to it.
- expect(await first.json()).toEqual(await second.json());
+ expect(withoutRequestId(await first.json())).toEqual(
+ withoutRequestId(await second.json()),
+ );
// Spend a real budget looking for a second event rather than checking once.
const events = await collected(scoped.id, 2, 5_000);
expect(events).toHaveLength(1);
}, 30_000);@@ -13,12 +13,13 @@ import {
createApiKey,
createEnvironment,
recordAttemptOutcome,
Repository,
} from "../db/repository";
import { encryptSecret, mintSigningSecret } from "./secret";
+import { withoutRequestId } from "../isolation/compare";
// Proving an endpoint works again (FR-WHK-09, research R8).
//
// THIS SUITE PLAYS THE DISPATCHER. `POST /test` creates a real delivery and then
// watches the row, because the attempt happens in another process — so something
// has to make that attempt, and importing the dispatcher's build into the api's
@@ -475,13 +476,15 @@ describe("the test event", () => {
expect(response.status).toBe(404);
// The same answer a missing endpoint gets, so a probe cannot tell one from the
// other — and nothing was delivered.
const missing = await sendTest(randomUUID(), myKey.credential);
expect(missing.status).toBe(404);
- expect(await response.json()).toEqual(await missing.json());
+ expect(withoutRequestId(await response.json())).toEqual(
+ withoutRequestId(await missing.json()),
+ );
expect(received).toHaveLength(0);
}, 60_000);
it("answers honestly when nothing is there to make the attempt", async () => {
// No dispatcher, nobody playing one. The route must not hang for ever and must
// not claim the endpoint is unhealthy — it does not know that. This is the one@@ -70,12 +70,24 @@ export default defineConfig({
],
thresholds: {
// Constitution VI, first clause: 70% of business logic. Set to what the
// constitution says, not to what the code achieves — a threshold tuned
// down to pass measures nothing. Currently met with room to spare
// (86.55% statements, 78.07% branches at the time of writing).
+ //
+ // THE LIMITS CHAPTER'S TEN FILES MOVE BOTH FIGURES UP, which is not the usual
+ // direction for a chapter that adds code and is worth naming for that reason:
+ // eight of the ten are small and heavily branched.
+ //
+ // MEASURED ON THIS TREE at the limits chapter's close: 80 files, 1,183 tests,
+ // 368.17 s, exit 0 — **92.39% statements, 86.50% branches**, 92.09% functions,
+ // 93.83% lines. Published Part 3 read 89.50 / 82.73 at the same chapter, and the
+ // two are NOT comparable: this tree reaches this chapter with a different file
+ // set in a different order. Both are recorded because the pins below were taken
+ // from published's readings and held on this one, which is the fact that
+ // mattered when they were ported.
lines: 70,
functions: 70,
statements: 70,
branches: 70,
// Constitution VI, second clause: ordering, idempotency and tenant
@@ -554,12 +566,82 @@ export default defineConfig({
"services/gateway/src/typing.ts": {
branches: 100,
functions: 100,
lines: 100,
statements: 100,
},
+
+ // The rate-limit chapter's limiter. Pinned at what the work achieves, which for the
+ // three pure files is everything — they hold no clock, no store and no
+ // framework, so a branch they miss is a case nobody thought of rather
+ // than a case nobody could reach.
+ //
+ // `bucket.ts`, `policy.ts` and `fallback.ts` are here at 100 on every
+ // metric. `fallback.ts` earns the strictest reading of constitution VI
+ // available: it is the mechanism the AUTH limiter degrades to, and R3's
+ // whole argument is that this one counter must not fail open. An
+ // unmeasured branch in it is a hole in the thing the chapter is about.
+ "services/api/src/limits/bucket.ts": {
+ branches: 100,
+ functions: 100,
+ lines: 100,
+ statements: 100,
+ },
+ "services/api/src/limits/policy.ts": {
+ branches: 100,
+ functions: 100,
+ lines: 100,
+ statements: 100,
+ },
+ "services/api/src/limits/fallback.ts": {
+ branches: 100,
+ functions: 100,
+ lines: 100,
+ statements: 100,
+ },
+
+ // The four that touch a store, a clock or Nest's request pipeline, pinned
+ // at measurement rather than at 100. Each shortfall is one branch that
+ // needs a real outage at a real instant to reach, and chasing it would
+ // mean mocking the thing under test.
+ //
+ // `store.ts` misses its `downUntil` reset; `auth-limiter.ts` misses the
+ // arm where the store answers AND the fallback has an entry;
+ // `client-address.ts` misses one shape of malformed body. The gateway's
+ // `limits.ts` misses the arm where a recovered store clears `downUntil`
+ // mid-window.
+ "services/api/src/limits/store.ts": {
+ branches: 91,
+ functions: 100,
+ lines: 100,
+ statements: 100,
+ },
+ "services/api/src/limits/auth-limiter.ts": {
+ branches: 87,
+ functions: 100,
+ lines: 100,
+ statements: 100,
+ },
+ "services/api/src/limits/client-address.ts": {
+ branches: 90,
+ functions: 100,
+ lines: 100,
+ statements: 100,
+ },
+ "services/api/src/limits/rate-limit.middleware.ts": {
+ branches: 85,
+ functions: 100,
+ lines: 96,
+ statements: 97,
+ },
+ "services/gateway/src/limits.ts": {
+ branches: 90,
+ functions: 100,
+ lines: 100,
+ statements: 100,
+ },
},
},
},
plugins: [
swc.vite({
module: { type: "es6" },The stack gets the address, and the api gets a dependency on the service:
@@ -75,13 +75,13 @@ services:
mailpit:
image: axllent/mailpit:v1.28
# An SMTP server that accepts everything and delivers nothing,
# with an HTTP API for reading what it caught.
#
- # WHY A CONTAINER RATHER THAN A FAKE. FR-021 says an email must not contain a
+ # WHY A CONTAINER RATHER THAN A FAKE. FR-WHK-07 says an email must not contain a
# signing secret, and the only artefact that can settle that is the message a
# server RECEIVED — a stub records what the sender passed, which is the same
# object the assertion would be reading, so a mailer that dropped the secret
# into a header the stub does not model would pass. Constitution VII asks a
# fifth container to justify itself; this is the justification (research R9).
#
@@ -109,22 +109,33 @@ services:
build:
context: .
dockerfile: services/api/Dockerfile
environment:
DATABASE_URL: postgres://relay:relay@postgres:5432/relay
RELAY_NATS_URL: nats://nats:4222
+ # Container names, not localhost — the api's own default is
+ # `redis://localhost:6379`, which inside this container is not the Redis
+ # service. And the tenant limiter FAILS OPEN by design (SAD §6.3), so a
+ # missing address would not crash anything: the composed stack would serve
+ # every request unlimited while reporting a limit. The constitution
+ # requires the full stack to start with one command, and this is what makes
+ # that true rather than merely quiet (research R24).
+ #
+ # RELAY_SMTP_URL joins in the transport phase, with the container it names.
+ RELAY_REDIS_URL: redis://redis:6379
# Development values. Both are secrets in anything that is not a laptop,
# and the api refuses to start in production without the first.
RELAY_WEBHOOK_SECRET_KEY: ${RELAY_WEBHOOK_SECRET_KEY:-}
RELAY_INTERNAL_CREDENTIAL: ${RELAY_INTERNAL_CREDENTIAL:-rk_svc_local_development_credential_0000}
PORT: "4000"
ports:
- "${RELAY_API_PORT:-4000}:4000"
depends_on:
postgres: { condition: service_healthy }
nats: { condition: service_healthy }
+ redis: { condition: service_healthy }
healthcheck:
test: ["CMD", "wget", "-q", "-O", "-", "http://localhost:4000/healthz"]
interval: 5s
timeout: 3s
retries: 5
start_period: 20sContainer names rather than localhost, because the api's own default —
redis://localhost:6379 — is not the Redis service from inside a container. And
the tenant limiter fails open by design, so a missing address would not crash
anything: the composed stack would serve every request unlimited while reporting a
limit. The constitution requires docker compose up to bring the full stack up,
and this is what makes that true rather than merely quiet.
The ioredis restriction is the same shape chapter 2.4 gave pg: a store keyed
per tenant gets one home, and an unrestricted client anywhere else is a
cross-tenant read waiting to be written.
AND IT ARRIVES TWELVE FILES LATE, WHICH IS THE INTERESTING PART. A rule
normally lands before the code it governs. This one lands after: by the time the
counter store exists, twelve files in this repository already import ioredis —
the api's two publishers, five gateway modules, and five suites that need a client
belonging to neither service. Every one of them has to be exempted in the same
commit that adds the rule, or the rule reddens a chapter nobody is editing.
The exemption list's other half is the test that reads it:
@@ -12,12 +12,18 @@ import { describe, expect, it } from "vitest";
// that no longer needs one.
//
// There is no second list to compare against. What the list has to agree with is
// the TREE, which makes the assertion here read the config's own text — and the
// restricted module names are read out of the rule rather than restated, so
// adding a third restricted module does not need this file edited.
+//
+// AND THE THIRD ONE ARRIVED, WHICH TESTED THAT CLAIM. The rate-limit chapter
+// restricted `ioredis`; the two checks that read the module names off the rule
+// picked it up with no edit here, exactly as intended. The last check needed one,
+// because it is not about modules at all — it is about which entries may be
+// PATTERNS, and the counter store is a second data-access layer.
const ROOT = join(import.meta.dirname, "..", "..", "..");
const CONFIG = join(ROOT, "eslint.config.mjs");
function config(): string {
return readFileSync(CONFIG, "utf8");
@@ -75,13 +81,13 @@ describe("the driver exemption is checked in both directions", () => {
);
expect(uses, `${path} is exempt from the driver rule and imports none of ${modules.join(", ")}`)
.not.toEqual([]);
}
});
- it("exempts the repository layer as a directory and everything else by path", () => {
+ it("exempts the two data-access layers as directories and everything else by path", () => {
const text = config();
// FOUND BY SCANNING BACK FROM THE RULE, NOT BY A WINDOW. This read `indexOf("ignores:
// [", indexOf("no-restricted-imports") - 2000)` and the 2000 was the whole check: the
// block grew by a comment, the real `ignores` fell 2,027 characters before the anchor
// — 27 outside the window — and the search silently found the NEXT one instead and
// reported `[]`. An empty list is a legitimate-looking answer, so nothing said broken.
@@ -93,12 +99,19 @@ describe("the driver exemption is checked in both directions", () => {
(m) => m[1]!,
);
// THE POSITIVE CONTROL. Every assertion below is about which of these are globs, and
// a parse that found nothing would satisfy all of them.
expect(entries.length, "parsed no entries at all — this test is broken, not passing")
.toBeGreaterThan(0);
- // The directory pattern is legitimate — `services/api/src/db` IS the layer the
- // rule carves out. Any OTHER pattern would silently absorb the next file added
- // under it, which is the thing this list exists instead of.
- expect(entries.filter((g) => g.includes("*"))).toEqual(["services/api/src/db/**"]);
+ // TWO directory patterns and they are the two data-access LAYERS: `db/**` for the
+ // driver and the engine, `limits/**` for the counter store, each of them the thing
+ // the rule carves out rather than a file that happens to need it. Any other pattern
+ // would silently absorb the next file added under it, which is what this list
+ // exists instead of — and the rate-limit chapter arrived with twelve older Redis
+ // clients in the tree, every one of them listed by path above rather than swept up
+ // by `services/gateway/src/**`.
+ expect(entries.filter((g) => g.includes("*"))).toEqual([
+ "services/api/src/db/**",
+ "services/api/src/limits/**",
+ ]);
});
});It asserts in both directions — every listed path must exist and must still import something the rule restricts — which is how a thirteenth Redis client cannot quietly inherit somebody else's exemption, and how a stale entry cannot sit there holding an exemption over a file that stopped needing one. One assertion in it had to be amended, and that is the only kind of change this file should ever need: it said exactly one entry may be a directory pattern, and there are two now. The two checks that read the restricted module names off the rule needed no edit at all, which is what they were written for.
@@ -14,12 +14,23 @@ export default tseslint.config(
languageOptions: { globals: globals.nodeBuiltin },
},
{
// Isolation lives in data access, not in handlers (constitution I):
// only the repository layer may touch the driver.
//
+ // THE RATE-LIMIT CHAPTER ADDS THE SECOND PER-TENANT STORE and the same argument
+ // applies to it. The counters are keyed `rl:{environment_id}:…`, so an
+ // unrestricted client would let any handler read or write another tenant's
+ // counter — which is the access this rule exists to prevent, and constitution I
+ // calls that a correctness property rather than a convention.
+ // `services/api/src/limits/**` is the Redis analogue of the repository layer, and
+ // it is exempt as a DIRECTORY for the same reason `db/**` is: it IS the layer the
+ // rule carves out. Every other Redis client in the tree is listed by path —
+ // including the gateway's half of this same store, `services/gateway/src/limits.ts`,
+ // which is one file rather than a layer.
+ //
// AND THE LANE'S OWN INFRASTRUCTURE, NAMED FILE BY FILE. The harness opens raw
// connections deliberately: one carrying the guard's exemption and one without,
// which is the distinction its tests are about, and `createPool()` cannot express
// it. So these three are exempt — as PATHS, not as a `packages/test-harness/**`
// pattern, because a pattern would silently absorb the next file added there and
// that is the failure mode the guard itself exists to remove.
@@ -30,13 +41,18 @@ export default tseslint.config(
// exemption forever. `driver-exempt.test.ts` reads this array and asserts each
// path exists and still imports a module the rule below restricts — with those
// module names read out of the rule rather than restated.
files: ["**/*.ts"],
ignores: [
"services/api/src/db/**",
- // DRIVER_EXEMPT — the lane's own infrastructure. Reasons, one per path:
+ "services/api/src/limits/**",
+ // DRIVER_EXEMPT — every path below is exempt from all three restricted modules,
+ // the driver's name on the marker notwithstanding: `driver-exempt.test.ts` reads
+ // the module names out of the rule, so this list governs whatever the rule names.
+ //
+ // First the lane's own infrastructure. Reasons, one per path:
// global-setup.ts installs the guard against a database vitest names
// setup.ts rewrites the connection string to carry the exemption
// guard.itest.ts holds one exempt client and one plain one, and the
// difference between them is the whole test
"packages/test-harness/src/global-setup.ts",
"packages/test-harness/src/setup.ts",
@@ -64,12 +80,89 @@ export default tseslint.config(
// "the state under test is one the repository is now unable to reach". Both are
// listed by path rather than reached through a shared helper, because a helper in
// another file names none of these specifiers and this rule sees only imports —
// an invisible exemption is worse than a listed one.
"services/api/src/internal/backfill.itest.ts",
"services/api/src/messages/history.itest.ts",
+ // ── AND EVERY OTHER REDIS CLIENT, BY PATH, WITH THE ARGUMENT IT NEEDS ──
+ //
+ // The rule arrives here and TWELVE files older than it already import `ioredis`.
+ // A missing exemption is not a silent one — this rule goes red on a chapter
+ // nobody is editing — so all of them land in the commit that adds the rule.
+ // FIVE DIFFERENT ARGUMENTS, and a blanket "the gateway's Redis files" would
+ // erase all five distinctions the rule exists to make.
+ //
+ // (1) NO KEY IS TOUCHED AT ALL — these name a pub/sub SUBJECT and never a key,
+ // which is the property, not whether they publish or subscribe. The subjects are
+ // `chan:{channel_id}`, `member:{channel_id}` and `typing:{channel_id}`: a channel
+ // UUID, and a subject is not readable at all, only listened to by whoever already
+ // subscribed. There is no key here for a cross-tenant read to reach. (The api's
+ // two publishers publish; the gateway's `membership.ts` only ever subscribes,
+ // because the api publishes that fabric — and the argument is the same either
+ // way.)
+ "services/api/src/fanout/publisher.ts",
+ "services/api/src/membership/publisher.ts",
+ //
+ // (2) THE COUNTER STORE'S OTHER HALF. `rl:{environment_id}:…` is the key shape
+ // the whole restriction is about, and this file composes it — so it is exempt as
+ // the rule's own subject, not against its reason. `limits.itest.ts` is listed
+ // beside it for something the rule cannot express at all: its subject is that
+ // the api and the gateway increment the SAME key, and the only way to check that
+ // is to read the key with NEITHER of their code.
+ "services/gateway/src/limits.ts",
+ "services/gateway/src/limits.itest.ts",
+ "services/gateway/src/fanout.ts",
+ // `member:{env}:{user}` — the principal-addressed half of that fabric — DOES
+ // carry an environment id, and that still does not make it the limiter's case:
+ // a subject is not readable, and the id is composed from the repository's own
+ // scope on the way out and from the authenticated connection's identity on the
+ // way in, never read from a payload.
+ "services/gateway/src/membership.ts",
+ // `typing.ts` both publishes and subscribes and composes no key at all — the
+ // environment travels INSIDE the payload, where the receiving gateway checks it
+ // against the connection it is about to act on.
+ "services/gateway/src/typing.ts",
+ //
+ // (2) KEYS ARE COMPOSED AND THEY ARE ENVIRONMENT-SCOPED — the limiter's own
+ // argument rather than the publishers'. `presence:{env}:{user}` is exactly the
+ // shape the restriction guards. Every key is composed from the environment id on
+ // the authenticated connection's own identity; no path takes one from a client,
+ // and there is no scan, `KEYS` or pattern read that could reach another tenant's.
+ "services/gateway/src/presence.ts",
+ //
+ // (3) THE ENVIRONMENT COMES FIRST IN THE KEY, which is the strongest case on
+ // this list rather than the weakest. `conn:{env}:{user}:{slot}` makes
+ // constitution I structural in the key itself: reaching across a tenant needs a
+ // caller to hand this module another environment's id, and the session layer
+ // takes that from the api's verified identity. The other entries argue about
+ // what they touch; this one cannot be wrong without being lied to.
+ "services/gateway/src/connections.ts",
+ //
+ // (4) THE SUBJECT IS WHAT REACHES THE FABRIC, so the oracle cannot be either
+ // service's own client. A spy on `createFanout` or on `createPresence` proves
+ // that an object was asked to publish, not that a frame arrived — and these
+ // suites' receive halves have rejection paths (a body that is not JSON, a body
+ // that is JSON and not a transition) that no module-level API can produce,
+ // because each only ever publishes payloads its own schema built.
+ "services/api/src/fanout/fanout.itest.ts",
+ "services/gateway/src/presence.itest.ts",
+ "services/gateway/src/membership.itest.ts",
+ "services/gateway/src/typing.itest.ts",
+ //
+ // (5) THE RAW CLIENT IS THE STIMULUS, NOT THE ORACLE — a fifth reason, and the
+ // rule cannot express it. This suite's subject is delivery and it asserts on
+ // sockets. It needs a client to CAUSE a membership change: `Membership` exposes
+ // `onChange`, `subscribeChannel` and `watch` and no `publish`, because the api
+ // publishes and the gateway only ever subscribes.
+ "services/gateway/src/connections.itest.ts",
+ //
+ // `services/gateway/src/connections.test.ts` IS DELIBERATELY ABSENT, and the
+ // ledger that owed these entries said to add it. It reads the module's own
+ // source off disk and imports nothing restricted, so the exemption would be one
+ // over nothing — and `driver-exempt.test.ts`'s stale-entry check is the half of
+ // this list that goes red when a listed file stops needing it.
],
rules: {
"no-restricted-imports": [
"error",
{
paths: [
@@ -80,12 +173,17 @@ export default tseslint.config(
},
{
name: "drizzle-orm",
message:
"The query engine lives inside the repository layer only (constitution I, ADR-16).",
},
+ {
+ name: "ioredis",
+ message:
+ "The counter store lives in services/api/src/limits and services/gateway/src/limits.ts only (constitution I). Its keys are per environment; an unrestricted client is a cross-tenant read.",
+ },
],
patterns: [
{
group: ["drizzle-orm/*"],
message:
"The query engine lives inside the repository layer only (constitution I, ADR-16).",@@ -43,12 +43,14 @@
"RELAY_REDIS_PORT",
"RELAY_NATS_URL",
"RELAY_NATS_PORT",
"RELAY_OUTBOX_RELAY",
"RELAY_DELIVERY_RELAY",
"RELAY_INTERNAL_CREDENTIAL",
+ "RELAY_AUTH_FAILURES_PER_MINUTE",
+ "RELAY_AUTH_KEY_PREFIX",
"RELAY_WEBHOOK_SECRET_KEY",
"RELAY_EVENT_CONSUMER",
"RELAY_NATS_REPLICAS",
"RELAY_E2E_API_PORT",
"RELAY_SMTP_URL",
"RELAY_MAILPIT_URL",@@ -397,12 +397,19 @@ export async function boot({ gateways = 2 } = {}): Promise<System> {
"RELAY_INTERNAL_CREDENTIAL",
// The rate-limit chapter's other half: where the notification relay posts its SMTP.
// The lane runs Mailpit on 11025 and the default is 1025, so an
// unforwarded variable is not a missing feature — it is a mailer talking
// confidently to a port nothing is listening on.
"RELAY_SMTP_URL",
+ // The failed-authentication threshold and the counter's key
+ // prefix. Forwarded for the reason this list exists at all — turbo runs
+ // tasks in STRICT env mode, so an undeclared variable reaches a child as
+ // `undefined` and the `??` behind it silently wins. A suite that raised the
+ // threshold would raise it in the parent and not in the api the child runs.
+ "RELAY_AUTH_FAILURES_PER_MINUTE",
+ "RELAY_AUTH_KEY_PREFIX",
),
// The api children run WITHOUT the outbox relay. This journey
// asserts message delivery, and a background loop draining the outbox while
// the outbox chapter's own suite asserts on that same table is a race between two test
// files, not a property of the system. The relay has its own suite, which
// drives it explicitly.AND NO NEW DEPENDENCY, which is worth one line because the limiter is the
chapter you would expect to add one. ioredis is already in services/api/package.json:
the fan-out chapter put it there for the REST send path's publisher. The counter store
reaches for the client the api already had, and the only thing this chapter adds to
that file is nothing at all.
Quotas. FR-RTL-05…08 — monthly caps, hard and soft spending limits, the 50/80/100% email — are the next chapter, and the dependency is the reason for that order rather than the length. A quota is metered consumption, and the metering it reads comes later still. Building quotas on a Redis counter that fails open would put money in a store this chapter has spent its length arguing is allowed to lose things, which is why the next chapter reaches for Postgres instead.
Anything about the connection CAP, and in this order that is a scope statement
rather than a missing dependency. The connection-cap chapter already built the
conn:{env}:{user}:{slot} registry, and it sits one layer away from the counter this
chapter adds: the cap asks how many are open, the limit asks how many were opened
this minute. Two questions, two mechanisms, and the door checks the cheap one
first — the limiter is one INCR, the cap a walk of up to five keys.
Worth saying because they read alike and fail differently. A client at its cap must
close a connection; a client over its connect limit must wait. The first is close
code 4004 after an error frame, the second an HTTP 429 before the handshake
completes — and the reason for that difference has a section of its own above.
Per-API-key limits. The SRS says per tenant, and the environment is the boundary constitution I enforces. A key is a credential, not a tenant.
A dashboard showing remaining allowance. There is no dashboard. The headers are the half of that promise constitution V requires.
A drain grace period, without which the connect limit and NFR-REL-03 only half
agree. Close code 4009 is declared and waiting.
A documentation site, which this chapter is the one to make cost something.
Naming a dependency is the difference between a scope decision and a silent gap.