ArgonDB v1.5.7.21 — The ArgonDB developer guide — cut v1.5.7.21 at a1ddf5f153868d1cbec935fc6a15ea80234f5370

The ArgonDB developer guide

GENERATED — never hand-edit. Regenerated at the release cut only (make regen-at-cut, [D-2026-09-26-6bea57]); the stamp above names the cut and the source commit. The drift gate (python3 scripts/generate-developer-guide.py --check, a make lint-nocargo leg) proves this file byte-identical to the guide of that STAMPED commit, naming any chapter that differs; make lint-at-cut proves it against HEAD at the cut. The seven chapters below are assembled, byte for byte apart from heading demotion and in-document anchors, from docs/2026-09-20-guide-00-… through -04-…, -07-… and docs/2026-10-04-guide-08-… — those files stay the cited sources (path:line citations elsewhere in the repo point into them), so edit a chapter there, never here.

Table of contents

• Developer guide 0 — the themes that run through ArgonDB
• Guide 1 — one database per tenant, and the wall between tenants
• Guide 2 — the per-tenant unit: the gate, wake and stand-down
• Guide 3 — the lake and its process model
• Guide 4 — continuous materialized views, and a view across the tenant boundary
• Guide 7 — agents and the MCP endpoint
• Guide 8 — building an application cloud on ArgonDBCloud: the ERP's configuration system and ERPCloud

Developer guide 0 — the themes that run through ArgonDB

Register: internal technical reference, written against ArgonDB v1.5.6. Every claim below carries a path:line into this repository, a section number (§4.0) into one of the guides that follow, or a ruling id [D-...] into the decision registry; nothing here was run against a box by this chapter. Naming: "sidecar" names only a tenant's own application; the process that turns the write log into the lake is the tail (docs/programs.md:106-112).

The five guides that follow this one each go deep on one region of ArgonDB: one database per tenant, and the wall, the per-tenant unit's gate, wake and stand-down, the lake and its process model, continuous materialized views, and agents and the MCP endpoint. This chapter does not summarize them one at a time — each already summarizes itself, in its own opening section. What follows instead is the six ideas that keep reappearing across those five regions: read together, they are what makes ArgonDB one mechanism rather than five, and they are the shape of what an application developer actually builds and experiences on top of it.

Each theme below has the same four parts: a plain-language paragraph first, then what the five guides say about it, then what it means for how you build an application, then what you experience running one — the commands, the things you stop having to do, and the refusals you will meet along the way.

1. One database per tenant, on its own timeline

In plain terms: every customer gets their own database, not a shared table with a customer-ID column. It's the difference between everyone keeping their money in one drawer with a label on each bill, versus everyone getting their own separate drawer. A bug in the label-reading code can empty the wrong drawer; a bug that reads the wrong drawer only ever reads an empty one, because the money was never in there.

Across the chapters. Guide 01 states the resulting invariant directly: "There is no query in this system that could return another company's row, because another company's rows are not in the database being queried" (docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md §2.1). That single-database-per-tenant topology was amended twice in September — first to give each tenant its own compute that sleeps and wakes on connect [D-2026-09-20-0d5ba3], then to give each tenant its own Postgres instance, own WAL and own timeline, with only the storage plane still shared (DESIGN.md:391 [D-2026-09-21-4107c5]) — and an operator-only switch, [compute] per_tenant, controls whether a given box actually turns the per-tenant compute on; it ships off, and no proof or gate flips it [D-2026-09-23-5122a6]. As of v1.5.6 this is the shape the product ships on one box, not a design: a tenancy (one multi-tenant system in the Estate — the CLI's word for it is system) and the tenants inside it are created by two provisioning commands (argondb system create, argondb tenant create; crates/argondb-ctl/src/main.rs:1059-1082, :610-640), and with per_tenant on, a company's compute is started at tenant create on a port recorded then, so a new company runs on its own compute from its first minute. Turning it on under companies that already hold data is not yet a complete move: the data half of provisioning — carrying an existing company's roles and rows onto its own branch — is not built, and DESIGN.md says so rather than implying otherwise (DESIGN.md:387). Guide 02 is the lifecycle mechanism this topology runs on: a unit is one compute plus at most one sidecar, sized so it is exactly what can be destroyed and recreated with no loss (docs/2026-09-20-guide-02-the-per-tenant-unit-gate-wake-and-stand-down.md §1, §2.4) — a stood-down unit measures 137 MB and about 0.6 s to wake (docs/2026-09-14-argondb-process-model-preliminary-design.md:114-119). The sidecar is the company's own application: one process per company under that company's own uid and cgroup [D-2026-09-21-7c4769], started only if the company was created with --sidecar-command — "a company with no application is a complete tenant" (crates/argondb-ctl/src/main.rs:640-647). Guide 03's tail is what makes the topology affordable: past one box it is a single process multiplexed across every tenant's timeline, never co-resident with one tenant's own instance, with a per-stream memory budget so the Nth tenant cannot starve the first (docs/2026-09-20-guide-03-the-lake-and-its-process-model.md §2.12 [D-2026-09-21-25c7c6]; §3.5 [D-2026-09-23-d17cbd]).

Building on ArgonDB. You never design multi-tenancy into your own schema. There is no tenant-ID column to remember on every table and no WHERE clause standing between a bug and a cross-customer leak — a company's rows live in a database that other companies' queries cannot reach at all. What ArgonDB does keep uniform across tenants is its own per-company schema, and in v1.5.6 it keeps it uniform across upgrades too: argondb estate migrate [--to <version>] walks every company in the estate directory, reads each one's version from that company's own argon.schema_version, skips a company already at the target, and brings the rest up with the same versioned apply tenant create uses (crates/argondb-ctl/src/estate_migrate.rs:1-50, crates/argondb-ctl/src/main.rs:563-575). Be precise about what that schema is: today it is ArgonDB's own identity table and its one write door (UNIFORM_SCHEMA_VERSION, crates/argondb-ctl/src/tenant_create.rs:1717-1748) — it does not migrate your application tables, which stay yours to version. The connection pooler, for its part, is optional: the box uses a platform PgBouncer when a supported one (1.25.2 or newer) is on PATH and serves every tenant on the direct port when it is not (DESIGN.md:389 [D-2026-09-24-f0e068]).

What you experience. Creating a tenancy is one command, argondb system create <system_key>, and so is creating a company inside it — argondb tenant create acme --display-name "Acme" --legal-name "Acme Holdings LLC" --system distributors --identity "tax_id=12-3456789" — which runs the timeline, compute, database, login role and grants, uniform schema, wall pass, identity record and (if you named one) the sidecar in a fixed order, is safe to re-run if it is interrupted, and finishes by minting the company's own MCP token (Theme 6) (crates/argondb-ctl/src/tenant_create.rs:206-215, :2955-2975; docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md §4.0 [D-2026-09-21-e781f5]). system create now publishes the new system's Identities to every other registered system in the same call, creating the Identities namespace itself when nothing has been published into it yet; it used to defer that grant and leave it to be run by hand later (crates/argondb-ctl/src/system_create.rs:1499-1510). After an upgrade, argondb estate migrate is the one command that brings every existing company to the new build's schema version; one company it cannot migrate is named as REFUSED, the pass carries on for the others, and the command exits non-zero naming how many were refused (crates/argondb-ctl/src/main.rs:4133-4148); the pass is a command you run, not something the box does on its own when a new build is deployed (crates/argondb-ctl/src/estate_migrate.rs:19-32). You never write idle-detection logic yourself: bytes at the gate are the only signal a unit uses to decide it is still awake, and the whole control is one config key, [compute] idle_stop (docs/2026-09-20-guide-02-the-per-tenant-unit-gate-wake-and-stand-down.md §2.1). The command REFUSES to adopt a database it did not create itself, rather than silently taking it over (docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md §4.0), and a client that connects while a tenant is mid-wake gets Postgres's own 57P03 cannot_connect_now — a signal every driver already knows to retry (docs/2026-09-20-guide-02-the-per-tenant-unit-gate-wake-and-stand-down.md §2.1, §3.6). A missing or too-old PgBouncer is never a boot refusal in any profile: the box starts on the direct port and says so, on the banner and in argondb status, with a pooler: NOT RUNNING line naming what it found and the minimum it wants; the pooled port, when a pooler runs, is the direct port plus 200 (5632 at the defaults), and [pooler] enabled = "off" runs none (crates/argondb-ctl/src/pooler.rs:79, :813-832). The release installer writes [pooler] enabled = "on" into the box's <data>/argon.toml only when the key is absent, and on, auto and an unset key are one code path (scripts/release/install.sh:98-106). The one pooler setting that still refuses the boot is a misspelled enabled value (crates/argondb-ctl/src/pooler.rs:221-240).

2. The wall — the thing every surface is measured against

In plain terms: imagine every tenant's data behind its own locked door, with its own key. A feature doesn't get to say "trust me, I checked the badge" — it has to actually not have a key to the other doors. That's the difference between a rule a programmer could forget and a wall a programmer cannot walk through by accident.

Across the chapters. Guide 01 rules the wall as three parts together, never a choice among them — catalog authorization, login, and a warehouse plus bucket plus credential scoped per tenant database (docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md §1, §2.6 [D-2026-09-11-21a6d7]) — and states plainly that the wall is "not a filter inside a query" but a different database, a different place in storage, a different credential (deploy/publish/architecture-site/content.md:374-381). One predicate, wall::decide, gates every object-store operation (docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md §3.4), and the same discipline reappears wherever a boundary is checked: Guide 04 evaluates a view's database with the identical predicate the pin uses, and refuses a cross-database join by name rather than letting the query planner narrow it away with a WHERE clause (docs/2026-09-20-guide-04-continuous-materialized-views.md §2.13, §3.4); the lake's read plane refuses a read of another database's table the same way, by name, unless that database has granted it (Theme 5; cross_database_refusal, crates/argondb-tail/src/lib.rs:967-983). Row-level security is explicitly a different, narrower mechanism: it stays inside one tenant's database and separates roles, not companies, so a defective policy is an over-permissive grant inside one company rather than a cross-customer breach; and a table's lake copy carries no row policies, so the read plane refuses a scan of a protected table's lake copy rather than serving it (docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md §2.9). Guide 01 states the one limit on all of this plainly: the box operator stands inside every wall, and a tenant never does (docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md §2.7, §6). It is equally direct about the storage plane under the databases: every company is a timeline of the box's one pageserver tenant, the pageserver refuses a caller with no token, and each compute's token names its one timeline, so the pageserver refuses a company's valid token on any other company's timeline — built [D-2026-09-23-b822c8] and asserted by the W5± cases of testing/box/cross-company-isolation-eight-surfaces.sh. With the default [compute] per_tenant = false, one compute serves every company's database and holds the box's own timeline, so the storage token is not what separates companies there; the connect wall and the password rule are (§2.6).

Building on ArgonDB. You do not write an authorization predicate to keep tenants apart — there is nothing to get wrong, because there is no code path that reaches across in the first place. The corollary is the design constraint that matters most when you build a feature that legitimately needs two parties' data in view together: it cannot be a query that spans the wall on its own. It has to be a component that stands with a foot on each side and makes an explicit, recorded copy — which is exactly what Theme 5 (consent and grants) exists to do, and Theme 4 (views) is how you keep that copy usable once it has landed.

What you experience. Nothing to configure — the wall holds without a setting once tenant warehousing is on by default, and the catalog authorization engine it needs (OpenFGA) ships inside the release artifact on both Linux and macOS, so a multi-database box never meets a boot refusal for a missing engine and there is nothing for you to install (scripts/release/build-artifact.sh:593-611, scripts/release/build-artifact-macos.sh:519-523; docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md §2.6). You probe the wall, you don't build it: argondb lake grant, argondb lake attach-clause and argondb lake revoke hand out and take back a reader's token (crates/argondb-ctl/src/main.rs:1235-1275; docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md §4.3); a revoke removes the catalog permission, and a token already issued still logs in until it expires but can no longer read anything (crates/argondb-ctl/src/main.rs:1246-1251). The refusals are specific and are part of the interface, not an edge case: a bucket access across the boundary answers 403 AccessDenied, naming the key and both buckets; a catalog warehouse you cannot see answers 404, never 403, and does not disclose its name at all (docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md §3.4, §3.5, §4.6); a view joining tables from two databases refuses cross-database-join, naming the remedy instead of a bare error (docs/2026-09-20-guide-04-continuous-materialized-views.md §3.4).

3. The lake as the shared read side, the tail as what feeds it

In plain terms: every change you make in the transactional database shows up, seconds later, in a second read-only copy built for big questions — without you writing a sync job. It's like a photocopier standing next to your filing cabinet that copies every page the moment you file it, so someone else can flip through the copies without ever touching your originals.

Across the chapters. The thesis is stated once, in one sentence: "every Postgres table is automatically an Iceberg table, at a known LSN, with sub-second freshness and zero impact on transactional performance" (DESIGN.md:36). Guide 03 is the mechanism: the tail decodes the write log once — "the single decode path" — and the ring, the changelog, the lake copy and every reader all feed from that one decode, which is what makes "the lake cannot drift" a structural fact rather than a hope (docs/2026-09-20-guide-03-the-lake-and-its-process-model.md §2.3). A read at position L is Iceberg@W ∪ tail(W, L] — the durable copy up to the watermark W, plus the tail's own decoded changes above it (docs/2026-09-20-guide-03-the-lake-and-its-process-model.md §1, §2.9). Guide 02 explains why the tail can sit resident beside a stood down tenant without waking it: with the compute asleep there is no new WAL to decode, so the tail idles at baseline (docs/2026-09-20-guide-02-the-per-tenant-unit-gate-wake-and-stand-down.md §2.3). Guide 04's view maintainer is a second consumer of the exact same watermark — it advances a view's answer only when the tail's own freshness action reports the watermark moved, never on a timer of its own (docs/2026-09-20-guide-04-continuous-materialized-views.md §2.2). The changelog the tail writes beside a published table is not optional bookkeeping — it is the same decode path, so an audit trail read from it is complete by construction rather than by discipline (docs/2026-09-20-guide-03-the-lake-and-its-process-model.md §2.6). And the lake has exactly one writer per territory, which v1.5.6 now enforces per table as well as per namespace: the committer alone writes the mirrored br_<branch> tables, ArgonDB's lake writer alone writes the derived dv_<branch> namespace (continuous views, lake-landed transforms and appends), and inside dv_<branch> each table has exactly one writer — a view's table is written only by the view maintainer, and the view maintainer never writes a table that is not a view (docs/2026-09-20-guide-03-the-lake-and-its-process-model.md §2.7; table_writer_refusal, crates/argondb-committer/src/lake_writer.rs:2310).

Building on ArgonDB. A reporting feature is not a project with its own ETL job, its own schema mapping and its own reconciliation step. It is a SELECT over a copy that is guaranteed structurally consistent with the transactional table, because both descend from one decode of one write log. The design question that actually matters is which tables to publish and at what freshness tolerance you can afford to read them, not how to keep a second system in sync with the first. A result you compute from the lake — a transform or a view — can land straight back into it, in dv_<branch>, with no Postgres transaction and no --pg-url (argondb transform run --landing lake, crates/argondb-ctl/src/main.rs:2051-2066); pick a name no other door already owns.

What you experience. Publishing a table is one command with no connector to configure: argondb lake publish <table> sets an argon:publish marker, backfills once at a pinned position, then follows incrementally (DESIGN.md:406; the same command run, docs/2026-09-20-guide-03-the-lake-and-its-process-model.md Example 1). You never write a CDC connector, define a Kafka topic, or run a reconciliation job. argondb lake list shows you exactly where each table stands — lake @ 0/1BD2748, backfill pending, or QUARANTINED @ ... — and freshness is a number you read, ingest_lag_seconds, never a promise you take on faith (crates/argondb-ctl/src/lake.rs:300-318; ingest_lag_seconds, crates/argondb-tail/src/metrics.rs:1557). The one refusal worth knowing in advance: a column type the decoder cannot handle is accepted and stored durably but never arrives in the lake, silently, with no SQL error — which is exactly why a publish_status check exists as a separate, explicit thing to run (docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md §4.7). The second: landing into a dv_* table another door owns — a lake-landed transform onto a view's table, say — refuses not-owner and writes nothing: "every dv_* table has exactly ONE writer; nothing was written. Land under another name, or remove {target} through the door that made it" (crates/argondb-committer/src/lake_writer.rs:1477-1480).

4. Views as a statement of what should stay current

In plain terms: instead of writing a script that re-runs a big report every night and hoping it finishes before someone asks for it, you write the report's definition once — "this is the number I want" — and the system keeps that number accurate for you, forever, updating only the parts that actually changed. You stop being the one who has to remember to press refresh.

Across the chapters. Guide 04 is the primary mechanism: a continuous materialized view keeps its answer as of a coordinate L_v and advances it by applying the change rows in (L_v, L_1] rather than re-running the query, and a reader gets the exact current answer or a named older one with its distance stated, never a silent guess (docs/2026-09-20-guide-04-continuous-materialized-views.md §1, §3.10). The definition surface is deliberately a function call, argon.create_view(name, sql, options), not a native Postgres MATERIALIZED VIEW — because Postgres owning the refresh is exactly the manual-refresh problem this exists to remove (docs/2026-09-20-guide-04-continuous-materialized-views.md §3.1). A classifier decides, from the query alone, which of four maintenance strategies applies — a keyed projection, an algebraic aggregate, a join, or (when none of those hold) a full recompute — and it over-approximates toward the safe, expensive option rather than the reverse (docs/2026-09-20-guide-04-continuous-materialized-views.md §2.6-§2.9, §3.5). The second mechanism, a transform, is the one-shot sibling of a continuous view — a SELECT computed once at a pin and landed with a mode (create, append, replace, drop), rather than kept current forever (argondb transform run, crates/argondb-ctl/src/main.rs:2044-2066; the same run over MCP, docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md §4.5). Because it is pinned, re-running it at the same coordinate gives the same output, where a hand-written batch that buckets against now() silently rewrites history when re-run after the fact. Both land in one of two places: the derived Postgres schema, reaching the lake through the tail, or the lake-owned dv_<branch> namespace directly; a view lands in dv_<branch> by default (docs/2026-09-20-guide-04-continuous-materialized-views.md §2.10, §2.11). The two compose directly: Guide 04's own worked example is a continuous view created over a table landed into a tenant's database from another party (docs/2026-09-20-guide-04-continuous-materialized-views.md §4.9) — Theme 5 and Theme 4 meeting in one example.

Building on ArgonDB. Per output, you choose one of three postures: an ordinary application table your own code writes; a transform, a pinned, reproducible one-shot SELECT for something tied to a business event like a period close; or a continuous view, a SELECT stated once that the system keeps current on its own. The fork that matters is whether the coordinate itself is the meaningful artifact (a transform, pinned at month-end) or the answer must never be allowed to go stale between requests (a view). Neither requires you to reason about incremental maintenance by hand — the classifier does, and it never guesses wrong in the unsafe direction.

What you experience. argon.create_view('name', 'SELECT ...', options) from SQL, argondb view create --name ... --sql ... from the CLI, or the create_view tool over MCP — the same body, three doors (docs/2026-09-20-guide-04-continuous-materialized-views.md §3.2). You never write a refresh cron job, never decide by hand whether a deleted row needs its group total decremented. Sixteen refusal kinds are closed and specific — cross-database-join, non-deterministic (a view over now() or random() is refused even for a recompute-class view), self-cycle, and others — each naming the offending construct and a remedy (ViewRefusalKind::ALL, crates/argondb-query/src/view_classify.rs:486; docs/2026-09-20-guide-04-continuous-materialized-views.md §3.4). A reader who wants an exact answer gets it or a stated refusal to the base data; the default staleness tolerance is zero (docs/2026-09-20-guide-04-continuous-materialized-views.md §3.10). A view's lake table belongs to the view: a lake-landed transform or append aimed at it refuses not-owner and writes nothing, so nothing but the maintainer can move a view's answer (Theme 3). Guide 04 keeps its own list of what is still red on this program, and names it rather than rounding it off (docs/2026-09-20-guide-04-continuous-materialized-views.md §9).

5. Consent and grants — how data crosses a boundary

In plain terms: a company's data never leaves its own room by an automatic back door. If another company is allowed to see part of it, that permission is written down — as a row in a table, in the room it came from — so anyone can later see exactly what was allowed, when, and to whom, and can take it back.

Across the chapters. Consent is a row that lives in the database of the party whose data crosses, written with that party's own ordinary Postgres privileges — never a config file and never an operator setting — and published, so the consent record reaches the lake beside the rows it governs. The general form, new since v1.5.5, is the mount grant: a database that wants another database to read one of its lake tables writes hub.mount_grant (grantee_database, source_schema, source_table), one row per table, with no wildcard, and a later CREATE TABLE is not shared by it (MOUNT_GRANT_DDL, crates/argondb-transform/src/hub_consent.rs:298-313). The lake's read plane then serves that one table to that one grantee out of the owner's warehouse, and refuses every other cross-database read by name (DatabaseScope, crates/argondb-tail/src/lib.rs:960-973). The state is not a boolean: grant_state() returns one of four facts — granted, no grant, revoked (with the instant), or granted under a contract version this build does not serve — because "not granted" and "revoked" and "granted under an unserved version" are different things for a reader of the log to know (GrantState, crates/argondb-transform/src/hub_consent.rs:104-121). The same predicate decides the older, flow-shaped grant table, hub.share_grant, which scheduled landings (argondb publication land|serve, argondb hub run) consult before they move rows from one party's database into another's (crates/argondb-transform/src/hub_consent.rs:40-48; crates/argondb-ctl/src/main.rs:2365-2400). The component that moves the data holds two identities without ever mixing them — it reads every tenant under one cross-tenant, read-only identity and writes as the box, and there is no flag that lets it read as the box on a walled database (crates/argondb-ctl/src/hub.rs:69-86). Guide 07 shows the same doctrine one layer up, at the level of what an agent is allowed to do rather than what data it can see: the admin mount's destructive calls require an explicit confirm=true and refuse, naming what would be lost, without it — "an agent proposes; the operator confirms" (docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md §2.3, §4.6; crates/argondb-mcp/src/admin.rs:17-20).

Building on ArgonDB. You never design an outbox table, a webhook, or a second replication path to move data between parties — that door is closed by design, not left to you to build carefully. Instead you decide what crosses: which table, to which database — and the owning party writes a consent row, in its own database, using its own ordinary write privileges. Because consent is data, it travels through the tail like any other row, so it is queryable, auditable and versioned the same way any other business fact is, not a side channel nobody can see.

What you experience. INSERT INTO hub.mount_grant (grantee_database, source_schema, source_table) VALUES (...) in the owner's database to grant a read; UPDATE hub.mount_grant SET revoked_at = now() ... to revoke it. A read of another database's table with no grant refuses cross-database lake read refused, naming the owner, the asker and the exact INSERT that would grant it; a read after a revoke refuses with a different sentence that says when the grant was revoked (cross_database_refusal, revoked_mount_refusal, crates/argondb-tail/src/lib.rs:924-960). Two limits, stated directly: a revoke takes effect within one schema-refresh cycle of the read plane, not at the instant of the UPDATE, so a read in between can still succeed (crates/argondb-tail/src/lib.rs:942-960); and revocation is forward-only — rows already landed by a flow grant stay where they landed. Pulling them back is a separate, deliberately destructive operator command, argondb hub recall, and it refuses while the grant is still in force, naming the UPDATE the granting party runs first (crates/argondb-ctl/src/hub.rs:1440-1458). A flow grant made at a contract version the running build does not serve refuses by name, ContractVersionUnserved, rather than landing nothing silently: the grantor did consent, just to something else (crates/argondb-transform/src/hub_consent.rs:115-120). argondb hub grants prints the current state of every flow grant and always exits 0 — it reports consent, it does not judge it (crates/argondb-ctl/src/main.rs:2554-2576).

6. Agents read through the same doors as people

In plain terms: an AI agent connecting to ArgonDB does not get a special side entrance. It gets handed a key to one specific door — the same kind of key you would hand a human analyst, a dashboard, or a batch job — and which door decides everything it can and cannot do from then on.

Across the chapters. Guide 07 is the primary mechanism: the MCP surface is ArgonDB's API, split into four mounts by trust tier — analyst, dba, admin, write — each with its own bearer token, its own port, its own bind default and its own backing Postgres role (docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md §1). "An operator composes an agent's capability by choosing which mounts it connects, so the risk analysis happens at connect time rather than by curating a tool list" (docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md §1). On the analyst mount, read-only is architectural, not a parser's opinion: the Postgres role is the boundary, and a write attempt through it fails with Postgres's own 42501 insufficient_privilege, the same error a human would get with the same grant (docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md §2.1, §4.9). v1.5.6 carries that same principle down to the company: every company now has its own MCP token for the analyst mount, and a request carrying it runs as that company's own read role, <key>_mcp, which can connect to that company's database and no other, and holds no write privilege at all (crates/argondb-mcp/src/company_token.rs:194-203; crates/argondb-ctl/src/mcp_company_token.rs:45-83). The box token is no longer the only credential an agent can be handed; it stays the operator's, with operator scope, by name (crates/argondb-mcp/src/lib.rs:6484-6490). Guide 04's rewrite rule makes no exception for who is asking: it is an optimizer rule installed by pin(), so the FDW scan on an analyst's compute, the MCP query tool, the CLI's own query path, and a transform runner's compute all inherit the identical view-rewrite treatment (docs/2026-09-20-guide-04-continuous-materialized-views.md §2.5). And Guide 03's lake-facing MCP mount is stated to be "always" reading through the same query plane as everything else, never a side channel of its own (docs/2026-09-20-guide-03-the-lake-and-its-process-model.md §2.11).

Building on ArgonDB. You never build a separate, elevated access path for AI. You compose an agent's entire blast radius by choosing which mount its process connects to — the same lever an operator already pulls to scope a human analyst, a read replica, or a batch job. An agent given the analyst mount is bound by exactly the tenant wall (Theme 2), the exact pin discipline (Theme 3), and the exact view-rewrite machinery (Theme 4) that bind a person using the same mount; there is no seventh theme reserved for "what agents get to skip." When the agent works for one company, hand it that company's token rather than the box's: its reach is then one database by construction, not by a prompt's good behavior.

What you experience. One claude mcp add per mount, each its own URL and token; a session runs schema_overview → pin → query (against that pin) → changes(since=<lsn>) to learn what has moved since — the same pin_id carried across the whole burst (docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md §7.2). You never write a SQL parser to keep an agent read-only — the grant system already does the whole job. A company's token is minted by argondb tenant create and written, never printed, to a 0600 file, <data>/tenants/<key>/mcp-token; argondb tenant mcp-token <key> re-issues it (the old one stops working) and argondb tenant mcp-token <key> --revoke revokes every token of that company and sets <key>_mcp to NOLOGIN (crates/argondb-ctl/src/tenant_create.rs:2079-2081, :2955-2975; crates/argondb-ctl/src/main.rs:677-686; crates/argondb-ctl/src/mcp_company_token.rs:90-94). The token is checked on every request against the live record, so a re-issue or revoke takes effect with no restart, and every response names the scope it ran under in an x-argondb-mcp-scope header, operator or company:<key> (crates/argondb-mcp/src/company_token.rs:53-56, :265-270). The limits, directly: a company token works on the analyst mount only — the dba, admin and write mounts keep the one operator token (crates/argondb-mcp/src/lib.rs:6240-6248) — and the two analyst tools that read box-wide state, search and publish_status, refuse under it by name, pointing at the operator (crates/argondb-mcp/src/lib.rs:4611-4621). A wrong bearer token on any mount refuses with 401 before any tool body runs — a forged, revoked or superseded company token each with its own sentence (crates/argondb-mcp/src/company_token.rs:102-124) — and a non-loopback bind on the admin or write mount with no token configured refuses to start the process at all, rather than starting it open (docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md §3.1).

How the six fit together

   one write log
        |
        v
   two shapes of one truth  --------->  the lake (theme 3)
   (Postgres table, Iceberg table)           |
        |                                    v
        v                              views state currency (theme 4)
   one database per tenant (theme 1)         |
        |                                    v
        v                              consent moves a copy
   the wall (theme 2)  <------------------- across it (theme 5)
        |
        v
   every reader goes through it the same way --- people and agents alike (theme 6)

Every theme above is a different vantage point onto the same handful of mechanisms: one write log, one wall, one pin, one classifier, one shape of consent row. An application built on ArgonDB is not five separate integrations — it is one mechanism, read from five directions, and this chapter is the map between them. Guides 01, 02, 03, 04 and 07 are where each direction goes deep; start with whichever theme above matches the feature in front of you.

Guide 1 — one database per tenant, and the wall between tenants

Audience: a senior developer who has to USE this. It explains the mechanism, states its limits and its failure modes, and gives worked examples. Every claim carries its source by path:line; a claim without one is a defect. Register: internal technical reference.

How to read the examples. Each one is labeled VERIFIED (run, with its real output pasted) or FROM SOURCE (the exact signature, flag, message or table quoted from the code or a design by path:line). This lane's box had no vendored PostgreSQL install and no built storage tier, so no box gate could run here; every example below is FROM SOURCE except where it says otherwise, and each names the gate that proves it on a box. The path:line coordinates are at commit c9415ae0a, except in §4.0 and §4.9-§4.12 and the rows and gotchas that cite them, which were re-read for v1.5.6 at commit e90da8c0c.

Labels the pictures use, and this guide uses with them. A solid outline is built and shipped. A dashed outline is designed and measured but not built. OPEN marks a decision still to be made. Where a source says UNCERTAIN or NOT MEASURED, this guide repeats it rather than rounding it off.

1. The diagram

The left panel is one PostgreSQL cluster on one box. A tenant is a DATABASE in it — Block 4 B [D-2026-09-11-1fc4b3], since amended twice over to B′ and then B′ (amended) [D-2026-09-20-0d5ba3] [D-2026-09-21-4107c5] — see §2.1 for the current wording — and the three columns are three member databases with the same shape: each has its own tables, its own SQL credential and its own lake key, its own place in the lake's address space, and its own published set. Two of the columns are distributors and one is a manufacturer, and the picture deliberately draws them identically: they are the same kind of thing, told apart by WHICH SCHEMA they publish in, not by a flag (crates/argondb-query/src/all_tenants.rs:91-112). Along the bottom of that panel are the three facts a box diagram cannot show: the provisioner runs a pass every 30 seconds and gives every new database its whole wall within one pass; the primary database is never a tenant and is never limited; and row-level security separates ROLES inside a database, never companies (docs/2026-09-04-erp-spec.md:328-351).

The right panel is the wall itself — Block 5, ruled D: A + B + C [D-2026-09-11-21a6d7], which means all three parts and not a choice among them. A is catalog authorization, so the catalog refuses to list or read a warehouse a caller is not entitled to and does not even disclose its name. B is a login in front of that catalog, because authorization with no authenticated subject authorizes nobody and the box would not be able to bootstrap its own catalog (docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:611-622, :790-794). C is a warehouse, a bucket and a credential per tenant, which is the only one of the three whose boundary is also a STORAGE boundary — the measurement behind that is that an Iceberg namespace has no storage prefix of its own, so there is nothing below the warehouse for a key to be scoped to (:168-189). Below those three sit the five principals and what each holds, and the bottom band is the one way rows cross: the box reads every tenant under one read-only identity, makes a COPY into the receiving party's own area, and the receiving party reads that copy under its own key. No query reaches across.

2. What it is

2.1 One PostgreSQL cluster, and a tenant is a database in it

The topology is a ruling, not an implementation detail: hundreds of tenant databases share one compute, one write-ahead log, one lake arm and one catalog (docs/2026-09-04-erp-spec.md:89-100). What that boundary buys is stated exactly at docs/2026-09-04-erp-spec.md:135-142: "There is no query in this system that could return another company's row, because another company's rows are not in the database being queried." That is a property of the topology, not of every predicate in every code path.

The topology ruling has been amended twice since, and the amended wording is the current standing rule — not what a box runs by default. [D-2026-09-20-0d5ba3] ratified "B′": each tenant database is served by its OWN compute that stops when idle and wakes on connect, rather than sharing the box's one compute. [D-2026-09-21-4107c5] amended B′ again, the same evening: a tenant is a database that is its own Postgres instance on its OWN TIMELINE with its own WAL; "one cluster" now names the SHARED STORAGE PLANE — one pageserver, one safekeeper and one lake serving every tenant's timeline — not one shared write-ahead log. So of the four things this section's first paragraph names as shared, two no longer are: compute and the write-ahead log are ruled PER TENANT database. That is the standing topology; it is ratified and built, default off (§5; crates/argondb-ctl/src/tenant_compute.rs:105, HISTORY#per-tenant-compute-table), and the [compute] per_tenant switch that turns it on is the operator's own setting, never a system default [D-2026-09-23-5122a6]. Everything else in this chapter — the worked examples in §4 included — describes the box with that switch off, which is what ships today.

Two coordinates the picture could not carry. First, what counts as a database is one WHERE clause, spelled once: datallowconn AND NOT datistemplate (crates/argondb-ctl/src/lake_tenant.rs:787), and the tenant wall, the estate provisioner and publish discovery all use it, so the three cannot disagree about what a database is (crates/argondb-ctl/src/transform_estate.rs:59-63). Second, the primary database — the one --source-url names, postgres on a box — is not a tenant: it keeps the box warehouse argon and the box credential, it is never given a connection limit, and it is excluded from every tenant principal by the same predicate that refuses to grant on it (crates/argondb-ctl/src/lake_grant.rs:50-68).

Formerly OPEN, now ruled and built behind an operator's switch — the picture still draws it open. Whether hundreds of tenants keep sharing one compute or each tenant unit gets its own is no longer undecided: B′ (amended) rules that each tenant gets its own compute on its own timeline [D-2026-09-20-0d5ba3] [D-2026-09-21-4107c5]. What stays a live question is WHEN a given box turns that on — and that is the operator's call, never the system's [D-2026-09-23-5122a6]. Deep dive 2 covers the per-tenant unit's gate, wake and stand-down.

2.2 The member database, and what makes one a manufacturer

Each column of the left panel is a member database. A distributor's ERP tables live in its own schemas (public, and whatever the application uses). A manufacturer's published tables live in mfr. The hub's control tables live in hub, and what the hub lands back lands in derived. That is the whole distinction: all_tenants.<schema>__<table> unions every member database's copy of a name, so without a schema rule a manufacturer holding public.sales_order_lines would be read as a distributor by every flow (crates/argondb-query/src/all_tenants.rs:91-102).

The rule has a sharp edge worth stating at the start. The mfr schema is ESTATE: the per-database provisioning pass creates it in EVERY member database, distributor and manufacturer alike, because there is one provisioner path and no manufacturer-shaped branch in it (crates/argondb-ctl/src/transform_estate.rs:28-34). So holding the schema makes nothing a manufacturer; PUBLISHING in it does, and the product asks that question of the copies a pin composed rather than of a policy list that could disagree with the data (crates/argondb-query/src/all_tenants.rs:104-111).

2.3 Its role and its key — one SQL credential, one lake key, 20 connections

The "20 connections (fixed limit)" in the picture is DEFAULT_CONNECTION_LIMIT = 20 (crates/argondb-ctl/src/lake_tenant.rs:596), and it is a ruling: every tenant database gets the SAME number, the operator raises it per tenant, and nothing re-fits it as tenants come and go — the even-share alternative ((max_connections - reserved) / tenants) was considered and not chosen (crates/argondb-ctl/src/lake_tenant.rs:119-125). It is PostgreSQL's own ALTER DATABASE … CONNECTION LIMIT, so its semantics are PostgreSQL's: superusers are exempt, and the count is of connections to THAT DATABASE, not of the role (:136-141).

The lake key is argon-tenant-<bucket>, minted by the provisioner into lake-store-credentials.json under the data directory. The access key is readable on purpose and is not a secret — it appears in the store's refusal lines and in a SigV4 credential scope, and an opaque one would make every refusal a lookup (docs/reserved-identifiers.md:6732).

2.4 Its identity in the lake — database.schema.table

A published table's address has FOUR parts — branch, database, schema, table — and the database is always present, the box's own postgres included (docs/lake-foreign-read.md:430-439). SQL gives a reader three name parts, so the first three are written as one quoted identifier: lake."br_main.shop.sales".orders. That is not a style choice: without the quotes the engine sees four name parts and stops at the parser with Parser Error: NameListToString NOT IMPLEMENTED, which reads like a broken query rather than a quoting mistake (docs/lake-foreign-read.md:449-456, and the measurement it comes from is docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:220-226).

2.5 Its published set — a table comment, and the default is nothing

Publication is opt-in per relation and the marker is a table comment: COMMENT ON TABLE sales.orders IS 'argon:publish' (docs/upc-hub-recipe.md:470-478). Nothing is published by default. The consequence for this guide is the ordering one: a database that exists and has published nothing still gets its whole wall on the next provisioner pass, because provisioning is keyed on the DATABASE and not on the first publish (crates/argondb-ctl/src/lake_tenant.rs:22-32).

2.6 The wall's three parts, and which of them is on today

A and B are ONE landing, not two phases, and the reason is upstream's own sentence: "Authorization is only effective if Authentication is enabled" (docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:616). With AUTHZ_BACKEND=openfga and no authentication the catalog is not a wall with a door — it is a locked door with no key cut, and the measurement says so exactly: anonymous bootstrap 401, anonymous warehouse creation 403, anonymous listing 403 (:780-794).

Both platforms bundle it, named rather than implied: the release artifact stages bin/openfga on Linux — scripts/release/build-artifact.sh refuses the build if its license text is missing (:243-246) and then stages the binary itself (:584-591) — and on macOS — scripts/release/build-artifact-macos.sh runs the same license gate (:223-226) and stages the binary through the shared scripts/release/lib/openfga-bundle.sh leg (:519-523) against the openfga-darwin-arm64 content pin. So neither platform's stock multi-database box meets a boot refusal for a missing engine (docs/2026-09-15-cross-tenant-read-design.md:513-516).

A fourth surface, not one of these three: the storage plane underneath them. Parts A–C wall the LAKE. The OLTP side has its own wall, and on 2026-09-23 the isolation gate measured a real gap in it: the box's pageserver ran with no credential check at all, so a credential-less process on loopback read any company's timeline under the box's one pageserver tenant. The box now signs a NeonJWT for every compute, the tail and its own management calls, and the pageserver refuses a caller carrying no token or the wrong one — closing that credential-less read. Every company is still a TIMELINE of the box's ONE pageserver tenant, so the token also names a timeline: the pageserver token carries an ARGON: timeline_id claim (storage/pageserver/src/auth.rs), compute_token(tenant, timeline) mints each compute's token for its one timeline (crates/argondb-ctl/src/pageserver_auth.rs, called at crates/argondb-ctl/src/serve.rs:13567), and the pageserver refuses that token on any other company's timeline — HTTP 403 on the pageserver's HTTP port and a JWT authentication error on the page service (basebackup and pagestream) — and logs pageserver auth: timeline mismatch. This is built [D-2026-09-23-b822c8], merged 2026-09-24 (b2743bdc9). The W5− and W5+ cases of testing/box/cross-company-isolation-eight-surfaces.sh assert it: A's compute token is refused on B's timeline on both ports, and the same token reads A's own timeline. Two limits, stated plainly. With [compute] per_tenant off, the default, one compute serves every company's database and its token names the box's own timeline, so what separates companies there is the connect wall and the password rule, not the storage token. And the box's own processes — the tail and the management client — hold a pageserverapi token with no timeline, which reaches every timeline; no company is ever given it. The pageserver must be built with this change: an older pageserver treats the new token as tenant-wide.

2.7 The five principals

Full detail, including which operation each may perform on which bucket, is sub-diagram 1b below (§7.2). In one line each:

• the box — the root store key and the argondb-box catalog subject: every warehouse, and it WRITES. The committer is this principal. The operator is inside every wall, and this guide states that rather than implying otherwise.
• a tenant — argon-tenant-<bucket> at the store and argondb-tenant-<database> at the catalog: one bucket, one warehouse, one select. This is what a tenant's application and its people use.
• the cross-tenant reader — argon-reader / argondb-reader: GET, HEAD and LIST on every tenant bucket, never the box bucket, and it cannot write anything anywhere.
• a manufacturer group — hub-<m> at the store and argondb-manufacturer-<m> at the catalog: its own hub bucket only, never a distributor's warehouse.
• argon_transform — the PostgreSQL role, the only writer of derived.* in every database.

What never exists (the red box in the picture): a principal spanning two tenants with write, and a crossing that is not a copy the box made.

2.8 all_tenants.<schema>__<table> — the reader's view

Under the cross-tenant reader, every same-named table across tenant databases is served as ONE view: the union of every tenant's copy plus a tenant column naming the tenant DATABASE (never the warehouse name — the two differ under a [lake.warehouse_overrides]) (docs/reserved-identifiers.md:39). It is a view, not a copy: each query expands to one scan per tenant of that tenant's own Iceberg files, the WHERE is pushed into every scan, and branches run in parallel.

Two properties the picture had room only to name. There is no tenant list anywhere — the roster IS the estate, re-derived on every provisioner pass from the same predicate the writer resolves a database to its warehouse with, and a config key naming tenants was explicitly withdrawn (docs/2026-09-15-cross-tenant-read-design.md:207-218). And an unwalled database is NAMED absent, never silently included or silently dropped: the roster carries it as absent: <db> — <reason> and every session banner prints it (:232-238).

2.9 Row-level security, and where it stops

RLS stays inside a database and separates roles, not companies (docs/2026-09-04-erp-spec.md:328-334). Its blast radius changed with the topology: a defective policy predicate used to be a cross-customer breach and is now an over-permissive grant inside one company (:344-351).

The residual the picture draws is real and current. A table's LAKE COPY has no row policies — PostgreSQL will not let a policy be attached to a foreign table at all, and Iceberg has no row-policy concept (docs/lake-foreign-read.md:644-649). Since 2026-09-12 the READ PLANE refuses a scan of a protected table's lake copy rather than serving it, naming the table, the role and the remedy (:651-661) — but that wall is on the Flight door only. A reader holding the catalog URL and the store credentials reads every row of every published table, row security or not (:670-680). Block 6 ruled that the Postgres role will travel into the read plane [D-2026-09-11-c64ea4]; it has not traveled into the hub's in-process read yet, and the picture lists that as one of its two residuals.

2.10 What is box-granular, and what is per tenant

The granularities below are what a box with [compute] per_tenant OFF gives you — today's shipped default. The ratified target topology moves the first three rows to PER TENANT once an operator turns per-tenant compute on [D-2026-09-20-0d5ba3] [D-2026-09-21-4107c5] [D-2026-09-23-5122a6].

(docs/2026-09-15-cross-tenant-read-design.md:585-600.) So "a tenant cannot be rewound on its own" is true of the OLTP side and false of the lake side: a scoped pin reads one tenant at any LSN in the retention window today, without touching the timeline. Per-tenant timelines need a separate Postgres instance per tenant, because PostgreSQL has no per-database write-ahead log to branch (:576-583) — which is exactly what the topology ruling has since made the standing design, and it is built, default off: each tenant database is its own Postgres instance on its own timeline, sharing only the storage plane [D-2026-09-20-0d5ba3] [D-2026-09-21-4107c5], under [compute] per_tenant (crates/argondb-ctl/src/tenant_compute.rs:105), with the tail following every tenant timeline as its own stream (crates/argondb-tail/src/streams.rs; HISTORY#tail-multiplexes-tenant-timelines). The switch is the operator's, never a system default [D-2026-09-23-5122a6]; with it off, every row above still holds.

3. How it works

3.1 One provisioning pass, start to finish

Who runs a pass. Two callers, one function. The box's cadence loop runs ensure_all every DEFAULT_INTERVAL_SECS = 30 (crates/argondb-ctl/src/lake_tenant.rs:155, :2303), and argondb lake tenant provision runs the same function on demand (:2575). It matters that it is one function: on 2026-09-16 a hand provision run raced the box's own cadence pass over the same 72 new databases and produced 409s on the reader's catalog grants and a non-zero CLI exit while the box's pass finished the estate (:157-170).

The pass lock. Every cluster pass now holds one SESSION-LEVEL PostgreSQL advisory lock on the primary for its whole duration (crates/argondb-ctl/src/lake_tenant.rs:1458-1466). The key is i64::from_be_bytes(b"argon_tw") = 7021788497314804855 — the eight ASCII bytes read big-endian, so the number is derivable by hand and is not a magic literal (:172-176; registered at docs/reserved-identifiers.md:6734). Three properties follow from choosing a session lock: a dead holder frees it without a reaper, pg_locks joined to pg_stat_activity names the holder so a waiter can say who it is waiting for, and the primary is the one coordinate both parties already reach. A waiter waits PASS_LOCK_WAIT_DEFAULT_SECS = 900 by default (:183), overridable with ARGONDB_TENANT_WALL_LOCK_WAIT_SECS (:179), and refuses on the deadline naming the holder and stating that nothing was provisioned. Fifteen minutes is not arbitrary: an 80-tenant pass under the wall was measured in the minutes on a loaded box (:180-182).

The order, per principal, and why it is this order (crates/argondb-ctl/src/lake_tenant.rs:34-50):

1. The credential is minted and written FIRST. The store watches that file and creates a bucket directory for every bucket it names, so the credential is also the bucket's birth certificate — one owner of the store's layout.
2. The box creates the bucket directory itself too, on this process's clock, so step 3 does not race the store's reload. Both paths are create_dir_all and idempotent.
3. Only then the warehouse, because Lakekeeper VALIDATES a storage profile by writing and reading a probe object in the bucket — into a bucket that does not exist, that validation fails.
4. Success is asserted from POSITIVE EVIDENCE — the warehouse is read back out of GET /management/v1/warehouse — never from the POST falling through, which a re-provision answers 400 to.

The warehouse profile carries the ROOT credential, not the tenant's (:70-78). Lakekeeper holds the storage credential for the whole warehouse and uses it for its own probe writes and maintenance; a profile carrying a bucket-scoped key would work today and break the moment Lakekeeper needed to touch anything else. The tenant key is for the tenant's READER — DuckDB, Trino, a raw s3:// client — which is exactly the reader part C exists to wall, because the catalog vends an ENDPOINT and never key material.

Every row is a statement, and a refusal is per database. Outcome has four arms and nothing falls through silently: AlreadyProvisioned, Provisioned, BoxWarehouse, Refused(reason) (crates/argondb-ctl/src/lake_tenant.rs:425-435). One unprovisionable tenant never stops the others — that is written into the function's error contract (:1451-1457) and gated at both polarities by testing/box/tenant-wall-warehouse.sh P9, which puts two legal databases and one illegal name (zz_bad_name) into one pass and asserts two walled, one named with its remedies, and a non-zero exit for the named one only (testing/box/tenant-wall-warehouse.sh:58-61).

Two principal kinds, one function. ensure_principals (crates/argondb-ctl/src/lake_tenant.rs:2199) is what a database and a manufacturer group both go through; ensure_all is it over the cluster's databases and hub::provision is it over the ownership table's manufacturers (:52-62). The credential file's database field carries the principal's STORE KEY — a database's own name, a manufacturer's prefixed one — so the two kinds cannot collide in it, and WarehouseMap::resolve_estate refuses an estate in which they would (:64-69).

3.2 The two cluster-wide halves of the same pass

Both are keyed on DATABASES rather than on principals, which is why they live in ensure_all_locked and not in ensure_principals — the hub also calls ensure_principals over manufacturer groups, which have no pg_database row to limit (crates/argondb-ctl/src/lake_tenant.rs:1480-1484).

The OLTP half — ensure_connection_limits (:830). The decision is pure and unit-testable without a PostgreSQL (plan_connection_limit, :755), and it is three rules in order (:749-753):

1. an explicit override is applied EXACTLY, whatever is there;
2. the default is applied only over NO_LIMIT_SET (-1, PostgreSQL's own sentinel, :600);
3. everything else is kept as found — a limit an operator set by hand is never lowered, and never raised either.

Three rungs resolve the wanted value, top wins: --connection-limit <db>=<n>, then [lake.tenant_connection_limit].<db> in argon.toml, then the default (ConnectionLimits::resolve, :651). An override naming a database that is not in the cluster is REPORTED rather than silently ignored (overrides_naming_no_database, :661-678). A value outside PostgreSQL's domain is refused by name at the boundary rather than handed to ALTER DATABASE to refuse in its own words (validate_connection_limit, :687), and the argument parser splits on the LAST = so a database name holding = still parses (parse_connection_limit_override, :706). LimitOutcome has four arms — Set, Overridden, Kept, Failed — and again nothing falls through (:730-744).

The reader half — ensure_reader (:1693), run AFTER the tenants are walled, so a tenant this pass minted is in the roster this pass writes (:1485-1491). Its roster is reader_plan (:1537), which asks WarehouseMap::resolve_estate_partitioned — the same predicate the provisioner walls with, called rather than re-stated — and keeps every database resolving to a NON-box warehouse, in database order (:1541-1554). Every database the resolver refused is carried with its reason (:1555-1562). The store half is one ReaderCredential in the same credentials file at version 2, its scope rewritten whenever the roster moves, so the key FOLLOWS tenants with no restart and no re-mint (the store re-reads the file by inode). The catalog half is one select assignment per tenant warehouse, DIFFED against the granted record the previous pass left in the file: a new tenant is granted on the pass that walls it, a dropped tenant is revoked on the pass that finds it gone, and every change is one log line (:103-107).

No wall, no reader (:108-112). With [lake].warehouse_per_database off, every database resolves to the box warehouse, so a reader would resolve nothing; the pass mints nothing and REMOVES a reader an earlier configuration left behind. reader_plan returns None for that case and its doc comment is explicit that None is a REFUSAL and not an empty roster (:1529-1532) — there is no cross-tenant read without a wall, and the product says so rather than handing out the lake under a read-only label.

A reader failure is a row, not the end of the pass. The tenants are already walled; the reader is retried next cadence, and the failure is one logged line (:1490-1500). Every outcome refreshes the roster cache, Unchanged included, because a pass that PROVED the record current is as good a reason to restart the window as a rewrite (:1589-1594).

3.3 Which warehouse a table publishes into, and what happens when it is not there yet

The committer resolves a catalog handle PER TABLE, from that table's own database (crates/argondb-committer/src/catalog_set.rs:226-239). When the resolution fails — the commonest reason being that the database was created seconds ago and its warehouse does not exist yet — the path refuses and defers by name. It never falls back to the box warehouse. The refusal text is one string, lifted out so it is testable without a catalog (unreachable_warehouse_message, :243-257), and it says why:

database "tenantc" publishes to lake warehouse "tenantc" (bucket "tenantc"); it is not
reachable. ArgonDB does NOT fall back to the box warehouse "argon" here: that would put
this database's published bytes behind the box credential, which is the tenant wall
inverted. If the database was just created, the box's tenant-wall pass provisions its
warehouse within one cadence interval — `argondb lake tenant list` names the state and
`argondb lake tenant provision` forces a pass.

The tail retries every cadence, so a deferral is a stable repeated log line and not a one-shot message. Within one 30-second pass the wall exists and the next retry publishes into the tenant's own bucket. And the ordering is stronger than the requirement: provisioning happens on first DATABASE, not on first publish, so in practice the wall is up before there is anything behind it — that substitution is named on the spot rather than made silently (crates/argondb-ctl/src/lake_tenant.rs:22-32).

3.4 The store's one predicate

wall::decide(creds, access_key, op, target) is pure, total, and the only rule in the store about who may touch which bucket (crates/argondb-lake-store/src/wall.rs:117-191). It sits where it does for a measured reason: s3s 0.14.1 calls S3Access::check ONCE per request, after the signature is verified and after the route is resolved, before the operation's input is deserialized — so one implementation covers roughly a hundred operations including ones this store does not implement and ones a future s3s adds. Guarding each method by hand would be about twenty edits today and a silent hole on the next operation added (crates/argondb-lake-store/src/wall.rs:12-22).

The one thing the request path does not carry is a COPY's SOURCE bucket — s3_path is the destination. s3s generates a per-operation hook for that case and calls it with the deserialized input, so BucketWall::copy_object reads copy_source and applies the SAME predicate (:24-30). A cross-bucket copy is how a wall that guards only the destination is walked straight through.

The arms, in the order the predicate tests them:

• no access key → deny: anonymous request refused: <op> needs a signature on this store (:126-130).
• Principal::Unknown → deny, naming the key (:133-135).
• Principal::Root → allow, unconditionally (:132).
• Principal::Scoped { bucket, database } → the asked bucket must equal the scoped bucket; a mismatch is denied with the key, its bucket, its database and the bucket asked all named. At the store ROOT only ListBuckets is allowed, and it is FILTERED to the caller's own bucket (:136-152).
• Principal::Reader { buckets } → a bucket outside the roster is denied by name, with (the box bucket never is) in the line; a bucket inside the roster is allowed only for an operation in READER_OPS, and every other operation is denied naming the allowed set (:161-189).

READER_OPS is CLOSED and holds exactly seven names — GetObject, HeadObject, ListObjectsV2, ListObjects, HeadBucket, GetBucketLocation, GetObjectAttributes (:78-86). Anything not in the list is refused by name, so an operation s3s adds later does not arrive allowed. CopyObject is deliberately absent: a copy WRITES its destination, and the reader may write nothing (:71-77).

Why ListBuckets is filtered rather than refused (:32-40): refusing it outright would be a wall too, and it would break every S3 client that probes the bucket list before addressing a bucket. Filtering is the answer that is both a wall and usable. Every OTHER root-level operation is refused by name — a closed default.

3.5 The catalog's half

The catalog half is one select assignment per (subject, warehouse), written through POST /management/v1/permissions/warehouse/{warehouse_id}/assignments with {"writes":[{"type":"select","user":"oidc~<subject>"}],"deletes":[]} — read out of Lakekeeper's own OpenAPI document rather than guessed, and exercised end to end before it was built (crates/argondb-ctl/src/lake_grant.rs:13-20). select is READ access to the warehouse: the tenant lists and reads its own tables and cannot write them, which is the shape the wall is for, since the committer is the only writer and writes as the box (:35-41). Revoke is the same endpoint with the subject moved from writes to deletes (:22-25).

Four measured behaviors a developer will meet:

1. A warehouse the caller may not see answers 404, not 403. The name is not disclosed (docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:798-800, and the C7d/C7e rows of the measurement at :833-834).
2. lakekeeper migrate with the backend set creates the OpenFGA store and writes the whole authorization model itself — no provisioning step of ours (:795-798).
3. The box mints its own tokens. Lakekeeper never calls an identity provider; it fetches <provider>/.well-known/openid-configuration, then the JWKS, then verifies RS256 signatures. The box holds the signing key and serves those two documents, so the wall costs one new service (the policy engine) and no identity provider (:813-822, :846-849).
4. Revoking a grant does not invalidate a token. It removes the assignment, so the principal is refused from the next request onward; the JWT itself is valid until it expires and the box keeps no revocation list. With the grant gone the token authenticates and authorizes nothing, and --ttl bounds the other half (crates/argondb-ctl/src/lake_grant.rs:281-287).

And one refusal that is a design statement rather than a validation: lake grant REFUSES a database that resolves to the BOX warehouse, because the box warehouse holds every other database's tables as well and "granting" there would hand the caller the whole lake while reading like a per-tenant permission (:45-68).

3.6 A read session that resolves every tenant

pin is the one home, and converting it converted four surfaces at once: the transform runner, the hub, the MCP pin/redeem path and the Flight read plane; the FDW reads through Flight and follows (docs/2026-09-15-cross-tenant-read-design.md:356-360). What changed is the type: QueryOpts.lake became the principal's CatalogSet — the plural handle the WRITER already had — so a session registers every committer table across every warehouse its principal may read, each table's provider built from the catalog handle its own database resolves to (:290-296).

The principal decides the reach (:311-317): the box resolves every warehouse; a tenant resolves exactly its own; the cross-tenant reader resolves every TENANT warehouse — CatalogSet::warehouses minus the box's, with absent tenants named.

Read-only means no writer in the session (:345-354). hub run is one process holding TWO principals: the reader for reads and the box for writes, with the run banner naming both. Splitting them is the point.

The consistency claim, stated exactly (:324-343). A pin is (branch, LSN, scope, …) and the LSN is a position in the cluster's ONE write-ahead log, which every tenant database shares — so one LSN names one instant for every table in every warehouse. Per table the guarantee is unchanged. What is NOT claimed is cross-warehouse atomicity of a cut: a cut spanning two tenants is one transaction PER WAREHOUSE, so two warehouses' watermarks may advance at different moments. A cross-tenant pin therefore lands at the newest LSN every scoped table in every tenant warehouse serves, and the session discloses per warehouse which coordinate it resolved.

This claim's "one write-ahead log" premise is the shipped default, not the standing topology. With [compute] per_tenant on, the ruled target gives each tenant database its OWN write-ahead log on its OWN timeline [D-2026-09-20-0d5ba3] [D-2026-09-21-4107c5]; one LSN naming one instant for every table in every warehouse holds today because that switch is off by default and is the operator's to flip [D-2026-09-23-5122a6].

3.7 The view, and what it costs

all_tenants.<schema>__<table> is composed at pin time from the cached roster and the per-warehouse table listing. The name is the flat key minus its database, grouped STRUCTURALLY by (schema, table) rather than re-parsed out of the name — the tenant-relative key is not injective (crates/argondb-query/src/all_tenants.rs:152-159). Each branch is SELECT '<tenant>' AS "tenant", <the view's columns, or NULL AS "col" where this copy lacks it> FROM "<flat name>", UNION ALLed in tenant order, columns unioned BY NAME in first-seen order (docs/reserved-identifiers.md:39).

Pushdown is the engine's and is not re-implemented. A WHERE over the view is pushed by DataFusion through the Union into every branch and on into each branch's IcebergTableScan, where manifests, files and row groups are pruned against that tenant's own statistics. Because tenant is a per-branch CONSTANT, WHERE tenant = 'x' simplifies to false in every other branch and the optimizer drops those branches before a file is opened — that is how "subsetting is the query's WHERE" costs nothing over a named set (crates/argondb-query/src/all_tenants.rs:34-45).

Drift has a voice twice (:47-58). A missing column is NULL AS "col" on that tenant's projection line beside its 'tenant' AS tenant, so EXPLAIN names the tenant and the column together; and every drifted view is listed in the served warning rendered at the mint and on every result the session serves, so a client that never reads a plan still sees it. A tenant with NO copy contributes no branch and is named the same way. A column whose TYPE differs is named per tenant with both types: the engine's UNION coercion widens it when it can (Int32 with Float64 reads as Float64) and the view is REFUSED BY NAME when it cannot (Int32 with Boolean) — never silently narrowed. A copy that already carries a column named tenant is refused the same way, because the projected literal would shadow it (:60-64).

The measured cost (docs/lake-foreign-read.md:386-402, re-measured 2026-09-19 on testing/box/all-tenants-views.sh P5):

The split is the finding: roughly 70 % of a pin is the catalog WALK — one namespace_exists plus a list_tables and a list_namespaces at each of the branch, database and schema levels, so seven round trips per warehouse against one metadata load per table (about 29 ms per round trip at 80 warehouses). Narrowing the table set buys the smaller half; narrowing the walk buys the larger one. Earlier figures on this line — 17122 ms at 8 and 152654 ms at 80 — DO NOT REPRODUCE, and nothing in between was a deliberate optimization of this path; they are left in that document as the record and are not a baseline (:404-409). A reader sizing a deployment should re-measure.

A scoped pin (argondb-query --scope) walks only the namespaces its names could live in — three round trips instead of seven for a warehouse holding no database the scope names. In a cross-tenant estate it prunes nothing, and that is measured rather than assumed: the scope a cross-tenant read can name is all_tenants.<schema>__<table>, which names no database, so every database level is admitted. The lever bites where a scope names FEWER DATABASES than the roster has (docs/lake-foreign-read.md:415-428).

3.8 The three spellings of one identity

Four coordinates, three SQL name parts. The shape was ruled in three blocks: full nesting [D-2026-09-11-702e69], the primary regularized [D-2026-09-11-118b2e], __ kept as the separator [D-2026-09-11-f73865] (crates/argondb-ctl/src/lake_address.rs:19-21). So postgres.public.orders on branch main has a catalog identifier of namespace ["br_main","postgres","public"] with table orders, an engine address of lake."br_main.postgres.public".orders, a flat lake key of postgres__public__orders, and a dotted log label of br_main.postgres.public.orders.

Those are four different jobs and no one of them can stand in for another (:47-78). address is what a DuckDB user TYPES, and its namespace is one quoted identifier because a nested namespace is readable no other way. lake-table is the FLAT key the tail's tables action, the FDW mount and a pin session use. leaf is the bare relation name a caller talking to the REST catalog directly needs. label is what a HUMAN and a LOG read — no catalog alias, no quotes — and it exists because address's quoting makes it useless as a log matcher.

Why this is a product subcommand and not a library function (:4-15): a shell gate, a shipped starter and a customer doc cannot call lake_table_name_for. The census measured what that costs — 452 lines in 133 files spell the lake address, and three of the consumer groups (12 lines of shipped starters, 132 lines in 57 gate files, 148 lines in 35 docs) are shell, SQL and prose. argondb lake address is the door they call, and testing/lib/lake-table-name.sh is its first caller.

What has no database coordinate stays at the branch level and in the box warehouse: _argon_gaps, _argon_timeline, and the lake writer's own dv_<branch> territory (docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:354-370; docs/lake-foreign-read.md:186-188). A tenant key does not reach them either.

A catalog namespace is a NAMING hierarchy, not a directory hierarchy — measured, and it is the reason part C exists. Every namespace the probe created reported the SAME storage location, and a table's location is the warehouse prefix plus the table's own UUID; the warehouse storage profile carries "storage-layout":null. A credential scoped to a namespace-shaped prefix would open nothing, because no such prefix exists (docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:168-189). One thing stays UNVERIFIED in that measurement and this guide repeats it rather than rounding it: whether 0.13.x offers a non-default storage-layout that nests objects by namespace — the field exists in the profile and was not probed (:187-189).

4. How to use it

Thirteen worked examples. Each states its label and, where it is FROM SOURCE, the path:line the command, flag, message or sample output is quoted from, plus the gate that proves it on a box.

4.0 argondb tenant create — the whole tenant, in one ordered command — FROM SOURCE

This is the command to reach for when you are creating a COMPANY, rather than just a database. §4.1 below is still exactly right and still supported — it creates the database and lets the wall pass find it — but it leaves the tenant's timeline, its compute row, its login role, its uniform schema, its identity record, its application process and its MCP token to you. argondb tenant create runs all of those in the order the ERP spec §13.4 forces — the project owner's ruling on who creates a company and in what order [D-2026-09-21-e781f5] — and it is safe to re-run:

argondb tenant create acme \
  --display-name "Acme" \
  --legal-name   "Acme Holdings LLC" \
  --system       distributors \
  --identity     "tax_id=12-3456789" \
  --identity     "contact=ops@acme.example" \
  --data-dir "$DATA" --port 5433

(crates/argondb-ctl/src/main.rs:611-670.)

The two name flags are OPTIONAL, and a missing name is written EMPTY, never invented and never refused. This changed in v1.5.5: an earlier build refused a create with no name. A company whose human has not answered yet is written with its key and empty names, the transcript names the flags that were not supplied, and the tenant still reaches live (crates/argondb-ctl/src/tenant_create.rs:1862-1873). What is still refused, before anything is created, is an --identity key the identity table has no column for — country=US, for example — because an unknown key cannot be stored and dropping it quietly is how an address goes missing (IDENTITY_EXTRA_FIELDS, crates/argondb-ctl/src/tenant_create.rs:2024; the refusal, :424-438). tenant create --help marks both name flags OPTIONAL (crates/argondb-ctl/src/main.rs:624-634), as the code does.

The eight steps, and what each one is. The run prints them as it goes; the one place their order is written down is STEPS (crates/argondb-ctl/src/tenant_create.rs:206-215):

Measured (2026-10-30, compose-2's node, [compute] per_tenant on, the estate pass and maintainer running): the whole command took 3.4 s on a fresh box with one system and 4.4 s at 300 tenants, with the compute start (step 2) about 1 s and the wall pass (step 6) under 1.1 s at every count, no step ever reaching 60 s (docs/lane-reports/product-measure-2026-10-30-tenant-create-timed-on-compose-2-at-the-reference-estates-tenant-count-against-a-fresh-box-and-the-954-second-idle-wait-named.md).

After the eight steps the run mints the company's own MCP token and writes it to a 0600 file, <data>/tenants/acme/mcp-token (§4.11). A failure there is named and does not undo the tenant; argondb tenant mcp-token acme is the remedy (crates/argondb-ctl/src/tenant_create.rs:2950-2975).

The identity table has one door. Step 7 writes the tenant's row and then its system's Identities row with the tenant commit's LSN as source_lsn; a trigger refuses a row write to identity from any role but the tenant API's own, and tenant_key is not editable at all. A rename goes through argondb tenant identity acme --display-name …, the same door (crates/argondb-ctl/src/main.rs:725-744).

RE-RUNNING IT IS SAFE, AND THAT IS THE POINT. Every step probes before it acts, so a second run creates nothing and says so (crates/argondb-ctl/src/tenant_create.rs:2244, :2938-2948):

[argondb]   step 1/8 timeline — already present (§13.4 step 5); not applied
...
[argondb] tenant create: acme is live — 0 step(s) applied, 8 already present. Journal: <data>/tenants/acme/create-journal.json
[argondb] tenant create: NO-OP — this run created nothing (every step probed present). §13.4 idempotence.

AND A RUN THAT DIES HALFWAY IS RESUMED BY RE-RUNNING IT. Progress is recorded in a step journal at <data>/tenants/acme/create-journal.json, and a tenant that is half-built says so in its own status column (provisioning, not live). Re-run the same command and it continues where it stopped:

[argondb] tenant create: RESUMING at step 4 (roles) — the journal names step 3 (database) as the last completed step

WHAT IT WILL NOT DO: adopt. If a database of that name already exists and this command did not create it, it REFUSES and points you at the provisioner. That is not an obstacle — the wall pass adopts it within one cadence, which is §4.1's whole story:

refusing: a database of this name already exists and this sequence did not create it
  (there is no step journal beside it). `argondb tenant create` CREATES; it does not adopt.

What this box does NOT give you. The run's own banner says it: on one development box there is one safekeeper, no WAL backup and no zone diversity. A quorum of one is not redundancy. Every §13.4 step that has no meaning on one box — the tenant's EC2 instance, its subnet and security group, its volume, its elastic address and the three-zone safekeeper member set — is printed as a SKIPPED step with the one-box substitute that stands in for it, so nothing is silently absent (skipped_cloud_steps, crates/argondb-ctl/src/tenant_create.rs:318). The Identities write and the application sidecar were on that list before v1.5.5; both are now real steps (7 and 8 above).

Where the estate control plane, the estate directory and Identities actually run — never co-resident with a tenant. [D-2026-09-21-8b9d22] ruled this on the model of estate instances, following the storage plane's own practices: at least two estate instances across two availability zones (three is the natural number, since the safekeeper quorum already needs three) run the provisioning API (one leader, the rest forwarding rather than serving), the Identities projection, and the multiplexed tail's own lease — never on a tenant's own instance, because a control plane sharing a tenant's instance dies with that tenant and cannot be the thing that recreates it. It does not scale to zero: the API must answer a tenant create call that can by definition arrive when nothing else is awake, so the estate control plane is always-resident, like the storage plane, not stop-when-idle like a tenant compute. On one development box every one of those coordinates collapses to a single process and a schema in the box's own primary (argon_estate) — which is why the bar for "one-box complete" now also requires showing that control plane's leader and its forwarding running as two processes on the box, not just the wall [D-2026-09-23-21780c].

Gate: testing/box/tenant-create-sequence.sh (both polarities, on ms-01). Its v1.5.5 run measured a created tenant reaching its system's Identities — all_tenants.public__identity, read through the cross-tenant reader — in 59 seconds with no administrator act (docs/releases/v1.5.5.md §15).

Fixed: the uniform schema's target is a step it defines. The v1.5.6 release shipped UNIFORM_SCHEMA_VERSION at 2 with a sequence that defined only version 1, so a fresh tenant create (step 5) and a default argondb estate migrate (§4.9) both stopped at the uniform schema has no step defining version 2. The sequence now carries step 2 — the identity table's one door (uniform_schema_steps in crates/argondb-ctl/src/tenant_create.rs) — a unit test holds the target to the last step, and testing/box/uniform-schema-fresh-create-and-migrate-from-v1.sh runs both commands on a box. Post-mortem: docs/postmortems/2026-09-28-uniform-schema-version-declared-without-its-step.md.

4.1 Create a tenant database and watch the provisioner wall it — FROM SOURCE

Nothing about creating a tenant is special. Create the database; the pass does the rest within one cadence.

# 1. the database. Nothing else is special about it.
createdb -p 5433 -h localhost -U cloud_admin tenanta
#   (or: psql -c 'CREATE DATABASE tenanta')

# 2. wait for the box's own pass (<= 30 s), or force one now:
argondb lake tenant provision --data-dir "$DATA" --port 5433

# 3. read the estate back
argondb lake tenant list --data-dir "$DATA" --port 5433

The listing's shape is the format string at crates/argondb-ctl/src/lake_tenant.rs:2384-2392 — seven columns on a box with a catalog login, six without (the CATALOG GRANTS column is ABSENT rather than a row of "none", because "none" would read as "this box does not have the feature"; :2377-2382). A worked listing, quoted from the customer recipe at docs/lake-foreign-read.md:141-145:

$ argondb lake tenant list
DATABASE       WAREHOUSE      BUCKET         ACCESS KEY               STATE
postgres       argon          argon          (the box credential)     primary — the box warehouse
shop           shop           shop           argon-tenant-shop        walled
tenantb        tenantb        tenantb        argon-tenant-tenantb     walled

Every STATE is a statement with a producer named (crates/argondb-ctl/src/lake_tenant.rs:2411-2424):

Then the per-database estate, which is a different provisioner path with the same cadence. Check it rather than assume it (docs/upc-hub-recipe.md:453-462):

psql -d tenanta -c "SELECT to_regnamespace('derived') IS NOT NULL
                      AND to_regnamespace('mfr')     IS NOT NULL
                      AND to_regclass('derived.transform_runs') IS NOT NULL
                      AND to_regclass('argon.view_state')       IS NOT NULL"

That is exactly the probe transform_estate::is_provisioned runs (crates/argondb-ctl/src/transform_estate.rs:127-147), and a unit test asserts the probe asks for precisely what deploy/sql/provision-transform-role.sql creates, so the two cannot drift (:298-317). Note the cluster/database split that made this gap invisible once: CREATE ROLE argon_transform LOGIN is a CLUSTER object, visible from every database, so "does the role exist?" asked alone returns a false CLEAN (:19-26).

Gate: testing/box/tenant-wall-warehouse.sh P1, P3, P4 and P7 (testing/box/tenant-wall-warehouse.sh:19-26, :41-57) — the credential, the bucket directory, the warehouse, a distinct key per tenant, a REFUSAL for the primary, and which bucket the published parquet actually lands in, every leg re-run after a box restart.

4.2 The three spellings of one identity — FROM SOURCE

Ask the box; do not assemble it. The examples below are quoted verbatim from the module's own doc comment, crates/argondb-ctl/src/lake_address.rs:25-36 and :105-110:

$ argondb lake address postgres.public.orders
lake."br_main.postgres.public".orders
$ argondb lake address --field namespace postgres.public.orders
br_main.postgres.public
$ argondb lake address --field lake-table postgres.public.orders
postgres__public__orders
$ argondb lake address --field qualified postgres.public.orders
"public"."orders"
$ argondb lake address --field label postgres.public.orders
br_main.postgres.public.orders

$ argondb lake address --changes postgres.public.orders
lake."br_main.postgres.public".orders__changes
$ argondb lake address --changes --field lake-table postgres.public.orders
postgres__public__orders__changes

--branch moves the branch coordinate, and --format json answers a machine (crates/argondb-ctl/src/lake_address.rs:278-281 for the two formats; docs/lake-foreign-read.md:462-466 for the branch form):

$ argondb lake address --branch dev shop.sales.orders
lake."dev.shop.sales".orders

Which field you want, by consumer: address for a DuckDB user and for a doc; namespace for a starter's LAKE_NAMESPACE constant; leaf for a caller talking to the REST catalog directly; lake-table for a pin scope, the FDW mount and the tail's tables action; label for a log matcher; qualified for the tail's own <pg> left the published set lines; bucket for "which bucket do this table's files land in" (crates/argondb-ctl/src/lake_address.rs:156-196; testing/box/tenant-wall-warehouse.sh:53).

The failure you will hit if you assemble it (docs/lake-foreign-read.md:453-456):

D SELECT count(*) FROM lake.br_main.postgres.public.orders;
Parser Error: NameListToString NOT IMPLEMENTED

Four name parts, and SQL has three. The quotes are not optional.

4.3 Hand an analyst a tenant token, and see what it can and cannot list — FROM SOURCE

# mint + grant in one call; the TOKEN is the only thing on stdout
TOKEN="$(argondb lake grant tenanta --data-dir "$DATA" --ttl 30)"

# the clause a reader pastes, minted and granted through the same path
argondb lake attach-clause --database tenanta --data-dir "$DATA"

# take it back
argondb lake revoke tenanta --data-dir "$DATA"

The lines on stderr are the format strings at crates/argondb-ctl/src/lake_grant.rs:256-275:

[argondb] granted `select` on warehouse tenanta (<uuid>) to oidc~argondb-tenant-tenanta, valid 30 day(s).
[argondb] the analyst reads it with:
    ATTACH 'tenanta' AS lake (TYPE iceberg, ENDPOINT '<catalog>/catalog', AUTHORIZATION_TYPE 'oauth2', TOKEN '<the token on stdout>');
[argondb] revoke it with: argondb lake revoke tenanta

What that token can do, measured (the C-series rows at docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:826-838):

C4a/C4b tenant principals BEFORE any grant          HTTP 403 / HTTP 403
C6a/C6b grant select on one warehouse each          HTTP 204 / HTTP 204
C7a tenant-a lists warehouses                       HTTP 200  -> tenanta ONLY
C7b tenant-b lists warehouses                       HTTP 200  -> tenantb ONLY
C7d tenant-a reads tenantB config                   HTTP 404
C7e tenant-b reads tenantA config                   HTTP 404
C7f/C7g committer reads BOTH configs (the control)  HTTP 200 / HTTP 200
C7h ANONYMOUS reads tenantA config                  HTTP 401
C8a FORGED token, right shape wrong signer          HTTP 401

Read the 404s: a warehouse this subject may not see is not merely refused, its NAME is not disclosed. And read the negative controls as controls — each is only one because its positive twin passed in the same run on the same box.

The refusal you will meet if you point it at the primary (crates/argondb-ctl/src/lake_grant.rs:59-67):

database "postgres" resolves to the BOX warehouse "argon", which holds every other
database's tables as well — granting there would hand the caller the whole lake while
reading like a per-tenant permission. Turn on `[lake] warehouse_per_database` (it is on
by default) and let the provisioner wall this database first: `argondb lake tenant provision`.

Revoke does not invalidate the token — it removes the assignment, so the subject is refused from the next request onward while the JWT stays valid until it expires; --ttl is what bounds that half (crates/argondb-ctl/src/lake_grant.rs:281-287).

4.4 Read every tenant at once under the reader, and see an unwalled database named — FROM SOURCE

argondb lake reader credential --data-dir "$DATA"            # the roster, the grants, a token
eval "$(argondb lake reader credential --data-dir "$DATA" --env)"   # exports ARGONDB_LAKE_WAREHOUSES
argondb-query "SELECT tenant, count(*) FROM all_tenants.public__orders GROUP BY tenant ORDER BY tenant"

The credential surface, quoted from docs/lake-foreign-read.md:228-241:

$ argondb lake reader credential
principal    cross-tenant reader (read-only over every tenant warehouse, never the box warehouse)
access-key   argon-reader
secret-key   <48 hex characters>
endpoint     http://127.0.0.1:9011
region       local
catalog      http://127.0.0.1:8181/catalog
warehouses   shop, tenantb
buckets      shop, tenantb
granted      shop, tenantb
pass-unix    1789514071
token-ttl    30 day(s)
token        <a JWT for the subject argondb-reader>

And the session, quoted from docs/lake-foreign-read.md:322-336 — note that the banner, the census line and the drift note all arrive before the rows:

$ eval "$(argondb lake reader credential --env)"
$ argondb-query "SELECT tenant, count(*) FROM all_tenants.public__orders GROUP BY tenant ORDER BY tenant"
-- pinned at 0/1C2A3F0 (min watermark 0/1C2A3F0)
-- note: this session resolves 4 warehouses — tenanta, tenantb, tenantc, tenantd — …
-- pin built in 412 ms over 4 warehouse(s), 5 lake table(s)
-- all_tenants: 2 view(s) served over 4 tenant(s) [tenanta, tenantb, tenantc, tenantd]: all_tenants."public__orders", all_tenants."public__other"
-- note: 2 cross-tenant view(s) DRIFTED — a row from a named tenant is unioned by column name with NULL for what that tenant lacks, and a tenant with no copy contributes no rows: all_tenants."public__orders" — tenant tenantc lacks column(s) "v" (served as NULL); tenant(s) tenantd have no copy of this table (no branch) | all_tenants."public__other" — tenant(s) tenanta, tenantb, tenantc have no copy of this table (no branch)
+---------+----------+
| tenant  | count(*) |
+---------+----------+
| tenanta | 3000     |
| tenantb | 3000     |
| tenantc | 50       |
+---------+----------+

An unwalled database is ABSENT BY NAME, in the list line and in the roster. The lake tenant list reader line is built at crates/argondb-ctl/src/lake_tenant.rs:2484-2498 and renders the absent entries as <database> (<reason>):

cross-tenant reader: argon-reader covers [shop, tenantb] catalog grants [shop, tenantb] absent [zz_bad_name (<the resolver's reason>)]

The other two states of that line are statements too (:2469-2476):

cross-tenant reader: none — [lake].warehouse_per_database is off (no wall, no reader)
cross-tenant reader: not yet minted — the provisioner mints it on the pass that walls the first non-primary database (within 30s), or run `argondb lake tenant provision`

Subsetting is the query's WHERE, and it is free. WHERE tenant IN ('tenanta','tenantb') folds to false in every other branch and those branches are dropped before a file is opened (crates/argondb-query/src/all_tenants.rs:40-43). There is no named tenant set to configure, and the config key that would have been one was withdrawn (docs/2026-09-15-cross-tenant-read-design.md:214-218).

The primary is not in the view. The session never resolves the box warehouse, so postgres__public__house answers "table not found", and naming the box warehouse in the roster by hand is refused by the catalog under the reader's login with the session saying which warehouse refused (docs/lake-foreign-read.md:277-281).

Gates: testing/box/cross-tenant-reader.sh (8 phases — the store wall, the catalog, a session, a tenant joining and leaving the roster, a restart; testing/box/cross-tenant-reader.sh:17-57) and testing/box/all-tenants-views.sh (8 phases — the view, pushdown read off EXPLAIN, zero bytes written, the cost measurement, a memory bound, a DuckDB reader; testing/box/all-tenants-views.sh:18-80).

4.5 Set, override and read back a connection limit — FROM SOURCE

# the default (20) is applied only where datconnlimit = -1
argondb lake tenant provision --data-dir "$DATA" --port 5433

# an explicit override, applied EXACTLY, on this pass and every later one
argondb lake tenant provision --data-dir "$DATA" --port 5433 --connection-limit tenantb=50

or, durably, in argon.toml:

[lake.tenant_connection_limit]
tenantb = 50

The CLI rung wins over the file rung, and both win over the default (crates/argondb-ctl/src/lake_tenant.rs:648-659). Read it back on the operator surface — lake tenant list carries a CONN LIMIT column precisely so an unset or hand-raised limit is visible outside the box log (:2366-2370), and the primary's cell reads unlimited (testing/box/tenant-wall-warehouse.sh:633-639).

The three rules, and both polarities of each, are what testing/box/tenant-wall-warehouse.sh P10 proves (testing/box/tenant-wall-warehouse.sh:62-74): 20 held sessions on tenant A as a NON-superuser role and the 21st refused NAMING THE DATABASE; tenant B still connecting (the positive twin); a superuser still connecting (PostgreSQL's documented exemption); a CLI override winning and printed with the sibling untouched; a hand-set value surviving a plain pass, neither lowered nor raised; an argon.toml override winning, printed, and printed again as "in force" on the next pass; a malformed value and an override naming no database refused by name with nothing applied; and the UPGRADE shape — a tenant with no limit getting 20 on the next pass, with the pass saying so.

Two edges worth knowing before you rely on the number. It is PostgreSQL's own limit, so superusers are exempt and the count is of connections to that DATABASE, not of the role (crates/argondb-ctl/src/lake_tenant.rs:136-141). And default = -1 is a legal setting meaning "set no default" — an operator who wants PostgreSQL's unlimited behavior back says so with -1 (:626-630).

4.6 Turn the catalog login on, and read a refused cross-tenant access — FROM SOURCE

# argon.toml — but note the DEFAULT already resolves this way on a
# multi-database box, and an explicit value always wins
[lake]
catalog_authz = "openfga"

The default was ruled on 2026-09-15: openfga on a box holding more than one database, off on a single-database box, explicit always wins (docs/2026-09-15-cross-tenant-read-design.md:502-504). The box says which one it is running, and says when the count has crossed since boot — one predicate, called by both the cadence loop and the list surface so they cannot disagree (crates/argondb-ctl/src/lake_tenant.rs:2344-2356, :2460-2465):

catalog authz: <what this box runs, and why> (as decided at boot)
<…and, when the database count has crossed 1 since boot, that the change takes effect at the next boot>

With the login OFF, the served refusal text states the position exactly — authenticates nobody and authorizes everybody (crates/argondb-ctl/src/lake_grant.rs:356, quoted at docs/2026-09-15-cross-tenant-read-design.md:520-521). Data is walled; metadata is not.

What a refused cross-tenant read prints, at each of the two doors. At the store, the exact Deny strings are the format strings at crates/argondb-lake-store/src/wall.rs:148-151, :166-170, :173-177 and :182-186:

access key "argon-tenant-tenanta" is scoped to bucket "tenanta" (database "tenanta") and GetObject addresses bucket "tenantb"
access key "argon-reader" is the cross-tenant READER over ["tenanta", "tenantb"] and GetObject addresses bucket "argon", which is not a tenant bucket in its roster (the box bucket never is)
access key "argon-reader" is the cross-tenant READER and is READ-ONLY: PutObject on bucket "tenanta" is refused (allowed: GetObject, HeadObject, ListObjectsV2, ListObjects, HeadBucket, GetBucketLocation, GetObjectAttributes)
access key "argon-reader" is the cross-tenant READER (read-only over 2 tenant bucket(s)) and CreateBucket addresses the whole store

Each surfaces to the client as 403 AccessDenied and is logged by the store with the key, the operation and both buckets named (docs/lake-foreign-read.md:190-194, :243-249). At the catalog, the refusal is a 404 and does not disclose the name (docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:798-800).

Gate: testing/box/tenant-wall-warehouse.sh P5 runs the whole matrix with both polarities and a restart afterwards — key A on bucket A 200, key A on bucket B 403 on list AND on get, the symmetric pair, key A on the BOX bucket 403, root on every bucket 200 as the control, ListBuckets filtered per key, anonymous 403 (testing/box/tenant-wall-warehouse.sh:27-40).

4.7 Stand up a manufacturer, and see the party rule bite — FROM SOURCE

A manufacturer is a member database beside the distributors; the SCHEMA is what keeps the parties apart (docs/upc-hub-recipe.md:430-441).

createdb mfracme           # the box's pass walls it exactly as it does a distributor
psql -d mfracme -c "SELECT to_regnamespace('mfr') IS NOT NULL"   # estate; you do not create it

CREATE TABLE mfr.price_list(
  upc             text PRIMARY KEY,
  list_price      double precision,
  effective_from  date,
  effective_to    date,
  manufacturer_id text);
COMMENT ON TABLE mfr.price_list IS 'argon:publish';

CREATE TABLE mfr.share_grant(
  flow             text NOT NULL,
  manufacturer_id  text NOT NULL,      -- a recipient, or '*'
  contract_version int  NOT NULL,
  granted_at       timestamptz NOT NULL DEFAULT now(),
  revoked_at       timestamptz,        -- NULL = in force
  PRIMARY KEY (flow, manufacturer_id, contract_version));
COMMENT ON TABLE mfr.share_grant IS 'argon:publish';
INSERT INTO mfr.share_grant(flow, manufacturer_id, contract_version)
  VALUES ('upc_prices', '*', 1);

(Both DDL blocks quoted from docs/upc-hub-recipe.md:470-496.) Then ask the product which parties it sees — the output is quoted from docs/upc-hub-recipe.md:508-514:

$ argondb hub parties
pin 0/3A1B2C8 — 4 member database(s), 1 manufacturer(s)
hub: party hubcontrol: distributor (publishes no mfr table at this pin)
hub: party mfracme: manufacturer (publishes 2 mfr table(s) at this pin)
hub: party tenanta: distributor (publishes no mfr table at this pin)
hub: party tenantb: distributor (publishes no mfr table at this pin)

Every member database gets a line, distributor included — so "no manufacturers" and "nothing looked" can never read the same (docs/upc-hub-recipe.md:516-517).

The failure mode, and its one remedy. A manufacturer database publishing outside mfr becomes a branch of the distributors' own cross-tenant views. The product names it and exits non-zero (docs/upc-hub-recipe.md:524-533):

hub: CROSSES PARTIES mfracme.public — served as "mfracme__public__products", a branch of all_tenants.public__products

…and a hub run whose source scans that branch is REFUSED before anything lands. It is a REFERENCE question and deliberately not a pruning one: the LOGICAL plan is walked before optimization, so a WHERE tenant IN (…) over the view does NOT clear it. The remedy is the one the design states — publish it in mfr. A source over a distributor's own table references no manufacturer copy and is unaffected (docs/reserved-identifiers.md:6735).

Gates: testing/box/upc-hub-warehouse.sh P20 and testing/box/all-tenants-views.sh P8, both polarities each (docs/reserved-identifiers.md:6735).

4.8 Find out who is holding the provisioner pass lock — VERIFIED (the key) + FROM SOURCE (the SQL)

The list surface prints the lock line itself (crates/argondb-ctl/src/lake_tenant.rs:2501-2506, rendered by pass_lock_list_line, :2512):

provisioner lock: pg advisory key 7021788497314804855 on the primary — free / HELD by …

To ask PostgreSQL directly you need the key's two halves, because pg_locks stores a single-bigint advisory key as classid (high 32 bits) and objid (low 32), with objsubid = 1 (crates/argondb-ctl/src/lake_tenant.rs:262-274). The derivation is reproducible by hand, which is the whole reason the key is eight ASCII bytes and not a magic literal — VERIFIED, run in this lane:

$ python3 -c 'k=int.from_bytes(b"argon_tw","big",signed=True); print(k, hex(k>>32), hex(k & 0xffffffff))'
7021788497314804855 0x6172676f 0x6e5f7477

which matches the registered values at docs/reserved-identifiers.md:6734 (classid = 0x6172676f, objid = 0x6e5f7477, objsubid = 1). The query the product itself runs is PASS_LOCK_HOLDER_SQL (crates/argondb-ctl/src/lake_tenant.rs:276-281) — FROM SOURCE, quoted:

SELECT a.pid, coalesce(a.application_name, ''), coalesce(a.state, ''),
       to_char(a.backend_start, 'YYYY-MM-DD HH24:MI:SS TZ')
  FROM pg_locks l JOIN pg_stat_activity a ON a.pid = l.pid
 WHERE l.locktype = 'advisory' AND l.granted AND l.objsubid = 1
   AND l.classid = (7021788497314804855 >> 32)::oid
   AND l.objid   = (7021788497314804855 & x'ffffffff'::bigint)::oid
 ORDER BY a.pid LIMIT 1;

The holder's application_name tells you which caller it is: the box's own pass wears <INTERNAL_APP_PREFIX>_tenant_wall and the CLI wears argondb_lake_tenant_cli (crates/argondb-ctl/src/lake_tenant.rs:240-242). Raise the wait with ARGONDB_TENANT_WALL_LOCK_WAIT_SECS if an 80-tenant pass is in front of you (:179-183).

Gate: testing/box/provisioner-exclusive-up-after-tail.sh, 6 phases, both polarities (docs/reserved-identifiers.md:6734).

4.9 Migrate every tenant's schema in one pass — FROM SOURCE

One database per tenant has a cost that shows up the first time the schema changes: a company created last month and a company created today are each at whatever version their own tenant create shipped. argondb estate migrate closes that gap. It brings EVERY tenant on the box to one version of the uniform schema in one pass, through the same per-database function step 5 of tenant create calls (tenant_create::apply_uniform_schema), so no tenant is migrated by a path the others were not (crates/argondb-ctl/src/estate_migrate.rs:1-17).

# bring every tenant to the version this build ships (the default --to)
argondb estate migrate --data-dir "$DATA" --port 5433

# or to a version you name
argondb estate migrate --to 2 --data-dir "$DATA" --port 5433

(crates/argondb-ctl/src/main.rs:555-572.) What one pass does, in order (crates/argondb-ctl/src/estate_migrate.rs:290-315):

1. It enumerates the estate directory, argon_estate.tenant, and asks pg_database which of those databases really exist. A directory row whose database is gone is REFUSED by name. A database with no directory row is not migrated at all: adopting it is a decision, not a side effect of a pass (:34-40, :342-353, :372).
2. It asks each tenant its own version, from that tenant's own argon.schema_version — never from the directory row. "Never applied" and "version 0" are different answers and are kept apart (:42-49).
3. It decides apply, skip or refuse for every tenant before changing anything (plan, :175). A tenant at or above the target is SKIPPED and counted; nothing is ever rolled back. A target above the highest version this build ships is refused for every tenant before any schema moves.
4. It applies, and a failure is that tenant's refusal, never the end of the pass. One company whose migration fails does not leave the other N-1 un-migrated (:80-91).
5. It records. The version is read BACK out of the tenant and only then written to the directory row, so the row can lag its database and can never run ahead of it (directory_value, :216); and one row per tenant per pass goes into argon_estate.schema_migration, outcome applied, at-target or refused with the reason (JOURNAL_DDL, :54-70).

Every pass ends with one count line, zeros included, so "nothing refused" is said rather than implied (summary_line, :277-286):

[argondb-box] estate migrate --to 2: 2 tenant(s) applied, 1 already at target (skipped), 0 refused (named, pass continued), 3 row(s) walked

The exit code is non-zero if any tenant was refused, after the pass has migrated every other tenant and named every refusal on stdout (crates/argondb-ctl/src/main.rs:4133-4147). A second run at the same target applies nothing.

What it does not do. It is a command an operator runs, not a reconciler arm on a timer: a schema change to every company at once needs an operator in the loop (crates/argondb-ctl/src/estate_migrate.rs:19-32). And the spec's rehearsal step — try the migration on a branch of that company's timeline first — is NOT built; the pass goes straight from reading the version to applying it (docs/2026-09-04-erp-spec.md §16). Read the defect paragraph at the end of §4.0 before you rely on the default --to.

Gate: testing/box/uniform-schema-migration-across-tenants.sh (P0-P10 — three tenants at two versions, a planted refusal in the middle tenant with the tenants on both sides still migrated, the journal, a re-run as a no-op by count, and an unshipped target refused).

4.10 Create a tenancy, and publish its Identities to every other tenancy — FROM SOURCE

A tenant belongs to a TENANCY — one multi-tenant system in the Estate, the Distributor Tenancy distributors or the Manufacturer Tenancy manufacturers (--system in §4.0; the CLI's word for a tenancy is system, and the command name stays argondb system create). Creating the tenancy is the other §13.4 API, and like tenant create it is ordered, journaled, probed and safe to re-run (crates/argondb-ctl/src/main.rs:1060-1083):

argondb system create manufacturers --data-dir "$DATA" --port 5433

(crates/argondb-ctl/src/system_create.rs:246-257.)

New in v1.5.6: step 4 grants in the same call. It used to grant only on a namespace the tail had already published into, so a system created moments ago printed DEFERRED, granted nothing, and journaled itself done — leaving the operator to run argondb system grant-identities later with nothing to say when. Step 4 now creates the Identities namespace itself when it is absent (the same act the tail's first publish performs) and grants in the same call (crates/argondb-ctl/src/system_create.rs:1486-1512, system_identities_grant::ensure_namespace_id). And the probe now reads an absent namespace, with another system registered, as step 4 UNFINISHED, so a create killed after step 3 runs step 4 again on resume instead of skipping it (:931-959).

Step 4 publishes in ONE direction, and this is the part to get right. It grants the OTHER systems' readers on THIS system's Identities. An earlier system's Identities is not granted to a system created later — the later system's create grants in the opposite direction only. So the first system you create publishes to nobody. Create the Distributor Tenancy first: each Manufacturer Tenancy created after it then grants the Distributors' reader its Identities, which is the Distributors' estate-wide read for discovery.

New in v1.5.7: a Manufacturer tenant sees a Distributor's identity once that Distributor invites it, with no operator step [D-2026-09-28-65b0a2]. The invitation is the shared-data grant you already know. The Distributor's pub.publication row names the Manufacturer's database: that is PENDING. The Manufacturer's own pub.subscription to it makes it ACCEPTED. Either one is enough. On every pass the box writes each inviting Distributor's Identities row into the Manufacturer's own derived.identities, which lists exactly those Distributors and no others (invited_by and invitation say who invited it and in what state). When the Distributor revokes (argondb publication revoke), the refresh stops. The row already landed stays, and the box log names it: revoked at <ts> — refresh stopped, the landed row is RETAINED (crates/argondb-ctl/src/invited_identities.rs).

grant-identities is still there for boxes that already depend on it. It opens the publishing system's WHOLE Identities namespace to the reading system's reader, which is wider than any invitation, so use it only when the reading system really needs every identity of the publishing one (crates/argondb-ctl/src/main.rs, SystemCmd::GrantIdentities):

argondb system grant-identities  --from distributors --to manufacturers
argondb system identities-grants                       # every recorded grant, LIVE or REVOKED
argondb system revoke-identities --from distributors --to manufacturers

Mind the first system's own message: it says "a system registered later gets its grant from its own create", which reads as if the later create covers both directions; it covers the later system's Identities only (crates/argondb-ctl/src/system_create.rs:1470-1478). A revoke refuses the foreign reader from its next request onward; what that system already landed into its own schema stays — it is that system's data (crates/argondb-ctl/src/main.rs:1145-1149).

What is not built, and printed as such rather than faked. Step 4's CROSS-ACCOUNT half — a bucket policy and an assumable role — has nothing to name on one box with no second account or region; every step 4 run prints CROSS-ACCOUNT: NOT BUILT on this box. And a system's reader is a CATALOG principal only: the credentials file still carries one box-wide reader key, not a key per system (crates/argondb-ctl/src/main.rs:1069-1073, crates/argondb-ctl/src/system_create.rs:1513-1520).

Gate: testing/box/system-create-sequence.sh — P3 asserts step 4 created the Identities namespace, P5 asserts a resume killed after step 3 runs exactly steps 4 and 5, and the system boundary is proven by a refused read.

4.11 Give each company its own MCP token — FROM SOURCE

Before v1.5.6 the box's one MCP token, ARGONDB_AUTH_TOKEN, was the only credential the MCP endpoint took, so anything you handed a company's agent could read every company on the box. Now every company carries its own token, and a session under it reads ONLY that company's database (crates/argondb-ctl/src/main.rs:672-687):

# tenant create mints the first one, into a 0600 file (§4.0)
cat "$DATA/tenants/acme/mcp-token"

# re-issue: prints a fresh token on stdout; every earlier token of acme is refused from the next request on
TOKEN="$(argondb tenant mcp-token acme --data-dir "$DATA")"

# revoke every token of acme
argondb tenant mcp-token acme --revoke --data-dir "$DATA"

What the token is and what it reads. A company token starts amc1. (company_token::PREFIX, crates/argondb-mcp/src/company_token.rs:46) and is signed by the box. A session under it reads through the company's own read role, <key>_mcp, on the company's own database: LOGIN, CONNECT on that one database, and SELECT on public including tables created later (crates/argondb-ctl/src/mcp_company_token.rs:45-87). So a read of another company's table is refused by PostgreSQL at the company's own connection, not by a filter. Exactly one token per company is live: a re-issue supersedes the last one, and --revoke removes the record and makes <key>_mcp NOLOGIN (:217-229).

What it refuses, by name. A bad token is an HTTP 401 whose body says why (crates/argondb-mcp/src/company_token.rs:102-122):

company token refused: its signature does not match its claim company="acme" (forged or re-labelled)
company token refused: company "acme" has no live MCP token (revoked, or never minted)
company token refused: company "acme"'s token was revoked by a re-issue (this is not its live token)

Two tools read box-wide state and refuse under a company token: search (crates/argondb-mcp/src/lib.rs:5028) and publish_status (:4767), both through one refusal, company_refusal (:4613-4625) — a company session gets no text search at all, not a company-scoped index. Every response carries x-argondb-mcp-scope: operator or company:<key>, so a client can see which scope served it (crates/argondb-mcp/src/company_token.rs:52-56).

Where it applies. Company tokens are accepted on the ANALYST surface only; the admin mount keeps its one operator token (crates/argondb-mcp/src/lib.rs:6244-6248). argondb serve turns them on by setting ARGONDB_MCP_COMPANY_AUTH_DIR; with it unset, company tokens are not accepted at all (crates/argondb-mcp/src/company_token.rs:48-50). The box token keeps operator scope — treat it as the operator credential it is, and hand a company's agent the company's token instead. A company session reads PostgreSQL only: the lake half of carrying the role into the read plane (Block 6 [D-2026-09-11-c64ea4]) is not built, so a company token reads no lake table — with one exception: its query reads the company's OWN <table>__changes companions, read-only and under the tenant wall [D-2026-10-03-ada8e3] (guide 4, §3.4).

Gate: testing/box/cross-company-isolation-eight-surfaces.sh, surfaces S4+ and S4- (:738-795).

4.12 Hand each manufacturer its own copy of the global aggregate — FROM SOURCE

A hub run's global aggregate (--aggregate <name>) always landed in the BOX warehouse — which no manufacturer's key opens. Now each manufacturer also gets its OWN COPY, in its own hub-<m> warehouse, so the manufacturer reads it under its own credential and the wall is not crossed (crates/argondb-ctl/src/hub.rs:870-945):

argondb hub run --source "$SRC" --ownership "$OWN_TBL" --flow upc_prices \
  --hub-table upc_hub --aggregate upc_hub_totals \
  --data-dir "$DATA" --port 5433

(The shape testing/box/upc-hub-warehouse.sh P14b runs, :1464-1468.)

A copy counts only what that manufacturer was granted. Each manufacturer's copy is computed over exactly the tenants whose grant of the flow to that manufacturer is in force at the run's pin — the run's own consent plan, not a second reading of the grants. Two states land NOTHING, not even an empty table, and say so on the run and in the ledger (hub.run_ledger.aggregate_copies):

hub: aggregate copy for <m>: NOT LANDED — no-flow: this run names no --flow, so no consent plan exists and no tenant granted "<m>" anything; a copy of the aggregate is handed only under a grant
hub: aggregate copy for <m>: NOT LANDED — no-consent: no tenant's grant of flow "upc_prices" to "<m>" is in force at pin <lsn>, so "<m>" receives no copy of the aggregate

The copy lands like the box's aggregate: created on first sight and replaced after, under the token <run>:__aggregate:<m>, so a re-run at the same token changes nothing. An aggregate whose columns would carry a tenant or a retailer identity — tenant, customer_id, customer_name, customer, name — is refused by name (check_aggregate_columns, crates/argondb-transform/src/hub.rs:184). One stale line: the --aggregate help still says the aggregate lands "in the BOX warehouse" only (crates/argondb-ctl/src/main.rs:2710-2711).

The open question, stated plainly. A copy counted over a SINGLE granting tenant shows that tenant's total volume across every manufacturer, competitors included. Whether that is acceptable is waiting on a ruling (docs/history/folded/2026-09-24-each-manufacturer-receives-its-own-copy-of-the-global-aggregate.md).

Gate: testing/box/upc-hub-warehouse.sh P14b (:1444-1549) — the granted manufacturer receives a copy and the ungranted one receives none, with the ledger saying why.

5. Built today versus designed

6. Gotchas

A tenant table published before its wall exists is DEFERRED, and the log line repeats. That is the design, not a stall: the alternative is publishing the bytes behind the box credential, which is the wall inverted. Wait one cadence or run argondb lake tenant provision (crates/argondb-committer/src/catalog_set.rs:243-257).

A database whose name is not a legal object-store bucket name is REFUSED, by name, and the pass continues. Nothing rewrites the name to fit, because a rewrite would merge tenant_a and tenant-a into one bucket — which is the wall this feature exists to build. The remedy is [lake.tenant_warehouse].<database> = { warehouse, bucket } in argon.toml, the only door for such a database (docs/reserved-identifiers.md:6732). Gate P9 proves both polarities (testing/box/tenant-wall-warehouse.sh:58-61).

warehouse_per_database = false removes the wall AND the reader. Every database resolves to the box warehouse, so reader_plan returns None and the pass REMOVES a reader an earlier configuration left behind. There is no cross-tenant read without a wall (crates/argondb-ctl/src/lake_tenant.rs:108-112, :1529-1532). An explicit true against a bring-your-own object store is also refused: the box can mint a bucket-scoped key only in a store it runs (docs/reserved-identifiers.md:6732).

With catalog_authz = "off", names are public even though bytes are not. Anything that can reach the catalog port lists every warehouse and every table name with no credential at all. The served text says so in those words. Bind the catalog to loopback and treat its port as privileged (docs/lake-foreign-read.md:196-202).

OpenFGA ships in both platforms' release artifacts. scripts/release/build-artifact.sh stages bin/openfga on Linux (license gate :243-246, staging :584-591) and scripts/release/build-artifact-macos.sh does the same on macOS (license gate :223-226, staging :519-523), so neither platform's stock multi-database box meets a boot refusal for a missing engine (docs/2026-09-15-cross-tenant-read-design.md:513-516).

A continuous view over a table you hold only through a mount grant is REFUSED at create, by name. A lake-landed view is scoped to its inputs' OWNER, so a view a grantee defined over the grantor's table would be created and then be unreadable in the grantee's own database; whether a grantee may build one at all is an open ruling (views design OPEN-V3). Until it is ruled, the create door refuses before the pin and before anything is enrolled, on both landings, with kind view-input-granted and this text (crates/argondb-query/src/view_classify.rs:2206-2218; crates/argondb-transform/src/view_door.rs:493-507):

view `<v>`: input <db>.<schema>.<table> is a mounted grant from <grantor>; views over granted tables are not ruled — [ruling pending]. A lake-landed view is scoped to its inputs' OWNER, so this one would be created and then unreadable here; define the view in <grantor>, or over tables this database owns

The remedy is in the last clause: define the view in the grantor's database, or over tables yours owns. Two edges. The check runs only when the create request names a database (crates/argondb-transform/src/view_door.rs:504), so a request with none is not checked. And the lane that built the refusal escalated three defects it did not fix, reproduced with an OWNED input as well: the tail panics on a view that was declared and never refreshed (crates/argondb-tail/src/lake_owned.rs:114), a view created through argon.create_view fails every refresh on its recorded lake_main. qualifier, and under the tenant wall the create door cannot plan any tenant database's table (docs/history/folded/2026-09-24-a-view-over-a-granted-table-is-refused-by-name-until-the-reader-scope-is-ruled.md). Gate: testing/box/view-over-granted-table.sh.

Hand a company's agent its own MCP token, never the box token. The box's ARGONDB_AUTH_TOKEN keeps operator scope over every company; the company token reads that company's database only (§4.11). A company session has no search and no publish_status — both read box-wide state and refuse by name — and reads no lake table, because the lake half of the role's propagation is not built (company_refusal, crates/argondb-mcp/src/lib.rs:4750-4762).

argondb system create publishes one way. A system's Identities reaches the systems that existed when it was created, not the ones created after it. A Manufacturer tenant does not need that grant to see the Distributors that invited it: since v1.5.7 an invitation lands their identities in its own derived.identities with no operator step [D-2026-09-28-65b0a2]. argondb system grant-identities --from <earlier> --to <later> opens the whole namespace and is wider than an invitation (§4.10).

Revoking a grant leaves the token valid until it expires. Nothing here keeps a revocation list. Use --ttl (crates/argondb-ctl/src/lake_grant.rs:281-287).

The operator is inside every wall. The root store key and the argondb-box catalog subject are plural over everything, the box warehouse included, and they write. That is drawn in the main picture and listed as one of its two residuals; it is not something the wall is intended to stop (docs/2026-09-15-cross-tenant-read-design.md:190-195).

Row-level security does not reach the lake copy, and the Flight-door refusal does not cover the catalog door. A reader holding the catalog URL and the store credentials reads every row of every published table, row security or not. Those credentials are the only thing standing between such a reader and the data (docs/lake-foreign-read.md:670-680). Two related surprises: a table already protected the first time the lake mounts it gets no mount at all (the type-learning read is subject to the same wall; run SELECT argondb_lake_sync() as a role row security does not apply to), and toggling row security reaches the wall on the schema cadence rather than instantly (:693-705).

The read plane's assertion is believed, not verified. The compute asserts "this read is for role R and row security does not apply to R here", and the read plane cannot check it, because the only credential on the wire is one process-wide bearer token. Anyone holding ARGONDB_AUTH_TOKEN can assert any role: do not hand it to analysts (docs/lake-foreign-read.md:682-691).

SHOW ALL TABLES does not list a nested table. DuckDB enumerates GET /namespaces with no ?parent and never recurses; a table you have not NAMED is absent from the listing. USE lake."br_main.<db>.<schema>" first and it appears (docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:246-250; the catalog-walk recipe is docs/lake-foreign-read.md:468-520).

A copy that already has a column named tenant cannot be unioned, and neither can a column whose types across tenants the engine cannot coerce (Int32 with Boolean). Both are refused BY NAME and the view is absent from the namespace — never silently narrowed (crates/argondb-query/src/all_tenants.rs:53-58, :60-64).

A session with no roster has no all_tenants namespace at all. The namespace is composed at pin time from what the session registered, so a roster-less session is not a session with an empty view set (docs/lake-foreign-read.md:315-318).

Do not size a deployment from the numbers in §3.7 without re-measuring. The earlier figures on that same line do not reproduce, and the document that carries them says so (docs/lake-foreign-read.md:404-409).

Two connection-limit surprises. Superusers are exempt, and the limit counts connections to the DATABASE rather than by role (crates/argondb-ctl/src/lake_tenant.rs:136-141). A limit an operator set by hand is never touched by a plain pass — neither lowered nor raised — so a tenant sitting at a hand-set 5 stays at 5 until an explicit override says otherwise (:749-753).

Cross-warehouse atomicity of a cut is not claimed. A cut spanning two tenants is one transaction per warehouse; a cross-tenant pin lands at the newest LSN every scoped table in every tenant warehouse serves, and says per warehouse which coordinate it resolved (docs/2026-09-15-cross-tenant-read-design.md:324-343).

Lake freshness is box-granular — with per-tenant compute off, the shipped default. Every tenant is then on the primary's one timeline, so one tenant's write volume gates every other tenant's watermark (docs/2026-09-15-cross-tenant-read-design.md:591). Per-tenant freshness needs a separate Postgres instance and timeline per tenant; that is the ratified topology [D-2026-09-20-0d5ba3] [D-2026-09-21-4107c5], built default off, with the tail running one stream per tenant timeline (HISTORY#tail-multiplexes-tenant-timelines), and the operator turns it on, not the system [D-2026-09-23-5122a6].

The catalog's REST {prefix} is the warehouse's UUID, not its name. /catalog/v1/argon/namespaces answers 400 {"error":{"type":"WarehouseIdIsNotUUID"}}; the resolver is the config response's defaults.prefix, read from GET /catalog/v1/config?warehouse=<name> (docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:191-196).

Not measured, and named as such: Trino. No Trino was stood up. Its Iceberg REST connector is documented to map a namespace to a schema with the same three-part budget, so the same arithmetic is EXPECTED to apply — that is an inference, not a measurement (docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:257-262).

7. Sub-diagrams

Three deeper pictures of parts the main diagram compresses. Each is generated, not drawn: the source is a Python script beside the others, using the same palette and legend, and python3 <script>.py regenerates the SVG.

7.1 One provisioning pass, and the table that arrives before its wall

Source docs/diagrams/argondb-and-the-erp/generate/d01a_provisioner_pass.py → docs/diagrams/argondb-and-the-erp/01a-provisioner-pass.svg. Embedded at §3.1. It carries the two callers and the pass lock; the four artifacts in the order they are made and why that order; the two cluster-wide halves (the connection limit and the reader); and the deferred-by-name path end to end, with the refusal text's own reasoning beside it.

7.2 The five principals, and what each one can reach

Source docs/diagrams/argondb-and-the-erp/generate/d01b_principals_and_buckets.py → docs/diagrams/argondb-and-the-erp/01b-principals-and-buckets.svg. Embedded at §3.4. The store's predicate arm by arm with its refusal wording, the catalog's four subjects and the grant verb, and a reach table reading every principal against every bucket and two warehouses.

7.3 One relation, its spellings, and the door that answers each

Source docs/diagrams/argondb-and-the-erp/generate/d01c_identity_three_spellings.py → docs/diagrams/argondb-and-the-erp/01c-identity-three-spellings.svg. Embedded at §3.8. Where an identity starts (a PostgreSQL relation plus a publish marker), the four spellings it has, which argondb lake address field answers each, the two failures of assembling it by hand, and what the regularization of the primary changed.

8. References

Read in full for this guide; cited above by path:line.

• docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md — the three-level identity and the tenant-wall options, with the measurements behind them.
• docs/2026-09-15-cross-tenant-read-design.md — the cross-tenant reader, the all_tenants views, the catalog-authz default (T8) and the timeline tiers (T7); all three blocks ruled 2026-09-15.
• docs/2026-09-04-erp-spec.md §2 (tenant = a database in one shared cluster), §4 (why the lake is where tenancy usually breaks), §8 (RLS, demoted).
• docs/upc-hub-recipe.md — the wall as the hub sees it, and the manufacturer-side operator steps.
• crates/argondb-ctl/src/lake_tenant.rs — the provisioner: ensure_all, ensure_principals, ensure_connection_limits, ensure_reader, reader_plan, the 30 s pass and the pass lock.
• crates/argondb-ctl/src/lake_grant.rs — lake grant / revoke / attach-clause, and the reader's token.
• crates/argondb-ctl/src/lake_address.rs — argondb lake address and its fields.
• crates/argondb-ctl/src/transform_estate.rs — the per-database estate (derived, mfr, the write surface) and the cluster/database split.
• crates/argondb-lake-store/src/wall.rs — the store's one predicate, READER_OPS, and the copy hook.
• crates/argondb-committer/src/catalog_set.rs — the per-table catalog handle and the deferral message.
• crates/argondb-query/src/all_tenants.rs — the all_tenants composer, the tenant column, MANUFACTURER_SCHEMA.
• crates/argondb-ctl/src/tenant_create.rs — argondb tenant create: the eight steps, the identity fields, the uniform schema's versioned apply and the company's MCP token.
• crates/argondb-ctl/src/estate_migrate.rs — argondb estate migrate, the multi-database migration driver.
• crates/argondb-ctl/src/system_create.rs and crates/argondb-ctl/src/system_identities_grant.rs — argondb system create and the Identities grants.
• crates/argondb-ctl/src/mcp_company_token.rs and crates/argondb-mcp/src/company_token.rs — the per-company MCP token, its read role and its refusals.
• testing/box/tenant-wall-warehouse.sh — the wall on a fresh box, both polarities, across a restart (P0-P10).
• testing/box/all-tenants-views.sh — the views, pushdown, zero bytes, cost, memory bound, a foreign reader (P0-P8).
• testing/box/cross-tenant-reader.sh — the reader's two halves, a session, the roster following the pass, a restart (P0-P7).
• docs/lake-foreign-read.md — the customer-facing recipe: per-tenant warehouses, the reader, the views, the address, RLS and the lake copy.
• docs/reserved-identifiers.md — the registered names: the tenant wall's, the reader's, all_tenants and its tenant column, the mfr schema and the party rule, the pass lock.
• Diagrams: docs/diagrams/argondb-and-the-erp/01-one-database-per-tenant-and-the-wall.svg and its generator generate/d01_tenancy.py; the three sub-diagrams of §7 and their generators; the shared drawing library generate/blocklib.py.

Guide 2 — the per-tenant unit: the gate, wake and stand-down

Audience. A senior software developer who has to USE this system: set the knobs, predict what a client sees, size a box, run an application beside a tenant's database, and know which of these capabilities exist today.

Register. Internal technical reference. Every claim carries its source by path:line or by document section; a claim with no source is a defect in this document. Where the picture says a thing is designed and not built, this guide says so in the same words and does not soften it.

Two names, ruled. Our WAL-to-Iceberg process is the tail (argondb-tail). "Sidecar" names only the tenant's own application — block S4, RATIFIED [D-2026-09-18-07c9d9] (docs/2026-09-17-process-sidecar-design.md:1177). The bare word never names our process anywhere below.

1. The diagram

Read the picture left to right along the top and you have the control path. A visitor — a browser, an application's API client, an SQL client, an AI agent — reaches the gate, a single process that fronts both the tenant's SQL port and the tenant's application port (the second listener is built: see the HTTP 503 paragraph at the end of §3.6). The gate is the only component that sees every byte to either, and that is what entitles it to decide who is awake. Its trigger is the first byte, never the accept(): a connection that arrives and sends nothing is dropped after thirty seconds with nothing spawned (crates/argondb-ctl/src/idle_stop.rs:101, :517). Below the gate is the supervisor's process table, and the table has two halves. On the left is the resident plant — broker, pageserver, safekeeper, the lake's object store and catalog, the policy engine, the tail, the maintenance worker, the agent endpoint — which never stops and is paid for once per box, measured at 265 MB (docs/2026-09-12-tenant-runtime-sidecar-design.md, §9.1, the M1 storage+supervisor line, 231 116 kB). In the middle is one tenant's unit: compute:<db> and at most one sidecar:<db>, and nothing else. That unit is the thing that stops.

The second thing the picture is saying is about its own outlines. Solid means built and shipped; dashed means designed, measured and not built. The picture's dashed unit box is stale. When it was drawn the tenancy topology B′ — one compute per tenant database — had been put to a block on 2026-09-12 and NOT selected [D-2026-09-12-44e647]. It was RATIFIED on 2026-09-20 [D-2026-09-20-0d5ba3] and amended [D-2026-09-21-4107c5], and the N-computes-one-per-tenant shape is BUILT: [compute] per_tenant gives every tenant database its own compute:<db> row, its own port pair, its own idle clock and its own gate (crates/argondb-ctl/src/tenant_compute.rs; §2.4, §2.8). It is default OFF and it stays the operator's setting, never the system's [D-2026-09-23-5122a6]. With the knob off the box runs the v1.4 shape, one primary compute per box, with [compute] idle_stop default OFF (ratified [D-2026-09-08-c60bce]). The per-company application process is built too, in a different shape from the row this chapter's §2.5 designs (§2.5 says how). So the reading this guide takes is that the mechanism (the gate, stand-down and wake) is shipped and proven, and the multiplicity is shipped behind an operator's switch, with the limits §2.7 and §6 name. The three boxes on the right that this diagram draws OPEN are RULED now, two of them against this design's own recommendation: G1 — the on-prem box keeps rung 1, but the one-click cloud VM requires rung 2, a container per tenant, AGAINST the recommendation [D-2026-09-20-3425d9]; G2 — a tenant's application is always a SEPARATE OS PROCESS in the box supervisor's table, never inside the database's process, as recommended [D-2026-09-20-4faa8c]; G4 — bytes are the only idle signal, AGAINST the recommendation: a silent WebSocket does not hold a tenant awake [D-2026-09-20-78f71e]. §2.6, §2.8 and §5 carry the detail; the picture's own OPEN styling on these three boxes is stale.

2. What it is

One paragraph per box in the picture, carrying the coordinates, states, limits, defaults and failure modes a block diagram cannot show.

2.1 The gate

The gate is idle_stop::serve_gate (crates/argondb-ctl/src/idle_stop.rs:693) and it is about ninety lines. With [compute] idle_stop ON the supervisor owns the public PG port and Postgres is moved to public + 100 (PG_INTERNAL_DELTA, idle_stop.rs:89; for the default box that is 5432 → 5532, and an overridden ARGONDB_PG_PORT shifts both by the same amount, so two boxes with distinct primaries get distinct internals). It binds exactly the address Postgres would have bound with the knob off (GateConfig::bind, idle_stop.rs:410), so a client sees no difference in its connection string. Per connection it spawns one tokio task (handle_conn, idle_stop.rs:822) and does exactly four things: buffer the first bytes, ask the supervisor to wake if the compute is down, connect to the internal port, and splice. The connect goes to the internal port ON THE ADDRESS THE CLIENT REACHED THE GATE ON (upstream_addr), never a fixed 127.0.0.1: the box hba trusts its own roles on loopback, so a gate that dialed loopback for everyone would present an off-box client as a trusted local one — which it did until 2026-09-24 (testing/box/bind-policy.sh Arm E is the gate). It parses no protocol — a SSLRequest, a GSSENCRequest, the startup packet and a cancel request all reach the real backend unaltered, because the gate never speaks Postgres to the client on the happy path. Two failure modes, both with real messages: a client that sends nothing inside FIRST_BYTES_TIMEOUT = 30 s (idle_stop.rs:101) is closed silently and wakes nothing; a wake that fails gets a genuine Postgres wire ErrorResponse carrying SQLSTATE 57P03 cannot_connect_now (idle_stop.rs:425, :448). A per-connection accept() error (EMFILE, ECONNABORTED) is logged and the loop continues (idle_stop.rs:486-492) — abandoning the port would leave the box's only client surface dead with the supervisor still up; only the initial bind failure is fatal.

2.2 The supervisor's process table

serve_inner builds ONE Vec<(&'static str, Child)> at boot and never reshapes it. Three parallel vectors are keyed on the same index: policies[i] (the restart policy), tripped[i] (the breaker), and stood_down[i] (crates/argondb-ctl/src/serve.rs:8866). The table is already heterogeneous — four inherited Rust binaries, two PostgreSQL clusters, a Go policy engine, a Rust catalog server and three of our own workers (docs/2026-09-12-tenant-runtime-sidecar-design.md, §2.1's row table) — and every row shares four things: a command rebuilt identically on respawn by respawn_command, a port resolved by the one predicate child_port, a log through spawn_logged (stdout AND stderr into one O_APPEND file, Fresh on first spawn and Append on respawn, so a panic outlives its own recovery), and an ownership stamp — ARGONDB_BOX_INSTANCE in the child's environment plus (name, pid, start-time, port) in the children ledger, which is how the reaper proves what belongs to this box without matching a process NAME. Every child leads its own process group (.process_group(0)), which is what makes kill(-pid, SIGTERM) on a compute_ctl also reach its postmaster. An application sidecar is not a new kind of row; it needs those four plus two the others do not — a health probe and an isolation spec. It also settles where the row lives: a tenant's application runs as a SEPARATE OS PROCESS in this same table, under its own uid and cgroup, never inside the database's process — RULED [D-2026-09-20-4faa8c] (block G2); the in-process Rust embedding tier this rules out is reconsidered only on a measurement showing the loopback round trip moves a real workload, and that measurement is not commissioned.

2.3 The resident plant

Broker, pageserver, safekeeper, lake-store, catalog-pg, openfga, lakekeeper, argondb-tail, argondb-maintenance and argondb-mcp are resident: they are never stood down and a second tenant does not pay for them again. Measured on the box with one database, lake off, pool off: the whole box is 368 205 kB PSS, of which 231 116 kB is storage plus supervisor (docs/2026-09-12-tenant-runtime-sidecar-design.md, §9.1's verbatim M1 block). Two rows deserve their own note. catalog-pg is the bundled catalog's OWN Postgres, and it exists because until 2026-09-08 the bundled Lakekeeper stored its Iceberg metadata IN the primary that idle-stop stops — so idle_stop ON REFUSED that shape at config load. That refusal is gone, and its removal is the whole point of [D-2026-09-08-ca2e96] option B (crates/argondb-ctl/src/serve.rs:5759-5772). argondb-tail is resident by decision, not by omission: when the tenant's compute is stood down there is no WAL to decode, the tail sits at baseline, and its only use of the primary is schema introspection over the internal port with the internal application name — so it can never wake a stood-down tenant and never resets its idle clock (docs/2026-09-17-process-sidecar-design.md:254-265, :757-770). This residency — one process per tenant BOX, never idle-stopped, not moved into the supervisor's own process and its cache not moved to the pageserver — is RULED, not just built: Block S1, the process boundary [D-2026-09-18-515279]. Its row carries a stated memory bound, 6 144 MiB with the alarm line at 5 632 MiB (docs/2026-09-17-process-sidecar-design.md, §4.2's delta table); the budget per buffer, the 6 GiB cgroup memory.max bound alarmed at 5.5 GiB, and the 512 MiB floor a low share derives down to before the box refuses are Block S2, adopted as all three [D-2026-09-18-ff217a]. One tail per tenant DATABASE, as opposed to per box, is not claimed here: the ruled row is one tail per box, and it stays one tail per box with per_tenant on.

2.4 compute:<db> — the tenant's database

compute_ctl plus its postmaster, with its own pgdata, its own spec and its own port. The property that makes stopping it a memory decision rather than a durability decision is that a compute is stateless over shared storage — analyst_pool.rs wipes pgdata before every slot spawn precisely so durability lives in the storage tier and nowhere else (docs/2026-09-12-tenant-runtime-sidecar-design.md, §2.3, citing analyst_pool.rs:1664). Measured: one idle compute returns 133 906 kB to the box when it stands down (box PSS 361 561 kB UP → 227 655 kB DOWN), and is re-paid in 619 ms p50 (min 613, max 636, ten cycles) when someone comes back (§9.3a and §9.3 of the same document). Failure mode: a compute that never accepts TCP fails the wake loudly with its own log tail, charges the wake governor with healthy_reset = false, and — after six failures in 120 s — opens the wake breaker, after which every connection is refused by name until the box is restarted (crates/argondb-ctl/src/serve.rs:9161-9166, :9250-9262).

A company's compute starts at tenant create, on a port recorded at create. Ruled by block 2026-09-25 ~11:1xam ET, verbatim: "(a), a compute started at tenant create with ports assigned and recorded at create so a new company runs on its own compute from its first minute, is chartered now and is a ship item for the release after v1.5.5." Step 2 of tenant create assigns the company the lowest free slot of the tenant band and records it in tenants/<db>/compute-slot.json (crates/argondb-ctl/src/tenant_compute.rs fn assign_slot), then asks the RUNNING supervisor to start its compute: the request and the answer are two files under <data>/tenant-compute-start/, and the supervisor adds the row, its per-child state, its idle clock and its wake gate to the live table and publishes the port to the estate directory before it answers (crates/argondb-ctl/src/serve.rs fn start_tenant_row_at_create). The create returns only once that is true, so the company's first write already goes to its own compute and the primary refuses it. A compute that cannot start fails the create BY NAME at step 2, taken back out of every table — no database, roles, wall or published port exist yet — and re-running the command resumes onto the same slot. The slot, not a sorted position, is the port: inserting a company into the sorted list used to move every later company's port. Companies that predate the record keep the port they had — the boot's one-time migration records each one's sorted position once (fn migration_plan). v1.5.5 does carry this. The start at create is in the tagged release, and the v1.5.5 notes as cut say a company created while the box runs gets its own compute at creation (docs/releases/v1.5.5.md, section 26, item 9). An earlier draft of those notes, written on 2026-09-25 before the start at create merged, said "at the next restart", citing the data-half ruling [D-2026-09-25-f80e7e], and this chapter had repeated that draft. What remains of "the next restart" is a company created under v1.5.4: its pool entry comes at the upgraded box's first estate pass.

Rows written before the compute existed are copied once, and then the primary refuses the company's writes. Once a company has its own compute, the pass installs a refusal on the primary for that company's database: a statement trigger on every table refuses INSERT, UPDATE, DELETE and TRUNCATE, and an event trigger refuses DDL. Both name the company, its compute's port and the ruling (crates/argondb-ctl/src/tenant_data.rs fn refusal). Reads on the primary are unchanged. Only after that refusal is in place does fn copy_once copy the rows that had landed on the primary onto the company's own compute, once, rolled back whole on failure. The company's roles are replayed onto its compute by the same pass, from the primary's catalog (crates/argondb-ctl/src/tenant_roles.rs fn ensure_on_computes), so a connection routed there logs in as itself. §2.7 states the one known defect in the copy.

2.5 sidecar:<db> — the tenant's application

A per-company application process is BUILT, but not in the shape this section designs. What ships (crates/argondb-ctl/src/tenant_sidecar.rs, box gate testing/box/erp-sidecar-per-company.sh) starts one process per company, from a step of tenant create that runs after the company's compute is up. The command is read from the estate directory's sidecar_command column for that company, never from a path convention. A company whose column is empty is a complete tenant with no process at all. The process runs under the company's own recorded uid and its own cgroup, and its state is the tenant's own <data>/tenants/<db>/sidecar.json. A later pass adopts a live one, restarts a dead one, and refuses a second restart inside the breaker window. It is not a row in the supervisor's table. Under [compute] per_tenant the tenant's gate fronts its recorded APPLICATION port (§3.6): a request there to a stood-down tenant wakes it and is told to retry. It does not yet hold the tenant awake — bytes on the application port do not reset the idle clock (§3.7). Only the application's own SQL connections do that, because they pass through the compute's gate like any other client. Its limit, stated plainly: two companies need two distinct uids, creating a uid is a root step (useradd), and the product refuses to share the supervisor's uid between companies. Since v1.5.7.2 that root step is the sudoers-scoped argondb-useradd helper's, not a human's [D-2026-10-03-37002a]: root runs sudo <prefix>/bin/argondb box install-useradd-helper --user <supervisor-user> once per box, and the sidecar then calls sudo -n /usr/local/sbin/argondb-useradd <key> itself when argon-t-<key> is absent. On a box without the helper the company's application refuses by name, naming that install command.

The rest of this section, and §4.5, is the supervisor-row design, which is still designed and not built. One more row per tenant, whose command is whatever <data>/tenants/<db>/run names. The supervisor runs what is there and does not build it, deploy it, update it, version it, roll it back, or fetch it; a command that does not resolve is a refusal at spawn with the path named, exactly like a missing compute_ctl today — never a silently absent row (docs/2026-09-12-tenant-runtime-sidecar-design.md:364-372). Its environment is .env_clear() plus an explicit set, which is the single most important line in its spawn, because Command inherits the parent's environment by default and the supervisor's holds ARGONDB_AUTH_TOKEN, the lake credentials and ARGONDB_BOX_INSTANCE. ARGONDB_BOX_INSTANCE is the one variable re-added BY HAND: a sidecar that cleared it would be unreapable, which is worse than the leak the stamp exists to prevent (§4.2c of the same document). It gets exactly one secret — its own tenant's SQL credential — and a health probe that is an HTTP GET or a TCP accept. Its restart policy is an ordinary RestartPolicy::new(), and a tripped sidecar cannot take the tenant's database down: the trip arm sets tripped[i] for that row, records DEGRADED, alarms, and explicitly keeps supervising the rest.

2.6 The isolation box

Rung 1 of a four-rung ladder: one unix user per tenant plus one cgroup carrying MemoryMax, CPUQuota and TasksMax. The cap half needs zero privilege and is measured; the uid half needs one root command per tenant at provisioning and none at run time. What it does not cover is stated rather than left for the word "isolation" to carry: the tenant's own data (the app reads it, that is what it is for), a kernel-level escape (a uid is not a VM), and what the application does with data it is entitled to. §4.6 has the recipe and §6 has the gotchas.

Which rung each deployment target gets is now RULED, block G1 [D-2026-09-20-3425d9] — against this design's own recommendation. The on-prem box keeps rung 1 as above (rung 0, process-only, when no sidecar is configured); the one-click cloud VM requires rung 2, a container per tenant, on top of rung 1 — designed in docs/2026-09-20-rung-2-container-per-tenant-on-the-cloud-vm-design.md, not built, with its own blocks R1–R5 unput. Hosted multi-tenant — our hardware running code we did not write — is NOT covered by this ruling; a uid is not proportionate to mutually hostile tenants, and its tier is decided when that product is designed. The kernel-level-escape gap stated above is unchanged by G1: it is stated as uncovered wherever the isolation is described, on every rung this ruling names.

2.7 The pooler

Built and shipped 2026-09-20, no longer dashed. One PgBouncer process per box, resident, transaction mode, one pooled port for the box routed by the startup packet's dbname, with the direct port still there — block P1, option A, RULED [D-2026-09-20-d5c0f2]. Pool size follows the tenant's own connection limit at one quarter, otherwise option A's defaults (reserve_pool_size 0, max_db_client_connections 200, box-wide max_client_conn 10000, server_idle_timeout = the tenant's idle_secs, min_pool_size 0, max_prepared_statements 200) — block P2, option C, RULED [D-2026-09-20-5b6a85]. The pooled port is the DEFAULT connection string the provisioner prints; the direct port is printed beside it, labeled for migrations, LISTEN/NOTIFY, session SET, session advisory locks and desktop tools — block P3, option A, RULED [D-2026-09-20-5f9e67]; the box's own tools (argondb, the tail, --pg-url defaults) use the direct port regardless. The pooled port is a derived sidecar of the PG port it fronts (PG_PORT + 200, pooler::POOL_PORT_DELTA), not a separate band entry, so it moves with whatever port the primary is on. crates/argondb-ctl/src/pooler.rs is the one home of the shape (sizing, the ini generator, the auth lookup, the two connection strings); the supervised child is pooler in serve.rs; the box gate is testing/box/pgbouncer-pooler-gate.sh.

The pooler is PgBouncer specifically, pinned at 1.25.2 (a source build — Ubuntu's packaged 1.25.1 predates the SCRAM fixes the pin needs). The pooler is OPTIONAL — AMENDED 9/24 [D-2026-09-24-f0e068]: it is used when a supported binary is present, and the box boots on the direct port and says so when one is absent; no profile refuses to boot over it. [pooler] enabled is on, auto or off, and on and auto now share one behavior — pool if a working binary is found, otherwise decline loudly and serve the direct port only; the installer writes enabled = "on" as intent, not as a refusal switch. A binary below the 1.25.2 floor (pooler::MIN_VERSION), or one that cannot report its own version, is treated the same as absent.

What that means for a developer, key by key (pooler::Enabled::parse):

The connection string a tenant is handed is the pooled one only when a pool actually came up; otherwise it is the direct one. The pooler serves non-superuser login roles only, so cloud_admin is refused on the pooled port by design. The release installer (scripts/release/install.sh, write_pooler_intent) writes [pooler] enabled = "on" into <data>/argon.toml only when the key is unset, and never over a value already there.

With [compute] per_tenant on, each tenant's pool dials that tenant's own compute. The one routing predicate is pooler::upstream_port. With per_tenant off, every [databases] entry dials the box's gate port, exactly the v1.4 bytes. With it on, each entry dials the compute_port the estate directory records for that database. A database with no recorded port gets no entry at all, and the box log names it (pooler::unrouted_line, "it is NOT routed to the primary"). It is never quietly sent to the primary. Because a company's compute now starts at tenant create (§2.4), the create's own pass writes its pool entry. The no-entry case is left for a company whose compute is not up yet, which is at most one pass's cadence.

What per-tenant pooling does NOT prove today, stated plainly. The routing is proven in both directions: the pooler's box gate (testing/box/pgbouncer-pooler-gate.sh) lands each company's pooled session on its own compute under per_tenant on (inet_server_port() is the company's port, never the primary's), and on the primary under off. The routing, the roles replay and the data half are all in v1.5.5. What has not passed on main is the cross-company isolation gate with per_tenant on. The lane that set out to prove it, attempt 3 of pooler-ini-dials-each-tenants-own-compute-port-when-per-tenant-is-on, merged with a FAIL verdict: its dependency measured cross-company-isolation-eight-surfaces: FAIL (25/26 assertions passed [per_tenant=on], with S2+ failing on relation "ledger" does not exist because at that moment a company's data was not yet on its own compute. The data half landed later. A fourth attempt then reached 26/26 [per_tenant=on] only on its own branch, and that branch is not on main. On a tree equal to main it first measured a defect that is therefore still there. The publish-status writer can create argon.publish_status on a company's newly awake compute before the one-time copy runs, and the copy then fails and is rolled back whole. The log says tenant data: <company> — the one-time copy NOT done this pass, and every later pass refuses the copy because the compute already holds that table. That company's pre-compute rows stay on the primary, readable there, and are not on its own compute. A box that has hit this is not repaired by any later pass. With per_tenant off (the default) none of this applies: the isolation gate reads cross-company-isolation-eight-surfaces: PASS (26/26 assertions [per_tenant=off]. Whether that binary is vendored into the release artifact or required from the platform was its own, separate block: RULED [D-2026-09-23-520b66] — require it from the platform, rather than bundle a third-party binary into the release — this half stands. That ruling's OTHER half — refuse to boot if the binary is absent — is SUPERSEDED by the 9/24 amendment above, and the release artifact already matches the platform-required half: scripts/release/build-artifact.sh refuses a --pgbouncer flag by name ("the artifact bundles NO pgbouncer"), so no un-vendoring work remains owed.

2.8 The TODAY vs DESIGNED column, and the tenancy question under it

B′, one compute per tenant database, is RATIFIED and BUILT. It was put to a block on 2026-09-12 and not selected then [D-2026-09-12-44e647]. It was ratified on 2026-09-20 [D-2026-09-20-0d5ba3] as "each tenant database served by its own compute that stops when idle and wakes on connect", and amended on 2026-09-21 [D-2026-09-21-4107c5]. The code is crates/argondb-ctl/src/tenant_compute.rs, switched by [compute] per_tenant (§3.1). Every tenant database in the provisioner's record gets a compute:<db> row. On Linux and Mac there is no count cap [D-2026-09-28-715d12]: each new tenant compute takes the next free pair of ports at or above tenant_port_base (default 25450, tenant_compute::DEFAULT_PORT_BASE), skipping every port the box reserves for itself and every port another listener already holds (tenant_compute::next_free_pair). The first port of the pair is the public one, owned by that tenant's own gate; the second is the port its Postgres really listens on. The pair is recorded in tenants/<db>/compute-slot.json (SlotRecord) and reused on every restart; if another listener holds a recorded port at boot, that tenant alone is refused, by name, naming the port and its holder, and the box keeps serving every other tenant (recorded_pair_conflict). A compute that cannot start for lack of memory is refused the same way (memory_refusal). The AWS/Azure approach is a separate v1.5.7 question and is not built. Each row has its own idle clock and stands down and wakes exactly as §3.3 and §3.4 describe for the primary.

The amendment [D-2026-09-21-4107c5] puts each tenant compute on its OWN TIMELINE with its own WAL, a branch cut at tenant create, and "one cluster" means the shared storage plane: one pageserver, one safekeeper, one lake (HISTORY#tenant-timeline-per-database). The tail follows every tenant timeline as its own stream inside one process (crates/argondb-tail/src/streams.rs; HISTORY#tail-multiplexes-tenant-timelines). The ERP spec's open item 7 was closed by the rework against this topology (HISTORY#erp-spec-topology-rework, HISTORY#erp-spec-reworked-to-per-tenant-timelines).

Separately, whether the box's SHIPPED DEFAULT ever flips to per-tenant compute is settled, and the answer is no system default at all: "Per-tenant compute is the operator's setting, never the system's" — RULED [D-2026-09-23-5122a6] (the amended B-MT1). Whatever [compute] per_tenant resolves per box, it is something an operator turns on, never something the box defaults to on its own.

3. How it works

The complete walk of the control and data paths, with what is atomic, what is idempotent, what is refused and with which message, what is disclosed, and what is measured.

3.1 Configuration resolution, at boot

The [compute] section of argon.toml has three keys (ComputeSection, crates/argondb-ctl/src/serve.rs:106-133). The two per-tenant keys come first, because they depend on the third:

• per_tenant, a TOML boolean, default false, turns on the per-tenant compute table (§2.8). ARGONDB_COMPUTE_PER_TENANT wins over it and takes true/false, on/off, 1/0. Any other value is refused by name (tenant_compute::resolve_per_tenant). per_tenant on with idle_stop off is REFUSED at config load, because the ruling is a compute "that stops when idle", and N computes that never stand down would be the cost with none of the point. The message names both knobs and both remedies: "Set idle_stop to a number of seconds (or ARGONDB_PRIMARY_IDLE_SECS), or turn per_tenant off."
• tenant_port_base, default 25450, is the base of the tenant band. ARGONDB_TENANT_PG_PORT wins over it. The band is validated at config load against every port the box already owns, so an overlap is refused before anything binds, and 0 is refused as not a TCP port.

[compute] idle_stop itself is a string, deliberately: "off" (the default) or a whole number of seconds, one spelling for both arms so that one parser — idle_stop::parse_idle_stop (crates/argondb-ctl/src/idle_stop.rs:142) — serves both the file key and the environment knob and the two cannot drift. idle_stop = 300 as a TOML integer is refused by serde with the type in the message and the remedy is quotes. Precedence is ARGONDB_PRIMARY_IDLE_SECS > argon.toml > default OFF (resolve_idle_stop, idle_stop.rs:166), matching the box's existing shape for port and memory_budget_mb, and the env spelling wins including when it is the disabling one: an operator who exports off has turned it off whatever the file says (idle_stop.rs, the env_wins_over_toml test). Two values are refused loudly rather than defaulted, because a knob that silently becomes its default is a trap and this one decides whether the database stops:

• a value that is neither off nor a whole number, naming the source;
• 0, which "would stand the primary down the instant it goes idle, which is a restart loop rather than a policy".

The whole section is deny_unknown_fields, as is every section of argon.toml (serve.rs:48-57): a typo'd key is refused with the accepted set named, on the doctrine that a knob the operator believes in and that does nothing is the worst outcome a config surface can produce.

Resolution happens at serve.rs:5752, with the other config MUSTs, because it decides which ports the box binds and the preflight plans them from it.

3.2 The idle predicate

The supervisor runs ONE long-lived probe connection to the primary's internal port and executes ONE query per tick — ACTIVITY_QUERY (crates/argondb-ctl/src/idle_stop.rs:205) — at PROBE_CADENCE_SECS = 2 (:95). One round trip on a held connection, because a fresh backend every two seconds would itself perturb pg_stat_activity, which is the thing being measured. The query returns five numbers: busy client backends, the age of the most recent idle client backend, walsenders (excluding walproposer), logical subscriptions with a live pid, and autovacuum workers. Every arm mirrors the inherited compute_ctl monitor and is cited to it at the arm.

Two exclusions are load-bearing, and they are in the SQL itself rather than in the caller: application_name NOT LIKE 'argondb_internal%' (INTERNAL_APP_PREFIX, idle_stop.rs:79 — every connector the box itself makes, the probe included, which names itself argondb_internal_idle_probe, :82) and NOT LIKE 'compute_ctl%'. Without the first, the monitor would count as activity and hold the box awake forever.

Reading::hold() (idle_stop.rs:267-286) returns the first of four AwakeHold reasons in the monitor's own precedence order — BusyClient, Walsender, LogicalSubscription, Autovacuum — each of which resets the idle clock to now. A hold is a reason the database cannot be stopped. decide (:307) is pure and takes last_active by &mut, and it only ever advances it, never retreats it, so a stale reading cannot pull the clock backwards; a negative age (clock skew inside the database) is treated as "now", which is the conservative direction because it keeps the box awake. The window boundary is inclusive (recent_activity, :288): exactly idle_secs since the last activity still counts as recent, one second past does not — the identical rule, for the identical reason, as analyst_pool::recent_activity (crates/argondb-ctl/src/analyst_pool.rs:540).

A primary the probe cannot measure is never stood down. A failed probe connect logs and continues, leaving the clock alone (crates/argondb-ctl/src/serve.rs:9016-9024); a failed read drops the probe, reconnects next tick and leaves the primary up (:9029-9036). A measurement that cannot tell "zero" from "I failed to look" must fail toward AWAKE, and here it does.

3.3 Stand-down, in five steps, and the order is the fix

The arm is serve.rs:9049-9144. It runs only when idle_secs.is_some() and the compute is not already stood down (:9007).

1. Drop the probe's own session first. Otherwise Postgres waits on us to disconnect during its own shutdown.
2. compute_up.store(false) BEFORE any signal (:9079). This is the step whose ORDER is the whole fix. A connection arriving between the SIGTERM and the port's release would otherwise be spliced into a Postgres that is shutting down and the client would get FATAL: the database system is shutting down instead of a wake. Flipping the flag first makes that window a wake request, which the supervisor serves as soon as it is back in select. This was witnessed on the gate's second run; it is not theoretical.
3. kill(-pid, SIGTERM) — the process GROUP, not the leader, because compute_ctl's postmaster is its process-group child and signaling compute_ctl alone can leave Postgres behind.
4. A double-bounded wait: STAND_DOWN_GRACE = 60 s (idle_stop.rs:112) AND the child's own exit. Child::wait also REAPS, which is what lets the child be excluded from the supervisor's wait set without leaving a zombie. On expiry: SIGKILL the group, and print the log tail.
5. Wait for the PORT to be free — poll connect(127.0.0.1:<internal>) every 100 ms for up to another 60 s (:9115-9140). This is the step a naive implementation omits: compute_ctl can exit before its Postgres has finished its shutdown checkpoint and released the socket, and a wake that spawned into that window would lose the bind to a corpse. On expiry the group is SIGKILLed with a WARNING naming the port.

Then stood_down[compute_idx] = true. stood_down is a second vector, never a third state on tripped, and the distinction matters: a stood-down child is not a tombstoned one — it is respawnable, it has charged no breaker, and the "every child tripped ⇒ the box is dead" bail must not fire because the primary is asleep.

Measured cost: 34 s against a 20 s window. The extra 14 s is Postgres's shutdown checkpoint plus step 5, and it lands on the stand-down, never on the wake (docs/2026-09-12-tenant-runtime-sidecar-design.md, §9.3a).

3.4 Wake, from the first packet

1. Bytes arrive at the gate — and bytes, not accept(), are the trigger (idle_stop.rs:517). This is load-bearing twice: argondb status's reachability probe connects and drops, so waking on accept would make the box's own status command hold it awake forever; and every modern browser opens speculative pre-connects that send nothing, so a sidecar-fronting gate that woke on accept would hold every tenant awake forever. A read timeout or a zero-byte read returns immediately and spawns nothing.
2. The gate checks compute_up (an AtomicBool, SeqCst) and, if it is false, sends a WakeRequest on a 64-deep channel (serve.rs:8860) and holds the client connection open while awaiting the reply, bounded by WAKE_DEADLINE = 180 s (idle_stop.rs:107).
3. The supervisor acts (serve.rs:9152 onward), because child handles live there and nowhere else — which is what keeps ONE owner of the compute process across boot, crash-respawn and wake. It rebuilds the command with respawn_command (not a second, drifting recovery path) and spawns with LogMode::Append, so the previous incarnation's shutdown and any failure that preceded it outlive their own recovery.
4. wait_tcp_accept, 120 × 500 ms, double-bounded on the compute pid (serve.rs:11253): it fails IMMEDIATELY with the compute's log tail when the producer dies rather than running out the clock. The deadline is the backstop, not the mechanism.
5. On success: stood_down = false, compute_up = true, policies[compute_idx].note_spawn(now), last_active = now, a mark in the child log reading woken by a client connection on the public PG port, and the reply goes back to the gate.
6. The gate splices: set_nodelay(true) on both halves (idle_stop.rs:584-585 — Nagle off, because a 40 ms coalescing delay per round trip is exactly what a splice-overhead measurement would find), the buffered first bytes are forwarded, and then copy_bidirectional (:590). After that the gate speaks no protocol again.

Three idempotence and race properties, each deliberate. A wake request that arrives when the compute is already up replies Ok(()) immediately — it raced another connection's wake or a crash-respawn (serve.rs:9154-9159). A wake inside the governor's backoff is refused cheaply rather than spawning, which is what turns a client retry storm into one spawn per backoff step instead of one spawn per connection. And a failed wake cleans up before it charges the governor: SIGTERM the group, wait up to 10 s, set stood_down back to true, then on_exit_healthy(now, false).

3.5 The wake governor is the restart policy, not a second breaker

wake_policy (serve.rs:11603) is built from the same RestartPolicy::new() every child respawn uses — its third consumer, after the supervised children and the analyst pool's slot recovery. The constants are one definition (crates/argondb-ctl/src/restart_policy.rs:35-43): BASE_BACKOFF_SECS = 2, MAX_BACKOFF_SECS = 30, HEALTHY_RESET_SECS = 60, BREAKER_MAX_RESTARTS = 6 in BREAKER_WINDOW_SECS = 120. Per-row constants are a supported shape: argondb-mcp trips on the THIRD exit inside sixty seconds (MCP_BREAKER_MAX_RESTARTS = 2, restart_policy.rs:67-68), because its characteristic failure is a deterministic CONFIGURATION refusal — it resolves and fail-closes every configured surface before it binds any of them, so a bad knob makes it exit in milliseconds, forever, and retrying re-reads the same environment and reaches the same conclusion.

For N tenants the consequence is the one that matters: a tenant whose compute cannot start burns its own breaker and nobody else's.

A trip does four things and no more: stop respawning that row, alarm loudly, record a DEGRADED box state readable by argondb status, and keep supervising everything else. Never a silent infinite loop — CLAUDE.md's expensive-automatic-recovery rule, N recoveries in M minutes ⇒ stop and alarm.

3.6 What a failed wake says, and why it says it in Postgres

The gate answers with a real Postgres wire ErrorResponse built by error_response (idle_stop.rs:656), written against the protocol 3.0 message grammar: byte 'E', an Int32 length counting itself but not the type byte, then NUL-terminated field entries (S=FATAL, V=FATAL, C=the SQLSTATE, M=the message), then a final zero byte. The SQLSTATE is 57P03 cannot_connect_now (:448) — the code Postgres itself uses while it is starting up, which every driver already classifies as "retry later" rather than "your credentials are wrong." There are four distinct messages, and each names its own remedy:

Two more come from the supervisor as the {why} of the second row: the rate limit (the primary failed to start and is in a {n}s backoff; retry shortly) and the open breaker (the primary's wake breaker is OPEN after repeated start failures ({n} attempts); see logs/compute.log. It will not be retried automatically — restart the box once the cause is fixed).

The application port has the other vocabulary: HTTP 503 with Retry-After. Under [compute] per_tenant each tenant's gate owns a SECOND listener, the tenant's APPLICATION port, in front of its sidecar. The port is RECORDED, like the tenant's other ports: tenants/<db>/compute-slot.json carries app_public_port (the gate's) and app_internal_port (where the sidecar listens, loopback), each the next free port the allocator found at create — in the same pass and under the same lock as the PG and compute_ctl HTTP pairs, starting where the PG pair starts — and reused as recorded on restart [D-2026-09-28-715d12] (tenant_compute::TenantTable::app_pair_of). A record written before the pair existed gets one at boot and keeps every other port it had. The vocabulary is chosen by WHICH LISTENER accepted, never by sniffing the first bytes, because sniffing would be a grammar and a grammar is machinery (docs/2026-09-12-tenant-runtime-sidecar-design.md §5). The first-bytes rule is the SQL port's, unchanged: a browser's pre-connect wakes nothing. What differs is that the application port does NOT hold the client through the wake. First bytes to a stood-down tenant begin the wake and are answered at once (idle_stop::handle_app_conn):

HTTP/1.1 503 Service Unavailable
Retry-After: 1
Content-Type: text/plain; charset=utf-8
Connection: close

tenant acme: the database compute is waking (4 ms in; its last start took 640 ms); retry after 1s

Retry-After is computed from a MEASUREMENT, never a constant (idle_stop::retry_after_secs). It is what remains of the tenant's last measured start, rounded up and never below 1. The row's WakeClock is seeded with the compute's measured start (spawn to accepting TCP, at boot or at create — the same two steps a wake runs) and overwritten by every successful wake the gate times, on either port. If a wake fails, the next 503 names why. Once the compute is up, the connection is spliced to the sidecar's internal port exactly as the SQL port splices to Postgres. A sidecar that is not listening gets 502 Bad Gateway naming the APPLICATION and no Retry-After, because no wake can fix it. The SQL port's four messages above are unchanged. Gate: testing/box/erp-sidecar-per-company.sh P12-P15. Not built: bytes on the application port do not yet reset the tenant's idle clock (design §7.2's last_sidecar_bytes), so a tenant used ONLY through its application stands down after its window and its next request is a 503 and a wake. The supervisor also does not yet start or stop the sidecar with the compute (§3.8); the provisioning pass (argondb tenant sidecar) runs it. The sidecar learns its internal port from its own command line; the box prints it at boot (tenant compute <db> application gate listening on …).

3.7 The idle predicate with an application in it

Designed, and its shape is now RULED, block G4 [D-2026-09-20-78f71e] — AGAINST this design's own recommendation, which would have added a fourth term for an open-but-silent sidecar connection. Bytes are the only idle signal. No connection-count hold, no second, longer window for a held-open socket, no busy flag the application can raise. The rule is one component sees all the traffic, and it is a rule rather than a preference because an application that keeps its own idle timer is a second control loop over one lifecycle, and it fails in the expensive direction: an app with a 30-second poll loop is never idle by its own reckoning, so the tenant's database never stands down and per-tenant scale-to-zero buys nothing. The gate fronts both ports, so:

tenant_idle  =  sql_reading.hold().is_none()
             && now - last_sql_activity      >  idle_secs
             && now - last_sidecar_bytes     >  idle_secs

last_sidecar_bytes is the gate's OWN number — it is already counting first bytes — so this needs no probe, no agent inside the application, and no cooperation from it at all. The idle signal for foreign code is measured at our socket, not inside their process. A fifth AwakeHold is deliberately NOT added: a hold is a reason the DATABASE cannot be stopped and a busy application is not one; it is a reason the TENANT is not idle, which the predicate's third term already carries. The consequence of G4's ruling: a browser holding a WebSocket open, silent for the idle window, is not a hold — the tenant stands down under it, the client reconnects on its own next bytes and the gate wakes the tenant, and server-push over a silent socket does not survive a stand-down. Every WebSocket client must already handle a reconnect, which is the ruling's own argument for it.

3.8 Wake and stand-down order with an application

Wake: compute first, then the application. wait_tcp_accept proves the compute is serving, then the sidecar is spawned and its health probe is waited on under the same double-bounding rule — an explicit deadline AND kill -0 on its pid, failing loudly with its own log tail. Stand-down is the mirror: the application first (SIGTERM to its process group, its own grace), then the compute's existing five-step sequence unchanged. The asymmetry is the design: the app must never be up with its database down, at either end.

A SQL-port connection is released at the compute step, not the application step. A psql session does not block on somebody's web server coming up. A connection that arrived on the application port waits for the health probe, because there is nothing for it to be spliced to until the app answers. This is stated because it is the kind of thing that otherwise gets decided by accident in code.

The application half of the wake latency is not ours to bound. A Go binary answering /healthz in 20 ms and a JVM taking 8 seconds are both legitimate applications. The obligation is to MEASURE and DISCLOSE the total, not to bound it.

3.9 Routing, when there is more than one tenant

The shipped gate maps one port to one compute. A multi-tenant gate needs exactly one new thing: routing, and there are three places that information can come from — a port per tenant, the startup packet's database field, or TLS SNI. The recommendation is a port per tenant: zero protocol parsing, zero new grammar, it is what the analyst pool already does, and it composes with the application row without extra thought because that row gets a port in the same band. Parsing the startup packet would put our code in front of the TLS handshake; SNI means terminating TLS at the gate. The routing problem is already solved by the operating system.

That recommendation is what shipped. With [compute] per_tenant on, each tenant's gate owns its own public port in the tenant band (§2.8), and the pooler's one pooled port routes by the startup packet's dbname to that tenant's port (§2.7). PgBouncer does that parsing, and the gate still parses nothing.

3.10 What is measured, and under which conditions

What an idle box still does. The hour above is the measurement behind a plain statement: a booted box with nothing to do raises nothing and advances nothing in the view maintainer. Two things still move, and neither is a fault. The decoded WAL head creeps forward with no write and no cut (1,205,552 bytes over the hour, about 335 B/s), so watch cuts, not the head. And the maintenance sweep in the same tail process WARNs `maintenance pin

PSS and not RSS, because every Postgres backend maps the same shared_buffers segment and summing RSS counts it once per process.

The 630-resident-computes figure that follows from 87 GB of available memory is arithmetic, not a measured ceiling, and the source says so: PSI stayed flat at three computes, so the pressure knee was never approached. Memory is also unlikely to be the binding constraint — 600 computes is 600 listeners, 600 pgdata directories, 600 timelines on one pageserver and 600 × max_connections file descriptors, and none of those is measured. The per-tenant table no longer sets a count of its own on Linux and Mac [D-2026-09-28-715d12]: the number of tenant computes on one box is bounded by free ports and by memory, and a compute refused for either is refused for its tenant alone (tenant_compute::next_free_pair, tenant_compute::memory_refusal).

4. How to use it

Every example below is labeled VERIFIED — run on the merged tree or on the agent node this guide was written on, with its real output pasted — or FROM SOURCE — the exact signature, flag, message or table quoted from the code or the design by path:line. An example that could be grounded neither way is not in this document.

4.1 Configure [compute] idle_stop, and both ways it refuses — FROM SOURCE

The section's idle_stop key, quoted from crates/argondb-ctl/src/serve.rs:106-133 (the per-tenant keys follow at the end of this section):

# argon.toml — the box's whole file surface.
# Every section is deny_unknown_fields: a typo'd key is REFUSED with the
# accepted set named, never warned about and ignored.

[compute]
# "off" (the default: the primary never stands down), or a whole number of
# seconds as a STRING. A TOML integer is refused by serde with the type in
# the message; the remedy is quotes.
idle_stop = "300"

[resources]
memory_budget_mb = 2048

Precedence, from idle_stop::resolve_idle_stop (crates/argondb-ctl/src/idle_stop.rs:166) — env beats file beats default:

# 1. the file's 300 s applies
argondb serve --data-dir /srv/argon

# 2. the env wins: 60 s, whatever argon.toml says
ARGONDB_PRIMARY_IDLE_SECS=60 argondb serve --data-dir /srv/argon

# 3. the env wins EVEN WHEN IT IS THE DISABLING SPELLING.
#    An operator who exports `off` has turned it off.
ARGONDB_PRIMARY_IDLE_SECS=off argondb serve --data-dir /srv/argon

The two refusals, verbatim from parse_idle_stop (crates/argondb-ctl/src/idle_stop.rs:142-165). Expected output for idle_stop = "5 minutes":

[compute] idle_stop="5 minutes" is neither `off` nor a whole number of seconds.
Use `off` (the default: the primary never stands down) or a positive idle window
in seconds, e.g. 300.

And for idle_stop = "0":

[compute] idle_stop=0 would stand the primary down the instant it goes idle,
which is a restart loop rather than a policy. Use `off` to disable idle-stop,
or a positive idle window in seconds.

Note the source name in the message. Set the same bad value through the environment and the first token is ARGONDB_PRIMARY_IDLE_SECS instead — one parser, two sources, and the message tells you which one you are fighting.

Turning on a compute per tenant. per_tenant is a boolean and needs an idle window beside it (§3.1):

[compute]
idle_stop        = "300"   # REQUIRED with per_tenant: each tenant's compute stands down after this
per_tenant       = true    # default false; ARGONDB_COMPUTE_PER_TENANT=on|off wins over it
tenant_port_base = 25450   # the default; slot i = base+2i public (its gate), base+2i+1 internal

At boot the box prints one line per recorded company slot and then one summary line: per-tenant computes: ON, the number of tenant databases taken from the provisioner's record, the band as <lo>-<hi>, and the idle window each compute stands down after (serve.rs, after resolve_boot_slots). A company created while the box runs gets its slot and its compute from tenant create (§2.4). Point its clients at the pooled port or at its own public port in the band, not at the primary's port. The primary still answers reads of that database, but it refuses every write to it, by name.

4.2 Observe a stand-down and a wake, with timestamps — FROM SOURCE

What the box prints, in order, with the emitting site. Every string below is quoted from the code; the wall-clock stamps are the ones a reader will see in their own log and are shown as HH:MM:SS placeholders rather than invented values.

# Turn it on with a short window and watch the supervisor's stderr.
ARGONDB_PRIMARY_IDLE_SECS=20 argondb serve --data-dir /srv/argon 2>&1 | ts

Boot — the gate announces the port split (idle_stop.rs:478-482):

[argondb-box] primary gate listening on 127.0.0.1:5432 → postgres :5532 (idle_stop ON)

A client connects and runs a query; the predicate holds the box awake and says which of the four reasons it is (serve.rs:9039-9047, AwakeHold::why() at idle_stop.rs:242-250). The line is printed only when the reason CHANGES, so a long query does not spam the log:

[argondb-box] primary held awake: a client query is running (idle_stop 20s)

The client disconnects. Twenty seconds later the stand-down arm runs (serve.rs:9061-9071), writing one mark into the compute's own child log and one line to the supervisor's stderr:

[argondb-box] primary idle 20s — standing it down (storage stays up; the next client connection on :5432 wakes it)

and, in <data>/logs/compute.log:

standing the primary down after 20s idle ([compute] idle_stop); the next client connection on :5432 wakes it

If the compute does not stop in STAND_DOWN_GRACE = 60 s, or if the internal port still accepts 60 s after the signal, the box says which and SIGKILLs the group (serve.rs:9104-9108, :9129-9137):

[argondb-box] the primary did not stop within 60s of SIGTERM — SIGKILLing its process group. Log tail:
[argondb-box] WARNING: :5532 still accepts connections 60s after the primary's stand-down — SIGKILLing its process group

The next connection's first byte wakes it (idle_stop.rs:526, serve.rs:9180, :9226):

[argondb-box] primary gate: 127.0.0.1:41922 asked for a stood-down primary — waking
[argondb-box] waking the primary (attempt 1)
[argondb-box] primary awake (attempt 1)

and the compute's child log carries the mark woken by a client connection on the public PG port.

To see the split with your own eyes rather than in a log, the box gate's phase A/B assertion is the shape to copy — with the knob OFF the PUBLIC port is owned by a postgres process and the internal port is not bound at all; with it ON the supervisor owns the public port and postgres owns public + 100 (testing/box/idle-stop-wake.sh:17-24):

ss -tlnp | grep -E ':(5432|5532)\b'
# idle_stop OFF:  127.0.0.1:5432  users:(("postgres",...))      and :5532 absent
# idle_stop ON :  127.0.0.1:5432  users:(("argondb",...))       and :5532 owned by postgres

The measured wake, ten cycles, quoted from docs/2026-09-12-tenant-runtime-sidecar-design.md, §9.3:

  cycle  1:    636 ms   cycle  6:    616 ms
  cycle  2:    625 ms   cycle  7:    617 ms
  cycle  3:    634 ms   cycle  8:    613 ms
  cycle  4:    618 ms   cycle  9:    620 ms
  cycle  5:    621 ms   cycle 10:    619 ms
M4  wake latency over 10 cycles: min 613 ms · p50 619 ms · max 636 ms

4.3 What a client sees while waking, and how a driver should retry — FROM SOURCE

On the happy path a client sees nothing. The gate holds the connection for ~619 ms and then splices; there is no error to handle and no pre-warm ping is needed, which is a property of the gate rather than of the number — the gate holds the first connection instead of failing it. (The published guidance for platforms whose control plane is NOT on the connection path is the opposite: send a lightweight SELECT 1 to pre-warm. That guidance does not apply here.)

On a failed wake the client gets SQLSTATE 57P03. Handle it as "retry later", not as an authentication or configuration error. The retry budget should respect the supervisor's own governor: backoff doubles 2 s → 30 s and the breaker opens after 6 failures in 120 s, so a client retrying faster than that gains nothing — the refusals inside the backoff window are answered without a spawn.

Python, psycopg 3 — sqlstate is 57P03:

import time, psycopg

def connect_with_wake_retry(dsn, attempts=6, base=1.0, cap=30.0):
    """57P03 cannot_connect_now == 'the tenant is waking, or its wake failed'.
    Anything else is a real error and is re-raised immediately."""
    delay = base
    for n in range(attempts):
        try:
            return psycopg.connect(dsn)
        except psycopg.OperationalError as e:
            if getattr(e, "sqlstate", None) != "57P03":
                raise
            if n == attempts - 1:
                raise
            time.sleep(delay)
            delay = min(delay * 2, cap)

Go, pgx v5 — the same classification on *pgconn.PgError.Code:

func connectWithWakeRetry(ctx context.Context, dsn string, attempts int) (*pgx.Conn, error) {
    delay := time.Second
    var err error
    for n := 0; n < attempts; n++ {
        var c *pgx.Conn
        c, err = pgx.Connect(ctx, dsn)
        if err == nil {
            return c, nil
        }
        var pgErr *pgconn.PgError
        if !errors.As(err, &pgErr) || pgErr.Code != "57P03" {
            return nil, err // not a wake failure — do not retry
        }
        select {
        case <-ctx.Done():
            return nil, ctx.Err()
        case <-time.After(delay):
        }
        if delay < 30*time.Second {
            delay *= 2
        }
    }
    return nil, err
}

JDBC exposes the same string through SQLException.getSQLState(). The point of choosing 57P03 rather than inventing a code is exactly this: the classification already exists in every driver (crates/argondb-ctl/src/idle_stop.rs:445-448).

What you will NOT see is a bare socket close on a failed wake, and that is what error_response exists for. The field codes it writes are S, V, C, M — severity, severity-non-localized, SQLSTATE, message — so e.diag.message_primary (psycopg) / pgErr.Message (pgx) carries the box's own sentence, including the path to logs/compute.log on a deadline failure.

One trap, stated because it is the failure mode a reader will hit. A silent connection is not a wake. A TCP health check that connects and drops, or a browser's speculative pre-connect, will be closed after 30 s having woken nothing (idle_stop.rs:101, :517-522). If you want a monitor to keep a tenant awake, it must SEND something.

4.4 The analyst pool: lazy mode, cap and port band — FROM SOURCE

The analyst pool is the shipped precedent for several computes on one box with their own idle rule, and you configure it entirely through the environment (crates/argondb-ctl/src/analyst_pool.rs:168-250):

# mode: off | lazy | always | auto (the default).
#   auto resolves by detected RAM: `always` at or above 16 GiB, `lazy` below
#   (analyst_pool.rs:55, :300-317).
#   lazy  — computes start on the first analyst query and idle-stop after the
#           analyst idle window (DEFAULT_IDLE_SECS = 300, analyst_pool.rs:103)
#   always— cap computes warmed at boot
#   off   — no replica computes, no static computes, no pin-redemption endpoint
ARGONDB_ANALYST_POOL=lazy \
ARGONDB_ANALYST_PG_PORT=25440 \
  argondb serve --data-dir /srv/argon

The port band is arithmetic from one base, and this is the shape a per-tenant compute band copies (analyst_pool.rs:124-129, :251-264):

Defaults: DEFAULT_PORT_BASE = 25440 (:124), DEFAULT_CAP = 2 (:107), DEFAULT_STATIC_CAP = 2 (:118), RECONCILE_CADENCE_SECS = 2 (:113).

Three refusals you can provoke, each at config load so the box never binds a compute whose coordinates are wrong (analyst_pool.rs:173-230):

# ARGONDB_ANALYST_POOL=lazy with a cap of 0:
… ARGONDB_ANALYST_POOL=lazy asks for one — set the pool to `off`, or a cap ≥ 1

# ARGONDB_ANALYST_PG_PORT=0 or a non-port:
ARGONDB_ANALYST_PG_PORT=0 is not a valid TCP port (1-65535)

# a band that would overflow, or whose members would collide:
ARGONDB_ANALYST_PG_PORT=<n> + --port-offset <k> overflows …

validate_port_band(cap, static_cap) (:269) is called before anything binds, and static_ports_never_collide_with_the_redeem_port is a unit test rather than a comment (:2950).

Two behaviors of lazy that surprise people. want_ready(mode, cap, recent_activity) (:522) returns cap while there was recent activity and 0 otherwise — that IS per-compute scale-to-zero. And a lazy pool reports its computes as DOWN in argondb status, which is the configured steady state, not a fault; the status walk says so in its own words (crates/argondb-ctl/src/serve.rs:5169-5172):

the pool is `lazy` — computes start on the first analyst query and idle-stop after the analyst idle window

The pool's computes are REPLICAS of one timeline, not databases of their own. That single sentence was the whole distance between the analyst pool and a per-tenant compute table; the table is now built separately, each tenant compute on its own timeline (§2.8).

One coupling to know: the pool's only consumer is the MCP child, so ARGONDB_MCP_PORT=0 disables the pool regardless of ARGONDB_ANALYST_POOL, and status says which switch did it (serve.rs:5158-5163).

4.5 The application sidecar row, as designed — FROM SOURCE

Design; a first version is built (§2.5): one process per company, recorded in <data>/tenants/<db>/sidecar.json (crates/argondb-ctl/src/tenant_sidecar.rs), with the command read from the estate directory rather than the run path below, and without the per-company uid. The rest is the contract a tenant's application is held to, quoted from docs/2026-09-12-tenant-runtime-sidecar-design.md §4.1–§4.3 and docs/2026-09-14-argondb-process-model-preliminary-design.md §3.2.

The supervisor reads exactly one path per tenant:

<data>/tenants/<db>/run        # a command line. The supervisor runs what is there.

A small HTTP application as a tenant's sidecar — the whole contract is: bind the port you are given, answer the health path, read your DSN from the environment, and exit on SIGTERM.

# /srv/argon/tenants/acme/run      (mode 0755, owned by argon-t-acme)
#!/bin/sh
exec /srv/argon/tenants/acme/bin/acme-app \
     --listen "127.0.0.1:${ARGONDB_SIDECAR_PORT}" \
     --health-path /healthz

# acme-app, in ~20 lines. Nothing here is ArgonDB-specific except the two
# environment variables it reads.
import os, http.server, psycopg

DSN  = os.environ["ARGONDB_SIDECAR_DSN"]      # the tenant's OWN credential, and the only secret
PORT = int(os.environ["ARGONDB_SIDECAR_PORT"])

class App(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/healthz":           # the supervisor's health probe
            self.send_response(200); self.end_headers(); self.wfile.write(b"ok"); return
        with psycopg.connect(DSN) as c:
            n = c.execute("select count(*) from orders").fetchone()[0]
        self.send_response(200); self.end_headers()
        self.wfile.write(f"{n} orders\n".encode())

# SIGTERM to the process GROUP is the stop signal. Default handling (terminate)
# is correct; install a handler only if you need a drain.
http.server.HTTPServer(("127.0.0.1", PORT), App).serve_forever()

The row the supervisor builds for it, field by field (docs/2026-09-12-tenant-runtime-sidecar-design.md, §4.1's table):

The environment it receives is a CLOSED set. It gets its DSN, its port, and ARGONDB_BOX_INSTANCE re-added by hand. It does not get ARGONDB_AUTH_TOKEN, the lake credentials, or anything else the supervisor was started with.

What the supervisor will not do for it: build it, deploy it, update it, version it, roll it back, or fetch it. A run that does not resolve is a refusal at spawn with the path named — exactly like a missing compute_ctl today — never a silently absent row.

4.6 The isolation rung-1 recipe: one root command per tenant — VERIFIED (the probes) and FROM SOURCE (the two root commands)

Run on the primary agent node on 2026-09-20 as an unprivileged user, no sudo anywhere. Verbatim:

$ stat -fc %T /sys/fs/cgroup
cgroup2fs

$ cat /proc/self/cgroup
0::/user.slice/user-1000.slice/session-63939.scope

$ cat /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers
cpu memory pids

$ loginctl show-user "$(id -un)" -p Linger
Linger=yes

$ systemd-run --user --scope -p MemoryMax=32M --unit=guide02-cap-probe \
      bash -c 'cat /sys/fs/cgroup/$(awk -F: "{print \$3}" /proc/self/cgroup)/memory.max'
Running as unit: guide02-cap-probe.scope; invocation ID: 3616a8fc799a4d44b15f60245641f8d5
33554432

$ systemctl --version | head -1
systemd 259 (259.5-0ubuntu3.4)

Read the cap probe. The scope was created with no privilege, and the process inside it read its own memory.max back as 33554432 — 32 MiB, the value that was asked for. That is a positive control, not an absence of an error: MemoryMax, CPUQuota and TasksMax for a tenant's application cost zero privilege on a cgroup2 box with cpu memory pids delegated.

The uid half is one root step per box, then none per tenant [D-2026-10-03-37002a]. Root installs the helper and its sudoers drop-in once:

# once per box, as root — writes /usr/local/sbin/argondb-useradd (root, 0755)
# and /etc/sudoers.d/argondb-useradd (root, 0440, checked by visudo -cf),
# letting the supervisor's user run exactly that helper with NOPASSWD:
sudo <prefix>/bin/argondb box install-useradd-helper --user <supervisor-user>

From then on the sidecar creates argon-t-<key> itself — sudo -n /usr/local/sbin/argondb-useradd <key>, which validates the key against the product's key grammar and runs the one useradd below and nothing else. The hand command stays the FALLBACK for a box where the helper is not installed, and the sidecar's refusal prints both. The fallback and the separate setcap step, quoted from docs/2026-09-12-tenant-runtime-sidecar-design.md:663-673:

# once per tenant, at provisioning, as root:
useradd --system --no-create-home --shell /usr/sbin/nologin argon-t-<db>

# once per box, at install, as root — ONLY if the supervisor is to switch uid
# itself (it does not run as root and should not start as root):
setcap cap_setuid,cap_setgid+ep /usr/local/bin/argondb

setcap cap_setuid on the supervisor binary is a real privilege grant and the source argues for it rather than slipping it in. The alternatives are a setuid-root helper (worse) or systemd-run --user under a user manager that is itself the tenant's user, which moves the problem to "who creates the user" and lands back at useradd. There is no path to a per-tenant uid that does not cross root once.

Putting them together, the shape of a rung-1 spawn:

# per tenant, at run time, no privilege beyond the uid switch:
systemd-run --uid=argon-t-acme --gid=argon-t-acme \
  --scope --unit=argon-sidecar-acme \
  -p MemoryMax=512M -p CPUQuota=50% -p TasksMax=64 \
  /srv/argon/tenants/acme/run

An anomaly this guide found and did not resolve. The source designs both record, from 2026-09-12 on this same node, that the unprivileged-user-namespace route is CLOSED — unshare -U -r failing with write failed /proc/self/uid_map: Operation not permitted because kernel.apparmor_restrict_unprivileged_userns = 1 (docs/2026-09-12-tenant-runtime-sidecar-design.md:685-699, docs/2026-09-14-argondb-process-model-preliminary-design.md, §2.3). Re-run on 2026-09-20 it is now OPEN:

$ sysctl -n kernel.apparmor_restrict_unprivileged_userns
0

$ unshare -U -r /bin/sh -c 'id -u; cat /proc/self/uid_map'
0
         0       1000          1

$ grep "^$(id -un):" /etc/subuid /etc/subgid     # account name redacted below
/etc/subuid:<box-user>:100000:65536
/etc/subgid:<box-user>:100000:65536

The AppArmor restriction has been lifted on this node since the measurement the designs rest on. This does not change the recommendation — the source's argument against the namespace route was never only that it failed here, it was that lifting the restriction is itself a root step and that the policy differs across distributions, so useradd is the portable answer. But the two documents state the refusal as a current measurement and it is no longer true on this node. Escalated, not resolved: changing those documents is outside this lane. Re-measured 2026-09-22, the namespace route still yields no second uid, for a different reason: newuidmap/newgidmap are not installed, so the /etc/subuid range above cannot be mapped (HISTORY#erp-sidecar-per-company; docs/2026-09-04-erp-spec.md §15.7).

4.7 The restart policy, and how a DEGRADED row reads — FROM SOURCE

The constants are one definition and both consumers' profile tests assert against it (crates/argondb-ctl/src/restart_policy.rs:33-43):

pub(crate) const BASE_BACKOFF_SECS:    u64   = 2;
pub(crate) const MAX_BACKOFF_SECS:     u64   = 30;
pub(crate) const HEALTHY_RESET_SECS:   u64   = 60;
pub(crate) const BREAKER_MAX_RESTARTS: usize = 6;
pub(crate) const BREAKER_WINDOW_SECS:  u64   = 120;

So the sequence a crash-looping row produces is: exit, wait 2 s, respawn; exit, wait 4 s; 8 s; 16 s; 30 s; 30 s — and on the sixth restart inside 120 s, Decision::Trip { count }. A row that stays up longer than 60 s since its last respawn clears its streak, so a genuinely flaky-but-working child is not tripped by a one-off crash.

One row has its own, tighter pair, and the reason is a design rule rather than impatience (restart_policy.rs:45-68):

pub(crate) const MCP_BREAKER_MAX_RESTARTS: usize = 2;   // trips on the THIRD exit
pub(crate) const MCP_BREAKER_WINDOW_SECS:  u64   = 60;  // inside sixty seconds

argondb-mcp is the only supervised child whose characteristic failure is a deterministic CONFIGURATION refusal: it resolves and fail-closes every configured surface before it binds any of them, so a bad knob makes it exit in milliseconds, forever, and each retry re-reads the same environment and reaches the same conclusion. The default profile would take seven exits and ~90 s of accumulated backoff to reach the same verdict — 90 s of a box presenting as healthy with its agent surface missing.

How to read it in status. A stood-down or lazily-cold component is reported with an expected_down note — the configured steady state, not a fault (crates/argondb-ctl/src/serve.rs:5084-5096):

primary Postgres (internal; idle_stop ON)   [down]
  the primary compute is stood down after `[compute] idle_stop` — the next
  client connection on the public PG port wakes it

A tripped row is different: the supervisor has stopped respawning it and the box carries a DEGRADED state. A DEGRADED box boots, says so on every surface, and keeps serving what it can — the catalog-authz arm is the worked example of the shape, and it prints its cause, its since-stamp and its retry policy on one line (serve.rs:7801-7808), tripping its own breaker after a bounded number of recovery attempts rather than looping:

[argondb-box] catalog authz: BREAKER TRIPPED — 10 recovery attempts over 300s all failed;
no further automatic retry. The box stays DEGRADED …

When the application sidecar row lands, the one edit the design names is to the DEGRADED record's text, so that argondb status names WHICH TENANT's sidecar was given up on — the mechanism is already there.

4.8 The refusal that protects the lake, both polarities — FROM SOURCE

Removing the old bundled-catalog refusal made a configuration reachable that never had been: a lake-ON box with idle-stop ON. A narrower MUST replaces it (crates/argondb-ctl/src/serve.rs:5789-5800, crates/argondb-common/src/schema_refresh.rs:80-108).

The negative polarity — [lake] enabled = true, schema refresh on, and a window shorter than the tail's 30 s refresh cadence (TAIL_REFRESH_CADENCE_SECS, schema_refresh.rs:45). Refused at preflight with nothing spawned:

idle_stop is 20s, which is SHORTER than the tail's schema-refresh cadence (30s), so this
lake-ON box would silently stop enrolling newly published tables. The tail discovers a new
relation by reading the PRIMARY (`refresh_tables`, once per 30s cadence) over an INTERNAL
connection that deliberately cannot wake the primary and cannot hold it awake — so at a 20s
window the primary is always stood down when the tick fires. Tables already tracked keep
materializing (their rows come from the safekeeper and the pageserver, not the primary),
which is why this looks like a healthy box. Remedy, pick one: raise `[compute] idle_stop`
(or ARGONDB_PRIMARY_IDLE_SECS) to at least 30; or set `[lake] enabled = false` if this box
does not publish to the lake; or set ARGONDB_SCHEMA_REFRESH=off to declare the published set
STATIC, after which new tables are never discovered by your instruction rather than by
accident.

The positive polarities, all three of which boot:

# (a) EXACTLY the cadence — the boundary-equal must-NOT-apply control.
#     The rule is "strictly shorter"; 30 is not shorter than 30.
ARGONDB_PRIMARY_IDLE_SECS=30 argondb serve --data-dir /srv/argon

# (b) the same short window, with the published set DECLARED static.
ARGONDB_PRIMARY_IDLE_SECS=20 ARGONDB_SCHEMA_REFRESH=off argondb serve --data-dir /srv/argon

# (c) the same short window with the lake off entirely ([lake] enabled = false).
ARGONDB_PRIMARY_IDLE_SECS=20 argondb serve --data-dir /srv/argon

Both polarities of that boundary are unit-tested against the shared constant (schema_refresh.rs, boundary_is_strictly_shorter) and gated live in both directions by testing/box/idle-stop-wake.sh phase I (testing/box/idle-stop-wake.sh:48-54). The predicate lives in argondb-common and not in either consumer because the crate that owns the cadence (argondb-tail) and the crate that refuses the box (argondb-ctl) do not depend on each other, and the same constant is passed to the tail on --cadence-secs at its spawn — so the check can never be against a number the running tail does not sleep on.

5. Built today versus designed

What is still open, stated once. B′ is ruled and the per-tenant compute table is built, so the old blocking dependency is gone. Three things remain. The cross-company isolation gate has not passed under per_tenant on at main's tip (§2.7). The application's own port has no gate, so a request to it neither wakes nor holds its tenant (§2.5). And per_tenant stays the operator's switch, default off [D-2026-09-23-5122a6].

6. Gotchas

Every refusal, limit and residual gap a developer will hit, with its remedy.

On v1.5.7, a box that has run tenant create may refuse its next boot. tenant create cuts each tenant its own timeline in the box's storage tenant, whether or not a compute is ever bound to it, and on v1.5.7 the box's argondb-tail bound the FIRST timeline the pageserver listed rather than the box primary's. The listing order changes from boot to boot, so a restart can bind a tenant's timeline: the boot quarantines the estate's own tables (scan failed: nblocks), the safekeeper answers Timeline … was not found in global map, and the box says argondb-tail exited before it bound Flight … refusing the boot. A tenant removed with a hand DROP DATABASE … WITH (FORCE) leaves its timeline behind and triggers the same refusal. Fixed in v1.5.7.1: argondb serve names the primary's timeline to the tail (--primary-tenant/--primary-timeline, from the box's identity.json), the tail binds exactly that one, and every other timeline is named once in the tail log as a tenant's (named by its tenants/<key>/identity.json) or as an ORPHAN and is not streamed (crates/argondb-tail/src/service.rs, fn choose_primary_timeline). Remedy on an existing v1.5.7 box that holds a tenant: do not restart it on the v1.5.7 binaries. Stop it (SIGTERM to the box pid, wait for it to exit), install v1.5.7.1, and start it on the same data directory — nothing in the data directory needs changing, and a box v1.5.7 already refused boots the same way. To remove a tenant, use argondb tenant delete <key>, never a hand DROP DATABASE: it removes the estate rows (through destroy-tenant's sequence), the database, the tenant's timeline on the pageserver and on the safekeeper, and its tenants/<key>/ directory, each step probed first so a re-run removes nothing twice (crates/argondb-ctl/src/tenant_delete.rs). It refuses a tenant with its own compute bound (per_tenant on), with nothing removed, and it leaves the tenant's lake data for argondb lake drop-orphan.

A connection that sends nothing wakes nothing. FIRST_BYTES_TIMEOUT = 30 s and then a silent close. A TCP-connect health check, a browser pre-connect, or nc -z will never wake a tenant and will never keep one awake. Remedy: make the monitor send a real query. This is a feature — it is what stops argondb status from holding every box awake forever (idle_stop.rs:101, :517-522).

The internal port is public + 100 only while idle-stop is ON. With the knob off there is no internal port at all and Postgres owns the public one. Anything that hard-codes 5532 breaks when the knob is turned off, and anything that hard-codes "postgres owns 5432" breaks when it is turned on. Remedy: resolve the port through argondb status, whose walk carries both shapes (serve.rs:5084-5096), and remember that an overridden ARGONDB_PG_PORT shifts both by the same 100 (idle_stop.rs:84-89). A tenant compute's split is + 1, not + 100: slot i is tenant_port_base + 2i public and the next port internal (§2.8).

per_tenant = true without idle_stop does not boot. It is refused at config load, with both knobs named. Remedy: set idle_stop to a number of seconds, or leave per_tenant off.

Under per_tenant, a company with no known compute port has no pooled entry at all. Its pooled connection fails at PgBouncer, and the box log names the company and says it is "NOT routed to the primary". That is deliberate: the alternative is a company's traffic quietly landing on the primary. Remedy: wait one estate pass after tenant create, or read the log line. How long that is: the estate member runs a pass every --pass-interval-secs (default 300 s) AFTER the last one ends, so the real gap is 300 s plus the pass's own run time; it runs on the leader only, /v1/estate/pass runs one on request, tenant create and tenant destroy run their own, and the member's log says "next pass at " at start and "next timed pass at " after every pass (--pass-interval-secs 0 means on request only).

Under per_tenant, the one-time copy of pre-compute rows can be refused for good. If the publish-status writer creates argon.publish_status on a company's compute before the copy runs, the copy rolls back and every later pass refuses it by name (§2.7). The fix exists only on an unmerged branch. Remedy: none on main today. Do not turn per_tenant on for a box whose companies already hold rows on the primary until that fix lands, or check the log for the one-time copy NOT done after the first pass.

An idle box WARNs every ~120 s, and it is not a fault. The tail's maintenance sweep logs maintenance pin <table>: CONSTRAINED … on a landed view's table even with no writes: 21 times in the measured idle hour (§3.10). It is escalated, not fixed. Remedy: do not alert on that line alone for an idle box; alert on a view-maintainer WARN, which stayed at zero.

idle_stop = 300 (a TOML integer) is refused. The key is a string so that one parser serves both the file and the environment. Remedy: quotes.

idle_stop = "0" is refused, not treated as "always off" or "always on". Remedy: "off" to disable, or a positive window.

A lake-ON box refuses a window shorter than 30 s. Not a warning — the box does not boot. Remedy: one of the three the message names (raise the window to ≥ 30; turn the lake off; declare the published set static with ARGONDB_SCHEMA_REFRESH=off). §4.8 has both polarities.

The default is OFF, and that is deliberate. On a laptop or a single VM a stopped compute saves no money, so the knob buys nothing there and costs a wake latency; it is the multi-instance estate that wants it (idle_stop.rs:115-117).

A stand-down takes longer than its window. 34 s against a 20 s window, measured. The extra is the shutdown checkpoint plus the wait-for-port-free step. Consequence: do not write a test that asserts the pid is gone at window + epsilon; assert it is gone within window + STAND_DOWN_GRACE.

A stood-down compute is not a tripped one. stood_down is a separate vector; a stood-down child has charged no breaker and the "every child tripped ⇒ the box is dead" bail deliberately does not fire. Do not infer health from "asleep".

An open wake breaker is not retried automatically. Six failed wakes inside 120 s and every subsequent connection is refused by name until the box is restarted. Remedy: fix the cause, read logs/compute.log, restart the box. The refusal is deliberate — an expensive automatic recovery sits behind a circuit breaker, never a loop.

The 619 ms wake is conditional. Small empty database, fast NVMe box, pageserver warm, lake off. A large database with a cold pageserver is not this number, and the source states the conditions for exactly this reason.

The application half of the wake is not measured and is not bounded. It is the app's number. The obligation is disclosure, not a bound.

~630 resident computes is arithmetic, not a ceiling. The pressure knee was never approached. The first real wall is more likely the pageserver's per-timeline cost, the port band, or file descriptors — none of them measured.

The gate has one failure vocabulary. An application-port client whose wake fails today would receive a Postgres ErrorResponse over what it believes is HTTP. The second vocabulary is proposed and not built; do not front an application with this gate yet.

A busy application does not create an AwakeHold. Deliberately — a hold is a reason the DATABASE cannot be stopped. It shows up as the predicate's third term instead, and that term does not exist in shipped code.

A silent WebSocket does not hold a tenant awake — RULED, not unsolved. Block G4 [D-2026-09-20-78f71e]: bytes are the only idle signal, against this design's own recommendation. A browser holding a socket open with no bytes crossing for the idle window is standing its tenant down like anything else; the client reconnects on its own next bytes and the gate wakes the tenant, but server-push over that silent socket does not survive the stand-down. Remedy: a client that depends on server-push must already handle a reconnect — there is no configurable hold to raise instead.

A box without a supported pgbouncer boots on the direct port — it does not refuse. The pooler is OPTIONAL (B-MT5 amended [D-2026-09-24-f0e068]): a platform with no pgbouncer, one below 1.25.2, or one that cannot report its own version boots the box on the direct port only, under [pooler] enabled = "on", "auto" or no key alike, and the tenant is handed the direct connection string. It is never SILENT: the banner and argondb status print pooler: NOT RUNNING, naming the binary, what was found and the install line. Remedy: read that line (or the boot line's resolved path and probed version when it says pooler: ON) before assuming the pooled port is live.

pgbouncer does not ship inside the release artifact. The platform provides it [D-2026-09-23-520b66]; the release census fails a tarball carrying bin/pgbouncer, and scripts/release/build-artifact.sh refuses a --pgbouncer flag by name. The installer writes [pooler] enabled = "on" into the recipient's argon.toml as a record of intent, never over an existing value. Remedy: a provisioning script that wants the pooled port installs a PgBouncer 1.25.2 or newer on the host (the Ubuntu line is on the banner).

Rung 1 alone is not enough on the one-click cloud VM. Block G1 [D-2026-09-20-3425d9], against the recommendation: the cloud VM target needs rung 2, a container per tenant, on top of rung 1 — and rung 2 is DESIGNED, NOT BUILT. §4.6's recipe below is the on-prem shape only.

Rung 1 does not cover a kernel escape. A uid is not a VM. If tenants are ever mutually hostile rather than one customer's own tenants, rung 1 is not sufficient and the microVM question reopens on different evidence.

Rung 1's uid half crosses root once, per tenant. There is no path that does not. Budget one useradd per tenant at provisioning.

The user-namespace escape hatch is kernel-policy dependent, in both directions. It was measured CLOSED on this node on 2026-09-12 and measures OPEN on 2026-09-20 (§4.6). Do not build a provisioning path that assumes either state; probe it, and keep useradd as the portable answer.

ARGONDB_MCP_PORT=0 silently disables the analyst pool — not silently, in fact: status says so by name. But the coupling surprises people, because the pool's only consumer is the MCP child.

A lazy pool's computes read as DOWN in status. That is the configured steady state. The note in the walk says so.

A sidecar that clears ARGONDB_BOX_INSTANCE would be unreapable. The .env_clear() is right and the stamp must be re-added by hand. This is the one exception to the closed environment and it is worse to forget than the leak the stamp prevents.

compute_ctl's INTERNAL HTTP server has no authorization layer. It carries /extensions, /extension_server/{*filename}, /grants and /refresh_configuration, and it answered an unauthenticated POST /extensions with a 422 naming the field it wanted. The hole is CLOSED by control_http_bind(), which returns 127.0.0.1 and takes no argument so no caller can widen it. The gotcha that remains: the SQL port still honors ARGONDB_LISTEN_ADDRESS, so widening it widens the SQL surface and nothing else — verify with testing/box/bind-policy.sh Arm B′ rather than by reading.

7. Sub-diagrams

Three deeper pictures of parts the main diagram compresses. Each is generated by a script beside it; python3 <script>.py from docs/diagrams/argondb-and-the-erp/generate/ rewrites its SVG.

They share the main picture's palette and legend: solid outline = built and shipped, dashed = designed and measured but not built, OPEN = a decision still to be made.

The earlier process-model set is still current and covers overlapping ground from the design's own angle: docs/diagrams/process-model/01-tenant-unit.svg, 02-wake-path.svg, 03-isolation-ladder.svg, 04-sidecar-reach.svg, 05-cloud-topologies.svg.

8. References

Diagrams

• docs/diagrams/argondb-and-the-erp/02-the-per-tenant-unit-gate-wake-and-stand-down.svg — the diagram this guide explains
• docs/diagrams/argondb-and-the-erp/generate/d02_unit.py — its generator; every label in the picture is a claim explained above
• docs/diagrams/argondb-and-the-erp/generate/blocklib.py — the drawing library and the palette
• docs/diagrams/argondb-and-the-erp/02a-wake-and-stand-down-sequence.svg, 02b-supervisor-process-table.svg, 02c-isolation-ladder.svg — this guide's sub-diagrams, with generators generate/d02a_wake_standdown.py, generate/d02b_process_table.py, generate/d02c_isolation_ladder.py
• docs/diagrams/process-model/01-tenant-unit.svg, 02-wake-path.svg, 03-isolation-ladder.svg, 04-sidecar-reach.svg, 05-cloud-topologies.svg — the earlier set

Designs

• docs/2026-09-12-tenant-runtime-sidecar-design.md — the process table as it is (§2.1), the restart policy and the breaker (§2.2), the analyst pool as the precedent (§2.3), the v1.4 idle-stop and wake (§2.4), the per-tenant compute table (§3), the application sidecar row (§4), the gate and routing (§5), the firewall property and the two precisions (§6), idle semantics (§7), uid and cgroups (§8), the measurements (§9), the blocks (§10)
• docs/2026-09-14-argondb-process-model-preliminary-design.md — the unit as one picture and the latency budget (§1), the isolation ladder (§2), the sidecar sharing the unit (§3), same-process Rust as an optional tier (§4), AWS and Azure (§5), the pooler (§6a), the open decisions G1–G4 (§7)
• docs/2026-09-17-process-sidecar-design.md — what the tail does not own (§4.3), the tail's row as it becomes (§4.2), idle-stop and why the tail is resident (§6.4), the two names ruled by block S4 (§11)

Code

• crates/argondb-ctl/src/idle_stop.rs — the whole feature: constants (:72–:112), the parser and precedence (:142, :166), the activity query (:205), AwakeHold (:230), recent_activity (:288), decide (:307), the probe (:336), error_response and CANNOT_CONNECT_NOW (:425, :448), serve_gate (:462), handle_conn (:506)
• crates/argondb-ctl/src/serve.rs — ComputeSection and the idle_stop key (:98–:116), config resolution (:5752), the refresh-cadence MUST (:5789), the status walk's idle-stop and pool arms (:5084–:5096, :5145–:5175), the wake channel and governor (:8860, :8866, :8875), the idle tick and stand-down (:9007–:9144), the wake (:9152–:9270), wait_tcp_accept (:11253)
• crates/argondb-ctl/src/analyst_pool.rs — the knob and the band (:103–:129, :168–:264), validate_port_band (:269), resolve_mode (:303), want_ready (:522), recent_activity (:540), stop_slot (:1727)
• crates/argondb-ctl/src/restart_policy.rs — the shared constants (:33–:43), the mcp profile and its reasoning (:45–:68), Decision (:70–:78)
• crates/argondb-common/src/schema_refresh.rs — TAIL_REFRESH_CADENCE_SECS (:45), idle_stop_refresh_refusal (:80), BOX_PRIMARY_DATABASE (:127)
• crates/argondb-ctl/src/tenant_compute.rs — the per-tenant switch (resolve_per_tenant), where tenant ports start (TenantBand, DEFAULT_PORT_BASE), the next-free-pair allocator (next_free_pair), the recorded slot and ports (assign_slot, recorded_pair_conflict, migration_plan), the memory refusal (memory_refusal)
• crates/argondb-ctl/src/tenant_data.rs — the primary's refusal (refusal) and the one-time copy (copy_once); crates/argondb-ctl/src/tenant_roles.rs — the roles replay (ensure_on_computes)
• crates/argondb-ctl/src/pooler.rs — the enabled key (Enabled::parse), the probe (resolve_binary, MIN_VERSION), the per-tenant route (upstream_port, unrouted_line)
• crates/argondb-ctl/src/tenant_sidecar.rs — the per-company application process as built

Gates

• testing/box/idle-stop-wake.sh — phases A–I; the claim under test and why it is a live gate are in its header (:2–:54)
• testing/box/bind-policy.sh — Arm B′, the control-server bind, both polarities
• testing/box/tenant-runtime-memory-measure.sh — the measurement script behind §9 of the sidecar design; it measures and does not assert
• testing/box/pgbouncer-pooler-gate.sh — the pooled path end to end, per_tenant on and off
• testing/t3/view-idle-hour.sh — the idle-hour measurement (§3.10); runner testing/box/ms01-view-idle-hour-run.sh

Build and operation

• docs/build-and-test.md — the two cargo workspaces, the vendored Postgres build, the box/compose-2/live-demo rule, gate placement
• docs/fleet-lanes.md — fleet placement for builds, tests and boxes

Decisions cited

[D-2026-09-08-c60bce] primary idle-stop and wake · [D-2026-09-08-ca2e96] the bundled catalog's own Postgres · [D-2026-09-12-44e647] tenancy B′ put and not selected then · [D-2026-09-20-0d5ba3] B′ adopted · [D-2026-09-21-4107c5] B′ amended, a timeline per tenant · [D-2026-09-25-a2d0a3] a company's compute owns its writes · [D-2026-09-18-07c9d9] block S4, the two names · [D-2026-09-11-21a6d7] the tenant wall in the lake · [D-2026-09-15-917bb4] the catalog-authz default · [D-2026-09-18-515279] block S1, the process boundary · [D-2026-09-18-ff217a] block S2, the tail's budget and bound · [D-2026-09-20-3425d9] block G1, the isolation tier per target · [D-2026-09-20-4faa8c] block G2, a separate OS process · [D-2026-09-20-a7fb70] block G3, AWS first · [D-2026-09-20-78f71e] block G4, bytes are the only idle signal · [D-2026-09-20-d5c0f2] block P1, one pooler per box · [D-2026-09-20-5b6a85] block P2, pool size at one quarter · [D-2026-09-20-5f9e67] block P3, the pooled port is the default · [D-2026-09-23-520b66] B-MT5, pgbouncer required from the platform · [D-2026-09-23-5122a6] the amended B-MT1, per-tenant compute is never a system default · [D-2026-09-20-0d5ba3] B′ ratified · [D-2026-09-21-4107c5] B′ amended · [D-2026-09-24-f0e068] B-MT5 amended, the pooler is optional · [D-2026-09-24-093abd] the installer writes the key · [D-2026-09-25-f80e7e] the data half, a company created while serve runs.

Guide 3 — the lake and its process model

Audience: a senior developer who has to build against this, operate it, or debug it. Every claim carries its source by path:line. Every worked example is labelled VERIFIED (run, with its real output pasted) or FROM SOURCE (the exact signature, flag, schema or constant quoted from the code or the design). Nothing here is labelled neither.

Naming, once: our process is the tail (argondb-tail). The word "sidecar" in this system names the tenant's application process and nothing of ours (block S4 [D-2026-09-18-07c9d9], docs/2026-09-17-process-sidecar-design.md:1177).

1. The diagram

The picture is four frames, and the order matters. The top frame is the write path: an ordinary INSERT commits in Postgres, the WAL goes to the safekeepers, argondb-tail reads that WAL as a physical standby, decodes heap and catalog pages, CLOG and WAL records into typed row changes as Arrow, holds them in an in-memory ring, and on a cadence lands everything the ring holds as one atomic multi-table commit into Iceberg. Nothing in that path is a pipeline you configure: there is no connector, no topic, no sink, no second copy of the schema. The left-middle frame is what lands — a standard Iceberg table per published relation, optionally a __changes changelog beside it, and two branch-level system tables. The right-middle frame is who reads it, and its whole content is one rule: every read names one position L, and every scoped table is served at exactly that position or the read is refused. The bottom frame is the process model: five supervised rows on one box, each with a memory cap and a restart rule.

The two coordinates carry the design. H is the head — the newest commit fully decoded and available in memory. W is the watermark — the newest position durably committed to Iceberg, stamped on every snapshot the cut touches. W ≤ H always, and the interval (W, H] is the tail: decoded rows that exist only in the tail's memory, served over Arrow Flight, evictable the instant W advances past them (DESIGN.md:481-483). A read at L is Iceberg@W ∪ tail(W, L] merged by primary key, which is why the lake never drifts from the database: the lake copy and the un-landed remainder are two halves of one decode, not two systems that have to be reconciled. The picture's dashed outlines mean designed, measured, not built yet (docs/diagrams/argondb-and-the-erp/generate/blocklib.py:4); on this diagram the D18 Iceberg-v3 line and the D19 speed trigger are the two claims carrying that status, and §5 states each one's evidence.

2. What it is

2.1 The application and the primary Postgres

The application writes to real Postgres and nothing about that changes. G1 — Postgres fidelity (MUST): any SQL, type, extension behaviour or driver behaviour the Neon Postgres fork supports works unchanged, and ArgonDB's additions MUST NOT alter the transactional write path or the OLTP read path (DESIGN.md:391). There is no trigger, no logical-replication slot, no publication, and by default no REPLICA IDENTITY FULL. The decode is physical — heap pages, catalog pages, CLOG and WAL records — which is exactly why the customer's schema, WAL settings and writes stay untouched (DESIGN.md:152). The one exception is an operator's per-table opt-in: argondb lake publish <table> --old-row-from-wal sets that table's REPLICA IDENTITY FULL so the tail takes the old row from the WAL instead of the pageserver (crates/argondb-ctl/src/lake.rs:1131, opt_in_old_row_from_wal at :1194; HISTORY#tail-old-row-from-the-wal). It is off unless asked for, and §3.9 has its numbers. G1 is deliberately kept separate from G7 — Isolation (MUST): G1 forbids altering the write path's behaviour; G7 forbids loading it. A change can satisfy one and violate the other (DESIGN.md:391).

2.2 The safekeepers and the pageserver

Each tenant database is its own Postgres instance on its own timeline with its own WAL; the shared storage plane — one pageserver, one safekeeper set, one lake — serves every tenant timeline (B′ amended [D-2026-09-21-4107c5]). The WAL is durable in the safekeepers (Paxos, ×3 in the separated topology) and the pageserver serves versioned pages by GetPage@LSN (DESIGN.md:83-97). The tail is a consumer of both: each timeline's WAL comes from the safekeeper and its pages from the pageserver (crates/argondb-ctl/src/serve.rs:8025-8031), one stream per tenant timeline (crates/argondb-tail/src/streams.rs; §2.12). On the box the pageserver asks for a token on both of its ports: the tail and the management path carry the box-wide pageserverapi token, and each tenant compute's token names its one timeline and is refused on any other (storage/pageserver/src/auth.rs check_permission_for_timeline; HISTORY#pageserver-token-auth-on-the-box, HISTORY#pageserver-token-timeline-claim). It also holds one Postgres connection per stream — to the primary on a single-timeline box, and to the tenant's OWN compute for a tenant timeline's stream, since the primary's catalog does not hold a tenant's tables (crates/argondb-tail/src/streams.rs:120-128) — and that connection is the one thing about the tail that is easy to get wrong when reasoning about isolation. Its only use is schema introspection: at boot, once per cadence in refresh_tables, and at the write-floor flip. It uses an internal port and an internal application_name precisely so it never wakes the primary and never resets its idle clock, and a refresh that fails while the primary is stood down is a warn! and a retry next cadence — no schema can change while the compute that would change it is stopped (crates/argondb-ctl/src/serve.rs:8034-8050).

The tail is also a term of the safekeeper's WAL removal horizon, which is the mechanism most likely to surprise you. It reports its resume point as the flush_lsn/apply_lsn of the standby status update it already sends, under application_name=argondb-tail; the safekeeper takes the min over live argondb-tail sessions, persists it to argon_tail_hold.json beside safekeeper.control, and mins it into the removal horizon — capped at ARGONDB_TAIL_WAL_HOLD_CAP_BYTES (default 4 GiB) below remote_consistent_lsn (docs/reserved-identifiers.md:76). The persistence is the point: a box restart is exactly when no live session exists, and the file is loaded before the first removal pass. Over the cap, removal advances to rcl - cap and one ARGON tail WAL hold OVER CAP line per advanced floor segment names the forfeit the next restart will report as RESUME POINT UNSERVABLE at boot. It is not a lease: a removed tail pins up to the cap until the sidecar is deleted with the box stopped.

2.3 argondb-tail: decode

The decoder turns physical storage into logical, schema-typed row changes in Arrow — inserts, updates and deletes, with MVCC visibility resolved so uncommitted and aborted transactions never appear in any output, lake or tail (DESIGN.md:417, DESIGN.md:98). This is the single decode path: the ring, the changelog, the lake copy and every read all feed from it. There is no second derivation anywhere in the system, which is what makes "the lake cannot drift" a structural statement rather than a quality claim.

Row identity is the primary key when there is one (D7, DESIGN.md:148). A table with a PK gets full incremental insert/update/delete semantics via a RowDelta commit — data and equality-delete files in one snapshot. A table without one is a supported contract, not a gap: accuracy is identical, but the lake copy is maintained by full-table scan-replace in any cadence in which the table changed, so a missing PK costs update performance in both time and whole-system load. Scan-replace is also the path during catch-up and the automatic per-table fallback on any error (DESIGN.md:492). The operational guidance is one line: put a primary key on every published table.

2.4 The ring (W, H]

The ring holds decoded, committed rows that are not yet in Iceberg. Its budget is ARGONDB_TAIL_DRAIN_QUEUE_MB, 2048 MiB by default (crates/argondb-tail/src/lib.rs:133). Two things about it are worth knowing before you size anything.

First, what is counted is allocated bytes, not rows × logical width — RecordBatch::get_array_memory_size(), which walks every column's ArrayData and sums the allocated buffer capacity (crates/argondb-tail/src/lib.rs:206). Deliberately: the ring's problem is RSS, and a batch of three rows whose builders rounded twenty columns up to 64-byte buffers costs the process what it allocated. The ring holds one RecordBatch per (commit, table) pair, and at the ERP mix's ~36 rows per transaction spread over ten published tables, that is mostly single-digit-row batches — which is where the cost comes from. Measured 2026-09-19 (crates/argondb-tail/tests/ring_per_row_cost.rs): per-commit batches 364.2 B/row, the same rows coalesced into one batch 76.7 B/row, ratio 4.7×, 78.9 % of the ring's accounted bytes being batch overhead at ~1,150 B fixed per batch (docs/2026-09-17-process-sidecar-design.md:294-340). That measurement is what ARGONDB_TAIL_RING_COALESCE_ROWS (default 512, crates/argondb-tail/src/lib.rs:215-238) exists to remove.

Second, accounted bytes are not resident bytes. The relationship was measured directly: drain_queue_bytes_high_water 2,064.4 MiB against tail VmHWM 3,217.2 MiB — a ratio of 1.56× (docs/2026-09-17-process-sidecar-design.md:340). Size against the process bound, not against the budget. The same paragraph withdraws the 22 KiB per-ring-row figure the design used to carry: it was arithmetic over an RSS peak divided by a row count sampled by a different loop, the two numbers were never a paired reading, and nothing in the current design rests on it.

0 is refused on this knob, not read as "unbounded" — unbounded is the behaviour the budget exists to remove — and an unparseable value falls back to the default with a WARN saying so, never to off (crates/argondb-tail/src/lib.rs:165-187). It is read once at ingest-loop start, so the bound cannot move under a running loop.

2.5 The cut

A cut is one pass of the cadence loop. It (1) chooses a target commit-boundary LSN W', never splitting a Postgres transaction; (2) computes the net per-PK delta and prepares a RowDelta commit per PK-bearing table, or a scan-replace for the rest; (3) prepares the D9 changelog appends; (4) lands everything touched in (W, W'] as one atomic multi-table commit via the REST /transactions/commit endpoint, carrying Argon.watermark-lsn = W'; and (5) on success advances W, after which the tail evicts ≤ W' (DESIGN.md:489-495).

Atomic multi-table commit is a catalog requirement, not a preference. G3 cannot be honoured without it and no weaker mode is offered; the rejected alternative — per-table commits letting tables trickle independently, which is what every CDC pipeline ships — would make the consistency contract silently depend on which catalog the customer chose, and a guarantee that holds only on some catalogs is not a guarantee. A startup capability probe refuses clearly against catalogs lacking it (DESIGN.md:494; the probe is crates/argondb-committer/src/catalog_probe.rs).

One transaction per warehouse: under D30's tenant wall each non-primary database has its own Lakekeeper warehouse, so per-tenant atomicity is what the single commit buys, not cluster-wide atomicity across tenants (DESIGN.md:382).

A cut normally runs once per [lake] cadence_secs (default 30 s), but a caller can ask for one sooner. A publication landing's data pin must read at or after the landing's run start. When its first pin reads below that point, it sends the tail's cut_now Flight action. The tail does not have a second cut path: the request wakes the same cadence loop early, through the kick the drain budget uses, and the landing waits for the pass numbered after its request. The pin then re-reads the watermark and pins the coordinate that cut landed. Requests that arrive together share one pass. A request that arrives while a pass is already running is answered by that pass, with no extra wake-up, when its cut reaches the requested coordinate. A loop that nobody asks keeps cutting on its normal cadence. If the requested cut is refused, the refusal comes back to the pin with the database and warehouse named. The pin then waits for the next cadence, as it did before cut on demand existed, and says why on its publication data: line. A tail too old to have the action, or a pass that ended before cutting, is handled the same way. freshness serves the loop's pass, cut, on-demand-pass and request counts under cut_on_demand (crates/argondb-tail/src/cut_on_demand.rs; the pin's side is cover_data_floor in crates/argondb-ctl/src/landing_runner.rs).

2.6 The published table, and the __changes changelog

A published relation lands as a standard Iceberg table at branch / database / schema + the bare relation name as the leaf — br_main/postgres/public + orders (DESIGN.md:521). Each cut that touches it writes one snapshot carrying Argon.watermark-lsn, Argon.branch and Argon.txn-range (DESIGN.md:482).

The changelog is opt-in per table (D9, DESIGN.md:156), configured by --changelog / ARGONDB_CHANGELOG on the tail (crates/argondb-tail/src/main.rs:39-43) — a comma-separated list of flat lake keys, not relation names. It is an append-only table of rows: no row is ever modified or removed except by retention, and its schema is the source columns followed by four appended fields (crates/argondb-committer/src/iceberg_sink.rs:10204-10226):

It is emitted from the same single decode path and committed in the same watermark-stamped atomic transactions as the base table, which is why a changelog can never disagree with its table about what happened at a position. Retention floor is the pin window (48 h, configurable upward): time travel reconstructs an instant from a retained snapshot plus changelog replay, so a changelog that expires early strands valid pins.

It is not append-only at the snapshot level: D10 compaction may commit a Replace. Rows are preserved exactly and every ArgonDB consumer is unaffected, but a foreign engine doing standard Iceberg incremental reads must fall back to a full re-read on a replace snapshot (DESIGN.md:156).

2.7 dv_<branch>, and the single-writer rule

Three writers, three territories, and the rule is MUST. The committer alone writes br_<branch>; the lake writer alone writes dv_<branch> — continuous views, run_transform(landing: lake) and append_rows; and argon_transform alone writes the derived Postgres schema, enforced by grants (DESIGN.md:383, D29 ruling 2). The reason dv_ exists as a separate namespace is ownership: the tail's maintenance plane reads ownership at the namespace root, so a foreign writer nesting under our root is still Foreign, and a lake-owned table is never a thing the committer might also decide to write (DESIGN.md:521).

Inside dv_<branch> the rule goes one level finer: every dv_* table has exactly one writer, and the lake writer enforces it with one predicate, table_writer_refusal (crates/argondb-committer/src/lake_writer.rs:2310), asked on the staging path and on the transform landing (crates/argondb-transform/src/land.rs:642). A continuous view's table is written only by the view maintainer. A table that is not a view is written only by run_transform(landing: lake) or append_rows, and those two count as one owner, so an append into a transform output is allowed. Any other door is refused with the wire kind not-owner, and nothing is written:

lake-writer/not-owner: <target> is a continuous view — its one writer is the view maintainer, and the `<door>` door is not its writer — every dv_* table has exactly ONE writer; nothing was written. Land under another name, or remove <target> through the door that made it

(crates/argondb-committer/src/lake_writer.rs:1477-1480). A table that ensure_table has just created, with no snapshot and no view declaration yet, has no owner, and every door may take it. Otherwise the refusal would block the maintainer's own first refresh. The four polarities (a transform landed in dv_main is readable by DuckDB; not-owner in both directions; a br_ landing is refused; dv_* is enrolled in maintenance and kept out of the committer's territory) are proven on a box by testing/box/dv-lake-landing-polarities.sh.

Nothing may enter a published br_<branch> table from outside the WAL — ruled never, and enforce it: "No analytics-side write ever lands in br_*. ArgonDB detects a foreign snapshot on a published table, refuses to treat that table as unmaterialized, and reports it. Bulk-load needs are served by a fast path through Postgres, so the rows have real LSNs" (DESIGN.md:719-770, §12.22 [D-2026-09-20-4693b1]). A row with no LSN and no Postgres transaction is refused, never re-baselined into place; where a watermark is recoverable the table keeps flowing, and where none is, the table is held (quarantined) with a gap record.

2.8 _argon_gaps and _argon_value_notes

_argon_gaps sits at the branch level of the namespace and records every re-baseline the tail performs. Schema (crates/argondb-committer/src/iceberg_sink.rs:2833-2842, :2684-2700):

_argon_value_notes records D14's lake-unrepresentable values — NaN in bounded numeric(p≤38,s), ±infinity in date/timestamp[tz] — which map to NULL in the lake with a durable record and a disclosure on reads on both serving paths (DESIGN.md:181-189). Floats are not in that class: IEEE 754 on both sides, NaN/±Inf pass through untouched. Both tables ride along in every pin session, scoped or not, because a digest must see recorded gaps and a value note must reach every read it affects (crates/argondb-query/src/lib.rs:4157-4160).

2.9 The query plane

argondb-query implements the unified read as a DataFusion TableProvider: Iceberg@W ∪ tail(W, L] merged by primary key (DESIGN.md:126). G3 — LSN-pinned snapshot consistency (MUST): a unified read at L returns, for every table in the pin's scope, exactly the committed state Postgres would show at L. Effects of transactions with commit LSN ≤ L are fully visible; all others fully invisible, across the scoped tables. Consistency claims never extend beyond the scope that was pinned (DESIGN.md:417).

G3 is MUST while G4 (freshness — tail head within ~1 s of the compute's flush LSN; Iceberg watermark trailing by the commit cadence, default target ≤ 60 s) and G5 (read-your-writes) are SHOULD, and the asymmetry is the contract: correctness is absolute, timeliness is negotiable. A slow tail degrades G4 and G5 and that degradation is visible and bounded; a read that mixes LSNs violates G3, and there is no acceptable degraded form of it (DESIGN.md:419-427).

2.10 The pin

A pin is a signed value token, not a server-side resource handle: (branch, LSN, scope, minted_at, coordinate-provenance), where provenance records whether the LSN was server-chosen at the protected default or caller-named (D12, DESIGN.md:163). That last field is what decides whether a later redemption may refuse.

• Scope is the table set the consistency claim covers. Explicit pin() defaults to all — the atomic-cut guarantee; unpinned reads auto-scope to the tables the query touches. A pin MUST NOT be refused because of data it does not cover.
• Freshness defaults to latest_servable: the newest coordinate that will still serve for the token's whole validity window, not merely the newest that serves at the instant of minting (ratified [D-2026-08-05-ef1644]). How D12's "durable for every tail-tracked table" is proved since 2026-10-22: a landing cut writes ONE property-only commit on the branch-scoped _argon_estate table — the ESTATE RECORD, "every tail-tracked table not excluded here was unchanged through cut L" — instead of a cursor-pair stamp on every quiet table (argondb_committer::estate_record; the project owner's block of 2026-10-21 [D-2026-10-21-a896f2]). The pin's probe reads it once per pin — from the TAIL, through its estate_record Flight action (the process that wrote it, or read it at boot; the catalog only for a tail older than the action, since 2026-10-22) — and treats a covered table whose own stamp is at or below L as durable through L, served from that stamp's snapshot; a table's own landing snapshot carries Argon.prior-held-through-lsn so a token minted inside that window still redeems after the table lands. A table gets its own stamp only when it landed, when a cut-on-demand ask names it, or when its cursor is absent. The record is applied only to the tracked set it names (hash + count), and a reader on older bytes (v1.5.7.15) answers not-yet-durable for a covered table — stale, never wrong — so a tail and the readers of one estate upgrade together.
• Validity is the retention window (default 48 h), independent of any server process lifetime. A token minted at the default survives tail restarts and watermark advances for the whole window. Redemption rebuilds the session on demand; the session cache is a performance artifact, never correctness.
• There is no heartbeat and no live-pin registry. GC safety is arithmetic: snapshot expiry keeps the newest N snapshots plus the travel anchor (the newest snapshot at or older than the window), plus — on a changelog-less table — every watermark-stamped snapshot inside the window, because there the servable coordinates are those watermarks. No valid token can point below the window, so orphan cleanup needs no registry (DESIGN.md:174).

2.11 The MCP endpoint and the analytical Postgres

argondb-mcp on :8018 serves four mounts, one token each — argondb (read-only), argondb-dba, argondb-admin, argondb-write (DESIGN.md:195, D15/D16). The analytical Postgres reads the lake through lake_<branch>.* foreign tables, mounted by the argondb_lake FDW extension and always reading through the query plane (crates/argondb-lake-fdw/src/catalog.rs:1-18). The analyst computes are a different surface again: they read pages, not the lake, so the publish choice does not constrain them (DESIGN.md:393).

argondb_lake is one of a fixed, ratified set — the box's compute is a separate OS process whose only entry is the SQL wire protocol, which is false if a client can CREATE EXTENSION its own C code, so the shipped extension set is narrowed to a rule rather than left to what the build happens to install. The final set: core PostgreSQL (vendored), pgcrypto, btree_gin, btree_gist, our pure-SQL uuid-ossp shim, the neon preload with its required neon_rmgr (the preload loads it internally; a tier without it spawns no compute at all), pg_trgm (required because Lakekeeper's own catalog migration creates it and the box will not boot without it), and the lake reader, argondb_lake — ten of the eighteen extension-shaped members the build used to install are removed from the build entirely, not merely from the preload allowlist (docs/2026-09-12-tenant-runtime-sidecar-design.md:523-618, [D-2026-09-20-6b4639] amended [D-2026-09-21-c514c3]).

2.12 The five processes on the box

argondb serve supervises: argondb-tail (decoder, committer, the Flight ring — resident, durability per cut); argondb-maintenance (compaction, clustering, Puffin statistics, snapshot expiry, orphan cleanup — every 120 s, no off switch); Lakekeeper + catalog-pg + OpenFGA (the Iceberg REST catalog, with atomic multi-table commit probed at boot); argondb lake-store (the bundled S3 surface); and argondb-mcp + the analyst pool (crates/argondb-ctl/src/serve.rs:8225, :8332-8360, :8380). Scale-to-zero applies unevenly and for stated reasons: the tail no — a tail at zero is a stale lake; maintenance yes, on a schedule; the agent endpoint technically yes but in practice no (cold start ~35–49 s); the catalog no.

Past a single box, the tail is one process, multiplexed across every tenant's timeline, not one tail per tenant and not one tail per box: it runs on a shared estate instance, holds N streams — one per tenant timeline, each with its own member set, its own watermarks and its own replay budget — and never runs on a tenant's own instance, because a tail co-resident with one tenant could decode every other tenant's WAL. Under the deployment's three-member safekeeper quorum, "redundant" was ratified to mean one multiplexed tail, warm standby: a second tail on an estate instance in another availability zone, taking over on a fenced lease with no committed transaction lost, rather than one tail process per tenant (docs/2026-09-04-erp-spec.md:2447-2609, block E3″ [D-2026-09-21-25c7c6], reasoning trail E3 [D-2026-09-21-8bb144] and E3′ [D-2026-09-21-5a7dc6]). The multiplexing itself is built — one tail process following every tenant timeline as its own stream on one box; the warm standby and its fencing lease are not yet built.

3. How it works

3.1 The ring, the cut, and the coordinates over time

A cut fires at whichever comes first of four conditions, and knowing all four is the difference between predicting the snapshot rate and guessing at it.

1. The clock. --cadence-secs, declared by the box at 30 s rather than left to the tail's own default, because the number the box checked [compute] idle_stop against at preflight must be the number the running tail sleeps on, and "both defaults happen to be 30" is a coincidence, not an agreement (crates/argondb-ctl/src/serve.rs:8065-8075).
2. The ring's high-water, 50 % of the budget — 1 GiB at the default. The cadence loop is woken and cuts now; its sleep becomes a wait on (clock, high-water), whichever first, and nothing downstream changes (docs/2026-09-17-process-sidecar-design.md:373).
3. The writer's byte trigger, per table: DEFAULT_CADENCE_BYTES = 128 MiB of staged parquet (crates/argondb-committer/src/lake_writer.rs:148). The writer's own due check "can fire between two interval ticks".
4. The floor, which is the one that says no: ARGONDB_TAIL_MIN_CUT_SECS, default 5 — the writer's own DEFAULT_CADENCE_SECS (crates/argondb-committer/src/lake_writer.rs:148). It bounds the snapshot rate, and the reason is downstream: the maintenance worker's compaction thresholds are cadence multiples (crates/argondb-tail/src/maintenance.rs), so an unfloored byte-triggered cut every ~2 s at peak would be 15× the shipped manifest rate.

There is a fifth threshold that is backpressure, not cadence: the writer's staged_ceiling_bytes, 1 GiB per table (crates/argondb-committer/src/lake_writer.rs:153, :164-179). At the ceiling nothing is written and the refusal names the ceiling, the current staged bytes and the next cadence tick.

At 100 % of the ring budget the decode loop stops pulling from the WAL stream until the ring is back under high-water. Never a drop, never a batches.clear() outside the shipped trip/re-baseline arms; the safekeeper's TCP window backs up and the WAL stays under the hold (docs/2026-09-17-process-sidecar-design.md:375).

The subtle part is that backpressure is a named phase, not a stall. Three consumers ask "is this decoder stuck?" — ReplayVerdict::NotAdvancing, which feeds the lag governor's forfeit; heal_decision; and the liveness watchdog's StallVerdict::Wedged ("no WAL event and no WAL bytes for 60 s"). A decoder that is not reading because the ring is full is not stuck, and none of the three may see it as stuck. So the ingest loop prints ingest phase=backpressure on its liveness line beside await-event and the others, a backpressured window is excluded from the "did not advance" measurement, and the stall verdict's "no WAL bytes" clock does not run while backpressured — one predicate, in the place the three already share (docs/2026-09-17-process-sidecar-design.md:546-570). Deadlock is ruled out by construction and asserted: the cadence loop that cuts the ring runs on its own task and never waits on the decode loop, and the decode loop holds no lock the cut needs.

3.2 What is atomic, what is idempotent, what is refused

Atomic. One cut = one multi-table commit = one Iceberg snapshot per table touched, base table and changelog together, all carrying the same Argon.watermark-lsn. There is no partial step and tables never trickle (DESIGN.md:494).

Idempotent. Every stage is idempotent by LSN (G8, MUST, DESIGN.md:438). The durable source of truth is the Iceberg snapshot metadata itself — there is no sidecar progress store. On startup the committer reads each table's Argon.watermark-lsn and resumes from the min over them. Not the max, and the reason is specific: per-table watermarks are not guaranteed equal at a crash, because scan-replace fallbacks land per table, outside the atomic cut, so resuming from max would silently skip a laggard's unapplied window. Each table then applies only changes above its own watermark, so the overlap deduplicates by construction. Commits are conditional (catalog CAS), and a crashed half-commit is retried and deduplicated by watermark check (DESIGN.md:499).

Refused, and with which message. Only five things can refuse a read, and each names the blocking table and the remedy:

• A pin scope naming a table that does not exist: pin scope names unknown lake table(s) [...]; published tables: [...] (crates/argondb-query/src/lib.rs:4102, :4218).
• Time travel whose changelog replay window intersects a recorded gap: time travel of <t> to <L> refused: the changelog replay window (<W'> … <L>] intersects a recorded gap (<s> … <e>] (<reason>) — the range is unreconstructable; pin at or after <e>, or repin for a fresh view (crates/argondb-query/src/lib.rs:6014-6026). This one is permanent: the rows were never decoded and no amount of retention brings them back.
• Time travel on a changelog-less table to a coordinate that is not one of its retained watermarks: <t> has no changelog, so time travel can only land exactly on a retained snapshot watermark (nearest at or below <L> is <w>); enable the changelog for row-exact history (crates/argondb-query/src/lib.rs:5790-5799). Reachable only through freshness=head or an explicit at_lsn — the caller named the coordinate and owns its lifetime.
• Time travel to a coordinate below a history restart. A whole-table replace that cannot commit in place takes a staging swap: it drops the old table and renames a fresh one over it. That happens, for example, when a Postgres schema change cannot be expressed in Iceberg in place. The new table retains no snapshot below its own watermark and records that watermark as Argon.history-truncated-below-lsn, with the cause as Argon.history-truncated-reason (crates/argondb-committer/src/iceberg_sink.rs:87-119). A read below that coordinate is refused, and the refusal names the restart's first servable coordinate as the remedy: time travel of <t> to <L> refused: a whole-table materialize restarted this table's history at <R> (<reason>), so every coordinate from <w> down lost its base — this is NOT retention expiry and repinning at an older coordinate will never succeed; the new history's first servable coordinate is <R>: repin at <R> (truncated_history_note, crates/argondb-query/src/lib.rs:7512-7552; truncated_history_refusal, :5822-5833). This check runs ahead of the gap check, because when both apply, the gap's "pin at or after" coordinate can itself be below the restart, and only the restart names a coordinate that still has a base. A coordinate at or above <R> is served. The , so every coordinate from <w> down lost its base clause appears only when the destroyed watermark was recorded. The remedy points forward, to <R>, by ruling [D-2026-09-26-544ae0]; resolving the pin forward automatically was not taken, so you repin yourself.
• L outside the retention window.

A server-chosen coordinate has no refusal doors left. When latest_servable finds no coordinate every scoped table can serve, the pin is still granted and each table that cannot serve it is served at its own newest retained watermark at or below the grant, with the mint disclosing, per table, the older coordinate it reflects (DESIGN.md:167). The disclosure is what separates this from silent staleness, and the read path prints it verbatim (crates/argondb-query/src/lib.rs:1914-1926).

Disclosed. Reads without an explicit pin MUST disclose when the session serves meaningfully behind the live database, measured against the pageserver position (the tail's freshness action) rather than the decoded head alone, because catch-up parks the head at the watermark. Threshold 16 MiB, a constant since 2026-08-02 [D-2026-08-02-5a35c8]; an active catch-up always discloses (DESIGN.md:167).

3.3 The unified read, per table

![3b — the unified read: Iceberg@W ∪ the changes in (W, L]](diagrams/argondb-and-the-erp/03b-unified-read.svg)

The protocol is five steps (DESIGN.md:503-509):

1. Pick L — default the current H, or client-supplied for read-your-writes — and a scope.
2. Per scoped table, select the Iceberg snapshot with the greatest W ≤ L. Any retained snapshot qualifies, not only the current one; that is what makes time travel the same code path as a fresh read.
3. Fetch the changes in (W, L] — from the ring over Arrow Flight when L is above the table's current watermark, else from the __changes changelog, whose rows each carry _argon_lsn and _argon_op.
4. Merge: upserts and deletes applied over the Iceberg scan keyed by PK; append-only tables concatenate.
5. Refuse only in the five cases above.

The important structural fact is in step 3: the ring and the changelog are two sources for one merge, not two read paths. The merge does not know or care which one fed it, which is why time travel and read-your-writes have identical semantics and identical code.

There is an ordering subtlety worth naming because it looks like a bug when you hit it. A cadence commit can move a table's W past a pin taken moments earlier. For an implicit "latest" pin the answer is to re-pin at the fresh head and rebuild, not to fail; an explicit --at-lsn below W is served by time travel (crates/argondb-query/src/lib.rs:4278-4281). And a time-travel pin folds the tail's current pending D14 value notes into a historical read — so a note about a value that landed after the pinned coordinate can be disclosed on an answer that does not contain it. That is over-disclosure, named in the code as deliberate: the failure this mechanism prevents is an affected value served with no note, never a note with no value (crates/argondb-query/src/lib.rs:4270-4276).

3.4 The gap record and the rescan

Four doors lead to a re-baseline: a resume point unservable at boot (the hold went over cap); a startup laggard, per table, further behind head than ARGONDB_MAX_LAG_BYTES (256 MiB, the shipped default — docs/reserved-identifiers.md:77); an undecoded-window barrier episode; and a watchdog stall escalation on ReplayVerdict::NotAdvancing.

A fifth outcome is not a gap and is easy to confuse with one: a table whose decode fails is quarantined alone at its last-good watermark, alerted with the row's evidence and the remedy named, while the cadence, the catch-up rescan set and the read plane for every other table carry on unaffected (D14 containment, MUST, DESIGN.md:186). Nothing was lost — the table simply stopped advancing, and argondb lake list renders it as QUARANTINED @ <lsn> (frozen; cause=<c>: <reason>) (crates/argondb-ctl/src/lake.rs:316).

The ordering is the guarantee. The gap record MUST land before any of the rescan's table commits. A crash between the two can then only leave a conservative false positive — a gap recorded over a window that turns out to be fine, which replay-from-watermark heals — never an advanced watermark with an unrecorded gap (DESIGN.md:170). The reverse order produces a lake that is silently missing history and confident about it, and this ordering is what makes that state unreachable. It is listed as an invariant of every stage of D13's sharding design, not just of today's single pipeline (DESIGN.md:191).

Two consequences reach the reader, and neither is silence. A changes() digest whose window intersects a gap is still served, annotated: its counts are declared lower bounds over the intersecting range(s). A time-travel read inside a gap is refused permanently, naming the range and the reason (DESIGN.md:170).

Note also the defensive guard in the replay path. Structurally, a re-baseline resets snapshot history, so no base below a gap should survive; the gap check in travel_parts exists to convert any future violation of that invariant into a refusal instead of wrong rows (crates/argondb-query/src/lib.rs:6006-6011).

3.5 The memory budget, and where it turns into lag

The bound is a sum of seven named buffers plus allocator slack (docs/2026-09-17-process-sidecar-design.md:392-455):

   baseline              1,024 MiB   tokio, the Flight server, the schema, the pagestream connections
   ring budget           2,048 MiB   ARGONDB_TAIL_DRAIN_QUEUE_MB
   open-txn budget         512 MiB   ARGONDB_MAX_OPEN_TXN_BYTES
   scan-buffer budget      512 MiB   ARGONDB_MAX_SCAN_BUFFER_BYTES
   TOAST-index budget      256 MiB
   old-tuple cache         128 MiB   ARGONDB_TAIL_OLD_TUPLE_CACHE_MB
   writer staging        1,024 MiB   one table's ceiling
   ------------------------------------------------------------------
                         5,504 MiB
   + 25 % of the 4,480 MiB of BUDGETS (not the baseline)   1,120 MiB
   ------------------------------------------------------------------
   stated bound          6,624 MiB   ->  memory.max 6,144 MiB, memory.high 5,632 MiB

Ratified as block S2 [D-2026-09-18-ff217a]. Two things about that table are measured rather than assumed, and one thing in it did not ship.

The bound has a floor, and the box does not refuse under it. The tail's floor is 4,704 MiB. When a box's own memory budget would give the tail a smaller share than that, the share clamps up to the floor at boot — the boot line names the share, the budget and the floor — and the box refuses only when an operator sets ARGONDB_TAIL_MEMORY_MB explicitly below the floor. Read literally, the "refuses only below it" wording of block S2 item (3) would stop every budgeted box in the estate today from booting at all; this is that read corrected (block S5 [D-2026-09-19-1d3fca]).

The Flight scan budget sits outside this cap, not inside it. ARGONDB_TAIL_FLIGHT_SCAN_MB (default 8,192 MiB per process) bounds Arrow Flight scan reads on its own process-wide budget; the cgroup cap above does not cover them, and the boot line and the v1.5.4 limits line say so. Raising the 6,144 MiB cap to cover the read path (to roughly 16.5 GiB) was one of the three shapes considered and was rejected: it would undo the "≥ 24 GiB replaced by 6 GiB" limits-line claim, and this is the only one of the three that changes no ratified number (block S7 [D-2026-09-19-7a272e]).

Measured (the baseline). 2026-09-19, lane tail-rss-bound-gate on ms-01, the idle arm — box booted, seeded, published, drained, then one 120 s cadence at zero clerks: tail VmHWM 172.5 MiB, VmRSS 72.4 MiB, drain queue 0 bytes, backpressure 0 ms, ingest lag 0 bytes. That is 17 % of the 1,024 MiB constant, and the bound is not restated on it, for three reasons the design states explicitly: recomposing would move 6,624 → 5,772 MiB and with it a ratified cap, which one reading does not do; the reading is at ERP_SCALE=1 with ten published tables and the baseline's composition scales with the tenant, so one point is not the curve; and a baseline under the constant makes the bound conservative, which is the safe direction (docs/2026-09-17-process-sidecar-design.md:415-437).

Measured (the fail-before). The same 10-clerk level with the cache on and no ring budget: 19,007 MiB sampled, VmHWM 19,013 MiB (testing/box/tail-rss-bound.sh:12-17). That reading is what the budget removes, and the gate cites it character-for-character rather than spending 19 GiB of a shared node nightly to re-prove it.

Did not ship. The design asked for a tail memory: over the alarm line log line; L2 shipped the cap and the alarm without it, and the v1.5.4 notes §11 say so in those words (docs/2026-09-17-process-sidecar-design.md:451-455). Do not grep for that line.

Memory, lag and the cap are one system (docs/2026-09-17-process-sidecar-design.md:583-600):

   memory held by the tail      ≤  the bound          6,144 MiB cap
   WAL the tail has not read    ≤  the hold cap       4 GiB
   replay after any restart     ≤  the same cap       resume_replay_budget(cap, ARGONDB_MAX_LAG_BYTES) takes the LARGER

Backpressure moves bytes from the first line to the second. The second is where a tenant faster than its tail eventually forfeits. At 10 clerks, ~2.2 KiB of WAL per transaction, so 4 GiB ≈ 1.9 M transactions; at ~560 txn/s under backpressure, a tail whose offered rate exceeded its decode rate by 100 txn/s would take **~5 hours** to reach the cap.

At 50 clerks the picture is different and the design says so plainly: the tail decodes 659.6 txn/s against 2,900.4 tps offered and never drains (904 s, residual 497 MB). Doubling the cap doubles the runway — 13 minutes becomes 26 — and changes nothing, because the deficit is structural at that level. The cap is not what forfeits the tenant; the decode rate is (docs/2026-09-17-process-sidecar-design.md:1143-1165, block S6 [D-2026-09-19-9c0383]).

Under the multiplexed tail (§2.12), this budget is per stream, not shared across every tenant. ARGONDB_MAX_LAG_BYTES (256 MiB) is the constant each tenant timeline's stream owns on its own — ruled "per-stream, the current constant each. Simple, and the Nth company cannot be starved by the first; the estate's worst case is N × 256 MiB of headroom that must exist" (block B-MT2 [D-2026-09-23-d17cbd]) — never one 256 MiB budget divided across N companies. Because the resume budget actually applied is the larger of ARGONDB_MAX_LAG_BYTES and the WAL hold cap, and the hold is on by default at 4 GiB, the constant in force per stream is 4 GiB, not 256 MiB, so the estate's real worst case is N × 4 GiB of headroom — and under a three-member quorum that hold is held on all three members, tripling a lagging stream's storage cost relative to a single box (docs/2026-09-04-erp-spec.md:2654-2723).

3.6 Restart

There is no shutdown path, and that is deliberate. argondb-tail installs no signal handler, pinned as a source fact by service::tests::there_is_no_shutdown_path_so_every_exit_is_a_crash, which asserts the tail's crates/argondb-tail/src/main.rs never mentions SIGTERM, ctrl_c, signal_hook or SignalKind. The restart point is durable per cut, never at exit, because a clean LSN written at shutdown is produced only by the polite exit path — exactly the artifact that is missing when it is most needed. A drain-on-SIGTERM was considered and is not proposed: it would re-open the crash-between-write-and-exit class that test exists to keep unreachable (docs/2026-09-17-process-sidecar-design.md:687-700).

The sequence (docs/2026-09-17-process-sidecar-design.md:701-736):

1. The process dies where it stands. Ring, open-transaction buffers and cache are gone; the resume point is the last cut's.
2. The supervisor's exit arm asks RestartPolicy: Restart(backoff) 2 s → 30 s, or Trip at 6 in 120 s → DEGRADED naming the tail, alarm raised, and the tenant's database keeps serving. Under a cgroup cap the exit is the kernel's kill inside the cap and the DEGRADED text says so.
3. respawn_command rebuilds the same command and environment; the log is appended, so the panic that killed the previous incarnation is still there.
4. Boot: restart_start_lsn reads the per-table resume points, the replay budget is max(hold cap, ARGONDB_MAX_LAG_BYTES), prove_resume_servable probes the safekeeper as argondb-resume-probe before choosing, and the boot line names the arm. Servable → decode from the resume point, no re-baseline, no gap. Unservable → fresh baseline with a named gap.
5. Replay cost: at most the cap's worth of WAL at the decode rate — up to ~1.9 M transactions, ~45 min at ~700 txn/s. With the ring budget in place a crash mid-window re-decodes at most ~4 s of peak decode plus whatever lag the tail carried.
6. The cache is cold; the hit rate reads low for the first minutes. Nothing else differs.

A binary upgrade is the same sequence with step 1 chosen: stop the tail row (SIGTERM to its process group), swap the binary the box resolves (find_bin("argondb-tail", "ARGONDB_TAIL_BIN")), let the respawn arm bring the new one up from the same resume point. No state crosses between the two incarnations except through the lake and the hold, so there is nothing for a version to migrate.

argondb stop SIGTERMs every child's process group together with a 10 s budget for the set and a 10 s group grace, then SIGKILL — a 35 s teardown bound. The tail gets the same SIGTERM as everything else and dies. At the next serve the hold, loaded from argon_tail_hold.json before the safekeeper's first removal pass, has kept the WAL back to the resume point (docs/2026-09-17-process-sidecar-design.md:737-755).

Everything above is restart-from-watermark, and on the estate's multiplexed tail (§2.12) it is still the baseline: a failure is a restart, and the estate's lake freshness pauses for every stream until it comes back — nothing is lost, because nothing above depends on there being only one tenant. What "warm standby" adds under the ratified redundancy answer is a second tail process, in another availability zone, that takes over on a fenced lease before the restart-from-watermark sequence would otherwise have to run (block E3″ [D-2026-09-21-25c7c6]) — not yet built.

The lake writer's own restart seam is measured. The same rule, "the durable coordinate is the Iceberg commit", covers the dv_<branch> writer. A lake view refresh killed after its Iceberg commit returns, and before anything else is recorded, restarts cleanly. The restarted run reads the committed coordinate, so the view reads whole at the new position. A replay of the dead refresh's idempotency token answers already-landed and adds no snapshot and no rows. The next refresh continues from that coordinate, and the rows equal an oracle built from the deltas. The kill is a real process abort, not a simulation: a child process lands the refresh and aborts at the after-lake-commit hook in land_view_lake (crates/argondb-transform/src/land.rs:1759). The test is a_refresh_killed_after_the_lake_commit_is_wholly_present_and_not_reapplied (crates/argondb-transform/tests/landing_view.rs:980), and bash testing/t3/view-restart-await-matrix.sh --kill runs it and prints the kill's run token on its verdict line (measured on ms-01, 2026-09-24). Candor: the test's catalog is in memory and is reopened from the newest committed metadata file. That stands in for a REST catalog's durable pointer; it is not Lakekeeper itself.

3.7 The maintenance plane

ArgonDB owns it (D10, DESIGN.md:158). Compaction (RewriteFiles), argon:cluster-by clustering, exact-NDV Puffin statistics, snapshot expiry under explicit retention, orphan cleanup. It runs whole in every deployment on a fixed 120-second cadence — MAINTENANCE_SWEEP_SECS: u64 = 120 (crates/argondb-tail/src/maintenance.rs:837) — a code constant with no switch, no cadence setting and no off state. Tests reach it only through the test-override seam [D-2026-08-02-6e83a2]. Safety comes from the operations' own guards, never from optionality.

Why ArgonDB owns it rather than the customer: ArgonDB is the single writer and already knows table state from the WAL, so an external maintenance job would be a second writer racing the committer — and it would hand the customer back the operational pipeline this product exists to delete.

Clustering defaults to the table's single primary-key column when a published table has exactly one PK column and sets no explicit marker (ratified [D-2026-08-05-5caa39]): the shape read() serves is a primary-key equality, and the mechanism sat unused while a benchmark spent 15.90 s scanning for one row. Precedence: explicit argon:cluster-by=<col> wins; argon:cluster-by=none opts out; composite-PK and PK-less tables get no default — a leading-column default for composite PKs is deliberately not implemented, because it would optimize a prefix-equality shape the surface never issues. A single uuid key — every ERP table's UUID v7 — is measured, not assumed, to take this default: it lands as Iceberg's native uuid (arrow FixedSizeBinary(16)) and sorts through the same cluster_order path, proven by cargo test -p argondb-committer --lib uuid_cluster_tests and on a box by testing/box/uuid-v7-cluster-default.sh (declared sort order, monotone row groups; the composite-key twin takes no default).

Because clustering is a cost every table now pays, the snapshot-replace path's sort is part of the same ruling: it writes rows in key order through the parquet writer's slice feed rather than materializing a sorted copy, for any table that fits ARGONDB_MAX_SCAN_BUFFER_BYTES; above the budget it writes budget-sized windows each sorted by the cluster key and declares no sort order at all, because a global SortOrder over window-sorted files is false metadata a foreign reader may act on (amended wording ratified [D-2026-08-12-7e936d]).

A pass that overruns the cadence backs off rather than overlapping: on the live demo at a 120 s cadence, measured pass durations exceeded the cadence (crates/argondb-tail/src/maintenance.rs:5985-6080), and the pass deadline is 30 cadences = one hour (:5107-5113).

3.8 Branches

A branch maps to a distinct Iceberg namespace, materialized lazily: the lake does no work for a branch until something reads or publishes on it, so an unread branch costs the lake nothing while the Postgres side stays Neon copy-on-write (DESIGN.md:515). Once materialized, a branch's lake tables share nothing at the file level — the storage cost of a materialized branch is a full copy of what it materializes.

The rejected alternative is instructive: copy-on-write file sharing in the lake would cut that storage cost but couple every branch's maintenance, because compaction rewrites and orphan cleanup could no longer prove a file dead by looking at one table's retained snapshots. Every operation would need cross-branch reference counting, and a bug in that coordination corrupts a branch you were not touching (DESIGN.md:517).

The br_ prefix is conditional, and this is the single most common wrong assumption about lake addressing. Only main, temp and system take it — they are the three names DuckDB reserves as schema names. Every other branch is its namespace, bare. A br_$BRANCH composed in a shell is therefore right on three branch names and silently wrong on all the others; the dv_<branch> sibling is unconditional. Ask argondb lake namespace instead (§4, example 8).

3.9 What is measured, and the numbers

The 198 txn/s ceiling is one number with the 201 txn/s "before the cache" line above — the old-tuple cache exists precisely because oldpage_fetch is the whole cost, and serving old-row reads from memory is what turns 198-201 txn/s into 666.65. No tuning inside the tail moves the ceiling: every batching, allocation or lock change acts on the 3.5 % that is not oldpage_fetch, and sharding the fetch across more pagestream connections was already measured not to help (8 connections: 113-190 txn/s against 2,114 offered, no better than one) because the pageserver's per-tenant WAL-redo process is the serial resource. The fix therefore had to take the fetch out of decode's way or stop making it, not make it faster, and three levers have shipped on that basis. They are listed below in the order they landed. The question block that framed them is drafted in the design doc (docs/2026-09-17-process-sidecar-design.md:1391-1450, §11b).

1. Decode decoupled from the old-page read [D-2026-09-23-212703]. The tail encodes the rows that need no old page while the fetch is in flight. ARGONDB_TAIL_DECODE_DECOUPLED defaults on; 0, false, off or no run the two legs in series again (crates/argondb-tail/src/lib.rs:4739-4745). Measured, it hid only 2.0-2.7 % of decode time, because oldpage_fetch stayed 93.6-97.8 % of decode in every arm. The release claims no throughput from it (docs/releases/v1.5.5.md:548).
2. Lever A, the old row from the WAL [D-2026-09-23-dd9ca5]. This is a per-table opt-in and is off by default. argondb lake publish <table> --old-row-from-wal sets the table's REPLICA IDENTITY FULL, so Postgres writes the whole old row into the WAL and the tail never asks the pageserver for it. It is refused by name when wal_level is not logical, or when the table's identity was set to NOTHING or USING INDEX on purpose (crates/argondb-ctl/src/lake.rs:1131-1230). It costs WAL: ×1.047 at the ERP mix, 31.6-31.8 bytes per UPDATE, and a wide or TOASTed table pays its full row width on every UPDATE and DELETE (docs/releases/v1.5.5.md:530-547).
3. The zero-WAL lever, now the default. Under the DEFAULT replica identity, on a table with a primary key, Postgres logs an UPDATE's old key only when the key changed. An UPDATE record that carries no old key therefore proves the key did not change. The tail applies it as a same-key update and reads no old page at all. It changes no table's identity and adds no WAL byte. One predicate decides it, record_proves_same_key (crates/argondb-tail/src/lib.rs:3876), and both the fetch plan and the decode pass ask it, so a page is never skipped by one and wanted by the other. ARGONDB_TAIL_ZERO_WAL defaults on, is read once per process and logged at first use; off restores the old-page read (crates/argondb-tail/src/lib.rs:3717-3745). The ruling that made it the default was taken by block on 2026-09-24, option C, and its id lands at the next registry regeneration (docs/2026-09-17-process-sidecar-design.md:1574-1583).

The zero-WAL lever refuses by name and falls back to the old-page read, which is always correct, in each of these cases (ZeroWalRefusal, crates/argondb-tail/src/lib.rs:3756):

• the table has no primary key;
• its identity is NOTHING, FULL (lever A's case, where the record already carries the row) or USING INDEX;
• the switch is off;
• a catalog write that could change the identity (pg_class, pg_index, pg_constraint) has not yet been read by a schema refresh;
• the record is older than the catalog read the tail's cached identity came from.

Each refusal is counted by reason in argondb_tail_zero_wal_refused_rows (crates/argondb-tail/src/metrics.rs:1665). A DELETE never takes the lever: under DEFAULT identity its record already carries the key, so no page was read before this lever either. Lever A stays the answer for a table the zero-WAL lever cannot help, most importantly a table with no primary key, whose DELETEs otherwise need the page.

Measured on ms-01 at the P1 level, all arms on one binary. The zero-WAL decode ceiling was 10,009.7-10,388.9 txn/s against lever A's 9,695.8-10,067.7 txn/s in three clean pairs (zero-WAL ÷ A = 1.028-1.061), with 0 old-page reads and +0 B of WAL per UPDATE against lever A's +31.6. The same harness with neither lever read 748.0-754.8 txn/s (docs/2026-09-17-process-sidecar-design.md:1527-1560). Read these as decode ceilings, not end-to-end throughput. Each is transactions decoded divided by decode seconds. They say the old-page read is no longer what bounds decode on a primary-keyed, DEFAULT-identity table. They do not say a tenant sustains 10,000 txn/s through the whole path.

The diagram's bottom caption also carries "338 MiB drained in 578 s, zero gaps" and a "56-byte drain residual". That residual is argondb_tail_ingest_lag_bytes + argondb_tail_commit_lag_bytes (crates/argondb-tail/src/metrics.rs:1445, :1469), and it was not a floor: testing/box/tail-drain-residual-probe.sh at 10 clerks measured level c10 drain: drained after 31s (residual 0), trajectory shape=REACHED-ZERO, so the gate keeps residual 0 as the meaning of drained (docs/2026-09-17-process-sidecar-design.md:846-860). The 56 bytes were one standby-snapshot WAL record — a race, not a structural constant. Teaching the gate a floor would have been the re-baseline the design forbids, wearing a shared-predicate costume.

4. How to use it

Every example below is labelled VERIFIED or FROM SOURCE. VERIFIED means the command was run and its output pasted; the binary used for every VERIFIED example in this section is argondb built from this repo at /home/steveis/argon/target/release/argondb (2026-09-18 build), and the subcommands exercised are the pure ones — they open no Postgres connection, reach no catalog and need no running box, which is exactly what makes them runnable in a lane and callable from a gate. FROM SOURCE means the signature, flag, schema, constant or message is quoted from the code or the design by path:line and has not been executed here.

Example 1 — publish a table, and the marker that does it

FROM SOURCE. Publishing is one command and no connector configuration (G2, DESIGN.md:393):

$ argondb lake publish public.orders
published public.orders — the full table backfills at the tail's next cadence, then follows incrementally

The exact println! is crates/argondb-ctl/src/lake.rs:1190-1193. What the command actually does is set a Postgres table comment containing the marker argon:publish:

-- what `argondb lake publish` sets (crates/argondb-ctl/src/lake.rs:1176-1181):
COMMENT ON TABLE public.orders IS 'argon:publish';
-- an existing comment is preserved and the marker appended:
COMMENT ON TABLE public.orders IS '<existing comment> argon:publish';

Three consequences follow from the marker living in a comment rather than a config file, and all three matter in practice.

1. The published set lives in the branch's catalog and branches with the database. Branch the database, and the branch starts with the same published set (DESIGN.md:393).
2. Re-publishing is idempotent and says so: public.orders is already published (crates/argondb-ctl/src/lake.rs:1173).
3. Any client can set that comment without going through the CLI, so the marker can land on things that are not publishable. That path is accepted and announced, never silently dropped and never refused at COMMENT time, because refusing would need an event trigger and therefore superuser on the primary — which cannot hold on a bring-your-own-Postgres deployment. See §6.3.

If the table has columns no Iceberg type can carry yet, the marker is still set and durable; the table auto-backfills when its columns become lake-eligible, with no re-publish, and the command says that explicitly instead of implying an imminent backfill (crates/argondb-ctl/src/lake.rs:1194-1201).

Example 2 — see it appear, from both directions

FROM SOURCE. There are two listings and they answer different questions. From Postgres — which published relations have a lake table, and at what watermark (crates/argondb-ctl/src/lake.rs:1543-1545, three TAB-separated cells: name, state, key):

$ argondb lake list --tail http://localhost:8817
public.orders   lake @ 0/1BD2748    postgres__public__orders
public.line_items   not mirrored    postgres__public__line_items
public.legacy_audit QUARANTINED @ 0/1A9F210 (frozen; cause=decode: numeric out of range)    postgres__public__legacy_audit

The state cell's spellings come from crates/argondb-ctl/src/lake.rs:305-316. Pass --tail or the D9 changelog configuration is not checked, and the command says on stderr that it did not look — rather than leaving you to infer a clean bill of health from silence (argondb lake list --help, run below in example 3).

From the catalog — what the lake actually holds, including tables no Postgres relation is behind:

$ argondb lake catalog-tables --branch main
postgres__public__orders    br_main.postgres.public.orders    nested
_argon_gaps                 br_main._argon_gaps               branch-system

Three cells: the flat lake key, the catalog address, and the shape — nested | old-shape | branch-system | unclassified. The shape cell is not decoration: an old-shape table's flat key is its bare leaf, which cannot match what the Postgres side is told, so a consumer that ignores cell 3 reads a live lake table as missing. A catalog that will not resolve is an error, never an empty listing; an unborn branch namespace is an empty listing (argondb lake catalog-tables --help).

From SQL, and where a company's rows live. Every database carries its own argon.publish_status and argon.gaps, written by the box on a cadence (crates/argondb-ctl/src/publish_status.rs). A company that has its own compute (B′, [compute] per_tenant) has them written ON THAT COMPUTE, and only while it is awake. The pass never wakes a sleeping company to refresh a status row, so a stood-down company's rows are brought current by the first pass after its next wake, seconds after the wake, not before the wake's own first read. Read as_of to see how old they are. The CLI reads from the right place and says which:

$ argondb publish-status --database alphaco
source: alphaco's own compute on :27132 (the estate directory gives it one; its rows are written there while it is awake, and this read wakes it if it was stood down)
database: alphaco
relations: 2
  public.identity  state=mirrored  watermark=0/36BF298  as_of=2026-09-25 19:28:18.285429+00
  public.ledger  state=mirrored  watermark=0/36BF298  as_of=2026-09-25 19:28:18.285429+00

(Measured on ms-01 by testing/box/publish-status-awake-compute-gate.sh, 2026-09-25.) A company with no compute of its own reads source: the primary (<db> has no compute of its own), and a database the writer has not reached yet says so instead of erroring.

Example 3 — argondb lake address: the three spellings of one table

VERIFIED. A relation has three spellings and since the three-level identity landed they are no longer the same string. Ask, do not assemble:

$ argondb lake address postgres.public.orders
lake."br_main.postgres.public".orders

$ argondb lake address --field namespace postgres.public.orders
br_main.postgres.public
$ argondb lake address --field lake-table postgres.public.orders
postgres__public__orders
$ argondb lake address --field qualified postgres.public.orders
"public"."orders"
$ argondb lake address --field label postgres.public.orders
br_main.postgres.public.orders
$ argondb lake address --field leaf postgres.public.orders
orders

Which spelling you want depends on who is asking:

• address — the CATALOG identity a DuckDB, Spark or Trino user types. It is quoted, and it has to be: DuckDB reads a nested namespace only when the whole dotted path is one quoted identifier (crates/argondb-ctl/src/lake_address.rs:57-63).
• lake-table — the FLAT key, which is what has to survive where only two name parts exist: the tail's per-table maps and its Flight tables action, the FDW's foreign tables, a pin session's registered relations, and the --scope argument.
• label — the maintenance plane's log spelling, namespace levels joined by . with no catalog alias and no quotes. It exists because address's quoting makes it useless as a log matcher (crates/argondb-ctl/src/lake_address.rs:65-72).
• qualified — the TAIL's log spelling, the argument of <pg> left the published set and its siblings.

The changelog sibling is a flag, not a --field family, because the changelog is a different relation carrying every one of those coordinates:

$ argondb lake address --changes postgres.public.orders
lake."br_main.postgres.public".orders__changes
$ argondb lake address --changes --field lake-table postgres.public.orders
postgres__public__orders__changes

Note which halves move: the lake-side answers become the sibling's, namespace does not (a changelog always sits in its base table's namespace), and the Postgres-side target does not either (a changelog has no pg_class row), which keeps the lake list | … | lake address round trip exact (crates/argondb-ctl/src/lake_address.rs:120-130).

A second database, and the JSON form a script should consume:

$ argondb lake address shop.sales.orders
lake."br_main.shop.sales".orders
$ argondb lake address --field lake-table shop.sales.orders
shop__sales__orders

$ argondb lake address --format json postgres.public.orders
{
  "box_warehouse": "argon",
  "branch": "main",
  "catalog": "lake",
  "changelog": false,
  "primary": "postgres",
  "tables": [
    {
      "address": "lake.\"br_main.postgres.public\".orders",
      "bucket": "argon",
      "changelog": false,
      "database": "postgres",
      "label": "br_main.postgres.public.orders",
      "lake_table": "postgres__public__orders",
      "leaf": "orders",
      "namespace": "br_main.postgres.public",
      "qualified": "\"public\".\"orders\"",
      "schema": "public",
      "table": "orders",
      "target": "public.orders",
      "warehouse": "argon",
      "warehouse_refusal": null
    }
  ],
  "warehouse_per_database": true
}

"changelog": true|false is carried so a consumer never has to infer which relation it was handed.

Example 4 — unpublish, and the two tiers

VERIFIED (the help text; the effects are FROM SOURCE via that text):

$ argondb lake unpublish --help
Stop publishing a table. Lake files and history are KEPT (re-publish re-baselines). With --purge, also drops the Iceberg table and its changelog and deletes their files — irreversible

Usage: argondb lake unpublish [OPTIONS] <TABLE>

Arguments:
  <TABLE>

Options:
      --pg-url <PG_URL>      Omitted = this box's primary (see `lake publish --pg-url`)
      --purge                Also delete the lake table, changelog, and all their files
      --yes                  Skip the interactive confirmation (scripts)
      --branch <BRANCH>      Branch whose lake table to purge [default: main]
      --data-dir <DATA_DIR>  This box's data directory (argondb-lake-list-port-defect: only consulted with --purge, which is the arm that reaches the catalog). Same default as `argondb serve`/`status` [env: ARGONDB_DATA_DIR=] [default: ~/argon]
      --port <PORT>          This box's Postgres port, for the SAME port-precedence `argondb status` uses (`--port` / `ARGONDB_PG_PORT` / argon.toml `port`; presence-based, lake-visibility-pair finding b)
  -h, --help                 Print help

The two tiers mirror DROP TABLE vs DROP TABLE PURGE exactly (DESIGN.md:393): plain unpublish stops capture and keeps every lake file and all history — instant and reversible, and re-publishing re-baselines; --purge additionally drops the Iceberg table and deletes its files after an explicit confirmation showing what will be destroyed. DROP TABLE in Postgres behaves like the default unpublish: the lake data is kept.

Two tables can never be published: TEMP and UNLOGGED. They write no WAL, so there is nothing to capture (DESIGN.md:393).

Example 5 — read the lake copy from Postgres, through lake_<branch>.*

FROM SOURCE. The argondb_lake FDW extension mounts every table this database published as a foreign table in the lake_<branch> schema (crates/argondb-lake-fdw/src/catalog.rs:1-18, docs/reserved-identifiers.md:4118-4127):

CREATE EXTENSION argondb_lake;                    -- superuser: it creates a FOREIGN DATA WRAPPER

CREATE SERVER argondb_lake_srv
  FOREIGN DATA WRAPPER argondb_lake
  OPTIONS (tail 'http://localhost:8817', branch 'main');

SELECT argondb_lake_sync();                       -- one idempotent pass: CREATE FOREIGN TABLE IF NOT EXISTS
SELECT argondb_lake_start_sync();                 -- the cadence background worker, so later publishes mount
                                                  -- without a reconnect

SELECT count(*) FROM lake_main.postgres__public__orders;
SELECT argondb_lake_freshness();                  -- the behind-note, as queryable JSON

Three properties of this mount are worth internalising.

• It always reads through the query plane. A foreign table here is not a direct Iceberg scan; it is the unified read, so it sees Iceberg@W ∪ tail, not just the base table.

• It is scoped to its own database. Before 2026-09-11 the mount asked the tail for tracked tables, a cluster-wide set, so under the database-per-tenant topology tenant A's schema held tenant B's tables as ordinary foreign tables — readable by anyone with USAGE on the schema, with no policy, grant or wall consulted anywhere on the path. Both halves are now scoped by one predicate evaluated at the tail: the list is scoped, and every ticket carries the asking database, so a foreign table mounted some other way is refused by the read plane with a named message rather than served (crates/argondb-lake-fdw/src/catalog.rs:19-45).

• Tables another database GRANTED this one are mounted too. A member database mounts, read-only, a table another database granted it through hub.mount_grant, a fourth DatabaseScope::Granted arm; an ungranted table is refused by name at the read plane, and a revoke takes effect within one schema-refresh cycle (crates/argondb-lake-fdw/src/catalog.rs:229, crates/argondb-lake-fdw/src/flight_client.rs:245; HISTORY#fdw-cross-party-mount-built, HISTORY#fdw-cross-party-mount-box-proven).

• A mount outside this database's list is warned about, once per sync pass (sweep_mounts_outside_this_database, crates/argondb-lake-fdw/src/catalog.rs:514-579). Sync is CREATE FOREIGN TABLE IF NOT EXISTS, so nothing else removes such a mount. It may be a leftover from before 2026-09-11, a hand-made mount, a revoked grant, or a mount whose source has gone. Only one kind can be dropped: a stale cross-database mount, which sync made for another database's table that the read plane no longer tracks because the source was dropped, renamed or unpublished. It is dropped only when you opt in:

  SET argondb_lake.drop_stale_mounts = on;   -- superuser-only (Suset): it is destructive
  SELECT argondb_lake_sync();

  Each drop is itself a WARNING naming the foreign table, its lake table and its source database. With the setting off (the default), the warning names the setting instead, ending SET argondb_lake.drop_stale_mounts = on and re-run argondb_lake_sync() to have sync drop it (crates/argondb-lake-fdw/src/catalog.rs:360-365). Each drop runs in its own DO block, with no CASCADE. A mount that cannot be dropped, because a view depends on it or the role does not own it, is named and left in place rather than aborting the sync.

  A revoked grant's mount is never dropped. Its source is still alive, so it keeps its warning. Whether sync should drop revoked mounts is an open question (docs/2026-09-22-fdw-cross-party-mount-design.md §4.4). Mounts sync did not make, and every mount classified while the read plane cannot say what it serves, are also kept: the classification fails closed (classify_mount, crates/argondb-lake-fdw/src/catalog.rs:308). None of this is the wall. The read plane's scan refusal is the wall, and the drop is housekeeping.

Example 6 — read the same table from DuckDB, with nothing of ours running

FROM SOURCE (docs/lake-foreign-read.md:430-470, :468-530). This is G6 — open-format exit (MUST): the tables are standard, spec-compliant Iceberg, readable by any Iceberg-capable engine with no ArgonDB component running, and ArgonDB metadata rides in Argon.* snapshot properties that never break a foreign reader (DESIGN.md:434).

-- in DuckDB, after ATTACHing the Iceberg REST catalog as `lake`
SELECT count(*) FROM lake."br_main.postgres.public".orders;
SELECT count(*) FROM lake."br_main.shop.sales".orders;   -- database `shop`, schema `sales`

The quotes are not optional, and the failure does not look like a quoting mistake:

D SELECT count(*) FROM lake.br_main.postgres.public.orders;
Parser Error: NameListToString NOT IMPLEMENTED

Discovery is the part that is missing, and it is measured. DuckDB enumerates the catalog's top level only and never walks into a nested namespace, so a table you have not named is invisible to SHOW ALL TABLES, duckdb_schemas() and duckdb_tables():

D SHOW ALL TABLES;
┌──────────┬─────────┬───────┬──────────────┬──────────────┬───────────┐
│ database │ schema  │ name  │ column_names │ column_types │ temporary │
├──────────┴─────────┴───────┴──────────────┴──────────────┴───────────┤
│                            0 rows                                     │
└───────────────────────────────────────────────────────────────────────┘

Name the namespace once and it becomes visible for the rest of the session:

USE lake."br_main.postgres.public";
SHOW ALL TABLES;

To learn the namespaces without guessing, walk the REST catalog yourself — one level per call, which is exactly what DuckDB does not do for you (docs/lake-foreign-read.md:510-528):

curl -s "$CATALOG/v1/$WH/namespaces"                    | jq -r '.namespaces[][]'
curl -s "$CATALOG/v1/$WH/namespaces?parent=br_main"     | jq -r '.namespaces[][-1]'
curl -s "$CATALOG/v1/$WH/namespaces?parent=br_main%1Fpostgres" | jq -r '.namespaces[][-1]'
curl -s "$CATALOG/v1/$WH/namespaces/br_main%1Fpostgres%1Fpublic/tables" \
  | jq -r '.identifiers[].name'

%1F is the level separator the REST spec uses, and $WH is the warehouse's UUID, not its name — the catalog answers 400 WarehouseIdIsNotUUID for the name. Read it from GET /catalog/v1/config?warehouse=argon, field defaults.prefix.

What a foreign engine gets is the base table at its watermark. No ring, no pin, no merge. That is not staleness by accident — it is a different, simpler read than the unified one, and it is why example 5's FDW mount exists.

A dv_<branch> table reads the same way, but tells you which coordinate it is at differently. A continuous view's table, like a transform output, is plain Iceberg, and DuckDB reads it with nothing of ours running. Its snapshots carry Argon.view-current-to-lsn, the LSN the view is current to, and never Argon.watermark-lsn. The lake writer refuses a snapshot summary containing Argon.watermark-lsn, Argon.replay-from-lsn or Argon.changelog-through-lsn before it attempts the commit (FORBIDDEN_LAKE_SNAPSHOT_PROPS, crates/argondb-committer/src/iceberg_sink.rs:729, enforced at crates/argondb-committer/src/lake_writer.rs:1564). A view did not come from the WAL, so a reader that took its coordinate for a watermark would pin it wrong. A foreign consumer can therefore tell the two classes apart by which key the current snapshot carries: a published br_ table has Argon.watermark-lsn, and a view has Argon.view-current-to-lsn.

Proven with stock DuckDB v1.5.4 by testing/box/view-foreign-reader.sh (2026-09-24, ms-01, view-foreign-reader: PASS (ok=7 fail=0)). DuckDB read the view's 10 rows, equal to the maintainer's own count. The view coordinate on the current snapshot equaled argondb view list's, and the published control table showed the opposite pair. Two DuckDB v1.5.4 limits shape how you read those properties yourself. iceberg_snapshots() returns no summary column, so read the summary from the table's metadata.json (its location comes from iceberg_load_table_response). That function also refuses a multi-level namespace written as an identifier, so pass the address as a string literal.

Example 7 — a pinned read, and what the pin token carries

VERIFIED (the surface) / FROM SOURCE (the semantics). The bundled reader:

$ argondb-query --help
Usage: argondb-query [OPTIONS] <SQL>

Arguments:
  <SQL>  SQL to execute against the unified views

Options:
      --tail <TAIL>      The tail's Flight address. Read from `ARGONDB_TAIL_ADDR` when the flag is absent … [env: ARGONDB_TAIL_ADDR=] [default: http://localhost:8817]
      --branch <BRANCH>  [default: main]
      --at-lsn <AT_LSN>  Pin LSN (X/Y). Default: current tail head (freshest consistent view)
      --scope <SCOPE>    Pin scope (D12): comma-separated lake table names the pin covers. Default: all tables (the atomic cut)
      --csv              Print the result as CSV (header row first) …
  -h, --help             Print help
  -V, --version          Print version

So the three coordinates of a read are three flags:

# freshest consistent view, scope = all tables (the atomic cut)
argondb-query "SELECT count(*) FROM postgres__public__orders"

# the same question at a named position
argondb-query --at-lsn 0/1BD2748 "SELECT count(*) FROM postgres__public__orders"

# narrowed scope — the consistency claim now covers exactly these two tables
argondb-query --scope postgres__public__orders,postgres__public__line_items \
  "SELECT o.id, sum(l.qty) FROM postgres__public__orders o
     JOIN postgres__public__line_items l ON l.order_id = o.id GROUP BY 1"

# machine-readable, for a foreign consumer
argondb-query --csv "SELECT * FROM postgres__public__orders LIMIT 10"

Note that --scope takes flat lake keys — argondb lake address --field lake-table is the door that produces them. Narrowing the scope is also the remedy every scope-related refusal names, which is why it is worth knowing before you need it.

The token itself is (branch, LSN, scope, minted_at, coordinate-provenance), signed and stateless (DESIGN.md:163). The field that does the work is the last one: whether the LSN was server-chosen at the protected default or caller-named. A server-chosen coordinate has no refusal doors left — when no coordinate serves every scoped table, the pin is granted anyway and the tables that cannot serve it are served at their own newest retained watermark, with a per-table disclosure. A caller-named coordinate can refuse, because the caller named it and owns its lifetime.

The disclosure a non-atomic pin prints is verbatim from crates/argondb-query/src/lib.rs:1914-1926:

note: this pin is NOT one atomic cut — no coordinate serves every scoped table, so 2 scoped
table(s) are reflected as of an OLDER coordinate than 0/1BD2748: postgres__public__audit,
postgres__public__legacy. Rows committed to those tables after their own coordinate are NOT
visible here; every other scoped table is at 0/1BD2748. A table is listed here for one of two
reasons: it has no D9 changelog, so it can only be read exactly at a retained snapshot watermark
and it retains none at 0/1BD2748; or its lake watermark has run ahead of the live decoder, so its
changelog holds nothing between its own coordinate and this one and it is served base-only.
Enable the D9 changelog on the first kind for row-exact history at any coordinate, narrow the
scope to tables that share a coordinate, or repin — the second kind clears itself as soon as the
decoder catches up.

Example 8 — branches, and the br_ prefix that is not always there

VERIFIED. This is the one piece of addressing a shell script gets wrong, so there is a pure subcommand for it:

$ argondb lake namespace
br_main
$ argondb lake namespace postgres public
br_main.postgres.public
$ argondb lake namespace --branch feature-x
feature-x
$ argondb lake namespace --territory lake-writer
dv_main
$ argondb lake address --branch feature-x postgres.public.orders
lake."feature-x.postgres.public".orders

Read the third line again. feature-x has no br_ prefix. Only main, temp and system take it — the three names DuckDB reserves as schema names — so a br_$BRANCH composed in a shell is right on three branch names and silently wrong on all the others. The dv_<branch> lake-writer territory is unconditional, which a literal cannot carry either (argondb lake namespace --help).

Give levels below the branch to compose deeper; the result is the same string argondb lake address --field namespace answers for a relation in that schema. Use lake namespace when you have no relation to name and lake address when you do.

Operationally: branching the database branches the published set with it, the lake does no work for the branch until something reads or publishes on it, and once materialized the branch's lake tables share nothing at the file level (DESIGN.md:515).

Example 9 — time travel, and what a gap refusal prints

FROM SOURCE. Below the watermark, a read is served from the newest retained snapshot with W' ≤ L plus changelog replay over (W', L]:

argondb-query --at-lsn 0/1A00000 "SELECT count(*) FROM postgres__public__orders"

Four ways that can refuse, each with its exact text.

Inside a recorded gap — permanent, because the rows were never decoded (crates/argondb-query/src/lib.rs:6014-6026):

time travel of postgres__public__orders to 0/1A00000 refused: the changelog replay window
(0/19C0000 … 0/1A00000] intersects a recorded gap (0/19F1200 … 0/1A0C440] (watchdog
stall-escalation re-baseline) — the range is unreconstructable; pin at or after 0/1A0C440,
or repin for a fresh view

No changelog, and L is not one of the table's retained watermarks (crates/argondb-query/src/lib.rs:5790-5799):

postgres__public__orders has no changelog, so time travel can only land exactly on a retained
snapshot watermark (nearest at or below 0/1A00000 is 0/19C0000); enable the changelog for
row-exact history

This one is reachable only through freshness=head or an explicit at_lsn. The latest_servable default snaps down to a durably servable coordinate at mint precisely so the ordinary path never lands here.

Below a history restart, the one refusal whose remedy points forward (crates/argondb-query/src/lib.rs:5738-5777). The table, coordinates and cause below are the ones the unit test uses (the_refusal_below_a_history_restart_names_the_first_servable_coordinate_and_only_then):

time travel of postgres__public__vio_dim to 1/D54A4238 refused: a whole-table materialize
restarted this table's history at 1/D54B78F0 (schema-change-not-expressible-in-place) — this is
NOT retention expiry and repinning at an older coordinate will never succeed; the new history's
first servable coordinate is 1/D54B78F0: repin at 1/D54B78F0

Repinning older never helps here, and that is exactly the difference from retention expiry, which the message calls out. The cause is one of schema-change-not-expressible-in-place, format-version-downgrade or eligibility-probe-failed (SwapReason::why, crates/argondb-committer/src/iceberg_sink.rs:7111-7117). A caller that has to recover programmatically should parse the restart coordinate with the one parser, argondb_query::history_restart_of (crates/argondb-query/src/lib.rs:5810), rather than re-spelling the sentence.

A scope naming something that does not exist (crates/argondb-query/src/lib.rs:4102, :4218):

pin scope names unknown lake table(s) ["postgres__public__ordres"]; published tables:
["postgres__public__orders", "postgres__public__line_items"]

The misspelling is deliberate: the refusal prints the published set, which is what makes it self-correcting.

To record a gap that predates gap recording — known from logs — there is an ops escape hatch that only ever adds rows (VERIFIED, the help):

$ argondb lake gap-record --help
Record a changelog gap in `_argon_gaps` (D12). Ops escape hatch: the tail records its own gaps; this backfills gaps that predate gap recording (known from logs) so changes() stops being confidently wrong over those windows

Usage: argondb lake gap-record [OPTIONS] --start <START> --end <END> --reason <REASON>

Options:
      --start <START>        Gap range is (start, end]: LSNs as X/Y
      --end <END>
      --reason <REASON>      Why the range is unreconstructable (shown verbatim in digests and refusals)
      --table <TABLE>        Affected lake table; omit = every tracked table
      --forfeits <FORFEITS>  How many commits an undecoded-window barrier episode coalesced into this ONE record. Omit for every gap class that is not a barrier episode — omitted is NOT the same claim as 0
      --at <AT>              When the gap happened, RFC3339 (default: now)
      --branch <BRANCH>      [default: main]
      …

Note --reason is shown verbatim in digests and refusals: whatever you type is what a developer hitting that window six months from now will read.

Example 10 — the change log's columns, and a changes() digest

FROM SOURCE. Enable the changelog on the tail, by flat lake key (crates/argondb-tail/src/main.rs:39-43):

export ARGONDB_CHANGELOG=postgres__public__orders,postgres__public__line_items
# or, equivalently, on the tail's own argv:
argondb-tail --changelog postgres__public__orders --changelog postgres__public__line_items …

The tail accepts any string here without checking it, so an entry that names nothing produces no changelog. That is why argondb lake list --tail <addr> reads the configuration back and names every entry matching no published key — and why, without --tail, it tells you it did not look.

The table that results is queryable like any other, and its four appended columns are the schema in §2.6:

-- from a foreign engine
SELECT _argon_lsn, _argon_op, _argon_seq, _argon_ts, id, status
FROM lake."br_main.postgres.public".orders__changes
WHERE _argon_lsn > 0/19C0000
ORDER BY _argon_lsn, _argon_seq;

The replay the query plane itself issues is exactly that shape (crates/argondb-query/src/lib.rs:6037):

SELECT * FROM chg WHERE _argon_lsn > {w_base} AND _argon_lsn <= {lsn}

Through the agent endpoint, the digest tool takes a window in three interchangeable forms (crates/argondb-mcp/src/lib.rs:2658-2671):

{
  "name": "changes",
  "arguments": {
    "since": "now-1h",                       // or "0/1A2B3C", or "2026-07-08T14:00:00Z" (μs resolution)
    "until": null,                           // defaults to now (head)
    "tables": ["postgres__public__orders"],  // default: all tracked tables with history
    "budget_tokens": 4000                    // default
  }
}

If that window intersects a recorded gap, the digest is still served and its counts are annotated as lower bounds over the intersecting range(s) (DESIGN.md:170). A digest is never silently short.

Example 11 — the tail's environment knobs

FROM SOURCE. The tail's knobs, with what each actually changes:

A worked sizing pass, and the trade in each direction:

# A memory-constrained box: halve the ring. The cut rate RISES (high-water is
# hit sooner), backpressure milliseconds rise, and the tail holds less.
export ARGONDB_TAIL_DRAIN_QUEUE_MB=1024

# A box with headroom and a bursty writer: raise the ring and let the clock
# do more of the cutting. Snapshots get larger and less frequent; the tail
# holds more, and a crash re-decodes more.
export ARGONDB_TAIL_DRAIN_QUEUE_MB=4096

# Turn coalescing OFF to reproduce the pre-L6 ring cost (a measurement, not a
# configuration): expect ~4.7x the accounted bytes for the same rows.
export ARGONDB_TAIL_RING_COALESCE_ROWS=0

# Restore the pre-2026-09-17 WAL horizon exactly, hold and replay budget both.
# A restart under lag then re-baselines with a gap, as it did before the hold.
export ARGONDB_TAIL_WAL_HOLD_CAP_BYTES=0

The knob's effect is read back, not assumed: the box gate's N1 arm runs a second arm at ARGONDB_TAIL_DRAIN_QUEUE_MB=512 and requires drain_queue_budget_bytes read back as 512 MiB, a high-water at or under 512 MiB, a higher drain_queue_backpressure_millis_total than the default arm, and still drained (testing/box/tail-rss-bound.sh §7 table, N1).

What to read while it runs — the shipped metric names are drain_queue_bytes_high_water and drain_queue_backpressure_millis_total (crates/argondb-tail/src/metrics.rs). Two log lines carry them (crates/argondb-tail/src/service.rs:6753-6761, :6557-6562):

tail drain queue: bytes=1073741824 (1024 MiB) high_water=1024 MiB budget=2048 MiB rows=46355 backpressure_holds=3 backpressure_secs=1.8 backpressure_expiries=0

ingest phase=backpressure for 2s, ring_rows=46355, drain_queue=2048 MiB (budget 2048 MiB, 3 hold(s)), open_txns=1 holding 12 MiB (peak 40 MiB, budget 512 MiB), evicted_open_txns=0, toast 0 value(s) / 0 block(s) scanned + 0 hint block(s)

ring_bytes_peak and ring_backpressure_secs_total are retired spellings that no producer emits; they survive only in the design's record of what two lanes were asked for (docs/2026-09-17-process-sidecar-design.md:800-816).

Example 12 — the maintenance plane's cadence, and what it writes

FROM SOURCE. The cadence is a code constant:

// crates/argondb-tail/src/maintenance.rs:837
pub const MAINTENANCE_SWEEP_SECS: u64 = 120;

No switch, no setting, no off state; tests reach it only through the test-override seam (DESIGN.md:158). In the split production shape it is its own supervised process, spawned only when the tail is also spawned (crates/argondb-ctl/src/serve.rs:8332-8360) — a maintenance worker with no tail would be a second writer with no ingestion to coordinate with.

What a pass writes, per table: a compaction (RewriteFiles) snapshot when the file count crosses a cadence-multiple threshold; a clustered rewrite when argon:cluster-by (explicit, or the single-PK-column default) applies; a Puffin statistics file carrying exact NDV; a snapshot-expiry commit under explicit retention, which keeps the newest N snapshots plus the travel anchor plus, on a changelog-less table, every watermark-stamped snapshot inside the retention window; and an orphan-file deletion pass limited to data files unreferenced by any retained snapshot and older than a writer grace period (DESIGN.md:158, :174).

Statistics are re-scanned only when a table's rows moved. A statistics scan reads every data file of a table, so the pass first asks iceberg_sink::stats_freshness whether the statistics already registered still describe the table's current rows, and counts each table it does not re-scan as stats_skipped_fresh= on the maintenance pass cost: line. Fresh means registered at the current snapshot, or at an older snapshot when every snapshot after it carries Argon.rows-unchanged-since naming its own parent. That mark is what a quiet cut writes. A cut moves the lake watermark of every clean table through a property-only snapshot (the quiet advance), but it moves a table's row watermark only when it carries that table's rows, and statistics describe rows. Anything else ends the walk and owes a scan: a snapshot that carried rows, a compaction rewrite, a foreign writer's snapshot with no mark, or a statistics snapshot that expiry has already removed. Before 2026-10-05 the predicate asked only the first question. Every cut therefore staled every table, and a quiet 385-table box re-read all 20 MB of its tables on every pass (stats_scanned=385 stats_skipped_fresh=0). The profile reader calls statistics current by the same function, so it never calls a table stale that the sweep declined to re-scan.

Its log lines are keyed by the label spelling of example 3 (crates/argondb-ctl/src/lake_address.rs:72-76):

maintenance br_main.postgres.public.cas_zoo__changes: changelog compaction rewrote 7 data file(s) -> 1

A pass that overruns the cadence backs off rather than overlapping; on the live demo at the 120 s cadence, passes exceeding the cadence were measured and are kept as the back-off fixture (crates/argondb-tail/src/maintenance.rs:5985-6080). The pass deadline is 30 cadences — one hour (:5107-5113).

Clustering is a table comment, same idiom as the publish marker:

COMMENT ON TABLE public.orders IS 'argon:publish argon:cluster-by=customer_id';
COMMENT ON TABLE public.events IS 'argon:publish argon:cluster-by=none';  -- opt out of the PK default

Example 13 — force a clean re-derive of one table

FROM SOURCE (argondb lake --help, quoted in §6.7). The tail's own self-heal — snapshot-replace on a failed row-delta commit — only fires when a write fails, so a table left duplicated or drifted by an already-fixed cause never self-heals:

$ argondb lake rederive public.orders             # DRY RUN by default: prints the plan, touches nothing
$ argondb lake rederive public.orders --execute

It automates the proven path — unpublish (never --purge), then re-publish — which routes the table through the tail's own mid-run join, keeping the tail the single writer. It deliberately does not replace the table from the CLI process, which would be a second writer racing the cadence on the same table.

Example 14 — the whole path, end to end

FROM SOURCE. One change, and where the time goes. The architecture site traces one price change across six hops, from one company's database to another's (deploy/publish/architecture-site/content.md:563-565). Within a single tenant the path is shorter, and it is worth writing out because it is the shape every question about freshness reduces to:

1. an ordinary COMMIT in Postgres            <- the application's latency; unchanged (G1)
2. the WAL reaches the safekeepers           <- durability
3. the tail decodes it into the ring         <- H advances; typically well under a second behind (G4)
4. a cut lands it in Iceberg                 <- W advances; ≤ 30 s by clock, sooner on a byte trigger
5. a reader pins L and reads                 <- Iceberg@W ∪ tail(W, L], merged by PK

Between steps 3 and 4 the row is already readable at full consistency — that is what the ring is for. Step 4 is about durability and file health, not about visibility. A reader that pins at head sees a row the instant step 3 completes; a foreign engine reading the Iceberg table directly sees it after step 4.

5. Built today vs designed

Solid outlines in the diagrams mean built and shipped; dashed means designed, measured, not built yet (docs/diagrams/argondb-and-the-erp/generate/blocklib.py:4). This table is the same statement per feature, with the evidence.

6. Gotchas

6.1 br_ is conditional

Only main, temp and system take the prefix. br_$BRANCH composed in a shell is right three times and wrong every other time; dv_<branch> is unconditional. Remedy: argondb lake namespace [--branch B] [--territory lake-writer] (example 8).

6.2 A nested namespace needs one quoted identifier

lake.br_main.postgres.public.orders is four SQL name parts and DuckDB stops at the parser: Parser Error: NameListToString NOT IMPLEMENTED. It reads like a broken query, not a quoting mistake. Remedy: argondb lake address <target>, which quotes by construction (crates/argondb-ctl/src/lake_address.rs:57-63).

6.3 SHOW ALL TABLES does not list your tables

Measured, not theoretical: DuckDB enumerates only the catalog's top level. Reading a table you can name works perfectly; discovery is the only thing missing. Remedy: USE lake."br_main.postgres.public"; once per namespace per session, or walk the REST catalog, or argondb lake catalog-tables (docs/lake-foreign-read.md:468-530).

6.4 The publish marker is a COMMENT, so it can land on non-tables

A view, a materialized view, a foreign table, a partitioned parent (which holds no storage — publish its leaf partitions), a sequence and an index are none of them capturable, and any client can COMMENT the marker onto them without going through the CLI. Refusing the DDL would need a Postgres event trigger and therefore superuser on the primary, which cannot hold on a bring-your-own-Postgres deployment. So the marker is accepted and announced: one rate-limited WARN per (database, relation) naming the relation and its kind, and argondb lake list shows it as not-publishable (<kind>) (DESIGN.md:393). The WARN names the database unconditionally (postgres.pmv.v) because publish discovery reads the whole cluster and a relation name alone no longer identifies a relation.

Also: TEMP and UNLOGGED tables cannot be published at all — no WAL, nothing to capture. And new tables are not auto-enrolled: publishing is per table, by design.

6.5 No primary key means scan-replace, every cadence the table changed

Accuracy is identical either way — ArgonDB guarantees 100 % agreement between the database and the lake with or without a PK. What a missing PK costs is update performance, in time and in whole-system load, because the copy is maintained by full-table scan-replace (DESIGN.md:148). Remedy: put a primary key on every published table. Corollary for maintenance: composite-PK and PK-less tables get no clustering default, so the read-point-lookup win does not apply to them (DESIGN.md:158).

6.6 0 is refused on the ring budget, and on the open-txn budget

Neither reads as "unbounded" — unbounded is the behaviour the budget exists to remove. An operator who wants effectively-no-budget says so with a large number. Unparseable values fall back to the default and log why (crates/argondb-tail/src/lib.rs:165-187). By contrast, ARGONDB_TAIL_WAL_HOLD_CAP_BYTES=0 does mean off — it is the documented "restore the pre-hold horizon exactly" escape — so the same digit means opposite things on two knobs in the same process. Read the table in example 11 before setting either.

6.7 The tail's self-heal only fires when a write fails

A table left duplicated or drifted by an already-fixed cause never self-heals, because the snapshot-replace heal is triggered by a failed row-delta commit and nothing else. Remedy: argondb lake rederive <table> --execute, which automates unpublish-then-republish (never --purge) so the table goes back through the tail's own mid-run join rather than being rewritten by a second process (argondb lake --help, the rederive entry).

6.8 A time-travel refusal inside a gap is permanent

Retention will not fix it and neither will waiting: the rows were never decoded. The refusal names the recorded range and the reason, and the remedy it prints — pin at or after <end_lsn>, or repin for a fresh view — is the whole of what is available (crates/argondb-query/src/lib.rs:6014-6026). A changes() digest over the same window is different: it is served, annotated as a lower bound.

A refusal below a history restart looks similar and is not. The rows there were decoded, but a whole-table replace that could not commit in place dropped the base below its new watermark. That refusal is checked first, says this is NOT retention expiry, and names the one coordinate to repin at, the restart's first servable coordinate (§3.2, example 9). If you script around these refusals, go forward to that coordinate. Repinning older, which is the usual reflex, never succeeds here.

6.9 Read-your-writes is lost at the default on changelog-less tables

latest_servable snaps down to the newest watermark every changelog-less scoped table retains, so a pin on such a table is as fresh as the last lake commit, not the last write. That cost is accepted and named in the ruling (DESIGN.md:167). Remedies, in order of preference: enable the D9 changelog on the table; narrow the scope; or use freshness=head, which opts into head-or-refuse and discloses the transience at mint.

6.10 A pin token can outlive what it names — in exactly one direction

A token minted at the latest_servable default survives tail restarts and watermark advances for its whole window. A token minted at a caller-named coordinate — freshness=head above a changelog-less table's W, or an explicit at_lsn in that transient range — survives the restart but not the watermark advance past it. The mint discloses that when it happens, and the provenance field in the token is what later lets redemption tell the two apart (DESIGN.md:163-166).

6.11 Over-disclosure on a time-travel pin

An at_lsn pin folds the tail's current pending D14 value notes into a historical read, so a note about a value that landed after the pinned coordinate can be disclosed on an answer that does not contain it. Named in the code as deliberate and as the safe direction: the failure being prevented is an affected value served with no note (crates/argondb-query/src/lib.rs:4270-4276).

6.12 Do not read total-records to reconcile a table

It fails in the direction that looks like a broken lake. Count rows (docs/lake-foreign-read.md:556).

6.13 Backpressure is not a stall, and the metrics say so

If you are writing an alert, do not treat a non-advancing read position as stuck without excluding backpressured windows — the product already does this in one shared predicate, and a second copy in your monitoring will fire on healthy backpressure (docs/2026-09-17-process-sidecar-design.md:546-570). The lines to read are ingest phase=... and tail drain queue: ... (example 11).

6.14 Two metric names in the design do not exist

ring_bytes_peak and ring_backpressure_secs_total are retired spellings that no producer emits. The shipped names are drain_queue_bytes_high_water and drain_queue_backpressure_millis_total. The design keeps the old spellings only in its record of what two lanes were asked for (docs/2026-09-17-process-sidecar-design.md:800-816).

6.15 There is no clean shutdown, and that is the design

argondb-tail installs no signal handler; a unit test asserts crates/argondb-tail/src/main.rs never mentions SIGTERM, ctrl_c, signal_hook or SignalKind. Every exit is a crash and the restart point is durable per cut. Do not add a drain-on-SIGTERM: it re-opens the crash-between-write-and-exit class that test exists to keep unreachable (docs/2026-09-17-process-sidecar-design.md:687-700).

6.16 The decode rate, not the cap, is what forfeits a fast tenant

At 50 clerks the tail decodes 659.6 txn/s against 2,900.4 tps offered and never drains. Doubling the WAL hold cap doubles the runway — 13 minutes becomes 26 — and changes nothing, because the deficit is structural at that level. Size the decode, not the buffer (docs/2026-09-17-process-sidecar-design.md:1143-1165).

Profiled to its stage, the pre-cache ceiling is one cost: the pageserver round-trip for old tuples (oldpage_fetch) is 96.5 % of charged decode time and bounds decode at 198.0 txn/s by itself, no matter what else in the tail changes. On a table with a primary key and the DEFAULT replica identity, the zero-WAL lever (on by default) now skips that read for every UPDATE that keeps its key. A table without a primary key, or with identity NOTHING or USING INDEX, still takes the page. For those tables the remedy is lever A, argondb lake publish <table> --old-row-from-wal, which you pay for in WAL (§3.9). If argondb_tail_zero_wal_refused_rows climbs, its reason label names which case the refused rows are in; the counter is per reason, not per table.

6.17 Snapshot expiry keeps more than you might expect, and for a reason

The keep-list is: the newest N snapshots, plus the travel anchor (the newest snapshot at or older than the retention window, or the oldest snapshot when the table is younger than the window), plus, on a table with no D9 changelog, every watermark-stamped snapshot inside the window — because there the servable coordinates are those watermarks and replay cannot cover any distance from the anchor (DESIGN.md:174). If you are estimating metadata growth on changelog-less tables, that third term is the one that dominates.

6.18 A quarantined table is not a gap

A table whose decode fails is quarantined alone at its last-good watermark: the cadence, every other table and the read plane are unaffected, and nothing was lost. argondb lake list renders it as QUARANTINED @ <lsn> (frozen; cause=<c>: <reason>), and a read of it is served base-only at that watermark and says so (crates/argondb-ctl/src/lake.rs:305-357). Before that state had a voice on the list surface, the watermark simply stopped moving and the customer had to read the tail log to learn why.

6.19 Two things about the changelog that surprise foreign readers

It is append-only at the row level, not the snapshot level: D10 compaction may commit a Replace. Rows are preserved exactly and every ArgonDB consumer is unaffected, but a foreign engine doing standard Iceberg incremental reads must fall back to a full re-read on a replace snapshot (DESIGN.md:156). And a delete is a tombstone carrying the key, not the old image — old-image capture is not built, and is accepted as not built.

6.20 A too-small memory budget clamps up at boot, it does not refuse

The tail's floor is 4,704 MiB. A box whose own memory budget would give the tail less than that does not fail to boot: the share clamps up to the floor, and the boot line names the share, the budget and the floor. The box refuses only when an operator sets ARGONDB_TAIL_MEMORY_MB explicitly below the floor. Read the cgroup-cap rule (§3.5) as "refuses below the floor" and you will expect every small deployment to fail to boot; it does not (block S5 [D-2026-09-19-1d3fca]).

7. Sub-diagrams

Four deeper block diagrams of parts the main picture compresses. Each is generated by a Python script beside it, using the same drawing library, palette and legend as the main set; re-render any of them with python3 <script> from docs/diagrams/argondb-and-the-erp/generate/.

8. References

Design and doctrine

• DESIGN.md — §3 architecture (:83-131), §4 decisions D7 (:148), D9 (:156), D10 (:158), D12 (:163-180), D13 (:186-194), D14 (:181-189), D15/D16 (:195-299), D17 (:318), D18 (:320), D19 (:333), D29 (:383), D30 (:382); §5 guarantees G1–G8 (:391-438); §7 the watermark and tail protocol (:477-521); §8 distributions (:525-537); §10 V1 (:557-576).
• docs/2026-09-17-process-sidecar-design.md — §5.2 the ring's budget and its three thresholds (:294-390), §5.3 the bound and the measured baseline (:392-455), §5.4 backpressure as a named phase (:546-580), §5.5 memory, lag and the cap as one system (:583-604), §6 restart and handoff (:685-768), §7 the box gate (:770-870), §11 blocks S1–S7 (:1061-1300).
• docs/2026-09-17-tail-resume-one-bound.md — the hold and the replay budget as one bound (:23, :114-166, :210).
• docs/2026-09-17-tail-wal-retention-hold.md — the safekeeper half (:83).
• docs/2026-09-17-tail-old-tuple-cache.md — the cache reading (:178-202).
• docs/lake-foreign-read.md — the address (:430-467), discovery (:468-555), reconciliation (:556-641), row security does not travel (:642-705).
• docs/reserved-identifiers.md — the __changes suffix and its three spellings (:30), the WAL hold's full register (:76), its gate (:77), the drain-queue knob (:7265), the FDW's SQL surface (:4118-4127).
• deploy/publish/architecture-site/content.md — "one change, six hops" (:540-640).

Code

• crates/argondb-tail/src/lib.rs — the ring: DEFAULT_TAIL_DRAIN_QUEUE_BYTES (:133), drain_queue_over_budget (:144), drain_queue_budget_bytes (:165), ring_batch_bytes (:206), RING_LSN_COLUMN (:212), default_coalesce_rows (:224).
• crates/argondb-tail/src/service.rs — the cadence loop, the liveness lines (:6557-6562, :6753-6761), the no-shutdown-path test.
• crates/argondb-tail/src/maintenance.rs — MAINTENANCE_SWEEP_SECS (:837), the overrun/back-off fixtures (:5985-6080), the pass deadline (:5105-5113).
• crates/argondb-committer/src/lake_writer.rs — the lake-owned writer's cadence (:146), byte trigger (:148), staged ceiling (:153), CadenceConfig (:157-230).
• crates/argondb-committer/src/iceberg_sink.rs — the changelog schema (:10204-10226), GapRecord (:2684-2700), the _argon_gaps Iceberg schema (:2833-2842), ident_label (:4670).
• crates/argondb-query/src/lib.rs — pin() (:4234), scope retention and the system tables (:4141-4233), SCOPE_UNKNOWN_TABLE (:4102), the non-atomic-pin disclosure (:1914-1926), the transient-note remedies (:1808-1822), the gap refusal (:6014-6026), the no-changelog refusal (:5790-5799), the history-restart refusal (truncated_history_note, :5738-5776), the changelog replay query (:6037).
• crates/argondb-ctl/src/lake.rs — publish (:1172-1202), the PublishState spellings (:305-357), the three-cell listing (:1506-1546).
• crates/argondb-ctl/src/lake_address.rs — the whole address surface (:1-140).
• crates/argondb-ctl/src/serve.rs — the tail spawn arm and its environment (:8020-8330), the maintenance spawn arm (:8332-8360), the MCP child (:8380-8400).
• crates/argondb-lake-fdw/src/catalog.rs — the mount, the database scope, the dv_ sibling (:1-60), classify_mount (:308), the sweep and the opt-in drop (:514-579, :630); the GUC itself is crates/argondb-lake-fdw/src/lib.rs:62-67, :170-181.
• crates/argondb-committer/src/lake_writer.rs — table_writer_refusal (:2294), the not-owner text (:1477-1480), the forbidden-property guard (:1564).
• crates/argondb-tail/src/lib.rs — the zero-WAL lever (:3717-3745), ZeroWalRefusal (:3756), record_proves_same_key (:3876), ARGONDB_TAIL_DECODE_DECOUPLED (:4739-4767).
• crates/argondb-mcp/src/lib.rs — ChangesParams (:2658-2671).

Gates

• testing/box/tail-rss-bound.sh — the memory bound, three arms, P1–P5 / N1–N3; its judge is testing/box/tail-rss-bound-verdict.py and its hermetic driver testing/box/tail-rss-bound-selftest.sh.
• testing/box/tail-wal-hold-restart-proof.sh — the hold, both polarities.
• testing/box/tail-drain-residual-probe.sh — drained means residual 0, and zero is reachable.
• testing/box/zero-wal-lever-arms.sh — the zero-WAL lever against lever A, same binary.
• testing/box/dv-lake-landing-polarities.sh — the dv_ landing and its one writer per table.
• testing/box/lake-mount-scoped-wall-matrix.sh — the database-scoped mount, including the stale-mount drop under the opt-in.
• testing/box/view-foreign-reader.sh — DuckDB reads a view table; the snapshot carries Argon.view-current-to-lsn, not Argon.watermark-lsn.
• testing/t3/view-restart-await-matrix.sh --kill — the after-lake-commit seam, killed for real.

Diagrams

• docs/diagrams/argondb-and-the-erp/03-the-lake-and-its-process-model.svg, generator docs/diagrams/argondb-and-the-erp/generate/d03_lake.py, library docs/diagrams/argondb-and-the-erp/generate/blocklib.py.
• The four sub-diagrams and their generators, §7.

Guide 4 — continuous materialized views, and a view across the tenant boundary

Who this is for. A senior developer who has to USE this subsystem: define views, read them, predict what they cost, and diagnose them when they stop advancing. It is one guide of a set, each drawn around one block diagram. It is an internal technical reference; it explains the mechanism and does not argue for it.

How to read a claim in here. Every statement is sourced by path:line to the code or to docs/2026-09-18-continuous-materialized-views-design.md (cited below as "the design", by section). Worked examples in §4 are labelled VERIFIED (run in this lane, with the real output pasted) or FROM SOURCE (the exact signature, flag, constant or table quoted by path:line). Where the picture is dashed, the guide says designed-and-not-built, and §5 has the table.

1. The diagram

Read the picture as four horizontal bands, top to bottom, and read the top band left to right as one refresh happening. The tail decodes every committed change to every published table into a change row carrying _argon_lsn, _argon_op (I/U/D) and _argon_seq. Those rows are reachable from exactly two durable places — the tail's in-memory ring above the lake watermark W, and the D9 <t>__changes changelog at and below it — and which one a consumer should use is decided by when the consumer runs, not by preference. The view maintainer runs inside argondb-maintenance, notices that W has advanced past some view's coordinate L_v, and schedules that view at L_1 = W. It applies the class's arithmetic over the window (L_v, L_1] and lands ONE snapshot whose rows and whose coordinate stamp — Argon.view-current-to-lsn = L_1 — are in the same commit. A read pinned at some L may then be answered from that view: the rightmost box is dashed-on-the-shipped-legend only in the sense that the whole rewrite path is newer than the rest of the band, and §5 states exactly what of it has landed.

The three bands below are the three axes a developer actually chooses on. The CLASSES band is the arithmetic: whether a view can be brought up to date from what changed (C1, C1a, C2, C2m, C3) or has to be recomputed from its inputs (C4), and the classifier decides that at create and prints it. The LANDINGS band is where the rows physically sit: one Iceberg table in dv_<branch> read through the same FDW as every other lake table, or one real Postgres table in derived.* on the primary — one landing per view, never both. The bottom band is the tenancy answer, and its shape is the point: a view never reaches across a wall. It is an ordinary join of two LOCAL tables, one of which happens to be a reference table the hub COPIED into this database earlier. The "no query reaches across" box is not a policy bolted onto views; it is the same wall the FDW mount and every scan ticket already carry, evaluated once more by the maintainer.

Three judgement calls the picture makes and this guide inherits. Dashed outline means designed, or designed and partly built, on this tree — it is not a promise about a date. OPEN means a decision nobody has made; the picture carries OPEN-V2 (retention for view snapshots) and this guide does not resolve it. And the word "sidecar" in this subsystem names only the tenant's own application process; our process is the tail.

2. What it is

One paragraph per box in the picture, carrying the coordinates, states, defaults, limits and failure modes the picture had no room for.

2.1 The tail — the change rows, and the two things they do not carry

The tail decodes the WAL and emits, per published table, a row stream where each row is the table's own columns — the NEW image — plus _argon_lsn, _argon_op (I/U/D), _argon_seq (the order within one commit) and _argon_ts (the design, §3.1). Two absences shape everything downstream. A D row carries the key, not the old image, and a U row carries the new image, not the old; the tail's old-tuple cache is internal to decoding and is not on the wire. So any arithmetic that needs −f(old) — every aggregate class — has to go and fetch the old image, and §3.4 is where that fetch is costed. A third property is load-bearing and easy to miss: a primary-key UPDATE is not a special case. The tail emits it as a D on the old key and an I/U on the new one, because the merge plan treats a key as a partition; the classifier relies on that and the box gate asserts it (testing/box/view-maintainer-incremental-oracle.sh:66-68).

2.2 Two delta sources, and the rule that picks between them

Above W the change rows are in the tail's ring, served over Flight as (from_lsn, to_lsn] tickets — and evicted the moment a cut lands. At and below W they exist only if the table is enrolled in the D9 changelog <t>__changes, an append-only Iceberg table of exactly those rows, committed in the same atomic cut as the base table and retained at least the pin window (48 h). The design (§3.1) states the consequence as a rule: the ring is the right source for the READ-TIME compensation of §4, and the changelog is the right source for MAINTENANCE — because a refresh runs at a cut, and at a cut the ring has just been evicted below it. Therefore a view is maintained incrementally only over inputs whose changelog is enrolled, and a view whose inputs are not enrolled is recompute class whatever its shape. That is the first thing the classifier prints. The first refresh's own coordinate is PINNED AT OR AFTER the enrolment floor, never left floating — Block V2a [D-2026-09-23-fd53b9], amending V2 [D-2026-09-18-b2dc92]: a first refresh raises its pin to the HIGHEST of its inputs' changelog-enrolment floors (costing V1 and D12's protected latest_servable default), rather than pinning at the branch's own servable coordinate, which could sit inside an input's enrolment gap — a window that starts there is reconstructable by neither the changelog (the drain only starts at enrolment) nor the ring (evicted below a cut). A third source exists for a downstream view in a chain — the upstream's own committed snapshot — and §3.8 covers it. And a window that intersects a recorded gap (_argon_gaps) on any input, or that begins below an input's changelog retention, is not a delta: the view recomputes at L_1 with the cause recorded (V4, DESIGN.md:49-54 of §11). Both cases are read from one record, _argon_gaps, by one predicate, argondb_query::view_gap::gap_block, and the cause is named from the record's reason (GapCause::of — Rebaseline or Retention). A changelog retention floor is itself an _argon_gaps row (0, floor], written BEFORE the changelog is pruned (changelog_retention::prune_changelog_through), so a reader never sees rows gone without the floor that explains them (§4.12).

2.3 The view maintainer — where it runs, and the three things it will not do

The maintainer is a task inside the argondb-maintenance process, beside — never inside — the D10 sweep loop, because the two have different clocks: the sweep runs on a 120 s cadence and a view refresh runs on the CUT (crates/argondb-tail/src/view_maintainer.rs:1-16; spawned at crates/argondb-tail/src/bin/argondb-maintenance.rs:139-155). It is a cut follower: it reads the tail's shipped freshness action every ARGONDB_VIEWS_POLL_SECS (default 5 — the resolution at which it NOTICES an advance, not a refresh interval, view_maintainer.rs:85-93) and acts only on a watermark advance. A NEW VIEW REFRESHES ONCE AT CREATION even on an IDLE branch: creating a view moves no watermark, so "acts only on a watermark advance" would leave a view created after its branch's writers finished at zero refreshes forever — the follower therefore re-runs an already-passed cut while any view of the branch has never landed a refresh, and stops as soon as that first refresh lands. A view set that has ALL landed is still never recomputed on an unmoved watermark. Its queue is ordered by staleness: never-refreshed first, then by W − L_v in bytes, then by name, so the order is TOTAL and a gate can assert it (view_maintainer.rs:648-650, the doc comment on schedule at :651). ARGONDB_VIEWS_CONCURRENCY (default 1, :85-88) bounds how many refreshes run at once.

Under per-tenant compute ([compute] per_tenant = true) a tenant's views are maintained by the tenant's own tail, which runs this same maintainer for that database's views and cuts them at the tenant's own coordinate; the box maintainer skips a tenant database by name, one line per pass, and never crosses the wall [D-2026-10-03-f3cf87].

The three promises it makes are in its own module header (view_maintainer.rs:42-43): it holds no ring, it applies no backpressure to the tail, and it never blocks a cut. A maintainer that falls behind simply targets a later cut — nothing is skipped, the window (L_v, L_1] just grows, and the changelog holds it for the pin window. If it falls far enough behind, the views are behind, that is disclosed, and reads are still served.

Two breakers sit around it. The per-view breaker is a STREAK, not a count: ARGONDB_VIEWS_BREAKER_FAILURES (default 3, :79-83) CONSECUTIVE failures mark one view FAILED (<cause>); it stops being scheduled, keeps serving at its own L_v, one success clears the streak, and the rewrite rule stops using it (BreakerVerdict, :210-222). Outside it is the row's ordinary RestartPolicy — backoff 2 s → 30 s, breaker at 6 restarts in 120 s. A tripped maintainer means views stop advancing and reads keep serving at L_v with the disclosure; it is never a refused read.

Its memory is its own pool: ARGONDB_VIEWS_MEMORY_MB bounds a FairSpillPool spilling under ARGONDB_VIEWS_SPILL_DIR (crates/argondb-query/src/memory_pool.rs:104-110), resolved EAGERLY at boot so a bad value refuses the process rather than the first refresh (argondb-maintenance.rs:130-137). The pool is a fourth even part of the box reserve; the cost of adding that fourth claimant is stated rather than smoothed — the read plane's own share moved 3,641 → 2,730 MB on a 32 GB box, and 142 → 106 MB on the 1280 MB floor (the design, §12 L4).

2.4 One refresh = one snapshot, stamped in the same commit

A refresh pins at L_1 = W, plans the view's own definition text, computes under the views pool, and lands through argondb_transform::land::land_view_lake — which is the committer's LakeWriter and nothing else. The snapshot carries the coordinate in the SAME commit as the rows, which is V1's whole content: Argon.view-current-to-lsn = L_1, beside Argon.view-definition-sha256, Argon.view-class, Argon.view-inputs (each input with the W it was read at) and Argon.run-token (so a retry after state-unknown finds its own commit). A view MUST NOT carry Argon.watermark-lsn, Argon.replay-from-lsn or Argon.changelog-through-lsn — those are claims about WAL COVERAGE that only the committer may make (V1, DESIGN.md §11). That is the "one timeline, two clocks" line in the picture: the WAL LSN is the one timeline every ArgonDB datum sits on; an Iceberg snapshot id or committed-at is a clock of the STORAGE.

A refresh killed after its lake commit is wholly present and not re-applied. Because the coordinate rides in the same commit as the rows, there is no second write that a crash can leave behind: the one seam after the commit returns (after-lake-commit, land_view_lake in crates/argondb-transform/src/land.rs) is killed for real by a_refresh_killed_after_the_lake_commit_is_wholly_present_and_not_reapplied (crates/argondb-transform/tests/landing_view.rs), run as leg H of testing/t3/view-restart-await-matrix.sh --kill. After the kill the view is at the new coordinate and the maintainer treats it as current; replaying the dead refresh's run token lands no snapshot and no rows (AlreadyLanded); the next refresh continues from the committed coordinate; and at every step the rows equal an oracle built from the deltas alone. The limit, stated: that test's catalog is an in-memory one re-opened from the newest committed metadata file, standing in for a REST catalog's durable pointer — it is not Lakekeeper. A Postgres-landed view has no such seam (its rows and its argon.view_state row commit in one transaction), and its kill between COPY and COMMIT is not yet proven on a box (§5).

2.5 A read pinned at L — the rewrite rule

The rewrite rule is an OptimizerRule installed by pin(), so every caller of the read plane inherits it: the FDW scan on the analyst compute, the MCP query tool, argondb-query on the CLI, the transform runner's own compute. It is installed FIRST in the optimizer rather than appended, because DataFusion's own filter and projection pushdown rewrite the very sub-plans the rule matches, and a filter pushed under a table's alias would be invisible to a later match and dropped (the design, §4.6). A match replaces the sub-plan with a OneOf extension node; the choice is made at match time by construction (a matched view is by construction a subset of the base scan), so nothing plans the original twice. The rule's answers are the Coordinate enum (crates/argondb-query/src/view_rewrite.rs:1260-1272): Exact, Compensated { from, plan }, Disclosed { behind, age_secs }, Never(reason). Never is not an error — the base plan runs.

2.6 C1 / C1a — the keyed projection

Select-project-filter over ONE table with that table's key passed through (or declared). A refresh applies π σ to the change rows: I/U upsert by key, D delete by key. Exact, cost proportional to |Δ|, no old images (view_classify.rs:55-63, :140-149). C1a is the same view on a window whose delta holds I rows only — checked per refresh, never assumed: classify never returns C1a, and window_specialisation(class, delta_is_insert_only) is its one producer (view_classify.rs:198-203). The picture's "refuse today (oracle gate RED)" caption is the measured status of update/delete windows on this build; §6.1 has it with the remedy.

2.7 C2 / C2m — the algebraic aggregate

GROUP BY g with count, sum, avg (kept as the sum,count pair), min/max. Per group the refresh applies +f(new) for an I, −f(old) + f(new) for a U, −f(old) for a D; a group whose count reaches zero is DELETED. It needs old images for U/D rows, which is the one place a refresh touches base data other than the delta. C2m is the min/max case: when a D or U held a group's current extreme, the AFFECTED GROUPS are recomputed from the base at L_1 (WHERE g IN (…)) and the rest run incrementally. The class is static even though the trigger is per refresh, because the maintainer must carry the affected-group arm for the view at all (view_classify.rs:68-72, :174-186). An aggregate with no GROUP BY is keyed on the empty tuple — one row — and is fine (ViewKey::EmptyTuple, :223-225). A view's key names its OUTPUT columns, and a key that is an alias of a source column is supported (s.txn_month AS month, keyed (upc, month)): the maintainer follows each key column down the view's own plan to its GROUP BY term and reads every group set — the new side, the old side and an insert-only window's empty old side — through that projection, never by applying the key's names to the base (view_delta::group_keys, view_delta::empty_groups; the v1.5.7.9 regression that did is in HISTORY.md).

2.8 C3 — join, changed side known

Fact joined to dimension on an FK (N:1), optionally with C1/C2 on top. The delta is the standard two-term form ΔV = ΔF ⋈ D@L_1 ∪ F@L_0 ⋈ ΔD. As built (view_maintainer::refresh_join, :2077) it is one term per input: each input is rebound to its own collapsed new images while every other input is read at L_1, the view's own definition text is planned unchanged over each binding, and the terms are unioned as a SET. Every term is evaluated at L_1 — a named deviation from the literal F@L_0 ⋈ ΔD, and in the only safe direction: an upsert must land the fact row's image at L_1, and F@L_0 would silently revert a fact change the first term just computed. It is also what makes the two terms produce identical rows for a key both touched, so UNION (never UNION ALL) dedupes them. The delete set is the OLD side, not the tombstones, because a deleted DIMENSION row leaves no tombstone on the FACT and its fact rows must still leave the view.