---
class: reference
purpose: "GENERATED — the whole ArgonDB developer guide: chapter 0 (the themes) plus chapters 1-4 and 7-8, in one document."
template: product
---
# The ArgonDB developer guide
**GENERATED — never hand-edit.** Regenerated at the release cut only
(`make regen-at-cut`, [D-2026-09-26-6bea57]); the stamp above names the cut
and the source commit. The drift gate
(`python3 scripts/generate-developer-guide.py --check`, a `make lint-nocargo`
leg) proves this file byte-identical to the guide of that STAMPED commit,
naming any chapter that differs; `make lint-at-cut` proves it against HEAD
at the cut. The seven chapters below are assembled,
byte for byte apart from heading demotion and in-document anchors, from
`docs/2026-09-20-guide-00-…` through `-04-…`, `-07-…` and `docs/2026-10-04-guide-08-…` — those files stay the cited
sources (`path:line` citations elsewhere in the repo point into them), so
edit a chapter there, never here.
## Table of contents
- [Developer guide 0 — the themes that run through ArgonDB](#developer-guide-0-the-themes-that-run-through-argondb)
- [Guide 1 — one database per tenant, and the wall between tenants](#guide-1-one-database-per-tenant-and-the-wall-between-tenants)
- [Guide 2 — the per-tenant unit: the gate, wake and stand-down](#guide-2-the-per-tenant-unit-the-gate-wake-and-stand-down)
- [Guide 3 — the lake and its process model](#guide-3-the-lake-and-its-process-model)
- [Guide 4 — continuous materialized views, and a view across the tenant boundary](#guide-4-continuous-materialized-views-and-a-view-across-the-tenant-boundary)
- [Guide 7 — agents and the MCP endpoint](#guide-7-agents-and-the-mcp-endpoint)
- [Guide 8 — building an application cloud on ArgonDBCloud: the ERP's configuration system and ERPCloud](#guide-8-building-an-application-cloud-on-argondbcloud-the-erps-configuration-system-and-erpcloud)
---
## Developer guide 0 — the themes that run through ArgonDB
**Register:** internal technical reference, written against ArgonDB v1.5.6.
Every claim below carries a `path:line` into this repository, a section
number (`§4.0`) into one of the guides that follow, or a ruling id `[D-...]`
into the decision registry; nothing here was run against a box by this
chapter. **Naming:**
"sidecar" names only a tenant's own application; the process that turns the
write log into the lake is **the tail**
(`docs/programs.md:106-112`).
The five guides that follow this one each go deep on one region of ArgonDB:
[one database per tenant, and the wall](#guide-1-one-database-per-tenant-and-the-wall-between-tenants),
[the per-tenant unit's gate, wake and stand-down](#guide-2-the-per-tenant-unit-the-gate-wake-and-stand-down),
[the lake and its process model](#guide-3-the-lake-and-its-process-model),
[continuous materialized views](#guide-4-continuous-materialized-views-and-a-view-across-the-tenant-boundary),
and [agents and the MCP endpoint](#guide-7-agents-and-the-mcp-endpoint).
This chapter does not summarize them one at a time — each already summarizes
itself, in its own opening section. What follows instead is the six ideas
that keep reappearing *across* those five regions: read together, they are
what makes ArgonDB one mechanism rather than five, and they are the shape of
what an application developer actually builds and experiences on top of it.
Each theme below has the same four parts: a plain-language paragraph first,
then what the five guides say about it, then what it means for how you
*build* an application, then what you *experience* running one — the
commands, the things you stop having to do, and the refusals you will meet
along the way.
---
### 1. One database per tenant, on its own timeline
**In plain terms:** every customer gets their own database, not a shared
table with a customer-ID column. It's the difference between everyone
keeping their money in one drawer with a label on each bill, versus everyone
getting their own separate drawer. A bug in the label-reading code can empty
the wrong drawer; a bug that reads the wrong drawer only ever reads an empty
one, because the money was never in there.
**Across the chapters.** Guide 01 states the resulting invariant directly:
*"There is no query in this system that could return another company's row,
because another company's rows are not in the database being queried"*
(`docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md` §2.1).
That single-database-per-tenant topology was amended twice in September —
first to give each tenant its own compute that sleeps and wakes on connect
[D-2026-09-20-0d5ba3], then to give each tenant its own Postgres instance,
own WAL and own timeline, with only the storage plane still shared
(`DESIGN.md:391` [D-2026-09-21-4107c5]) — and an operator-only switch,
`[compute] per_tenant`, controls whether a given box actually turns the
per-tenant compute on; it ships off, and no proof or gate flips it
[D-2026-09-23-5122a6]. As of v1.5.6 this is the shape the product ships on one
box, not a design: a *tenancy* (one multi-tenant system in the Estate — the
CLI's word for it is `system`) and the *tenants* inside it are created by two
provisioning commands (`argondb system create`, `argondb tenant create`;
`crates/argondb-ctl/src/main.rs:1059-1082`, `:610-640`), and with
`per_tenant` on, a company's compute is started at `tenant create` on a port
recorded then, so a new company runs on its own compute from its first
minute. Turning it on under companies that already hold data is not yet a
complete move: the data half of provisioning — carrying an existing
company's roles and rows onto its own branch — is not built, and DESIGN.md
says so rather than implying otherwise (`DESIGN.md:387`). Guide 02 is the lifecycle mechanism this topology runs on:
a *unit* is one compute plus at most one sidecar, sized so it is exactly what
can be destroyed and recreated with no loss
(`docs/2026-09-20-guide-02-the-per-tenant-unit-gate-wake-and-stand-down.md`
§1, §2.4) — a stood-down unit measures 137 MB and about 0.6 s to wake
(`docs/2026-09-14-argondb-process-model-preliminary-design.md:114-119`). The
sidecar is the company's own application: one process per company under that
company's own uid and cgroup [D-2026-09-21-7c4769], started only if the
company was created with `--sidecar-command` — "a company with no application
is a complete tenant" (`crates/argondb-ctl/src/main.rs:640-647`). Guide 03's
tail is what makes the topology affordable: past one box it is a single
process multiplexed across every tenant's timeline, never co-resident with
one tenant's own instance, with a per-stream memory budget so the Nth tenant
cannot starve the first
(`docs/2026-09-20-guide-03-the-lake-and-its-process-model.md` §2.12
[D-2026-09-21-25c7c6]; §3.5 [D-2026-09-23-d17cbd]).
**Building on ArgonDB.** You never design multi-tenancy into your own
schema. There is no tenant-ID column to remember on every table and no
`WHERE` clause standing between a bug and a cross-customer leak — a company's
rows live in a database that other companies' queries cannot reach at all.
What ArgonDB *does* keep uniform across tenants is its own per-company
schema, and in v1.5.6 it keeps it uniform across upgrades too:
`argondb estate migrate [--to ]` walks every company in the estate
directory, reads each one's version from that company's own
`argon.schema_version`, skips a company already at the target, and brings the
rest up with the same versioned apply `tenant create` uses
(`crates/argondb-ctl/src/estate_migrate.rs:1-50`,
`crates/argondb-ctl/src/main.rs:563-575`). Be precise about what that schema
is: today it is ArgonDB's own identity table and its one write door
(`UNIFORM_SCHEMA_VERSION`, `crates/argondb-ctl/src/tenant_create.rs:1036-1067`)
— it does not migrate *your* application tables, which stay yours to
version. The connection pooler, for its part, is optional: the box uses a
platform PgBouncer when a supported one (1.25.2 or newer) is on `PATH` and
serves every tenant on the direct port when it is not
(`DESIGN.md:389` [D-2026-09-24-f0e068]).
**What you experience.** Creating a tenancy is one command, `argondb system
create `, and so is creating a company inside it —
`argondb tenant create acme --display-name "Acme" --legal-name "Acme
Holdings LLC" --system distributors --identity "tax_id=12-3456789"` — which
runs the timeline, compute, database, login role and grants, uniform schema,
wall pass, identity record and (if you named one) the sidecar in a fixed
order, is safe to re-run if it is interrupted, and finishes by minting the
company's own MCP token (Theme 6)
(`crates/argondb-ctl/src/tenant_create.rs:206-215`, `:2955-2975`;
`docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md` §4.0
[D-2026-09-21-e781f5]). `system create` now publishes the new system's
Identities to every other registered system in the same call, creating the
Identities namespace itself when nothing has been published into it yet; it
used to defer that grant and leave it to be run by hand later
(`crates/argondb-ctl/src/system_create.rs:1499-1510`). After an upgrade,
`argondb estate migrate` is the one command that brings every existing
company to the new build's schema version; one company it cannot migrate
is named as `REFUSED`, the pass carries on for the others, and the command
exits non-zero naming how many were refused
(`crates/argondb-ctl/src/main.rs:4133-4148`); the pass is a command you run,
not something the box does on its own when a new build is deployed
(`crates/argondb-ctl/src/estate_migrate.rs:19-32`). You never write
idle-detection logic yourself: bytes at the gate are the only signal a unit uses to decide it is still awake, and
the whole control is one config key, `[compute] idle_stop`
(`docs/2026-09-20-guide-02-the-per-tenant-unit-gate-wake-and-stand-down.md`
§2.1). The command REFUSES to adopt a database it did not
create itself, rather than silently taking it over
(`docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md` §4.0),
and a client that connects while a tenant is mid-wake gets Postgres's own
`57P03 cannot_connect_now` — a signal every driver already knows to retry
(`docs/2026-09-20-guide-02-the-per-tenant-unit-gate-wake-and-stand-down.md`
§2.1, §3.6). A missing or too-old PgBouncer is never a boot refusal in any
profile: the box starts on the direct port and says so, on the banner and in
`argondb status`, with a `pooler: NOT RUNNING` line naming what it found and
the minimum it wants; the pooled port, when a pooler runs, is the direct port
plus 200 (5632 at the defaults), and `[pooler] enabled = "off"` runs none
(`crates/argondb-ctl/src/pooler.rs:79`, `:813-832`). The release installer
writes `[pooler] enabled = "on"` into the box's `/argon.toml` only
when the key is absent, and `on`, `auto` and an unset key are one code path
(`scripts/release/install.sh:98-106`). The one pooler setting that still
refuses the boot is a misspelled `enabled` value
(`crates/argondb-ctl/src/pooler.rs:221-240`).
---
### 2. The wall — the thing every surface is measured against
**In plain terms:** imagine every tenant's data behind its own locked door,
with its own key. A feature doesn't get to say "trust me, I checked the
badge" — it has to actually not have a key to the other doors. That's the
difference between a rule a programmer could forget and a wall a programmer
cannot walk through by accident.
**Across the chapters.** Guide 01 rules the wall as three parts together,
never a choice among them — catalog authorization, login, and a warehouse
plus bucket plus credential scoped per tenant database
(`docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md` §1, §2.6
[D-2026-09-11-21a6d7]) — and states plainly that the wall is "not a filter
inside a query" but a different database, a different place in storage, a
different credential (`deploy/publish/architecture-site/content.md:374-381`).
One predicate, `wall::decide`, gates every object-store operation
(`docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md` §3.4),
and the *same discipline* reappears wherever a boundary is checked: Guide 04
evaluates a view's `database` with the identical predicate the pin uses, and
refuses a cross-database join by name rather than letting the query planner
narrow it away with a `WHERE` clause
(`docs/2026-09-20-guide-04-continuous-materialized-views.md` §2.13, §3.4);
the lake's read plane refuses a read of another database's table the same
way, by name, unless that database has granted it (Theme 5;
`cross_database_refusal`, `crates/argondb-tail/src/lib.rs:924-940`).
Row-level security is explicitly a *different, narrower* mechanism: it stays
inside one tenant's database and separates roles, not companies, so a
defective policy is an over-permissive grant inside one company rather than a
cross-customer breach; and a table's lake copy carries no row policies, so the
read plane refuses a scan of a protected table's lake copy rather than serving
it (`docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md` §2.9).
Guide 01 states the one limit on all of this plainly: the box operator stands
inside every wall, and a tenant never does
(`docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md` §2.7,
§6). It is equally direct about the storage plane under the databases: every
company is a timeline of the box's one pageserver tenant, the pageserver
refuses a caller with no token, and each compute's token names its one
timeline, so the pageserver refuses a company's valid token on any other
company's timeline — built [D-2026-09-23-b822c8] and asserted by the W5±
cases of `testing/box/cross-company-isolation-eight-surfaces.sh`. With the
default `[compute] per_tenant = false`, one compute serves every company's
database and holds the box's own timeline, so the storage token is not what
separates companies there; the connect wall and the password rule are
(§2.6).
**Building on ArgonDB.** You do not write an authorization predicate to keep
tenants apart — there is nothing to get wrong, because there is no code path
that reaches across in the first place. The corollary is the design
constraint that matters most when you build a feature that legitimately
needs two parties' data in view together: it cannot be a query that spans
the wall on its own. It has to be a component that stands with a foot on
each side and makes an explicit, recorded copy — which is exactly what Theme
5 (consent and grants) exists to do, and Theme 4 (views) is how you keep
that copy usable once it has landed.
**What you experience.** Nothing to configure — the wall holds without a
setting once tenant warehousing is on by default, and the catalog
authorization engine it needs (OpenFGA) ships inside the release artifact on
both Linux and macOS, so a multi-database box never meets a boot refusal for
a missing engine and there is nothing for you to install
(`scripts/release/build-artifact.sh:593-611`,
`scripts/release/build-artifact-macos.sh:519-523`;
`docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md` §2.6).
You probe the wall, you don't build it: `argondb lake grant`, `argondb lake
attach-clause` and `argondb lake revoke` hand out and take back a reader's
token (`crates/argondb-ctl/src/main.rs:1235-1275`;
`docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md` §4.3); a
revoke removes the catalog permission, and a token already issued still logs
in until it expires but can no longer read anything
(`crates/argondb-ctl/src/main.rs:1246-1251`).
The refusals are specific and are part of the interface, not an edge case: a
bucket access across the boundary answers `403 AccessDenied`, naming the key
and both buckets; a catalog warehouse you cannot see answers `404`, never
`403`, and does not disclose its name at all
(`docs/2026-09-20-guide-01-one-database-per-tenant-and-the-wall.md` §3.4,
§3.5, §4.6);
a view joining tables from two databases refuses `cross-database-join`,
naming the remedy instead of a bare error
(`docs/2026-09-20-guide-04-continuous-materialized-views.md` §3.4).
---
### 3. The lake as the shared read side, the tail as what feeds it
**In plain terms:** every change you make in the transactional database
shows up, seconds later, in a second read-only copy built for big questions
— without you writing a sync job. It's like a photocopier standing next to
your filing cabinet that copies every page the moment you file it, so
someone else can flip through the copies without ever touching your
originals.
**Across the chapters.** The thesis is stated once, in one sentence:
*"every Postgres table is automatically an Iceberg table, at a known LSN,
with sub-second freshness and zero impact on transactional performance"*
(`DESIGN.md:36`).
Guide 03 is the mechanism: the tail decodes the write log once — "the single
decode path" — and the ring, the changelog, the lake copy and every reader
all feed from that one decode, which is what makes "the lake cannot drift"
a structural fact rather than a hope
(`docs/2026-09-20-guide-03-the-lake-and-its-process-model.md` §2.3). A
read at position `L` is `Iceberg@W ∪ tail(W, L]` — the durable copy up to
the watermark `W`, plus the tail's own decoded changes above it
(`docs/2026-09-20-guide-03-the-lake-and-its-process-model.md` §1, §2.9). Guide
02 explains why the tail can sit resident beside a *stood down* tenant
without waking it: with the compute asleep there is no new WAL to decode, so
the tail idles at baseline
(`docs/2026-09-20-guide-02-the-per-tenant-unit-gate-wake-and-stand-down.md`
§2.3). Guide 04's view maintainer is a second consumer of the exact same
watermark — it advances a view's answer only when the tail's own freshness
action reports the watermark moved, never on a timer of its own
(`docs/2026-09-20-guide-04-continuous-materialized-views.md` §2.2). The
changelog the tail writes beside a published table is not optional
bookkeeping — it is the same decode path, so an audit trail read from it is
complete by construction rather than by discipline
(`docs/2026-09-20-guide-03-the-lake-and-its-process-model.md` §2.6). And the
lake has exactly one writer per territory, which v1.5.6 now enforces per
*table* as well as per namespace: the committer alone writes the mirrored
`br_` tables, ArgonDB's lake writer alone writes the derived
`dv_` namespace (continuous views, lake-landed transforms and
appends), and inside `dv_` each table has exactly one writer — a
view's table is written only by the view maintainer, and the view maintainer
never writes a table that is not a view
(`docs/2026-09-20-guide-03-the-lake-and-its-process-model.md` §2.7;
`table_writer_refusal`, `crates/argondb-committer/src/lake_writer.rs:2310`).
**Building on ArgonDB.** A reporting feature is not a project with its own
ETL job, its own schema mapping and its own reconciliation step. It is a
`SELECT` over a copy that is guaranteed structurally consistent with the
transactional table, because both descend from one decode of one write log.
The design question that actually matters is which tables to publish and at
what freshness tolerance you can afford to read them, not how to keep a
second system in sync with the first. A result you compute *from* the lake —
a transform or a view — can land straight back into it, in `dv_`,
with no Postgres transaction and no `--pg-url`
(`argondb transform run --landing lake`,
`crates/argondb-ctl/src/main.rs:2051-2066`); pick a name no other door
already owns.
**What you experience.** Publishing a table is one command with no
connector to configure: `argondb lake publish ` sets an
`argon:publish` marker, backfills once at a pinned position, then follows
incrementally
(`DESIGN.md:406`; the same command run,
`docs/2026-09-20-guide-03-the-lake-and-its-process-model.md` Example 1). You
never write a CDC connector, define a Kafka topic, or run a reconciliation
job. `argondb lake list` shows you exactly where each table stands —
`lake @ 0/1BD2748`, `backfill pending`, or `QUARANTINED @ ...` — and freshness is
a number you read, `ingest_lag_seconds`, never a promise you take on faith
(`crates/argondb-ctl/src/lake.rs:300-318`; `ingest_lag_seconds`,
`crates/argondb-tail/src/metrics.rs:1557`). The one
refusal worth knowing in advance: a column type the decoder cannot handle is
accepted and stored durably but never arrives in the lake, silently, with no
SQL error — which is exactly why a `publish_status` check exists as a
separate, explicit thing to run
(`docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md` §4.7). The second:
landing into a `dv_*` table another door owns — a lake-landed transform onto
a view's table, say — refuses `not-owner` and writes nothing: *"every dv_\*
table has exactly ONE writer; nothing was written. Land under another name,
or remove {target} through the door that made it"*
(`crates/argondb-committer/src/lake_writer.rs:1477-1480`).
---
### 4. Views as a statement of what should stay current
**In plain terms:** instead of writing a script that re-runs a big report
every night and hoping it finishes before someone asks for it, you write the
report's definition once — "this is the number I want" — and the system
keeps that number accurate for you, forever, updating only the parts that
actually changed. You stop being the one who has to remember to press
refresh.
**Across the chapters.** Guide 04 is the primary mechanism: a continuous
materialized view keeps its answer as of a coordinate `L_v` and advances it
by applying the change rows in `(L_v, L_1]` rather than re-running the
query, and a reader gets the exact current answer or a named older one with
its distance stated, never a silent guess
(`docs/2026-09-20-guide-04-continuous-materialized-views.md` §1, §3.10).
The definition surface is deliberately a function call, `argon.create_view(name,
sql, options)`, not a native Postgres `MATERIALIZED VIEW` — because Postgres
owning the refresh is exactly the manual-refresh problem this exists to
remove
(`docs/2026-09-20-guide-04-continuous-materialized-views.md` §3.1).
A classifier decides, from the query alone, which of four maintenance
strategies applies — a keyed projection, an algebraic aggregate, a join, or
(when none of those hold) a full recompute — and it over-approximates toward
the safe, expensive option rather than the reverse
(`docs/2026-09-20-guide-04-continuous-materialized-views.md` §2.6-§2.9, §3.5).
The second mechanism, a **transform**, is the one-shot sibling of a
continuous view — a `SELECT` computed once at a pin and landed with a mode
(`create`, `append`, `replace`, `drop`), rather than kept current forever
(`argondb transform run`, `crates/argondb-ctl/src/main.rs:2044-2066`; the
same run over MCP,
`docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md` §4.5). Because it
is pinned, re-running it at the same coordinate gives the same output, where
a hand-written batch that buckets against `now()` silently rewrites history
when re-run after the fact. Both land in one of two places: the `derived`
Postgres schema, reaching the lake through the tail, or the lake-owned
`dv_` namespace directly; a view lands in `dv_` by default
(`docs/2026-09-20-guide-04-continuous-materialized-views.md` §2.10, §2.11).
The two compose directly: Guide 04's own worked example is a continuous view
created *over* a table landed into a tenant's database from another party
(`docs/2026-09-20-guide-04-continuous-materialized-views.md` §4.9) — Theme 5
and Theme 4 meeting in one example.
**Building on ArgonDB.** Per output, you choose one of three postures: an
ordinary application table your own code writes; a **transform**, a
pinned, reproducible one-shot `SELECT` for something tied to a business
event like a period close; or a **continuous view**, a `SELECT` stated once
that the system keeps current on its own. The fork that matters is whether
the coordinate itself is the meaningful artifact (a transform, pinned at
month-end) or the answer must never be allowed to go stale between requests
(a view). Neither requires you to reason about incremental maintenance by
hand — the classifier does, and it never guesses wrong in the unsafe
direction.
**What you experience.** `argon.create_view('name', 'SELECT ...', options)`
from SQL, `argondb view create --name ... --sql ...` from the CLI, or the
`create_view` tool over MCP — the same body, three doors
(`docs/2026-09-20-guide-04-continuous-materialized-views.md` §3.2). You
never write a refresh cron job, never decide by hand whether a deleted row
needs its group total decremented. Sixteen refusal kinds are closed and
specific — `cross-database-join`, `non-deterministic` (a view over `now()`
or `random()` is refused even for a recompute-class view), `self-cycle`, and
others — each naming the offending construct and a remedy
(`ViewRefusalKind::ALL`, `crates/argondb-query/src/view_classify.rs:486`;
`docs/2026-09-20-guide-04-continuous-materialized-views.md` §3.4). A
reader who wants an exact answer gets it or a stated refusal to the base
data; the default staleness tolerance is zero
(`docs/2026-09-20-guide-04-continuous-materialized-views.md` §3.10). A view's
lake table belongs to the view: a lake-landed transform or append aimed at it
refuses `not-owner` and writes nothing, so nothing but the maintainer can
move a view's answer (Theme 3). Guide 04 keeps its own list of what is still
red on this program, and names it rather than rounding it off
(`docs/2026-09-20-guide-04-continuous-materialized-views.md` §9).
---
### 5. Consent and grants — how data crosses a boundary
**In plain terms:** a company's data never leaves its own room by an
automatic back door. If another company is allowed to see part of it, that
permission is written down — as a row in a table, in the room it came from —
so anyone can later see exactly what was allowed, when, and to whom, and can
take it back.
**Across the chapters.** Consent is a row that lives in the database of the
party whose data crosses, written with that party's own ordinary Postgres
privileges — never a config file and never an operator setting — and
published, so the consent record reaches the lake beside the rows it governs.
The general form, new since v1.5.5, is the **mount grant**: a database that
wants another database to read one of its lake tables writes
`hub.mount_grant (grantee_database, source_schema, source_table)`, one row per
table, with no wildcard, and a later `CREATE TABLE` is not shared by it
(`MOUNT_GRANT_DDL`, `crates/argondb-transform/src/hub_consent.rs:298-313`).
The lake's read plane then serves that one table to that one grantee out of
the owner's warehouse, and refuses every other cross-database read by name
(`DatabaseScope`, `crates/argondb-tail/src/lib.rs:909-922`). The state is not
a boolean: `grant_state()` returns one of four facts — granted, no grant,
revoked (with the instant), or granted under a contract version this build
does not serve — because "not granted" and "revoked" and "granted under an
unserved version" are different things for a reader of the log to know
(`GrantState`, `crates/argondb-transform/src/hub_consent.rs:104-121`). The
same predicate decides the older, flow-shaped grant table, `hub.share_grant`,
which scheduled landings (`argondb publication land|serve`, `argondb hub
run`) consult before they move rows from one party's database into
another's (`crates/argondb-transform/src/hub_consent.rs:40-48`;
`crates/argondb-ctl/src/main.rs:2365-2400`). The component that moves the
data holds two identities without ever mixing them — it reads every tenant
under one cross-tenant, read-only identity and writes as the box, and there
is no flag that lets it read as the box on a walled database
(`crates/argondb-ctl/src/hub.rs:69-86`). Guide 07 shows the same doctrine one
layer up, at the level of what an agent is allowed to do rather than what
data it can see: the admin mount's destructive calls require an explicit
`confirm=true` and refuse, naming what would be lost, without it — "an agent
proposes; the operator confirms"
(`docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md` §2.3, §4.6;
`crates/argondb-mcp/src/admin.rs:17-20`).
**Building on ArgonDB.** You never design an outbox table, a webhook, or a
second replication path to move data between parties — that door is closed
by design, not left to you to build carefully. Instead you decide what
crosses: which table, to which database — and the owning party writes a
consent row, in its own database, using its own ordinary write privileges.
Because consent is data, it travels through the tail like any other row, so
it is queryable, auditable and versioned the same way any other business fact
is, not a side channel nobody can see.
**What you experience.** `INSERT INTO hub.mount_grant (grantee_database,
source_schema, source_table) VALUES (...)` in the owner's database to grant a
read; `UPDATE hub.mount_grant SET revoked_at = now() ...` to revoke it.
A read of another database's table with no grant refuses `cross-database lake
read refused`, naming the owner, the asker and the exact `INSERT` that would
grant it; a read after a revoke refuses with a *different* sentence that says
when the grant was revoked
(`cross_database_refusal`, `revoked_mount_refusal`,
`crates/argondb-tail/src/lib.rs:924-960`). Two limits, stated directly: a
revoke takes effect within one schema-refresh cycle of the read plane, not
at the instant of the `UPDATE`, so a read in between can still succeed
(`crates/argondb-tail/src/lib.rs:942-960`); and revocation is forward-only —
rows already landed by a flow grant stay where they landed. Pulling them back
is a separate, deliberately destructive operator command, `argondb hub
recall`, and it refuses while the grant is still in force, naming the `UPDATE`
the granting party runs first
(`crates/argondb-ctl/src/hub.rs:1440-1458`). A flow grant made at a contract
version the running build does not serve refuses by name,
`ContractVersionUnserved`, rather than landing nothing silently: the grantor
did consent, just to something else
(`crates/argondb-transform/src/hub_consent.rs:115-120`). `argondb hub grants`
prints the current state of every flow grant and always exits 0 — it reports
consent, it does not judge it (`crates/argondb-ctl/src/main.rs:2554-2576`).
---
### 6. Agents read through the same doors as people
**In plain terms:** an AI agent connecting to ArgonDB does not get a special
side entrance. It gets handed a key to one specific door — the same kind of
key you would hand a human analyst, a dashboard, or a batch job — and which
door decides everything it can and cannot do from then on.
**Across the chapters.** Guide 07 is the primary mechanism: the MCP surface
*is* ArgonDB's API, split into four mounts by trust tier — analyst, dba,
admin, write — each with its own bearer token, its own port, its own bind
default and its own backing Postgres role
(`docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md` §1). "An
operator composes an agent's capability by choosing which mounts it
connects, so the risk analysis happens at connect time rather than by
curating a tool list"
(`docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md` §1). On the
analyst mount, read-only is architectural, not a parser's opinion: the
Postgres role is the boundary, and a write attempt through it fails with
Postgres's own `42501 insufficient_privilege`, the same error a human would
get with the same grant
(`docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md` §2.1, §4.9).
v1.5.6 carries that same principle down to the company: every company now has
its own MCP token for the analyst mount, and a request carrying it runs as
that company's own read role, `_mcp`, which can connect to that
company's database and no other, and holds no write privilege at all
(`crates/argondb-mcp/src/company_token.rs:194-203`;
`crates/argondb-ctl/src/mcp_company_token.rs:45-83`). The box token is no
longer the only credential an agent can be handed; it stays the operator's,
with operator scope, by name (`crates/argondb-mcp/src/lib.rs:6484-6490`).
Guide 04's rewrite rule makes no exception for who is asking: it is an
optimizer rule installed by `pin()`, so the FDW scan on an analyst's compute,
the MCP `query` tool, the CLI's own query path, and a transform runner's
compute all inherit the identical view-rewrite treatment
(`docs/2026-09-20-guide-04-continuous-materialized-views.md` §2.5). And Guide
03's lake-facing MCP mount is stated to be "always" reading through the same
query plane as everything else, never a side channel of its own
(`docs/2026-09-20-guide-03-the-lake-and-its-process-model.md` §2.11).
**Building on ArgonDB.** You never build a separate, elevated access path
for AI. You compose an agent's entire blast radius by choosing which mount
its process connects to — the same lever an operator already pulls to scope
a human analyst, a read replica, or a batch job. An agent given the analyst
mount is bound by exactly the tenant wall (Theme 2), the exact pin
discipline (Theme 3), and the exact view-rewrite machinery (Theme 4) that
bind a person using the same mount; there is no seventh theme reserved for
"what agents get to skip." When the agent works for one company, hand it
that company's token rather than the box's: its reach is then one database
by construction, not by a prompt's good behavior.
**What you experience.** One `claude mcp add` per mount, each its own
URL and token; a session runs `schema_overview` → `pin` → `query` (against
that pin) → `changes(since=)` to learn what has moved since — the same
`pin_id` carried across the whole burst
(`docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md` §7.2). You
never write a SQL parser to keep an agent read-only — the grant system
already does the whole job. A company's token is minted by `argondb tenant
create` and written, never printed, to a `0600` file,
`/tenants//mcp-token`; `argondb tenant mcp-token ` re-issues
it (the old one stops working) and `argondb tenant mcp-token --revoke`
revokes every token of that company and sets `_mcp` to `NOLOGIN`
(`crates/argondb-ctl/src/tenant_create.rs:2079-2081`, `:2955-2975`;
`crates/argondb-ctl/src/main.rs:677-686`;
`crates/argondb-ctl/src/mcp_company_token.rs:90-94`). The token is checked
on every request against the live record, so a re-issue or revoke takes
effect with no restart, and every response names the scope it ran under in
an `x-argondb-mcp-scope` header, `operator` or `company:`
(`crates/argondb-mcp/src/company_token.rs:53-56`, `:265-270`). The limits,
directly: a company token works on the analyst mount only — the dba, admin
and write mounts keep the one operator token
(`crates/argondb-mcp/src/lib.rs:6240-6248`) — and the two analyst tools that
read box-wide state, `search` and `publish_status`, refuse under it by name,
pointing at the operator (`crates/argondb-mcp/src/lib.rs:4611-4621`). A
wrong bearer token on any mount refuses with `401` before any tool body runs
— a forged, revoked or superseded company token each with its own sentence
(`crates/argondb-mcp/src/company_token.rs:102-124`) — and a non-loopback bind
on the admin or write mount with no token configured refuses to start the
process at all, rather than starting it open
(`docs/2026-09-20-guide-07-agents-and-the-mcp-endpoint.md` §3.1).
---
### How the six fit together
```text
one write log
|
v
two shapes of one truth ---------> the lake (theme 3)
(Postgres table, Iceberg table) |
| v
v views state currency (theme 4)
one database per tenant (theme 1) |
| v
v consent moves a copy
the wall (theme 2) <------------------- across it (theme 5)
|
v
every reader goes through it the same way --- people and agents alike (theme 6)
```
Every theme above is a different vantage point onto the same handful of
mechanisms: one write log, one wall, one pin, one classifier, one shape of
consent row. An application built on ArgonDB is not five separate
integrations — it is one mechanism, read from five directions, and this
chapter is the map between them. Guides 01, 02, 03, 04 and 07 are where each
direction goes deep; start with whichever theme above matches the feature in
front of you.
---
## Guide 1 — one database per tenant, and the wall between tenants
**Audience: a senior developer who has to USE this.** It explains the
mechanism, states its limits and its failure modes, and gives worked examples.
Every claim carries its source by `path:line`; a claim without one is a defect.
Register: internal technical reference.
**How to read the examples.** Each one is labeled **VERIFIED** (run, with its
real output pasted) or **FROM SOURCE** (the exact signature, flag, message or
table quoted from the code or a design by `path:line`). This lane's box had no
vendored PostgreSQL install and no built storage tier, so no box gate could run
here; every example below is **FROM SOURCE** except where it says otherwise,
and each names the gate that proves it on a box. The `path:line` coordinates
are at commit `c9415ae0a`, except in §4.0 and §4.9-§4.12 and the rows and
gotchas that cite them, which were re-read for v1.5.6 at commit `e90da8c0c`.
**Labels the pictures use, and this guide uses with them.** A solid outline is
built and shipped. A dashed outline is designed and measured but not built.
`OPEN` marks a decision still to be made. Where a source says `UNCERTAIN` or
`NOT MEASURED`, this guide repeats it rather than rounding it off.
---
### 1. The diagram
The left panel is one PostgreSQL cluster on one box. A tenant is a DATABASE in
it — Block 4 B [D-2026-09-11-1fc4b3], since amended twice over to B′ and then
B′ (amended) [D-2026-09-20-0d5ba3] [D-2026-09-21-4107c5] — see §2.1 for the
current wording — and the three columns are three member
databases with the same shape: each has its own tables, its own SQL credential
and its own lake key, its own place in the lake's address space, and its own
published set. Two of the columns are distributors and one is a manufacturer,
and the picture deliberately draws them identically: they are the same kind of
thing, told apart by WHICH SCHEMA they publish in, not by a flag
(`crates/argondb-query/src/all_tenants.rs:91-112`). Along the bottom of that
panel are the three facts a box diagram cannot show: the provisioner runs a
pass every 30 seconds and gives every new database its whole wall within one
pass; the primary database is never a tenant and is never limited; and
row-level security separates ROLES inside a database, never companies
(`docs/2026-09-04-erp-spec.md:328-351`).
The right panel is the wall itself — Block 5, ruled `D: A + B + C`
[D-2026-09-11-21a6d7], which means all three parts and not a choice among them.
**A** is catalog authorization, so the catalog refuses to list or read a
warehouse a caller is not entitled to and does not even disclose its name.
**B** is a login in front of that catalog, because authorization with no
authenticated subject authorizes nobody and the box would not be able to
bootstrap its own catalog
(`docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:611-622`,
`:790-794`). **C** is a warehouse, a bucket and a credential per tenant, which
is the only one of the three whose boundary is also a STORAGE boundary — the
measurement behind that is that an Iceberg namespace has no storage prefix of
its own, so there is nothing below the warehouse for a key to be scoped to
(`:168-189`). Below those three sit the five principals and what each holds,
and the bottom band is the one way rows cross: the box reads every tenant under
one read-only identity, makes a COPY into the receiving party's own area, and
the receiving party reads that copy under its own key. No query reaches across.
---
### 2. What it is
#### 2.1 One PostgreSQL cluster, and a tenant is a database in it
The topology is a ruling, not an implementation detail: hundreds of tenant
databases share one compute, one write-ahead log, one lake arm and one catalog
(`docs/2026-09-04-erp-spec.md:89-100`). What that boundary buys is stated
exactly at `docs/2026-09-04-erp-spec.md:135-142`: *"There is no query in this
system that could return another company's row, because another company's rows
are not in the database being queried."* That is a property of the topology,
not of every predicate in every code path.
**The topology ruling has been amended twice since, and the amended wording is
the current standing rule — not what a box runs by default.**
[D-2026-09-20-0d5ba3] ratified "B′": each tenant database is served by its OWN
compute that stops when idle and wakes on connect, rather than sharing the
box's one compute. [D-2026-09-21-4107c5] amended B′ again, the same evening: a
tenant is a database that is its own Postgres instance on its OWN TIMELINE
with its own WAL; "one cluster" now names the SHARED STORAGE PLANE — one
pageserver, one safekeeper and one lake serving every tenant's timeline — not
one shared write-ahead log. So of the four things this section's first
paragraph names as shared, two no longer are: compute and the write-ahead log
are ruled PER TENANT database. That is the standing topology; it is ratified
and **built, default off** (§5; `crates/argondb-ctl/src/tenant_compute.rs:105`,
HISTORY#per-tenant-compute-table), and the `[compute] per_tenant` switch that
turns it on is the operator's own setting, never a system default
[D-2026-09-23-5122a6]. Everything else in this chapter — the worked examples
in §4 included — describes the box with that switch off, which is what ships
today.
Two coordinates the picture could not carry. First, what counts as a database
is one `WHERE` clause, spelled once: `datallowconn AND NOT datistemplate`
(`crates/argondb-ctl/src/lake_tenant.rs:787`), and the tenant wall, the estate
provisioner and publish discovery all use it, so the three cannot disagree
about what a database is (`crates/argondb-ctl/src/transform_estate.rs:59-63`).
Second, the primary database — the one `--source-url` names, `postgres` on a
box — is not a tenant: it keeps the box warehouse `argon` and the box
credential, it is never given a connection limit, and it is excluded from every
tenant principal by the same predicate that refuses to grant on it
(`crates/argondb-ctl/src/lake_grant.rs:50-68`).
**Formerly OPEN, now ruled and built behind an operator's switch — the picture
still draws it open.** Whether hundreds of tenants keep sharing one compute or
each tenant unit gets its own is no longer undecided: B′ (amended) rules that
each tenant gets its own compute on its own timeline
[D-2026-09-20-0d5ba3] [D-2026-09-21-4107c5]. What stays a live question is
WHEN a given box turns that on — and that is the operator's call, never the
system's [D-2026-09-23-5122a6]. Deep dive 2 covers the per-tenant unit's gate,
wake and stand-down.
#### 2.2 The member database, and what makes one a manufacturer
Each column of the left panel is a member database. A distributor's ERP tables
live in its own schemas (`public`, and whatever the application uses). A
manufacturer's published tables live in `mfr`. The hub's control tables live in
`hub`, and what the hub lands back lands in `derived`. That is the whole
distinction: `all_tenants.__` unions every member database's
copy of a name, so without a schema rule a manufacturer holding
`public.sales_order_lines` would be read as a distributor by every flow
(`crates/argondb-query/src/all_tenants.rs:91-102`).
The rule has a sharp edge worth stating at the start. The `mfr` schema is
ESTATE: the per-database provisioning pass creates it in EVERY member database,
distributor and manufacturer alike, because there is one provisioner path and
no manufacturer-shaped branch in it
(`crates/argondb-ctl/src/transform_estate.rs:28-34`). So **holding the schema
makes nothing a manufacturer; PUBLISHING in it does**, and the product asks
that question of the copies a pin composed rather than of a policy list that
could disagree with the data (`crates/argondb-query/src/all_tenants.rs:104-111`).
#### 2.3 Its role and its key — one SQL credential, one lake key, 20 connections
The "20 connections (fixed limit)" in the picture is
`DEFAULT_CONNECTION_LIMIT = 20`
(`crates/argondb-ctl/src/lake_tenant.rs:596`), and it is a ruling: every tenant
database gets the SAME number, the operator raises it per tenant, and nothing
re-fits it as tenants come and go — the even-share alternative
(`(max_connections - reserved) / tenants`) was considered and not chosen
(`crates/argondb-ctl/src/lake_tenant.rs:119-125`). It is PostgreSQL's own
`ALTER DATABASE … CONNECTION LIMIT`, so its semantics are PostgreSQL's:
superusers are exempt, and the count is of connections to THAT DATABASE, not of
the role (`:136-141`).
The lake key is `argon-tenant-`, minted by the provisioner into
`lake-store-credentials.json` under the data directory. The access key is
readable on purpose and is not a secret — it appears in the store's refusal
lines and in a SigV4 credential scope, and an opaque one would make every
refusal a lookup (`docs/reserved-identifiers.md:6732`).
#### 2.4 Its identity in the lake — `database.schema.table`
A published table's address has FOUR parts — branch, database, schema, table —
and the database is always present, the box's own `postgres` included
(`docs/lake-foreign-read.md:430-439`). SQL gives a reader three name parts, so
the first three are written as one quoted identifier:
`lake."br_main.shop.sales".orders`. That is not a style choice: without the
quotes the engine sees four name parts and stops at the parser with
`Parser Error: NameListToString NOT IMPLEMENTED`, which reads like a broken
query rather than a quoting mistake (`docs/lake-foreign-read.md:449-456`, and
the measurement it comes from is
`docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:220-226`).
#### 2.5 Its published set — a table comment, and the default is nothing
Publication is opt-in per relation and the marker is a table comment:
`COMMENT ON TABLE sales.orders IS 'argon:publish'`
(`docs/upc-hub-recipe.md:470-478`). Nothing is published by default. The
consequence for this guide is the ordering one: a database that exists and has
published nothing still gets its whole wall on the next provisioner pass,
because provisioning is keyed on the DATABASE and not on the first publish
(`crates/argondb-ctl/src/lake_tenant.rs:22-32`).
#### 2.6 The wall's three parts, and which of them is on today
| part | what it enforces | state today |
|---|---|---|
| **A** — catalog authorization (OpenFGA behind the Iceberg catalog) | the catalog refuses list/read on a warehouse the caller is not entitled to, and answers 404 rather than 403 so the NAME is not disclosed | built; DEFAULT is `openfga` on a box holding more than one database, `off` on a single-database box, explicit always wins (`docs/2026-09-15-cross-tenant-read-design.md:502-504`) |
| **B** — a login in front of the catalog | every catalog request carries a verified subject; the box holds the signing key and serves the two documents itself, so no identity provider is needed | built 2026-09-11 under Block 9 [D-2026-09-11-b5b145] (`docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:846-854`) |
| **C** — a warehouse, a bucket and a credential per tenant | a key opens one bucket, so the STORE enforces the wall even for a reader that bypasses the catalog entirely and speaks `s3://` | built; `warehouse_per_database = true` (`crates/argondb-lake-store/src/wall.rs:4-10`) |
A and B are ONE landing, not two phases, and the reason is upstream's own
sentence: *"Authorization is only effective if Authentication is enabled"*
(`docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:616`).
With `AUTHZ_BACKEND=openfga` and no authentication the catalog is not a wall
with a door — it is a locked door with no key cut, and the measurement says so
exactly: anonymous bootstrap 401, anonymous warehouse creation 403, anonymous
listing 403 (`:780-794`).
**Both platforms bundle it, named rather than implied:** the release artifact
stages `bin/openfga` on Linux — `scripts/release/build-artifact.sh` refuses
the build if its license text is missing (`:243-246`) and then stages the
binary itself (`:584-591`) — and on macOS — `scripts/release/build-artifact-macos.sh`
runs the same license gate (`:223-226`) and stages the binary through the
shared `scripts/release/lib/openfga-bundle.sh` leg (`:519-523`) against the
`openfga-darwin-arm64` content pin. So neither platform's stock
multi-database box meets a boot refusal for a missing engine
(`docs/2026-09-15-cross-tenant-read-design.md:513-516`).
**A fourth surface, not one of these three: the storage plane underneath
them.** Parts A–C wall the LAKE. The OLTP side has its own wall, and on
2026-09-23 the isolation gate measured a real gap in it: the box's pageserver
ran with no credential check at all, so a credential-less process on loopback
read any company's timeline under the box's one pageserver tenant. The box now
signs a NeonJWT for every compute, the tail and its own management calls, and
the pageserver refuses a caller carrying no token or the wrong one — closing
that credential-less read. Every company is still a TIMELINE of the box's ONE
pageserver tenant, so the token also names a timeline: the pageserver token
carries an ARGON: `timeline_id` claim (`storage/pageserver/src/auth.rs`),
`compute_token(tenant, timeline)` mints each compute's token for its one
timeline (`crates/argondb-ctl/src/pageserver_auth.rs`, called at
`crates/argondb-ctl/src/serve.rs:13567`), and the pageserver refuses that
token on any other company's timeline — HTTP 403 on the pageserver's HTTP port and a
JWT authentication error on the page service (basebackup and pagestream) — and
logs `pageserver auth: timeline mismatch`. This is built [D-2026-09-23-b822c8],
merged 2026-09-24 (b2743bdc9). The W5− and W5+ cases of
`testing/box/cross-company-isolation-eight-surfaces.sh` assert it: A's compute
token is refused on B's timeline on both ports, and the same token reads A's
own timeline. Two limits, stated plainly. With `[compute] per_tenant` off, the
default, one compute serves every company's database and its token names the
box's own timeline, so what separates companies there is the connect wall and
the password rule, not the storage token. And the box's own processes — the
tail and the management client — hold a `pageserverapi` token with no
timeline, which reaches every timeline; no company is ever given it. The
pageserver must be built with this change: an older pageserver treats the new
token as tenant-wide.
#### 2.7 The five principals
Full detail, including which operation each may perform on which bucket, is
sub-diagram 1b below (§7.2). In one line each:
- **the box** — the root store key and the `argondb-box` catalog subject: every
warehouse, and it WRITES. The committer is this principal. The operator is
inside every wall, and this guide states that rather than implying otherwise.
- **a tenant** — `argon-tenant-` at the store and
`argondb-tenant-` at the catalog: one bucket, one warehouse, one
`select`. This is what a tenant's application and its people use.
- **the cross-tenant reader** — `argon-reader` / `argondb-reader`: `GET`,
`HEAD` and `LIST` on every tenant bucket, never the box bucket, and it cannot
write anything anywhere.
- **a manufacturer group** — `hub-` at the store and
`argondb-manufacturer-` at the catalog: its own hub bucket only, never a
distributor's warehouse.
- **`argon_transform`** — the PostgreSQL role, the only writer of `derived.*`
in every database.
**What never exists** (the red box in the picture): a principal spanning two
tenants with write, and a crossing that is not a copy the box made.
#### 2.8 `all_tenants.__` — the reader's view
Under the cross-tenant reader, every same-named table across tenant databases
is served as ONE view: the union of every tenant's copy plus a `tenant` column
naming the tenant DATABASE (never the warehouse name — the two differ under a
`[lake.warehouse_overrides]`) (`docs/reserved-identifiers.md:39`). It is a
view, not a copy: each query expands to one scan per tenant of that tenant's
own Iceberg files, the `WHERE` is pushed into every scan, and branches run in
parallel.
Two properties the picture had room only to name. There is **no tenant list
anywhere** — the roster IS the estate, re-derived on every provisioner pass
from the same predicate the writer resolves a database to its warehouse with,
and a config key naming tenants was explicitly withdrawn
(`docs/2026-09-15-cross-tenant-read-design.md:207-218`). And **an unwalled
database is NAMED absent, never silently included or silently dropped**: the
roster carries it as `absent: — ` and every session banner prints
it (`:232-238`).
#### 2.9 Row-level security, and where it stops
RLS stays inside a database and separates roles, not companies
(`docs/2026-09-04-erp-spec.md:328-334`). Its blast radius changed with the
topology: a defective policy predicate used to be a cross-customer breach and
is now an over-permissive grant inside one company (`:344-351`).
The residual the picture draws is real and current. A table's LAKE COPY has no
row policies — PostgreSQL will not let a policy be attached to a foreign table
at all, and Iceberg has no row-policy concept
(`docs/lake-foreign-read.md:644-649`). Since 2026-09-12 the READ PLANE refuses
a scan of a protected table's lake copy rather than serving it, naming the
table, the role and the remedy (`:651-661`) — but that wall is on the Flight
door only. A reader holding the catalog URL and the store credentials reads
every row of every published table, row security or not (`:670-680`). Block 6
ruled that the Postgres role will travel into the read plane
[D-2026-09-11-c64ea4]; it has not traveled into the hub's in-process read yet,
and the picture lists that as one of its two residuals.
#### 2.10 What is box-granular, and what is per tenant
The granularities below are what a box with `[compute] per_tenant` OFF gives
you — today's shipped default. The ratified target topology moves the first
three rows to PER TENANT once an operator turns per-tenant compute on
[D-2026-09-20-0d5ba3] [D-2026-09-21-4107c5] [D-2026-09-23-5122a6].
| capability | granularity | why |
|---|---|---|
| OLTP point-in-time recovery | the box | one storage timeline per box |
| a rehearsal or upgrade branch | the box | a child timeline forks the whole cluster |
| lake freshness | the box | with the switch off every tenant is on the primary's one timeline, so one tenant's write volume gates every other's watermark |
| a D12 pin | **per scope** | a scope is a table set, and a tenant is a database's tables |
| lake time travel | **per table** | a retained snapshot plus `T__changes` replay |
| the tenant wall (bytes; and names under the catalog login) | **per database** | v1.5.1 |
(`docs/2026-09-15-cross-tenant-read-design.md:585-600`.) So "a tenant cannot be
rewound on its own" is true of the OLTP side and false of the lake side: a
scoped pin reads one tenant at any LSN in the retention window today, without
touching the timeline. Per-tenant timelines need a separate Postgres instance
per tenant, because PostgreSQL has no per-database write-ahead log to branch
(`:576-583`) — which is exactly what the topology ruling has since made the
standing design, and it is **built, default off**: each tenant database is its
own Postgres instance on its own timeline, sharing only the storage plane
[D-2026-09-20-0d5ba3] [D-2026-09-21-4107c5], under `[compute] per_tenant`
(`crates/argondb-ctl/src/tenant_compute.rs:105`), with the tail following every
tenant timeline as its own stream (`crates/argondb-tail/src/streams.rs`;
HISTORY#tail-multiplexes-tenant-timelines). The switch is the operator's, never
a system default [D-2026-09-23-5122a6]; with it off, every row above still
holds.
---
### 3. How it works
#### 3.1 One provisioning pass, start to finish
**Who runs a pass.** Two callers, one function. The box's cadence loop runs
`ensure_all` every `DEFAULT_INTERVAL_SECS = 30`
(`crates/argondb-ctl/src/lake_tenant.rs:155`, `:2303`), and
`argondb lake tenant provision` runs the same function on demand
(`:2575`). It matters that it is one function: on 2026-09-16 a hand `provision`
run raced the box's own cadence pass over the same 72 new databases and
produced 409s on the reader's catalog grants and a non-zero CLI exit while the
box's pass finished the estate (`:157-170`).
**The pass lock.** Every cluster pass now holds one SESSION-LEVEL PostgreSQL
advisory lock on the primary for its whole duration
(`crates/argondb-ctl/src/lake_tenant.rs:1458-1466`). The key is
`i64::from_be_bytes(b"argon_tw")` = `7021788497314804855` — the eight ASCII
bytes read big-endian, so the number is derivable by hand and is not a magic
literal (`:172-176`; registered at `docs/reserved-identifiers.md:6734`). Three
properties follow from choosing a session lock: a dead holder frees it without
a reaper, `pg_locks` joined to `pg_stat_activity` names the holder so a waiter
can say who it is waiting for, and the primary is the one coordinate both
parties already reach. A waiter waits `PASS_LOCK_WAIT_DEFAULT_SECS = 900` by
default (`:183`), overridable with `ARGONDB_TENANT_WALL_LOCK_WAIT_SECS`
(`:179`), and refuses on the deadline naming the holder and stating that
nothing was provisioned. Fifteen minutes is not arbitrary: an 80-tenant pass
under the wall was measured in the minutes on a loaded box (`:180-182`).
**The order, per principal, and why it is this order**
(`crates/argondb-ctl/src/lake_tenant.rs:34-50`):
1. **The credential is minted and written FIRST.** The store watches that file
and creates a bucket directory for every bucket it names, so the credential
is also the bucket's birth certificate — one owner of the store's layout.
2. **The box creates the bucket directory itself too**, on this process's
clock, so step 3 does not race the store's reload. Both paths are
`create_dir_all` and idempotent.
3. **Only then the warehouse**, because Lakekeeper VALIDATES a storage profile
by writing and reading a probe object in the bucket — into a bucket that
does not exist, that validation fails.
4. **Success is asserted from POSITIVE EVIDENCE** — the warehouse is read back
out of `GET /management/v1/warehouse` — never from the POST falling through,
which a re-provision answers `400` to.
**The warehouse profile carries the ROOT credential, not the tenant's**
(`:70-78`). Lakekeeper holds the storage credential for the whole warehouse and
uses it for its own probe writes and maintenance; a profile carrying a
bucket-scoped key would work today and break the moment Lakekeeper needed to
touch anything else. The tenant key is for the tenant's READER — DuckDB, Trino,
a raw `s3://` client — which is exactly the reader part C exists to wall,
because the catalog vends an ENDPOINT and never key material.
**Every row is a statement, and a refusal is per database.** `Outcome` has four
arms and nothing falls through silently: `AlreadyProvisioned`, `Provisioned`,
`BoxWarehouse`, `Refused(reason)`
(`crates/argondb-ctl/src/lake_tenant.rs:425-435`). One unprovisionable
tenant never stops the others — that is written into the function's error
contract (`:1451-1457`) and gated at both polarities by
`testing/box/tenant-wall-warehouse.sh` P9, which puts two legal databases and
one illegal name (`zz_bad_name`) into one pass and asserts two walled, one
named with its remedies, and a non-zero exit for the named one only
(`testing/box/tenant-wall-warehouse.sh:58-61`).
**Two principal kinds, one function.** `ensure_principals`
(`crates/argondb-ctl/src/lake_tenant.rs:2199`) is what a database and a
manufacturer group both go through; `ensure_all` is it over the cluster's
databases and `hub::provision` is it over the ownership table's manufacturers
(`:52-62`). The credential file's `database` field carries the principal's
STORE KEY — a database's own name, a manufacturer's prefixed one — so the two
kinds cannot collide in it, and `WarehouseMap::resolve_estate` refuses an
estate in which they would (`:64-69`).
#### 3.2 The two cluster-wide halves of the same pass
Both are keyed on DATABASES rather than on principals, which is why they live
in `ensure_all_locked` and not in `ensure_principals` — the hub also calls
`ensure_principals` over manufacturer groups, which have no `pg_database` row
to limit (`crates/argondb-ctl/src/lake_tenant.rs:1480-1484`).
**The OLTP half — `ensure_connection_limits`** (`:830`). The decision is pure
and unit-testable without a PostgreSQL (`plan_connection_limit`, `:755`), and
it is three rules in order (`:749-753`):
1. an explicit override is applied EXACTLY, whatever is there;
2. the default is applied only over `NO_LIMIT_SET` (`-1`, PostgreSQL's own
sentinel, `:600`);
3. everything else is kept as found — a limit an operator set by hand is never
lowered, and never raised either.
Three rungs resolve the wanted value, top wins: `--connection-limit =`,
then `[lake.tenant_connection_limit].` in `argon.toml`, then the default
(`ConnectionLimits::resolve`, `:651`). An override naming a database that is
not in the cluster is REPORTED rather than silently ignored
(`overrides_naming_no_database`, `:661-678`). A value outside PostgreSQL's
domain is refused by name at the boundary rather than handed to
`ALTER DATABASE` to refuse in its own words (`validate_connection_limit`,
`:687`), and the argument parser splits on the LAST `=` so a database name
holding `=` still parses (`parse_connection_limit_override`, `:706`).
`LimitOutcome` has four arms — `Set`, `Overridden`, `Kept`, `Failed` — and
again nothing falls through (`:730-744`).
**The reader half — `ensure_reader`** (`:1693`), run AFTER the tenants are
walled, so a tenant this pass minted is in the roster this pass writes
(`:1485-1491`). Its roster is `reader_plan` (`:1537`), which asks
`WarehouseMap::resolve_estate_partitioned` — the same predicate the provisioner
walls with, called rather than re-stated — and keeps every database resolving
to a NON-box warehouse, in database order (`:1541-1554`). Every database the
resolver refused is carried with its reason (`:1555-1562`). The store half is
one `ReaderCredential` in the same credentials file at version 2, its scope
rewritten whenever the roster moves, so the key FOLLOWS tenants with no restart
and no re-mint (the store re-reads the file by inode). The catalog half is one
`select` assignment per tenant warehouse, DIFFED against the `granted` record
the previous pass left in the file: a new tenant is granted on the pass that
walls it, a dropped tenant is revoked on the pass that finds it gone, and every
change is one log line (`:103-107`).
**No wall, no reader** (`:108-112`). With `[lake].warehouse_per_database` off,
every database resolves to the box warehouse, so a reader would resolve
nothing; the pass mints nothing and REMOVES a reader an earlier configuration
left behind. `reader_plan` returns `None` for that case and its doc comment is
explicit that `None` is a REFUSAL and not an empty roster (`:1529-1532`) —
there is no cross-tenant read without a wall, and the product says so rather
than handing out the lake under a read-only label.
**A reader failure is a row, not the end of the pass.** The tenants are already
walled; the reader is retried next cadence, and the failure is one logged line
(`:1490-1500`). Every outcome refreshes the roster cache, `Unchanged` included,
because a pass that PROVED the record current is as good a reason to restart
the window as a rewrite (`:1589-1594`).
#### 3.3 Which warehouse a table publishes into, and what happens when it is not there yet
The committer resolves a catalog handle PER TABLE, from that table's own
database (`crates/argondb-committer/src/catalog_set.rs:226-239`). When the
resolution fails — the commonest reason being that the database was created
seconds ago and its warehouse does not exist yet — the path **refuses and
defers by name. It never falls back to the box warehouse.** The refusal text is
one string, lifted out so it is testable without a catalog
(`unreachable_warehouse_message`, `:243-257`), and it says why:
```text
database "tenantc" publishes to lake warehouse "tenantc" (bucket "tenantc"); it is not
reachable. ArgonDB does NOT fall back to the box warehouse "argon" here: that would put
this database's published bytes behind the box credential, which is the tenant wall
inverted. If the database was just created, the box's tenant-wall pass provisions its
warehouse within one cadence interval — `argondb lake tenant list` names the state and
`argondb lake tenant provision` forces a pass.
```
The tail retries every cadence, so a deferral is a stable repeated log line and
not a one-shot message. Within one 30-second pass the wall exists and the next
retry publishes into the tenant's own bucket. **And the ordering is stronger
than the requirement:** provisioning happens on first DATABASE, not on first
publish, so in practice the wall is up before there is anything behind it —
that substitution is named on the spot rather than made silently
(`crates/argondb-ctl/src/lake_tenant.rs:22-32`).
#### 3.4 The store's one predicate
`wall::decide(creds, access_key, op, target)` is pure, total, and the only rule
in the store about who may touch which bucket
(`crates/argondb-lake-store/src/wall.rs:117-191`). It sits where it does for a
measured reason: `s3s` 0.14.1 calls `S3Access::check` ONCE per request, after
the signature is verified and after the route is resolved, before the
operation's input is deserialized — so one implementation covers roughly a
hundred operations including ones this store does not implement and ones a
future `s3s` adds. Guarding each method by hand would be about twenty edits
today and a silent hole on the next operation added
(`crates/argondb-lake-store/src/wall.rs:12-22`).
**The one thing the request path does not carry is a COPY's SOURCE bucket** —
`s3_path` is the destination. `s3s` generates a per-operation hook for that
case and calls it with the deserialized input, so `BucketWall::copy_object`
reads `copy_source` and applies the SAME predicate
(`:24-30`). A cross-bucket copy is how a wall that guards only the destination
is walked straight through.
The arms, in the order the predicate tests them:
- **no access key** → deny: `anonymous request refused: needs a signature
on this store` (`:126-130`).
- **`Principal::Unknown`** → deny, naming the key (`:133-135`).
- **`Principal::Root`** → allow, unconditionally (`:132`).
- **`Principal::Scoped { bucket, database }`** → the asked bucket must equal
the scoped bucket; a mismatch is denied with the key, its bucket, its
database and the bucket asked all named. At the store ROOT only
`ListBuckets` is allowed, and it is FILTERED to the caller's own bucket
(`:136-152`).
- **`Principal::Reader { buckets }`** → a bucket outside the roster is denied
by name, with `(the box bucket never is)` in the line; a bucket inside the
roster is allowed only for an operation in `READER_OPS`, and every other
operation is denied naming the allowed set (`:161-189`).
`READER_OPS` is CLOSED and holds exactly seven names — `GetObject`,
`HeadObject`, `ListObjectsV2`, `ListObjects`, `HeadBucket`,
`GetBucketLocation`, `GetObjectAttributes` (`:78-86`). Anything not in the
list is refused by name, so an operation `s3s` adds later does not arrive
allowed. `CopyObject` is deliberately absent: a copy WRITES its destination,
and the reader may write nothing (`:71-77`).
**Why `ListBuckets` is filtered rather than refused** (`:32-40`): refusing it
outright would be a wall too, and it would break every S3 client that probes
the bucket list before addressing a bucket. Filtering is the answer that is
both a wall and usable. Every OTHER root-level operation is refused by name — a
closed default.
#### 3.5 The catalog's half
The catalog half is one `select` assignment per (subject, warehouse), written
through `POST /management/v1/permissions/warehouse/{warehouse_id}/assignments`
with `{"writes":[{"type":"select","user":"oidc~"}],"deletes":[]}` —
read out of Lakekeeper's own OpenAPI document rather than guessed, and
exercised end to end before it was built
(`crates/argondb-ctl/src/lake_grant.rs:13-20`). `select` is READ access to the
warehouse: the tenant lists and reads its own tables and cannot write them,
which is the shape the wall is for, since the committer is the only writer and
writes as the box (`:35-41`). Revoke is the same endpoint with the subject
moved from `writes` to `deletes` (`:22-25`).
Four measured behaviors a developer will meet:
1. **A warehouse the caller may not see answers 404, not 403.** The name is not
disclosed (`docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:798-800`,
and the C7d/C7e rows of the measurement at `:833-834`).
2. **`lakekeeper migrate` with the backend set creates the OpenFGA store and
writes the whole authorization model itself** — no provisioning step of ours
(`:795-798`).
3. **The box mints its own tokens.** Lakekeeper never calls an identity
provider; it fetches `/.well-known/openid-configuration`, then the
JWKS, then verifies RS256 signatures. The box holds the signing key and
serves those two documents, so the wall costs one new service (the policy
engine) and no identity provider (`:813-822`, `:846-849`).
4. **Revoking a grant does not invalidate a token.** It removes the assignment,
so the principal is refused from the next request onward; the JWT itself is
valid until it expires and the box keeps no revocation list. With the grant
gone the token authenticates and authorizes nothing, and `--ttl` bounds the
other half (`crates/argondb-ctl/src/lake_grant.rs:281-287`).
And one refusal that is a design statement rather than a validation: `lake
grant` REFUSES a database that resolves to the BOX warehouse, because the box
warehouse holds every other database's tables as well and "granting" there
would hand the caller the whole lake while reading like a per-tenant permission
(`:45-68`).
#### 3.6 A read session that resolves every tenant
`pin` is the one home, and converting it converted four surfaces at once: the
transform runner, the hub, the MCP pin/redeem path and the Flight read plane;
the FDW reads through Flight and follows
(`docs/2026-09-15-cross-tenant-read-design.md:356-360`). What changed is the
type: `QueryOpts.lake` became the principal's `CatalogSet` — the plural handle
the WRITER already had — so a session registers every committer table across
every warehouse its principal may read, each table's provider built from the
catalog handle its own database resolves to (`:290-296`).
The principal decides the reach (`:311-317`): the box resolves every warehouse;
a tenant resolves exactly its own; the cross-tenant reader resolves every
TENANT warehouse — `CatalogSet::warehouses` minus the box's, with absent
tenants named.
**Read-only means no writer in the session** (`:345-354`). `hub run` is one
process holding TWO principals: the reader for reads and the box for writes,
with the run banner naming both. Splitting them is the point.
**The consistency claim, stated exactly** (`:324-343`). A pin is
`(branch, LSN, scope, …)` and the LSN is a position in the cluster's ONE
write-ahead log, which every tenant database shares — so one LSN names one
instant for every table in every warehouse. Per table the guarantee is
unchanged. What is NOT claimed is cross-warehouse atomicity of a cut: a cut
spanning two tenants is one transaction PER WAREHOUSE, so two warehouses'
watermarks may advance at different moments. A cross-tenant pin therefore lands
at the newest LSN every scoped table in every tenant warehouse serves, and the
session discloses per warehouse which coordinate it resolved.
**This claim's "one write-ahead log" premise is the shipped default, not the
standing topology.** With `[compute] per_tenant` on, the ruled target gives
each tenant database its OWN write-ahead log on its OWN timeline
[D-2026-09-20-0d5ba3] [D-2026-09-21-4107c5]; one LSN naming one instant for
every table in every warehouse holds today because that switch is off by
default and is the operator's to flip [D-2026-09-23-5122a6].
#### 3.7 The view, and what it costs
`all_tenants.__` is composed at pin time from the cached roster
and the per-warehouse table listing. The name is the flat key minus its
database, grouped STRUCTURALLY by `(schema, table)` rather than re-parsed out
of the name — the tenant-relative key is not injective
(`crates/argondb-query/src/all_tenants.rs:152-159`). Each branch is
`SELECT '' AS "tenant", FROM ""`, `UNION ALL`ed in tenant order, columns
unioned BY NAME in first-seen order (`docs/reserved-identifiers.md:39`).
**Pushdown is the engine's and is not re-implemented.** A `WHERE` over the view
is pushed by DataFusion through the `Union` into every branch and on into each
branch's `IcebergTableScan`, where manifests, files and row groups are pruned
against that tenant's own statistics. Because `tenant` is a per-branch
CONSTANT, `WHERE tenant = 'x'` simplifies to `false` in every other branch and
the optimizer drops those branches before a file is opened — that is how
"subsetting is the query's `WHERE`" costs nothing over a named set
(`crates/argondb-query/src/all_tenants.rs:34-45`).
**Drift has a voice twice** (`:47-58`). A missing column is `NULL AS "col"` on
that tenant's projection line beside its `'tenant' AS tenant`, so `EXPLAIN`
names the tenant and the column together; and every drifted view is listed in
the served warning rendered at the mint and on every result the session serves,
so a client that never reads a plan still sees it. A tenant with NO copy
contributes no branch and is named the same way. A column whose TYPE differs is
named per tenant with both types: the engine's `UNION` coercion widens it when
it can (`Int32` with `Float64` reads as `Float64`) and the view is REFUSED BY
NAME when it cannot (`Int32` with `Boolean`) — never silently narrowed. A copy
that already carries a column named `tenant` is refused the same way, because
the projected literal would shadow it (`:60-64`).
**The measured cost** (`docs/lake-foreign-read.md:386-402`, re-measured
2026-09-19 on `testing/box/all-tenants-views.sh` P5):
| estate | whole pin | catalog listing | table metadata |
|---|---|---|---|
| 8 warehouses, 8 lake tables | 2307 ms | 1702 ms | 261 ms |
| 80 warehouses, 80 lake tables | 22575 ms | 16123 ms | 2974 ms |
The split is the finding: roughly 70 % of a pin is the catalog WALK — one
`namespace_exists` plus a `list_tables` and a `list_namespaces` at each of the
branch, database and schema levels, so seven round trips per warehouse against
one metadata load per table (about 29 ms per round trip at 80 warehouses).
Narrowing the table set buys the smaller half; narrowing the walk buys the
larger one. **Earlier figures on this line — 17122 ms at 8 and 152654 ms at 80
— DO NOT REPRODUCE**, and nothing in between was a deliberate optimization of
this path; they are left in that document as the record and are not a baseline
(`:404-409`). A reader sizing a deployment should re-measure.
A scoped pin (`argondb-query --scope`) walks only the namespaces its names
could live in — three round trips instead of seven for a warehouse holding no
database the scope names. **In a cross-tenant estate it prunes nothing, and
that is measured rather than assumed**: the scope a cross-tenant read can name
is `all_tenants.__`, which names no database, so every database
level is admitted. The lever bites where a scope names FEWER DATABASES than the
roster has (`docs/lake-foreign-read.md:415-428`).
#### 3.8 The three spellings of one identity
Four coordinates, three SQL name parts. The shape was ruled in three blocks:
full nesting [D-2026-09-11-702e69], the primary regularized
[D-2026-09-11-118b2e], `__` kept as the separator [D-2026-09-11-f73865]
(`crates/argondb-ctl/src/lake_address.rs:19-21`). So `postgres.public.orders`
on branch `main` has a catalog identifier of namespace
`["br_main","postgres","public"]` with table `orders`, an engine address of
`lake."br_main.postgres.public".orders`, a flat lake key of
`postgres__public__orders`, and a dotted log label of
`br_main.postgres.public.orders`.
Those are four different jobs and no one of them can stand in for another
(`:47-78`). `address` is what a DuckDB user TYPES, and its namespace is one
quoted identifier because a nested namespace is readable no other way.
`lake-table` is the FLAT key the tail's `tables` action, the FDW mount and a
pin session use. `leaf` is the bare relation name a caller talking to the REST
catalog directly needs. `label` is what a HUMAN and a LOG read — no catalog
alias, no quotes — and it exists because `address`'s quoting makes it useless
as a log matcher.
**Why this is a product subcommand and not a library function**
(`:4-15`): a shell gate, a shipped starter and a customer doc cannot call
`lake_table_name_for`. The census measured what that costs — 452 lines in 133
files spell the lake address, and three of the consumer groups (12 lines of
shipped starters, 132 lines in 57 gate files, 148 lines in 35 docs) are shell,
SQL and prose. `argondb lake address` is the door they call, and
`testing/lib/lake-table-name.sh` is its first caller.
**What has no database coordinate stays at the branch level and in the box
warehouse:** `_argon_gaps`, `_argon_timeline`, and the lake writer's own
`dv_` territory
(`docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:354-370`;
`docs/lake-foreign-read.md:186-188`). A tenant key does not reach them either.
**A catalog namespace is a NAMING hierarchy, not a directory hierarchy** —
measured, and it is the reason part C exists. Every namespace the probe created
reported the SAME storage location, and a table's location is the warehouse
prefix plus the table's own UUID; the warehouse storage profile carries
`"storage-layout":null`. A credential scoped to a namespace-shaped prefix would
open nothing, because no such prefix exists
(`docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:168-189`).
One thing stays **UNVERIFIED** in that measurement and this guide repeats it
rather than rounding it: whether 0.13.x offers a non-default `storage-layout`
that nests objects by namespace — the field exists in the profile and was not
probed (`:187-189`).
---
### 4. How to use it
Thirteen worked examples. Each states its label and, where it is FROM SOURCE, the
`path:line` the command, flag, message or sample output is quoted from, plus
the gate that proves it on a box.
#### 4.0 `argondb tenant create` — the whole tenant, in one ordered command — FROM SOURCE
**This is the command to reach for when you are creating a COMPANY**, rather
than just a database. §4.1 below is still exactly right and still supported —
it creates the database and lets the wall pass find it — but it leaves the
tenant's timeline, its compute row, its login role, its uniform schema, its
identity record, its application process and its MCP token to you.
`argondb tenant create` runs all of those in the order the ERP spec §13.4
forces — the project owner's ruling on who creates a company and in what order
[D-2026-09-21-e781f5] — and it is safe to re-run:
```bash
argondb tenant create acme \
--display-name "Acme" \
--legal-name "Acme Holdings LLC" \
--system distributors \
--identity "tax_id=12-3456789" \
--identity "contact=ops@acme.example" \
--data-dir "$DATA" --port 5433
```
| flag | what it is | required |
|---|---|---|
| `` | the tenant's stable, never-reused key — and, on a box, its database name. Lowercase ASCII letters, digits and underscores | yes |
| `--display-name` | §13.1's `display_name`: the company's name as people write it | no — see below |
| `--legal-name` | §13.1's `legal_name` | no — same |
| `--system` | which tenancy the tenant belongs to (`distributors`, `manufacturers`) — a tenancy is one multi-tenant system in the Estate, and the CLI's word for it is `system` | no (`distributors`) |
| `--identity KEY=VALUE` | the rest of the company's identity. Repeatable. The key must be one of `address`, `tax_id`, `contact` | no |
| `--sidecar-command COMMAND` | the command that runs the company's APPLICATION as its own process (step 8) | no — omitted means the company runs no application |
| `--sidecar-uid UID` | the unix uid that process runs under | no — defaults to the `argon-t-` user |
| `--direct` | also print the DIRECT connection string; the POOLED one is what a tenant is handed by default | no |
| `--data-dir`, `--port` | the box, resolved by the same precedence every other subcommand uses | no |
(`crates/argondb-ctl/src/main.rs:611-670`.)
**The two name flags are OPTIONAL, and a missing name is written EMPTY, never
invented and never refused.** This changed in v1.5.5: an earlier build refused
a create with no name. A company whose human has not answered yet is written
with its key and empty names, the transcript names the flags that were not
supplied, and the tenant still reaches `live`
(`crates/argondb-ctl/src/tenant_create.rs:1862-1873`). What is still refused,
before anything is created, is an `--identity` key the identity table has no
column for — `country=US`, for example — because an unknown key cannot be
stored and dropping it quietly is how an address goes missing
(`IDENTITY_EXTRA_FIELDS`, `crates/argondb-ctl/src/tenant_create.rs:1321`;
the refusal, `:424-438`). `tenant create --help` marks both name flags OPTIONAL
(`crates/argondb-ctl/src/main.rs:624-634`), as the code does.
**The eight steps, and what each one is.** The run prints them as it goes; the
one place their order is written down is `STEPS`
(`crates/argondb-ctl/src/tenant_create.rs:206-215`):
| # | step | what exists afterwards |
|---|---|---|
| 1 | its **timeline** | a branch of the box timeline, recorded in `/tenants/acme/identity.json` |
| 2 | its **compute row** | a row in the per-tenant compute table — its own slot in the tenant port band and its own directory |
| 3 | its **database** | `acme` in `pg_database` |
| 4 | its **role and grants** | `acme_owner` in `pg_roles`, granted on the database and on `public` |
| 5 | its **uniform schema** | §13.1's `identity` table, applied through the versioned path §4.9 drives across every tenant, with its version recorded in `acme`'s own `argon.schema_version` |
| 6 | **the wall pass** | exactly what §4.1 does — warehouse, bucket, scoped credential, catalog grant, connection limit |
| 7 | its **identity row** | your answers in `acme`'s `identity` table, WRITTEN THROUGH to its system's Identities table in the same call |
| 8 | its **sidecar** | the company's application process, under its own uid and cgroup — or nothing at all, when no `--sidecar-command` was given |
After the eight steps the run mints the company's own **MCP token** and writes
it to a `0600` file, `/tenants/acme/mcp-token` (§4.11). A failure there is
named and does not undo the tenant; `argondb tenant mcp-token acme` is the
remedy (`crates/argondb-ctl/src/tenant_create.rs:2950-2975`).
**The identity table has one door.** Step 7 writes the tenant's row and then its
system's Identities row with the tenant commit's LSN as `source_lsn`; a trigger
refuses a row write to `identity` from any role but the tenant API's own, and
`tenant_key` is not editable at all. A rename goes through
`argondb tenant identity acme --display-name …`, the same door
(`crates/argondb-ctl/src/main.rs:725-744`).
**RE-RUNNING IT IS SAFE, AND THAT IS THE POINT.** Every step probes before it
acts, so a second run creates nothing and says so
(`crates/argondb-ctl/src/tenant_create.rs:2244`, `:2938-2948`):
```text
[argondb] step 1/8 timeline — already present (§13.4 step 5); not applied
...
[argondb] tenant create: acme is live — 0 step(s) applied, 8 already present. Journal: /tenants/acme/create-journal.json
[argondb] tenant create: NO-OP — this run created nothing (every step probed present). §13.4 idempotence.
```
**AND A RUN THAT DIES HALFWAY IS RESUMED BY RE-RUNNING IT.** Progress is
recorded in a step journal at `/tenants/acme/create-journal.json`, and a
tenant that is half-built says so in its own `status` column
(`provisioning`, not `live`). Re-run the same command and it continues where
it stopped:
```text
[argondb] tenant create: RESUMING at step 4 (roles) — the journal names step 3 (database) as the last completed step
```
**WHAT IT WILL NOT DO: adopt.** If a database of that name already exists and
this command did not create it, it REFUSES and points you at the provisioner.
That is not an obstacle — the wall pass adopts it within one cadence, which is
§4.1's whole story:
```text
refusing: a database of this name already exists and this sequence did not create it
(there is no step journal beside it). `argondb tenant create` CREATES; it does not adopt.
```
**What this box does NOT give you.** The run's own banner says it: on one
development box there is one safekeeper, no WAL backup and no zone diversity.
A quorum of one is not redundancy. Every §13.4 step that has no meaning on one
box — the tenant's EC2 instance, its subnet and security group, its volume, its
elastic address and the three-zone safekeeper member set — is printed as a
SKIPPED step with the one-box substitute that stands in for it, so nothing is
silently absent (`skipped_cloud_steps`,
`crates/argondb-ctl/src/tenant_create.rs:318`). The Identities write and the
application sidecar were on that list before v1.5.5; both are now real steps
(7 and 8 above).
**Where the estate control plane, the estate directory and Identities
actually run — never co-resident with a tenant.** [D-2026-09-21-8b9d22] ruled
this on the model of estate instances, following the storage plane's own
practices: at least two estate instances across two availability zones (three
is the natural number, since the safekeeper quorum already needs three) run
the provisioning API (one leader, the rest forwarding rather than serving),
the Identities projection, and the multiplexed tail's own lease — never on a
tenant's own instance, because a control plane sharing a tenant's instance
dies with that tenant and cannot be the thing that recreates it. It does not
scale to zero: the API must answer a `tenant create` call that can by
definition arrive when nothing else is awake, so the estate control plane is
always-resident, like the storage plane, not stop-when-idle like a tenant
compute. On one development box every one of those coordinates collapses to a
single process and a schema in the box's own primary (`argon_estate`) — which
is why the bar for "one-box complete" now also requires showing that control
plane's leader and its forwarding running as two processes on the box, not
just the wall [D-2026-09-23-21780c].
Gate: `testing/box/tenant-create-sequence.sh` (both polarities, on ms-01). Its
v1.5.5 run measured a created tenant reaching its system's Identities —
`all_tenants.public__identity`, read through the cross-tenant reader — in 59
seconds with no administrator act (`docs/releases/v1.5.5.md` §15).
**Fixed: the uniform schema's target is a step it defines.** The v1.5.6
release shipped `UNIFORM_SCHEMA_VERSION` at `2` with a sequence that defined
only version `1`, so a fresh `tenant create` (step 5) and a default `argondb
estate migrate` (§4.9) both stopped at `the uniform schema has no step
defining version 2`. The sequence now carries step 2 — the identity table's one
door (`uniform_schema_steps` in `crates/argondb-ctl/src/tenant_create.rs`) — a
unit test holds the target to the last step, and
`testing/box/uniform-schema-fresh-create-and-migrate-from-v1.sh` runs both
commands on a box. Post-mortem:
`docs/postmortems/2026-09-28-uniform-schema-version-declared-without-its-step.md`.
#### 4.1 Create a tenant database and watch the provisioner wall it — FROM SOURCE
Nothing about creating a tenant is special. Create the database; the pass does
the rest within one cadence.
```bash
# 1. the database. Nothing else is special about it.
createdb -p 5433 -h localhost -U cloud_admin tenanta
# (or: psql -c 'CREATE DATABASE tenanta')
# 2. wait for the box's own pass (<= 30 s), or force one now:
argondb lake tenant provision --data-dir "$DATA" --port 5433
# 3. read the estate back
argondb lake tenant list --data-dir "$DATA" --port 5433
```
The listing's shape is the format string at
`crates/argondb-ctl/src/lake_tenant.rs:2384-2392` — seven columns on a box with
a catalog login, six without (the `CATALOG GRANTS` column is ABSENT rather than
a row of "none", because "none" would read as "this box does not have the
feature"; `:2377-2382`). A worked listing, quoted from the customer recipe at
`docs/lake-foreign-read.md:141-145`:
```text
$ argondb lake tenant list
DATABASE WAREHOUSE BUCKET ACCESS KEY STATE
postgres argon argon (the box credential) primary — the box warehouse
shop shop shop argon-tenant-shop walled
tenantb tenantb tenantb argon-tenant-tenantb walled
```
Every STATE is a statement with a producer named
(`crates/argondb-ctl/src/lake_tenant.rs:2411-2424`):
| `(credential, warehouse)` | STATE |
|---|---|
| both present | `walled` |
| neither | ``not yet — the provisioner walls it within 30s, or run `argondb lake tenant provision` `` |
| credential only | `credential minted, WAREHOUSE MISSING — the last pass could not reach …` |
| the resolver refused the name | `REFUSED — ` |
Then the per-database estate, which is a different provisioner path with the
same cadence. Check it rather than assume it
(`docs/upc-hub-recipe.md:453-462`):
```bash
psql -d tenanta -c "SELECT to_regnamespace('derived') IS NOT NULL
AND to_regnamespace('mfr') IS NOT NULL
AND to_regclass('derived.transform_runs') IS NOT NULL
AND to_regclass('argon.view_state') IS NOT NULL"
```
That is exactly the probe `transform_estate::is_provisioned` runs
(`crates/argondb-ctl/src/transform_estate.rs:127-147`), and a unit test asserts
the probe asks for precisely what `deploy/sql/provision-transform-role.sql`
creates, so the two cannot drift (`:298-317`). Note the cluster/database split
that made this gap invisible once: `CREATE ROLE argon_transform LOGIN` is a
CLUSTER object, visible from every database, so "does the role exist?" asked
alone returns a false CLEAN (`:19-26`).
**Gate:** `testing/box/tenant-wall-warehouse.sh` P1, P3, P4 and P7
(`testing/box/tenant-wall-warehouse.sh:19-26`, `:41-57`) — the credential, the
bucket directory, the warehouse, a distinct key per tenant, a REFUSAL for the
primary, and which bucket the published parquet actually lands in, every leg
re-run after a box restart.
#### 4.2 The three spellings of one identity — FROM SOURCE
Ask the box; do not assemble it. The examples below are quoted verbatim from
the module's own doc comment, `crates/argondb-ctl/src/lake_address.rs:25-36`
and `:105-110`:
```text
$ argondb lake address postgres.public.orders
lake."br_main.postgres.public".orders
$ argondb lake address --field namespace postgres.public.orders
br_main.postgres.public
$ argondb lake address --field lake-table postgres.public.orders
postgres__public__orders
$ argondb lake address --field qualified postgres.public.orders
"public"."orders"
$ argondb lake address --field label postgres.public.orders
br_main.postgres.public.orders
$ argondb lake address --changes postgres.public.orders
lake."br_main.postgres.public".orders__changes
$ argondb lake address --changes --field lake-table postgres.public.orders
postgres__public__orders__changes
```
`--branch` moves the branch coordinate, and `--format json` answers a machine
(`crates/argondb-ctl/src/lake_address.rs:278-281` for the two formats;
`docs/lake-foreign-read.md:462-466` for the branch form):
```text
$ argondb lake address --branch dev shop.sales.orders
lake."dev.shop.sales".orders
```
Which field you want, by consumer: `address` for a DuckDB user and for a doc;
`namespace` for a starter's `LAKE_NAMESPACE` constant; `leaf` for a caller
talking to the REST catalog directly; `lake-table` for a pin scope, the FDW
mount and the tail's `tables` action; `label` for a log matcher; `qualified`
for the tail's own ` left the published set` lines; `bucket` for "which
bucket do this table's files land in"
(`crates/argondb-ctl/src/lake_address.rs:156-196`;
`testing/box/tenant-wall-warehouse.sh:53`).
**The failure you will hit if you assemble it**
(`docs/lake-foreign-read.md:453-456`):
```text
D SELECT count(*) FROM lake.br_main.postgres.public.orders;
Parser Error: NameListToString NOT IMPLEMENTED
```
Four name parts, and SQL has three. The quotes are not optional.
#### 4.3 Hand an analyst a tenant token, and see what it can and cannot list — FROM SOURCE
```bash
# mint + grant in one call; the TOKEN is the only thing on stdout
TOKEN="$(argondb lake grant tenanta --data-dir "$DATA" --ttl 30)"
# the clause a reader pastes, minted and granted through the same path
argondb lake attach-clause --database tenanta --data-dir "$DATA"
# take it back
argondb lake revoke tenanta --data-dir "$DATA"
```
The lines on stderr are the format strings at
`crates/argondb-ctl/src/lake_grant.rs:256-275`:
```text
[argondb] granted `select` on warehouse tenanta () to oidc~argondb-tenant-tenanta, valid 30 day(s).
[argondb] the analyst reads it with:
ATTACH 'tenanta' AS lake (TYPE iceberg, ENDPOINT '/catalog', AUTHORIZATION_TYPE 'oauth2', TOKEN '');
[argondb] revoke it with: argondb lake revoke tenanta
```
**What that token can do, measured** (the C-series rows at
`docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:826-838`):
```text
C4a/C4b tenant principals BEFORE any grant HTTP 403 / HTTP 403
C6a/C6b grant select on one warehouse each HTTP 204 / HTTP 204
C7a tenant-a lists warehouses HTTP 200 -> tenanta ONLY
C7b tenant-b lists warehouses HTTP 200 -> tenantb ONLY
C7d tenant-a reads tenantB config HTTP 404
C7e tenant-b reads tenantA config HTTP 404
C7f/C7g committer reads BOTH configs (the control) HTTP 200 / HTTP 200
C7h ANONYMOUS reads tenantA config HTTP 401
C8a FORGED token, right shape wrong signer HTTP 401
```
Read the 404s: a warehouse this subject may not see is not merely refused, its
NAME is not disclosed. And read the negative controls as controls — each is
only one because its positive twin passed in the same run on the same box.
**The refusal you will meet if you point it at the primary**
(`crates/argondb-ctl/src/lake_grant.rs:59-67`):
```text
database "postgres" resolves to the BOX warehouse "argon", which holds every other
database's tables as well — granting there would hand the caller the whole lake while
reading like a per-tenant permission. Turn on `[lake] warehouse_per_database` (it is on
by default) and let the provisioner wall this database first: `argondb lake tenant provision`.
```
**Revoke does not invalidate the token** — it removes the assignment, so the
subject is refused from the next request onward while the JWT stays valid until
it expires; `--ttl` is what bounds that half
(`crates/argondb-ctl/src/lake_grant.rs:281-287`).
#### 4.4 Read every tenant at once under the reader, and see an unwalled database named — FROM SOURCE
```bash
argondb lake reader credential --data-dir "$DATA" # the roster, the grants, a token
eval "$(argondb lake reader credential --data-dir "$DATA" --env)" # exports ARGONDB_LAKE_WAREHOUSES
argondb-query "SELECT tenant, count(*) FROM all_tenants.public__orders GROUP BY tenant ORDER BY tenant"
```
The credential surface, quoted from `docs/lake-foreign-read.md:228-241`:
```text
$ argondb lake reader credential
principal cross-tenant reader (read-only over every tenant warehouse, never the box warehouse)
access-key argon-reader
secret-key <48 hex characters>
endpoint http://127.0.0.1:9011
region local
catalog http://127.0.0.1:8181/catalog
warehouses shop, tenantb
buckets shop, tenantb
granted shop, tenantb
pass-unix 1789514071
token-ttl 30 day(s)
token
```
And the session, quoted from `docs/lake-foreign-read.md:322-336` — note that
the banner, the census line and the drift note all arrive before the rows:
```text
$ eval "$(argondb lake reader credential --env)"
$ argondb-query "SELECT tenant, count(*) FROM all_tenants.public__orders GROUP BY tenant ORDER BY tenant"
-- pinned at 0/1C2A3F0 (min watermark 0/1C2A3F0)
-- note: this session resolves 4 warehouses — tenanta, tenantb, tenantc, tenantd — …
-- pin built in 412 ms over 4 warehouse(s), 5 lake table(s)
-- all_tenants: 2 view(s) served over 4 tenant(s) [tenanta, tenantb, tenantc, tenantd]: all_tenants."public__orders", all_tenants."public__other"
-- note: 2 cross-tenant view(s) DRIFTED — a row from a named tenant is unioned by column name with NULL for what that tenant lacks, and a tenant with no copy contributes no rows: all_tenants."public__orders" — tenant tenantc lacks column(s) "v" (served as NULL); tenant(s) tenantd have no copy of this table (no branch) | all_tenants."public__other" — tenant(s) tenanta, tenantb, tenantc have no copy of this table (no branch)
+---------+----------+
| tenant | count(*) |
+---------+----------+
| tenanta | 3000 |
| tenantb | 3000 |
| tenantc | 50 |
+---------+----------+
```
**An unwalled database is ABSENT BY NAME, in the list line and in the roster.**
The `lake tenant list` reader line is built at
`crates/argondb-ctl/src/lake_tenant.rs:2484-2498` and renders the absent
entries as ` ()`:
```text
cross-tenant reader: argon-reader covers [shop, tenantb] catalog grants [shop, tenantb] absent [zz_bad_name ()]
```
The other two states of that line are statements too
(`:2469-2476`):
```text
cross-tenant reader: none — [lake].warehouse_per_database is off (no wall, no reader)
cross-tenant reader: not yet minted — the provisioner mints it on the pass that walls the first non-primary database (within 30s), or run `argondb lake tenant provision`
```
**Subsetting is the query's `WHERE`, and it is free.** `WHERE tenant IN
('tenanta','tenantb')` folds to `false` in every other branch and those
branches are dropped before a file is opened
(`crates/argondb-query/src/all_tenants.rs:40-43`). There is no named tenant set
to configure, and the config key that would have been one was withdrawn
(`docs/2026-09-15-cross-tenant-read-design.md:214-218`).
**The primary is not in the view.** The session never resolves the box
warehouse, so `postgres__public__house` answers "table not found", and naming
the box warehouse in the roster by hand is refused by the catalog under the
reader's login with the session saying which warehouse refused
(`docs/lake-foreign-read.md:277-281`).
**Gates:** `testing/box/cross-tenant-reader.sh` (8 phases — the store wall, the
catalog, a session, a tenant joining and leaving the roster, a restart;
`testing/box/cross-tenant-reader.sh:17-57`) and
`testing/box/all-tenants-views.sh` (8 phases — the view, pushdown read off
`EXPLAIN`, zero bytes written, the cost measurement, a memory bound, a DuckDB
reader; `testing/box/all-tenants-views.sh:18-80`).
#### 4.5 Set, override and read back a connection limit — FROM SOURCE
```bash
# the default (20) is applied only where datconnlimit = -1
argondb lake tenant provision --data-dir "$DATA" --port 5433
# an explicit override, applied EXACTLY, on this pass and every later one
argondb lake tenant provision --data-dir "$DATA" --port 5433 --connection-limit tenantb=50
```
or, durably, in `argon.toml`:
```toml
[lake.tenant_connection_limit]
tenantb = 50
```
The CLI rung wins over the file rung, and both win over the default
(`crates/argondb-ctl/src/lake_tenant.rs:648-659`). Read it back on the
operator surface — `lake tenant list` carries a `CONN LIMIT` column precisely
so an unset or hand-raised limit is visible outside the box log
(`:2366-2370`), and the primary's cell reads `unlimited`
(`testing/box/tenant-wall-warehouse.sh:633-639`).
The three rules, and both polarities of each, are what
`testing/box/tenant-wall-warehouse.sh` P10 proves
(`testing/box/tenant-wall-warehouse.sh:62-74`): 20 held sessions on tenant A as
a NON-superuser role and the 21st refused NAMING THE DATABASE; tenant B still
connecting (the positive twin); a superuser still connecting (PostgreSQL's
documented exemption); a CLI override winning and printed with the sibling
untouched; a hand-set value surviving a plain pass, neither lowered nor raised;
an `argon.toml` override winning, printed, and printed again as "in force" on
the next pass; a malformed value and an override naming no database refused by
name with nothing applied; and the UPGRADE shape — a tenant with no limit
getting 20 on the next pass, with the pass saying so.
Two edges worth knowing before you rely on the number. It is PostgreSQL's own
limit, so **superusers are exempt** and **the count is of connections to that
DATABASE, not of the role** (`crates/argondb-ctl/src/lake_tenant.rs:136-141`).
And `default = -1` is a legal setting meaning "set no default" — an operator
who wants PostgreSQL's unlimited behavior back says so with `-1`
(`:626-630`).
#### 4.6 Turn the catalog login on, and read a refused cross-tenant access — FROM SOURCE
```toml
# argon.toml — but note the DEFAULT already resolves this way on a
# multi-database box, and an explicit value always wins
[lake]
catalog_authz = "openfga"
```
The default was ruled on 2026-09-15: `openfga` on a box holding more than one
database, `off` on a single-database box, explicit always wins
(`docs/2026-09-15-cross-tenant-read-design.md:502-504`). The box says which one
it is running, and says when the count has crossed since boot — one predicate,
called by both the cadence loop and the list surface so they cannot disagree
(`crates/argondb-ctl/src/lake_tenant.rs:2344-2356`, `:2460-2465`):
```text
catalog authz: (as decided at boot)
<…and, when the database count has crossed 1 since boot, that the change takes effect at the next boot>
```
With the login OFF, the served refusal text states the position exactly —
`authenticates nobody and authorizes everybody`
(`crates/argondb-ctl/src/lake_grant.rs:356`, quoted at
`docs/2026-09-15-cross-tenant-read-design.md:520-521`). Data is walled;
metadata is not.
**What a refused cross-tenant read prints, at each of the two doors.** At the
store, the exact `Deny` strings are the format strings at
`crates/argondb-lake-store/src/wall.rs:148-151`, `:166-170`, `:173-177` and
`:182-186`:
```text
access key "argon-tenant-tenanta" is scoped to bucket "tenanta" (database "tenanta") and GetObject addresses bucket "tenantb"
access key "argon-reader" is the cross-tenant READER over ["tenanta", "tenantb"] and GetObject addresses bucket "argon", which is not a tenant bucket in its roster (the box bucket never is)
access key "argon-reader" is the cross-tenant READER and is READ-ONLY: PutObject on bucket "tenanta" is refused (allowed: GetObject, HeadObject, ListObjectsV2, ListObjects, HeadBucket, GetBucketLocation, GetObjectAttributes)
access key "argon-reader" is the cross-tenant READER (read-only over 2 tenant bucket(s)) and CreateBucket addresses the whole store
```
Each surfaces to the client as `403 AccessDenied` and is logged by the store
with the key, the operation and both buckets named
(`docs/lake-foreign-read.md:190-194`, `:243-249`). At the catalog, the refusal
is a `404` and does not disclose the name
(`docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:798-800`).
**Gate:** `testing/box/tenant-wall-warehouse.sh` P5 runs the whole matrix with
both polarities and a restart afterwards — key A on bucket A 200, key A on
bucket B 403 on list AND on get, the symmetric pair, key A on the BOX bucket
403, root on every bucket 200 as the control, `ListBuckets` filtered per key,
anonymous 403 (`testing/box/tenant-wall-warehouse.sh:27-40`).
#### 4.7 Stand up a manufacturer, and see the party rule bite — FROM SOURCE
A manufacturer is a member database beside the distributors; the SCHEMA is what
keeps the parties apart (`docs/upc-hub-recipe.md:430-441`).
```bash
createdb mfracme # the box's pass walls it exactly as it does a distributor
psql -d mfracme -c "SELECT to_regnamespace('mfr') IS NOT NULL" # estate; you do not create it
```
```sql
CREATE TABLE mfr.price_list(
upc text PRIMARY KEY,
list_price double precision,
effective_from date,
effective_to date,
manufacturer_id text);
COMMENT ON TABLE mfr.price_list IS 'argon:publish';
CREATE TABLE mfr.share_grant(
flow text NOT NULL,
manufacturer_id text NOT NULL, -- a recipient, or '*'
contract_version int NOT NULL,
granted_at timestamptz NOT NULL DEFAULT now(),
revoked_at timestamptz, -- NULL = in force
PRIMARY KEY (flow, manufacturer_id, contract_version));
COMMENT ON TABLE mfr.share_grant IS 'argon:publish';
INSERT INTO mfr.share_grant(flow, manufacturer_id, contract_version)
VALUES ('upc_prices', '*', 1);
```
(Both DDL blocks quoted from `docs/upc-hub-recipe.md:470-496`.) Then ask the
product which parties it sees — the output is quoted from
`docs/upc-hub-recipe.md:508-514`:
```text
$ argondb hub parties
pin 0/3A1B2C8 — 4 member database(s), 1 manufacturer(s)
hub: party hubcontrol: distributor (publishes no mfr table at this pin)
hub: party mfracme: manufacturer (publishes 2 mfr table(s) at this pin)
hub: party tenanta: distributor (publishes no mfr table at this pin)
hub: party tenantb: distributor (publishes no mfr table at this pin)
```
Every member database gets a line, distributor included — so "no
manufacturers" and "nothing looked" can never read the same
(`docs/upc-hub-recipe.md:516-517`).
**The failure mode, and its one remedy.** A manufacturer database publishing
outside `mfr` becomes a branch of the distributors' own cross-tenant views. The
product names it and exits non-zero (`docs/upc-hub-recipe.md:524-533`):
```text
hub: CROSSES PARTIES mfracme.public — served as "mfracme__public__products", a branch of all_tenants.public__products
```
…and a `hub run` whose source scans that branch is REFUSED before anything
lands. It is a REFERENCE question and deliberately not a pruning one: the
LOGICAL plan is walked before optimization, so a `WHERE tenant IN (…)` over the
view does NOT clear it. The remedy is the one the design states — publish it in
`mfr`. A source over a distributor's own table references no manufacturer copy
and is unaffected (`docs/reserved-identifiers.md:6735`).
**Gates:** `testing/box/upc-hub-warehouse.sh` P20 and
`testing/box/all-tenants-views.sh` P8, both polarities each
(`docs/reserved-identifiers.md:6735`).
#### 4.8 Find out who is holding the provisioner pass lock — VERIFIED (the key) + FROM SOURCE (the SQL)
The list surface prints the lock line itself
(`crates/argondb-ctl/src/lake_tenant.rs:2501-2506`, rendered by
`pass_lock_list_line`, `:2512`):
```text
provisioner lock: pg advisory key 7021788497314804855 on the primary — free / HELD by …
```
To ask PostgreSQL directly you need the key's two halves, because
`pg_locks` stores a single-`bigint` advisory key as `classid` (high 32 bits)
and `objid` (low 32), with `objsubid = 1`
(`crates/argondb-ctl/src/lake_tenant.rs:262-274`). The derivation is
reproducible by hand, which is the whole reason the key is eight ASCII bytes
and not a magic literal — **VERIFIED, run in this lane**:
```text
$ python3 -c 'k=int.from_bytes(b"argon_tw","big",signed=True); print(k, hex(k>>32), hex(k & 0xffffffff))'
7021788497314804855 0x6172676f 0x6e5f7477
```
which matches the registered values at `docs/reserved-identifiers.md:6734`
(`classid = 0x6172676f`, `objid = 0x6e5f7477`, `objsubid = 1`). The query the
product itself runs is `PASS_LOCK_HOLDER_SQL`
(`crates/argondb-ctl/src/lake_tenant.rs:276-281`) — FROM SOURCE, quoted:
```sql
SELECT a.pid, coalesce(a.application_name, ''), coalesce(a.state, ''),
to_char(a.backend_start, 'YYYY-MM-DD HH24:MI:SS TZ')
FROM pg_locks l JOIN pg_stat_activity a ON a.pid = l.pid
WHERE l.locktype = 'advisory' AND l.granted AND l.objsubid = 1
AND l.classid = (7021788497314804855 >> 32)::oid
AND l.objid = (7021788497314804855 & x'ffffffff'::bigint)::oid
ORDER BY a.pid LIMIT 1;
```
The holder's `application_name` tells you which caller it is: the box's own
pass wears `_tenant_wall` and the CLI wears
`argondb_lake_tenant_cli` (`crates/argondb-ctl/src/lake_tenant.rs:185-187`).
Raise the wait with `ARGONDB_TENANT_WALL_LOCK_WAIT_SECS` if an 80-tenant pass
is in front of you (`:179-183`).
**Gate:** `testing/box/provisioner-exclusive-up-after-tail.sh`, 6 phases, both
polarities (`docs/reserved-identifiers.md:6734`).
#### 4.9 Migrate every tenant's schema in one pass — FROM SOURCE
One database per tenant has a cost that shows up the first time the schema
changes: a company created last month and a company created today are each at
whatever version their own `tenant create` shipped. `argondb estate migrate`
closes that gap. It brings EVERY tenant on the box to one version of the
uniform schema in one pass, through the same per-database function step 5 of
`tenant create` calls (`tenant_create::apply_uniform_schema`), so no tenant is
migrated by a path the others were not
(`crates/argondb-ctl/src/estate_migrate.rs:1-17`).
```bash
# bring every tenant to the version this build ships (the default --to)
argondb estate migrate --data-dir "$DATA" --port 5433
# or to a version you name
argondb estate migrate --to 2 --data-dir "$DATA" --port 5433
```
(`crates/argondb-ctl/src/main.rs:555-572`.) What one pass does, in order
(`crates/argondb-ctl/src/estate_migrate.rs:290-315`):
1. **It enumerates the estate directory, `argon_estate.tenant`, and asks
`pg_database` which of those databases really exist.** A directory row whose
database is gone is REFUSED by name. A database with no directory row is not
migrated at all: adopting it is a decision, not a side effect of a pass
(`:34-40`, `:342-353`, `:372`).
2. **It asks each tenant its own version**, from that tenant's own
`argon.schema_version` — never from the directory row. "Never applied" and
"version 0" are different answers and are kept apart (`:42-49`).
3. **It decides apply, skip or refuse for every tenant before changing
anything** (`plan`, `:175`). A tenant at or above the target is SKIPPED and
counted; nothing is ever rolled back. A target above the highest version
this build ships is refused for every tenant before any schema moves.
4. **It applies, and a failure is that tenant's refusal, never the end of the
pass.** One company whose migration fails does not leave the other N-1
un-migrated (`:80-91`).
5. **It records.** The version is read BACK out of the tenant and only then
written to the directory row, so the row can lag its database and can never
run ahead of it (`directory_value`, `:216`); and one row per tenant per pass
goes into `argon_estate.schema_migration`, outcome `applied`, `at-target` or
`refused` with the reason (`JOURNAL_DDL`, `:54-70`).
Every pass ends with one count line, zeros included, so "nothing refused" is
said rather than implied (`summary_line`, `:277-286`):
```text
[argondb-box] estate migrate --to 2: 2 tenant(s) applied, 1 already at target (skipped), 0 refused (named, pass continued), 3 row(s) walked
```
**The exit code is non-zero if any tenant was refused**, after the pass has
migrated every other tenant and named every refusal on stdout
(`crates/argondb-ctl/src/main.rs:4133-4147`). A second run at the same target
applies nothing.
**What it does not do.** It is a command an operator runs, not a reconciler
arm on a timer: a schema change to every company at once needs an operator in
the loop (`crates/argondb-ctl/src/estate_migrate.rs:19-32`). And the spec's
rehearsal step — try the migration on a branch of that company's timeline
first — is NOT built; the pass goes straight from reading the version to
applying it (`docs/2026-09-04-erp-spec.md` §16). **Read the defect paragraph at
the end of §4.0 before you rely on the default `--to`.**
**Gate:** `testing/box/uniform-schema-migration-across-tenants.sh` (P0-P10 —
three tenants at two versions, a planted refusal in the middle tenant with the
tenants on both sides still migrated, the journal, a re-run as a no-op by
count, and an unshipped target refused).
#### 4.10 Create a tenancy, and publish its Identities to every other tenancy — FROM SOURCE
A tenant belongs to a TENANCY — one multi-tenant system in the Estate, the
Distributor Tenancy `distributors` or the Manufacturer Tenancy
`manufacturers` (`--system` in §4.0; the CLI's word for a tenancy is `system`,
and the command name stays `argondb system create`). Creating the tenancy is
the other §13.4 API, and like
`tenant create` it is ordered, journaled, probed and safe to re-run
(`crates/argondb-ctl/src/main.rs:1060-1083`):
```bash
argondb system create manufacturers --data-dir "$DATA" --port 5433
```
| # | step | what exists afterwards |
|---|---|---|
| 1 | register | the system's row in the estate directory |
| 2 | Identities | the system's own `identities-` database and its lake namespace |
| 3 | reader | the system's cross-tenant reader holds `select` on the system's tenant warehouses — never the box warehouse |
| 4 | publish | every OTHER registered system's reader holds `select` on THIS system's Identities namespace |
| 5 | live | the system is marked `live` |
(`crates/argondb-ctl/src/system_create.rs:246-257`.)
**New in v1.5.6: step 4 grants in the same call.** It used to grant only on a
namespace the tail had already published into, so a system created moments
ago printed `DEFERRED`, granted nothing, and journaled itself done — leaving
the operator to run `argondb system grant-identities` later with nothing to say
when. Step 4 now creates the Identities namespace itself when it is absent (the
same act the tail's first publish performs) and grants in the same call
(`crates/argondb-ctl/src/system_create.rs:1486-1512`,
`system_identities_grant::ensure_namespace_id`). And the probe now reads an
absent namespace, with another system registered, as step 4 UNFINISHED, so a
create killed after step 3 runs step 4 again on resume instead of skipping it
(`:931-959`).
**Step 4 publishes in ONE direction, and this is the part to get right.** It
grants the OTHER systems' readers on THIS system's Identities. An earlier
system's Identities is not granted to a system created later — the later
system's `create` grants in the opposite direction only. So the first system
you create publishes to nobody. Create the Distributor Tenancy first: each
Manufacturer Tenancy created after it then grants the Distributors' reader its
Identities, which is the Distributors' estate-wide read for discovery.
**New in v1.5.7: a Manufacturer tenant sees a Distributor's identity once that
Distributor invites it, with no operator step** [D-2026-09-28-65b0a2]. The
invitation is the shared-data grant you already know. The Distributor's
`pub.publication` row names the Manufacturer's database: that is PENDING. The
Manufacturer's own `pub.subscription` to it makes it ACCEPTED. Either one is
enough. On every pass the box writes each inviting Distributor's Identities row
into the Manufacturer's own `derived.identities`, which lists exactly those
Distributors and no others (`invited_by` and `invitation` say who invited it
and in what state). When the Distributor revokes (`argondb publication revoke`),
the refresh stops. The row already landed stays, and the box log names it:
`revoked at — refresh stopped, the landed row is RETAINED`
(`crates/argondb-ctl/src/invited_identities.rs`).
`grant-identities` is still there for boxes that already depend on it. It
opens the publishing system's WHOLE Identities namespace to the reading
system's reader, which is wider than any invitation, so use it only when the
reading system really needs every identity of the publishing one
(`crates/argondb-ctl/src/main.rs`, `SystemCmd::GrantIdentities`):
```bash
argondb system grant-identities --from distributors --to manufacturers
argondb system identities-grants # every recorded grant, LIVE or REVOKED
argondb system revoke-identities --from distributors --to manufacturers
```
Mind the first system's own message: it says "a system registered later gets
its grant from its own `create`", which reads as if the later create covers
both directions; it covers the later system's Identities only
(`crates/argondb-ctl/src/system_create.rs:1470-1478`). A revoke refuses the
foreign reader from its next request onward; what that system already landed
into its own schema stays — it is that system's data
(`crates/argondb-ctl/src/main.rs:1145-1149`).
**What is not built, and printed as such rather than faked.** Step 4's
CROSS-ACCOUNT half — a bucket policy and an assumable role — has nothing to
name on one box with no second account or region; every step 4 run prints
`CROSS-ACCOUNT: NOT BUILT on this box`. And a system's reader is a CATALOG
principal only: the credentials file still carries one box-wide reader key,
not a key per system (`crates/argondb-ctl/src/main.rs:1069-1073`,
`crates/argondb-ctl/src/system_create.rs:1513-1520`).
**Gate:** `testing/box/system-create-sequence.sh` — P3 asserts step 4 created
the Identities namespace, P5 asserts a resume killed after step 3 runs exactly
steps 4 and 5, and the system boundary is proven by a refused read.
#### 4.11 Give each company its own MCP token — FROM SOURCE
Before v1.5.6 the box's one MCP token, `ARGONDB_AUTH_TOKEN`, was the only
credential the MCP endpoint took, so anything you handed a company's agent
could read every company on the box. Now every company carries its own token,
and a session under it reads ONLY that company's database
(`crates/argondb-ctl/src/main.rs:672-687`):
```bash
# tenant create mints the first one, into a 0600 file (§4.0)
cat "$DATA/tenants/acme/mcp-token"
# re-issue: prints a fresh token on stdout; every earlier token of acme is refused from the next request on
TOKEN="$(argondb tenant mcp-token acme --data-dir "$DATA")"
# revoke every token of acme
argondb tenant mcp-token acme --revoke --data-dir "$DATA"
```
**What the token is and what it reads.** A company token starts `amc1.`
(`company_token::PREFIX`, `crates/argondb-mcp/src/company_token.rs:46`) and is
signed by the box. A session under it reads through the company's own read
role, `_mcp`, on the company's own database: `LOGIN`, `CONNECT` on that
one database, and `SELECT` on `public` including tables created later
(`crates/argondb-ctl/src/mcp_company_token.rs:45-87`). So a read of another
company's table is refused by PostgreSQL at the company's own connection, not
by a filter. Exactly one token per company is live: a re-issue supersedes the
last one, and `--revoke` removes the record and makes `_mcp` `NOLOGIN`
(`:217-229`).
**What it refuses, by name.** A bad token is an HTTP 401 whose body says why
(`crates/argondb-mcp/src/company_token.rs:102-122`):
```text
company token refused: its signature does not match its claim company="acme" (forged or re-labelled)
company token refused: company "acme" has no live MCP token (revoked, or never minted)
company token refused: company "acme"'s token was revoked by a re-issue (this is not its live token)
```
Two tools read box-wide state and refuse under a company token: `search`
(`crates/argondb-mcp/src/lib.rs:5028`) and `publish_status` (`:4767`), both
through one refusal, `company_refusal` (`:4613-4625`) — a company session gets
no text search at all, not a company-scoped index. Every response carries `x-argondb-mcp-scope: operator` or
`company:`, so a client can see which scope served it
(`crates/argondb-mcp/src/company_token.rs:52-56`).
**Where it applies.** Company tokens are accepted on the ANALYST surface only;
the admin mount keeps its one operator token
(`crates/argondb-mcp/src/lib.rs:6244-6248`). `argondb serve` turns them on by
setting `ARGONDB_MCP_COMPANY_AUTH_DIR`; with it unset, company tokens are not
accepted at all (`crates/argondb-mcp/src/company_token.rs:48-50`). The box token keeps operator scope
— treat it as the operator credential it is, and hand a company's agent the
company's token instead. A company session reads PostgreSQL only: the lake
half of carrying the role into the read plane (Block 6 [D-2026-09-11-c64ea4])
is not built, so a company token reads no lake table — with one exception: its
`query` reads the company's OWN `__changes` companions, read-only and
under the tenant wall [D-2026-10-03-ada8e3] (guide 4, §3.4).
**Gate:** `testing/box/cross-company-isolation-eight-surfaces.sh`, surfaces S4+
and S4- (`:738-795`).
#### 4.12 Hand each manufacturer its own copy of the global aggregate — FROM SOURCE
A hub run's global aggregate (`--aggregate `) always landed in the BOX
warehouse — which no manufacturer's key opens. Now each manufacturer also gets
its OWN COPY, in its own `hub-` warehouse, so the manufacturer reads it
under its own credential and the wall is not crossed
(`crates/argondb-ctl/src/hub.rs:870-945`):
```bash
argondb hub run --source "$SRC" --ownership "$OWN_TBL" --flow upc_prices \
--hub-table upc_hub --aggregate upc_hub_totals \
--data-dir "$DATA" --port 5433
```
(The shape `testing/box/upc-hub-warehouse.sh` P14b runs, `:1464-1468`.)
**A copy counts only what that manufacturer was granted.** Each manufacturer's
copy is computed over exactly the tenants whose grant of the flow to that
manufacturer is in force at the run's pin — the run's own consent plan, not a
second reading of the grants. Two states land NOTHING, not even an empty
table, and say so on the run and in the ledger (`hub.run_ledger.aggregate_copies`):
```text
hub: aggregate copy for : NOT LANDED — no-flow: this run names no --flow, so no consent plan exists and no tenant granted "" anything; a copy of the aggregate is handed only under a grant
hub: aggregate copy for : NOT LANDED — no-consent: no tenant's grant of flow "upc_prices" to "" is in force at pin , so "" receives no copy of the aggregate
```
The copy lands like the box's aggregate: created on first sight and replaced
after, under the token `:__aggregate:`, so a re-run at the same token
changes nothing. An aggregate whose columns would carry a tenant or a retailer
identity — `tenant`, `customer_id`, `customer_name`, `customer`, `name` — is
refused by name (`check_aggregate_columns`,
`crates/argondb-transform/src/hub.rs:184`). **One stale line:** the
`--aggregate` help still says the aggregate lands "in the BOX warehouse" only
(`crates/argondb-ctl/src/main.rs:2710-2711`).
**The open question, stated plainly.** A copy counted over a SINGLE granting
tenant shows that tenant's total volume across every manufacturer, competitors
included. Whether that is acceptable is waiting on a ruling
(`docs/history/folded/2026-09-24-each-manufacturer-receives-its-own-copy-of-the-global-aggregate.md`).
**Gate:** `testing/box/upc-hub-warehouse.sh` P14b (`:1444-1549`) — the granted
manufacturer receives a copy and the ungranted one receives none, with the
ledger saying why.
---
### 5. Built today versus designed
| feature | status | what proves it |
|---|---|---|
| tenant = a database in one shared cluster | **BUILT** | Block 4 B [D-2026-09-11-1fc4b3]; `docs/2026-09-04-erp-spec.md:89-100` |
| part C — a warehouse, a bucket and a scoped store credential per tenant database | **BUILT** (v1.5.1) | `crates/argondb-ctl/src/lake_tenant.rs` (the provisioner); `crates/argondb-lake-store/src/wall.rs:120` (the predicate); gate `testing/box/tenant-wall-warehouse.sh` P1-P7 |
| the committer resolving a catalog handle PER TABLE, so tenant bytes land in the tenant bucket | **BUILT** | `crates/argondb-committer/src/catalog_set.rs:226-239`; gate P7, re-run after a restart (`testing/box/tenant-wall-warehouse.sh:41-57`) |
| deferral by name when a table's warehouse does not exist yet | **BUILT** | `catalog_set.rs:243-257` |
| parts A + B — catalog authorization and a login, with the box minting its own tokens | **BUILT** 2026-09-11 | Block 9 [D-2026-09-11-b5b145]; `docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:846-854`; `crates/argondb-ctl/src/lake_grant.rs` |
| `catalog_authz` DEFAULT = on with more than one database | **BUILT** 2026-09-15 | ruled by question block 9/15; `docs/2026-09-15-cross-tenant-read-design.md:502-516` |
| bundling OpenFGA in the Linux release artifact | **BUILT** | `scripts/release/build-artifact.sh:243-246` (license-text gate), `:584-591` (staging `bin/openfga`) |
| bundling OpenFGA in the macOS release artifact | **BUILT** | `scripts/release/build-artifact-macos.sh:223-226` (license-text gate), `:519-523` (staging `bin/openfga` against the `openfga-darwin-arm64` pin) |
| three-level identity `database.schema.table` in the catalog, primary regularized | **BUILT** 2026-09-11 | Blocks 1 A / 2 B / 3 A [D-2026-09-11-702e69] [D-2026-09-11-118b2e] [D-2026-09-11-f73865]; `crates/argondb-ctl/src/lake_address.rs:19-36` |
| the identity version gate — migrate or refuse, never guess | **BUILT** 2026-09-11 | `docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:470-511` |
| the OLTP half — a per-database connection limit, fixed at 20 | **BUILT** 2026-09-15 | ruled 9/15; `crates/argondb-ctl/src/lake_tenant.rs:114-141`, `:596`; gate P10 |
| the provisioner pass lock (one pass at a time) | **BUILT** 2026-09-16 | `crates/argondb-ctl/src/lake_tenant.rs:157-183`; `docs/reserved-identifiers.md:6734` |
| the cross-tenant reader — one read-only key, one catalog login, roster = the estate | **BUILT** 2026-09-15 | Block A, ruled 9/15; `lake_tenant::ensure_reader`; gate `testing/box/cross-tenant-reader.sh` |
| `all_tenants.__` views with pushdown, drift named twice | **BUILT** 2026-09-16 | `crates/argondb-query/src/all_tenants.rs`; gate `testing/box/all-tenants-views.sh` |
| `hub run` reading under the reader and writing as the box | **BUILT** 2026-09-16 | `docs/reserved-identifiers.md:40`; `docs/2026-09-15-cross-tenant-read-design.md:653` |
| the `mfr` schema and the party rule | **BUILT** 2026-09-19 | `docs/reserved-identifiers.md:6735`; `crates/argondb-query/src/all_tenants.rs:91-112` |
| the hub return flow — a manufacturer's price list landing in each ACCEPTING distributor's `derived.mfr_price_list` | **BUILT** (`argondb hub return`, shipped 2026-09-19, box gate green on the merged tree) — the main diagram still draws it dashed, and that is stale | `crates/argondb-transform/src/hub_return.rs`; `docs/2026-09-17-distributor-manufacturer-data-sharing-design.md` §3.4; HISTORY#hub-return-landing-box-gate-on-the-merged-tree |
| the Postgres role travelling into the read plane (Block 6) | **PARTLY — refusal built, propagation into the hub's in-process read NOT** | [D-2026-09-11-c64ea4]; `docs/lake-foreign-read.md:651-668` (the refusal) and `:670-691` (the limit) |
| per-tenant storage timelines (per-tenant PITR, per-tenant freshness) — a tenant database as its own Postgres instance on its own timeline | **RATIFIED, BUILT, default OFF** — each tenant compute runs on a branch cut at `tenant create`, and the tail follows every tenant timeline as its own stream | B′ (amended) [D-2026-09-20-0d5ba3] [D-2026-09-21-4107c5]; `docs/2026-09-04-erp-spec.md` §2; `crates/argondb-tail/src/streams.rs`; HISTORY#tail-multiplexes-tenant-timelines |
| one compute per tenant unit | **RATIFIED, BUILT, default OFF** under `[compute] per_tenant` — the switch is the operator's, never a system default; once a tenant has its compute, its writes go there | [D-2026-09-20-0d5ba3] [D-2026-09-21-4107c5] [D-2026-09-23-5122a6] [D-2026-09-25-a2d0a3]; `crates/argondb-ctl/src/tenant_compute.rs:105`, `tenant_data.rs`; deep dive 2 |
| the estate control plane — provisioning API, Identities projection and the multiplexed tail's lease, on their own estate instances, leader plus forwarding | **DESIGNED, NOT BUILT** on the multi-instance shape; on one box it collapses to a single process and the `argon_estate` schema | [D-2026-09-21-8b9d22]; "one-box complete" also requires showing the leader and its forwarding [D-2026-09-23-21780c]; `docs/2026-09-04-erp-spec.md` §13.3 |
| a multi-tenant control plane provisioning N cloud instances (not "N boxes" — that framing was corrected) | **DESIGNED, NOT BUILT** | `docs/2026-09-04-erp-spec.md` §13, disposition §18, cited at `docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:892-894`; the cloud-not-boxes correction [D-2026-09-21-6e436d] |
| storage-prefix-scoped short-lived credentials (STS vending) | **NOT AVAILABLE with the bundled store** — a hosted/BYO-S3 option | `docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:876-883` |
| a cross-party mount — one tenant reads exactly the tables another tenant GRANTED it, one row per table; it CANNOT write, read an ungranted table, or reach a third party's (or any other database's) warehouse | **BUILT, box-proven** — both polarities (granted / ungranted), both tenant-wall states (on / off) | `hub.mount_grant (grantee_database, source_schema, source_table, granted_at, revoked_at)`, published by `MOUNT_GRANT_DDL` (`crates/argondb-transform/src/hub_consent.rs:304`) and decided by `grant_state` (`:204`) through `mount_grant_state` (`:353`); `DatabaseScope::Granted { owner }` (`crates/argondb-tail/src/lib.rs:908-918`); the one named refusal on the read plane, two sentences — never-granted keeps the `cross-database` anchor, revoked carries the instant (`crates/argondb-tail/src/flight.rs:655-677`); gate `testing/box/fdw-cross-party-mount.sh`: `fdw-cross-party-mount: PASS — 11 checks, tenant wall ON (revoke lag 28 s)` and the same line, tenant wall OFF (revoke lag 29 s), ms-01 |
| `argondb estate migrate` — one pass brings every tenant's uniform schema to one version, skip-at-target, a refused tenant never stops the pass | **BUILT** 2026-09-27 — but see the version defect at the end of §4.0 | `crates/argondb-ctl/src/estate_migrate.rs`; gate `testing/box/uniform-schema-migration-across-tenants.sh` |
| rehearsing a tenant's migration on a branch of its timeline first (ERP spec §16 step 2) | **NOT BUILT** | `docs/2026-09-04-erp-spec.md` §16 |
| `argondb system create` step 4 publishing a new system's Identities to every other system in the same call | **BUILT** 2026-09-27; one direction only, and the cross-account bucket policy is **NOT BUILT** | `crates/argondb-ctl/src/system_create.rs:1454-1522`; gate `testing/box/system-create-sequence.sh` |
| a per-company MCP token reading only that company's database | **BUILT** 2026-09-27 (PostgreSQL only; the lake half of Block 6 is **NOT BUILT**) | `crates/argondb-ctl/src/mcp_company_token.rs`, `crates/argondb-mcp/src/company_token.rs`; gate `testing/box/cross-company-isolation-eight-surfaces.sh` S4+/S4- |
| each manufacturer's own copy of the global aggregate, counted over its granted tenants only | **BUILT** 2026-09-27; the single-granting-tenant disclosure is **OPEN** | `crates/argondb-ctl/src/hub.rs:870-945`; gate `testing/box/upc-hub-warehouse.sh` P14b |
| a view defined over a table the database holds only through a mount grant | **REFUSED BY NAME** (`view-input-granted`) until the reader scope is ruled | `crates/argondb-query/src/view_classify.rs:2206-2218`; gate `testing/box/view-over-granted-table.sh` |
---
### 6. Gotchas
**A tenant table published before its wall exists is DEFERRED, and the log line
repeats.** That is the design, not a stall: the alternative is publishing the
bytes behind the box credential, which is the wall inverted. Wait one cadence
or run `argondb lake tenant provision`
(`crates/argondb-committer/src/catalog_set.rs:243-257`).
**A database whose name is not a legal object-store bucket name is REFUSED, by
name, and the pass continues.** Nothing rewrites the name to fit, because a
rewrite would merge `tenant_a` and `tenant-a` into one bucket — which is the
wall this feature exists to build. The remedy is
`[lake.tenant_warehouse]. = { warehouse, bucket }` in `argon.toml`,
the only door for such a database (`docs/reserved-identifiers.md:6732`). Gate
P9 proves both polarities (`testing/box/tenant-wall-warehouse.sh:58-61`).
**`warehouse_per_database = false` removes the wall AND the reader.** Every
database resolves to the box warehouse, so `reader_plan` returns `None` and the
pass REMOVES a reader an earlier configuration left behind. There is no
cross-tenant read without a wall
(`crates/argondb-ctl/src/lake_tenant.rs:108-112`, `:1529-1532`). An explicit
`true` against a bring-your-own object store is also refused: the box can mint
a bucket-scoped key only in a store it runs
(`docs/reserved-identifiers.md:6732`).
**With `catalog_authz = "off"`, names are public even though bytes are not.**
Anything that can reach the catalog port lists every warehouse and every table
name with no credential at all. The served text says so in those words. Bind
the catalog to loopback and treat its port as privileged
(`docs/lake-foreign-read.md:196-202`).
**OpenFGA ships in both platforms' release artifacts.**
`scripts/release/build-artifact.sh` stages `bin/openfga` on Linux (license
gate `:243-246`, staging `:584-591`) and `scripts/release/build-artifact-macos.sh`
does the same on macOS (license gate `:223-226`, staging `:519-523`), so
neither platform's stock multi-database box meets a boot refusal for a
missing engine (`docs/2026-09-15-cross-tenant-read-design.md:513-516`).
**A continuous view over a table you hold only through a mount grant is
REFUSED at create, by name.** A lake-landed view is scoped to its inputs'
OWNER, so a view a grantee defined over the grantor's table would be created
and then be unreadable in the grantee's own database; whether a grantee may
build one at all is an open ruling (views design OPEN-V3). Until it is ruled,
the create door refuses before the pin and before anything is enrolled, on both
landings, with kind `view-input-granted` and this text
(`crates/argondb-query/src/view_classify.rs:2206-2218`;
`crates/argondb-transform/src/view_door.rs:493-507`):
```text
view ``: input .. is a mounted grant from ; views over granted tables are not ruled — [ruling pending]. A lake-landed view is scoped to its inputs' OWNER, so this one would be created and then unreadable here; define the view in , or over tables this database owns
```
The remedy is in the last clause: define the view in the grantor's database,
or over tables yours owns. Two edges. The check runs only when the create
request names a database (`crates/argondb-transform/src/view_door.rs:504`), so a request with none is not
checked. And the lane that built the refusal escalated three defects it did not
fix, reproduced with an OWNED input as well: the tail panics on a view that was
declared and never refreshed (`crates/argondb-tail/src/lake_owned.rs:114`), a
view created through `argon.create_view` fails every refresh on its recorded
`lake_main.` qualifier, and under the tenant wall the create door cannot plan
any tenant database's table
(`docs/history/folded/2026-09-24-a-view-over-a-granted-table-is-refused-by-name-until-the-reader-scope-is-ruled.md`).
Gate: `testing/box/view-over-granted-table.sh`.
**Hand a company's agent its own MCP token, never the box token.** The box's
`ARGONDB_AUTH_TOKEN` keeps operator scope over every company; the company token
reads that company's database only (§4.11). A company session has no `search`
and no `publish_status` — both read box-wide state and refuse by name — and
reads no lake table, because the lake half of the role's propagation is not
built (`company_refusal`, `crates/argondb-mcp/src/lib.rs:4750-4762`).
**`argondb system create` publishes one way.** A system's Identities reaches
the systems that existed when it was created, not the ones created after it.
A Manufacturer tenant does not need that grant to see the Distributors that
invited it: since v1.5.7 an invitation lands their identities in its own
`derived.identities` with no operator step [D-2026-09-28-65b0a2].
`argondb system grant-identities --from --to ` opens the
whole namespace and is wider than an invitation (§4.10).
**Revoking a grant leaves the token valid until it expires.** Nothing here
keeps a revocation list. Use `--ttl`
(`crates/argondb-ctl/src/lake_grant.rs:281-287`).
**The operator is inside every wall.** The root store key and the
`argondb-box` catalog subject are plural over everything, the box warehouse
included, and they write. That is drawn in the main picture and listed as one
of its two residuals; it is not something the wall is intended to stop
(`docs/2026-09-15-cross-tenant-read-design.md:190-195`).
**Row-level security does not reach the lake copy, and the Flight-door refusal
does not cover the catalog door.** A reader holding the catalog URL and the
store credentials reads every row of every published table, row security or
not. Those credentials are the only thing standing between such a reader and
the data (`docs/lake-foreign-read.md:670-680`). Two related surprises: a table
already protected the first time the lake mounts it gets no mount at all (the
type-learning read is subject to the same wall; run `SELECT
argondb_lake_sync()` as a role row security does not apply to), and toggling
row security reaches the wall on the schema cadence rather than instantly
(`:693-705`).
**The read plane's assertion is believed, not verified.** The compute asserts
"this read is for role R and row security does not apply to R here", and the
read plane cannot check it, because the only credential on the wire is one
process-wide bearer token. Anyone holding `ARGONDB_AUTH_TOKEN` can assert any
role: do not hand it to analysts (`docs/lake-foreign-read.md:682-691`).
**`SHOW ALL TABLES` does not list a nested table.** DuckDB enumerates
`GET /namespaces` with no `?parent` and never recurses; a table you have not
NAMED is absent from the listing. `USE lake."br_main.."` first and
it appears (`docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:246-250`;
the catalog-walk recipe is `docs/lake-foreign-read.md:468-520`).
**A copy that already has a column named `tenant` cannot be unioned**, and
neither can a column whose types across tenants the engine cannot coerce
(`Int32` with `Boolean`). Both are refused BY NAME and the view is absent from
the namespace — never silently narrowed
(`crates/argondb-query/src/all_tenants.rs:53-58`, `:60-64`).
**A session with no roster has no `all_tenants` namespace at all.** The
namespace is composed at pin time from what the session registered, so a
roster-less session is not a session with an empty view set
(`docs/lake-foreign-read.md:315-318`).
**Do not size a deployment from the numbers in §3.7 without re-measuring.**
The earlier figures on that same line do not reproduce, and the document that
carries them says so (`docs/lake-foreign-read.md:404-409`).
**Two connection-limit surprises.** Superusers are exempt, and the limit counts
connections to the DATABASE rather than by role
(`crates/argondb-ctl/src/lake_tenant.rs:136-141`). A limit an operator set by
hand is never touched by a plain pass — neither lowered nor raised — so a
tenant sitting at a hand-set 5 stays at 5 until an explicit override says
otherwise (`:749-753`).
**Cross-warehouse atomicity of a cut is not claimed.** A cut spanning two
tenants is one transaction per warehouse; a cross-tenant pin lands at the
newest LSN every scoped table in every tenant warehouse serves, and says per
warehouse which coordinate it resolved
(`docs/2026-09-15-cross-tenant-read-design.md:324-343`).
**Lake freshness is box-granular — with per-tenant compute off, the shipped
default.** Every tenant is then on the primary's one timeline, so one tenant's
write volume gates every other tenant's watermark
(`docs/2026-09-15-cross-tenant-read-design.md:591`). Per-tenant freshness needs
a separate Postgres instance and timeline per tenant; that is the ratified
topology [D-2026-09-20-0d5ba3] [D-2026-09-21-4107c5], built default off, with
the tail running one stream per tenant timeline
(HISTORY#tail-multiplexes-tenant-timelines), and the operator turns it on, not
the system [D-2026-09-23-5122a6].
**The catalog's REST `{prefix}` is the warehouse's UUID, not its name.**
`/catalog/v1/argon/namespaces` answers
`400 {"error":{"type":"WarehouseIdIsNotUUID"}}`; the resolver is the config
response's `defaults.prefix`, read from
`GET /catalog/v1/config?warehouse=`
(`docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:191-196`).
**Not measured, and named as such:** Trino. No Trino was stood up. Its Iceberg
REST connector is documented to map a namespace to a schema with the same
three-part budget, so the same arithmetic is EXPECTED to apply — that is an
inference, not a measurement
(`docs/2026-09-10-multi-database-identity-and-tenant-isolation-design.md:257-262`).
---
### 7. Sub-diagrams
Three deeper pictures of parts the main diagram compresses. Each is generated,
not drawn: the source is a Python script beside the others, using the same
palette and legend, and `python3