--- class: reference purpose: "The ArgonDB developer guide, reader's edition — the same seven chapters, written for a human reader." template: product --- # The ArgonDB developer guide — reader's edition A reader's edition of the ArgonDB developer guide, assembled from `docs/readers-guide/`. ## Table of contents - [Developer guide 0 — the themes that run through ArgonDB](#developer-guide-0-the-themes-that-run-through-argondb) - [Guide 1 — one database per tenant, and the wall between tenants](#guide-1-one-database-per-tenant-and-the-wall-between-tenants) - [Guide 2 — the per-tenant unit: the gate, wake and stand-down](#guide-2-the-per-tenant-unit-the-gate-wake-and-stand-down) - [Guide 3 — the lake and its process model](#guide-3-the-lake-and-its-process-model) - [Guide 4 — continuous materialized views, and a view across the tenant boundary](#guide-4-continuous-materialized-views-and-a-view-across-the-tenant-boundary) - [Guide 7 — agents and the MCP endpoint](#guide-7-agents-and-the-mcp-endpoint) - [Guide 8 — building an application cloud on ArgonDBCloud: the ERP's configuration system and ERPCloud](#guide-8-building-an-application-cloud-on-argondbcloud-the-erps-configuration-system-and-erpcloud) --- ## Developer guide 0 — the themes that run through ArgonDB ArgonDB is PostgreSQL for software that serves many companies at once. Each company gets a database of its own, walled off from every other company's at the storage level, and every table you choose to publish also appears, seconds later, as a read-only analytics copy in a data lake (an Iceberg table) — with no sync job for you to write. On top of that sit three things most teams otherwise build by hand: reports that keep themselves current, a recorded way to share data between companies with consent, and a door for AI agents that is the same door people use. This chapter is for someone deciding whether to build on it. It does not walk through the five guides that follow one by one; each of those opens with its own summary. Instead it describes the six ideas that recur across all of them, because those six ideas are what you would actually be building on: 1. one database per tenant; 2. a wall between tenants that no feature can reach across; 3. a lake that is fed from the write log and cannot drift from it; 4. views that you state once and that stay current; 5. consent, recorded as data, for anything that crosses the wall; 6. agents that are bound by exactly the same rules as people. Each theme has the same four parts: what it is, in plain terms; what the deeper chapters say about it; what it changes about how you build; and what you meet when you run it — the commands you type, the work you stop doing by hand, and the refusals you will see. A few terms the chapters use throughout. **The box** is one machine running ArgonDB — its PostgreSQL, its storage, its lake and its catalog. **The operator** is whoever runs that box. **The sidecar** means only a tenant's own application process. The process that reads the write log and turns it into the lake is called **the tail**. The five deeper chapters: [one database per tenant, and the wall](#guide-1-one-database-per-tenant-and-the-wall-between-tenants), [the per-tenant unit's gate, wake and stand-down](#guide-2-the-per-tenant-unit-the-gate-wake-and-stand-down), [the lake and its process model](#guide-3-the-lake-and-its-process-model), [continuous materialized views](#guide-4-continuous-materialized-views-and-a-view-across-the-tenant-boundary), and [agents and the MCP endpoint](#guide-7-agents-and-the-mcp-endpoint). --- ### 1. One database per tenant, on its own timeline **In plain terms:** every customer gets their own database, not a shared table with a customer-ID column. Think of everyone keeping their money in one drawer with a label on each bill, versus everyone having a separate drawer. A bug in the label-reading code can hand out the wrong bills; a bug that opens the wrong drawer finds it empty, because that customer's money was never in there. **What the chapters say.** Chapter 1 puts the consequence in one sentence: *"There is no query in this system that could return another company's row, because another company's rows are not in the database being queried."* Where this is heading, and where it is today, are now closer together than "designed, not built." The more isolated shape — a company with its own compute, its own PostgreSQL instance, its own write-ahead log and its own timeline, sharing only the storage layer underneath — is something the product actually ships in this version, not only a plan. It travels behind one operator switch, `[compute] per_tenant`. Left off, which is the default, every tenant still shares one compute and one write-ahead log on the box, exactly as before. Turned on, a company created from that point on gets its own compute from its very first minute. What is not yet built is moving a company that already holds data onto that shape after the fact — the design says so plainly rather than leaving you to find out the hard way. Chapter 2 describes the lifecycle that runs underneath either shape. A *unit* is one compute plus at most one sidecar, and it is sized to be exactly the thing that can be destroyed and recreated without losing anything. Measured, a stood-down unit takes 137 MB and wakes in about 0.6 seconds. Chapter 3 explains what keeps this affordable across many machines: the tail is a single process that serves every tenant's timeline at once, never runs alongside any one tenant's own instance, and gives each stream its own memory budget so the hundredth tenant cannot starve the first. **What it changes about how you build.** You never design multi-tenancy into your own schema. There is no tenant-ID column to remember on every table and no `WHERE` clause standing between a bug and a leak between customers. What ArgonDB does keep uniform across every tenant is its own per-company schema — and, in this version, uniform across upgrades too: one command, `argondb estate migrate`, walks every company on the box, checks the schema version each one is actually running, skips any company already caught up, and brings the rest forward the same way `tenant create` itself would. That command only ever touches ArgonDB's own tables — its identity table and its one write door — never your own application's tables, which stay yours to version. The connection pooler in front of all this is optional: the box uses a compatible PgBouncer automatically when one is available, and serves every tenant on the direct port when it is not. **What you meet when you run it.** Creating a company is one command: ```bash argondb tenant create acme --display-name "Acme" --legal-name "Acme Holdings LLC" \ --system distributors --identity "tax_id=12-3456789" ``` It creates the tenant's timeline, its compute record, its database, its login role and grants, its uniform schema, its wall pass and its identity record — and, if you named one, its sidecar application — in that order. It is safe to run again if it was interrupted, and it finishes by minting the company its own token for the agent door (theme 6). What you stop doing by hand: you never write idle detection. Traffic reaching the tenant's own door is the only signal a unit uses to decide it is still in use, and the whole control is one setting, `[compute] idle_stop`. The refusals you will meet: the command refuses to take over a database it did not create, rather than silently adopting it. A client that connects while its tenant is still waking gets PostgreSQL's own `57P03 cannot_connect_now`, which every driver already knows to retry. `argondb estate migrate`, run after an upgrade, brings every existing company forward; one it cannot bring forward is named `REFUSED` rather than stopping the whole pass, and the command's own exit code says how many were refused. A missing or too-old connection pooler is never a reason the box refuses to start — it starts on the direct port and says so, on its own startup banner and in `argondb status`; the one pooler setting that does refuse to start the box is simply a misspelled one. --- ### 2. The wall — what every surface is measured against **In plain terms:** each tenant's data sits behind its own locked door, with its own key. A feature does not get to say "trust me, I checked the badge" — it simply has no key to the other doors. That is the difference between a rule a programmer could forget and a wall a programmer cannot walk through by accident. **What the chapters say.** Chapter 1 defines the wall as three parts that all apply at once, never a choice among them: - the catalog (the directory of lake tables) refuses to list or read another tenant's tables; - every request to the catalog carries a verified login; - each tenant database gets its own warehouse, its own storage bucket and its own storage credential, so the storage itself refuses a key that belongs to someone else. The wall is not a filter inside a query. It is a different database, a different place in storage and a different credential. One decision point gates every read and write against every storage bucket. The same approach reappears wherever a boundary is checked. Chapter 4's views check which database a view reads with the same rule, and refuse a join across two databases by name — rather than letting the query planner narrow it away with a `WHERE` clause. Row-level security is a separate and narrower tool. It works inside one tenant's own database and separates roles, not companies, so a mistake in a row policy is an over-permissive grant inside one company, never a leak between customers — and a table's copy in the lake carries no row policies at all, so a protected table's lake copy is refused to a reader rather than served unprotected. One limit applies to all of this: **the box operator stands inside every wall; a tenant never does.** The wall protects tenants from each other, not from whoever runs the machine. The chapters are equally direct about the storage layer underneath the databases: every company's data is really a timeline inside the box's one shared storage service. That service refuses any caller with no token, and each company's compute gets a token that names only its own timeline, so the service refuses it on any other company's timeline. That check is built, and the cross-company isolation test proves it. One caveat: with per-tenant compute off, the default, one compute serves every company's database, so the storage token is not what separates companies there; the connect wall and the password rule are. **What it changes about how you build.** You do not write an authorization check to keep tenants apart, because no code path reaches across in the first place. That has one consequence you should plan around: a feature that genuinely needs two parties' data side by side cannot be a single query that spans the wall. It has to be a component with a foot on each side that makes an explicit, recorded copy. Theme 5 (consent) is how that copy is made, and Theme 4 (views) is how you keep it useful once it has landed. **What you meet when you run it.** There is nothing to configure: once each tenant has its own warehouse — which is the default — the wall holds, and the catalog's authorization engine ships inside the release itself, on every platform ArgonDB runs on, so there is nothing extra to install. You hand out and take back a reader's access with `argondb lake grant`, `argondb lake attach-clause` and `argondb lake revoke`. The refusals are specific, and they are part of the interface rather than an edge case: - reaching into another tenant's bucket answers `403 AccessDenied`, naming the key and both buckets; - a catalog warehouse you may not see answers `404`, never `403`, so its name is not disclosed at all; - a view that joins tables from two databases is refused as `cross-database-join`, with the remedy named instead of a bare error. --- ### 3. The lake is the shared read side, and the tail feeds it **In plain terms:** every change you make in the transactional database shows up, seconds later, in a second, read-only copy built for big analytical questions — without you writing a sync job. Picture a photocopier beside your filing cabinet that copies each page the moment you file it, so other people can read the copies without ever touching your originals. **What the chapters say.** The promise is one sentence: *"every Postgres table is automatically an Iceberg table, at a known LSN, with sub-second freshness and zero impact on transactional performance."* (An LSN is a position in PostgreSQL's write-ahead log; it names an exact moment in the data's history.) Chapter 3 describes the mechanism. The tail decodes the write log exactly once, and everything downstream — its in-memory buffer, the change log, the lake copy and every reader — is fed from that one decode. That is why "the lake cannot drift from the database" is a consequence of the design rather than a hope. A read at position `L` is answered from the durable lake copy up to its watermark `W`, plus the tail's own decoded changes between `W` and `L`. Chapter 2 explains why the tail can stay running beside a tenant whose compute is asleep without waking it: an asleep compute writes nothing, so there is nothing to decode and the tail idles. Chapter 4's views are a second consumer of the same watermark: a view's answer moves forward only when the tail reports that the watermark moved, never on a timer of its own. The change log the tail produces beside a published table is not optional bookkeeping — it comes from that same single decode, so the audit trail it gives you is complete by construction rather than by discipline. This version also tightens who is allowed to write where in the lake: the process that copies a table over from Postgres is the only writer of that mirrored copy, ArgonDB's own lake writer is the only writer of anything it derives (a continuous view's table, a landed transform), and inside that derived area each table still has exactly one writer — a view's own table is written only by that view's own maintainer, and never by anything else. **What it changes about how you build.** A reporting feature stops being a project with its own ETL job, schema mapping and reconciliation step. It is a `SELECT` over a copy that is guaranteed to match the transactional table, because both come from one decode of one write log. The questions left for you are which tables to publish, and how stale a read you can accept — not how to keep a second system in step with the first. A result you compute from the lake — a transform or a view — can land straight back into it, with no separate database step, as long as you do not reuse a name another door already owns. **What you meet when you run it.** Publishing a table is one command, with no connector to configure: `argondb lake publish ` marks the table, copies it once at a fixed position, and then follows its changes. You never write a change-data-capture connector, define a Kafka topic or run a reconciliation job. `argondb lake list` shows where each table stands — `lake @ 0/1BD2748`, `backfill pending`, or `QUARANTINED @ ...` — and freshness is a number you can read, `ingest_lag_seconds`, rather than a promise. Two limits to know in advance. **A column type the decoder cannot handle is stored normally in PostgreSQL but never reaches the lake, and nothing raises a SQL error.** That is why a separate `publish_status` check exists, and why you should run it after publishing. And landing something into a piece of the lake another door already owns — say, a transform aimed at a table that belongs to a view — refuses by name and writes nothing, telling you to land under a different name or remove the existing table through the door that made it. --- ### 4. Views: say what should stay current, and it does **In plain terms:** instead of a script that re-runs a big report every night and hoping it finishes before someone asks, you write the report's definition once — "this is the number I want" — and the system keeps that number accurate, updating only the parts that actually changed. You stop being the person who has to remember to press refresh. **What the chapters say.** Chapter 4 covers continuous materialized views. A view holds its answer as of one log position and moves forward by applying only the changes since then, rather than re-running the whole query. A reader gets either the exact current answer or a named older one with its distance from current stated — never a silent guess. You define a view with a function call, `argon.create_view(name, sql, options)`, not PostgreSQL's native `MATERIALIZED VIEW`. The reason is that a native materialized view leaves the refresh to you, which is exactly the chore this exists to remove. From the query alone, ArgonDB decides which of four maintenance strategies applies: a keyed projection, an aggregate it can update arithmetically, a join, or — when none of those hold — a full recompute. When it is unsure, it chooses the safe, more expensive strategy, never the reverse. A **transform** is the one-time sibling of a view: a `SELECT` computed once at a fixed position and landed, rather than kept current forever. Because it is pinned to that one position, running it again gives you back the exact same answer — unlike a hand-written batch that buckets rows against "now," which quietly rewrites history if you run it again later. A transform and a view both land in one of two places: an ordinary Postgres schema that then reaches the lake the normal way, or straight into the lake's own derived area, which is where a view lands by default. The two mechanisms combine directly: one chapter's own worked example is a continuous view built over a table that another company's consent had already landed into this one — Themes 4 and 5 in one example. **What it changes about how you build.** For each output, you choose one of three shapes: - an ordinary table that your own code writes; - a **transform** — a reproducible, one-time `SELECT` fixed at a position, for something tied to a business event such as a period close; - a **continuous view** — a `SELECT` you state once and the system keeps current. The deciding question is whether the fixed position itself is the point (a transform at month-end) or the answer must never go stale between requests (a view). Neither asks you to work out incremental maintenance by hand. **What you meet when you run it.** The same definition goes through three doors: `argon.create_view('name', 'SELECT ...', options)` from SQL, `argondb view create --name ... --sql ...` from the command line, or the `create_view` tool for an agent. You never write a refresh cron job, and you never decide by hand whether a deleted row should reduce a group's total. There are sixteen kinds of refusal, each specific and each naming the offending construct and a remedy — among them `cross-database-join`, `non-deterministic` (a view over `now()` or `random()` is refused, even one that would be fully recomputed) and `self-cycle`. The default staleness tolerance is zero: a reader who asks for an exact answer gets it, or a stated refusal. A view's own table in the lake belongs only to it: a transform or an append aimed at that table refuses by name and writes nothing, so nothing but the view's own maintainer can ever move its answer (theme 3). The chapter that covers views keeps its own running list of what still does not work on this feature, and states it plainly rather than rounding it off. --- ### 5. Consent and grants — how data crosses a boundary **In plain terms:** a company's data never leaves its own room through an automatic back door. If another company is allowed to see part of it, that permission is written down — as a row in a table, in the room the data came from — so anyone can later see exactly what was allowed, when, and to whom, and the company can take it back. **What the chapters say.** Consent is a row in the database of the party whose data is crossing — never a configuration file and never an operator setting, and written with that party's own ordinary permissions. The general form is a **mount grant**: a database that wants another database to be able to read one of its lake tables writes one row naming exactly that table, with no wildcard — a table you create later is not automatically included. The lake's read side then serves that one table to that one grantee, and refuses every other cross-database read by name. An older, narrower shape of grant still exists for a scheduled hand-off of data between two parties, checked by the same rule. The state of a grant is not a yes/no. Asking for it returns one of four answers — granted; never granted; revoked, with the time; or granted under a contract version this build does not yet serve — because each of those means something different to whoever reads the log later. The component that moves data between parties holds two identities and never mixes them. It reads every tenant under one read-only identity that spans tenants, and writes results back as the box. There is no setting that lets it read as the box from a walled database. Chapter 7 applies the same principle to what an agent may *do* rather than what it may see: on the administration door, a destructive call needs an explicit `confirm=true`, and without it the call refuses and names what would be lost — "an agent proposes; the operator confirms." **What it changes about how you build.** You never design an outbox table, a webhook or a second replication path to move data between parties. That door is closed by design, not left for you to build carefully. You decide what crosses instead — which table, to which database — and the owning party writes a consent row in its own database, using its own ordinary write permissions. Because consent is data, it travels through the tail like any other row, so it can be queried, audited and versioned like any other business fact. **What you meet when you run it.** ```sql -- grant INSERT INTO hub.mount_grant(grantee_database, source_schema, source_table) VALUES (...); -- revoke UPDATE hub.mount_grant SET revoked_at = now() WHERE ...; ``` `argondb hub grants` reads the current state of every grant, and always succeeds — it reports consent, it does not judge it. The refusals you will meet: a read of a table with no grant refuses by name, naming the owner, the asker and the exact statement that would grant it; a read after a revoke gives a different message, naming when the grant was revoked. Two limits, stated directly: a revoke takes effect within one refresh cycle, not the instant you run it, so a read in between can still succeed; and revoking is forward-only — rows already delivered under a grant stay where they landed. Pulling those back is a separate, deliberately harder operator action, and it refuses while the grant is still in force, naming the exact step to take first. A grant made under a contract version the running build does not serve is refused by name, with a message that says the company did consent — just to a different version. --- ### 6. Agents use the same doors as people **In plain terms:** an AI agent connecting to ArgonDB does not get a special side entrance. It is handed a key to one specific door — the same kind of key you would hand a human analyst, a dashboard or a batch job — and that door decides everything it can and cannot do from then on. **What the chapters say.** Chapter 7 describes the agent endpoint (it speaks MCP, the Model Context Protocol). That endpoint *is* ArgonDB's API, split into four doors by level of trust — analyst, dba, admin and write — each with its own token, its own port, its own default network binding and its own PostgreSQL role behind it. In the chapter's words, "an operator composes an agent's capability by choosing which mounts it connects, so the risk analysis happens at connect time rather than by curating a tool list." On the analyst door, read-only is enforced by the database, not by a query parser's opinion: the PostgreSQL role is the boundary, and a write attempt fails with PostgreSQL's own `42501 insufficient_privilege` — the same error a person with the same grant would get. In this version, every company also gets its own token for the analyst door: a request carrying it runs as that one company's own read role, which can reach that company's database and no other, and holds no write privilege at all. The operator's own token still exists and still works everywhere, but it is no longer the only credential an agent can be handed. No reader gets an exception. The rule that redirects a query to a view's maintained answer is installed the same way regardless of who is asking, so an analyst's database connection, the agent's `query` tool, the command-line query path and a transform all get identical treatment. Chapter 3 states that the lake-facing agent door always reads through the same query path as everything else. **What it changes about how you build.** You never build a separate, elevated access path for AI. You set an agent's entire reach by choosing which door its process connects to — the same choice an operator already makes for a human analyst, a read replica or a batch job. An agent on the analyst door is bound by exactly the same tenant wall (Theme 2), the same fixed read positions (Theme 3) and the same view machinery (Theme 4) as a person on that door. There is no seventh theme for what agents get to skip. When the agent works for one company, hand it that company's own token rather than the operator's — its reach is then one database by construction, not by a prompt's good behavior. **What you meet when you run it.** One `claude mcp add` per door, each with its own URL and token. A session typically runs `schema_overview`, then `pin` (fix a read position), then `query` against that pin, then `changes(since=)` to see what has moved since — carrying the same `pin_id` through the whole exchange. A company's own token is minted when its tenant is created, and `argondb tenant mcp-token ` re-issues it (the old one stops working); `--revoke` turns off every token that company holds. Every response names the scope it ran under, so you can always tell whether a call ran as the operator or as one specific company. The refusals you will meet: a wrong token on any door is refused with `401 Unauthorized` before any tool runs. A company's own token only works on the analyst door — the dba, admin and write doors still need the operator's — and the two analyst tools that read box-wide state instead of one company's data refuse under a company token, naming the operator as the one who can run them. The admin and write doors refuse to start at all if they are bound to a non-local address with no token configured, rather than starting open. --- ### How the six fit together ```text one write log | v two shapes of one truth ---------> the lake (theme 3) (Postgres table, Iceberg table) | | v v views stay current (theme 4) one database per tenant (theme 1) | | v v consent moves a copy the wall (theme 2) <------------------- across it (theme 5) | v every reader goes through it the same way --- people and agents alike (theme 6) ``` The six themes are six views of a small set of mechanisms: one write log, one wall, one way of fixing a read position, one classifier for views, and one shape of consent row. An application built on ArgonDB is not five separate integrations; it is one mechanism seen from five directions. The chapters that go deep on each direction are chapters 1, 2, 3, 4 and 7 — start with the one that matches the feature in front of you. **Limits to weigh before you decide**, all stated in the themes above: - Per-tenant compute and a per-tenant timeline now ship, but off by default, behind one operator switch; moving a company that already holds data onto that shape is not yet built. - The box operator is inside every wall. With per-tenant compute off, the default, the timeline-limited storage token does not separate companies, because one compute serves them all; the connect wall and the password rule do. - A column type the lake cannot decode is silently absent from the lake copy until you check `publish_status`. - Revoking a grant stops future copies; rows already delivered stay until the operator deliberately pulls them back. --- ## Guide 1 — one database per tenant, and the wall between tenants ArgonDB gives every tenant — every company your software serves — a PostgreSQL database of its own, and then walls that database off in the lake as well: its published tables land in their own storage bucket, under their own key, in their own section of the catalog. Nothing you build on top has to filter by tenant, because another tenant's rows are never in the place being read. When data legitimately needs to be read across tenants, a separate, read-only identity does it, and when it needs to cross from one party to another, the box makes an explicit copy. This chapter is for a developer who has to use that. It covers how it is laid out, how the wall is built, how to create tenants and hand out access, what the refusals look like, and the limits. **Before you build: the limits that matter most.** Each is covered in detail below. - **Today all tenants share one PostgreSQL cluster** — one compute and one write-ahead log. A compute and a timeline per tenant are designed, not built. - **The PostgreSQL side is walled differently from the lake side.** The storage service under PostgreSQL refuses a caller with no valid token, and a company's compute token names only its own timeline, so the service refuses it on any other company's timeline. With per-tenant compute off, the default, one compute serves every company, so that token does not separate companies there (§2.6). - **The operator is inside every wall.** The wall separates tenants from each other, not from whoever runs the box. - **Row-level security does not reach the lake copy** through the catalog and storage door (§2.9). - **This chapter gives no measured ceiling on tenants per box.** The largest estate measured is 80, and reading all 80 at once took about 22.6 seconds just to set up (§3.7). - **This chapter does not cover removing a tenant.** When a tenant database is dropped, the next pass takes it out of the cross-tenant reader; what happens to its warehouse, bucket and storage key is not described here, so check it before you promise a customer deletion. - **This chapter does not cover moving an existing application onto this layout.** If your application today keeps every company in one database and separates them with a `tenant_id` column or row-level security, moving it to one database per company is a data migration you plan yourself; nothing here does it for you. **How to read the examples.** Each example is marked either **RUN** — run for this chapter, with its real output pasted — or **FROM THE CODE** — the exact command, flag, message or output quoted from the source code or its design, and not run for this chapter. No test box was available while this chapter was written, so every example is FROM THE CODE unless it says otherwise, and each one names the box test that proves it. **How to read the pictures.** A solid outline is built and shipped. A dashed outline is designed and measured but not built. The pictures also mark decisions that are still to be made. Where a measurement was not made, or its result is uncertain, this chapter says so rather than rounding it off. --- ### 1. The diagram ![One database per tenant, and the wall between tenants](diagrams/argondb-and-the-erp/01-one-database-per-tenant-and-the-wall.svg) The picture has two halves: on the left, how tenants are laid out on one box; on the right, the wall between them. **The left half** is one PostgreSQL cluster on one box. A tenant is a DATABASE in that cluster (§2.1 explains where this is heading). The three columns are three member databases with the same shape: each has its own tables, its own SQL login and its own lake key, its own place in the lake, and its own set of published tables. Two columns are distributors and one is a manufacturer, and the picture deliberately draws them the same: they are the same kind of thing, told apart by WHICH SCHEMA they publish in, not by a flag. Along the bottom are three facts a box diagram cannot show: - the provisioner runs every 30 seconds and gives each new database its whole wall within one pass; - the primary database is never a tenant and never gets a connection limit; - row-level security separates ROLES inside a database, never companies. **The right half** is the wall. It has three parts, and all three apply together — it is never a choice among them: - **A — catalog authorization.** The catalog refuses to list or read a warehouse the caller is not entitled to, and does not even disclose its name. - **B — a login in front of the catalog.** Authorization with no authenticated caller authorizes nobody, and without a login the box could not set up its own catalog. - **C — a warehouse, a bucket and a credential per tenant.** This is the only part whose boundary is also a STORAGE boundary. It has to exist because an Iceberg namespace has no storage location of its own — measured — so there is nothing smaller than a warehouse for a key to be limited to. Below those three sit the five identities that hold keys and what each can reach. The bottom band shows the one way rows cross between tenants: the box reads every tenant under one read-only identity, makes a COPY into the receiving party's own area, and the receiving party reads that copy under its own key. No query reaches across. --- ### 2. What it is #### 2.1 One PostgreSQL cluster, and a tenant is a database in it **What it is:** today, all your tenants live as separate databases inside one PostgreSQL cluster. Hundreds of tenant databases can share one compute, one write-ahead log, one lake writer and one catalog. What the separate databases buy you is this: *"There is no query in this system that could return another company's row, because another company's rows are not in the database being queried."* That is a property of how the data is laid out, not of every check in every code path. **Where it is heading.** The layout the project has committed to changes this twice over: 1. each tenant database gets its OWN compute, which stops when the tenant is idle and wakes when someone connects; 2. each tenant database becomes its own PostgreSQL instance, on its OWN TIMELINE with its own write-ahead log. "One cluster" then means only the SHARED STORAGE LAYER — one pageserver, one safekeeper and one lake serving every tenant's timeline. So of the four things shared today, two — compute and the write-ahead log — become per tenant. **This is built and off by default** (§5). It is switched on per box by `[compute] per_tenant`, which is the operator's setting and never a default. Everything else in this chapter, including every worked example in §4, describes a box with that setting off — which is what ships today. Chapter 2 covers the per-tenant unit: how it is gated, woken and stood down. Two details the picture could not carry. - **What counts as a database** is defined once, as `datallowconn AND NOT datistemplate`. The tenant wall, the provisioner and the discovery of published tables all use that same definition, so they cannot disagree about which databases exist. - **The primary database** — the one `--source-url` names, `postgres` on a box — is not a tenant. It keeps the box's own warehouse, `argon`, and the box's credential; it never gets a connection limit; and it is excluded from every tenant identity by the same check that refuses to grant access on it. #### 2.2 The member database, and what makes one a manufacturer **What it is:** every tenant database — distributor or manufacturer — has the same set of schemas, and which one it *publishes* in decides which kind of party it is. - A distributor's application tables live in its own schemas (`public`, and whatever the application uses). - A manufacturer's published tables live in `mfr`. - The sharing hub's control tables live in `hub`, and what the hub delivers lands in `derived`. This matters because the cross-tenant view `all_tenants.__
` combines every member database's copy of a same-named table. Without a schema rule, a manufacturer holding `public.sales_order_lines` would be read as a distributor by every sharing flow. One sharp edge. The provisioner creates the `mfr` schema in EVERY member database, distributors included, because there is one provisioning path with no manufacturer-specific branch. So **having the schema does not make a database a manufacturer; publishing tables in it does.** ArgonDB answers that question from the published copies it is actually reading, not from a separate list that could disagree with the data. #### 2.3 Its login and its key — one SQL login, one lake key, 20 connections **What it is:** each tenant database gets a connection limit and a key for its own part of the lake. **The connection limit** is `DEFAULT_CONNECTION_LIMIT = 20`. Every tenant database gets the SAME number; the operator raises it per tenant; and nothing recalculates it as tenants come and go. (Dividing the server's connections evenly among tenants was considered and not chosen.) It is PostgreSQL's own `ALTER DATABASE … CONNECTION LIMIT`, so it behaves exactly as PostgreSQL's does: superusers are exempt, and it counts connections to THAT DATABASE, not connections by a role. **The lake key** is named `argon-tenant-`. The provisioner writes it into `lake-store-credentials.json` in the data directory. The key's name is not secret, on purpose: it appears in the storage service's refusal messages and in request signatures, and an opaque name would turn every refusal into a lookup. (The secret half of the key is secret.) #### 2.4 Its address in the lake — `database.schema.table` **What it is:** a published table's lake address always includes its database, so two tenants' `public.orders` never collide. The address has FOUR parts — branch, database, schema, table — and the database is always present, including for the box's own `postgres`. SQL gives a reader only three name parts, so the first three are written as one quoted name: `lake."br_main.shop.sales".orders`. The quotes are not a style choice. Without them the engine sees four name parts and stops with `Parser Error: NameListToString NOT IMPLEMENTED`, which looks like a broken query rather than a quoting mistake. §4.2 shows how to have the box spell the address for you. #### 2.5 What it publishes — a table comment, and the default is nothing **What it is:** a table reaches the lake only if you mark it. The marker is a table comment: `COMMENT ON TABLE sales.orders IS 'argon:publish'`. Nothing is published by default. And a database gets its whole wall on the next provisioner pass even if it has published nothing yet, because provisioning is triggered by the DATABASE existing, not by its first published table. #### 2.6 The wall's three parts, and which of them is on today **What it is:** the three parts of the wall from §1, with their current state. | part | what it enforces | state today | |---|---|---| | **A** — catalog authorization (OpenFGA behind the Iceberg catalog) | the catalog refuses to list or read a warehouse the caller is not entitled to, and answers 404 rather than 403 so the NAME is not disclosed | built. Default: `openfga` on a box with more than one database, `off` on a single-database box; an explicit setting always wins | | **B** — a login in front of the catalog | every catalog request carries a verified identity; the box holds the signing key and serves the two documents that verification needs, so no separate identity provider is needed | built 2026-09-11 | | **C** — a warehouse, a bucket and a credential per tenant | a key opens one bucket, so the STORAGE SERVICE enforces the wall even against a reader that skips the catalog and addresses `s3://` directly | built; `warehouse_per_database = true` | A and B arrive together, never one without the other. The catalog's own documentation explains why: *"Authorization is only effective if Authentication is enabled."* With authorization on and no login, the catalog is not a wall with a door — it is a locked door with no key cut. The measurement says so exactly: anonymous setup answered 401, anonymous warehouse creation 403, anonymous listing 403. **OpenFGA ships in the release, on both platforms.** It is not something you install yourself: the release build for Linux and the one for macOS each stage the OpenFGA binary inside the release artifact, so a stock multi-database box never meets a boot refusal for a missing authorization engine. **Underneath these three is a fourth surface: the storage layer under PostgreSQL.** Parts A–C wall the LAKE. The transactional side has a wall of its own, and on 2026-09-23 the isolation test found a real gap in it: the box's pageserver (the service that stores PostgreSQL's pages) checked no credential at all, so any process on the machine could read any company's data. The box now signs a token for every compute, for the tail and for its own management calls, and the pageserver refuses a caller with no token or the wrong one. **What that does not yet close:** every company is still a timeline inside the box's ONE pageserver tenant, and the token does not name a timeline, so a company's own VALID compute token could have read every OTHER company's timeline. So the token now names the one timeline it may read. The pageserver refuses it on any other company's timeline, both over its web port and to the page service, and logs the mismatch. This is built, merged 2026-09-24, and the cross-company isolation test's W5 cases prove it: company A's compute token is refused on B's timeline, and reads A's own. Two limits. With per-tenant compute off, the default, one compute serves every company and its token names the box's own timeline, so the connect wall and the password rule separate companies there, not this token. And the box's own tail and management calls hold a token with no timeline that reaches every timeline; no company is given it. The pageserver must be built with this change, because an older one treats the new token as tenant-wide. #### 2.7 The five identities **What it is:** exactly five kinds of identity hold keys to the lake, and each can reach a fixed set of things. §3.4's picture shows each against every bucket. In one line each: - **the box** — the root storage key and the `argondb-box` catalog identity. It reaches every warehouse, and it WRITES; the process that writes the lake runs as the box. The operator is inside every wall, and this chapter says so rather than implying otherwise. - **a tenant** — `argon-tenant-` at the storage service and `argondb-tenant-` at the catalog: one bucket, one warehouse, and read access to it. This is what a tenant's application and its people use. - **the cross-tenant reader** — `argon-reader` / `argondb-reader`: may read and list every tenant bucket, never the box's own bucket, and cannot write anything anywhere. - **a manufacturer group** — `hub-` at the storage service and `argondb-manufacturer-` at the catalog: its own hub bucket only, never a distributor's warehouse. - **`argon_transform`** — a PostgreSQL role, and the only one allowed to write `derived.*` in any database. **What never exists** (the red box in the picture): an identity that can write to two tenants, and data crossing between tenants other than as a copy the box made. #### 2.8 `all_tenants.__
` — reading every tenant at once **What it is:** a view that lets the cross-tenant reader query a table across every tenant as if it were one table. Under the cross-tenant reader, every same-named table across tenant databases appears as ONE view: every tenant's copy combined, plus a `tenant` column holding the tenant's DATABASE name (never the warehouse name — the two can differ if an operator renamed a warehouse in `[lake.warehouse_overrides]`). It is a view, not a copy: each query turns into one scan per tenant of that tenant's own lake files, your `WHERE` is applied inside every scan, and the scans run in parallel. Two properties the picture could only name: - **There is no tenant list to maintain.** Which tenants the reader covers is worked out again on every provisioner pass, using the same check the lake writer uses to find a database's warehouse. There is no configuration key that lists tenants. - **A database without a wall is reported as absent by name** — never quietly included and never quietly dropped. The reader's roster lists it as `absent: — `, and every session prints it when it starts. #### 2.9 Row-level security, and where it stops **What it is:** PostgreSQL's row-level security (RLS) still works inside a tenant's database, but it separates roles within one company, not companies. Because companies are now separate databases, a mistake in a policy is an over-broad grant inside one company, not a leak between customers. **The gap to know about is in the lake copy.** A table's lake copy carries no row policies: PostgreSQL does not allow a policy on a foreign table, and Iceberg has no row-policy concept. Since 2026-09-12 ArgonDB's query service (the read plane, which answers lake reads over Arrow Flight) refuses to scan the lake copy of a table that has row security, naming the table, the role and the remedy. But that refusal is only on that one door. **A reader holding the catalog address and the storage credentials reads every row of every published table, row security or not.** The PostgreSQL role is meant to travel into the read plane; it does not yet reach the sharing hub's own internal reads. The picture lists this as one of its two remaining gaps. #### 2.10 What is shared by the whole box, and what is per tenant **What it is:** some things can be done to one tenant alone; others, today, only to the whole box. The table below is for a box with `[compute] per_tenant` off — which is the default. When an operator turns per-tenant compute on, the first three rows become per tenant. | capability | granularity | why | |---|---|---| | point-in-time recovery of the transactional data | the box | one storage timeline per box | | a rehearsal or upgrade branch | the box | a child timeline copies the whole cluster | | lake freshness | the box | with the setting off, every tenant is on the box's one timeline, so one tenant's write volume holds back every other tenant's watermark | | a fixed read position (a pin) | **per scope** | a scope is a set of tables, and a tenant is one database's tables | | lake time travel | **per table** | a retained snapshot plus replay of the table's change log | | the tenant wall (data; and names, when the catalog login is on) | **per database** | | So "a tenant cannot be rewound on its own" is true of the transactional side and false of the lake side: a pin limited to one tenant reads that tenant at any position in the retention window today, without touching the timeline. Per-tenant timelines need a separate PostgreSQL instance per tenant, because PostgreSQL has no per-database write-ahead log to branch — which is exactly why the per-tenant layout in §2.1 gives each tenant its own instance. That layout is built and off by default; until an operator turns it on, every row above holds. --- ### 3. How it works #### 3.1 One provisioning pass, start to finish ![One provisioning pass, and the table that arrives before its wall](diagrams/argondb-and-the-erp/01a-provisioner-pass.svg) **What it is:** the provisioner is the part of the box that gives every database its wall. It runs on its own every 30 seconds, and you can run it by hand; either way it makes the credential, the bucket and the warehouse for each new database, in a fixed order. **Who runs a pass.** Two callers run the same function. The box's own loop runs it every `DEFAULT_INTERVAL_SECS = 30`, and `argondb lake tenant provision` runs it on demand. It matters that there is only one function: on 2026-09-16, a hand-run `provision` raced the box's own pass over the same 72 new databases, which produced conflict errors (409s) on the reader's catalog grants and a non-zero exit from the command, while the box's own pass finished the job. **Only one pass runs at a time.** Every pass now holds one PostgreSQL advisory lock on the primary database for as long as it runs. The lock key is `7021788497314804855`: the eight ASCII bytes `argon_tw` read as one big-endian number, so you can work it out by hand rather than trust a magic number (§4.8 shows how). A session-level lock was chosen for three reasons: a holder that dies releases it without any cleanup process; PostgreSQL's `pg_locks` joined to `pg_stat_activity` names the holder, so a waiting pass can say who it is waiting for; and the primary is the one database both callers can already reach. A waiting pass waits 900 seconds by default (`PASS_LOCK_WAIT_DEFAULT_SECS`), which you can change with `ARGONDB_TENANT_WALL_LOCK_WAIT_SECS`. When the wait runs out it refuses, naming the holder and stating that nothing was provisioned. Fifteen minutes is not arbitrary: a pass over 80 tenants on a loaded box was measured in minutes. **The order of the steps, and why.** For each identity: 1. **The credential is created and written FIRST.** The storage service watches that file and creates a bucket directory for every bucket it names, so the credential also brings the bucket into being — one owner of the storage layout. 2. **The box also creates the bucket directory itself**, so step 3 does not race the storage service reloading the file. Both paths are safe to repeat. 3. **Only then the warehouse**, because the catalog (Lakekeeper) checks a warehouse's storage settings by writing and reading a test object in the bucket, and that check fails if the bucket does not exist yet. 4. **Success is confirmed by reading the warehouse back** from the catalog — never assumed because the create request did not fail, since re-creating an existing warehouse answers `400`. **The warehouse's storage settings carry the ROOT credential, not the tenant's.** The catalog holds one storage credential for the whole warehouse and uses it for its own test writes and maintenance. A tenant-limited key would work today and break the moment the catalog needed to touch anything else. The tenant's key is for the tenant's own readers — DuckDB, Trino, a raw `s3://` client — which are exactly the readers part C exists to wall, because the catalog gives out an address and never a key. **Every database gets a result, and a refusal affects only that database.** Each outcome is one of four — already provisioned, provisioned, the box's own warehouse, or refused with a reason — and nothing is skipped silently. One database that cannot be provisioned never stops the others. The box test for this puts two valid databases and one invalid name (`zz_bad_name`) into one pass and checks that two are walled, one is named with its remedies, and only the named one produces a non-zero exit. **Tenants and manufacturer groups go through the same steps.** The same code provisions a database and a manufacturer group; the regular pass runs it over the cluster's databases, and the sharing hub runs it over the manufacturers in its ownership table. In the credential file, the `database` field holds each identity's STORAGE KEY — a database's own name, or a manufacturer's prefixed one — so the two kinds cannot collide there, and the provisioner refuses a layout in which they would. #### 3.2 Two more jobs in the same pass **What it is:** besides walling each database, every pass also sets connection limits and keeps the cross-tenant reader up to date. Both work over DATABASES rather than over identities — a manufacturer group has no database to limit — so they run once per pass over the cluster, not once per identity. **The connection limit.** The decision follows three rules, in order: 1. an explicit override is applied EXACTLY, whatever the current value; 2. the default is applied only where no limit is set (`-1`, PostgreSQL's own "no limit"); 3. anything else is left as found — a limit an operator set by hand is never lowered, and never raised either. The wanted value comes from the first of these that exists: `--connection-limit =` on the command line, then `[lake.tenant_connection_limit].` in `argon.toml`, then the default. An override that names a database not in the cluster is reported, not silently ignored. A value PostgreSQL would not accept is refused by name before it reaches `ALTER DATABASE`, and the command line splits on the LAST `=`, so a database name containing `=` still works. Each database's result is one of four — set, overridden, kept, or failed — and again nothing is skipped silently. **The cross-tenant reader.** This runs AFTER the tenants are walled, so a tenant created in this pass is covered by the reader in the same pass. The reader covers every database whose warehouse is not the box's own, found by the same check the provisioner walls with — reused, not rewritten — in database order. Every database that check refused is recorded with its reason. At the storage service, the reader is one credential in the same credentials file, and its list of buckets is rewritten whenever the set of tenants changes, so the key follows the tenants with no restart and no new key. At the catalog, the reader gets read access to each tenant warehouse, compared against the list of grants the previous pass recorded: a new tenant is granted on the pass that walls it, a removed tenant is revoked on the pass that finds it gone, and every change is one log line. **No wall, no reader.** With `[lake].warehouse_per_database` off, every database uses the box's warehouse, so the reader would cover nothing. The pass creates no reader and REMOVES any reader an earlier configuration left behind. This is a deliberate refusal, not an empty result: there is no cross-tenant read without a wall, and ArgonDB says so rather than handing out the whole lake under a read-only label. **A reader failure does not end the pass.** The tenants are already walled; the reader is retried on the next pass, and the failure is one logged line. Every pass refreshes the reader's cached roster, including a pass that changed nothing, because a pass that confirmed the roster is current is as good a reason to trust it as one that rewrote it. #### 3.3 Where a table is published, and what happens if its wall is not ready **What it is:** each published table goes into its own database's warehouse. If that warehouse does not exist yet — usually because the database was created seconds ago — ArgonDB waits and retries rather than putting the table anywhere else. The lake writer looks up the right warehouse for EACH TABLE, from that table's own database. If the lookup fails, it **refuses, defers and says why. It never falls back to the box's warehouse.** The message: ```text database "tenantc" publishes to lake warehouse "tenantc" (bucket "tenantc"); it is not reachable. ArgonDB does NOT fall back to the box warehouse "argon" here: that would put this database's published bytes behind the box credential, which is the tenant wall inverted. If the database was just created, the box's tenant-wall pass provisions its warehouse within one cadence interval — `argondb lake tenant list` names the state and `argondb lake tenant provision` forces a pass. ``` The tail retries on every pass, so a deferral shows up as the same log line repeating, not a single message. Within one 30-second pass the wall exists and the next retry publishes into the tenant's own bucket. In practice the wall is usually up before there is anything to publish, because provisioning starts when the DATABASE is created, not when its first table is published. #### 3.4 The storage service's one rule ![The five identities, and what each one can reach](diagrams/argondb-and-the-erp/01b-principals-and-buckets.svg) **What it is:** the storage service (the S3-compatible store that holds lake files) has exactly one function that decides who may touch which bucket, and every request goes through it. That function, `wall::decide`, has no side effects and answers every possible input. It sits where it does for a measured reason: the S3 library the service is built on (`s3s` 0.14.1) calls one access check per request, after the signature is verified and the request is routed but before its body is read. So one check covers roughly a hundred operations — including ones this service does not implement and ones a future version of the library adds. Guarding each operation by hand would take about twenty edits today and leave a silent hole with the next operation added. **The one thing that check cannot see is a copy's SOURCE bucket** — the request names only the destination. The library provides a separate hook for copies, called with the parsed request, so the copy handler reads the source bucket and applies the SAME rule. A cross-bucket copy is exactly how a wall that guards only destinations gets walked through. The rule's cases, in the order it tests them: - **no access key** → refused: `anonymous request refused: needs a signature on this store`. - **an unknown key** → refused, naming the key. - **the root key** → allowed, always. - **a tenant key** → the requested bucket must be the key's own bucket. A mismatch is refused, naming the key, its bucket, its database and the bucket requested. At the top level of the store, only listing buckets is allowed, and the list is FILTERED to the caller's own bucket. - **the reader key** → a bucket outside the reader's roster is refused by name, with `(the box bucket never is)` in the message; a bucket inside the roster is allowed only for a read operation, and anything else is refused naming the allowed list. The reader's allowed operations are a FIXED list of exactly seven: `GetObject`, `HeadObject`, `ListObjectsV2`, `ListObjects`, `HeadBucket`, `GetBucketLocation`, `GetObjectAttributes`. Anything not on the list is refused by name, so an operation the library adds later does not arrive allowed. `CopyObject` is deliberately not on it: a copy WRITES its destination, and the reader may write nothing. **Why listing buckets is filtered rather than refused:** refusing it would also be a wall, but it would break every S3 client that lists buckets before opening one. Filtering is both a wall and usable. Every OTHER top-level operation is refused by name. #### 3.5 The catalog's half **What it is:** the catalog grants each identity read access to specific warehouses, one grant per identity per warehouse. Each grant is a `select` assignment written through the catalog's permissions API, `POST /management/v1/permissions/warehouse/{warehouse_id}/assignments`, with `{"writes":[{"type":"select","user":"oidc~"}],"deletes":[]}`. That shape was taken from the catalog's own API description rather than guessed, and exercised end to end before it was built. `select` is READ access: a tenant lists and reads its own tables and cannot write them — which is the point, since the lake writer is the only writer and it writes as the box. Revoking is the same call with the identity moved from `writes` to `deletes`. Four measured behaviors you will meet: 1. **A warehouse you may not see answers 404, not 403.** Its name is not disclosed. 2. **The catalog sets up its own authorization store.** Running `lakekeeper migrate` with authorization on creates the OpenFGA store and writes the whole authorization model; there is no provisioning step of ArgonDB's. 3. **The box issues its own login tokens.** The catalog never calls an outside identity provider: it fetches `/.well-known/openid-configuration`, then the signing keys, then checks RS256 signatures. The box holds the signing key and serves those two documents, so the wall costs one new service (the authorization store) and no identity provider. 4. **Revoking a grant does not cancel a token.** It removes the grant, so the identity is refused from its next request on; the token itself stays valid until it expires, and the box keeps no revocation list. With the grant gone, the token logs in and is allowed nothing. `--ttl` limits how long it lives. And one refusal that is a design statement rather than input checking: `argondb lake grant` REFUSES a database whose warehouse is the BOX's warehouse, because that warehouse holds every other database's tables as well, and "granting" there would hand over the whole lake while looking like a per-tenant permission. #### 3.6 A read session that covers every tenant **What it is:** when a session fixes its read position (a pin), it now sees every warehouse its identity may read — all of them for the box, one for a tenant, every tenant's for the cross-tenant reader. Pinning is done in one place, so changing it changed four things at once: the transform runner, the sharing hub, the agent endpoint's pin tools and the read plane; PostgreSQL's foreign-table access reads through the read plane and followed. A session now holds a set of catalog handles, one per warehouse — the same thing the lake writer already had — and registers every lake table across every warehouse its identity may read, each through its own database's warehouse. The identity decides the reach: the box sees every warehouse; a tenant sees exactly its own; the cross-tenant reader sees every TENANT warehouse — every warehouse except the box's — with absent tenants named. **Read-only means no writer in the session.** `argondb hub run` is one process holding TWO identities: the reader for reads and the box for writes, and its opening banner names both. Keeping them apart is the point. **What is guaranteed about consistency, exactly.** A pin names a branch, a log position and a scope, among other things, and today the log position is a point in the cluster's ONE write-ahead log, which every tenant database shares — so one position names one moment for every table in every warehouse. Within a table, nothing changes. What is NOT promised is that a cut across two warehouses is atomic: a cut covering two tenants is one transaction PER WAREHOUSE, so two warehouses' watermarks can move at different moments. A pin across tenants therefore lands at the newest position that every table in scope, in every tenant warehouse, can serve, and the session reports per warehouse which position it used. **That "one write-ahead log" is today's layout, not where it is heading.** With per-tenant compute built and `[compute] per_tenant` on, each tenant database gets its OWN write-ahead log on its OWN timeline. One position naming one moment everywhere holds today because that setting is off by default, and turning it on is the operator's decision. #### 3.7 The cross-tenant view, and what it costs **What it is:** how `all_tenants.__
` is put together, and how long it takes at 8 and 80 tenants. The view is built when the session pins, from the cached roster and each warehouse's list of tables. Tables are grouped by `(schema, table)` from the catalog's own structure rather than by re-parsing names, because the flat names are not guaranteed unique. Each tenant contributes one branch: `SELECT '' AS "tenant", FROM ""`, joined with `UNION ALL` in tenant order, with columns matched BY NAME in the order they are first seen. **Filtering is done by the query engine, not re-implemented.** A `WHERE` on the view is pushed by the engine (DataFusion) through the union into every branch and on into each branch's Iceberg scan, where files and row groups are skipped using that tenant's own statistics. Because `tenant` is a constant in each branch, `WHERE tenant = 'x'` becomes `false` in every other branch and those branches are dropped before any file is opened. That is how choosing a subset of tenants with `WHERE` costs nothing extra. **Differences between tenants are reported twice.** A missing column appears as `NULL AS "col"` on that tenant's line beside its `'tenant' AS tenant`, so `EXPLAIN` names the tenant and the column together; and every view with such a difference is listed in the warning shown when the pin is made and on every result the session returns, so a client that never reads a plan still sees it. A tenant with NO copy of the table contributes no branch and is named the same way. A column whose TYPE differs is named per tenant with both types: the engine widens it when it can (`Int32` with `Float64` reads as `Float64`), and the view is REFUSED BY NAME when it cannot (`Int32` with `Boolean`) — never silently narrowed. A copy that already has a column named `tenant` is refused the same way, because the added column would hide it. **The measured cost** (re-measured 2026-09-19 on the box test for these views): | estate | whole pin | catalog listing | table metadata | |---|---|---|---| | 8 warehouses, 8 lake tables | 2307 ms | 1702 ms | 261 ms | | 80 warehouses, 80 lake tables | 22575 ms | 16123 ms | 2974 ms | The split is the finding: about 70 % of a pin is walking the catalog — seven round trips per warehouse (one namespace check, plus a table listing and a namespace listing at each of the branch, database and schema levels) against one metadata load per table, at about 29 ms per round trip at 80 warehouses. Narrowing the set of tables saves the smaller part; narrowing the walk saves the larger one. **Earlier figures for the same measurement — 17122 ms at 8 and 152654 ms at 80 — DO NOT REPRODUCE**, and nothing between the two measurements was a deliberate optimization of this path. They are kept in the design record, not as a baseline. If you are sizing a deployment, measure it yourself. A pin limited to named tables (`argondb-query --scope`) walks only the namespaces those names could live in — three round trips instead of seven for a warehouse holding no database the scope names. **Across tenants this saves nothing, and that was measured, not assumed:** the name a cross-tenant read uses is `all_tenants.__
`, which names no database, so every database level has to be walked. The saving applies when a scope names FEWER DATABASES than the roster holds. #### 3.8 The spellings of one table's identity ![One relation, its spellings, and the door that answers each](diagrams/argondb-and-the-erp/01c-identity-three-spellings.svg) **What it is:** one published table has several names, each for a different job, and the box will spell any of them for you. There are four coordinates (branch, database, schema, table) and only three SQL name parts. The shape is fixed: full nesting, the primary database named like any other, and `__` as the separator in flat names. So `postgres.public.orders` on branch `main` has: - a catalog identifier of namespace `["br_main","postgres","public"]` with table `orders`; - an engine address of `lake."br_main.postgres.public".orders`; - a flat lake key of `postgres__public__orders`; - a dotted log label of `br_main.postgres.public.orders`. Each has its own job, and none can stand in for another. The **address** is what a DuckDB user TYPES; its namespace is one quoted name because a nested namespace cannot be read any other way. The **lake table** is the FLAT key used by the tail's table listing, PostgreSQL's foreign-table mount and a pin session. The **leaf** is the bare table name a caller talking to the catalog API directly needs. The **label** is what a PERSON and a LOG read — no catalog alias, no quotes — and it exists because the address's quoting makes it useless for matching log lines. **Why this is a command and not just a library function:** a shell test, a starter project and a customer document cannot call a Rust function. A survey found 452 lines in 133 files spelling the lake address, and three of the groups of callers (12 lines in shipped starters, 132 lines in 57 test files, 148 lines in 35 documents) are shell, SQL and prose. `argondb lake address` is what they call instead (§4.2). **What has no database stays at the branch level, in the box's warehouse:** `_argon_gaps`, `_argon_timeline`, and the lake writer's own `dv_` area. A tenant key does not reach them either. **A catalog namespace is a NAMING hierarchy, not a directory hierarchy** — measured, and it is why part C exists. Every namespace the test created reported the SAME storage location; a table's location is the warehouse's location plus the table's own UUID; and the warehouse's storage settings carry `"storage-layout":null`. A credential limited to a namespace-shaped prefix would open nothing, because no such prefix exists. **One thing was not checked:** whether catalog version 0.13.x offers a non-default `storage-layout` that nests files by namespace. The setting exists; it was not tested. --- ### 4. How to use it Thirteen worked examples. Each is marked RUN or FROM THE CODE, and each names the box test that proves it. #### 4.0 `argondb tenant create` — a whole company, in one command — FROM THE CODE **What it is:** the command to use when you are creating a COMPANY rather than just a database. It creates everything the company needs, in a fixed order, and it is safe to run again. §4.1 below is still correct and still supported — it creates the database and lets the provisioner find it — but it leaves the tenant's timeline, its compute record, its login role, its shared identity schema, its identity record, its application process and its own MCP token to you. `argondb tenant create` does all of that in the order the ERP application requires, because creating a company is a sequence where each step depends on the one before: ```bash argondb tenant create acme \ --display-name "Acme" \ --legal-name "Acme Holdings LLC" \ --system distributors \ --identity "tax_id=12-3456789" \ --identity "contact=ops@acme.example" \ --data-dir "$DATA" --port 5433 ``` | flag | what it is | required | |---|---|---| | `` | the tenant's permanent key, never reused — and, on a box, its database name. Lowercase ASCII letters, digits and underscores | yes | | `--display-name` | the company's name as people write it (the identity record's `display_name`) | no — see below | | `--legal-name` | the company's legal name (the identity record's `legal_name`) | no — same | | `--system` | which tenancy the tenant belongs to (`distributors`, `manufacturers`) — a tenancy is one multi-tenant system in the Estate, and the CLI's word for it is `system` | no (`distributors`) | | `--identity KEY=VALUE` | the rest of the company's identity. Repeatable. The key must be one of `address`, `tax_id`, `contact` | no | | `--sidecar-command COMMAND` | the command that runs the company's own application as its own process (step 8 below) | no — leave it out and the company runs no application | | `--sidecar-uid UID` | the unix user that process runs under | no — defaults to the company's own system user | | `--direct` | also print the direct connection string, not just the pooled one a tenant is handed by default | no | | `--data-dir`, `--port` | which box, found the same way every other subcommand finds it | no | **The two name flags are now OPTIONAL, and a name left out is written empty — never invented, and no longer refused.** Earlier builds refused a create with no name; that changed. A company whose human has not answered yet is created with its key and empty names, the run names the flags you left out, and the tenant still ends up live. What IS still refused, before anything is created, is an `--identity` key this chapter's table above does not list — `country=US`, for example — because there is nowhere to put a key nobody defined, and dropping it silently is how an address goes missing. **One thing to know:** the command's own `--help` text for the two name flags still says they are required; the behavior above is what the command actually does. **The eight steps.** The command prints each as it goes: | # | step | what exists afterwards | |---|---|---| | 1 | its **timeline** | a branch of the box's timeline, recorded in `/tenants/acme/identity.json` | | 2 | its **compute record** | a row in the per-tenant compute table — its own slot in the tenant port range and its own directory | | 3 | its **database** | `acme` in `pg_database` | | 4 | its **role and grants** | `acme_owner` in `pg_roles`, granted on the database and on `public` | | 5 | its **shared identity schema** | the same `identity` table shape every tenant gets, brought up to the version this build ships | | 6 | **the wall** | exactly what §4.1 does — warehouse, bucket, limited credential, catalog grant, connection limit | | 7 | its **identity row** | your two names and identity fields, written into `acme`'s own `identity` table AND into its system's shared Identities, in the same step | | 8 | its **application process** | the company's own application, running under its own user — or nothing at all, if you left out `--sidecar-command` | **Measured (2026-10-30, on compose-2's node, with per-tenant compute on and the estate pass and maintainer running):** the whole command took 3.4 s on a fresh box with one system and 4.4 s at 300 tenants, with the compute start (step 2) about 1 s and the wall pass (step 6) under 1.1 s at every count, and no step ever reaching 60 s. After all eight steps the command also mints the company's own MCP token and writes it to a private file only the box can read (§4.11). If that one step fails, the tenant is still there and nothing is undone; running `argondb tenant mcp-token acme` afterwards mints the token on its own. **Step 7 has one door.** A company's identity row can only be written by the tenant creation and update commands themselves, and its permanent key cannot be changed at all once set. To rename a company, use `argondb tenant identity acme --display-name …` — the same door, after the fact. **RUNNING IT AGAIN IS SAFE.** Every step checks before it acts, so a second run creates nothing and says so: ```text [argondb] step 1/8 timeline — already present (§13.4 step 5); not applied ... [argondb] tenant create: acme is live — 0 step(s) applied, 8 already present. [argondb] tenant create: NO-OP — this run created nothing (every step probed present). §13.4 idempotence. ``` (The `§13.4` in those lines refers to the ERP application's specification of this sequence.) **A RUN THAT STOPS HALFWAY IS FINISHED BY RUNNING IT AGAIN.** Progress is kept in a step journal at `/tenants/acme/create-journal.json`, and a half-built tenant says so in its own `status` column (`provisioning`, not `live`). Run the same command again and it picks up where it stopped: ```text [argondb] tenant create: RESUMING at step 4 (roles) — the journal names step 3 (database) as the last completed step ``` **WHAT IT WILL NOT DO: take over an existing database.** If a database of that name already exists and this command did not create it, it REFUSES and points you at the provisioner. That is not a dead end — the provisioner walls an existing database within one pass, which is §4.1: ```text refusing: a database of this name already exists and this sequence did not create it (there is no step journal beside it). `argondb tenant create` CREATES; it does not adopt. ``` **What a single box does NOT give you.** The command's own banner says it: on one development box there is one safekeeper, no write-ahead-log backup and no spread across availability zones. A quorum of one is not redundancy. Every step that means nothing on one box — the tenant's own cloud instance, its subnet and security group, its volume, and its fixed address, plus the three-zone safekeeper set — is printed as SKIPPED, with what stands in for it on one box, so nothing is missing silently. Writing to the shared Identities and running the application process used to be on that skipped list too; they are now real steps (7 and 8 above) even on one box. **Where the control plane runs — never alongside a tenant.** The estate control plane (the provisioning API, the estate directory and the identity projection, plus the lease that decides which instance runs the shared tail) is designed to run on its own estate instances, following the storage layer's own practice: at least two, across two availability zones — three is the natural number, since the safekeeper quorum already needs three. One is the leader and the others forward requests to it rather than answering them. It never runs on a tenant's own instance, because a control plane sharing a tenant's instance dies with that tenant and cannot be what recreates it. It does not scale to zero: it has to answer a `tenant create` that, by definition, can arrive when nothing else is awake, so it is always running, like the storage layer, rather than stopping when idle like a tenant's compute. This multi-instance layout is designed, not built. On one development box it collapses to a single process and a schema, `argon_estate`, in the box's own primary database; the single-box version is not counted complete until the leader and its forwarding are shown running as two processes on the box. **A defect that is now fixed.** In the v1.5.6 release, the company's shared identity schema was supposed to reach version 2 but the steps that apply it only went up to version 1, so a fresh `tenant create` and the default run of `argondb estate migrate` (§4.9) both stopped with a message saying the build had no step defining version 2. The missing step — the one that makes the identity table writable only through the tenant commands — is now in place, and a box test runs both commands to prove it. **Proof:** the tenant-create box test checks both outcomes of each of the four behaviors above (a create with no name, re-running, resuming, refusing to take over). Its most recent run measured a newly created tenant's identity reaching its whole system in 59 seconds, with no administrator having to do anything. #### 4.1 Create a tenant database and watch the provisioner wall it — FROM THE CODE **What it is:** the minimal way to add a tenant — create a database, and the provisioner gives it a wall within 30 seconds. ```bash # 1. the database. Nothing else is special about it. createdb -p 5433 -h localhost -U cloud_admin tenanta # (or: psql -c 'CREATE DATABASE tenanta') # 2. wait for the box's own pass (<= 30 s), or force one now: argondb lake tenant provision --data-dir "$DATA" --port 5433 # 3. read the result back argondb lake tenant list --data-dir "$DATA" --port 5433 ``` The listing has seven columns on a box with the catalog login on, and six without it: the `CATALOG GRANTS` column is left out rather than filled with "none", because "none" would read as "this box does not have that feature". A sample listing, from the customer guide: ```text $ argondb lake tenant list DATABASE WAREHOUSE BUCKET ACCESS KEY STATE postgres argon argon (the box credential) primary — the box warehouse shop shop shop argon-tenant-shop walled tenantb tenantb tenantb argon-tenant-tenantb walled ``` Every STATE is a sentence that tells you what will happen next: | credential and warehouse | STATE | |---|---| | both present | `walled` | | neither | ``not yet — the provisioner walls it within 30s, or run `argondb lake tenant provision` `` | | credential only | `credential minted, WAREHOUSE MISSING — the last pass could not reach …` | | the database name was refused | `REFUSED — ` | Each database also gets its own schemas and tables for transforms and views, from a separate provisioning step on the same 30-second schedule. Check it rather than assume it: ```bash psql -d tenanta -c "SELECT to_regnamespace('derived') IS NOT NULL AND to_regnamespace('mfr') IS NOT NULL AND to_regclass('derived.transform_runs') IS NOT NULL AND to_regclass('argon.view_state') IS NOT NULL" ``` That is exactly the check the box itself runs, and a unit test confirms it asks for precisely what `deploy/sql/provision-transform-role.sql` creates, so the two cannot drift. One trap that once hid a gap here: `CREATE ROLE argon_transform LOGIN` creates a CLUSTER-wide role, visible from every database, so asking only "does the role exist?" answers yes even for a database that was never set up. **Proof:** the tenant-wall box test checks the credential, the bucket directory, the warehouse, a different key for each tenant, a REFUSAL for the primary database, and which bucket the published files actually land in — and runs every check again after a box restart. #### 4.2 The spellings of one table's identity — FROM THE CODE **What it is:** `argondb lake address` spells a table's lake identity for whichever job you need. Ask the box; do not assemble it by hand. These examples are quoted from the command's own documentation: ```text $ argondb lake address postgres.public.orders lake."br_main.postgres.public".orders $ argondb lake address --field namespace postgres.public.orders br_main.postgres.public $ argondb lake address --field lake-table postgres.public.orders postgres__public__orders $ argondb lake address --field qualified postgres.public.orders "public"."orders" $ argondb lake address --field label postgres.public.orders br_main.postgres.public.orders $ argondb lake address --changes postgres.public.orders lake."br_main.postgres.public".orders__changes $ argondb lake address --changes --field lake-table postgres.public.orders postgres__public__orders__changes ``` `--branch` changes the branch, and `--format json` answers in a form a program can read: ```text $ argondb lake address --branch dev shop.sales.orders lake."dev.shop.sales".orders ``` Which field to use: - `address` — for a DuckDB user, and for documentation; - `namespace` — for a starter project's `LAKE_NAMESPACE` constant; - `leaf` — for a caller talking to the catalog API directly; - `lake-table` — for a pin scope, the PostgreSQL foreign-table mount and the tail's table listing; - `label` — for matching log lines; - `qualified` — for the tail's own ` left the published set` lines; - `bucket` — for "which bucket do this table's files land in?". **The failure you will hit if you assemble it yourself:** ```text D SELECT count(*) FROM lake.br_main.postgres.public.orders; Parser Error: NameListToString NOT IMPLEMENTED ``` Four name parts, and SQL allows three. The quotes are not optional. #### 4.3 Give an analyst a tenant token, and see what it can and cannot list — FROM THE CODE **What it is:** how you give a person or tool read access to one tenant's lake, and take it away. ```bash # create the token and grant access in one call; the TOKEN is the only thing on stdout TOKEN="$(argondb lake grant tenanta --data-dir "$DATA" --ttl 30)" # the ATTACH statement a reader pastes, created and granted the same way argondb lake attach-clause --database tenanta --data-dir "$DATA" # take it back argondb lake revoke tenanta --data-dir "$DATA" ``` What the grant prints on stderr: ```text [argondb] granted `select` on warehouse tenanta () to oidc~argondb-tenant-tenanta, valid 30 day(s). [argondb] the analyst reads it with: ATTACH 'tenanta' AS lake (TYPE iceberg, ENDPOINT '/catalog', AUTHORIZATION_TYPE 'oauth2', TOKEN ''); [argondb] revoke it with: argondb lake revoke tenanta ``` **What such a token can do, measured** (each line is one check from the measurement run; the codes at the left are that run's own labels): ```text C4a/C4b tenant principals BEFORE any grant HTTP 403 / HTTP 403 C6a/C6b grant select on one warehouse each HTTP 204 / HTTP 204 C7a tenant-a lists warehouses HTTP 200 -> tenanta ONLY C7b tenant-b lists warehouses HTTP 200 -> tenantb ONLY C7d tenant-a reads tenantB config HTTP 404 C7e tenant-b reads tenantA config HTTP 404 C7f/C7g committer reads BOTH configs (the control) HTTP 200 / HTTP 200 C7h ANONYMOUS reads tenantA config HTTP 401 C8a FORGED token, right shape wrong signer HTTP 401 ``` Read the 404s: a warehouse this identity may not see is not merely refused; its NAME is not disclosed. And read each refusal as meaningful only because its matching success passed in the same run on the same box. **The refusal you will meet if you point it at the primary database:** ```text database "postgres" resolves to the BOX warehouse "argon", which holds every other database's tables as well — granting there would hand the caller the whole lake while reading like a per-tenant permission. Turn on `[lake] warehouse_per_database` (it is on by default) and let the provisioner wall this database first: `argondb lake tenant provision`. ``` **Revoking does not cancel the token.** It removes the grant, so the identity is refused from its next request on, while the token stays valid until it expires; `--ttl` is what limits that. #### 4.4 Read every tenant at once, and see a database without a wall named — FROM THE CODE **What it is:** how the cross-tenant reader queries all tenants together, and what it tells you about tenants it cannot cover. ```bash argondb lake reader credential --data-dir "$DATA" # the roster, the grants, a token eval "$(argondb lake reader credential --data-dir "$DATA" --env)" # exports ARGONDB_LAKE_WAREHOUSES argondb-query "SELECT tenant, count(*) FROM all_tenants.public__orders GROUP BY tenant ORDER BY tenant" ``` What the credential command prints: ```text $ argondb lake reader credential principal cross-tenant reader (read-only over every tenant warehouse, never the box warehouse) access-key argon-reader secret-key <48 hex characters> endpoint http://127.0.0.1:9011 region local catalog http://127.0.0.1:8181/catalog warehouses shop, tenantb buckets shop, tenantb granted shop, tenantb pass-unix 1789514071 token-ttl 30 day(s) token ``` And the query session. Note that the banner, the count of tenants and the note about differences all arrive before the rows: ```text $ eval "$(argondb lake reader credential --env)" $ argondb-query "SELECT tenant, count(*) FROM all_tenants.public__orders GROUP BY tenant ORDER BY tenant" -- pinned at 0/1C2A3F0 (min watermark 0/1C2A3F0) -- note: this session resolves 4 warehouses — tenanta, tenantb, tenantc, tenantd — … -- pin built in 412 ms over 4 warehouse(s), 5 lake table(s) -- all_tenants: 2 view(s) served over 4 tenant(s) [tenanta, tenantb, tenantc, tenantd]: all_tenants."public__orders", all_tenants."public__other" -- note: 2 cross-tenant view(s) DRIFTED — a row from a named tenant is unioned by column name with NULL for what that tenant lacks, and a tenant with no copy contributes no rows: all_tenants."public__orders" — tenant tenantc lacks column(s) "v" (served as NULL); tenant(s) tenantd have no copy of this table (no branch) | all_tenants."public__other" — tenant(s) tenanta, tenantb, tenantc have no copy of this table (no branch) +---------+----------+ | tenant | count(*) | +---------+----------+ | tenanta | 3000 | | tenantb | 3000 | | tenantc | 50 | +---------+----------+ ``` **A database without a wall is listed as ABSENT, by name.** `argondb lake tenant list` prints a line for the reader, with absent databases shown as ` ()`: ```text cross-tenant reader: argon-reader covers [shop, tenantb] catalog grants [shop, tenantb] absent [zz_bad_name ()] ``` The other two forms of that line are just as explicit: ```text cross-tenant reader: none — [lake].warehouse_per_database is off (no wall, no reader) cross-tenant reader: not yet minted — the provisioner mints it on the pass that walls the first non-primary database (within 30s), or run `argondb lake tenant provision` ``` **Choosing a subset of tenants is a `WHERE`, and it costs nothing.** `WHERE tenant IN ('tenanta','tenantb')` becomes `false` in every other branch, and those branches are dropped before any file is opened. There is no named set of tenants to configure. **The primary database is not in the view.** The session never sees the box's warehouse, so `postgres__public__house` answers "table not found". Adding the box's warehouse to the roster by hand is refused by the catalog under the reader's login, and the session says which warehouse refused. **Proof:** two box tests. The cross-tenant reader test (8 phases) checks the storage wall, the catalog, a session, a tenant joining and leaving the roster, and a restart. The cross-tenant views test (8 phases) checks the view, that filters are pushed down (read from `EXPLAIN`), that nothing is written, the cost measurement, a memory limit, and a DuckDB reader. #### 4.5 Set, override and read back a connection limit — FROM THE CODE **What it is:** each tenant database has a connection limit; here is how to change it for one tenant. ```bash # the default (20) is applied only where datconnlimit = -1 argondb lake tenant provision --data-dir "$DATA" --port 5433 # an explicit override, applied EXACTLY, on this pass and every later one argondb lake tenant provision --data-dir "$DATA" --port 5433 --connection-limit tenantb=50 ``` or, to make it permanent, in `argon.toml`: ```toml [lake.tenant_connection_limit] tenantb = 50 ``` The command-line value wins over the file, and both win over the default. `argondb lake tenant list` has a `CONN LIMIT` column so that an unset or hand-raised limit is visible without reading the box's log; the primary database's cell reads `unlimited`. **Proof:** the tenant-wall box test checks both outcomes of each rule. As a non-superuser, 20 sessions on tenant A are held and the 21st is refused, NAMING THE DATABASE; tenant B still connects, as does a superuser (PostgreSQL's documented exception). A command-line override wins and is printed, with the other tenant untouched. A hand-set value survives a plain pass, neither lowered nor raised. An `argon.toml` override wins, is printed, and is printed again as "in force" on the next pass. A malformed value, and an override naming no database, are refused by name with nothing applied. And the upgrade case: a tenant with no limit gets 20 on the next pass, and the pass says so. Two things to know before you rely on the number. It is PostgreSQL's own limit, so **superusers are exempt**, and **it counts connections to that DATABASE, not by role**. And `default = -1` is a valid setting meaning "apply no default" — an operator who wants PostgreSQL's unlimited behavior back says so with `-1`. #### 4.6 Turn the catalog login on, and read a refused cross-tenant access — FROM THE CODE **What it is:** how to turn on catalog authorization (parts A and B of the wall), and what a refused access looks like at the catalog and at storage. ```toml # argon.toml — note the DEFAULT already works this way on a # multi-database box, and an explicit value always wins [lake] catalog_authz = "openfga" ``` The default, since 2026-09-15: `openfga` on a box holding more than one database, `off` on a single-database box; an explicit value always wins. The box says which it is running, and says so again if the number of databases has crossed one since it started. The box's loop and the listing command use the same check, so they cannot disagree: ```text catalog authz: (as decided at boot) <…and, when the database count has crossed 1 since boot, that the change takes effect at the next boot> ``` With the login OFF, the box's own message states the position exactly — it `authenticates nobody and authorizes everybody`. Data is walled; table names are not. **What a refused cross-tenant read prints.** At the storage service: ```text access key "argon-tenant-tenanta" is scoped to bucket "tenanta" (database "tenanta") and GetObject addresses bucket "tenantb" access key "argon-reader" is the cross-tenant READER over ["tenanta", "tenantb"] and GetObject addresses bucket "argon", which is not a tenant bucket in its roster (the box bucket never is) access key "argon-reader" is the cross-tenant READER and is READ-ONLY: PutObject on bucket "tenanta" is refused (allowed: GetObject, HeadObject, ListObjectsV2, ListObjects, HeadBucket, GetBucketLocation, GetObjectAttributes) access key "argon-reader" is the cross-tenant READER (read-only over 2 tenant bucket(s)) and CreateBucket addresses the whole store ``` Each reaches the client as `403 AccessDenied` and is logged by the storage service with the key, the operation and both buckets named. At the catalog, the refusal is a `404` and does not disclose the name. **Proof:** the tenant-wall box test runs the whole matrix, both outcomes of each, and again after a restart: key A on bucket A allowed; key A on bucket B refused, for listing AND for reading; the same the other way round; key A on the BOX bucket refused; the root key allowed on every bucket, as the control; bucket listing filtered per key; anonymous refused. #### 4.7 Set up a manufacturer, and see the party rule catch a mistake — FROM THE CODE **What it is:** a manufacturer is an ordinary member database; publishing in the `mfr` schema is what makes it one, and publishing anywhere else is caught. ```bash createdb mfracme # the box's pass walls it exactly as it does a distributor psql -d mfracme -c "SELECT to_regnamespace('mfr') IS NOT NULL" # provisioned for you; you do not create it ``` ```sql CREATE TABLE mfr.price_list( upc text PRIMARY KEY, list_price double precision, effective_from date, effective_to date, manufacturer_id text); COMMENT ON TABLE mfr.price_list IS 'argon:publish'; CREATE TABLE mfr.share_grant( flow text NOT NULL, manufacturer_id text NOT NULL, -- a recipient, or '*' contract_version int NOT NULL, granted_at timestamptz NOT NULL DEFAULT now(), revoked_at timestamptz, -- NULL = in force PRIMARY KEY (flow, manufacturer_id, contract_version)); COMMENT ON TABLE mfr.share_grant IS 'argon:publish'; INSERT INTO mfr.share_grant(flow, manufacturer_id, contract_version) VALUES ('upc_prices', '*', 1); ``` Then ask which parties the box sees: ```text $ argondb hub parties pin 0/3A1B2C8 — 4 member database(s), 1 manufacturer(s) hub: party hubcontrol: distributor (publishes no mfr table at this pin) hub: party mfracme: manufacturer (publishes 2 mfr table(s) at this pin) hub: party tenanta: distributor (publishes no mfr table at this pin) hub: party tenantb: distributor (publishes no mfr table at this pin) ``` Every member database gets a line, distributors included — so "no manufacturers" and "did not look" can never look the same. **The mistake, and its one fix.** A manufacturer database that publishes outside `mfr` becomes one of the branches of the distributors' own cross-tenant views. The box names it and exits non-zero: ```text hub: CROSSES PARTIES mfracme.public — served as "mfracme__public__products", a branch of all_tenants.public__products ``` …and an `argondb hub run` whose source reads that branch is REFUSED before anything is delivered. The check asks whether the query *refers* to that branch, not whether the branch would be filtered out: the query's plan is checked before it is optimized, so adding `WHERE tenant IN (…)` does NOT get past it. The fix is to publish the table in `mfr`. A source that reads only a distributor's own tables refers to no manufacturer copy and is unaffected. **Proof:** the sharing-hub box test and the cross-tenant views box test each check both outcomes. #### 4.8 Find out who holds the provisioner's lock — RUN (the key) + FROM THE CODE (the SQL) **What it is:** if a provisioning pass is waiting, this is how you see which process holds the lock. `argondb lake tenant list` prints the lock's state itself: ```text provisioner lock: pg advisory key 7021788497314804855 on the primary — free / HELD by … ``` To ask PostgreSQL directly you need the key's two halves, because `pg_locks` stores a single-number advisory key as `classid` (the high 32 bits) and `objid` (the low 32 bits), with `objsubid = 1`. You can work them out by hand, which is why the key is eight ASCII bytes rather than a magic number — **RUN for this chapter**: ```text $ python3 -c 'k=int.from_bytes(b"argon_tw","big",signed=True); print(k, hex(k>>32), hex(k & 0xffffffff))' 7021788497314804855 0x6172676f 0x6e5f7477 ``` That matches the registered values (`classid = 0x6172676f`, `objid = 0x6e5f7477`, `objsubid = 1`). The query the box itself runs — FROM THE CODE: ```sql SELECT a.pid, coalesce(a.application_name, ''), coalesce(a.state, ''), to_char(a.backend_start, 'YYYY-MM-DD HH24:MI:SS TZ') FROM pg_locks l JOIN pg_stat_activity a ON a.pid = l.pid WHERE l.locktype = 'advisory' AND l.granted AND l.objsubid = 1 AND l.classid = (7021788497314804855 >> 32)::oid AND l.objid = (7021788497314804855 & x'ffffffff'::bigint)::oid ORDER BY a.pid LIMIT 1; ``` The holder's `application_name` tells you which caller it is: the box's own pass shows `_tenant_wall`, and the command line shows `argondb_lake_tenant_cli`. If a pass over 80 tenants is ahead of you, raise the wait with `ARGONDB_TENANT_WALL_LOCK_WAIT_SECS`. **Proof:** the provisioner-exclusivity box test, 6 phases, both outcomes. #### 4.9 Bring every tenant's shared schema up to date in one pass — FROM THE CODE **What it is:** one database per tenant means a company created last month and a company created today can be at different versions of the identity schema they all share. `argondb estate migrate` brings every tenant on the box to one version in one pass, using the exact same per-database step that `tenant create` step 5 uses, so no tenant is brought up to date by a different path than the others. ```bash # bring every tenant to the version this build ships argondb estate migrate --data-dir "$DATA" --port 5433 # or name a version yourself argondb estate migrate --to 2 --data-dir "$DATA" --port 5433 ``` One pass, in order: 1. it looks at the estate's own list of tenants and checks each one still has a real database. A listed tenant whose database is gone is refused by name; a database with no listing is left alone — adopting it is a decision an operator makes, not something a pass does on its own. 2. it asks each tenant its own version, from that tenant's own database — never from the estate's list, so "never migrated" and "already at version 0" are not confused with each other. 3. it decides, for every tenant, whether to apply, skip or refuse — before changing anything. A tenant already at or past the target is skipped and counted. A target this build does not ship is refused for every tenant before any schema moves. 4. it applies the change. One company failing does not stop the rest — the other tenants are still migrated. 5. it records what happened: one row per tenant, and only after reading the version back out of that tenant's own database, so the record can never claim more than really landed. Every pass ends with one line of counts, zeros included, so "nothing refused" is said outright rather than left to be inferred: ```text [argondb-box] estate migrate --to 2: 2 tenant(s) applied, 1 already at target (skipped), 0 refused (named, pass continued), 3 row(s) walked ``` The command's exit code is non-zero if anything was refused, but only after every other tenant has already been migrated and every refusal named. Running it again at the same target changes nothing. **What it does not do.** It is a command an operator runs, not something that happens on its own timer — changing every company's schema at once needs a person deciding to. And it does not try the change on a copy first before doing it for real; that rehearsal step is designed, not built. **Read the limit at the end of §4.0 before you rely on the default target** — this build's own migration steps do not yet reach the version it targets by default. **Proof:** the box test for migrating every tenant's schema — three tenants at two different versions, one deliberately broken in the middle with the tenants on either side still migrated, a re-run that changes nothing, and an unshipped target refused for everyone. #### 4.10 Create a tenancy, and let it publish its Identities to every other tenancy — FROM THE CODE **What it is:** every tenant belongs to a TENANCY, one multi-tenant system in the Estate — the Distributor Tenancy `distributors` or the Manufacturer Tenancy `manufacturers` in the examples above. The CLI's word for a tenancy is `system`, and the command stays `argondb system create`. Creating the tenancy itself is its own command, and like `tenant create` it is ordered, safe to re-run, and picks up where it stopped. ```bash argondb system create manufacturers --data-dir "$DATA" --port 5433 ``` | # | step | what exists afterwards | |---|---|---| | 1 | register | the system's own row in the estate's list | | 2 | Identities | the system's own shared Identities database and its place in the lake | | 3 | reader | the system's own cross-tenant reader, scoped to its own tenants only | | 4 | publish | every OTHER already-registered system's reader can now read THIS system's Identities | | 5 | live | the system is marked live | **New in this release: step 4 grants access in the same call it runs in.** Before, step 4 only granted access to Identities that already existed, so a system created moments earlier printed "deferred," granted nothing, and marked itself done anyway — leaving an operator to run the grant by hand later with no clear signal for when. Step 4 now creates the Identities place itself if it is missing and grants access in the same step. And if a create is interrupted after step 3, running it again now correctly re-does step 4 instead of skipping it. **Step 4 only publishes in ONE direction, and this is the part to get right.** It grants every OTHER, already-registered system read access to THIS system's Identities. It does NOT reach back and grant this new system access to systems that came before it — the message it prints reads as if it might, but it only covers the new system's own Identities. So the first tenancy you ever create publishes to nobody — create the Distributor Tenancy first — and if you want a second tenancy to read the first one's Identities, you grant that by hand: ```bash argondb system grant-identities --from distributors --to manufacturers argondb system identities-grants # every grant on record, live or revoked argondb system revoke-identities --from distributors --to manufacturers ``` A revoke stops the OTHER system's reader from the next request on; whatever that system already copied into its own schema before the revoke stays — it is that system's own data now. **What is printed as not built, rather than faked.** Step 4's cross-account half — the kind of grant that would let a completely separate cloud account read the Identities — has nothing to attach to on one box with no second account, so every run prints that it is not built there. And every system's reader is one shared credential, not one key per system. **Proof:** the box test for creating a system — one check that step 4 actually created the Identities place, one check that an interrupted create resumes at the right step, and a check that the system boundary really holds by trying, and failing, a read across it. #### 4.11 Give each company its own MCP token — FROM THE CODE **What it is:** before this release, the box had exactly one MCP token, and anything you handed a company's agent with that token could read every company on the box. Now every company has its own token, and a session under it can only read that one company's database. ```bash # tenant create mints the first one, into a private file only the box can read (§4.0) cat "$DATA/tenants/acme/mcp-token" # mint a fresh one: prints the new token, and every earlier token for acme # stops working from the next request on TOKEN="$(argondb tenant mcp-token acme --data-dir "$DATA")" # take away every token acme has argondb tenant mcp-token acme --revoke --data-dir "$DATA" ``` **What the token can do.** A session under a company's token reads through that company's own read-only role, on that company's own database only. So a request for another company's table is refused by PostgreSQL itself, at the connection, not by a filter that could be bypassed. Only one token per company is live at a time: minting a fresh one immediately retires the last one, and revoking removes it and turns the company's read-only role off. **What it refuses, in its own words:** ```text company token refused: its signature does not match its claim company="acme" (forged or re-labelled) company token refused: company "acme" has no live MCP token (revoked, or never minted) company token refused: company "acme"'s token was revoked by a re-issue (this is not its live token) ``` Two things a session under a company's token cannot do at all: search across the box, and ask for publish status — both would read box-wide state, so both are refused by name rather than answering with an empty, company-scoped version. Every response also states plainly whether it was served at operator scope or at one company's scope, so a client can always tell which it got. **Where this applies.** Company tokens work on the ordinary agent surface only; the administrator surface still uses the one box-wide token, and company tokens are not accepted there at all. This has to be turned on by the operator — with it off, company tokens are not accepted anywhere. **Hand a company's agent its own token, never the box's own token** — the box token still reads every company. And a company token reads PostgreSQL only: the lake side of carrying a role's identity into reads (§2.9) is not built, so a company session cannot read any lake table yet, with one exception. Its `query` tool reads the company's OWN `
__changes` change logs. Those reads are read-only and stay under the tenant wall (guide 4, §3.4). **Proof:** the box test for cross-company isolation across eight different surfaces, both with a company token and without one. #### 4.12 Hand each manufacturer its own copy of the shared totals — FROM THE CODE **What it is:** when the sharing hub computes a shared total across every distributor (`--aggregate `), that total used to land only in the box's own area, which no manufacturer's own key can open. Now each manufacturer also gets its OWN COPY, in its own area, readable under its own key, with no need to cross the wall to see it. ```bash argondb hub run --source "$SRC" --ownership "$OWN_TBL" --flow upc_prices \ --hub-table upc_hub --aggregate upc_hub_totals \ --data-dir "$DATA" --port 5433 ``` **A manufacturer's copy counts only what it was actually granted.** Each manufacturer's copy adds up exactly the distributors whose grant of that data flow to that manufacturer was in force at the moment of that run — nothing is re-checked afterwards. Two situations land nothing at all for a given manufacturer, not even an empty table, and the run says exactly why: ```text hub: aggregate copy for : NOT LANDED — no-flow: this run names no --flow, so no consent plan exists and no tenant granted "" anything; a copy of the aggregate is handed only under a grant hub: aggregate copy for : NOT LANDED — no-consent: no tenant's grant of flow "upc_prices" to "" is in force at pin , so "" receives no copy of the aggregate ``` Running it again with nothing changed replaces the manufacturer's copy in place rather than adding a second one. And a total that would reveal one distributor or one customer by name is refused, by name, rather than shipped. **The open question, stated plainly.** If only ONE distributor has granted a manufacturer this data, that manufacturer's "shared total" is really just that one distributor's own volume, competitors included — which may or may not be acceptable, and is not yet ruled on. **Proof:** the box test for the manufacturer hub warehouse — the granted manufacturer receives its copy, the ungranted one receives none, and the ledger says why in both cases. --- ### 5. What exists today, and what is designed but not built Before you build on the tenant layout you need to know which parts of this chapter you can use now and which describe where it is going. Everything in §1–§4 that is not marked otherwise exists today. The table below lists every piece once, with its state and, where one exists, the box test that proves it on a fresh machine. | piece | state today | proven by | |---|---|---| | a tenant is a database in one shared PostgreSQL cluster | built | | | part C of the wall — a warehouse, a bucket and a limited storage key per tenant database | built (release 1.5.1) | the tenant-wall box test, phases 1–7 | | the lake writer choosing the catalog per table, so a tenant's files land in the tenant's bucket | built | the same test, phase 7, repeated after a restart | | a table whose warehouse does not exist yet is held back and named, not published | built | | | parts A and B — catalog authorization and a catalog login, with the box issuing its own tokens | built 2026-09-11 | | | catalog authorization on by default when a box holds more than one database | built 2026-09-15 | | | OpenFGA (the authorization service) bundled in the release, both platforms | built | | | the three-part name `database.schema.table` in the catalog, with the primary database named the same way | built 2026-09-11 | | | an old catalog layout is migrated or refused at startup, never guessed at | built 2026-09-11 | | | a connection limit per database, 20 by default | built 2026-09-15 | the tenant-wall box test, phase 10 | | only one provisioning pass at a time (the pass lock) | built 2026-09-16 | the provisioner-exclusivity box test | | the cross-tenant reader — one read-only storage key and one catalog login, covering exactly the tenants that have a wall | built 2026-09-15 | the cross-tenant-reader box test | | `all_tenants.__
` views, with filters pushed into each tenant's scan and differences named | built 2026-09-16 | the all-tenants-views box test | | `argondb hub run` reading as the cross-tenant reader and writing as the box | built 2026-09-16 | | | the `mfr` schema and the rule that a database cannot be both a distributor and a manufacturer | built 2026-09-19 | | | a token on every caller of the storage service under PostgreSQL | built, after the gap found on 2026-09-23 (§2.6) | | | a storage-service token limited to one company's timeline | built 2026-09-24 — a compute's token names its one timeline and the pageserver refuses it on any other (§2.6) | the cross-company isolation test, W5- and W5+ | | a mount between parties — one tenant reads exactly the tables another tenant granted it, one row per table, and cannot write, read an ungranted table or reach any third party's warehouse | built | the cross-party-mount box test, granted and ungranted, with the wall on and off: `fdw-cross-party-mount: PASS — 11 checks, tenant wall ON (revoke lag 28 s)`, and the same with the wall off (29 s) | | a manufacturer's price list returning into each accepting distributor's `derived.mfr_price_list` | built (`argondb hub return`) — the main picture still draws it dashed, which is out of date; chapter 6 covers it | | | the PostgreSQL role carried into the read plane | **partly built** — the refusal for tables with row security is built; carrying the role into the sharing hub's own reads is not (§2.9) | | | a separate PostgreSQL instance and timeline per tenant — per-tenant point-in-time recovery and per-tenant lake freshness | **built, off by default.** This is the layout ArgonDB is committed to; the operator turns it on, never a default, and the tail follows each tenant's timeline as its own stream | | | one compute per tenant | **built, off by default.** Turned on by the operator, never by default; once a tenant has its compute, its writes go there; chapter 2 covers it | | | an estate control plane — the provisioning API, the identity list and the lease on the shared tail, on their own instances with a leader | **designed, not built** for more than one instance. On one box it is a single process and the `argon_estate` schema | | | one control plane provisioning many cloud instances | **designed, not built** | | | short-lived storage keys limited to a path prefix | **not available with the bundled storage service**; an option only with a hosted or your own S3 | | | `argondb estate migrate` — bringing every tenant's shared schema up to one version in one pass, without one refused tenant stopping the rest | **built** — but see the limit at the end of §4.0 | the box test for migrating every tenant's schema | | trying a schema migration on a copy of a tenant's timeline before doing it for real | **designed, not built** | | | `argondb system create` publishing a new system's Identities to every other system in the same command | **built** — one direction only, and reaching a completely separate cloud account is **not built** | the box test for creating a system | | a company's own MCP token, reading only that company's database | **built** — PostgreSQL only; the lake side (§2.9) is **not built** | the box test for cross-company isolation | | each manufacturer's own copy of a shared total, counted only from what it was granted | **built** — the single-granter disclosure question is **open**, not yet ruled on | the box test for the manufacturer hub warehouse | | a view defined over a table a database holds only through a mount grant | **refused by name**, until it is ruled whether a grantee may build one at all | the box test for a view over a granted table | --- ### 6. Surprises you will meet These are the behaviors most likely to catch a first-time user. Each is deliberate unless it says otherwise, and each says what to do. **A tenant table published before its wall exists is held back, and the log line repeats.** This is not a stall. The alternative would be publishing the tenant's data under the box's own key, which is the wall turned inside out. Wait for the next pass (at most 30 seconds) or run `argondb lake tenant provision`. **A database whose name is not a legal bucket name is refused by name, and the pass carries on with the others.** Nothing rewrites the name to fit, because a rewrite would put `tenant_a` and `tenant-a` in the same bucket — exactly what the wall exists to prevent. Give such a database its own warehouse and bucket in `argon.toml` with `[lake.tenant_warehouse]. = { warehouse, bucket }`; that is the only way in for it. The tenant-wall box test, phase 9, proves both outcomes. **`warehouse_per_database = false` removes the wall and the cross-tenant reader together.** Every database then uses the box's own warehouse, so there is nothing for the reader to cover, and the pass removes a reader that an earlier configuration left behind. There is no cross-tenant read without a wall. Setting it to `true` against your own external storage service is also refused: the box can only create a bucket-limited key in a storage service it runs itself. **With `catalog_authz = "off"`, table names are public even though the data is not.** Anything that can reach the catalog's port can list every warehouse and every table name without any credential. ArgonDB's own output says so in those words. Bind the catalog to the local machine and treat its port as privileged. **OpenFGA ships in both platforms' release builds.** You do not install it yourself — the Linux and macOS release artifacts each stage the binary, so a stock multi-database box never meets a boot refusal for a missing authorization engine. **Revoking a grant leaves an issued token valid until it expires.** Nothing keeps a list of revoked tokens. Issue short-lived tokens with `--ttl`. **`argondb tenant create`'s two name flags are optional now, not required.** An earlier build refused a create with no name; this one writes the tenant with empty names instead and says which flags you left out. What the command still refuses is an `--identity` key it does not recognize — an unknown key has nowhere to be stored, so it is refused rather than silently dropped. The command's own `--help` text has not caught up to this and still calls the name flags required. **A new system only publishes its Identities forward, not backward.** `argondb system create` grants every already-existing system a read of the NEW system's Identities. It does nothing for the other direction — the new system does not automatically get to read anyone else's. Grant that yourself with `argondb system grant-identities` (§4.10). **Hand a company's agent its own MCP token, never the box's token.** The box's own token still reads every company; a company's own token reads only that company (§4.11). A session under a company's token also has no box-wide search and cannot ask for publish status — both are refused by name rather than answered with a company-scoped guess. **A continuous view over a table you only hold through a mount grant is refused when you try to create it.** A lake-landed view can only be read from the database that owns its inputs, so a view built over a table you were only granted would be created and then be unreadable from where you made it. Until that is ruled one way or the other, creating one is refused by name, with a message naming the grantor's database as the one place the view can be defined instead. **The operator is inside every wall.** The root storage key and the box's catalog identity reach everything, including the box's own warehouse, and they can write. The main picture draws this and lists it as one of its two remaining gaps. The wall separates tenants from each other; it is not meant to keep out whoever runs the box. **Row-level security does not reach the lake copy, and the read plane's refusal does not cover the catalog door.** A reader holding the catalog address and the storage credentials reads every row of every published table, whatever the row policies say — those credentials are the only thing in the way. Two related surprises: a table that already has row security the first time the lake mounts it gets no mount at all, because the read that learns its column types meets the same wall (run `SELECT argondb_lake_sync()` as a role that row security does not apply to); and turning row security on or off reaches the wall on the next schema pass, not instantly. **The read plane trusts the role it is told, and cannot check it.** The compute tells the read plane "this read is for role R, and row security does not apply to R here", and the read plane has no way to verify that, because the only credential it receives is one token shared by the whole process. Anyone holding `ARGONDB_AUTH_TOKEN` can claim any role, so never give it to an analyst. **`SHOW ALL TABLES` in DuckDB does not list a table nested under a database and schema.** DuckDB lists only the top level and does not look further, so a table you have not named is missing from the listing. Run `USE lake."br_main.."` first and the table appears. **Some tables cannot be combined across tenants.** If a tenant's copy already has a column named `tenant`, or a column whose types across tenants cannot be reconciled (`Int32` in one, `Boolean` in another), the view is refused by name and left out — never quietly narrowed. **A read session with no roster has no `all_tenants` views at all.** The views are built from what the session registered when it pinned, so a session without a roster is not a session with an empty set of views. **Do not size a deployment from §3.7's numbers without measuring your own.** Earlier figures for the same measurement did not reproduce. **Two connection-limit surprises.** Superusers are exempt, and the limit counts connections to the database, not per role. A limit an operator set by hand is never changed by an ordinary pass — neither lowered nor raised — so a tenant set to 5 by hand stays at 5 until an explicit override says otherwise. **A read position across tenants is not atomic across warehouses.** A cut spanning two tenants is one transaction per warehouse. A pin across tenants lands at the newest position that every limited table in every tenant warehouse can serve, and reports per warehouse which position it resolved. **Lake freshness is shared by the whole box by default.** With per-tenant compute off, every tenant writes to one log, so one busy tenant holds back how fresh every other tenant's lake copy is. Per-tenant freshness needs a PostgreSQL instance and timeline per tenant. That is the committed layout, it is built and off by default, and turning it on is the operator's choice. **The catalog's REST `{prefix}` is the warehouse's ID, not its name.** `/catalog/v1/argon/namespaces` answers `400 {"error":{"type":"WarehouseIdIsNotUUID"}}`. Get the right value from `defaults.prefix` in the response to `GET /catalog/v1/config?warehouse=`. **Not measured: Trino.** No Trino was set up for this chapter. Its Iceberg connector is documented to map a namespace to a schema under the same three-part naming, so the same rules are expected to apply — but that is an inference, not a measurement. --- ### 7. The detailed pictures Three pictures go deeper than the main diagram. Each appears in the section it explains: - **One provisioning pass, and the table that arrives before its wall** (§3.1): the two ways a pass is started and the lock between them; the four things a pass creates, in order, and why that order; the two box-wide jobs (the connection limit and the reader); and a table being held back and named, from start to finish, with the reasoning of the refusal beside it. - **The five identities, and what each one can reach** (§3.4): the storage service's rule case by case with its refusal wording, the catalog's four identities and how a grant is made, and a table of every identity against every bucket and two warehouses. - **One table, its spellings, and which door answers each** (§3.8): where a table's identity starts (a PostgreSQL table plus the marker that publishes it), its four spellings, which field of `argondb lake address` gives each, the two ways of getting it wrong by assembling it by hand, and what changed when the primary database started to be named the same way as the others. --- ### 8. Where to read further - **`docs/lake-foreign-read.md`** — the customer-facing recipe for reading the lake from outside: per-tenant warehouses, the cross-tenant reader, the `all_tenants` views, `argondb lake address`, and row security against the lake copy. - **`docs/upc-hub-recipe.md`** — the wall as the sharing hub sees it, and the steps an operator takes to set up a manufacturer. - **`docs/reserved-identifiers.md`** — every name ArgonDB reserves for this chapter's machinery: the wall's identities, the reader's, `all_tenants` and its `tenant` column, the `mfr` schema, and the provisioner's lock. - **The source code**, if you want to read how it is done: the provisioner is `crates/argondb-ctl/src/lake_tenant.rs`, grants and tokens are `crates/argondb-ctl/src/lake_grant.rs`, table addresses are `crates/argondb-ctl/src/lake_address.rs`, the storage service's rule is `crates/argondb-lake-store/src/wall.rs`, and the `all_tenants` views are `crates/argondb-query/src/all_tenants.rs`. `argondb tenant create` itself — its eight steps, its identity fields and its MCP token minting — is `crates/argondb-ctl/src/tenant_create.rs`; the multi-tenant schema migrator is `crates/argondb-ctl/src/estate_migrate.rs`; creating a system and its Identities grants are `crates/argondb-ctl/src/system_create.rs` and `crates/argondb-ctl/src/system_identities_grant.rs`; and the per-company MCP token is `crates/argondb-ctl/src/mcp_company_token.rs` and `crates/argondb-mcp/src/company_token.rs`. - **The box tests** that prove this chapter on a fresh machine: `testing/box/tenant-wall-warehouse.sh`, `testing/box/all-tenants-views.sh`, `testing/box/cross-tenant-reader.sh`, `testing/box/fdw-cross-party-mount.sh`, `testing/box/uniform-schema-migration-across-tenants.sh`, `testing/box/system-create-sequence.sh`, `testing/box/cross-company-isolation-eight-surfaces.sh` and `testing/box/upc-hub-warehouse.sh`. --- ## Guide 2 — the per-tenant unit: the gate, wake and stand-down ArgonDB lets a tenant's database go to sleep when nobody is using it and wake up again the moment someone connects. A sleeping database gives its memory back to the machine; the data itself stays safe, because it lives in a shared storage layer and not inside the database process. This chapter explains how that works: the small front door that notices a new connection (**the gate**), the rule that decides a database is idle, the order in which it is shut down and started again, and what your client sees while that happens. It is written for a developer who has to use this: turn the setting on, predict what a client will see, size a machine, run an application beside a tenant's database, and know which of these pieces exist today and which are still designs. **Two names used throughout.** ArgonDB's own process that copies database changes into the lake (the Iceberg tables described in the next chapter) is called **the tail** (`argondb-tail`). The word **sidecar** means only one thing in this guide: the tenant's own application running beside its database. It never names an ArgonDB process. **What you will type, what you stop doing, what you will be refused.** You type one setting — `[compute] idle_stop = "300"` in `argon.toml`, or `ARGONDB_PRIMARY_IDLE_SECS=300` in the environment. You stop writing your own "shut the database down at night" scripts and your own pre-warm pings: the gate wakes the database on the first real bytes of a connection and holds that connection until the database answers. You will be refused at startup for a malformed or zero window, for a window shorter than 30 seconds on a machine that publishes to the lake, and — at connect time — with the standard Postgres "try again later" error (`57P03`) if a wake fails. --- ### 1. The diagram ![The per-tenant unit: the gate, wake and stand-down](diagrams/argondb-and-the-erp/02-the-per-tenant-unit-gate-wake-and-stand-down.svg) The picture shows one machine running ArgonDB, and which parts of it can be switched off to save memory. Read it left to right along the top and you have the control path. A visitor — a browser, an application's API client, an SQL client, an AI agent — reaches **the gate**, a single process that fronts the tenant's SQL port and the tenant's application port (the second listener is built; see the HTTP `503` paragraph at the end of section 3.6). The gate is the only component that sees every byte headed for either, and that is why it is the one that decides whether the database needs to be awake. The gate wakes the database on the **first byte** a client sends, not on the bare TCP connection. A connection that arrives and sends nothing is dropped after thirty seconds, and nothing is started. Below the gate is the supervisor's process table — the list of every process the ArgonDB supervisor (`argondb serve`) starts and watches. It has two halves: - **The resident plant**, on the left: the broker, the pageserver and the safekeeper (the shared storage layer), the lake's object store and catalog, the policy engine, the tail, the maintenance worker and the agent endpoint. These never stop, and a machine pays for them once, however many tenants it holds. Measured, storage plus the supervisor came to 231 116 kB; the sizing arithmetic in section 3.10 uses 265 MB for the whole shared plant. - **One tenant's unit**, in the middle: that tenant's database process (`compute:`) and at most one application beside it (`sidecar:`), and nothing else. That unit is the thing that stops. The outlines carry meaning. **A solid outline means built and shipping; a dashed outline means designed and measured, but not built.** Section 5 lists every piece with its state. In short: stopping and waking a database is built and is the default shape, one database per machine; a compute per tenant, each stopping and waking on its own, is built and switched off by default (section 2.8); and the tenant's own application runs as its own process in a first version, while its full sleep-and-wake lifecycle is designed and not built. (The picture itself was drawn before the per-tenant compute was built, so it still shows that box dashed.) The three boxes on the right of the picture are drawn as open questions; all three are now settled, and the picture's styling for them is out of date: - **Isolation per deployment target.** An on-premises machine isolates a tenant's application with a unix user and a cgroup. The one-click cloud VM goes further and requires a container per tenant (section 2.6). - **Where a tenant's application runs.** Always as a separate operating-system process in the supervisor's table, never inside the database process (section 2.2). - **What counts as activity.** Only bytes crossing the gate. A silent WebSocket does not keep a tenant awake (section 3.7). --- ### 2. What it is This section takes the boxes in the picture one at a time and adds what a diagram cannot show: ports, defaults, limits and what goes wrong. #### 2.1 The gate The gate is a small proxy, about ninety lines of code, that sits on the database's public port so it can notice a new connection and wake the database before handing the connection over. When `[compute] idle_stop` is on, the supervisor takes the public Postgres port and moves Postgres itself to the public port plus 100. On a default machine that is 5432 for clients and 5532 for Postgres. If you change the public port with `ARGONDB_PG_PORT`, both move together, so two machines with different public ports also get different internal ones. The gate binds exactly the address Postgres would have bound with the setting off, so your connection string does not change. For each connection the gate does four things: it reads the first bytes, asks the supervisor to wake the database if it is down, connects to the internal port, and then copies bytes in both directions. It connects to the internal port on the same address your client used to reach the gate, never on a fixed `127.0.0.1`. That matters because the machine trusts its own roles when they connect over loopback: a gate that always dialed loopback would make a client on another machine look like a trusted local one, and it did exactly that until 2026-09-24. Now a client from another machine is asked for its password, as it should be. It does not parse the Postgres protocol. An SSL request, a GSS encryption request, the startup packet and a cancel request all reach the real database unchanged, because on the normal path the gate never speaks Postgres to your client. Two things can go wrong, and both are reported plainly: - A client that sends nothing within **30 seconds** is closed silently and wakes nothing. - A wake that fails is answered with a real Postgres error message carrying SQLSTATE **`57P03 cannot_connect_now`** (section 3.6). An error accepting one connection (for example the process running out of file descriptors, or a client aborting mid-handshake) is logged and the gate keeps listening; giving up the port would leave the machine with no way in while the supervisor was still running. Only failing to bind the port in the first place stops the supervisor. #### 2.2 The supervisor's process table The supervisor is the one process that starts, watches and restarts every other process on the machine; its process table is the fixed list of those processes. The list is built once at startup and never reshaped. Each row carries its own restart policy, its own circuit breaker (which stops restarts after repeated failure) and a separate "stood down" flag, which records that a process was stopped on purpose because it was idle. The rows are already of many kinds — four Rust storage binaries, two PostgreSQL clusters, a Go policy engine, a Rust catalog server and three of ArgonDB's own workers — and every row shares four things: - **a command** that is rebuilt identically every time the process is restarted; - **a port**, worked out by one shared rule; - **a log file** that receives both stdout and stderr in append mode. It is started fresh on first launch and appended to on every restart, so the output of a crash is still there after the process has recovered; - **an ownership stamp**: the `ARGONDB_BOX_INSTANCE` variable in the process's environment, plus a record of its name, process id, start time and port. This is how the supervisor proves which leftover processes belong to this machine when it cleans up, without guessing from process names. Every process leads its own process group, so a stop signal sent to the group reaches everything under it — for a database, both its controller (`compute_ctl`) and the Postgres server beneath it. **A tenant's application is not a new kind of row.** It needs the same four things plus two more: a health check and an isolation setting. It always runs as a **separate operating-system process** in this same table, under its own unix user and cgroup, never inside the database's process. Embedding it in the supervisor's own process would only be reconsidered if a measurement showed that the local network round trip slows a real workload, and no such measurement is planned. ![The supervisor's process table, row by row](diagrams/argondb-and-the-erp/02b-supervisor-process-table.svg) #### 2.3 The resident plant The resident plant is everything on the machine that is shared by all tenants and never stops: the broker, pageserver, safekeeper, `lake-store`, `catalog-pg`, `openfga`, `lakekeeper`, `argondb-tail`, `argondb-maintenance` and `argondb-mcp`. A second tenant does not pay for them again. Measured on a machine with one database, the lake switched off and the analyst pool switched off, the whole machine used **368 205 kB** of memory (proportional set size, PSS — see section 3.10 for why this measure), of which **231 116 kB** was storage plus the supervisor. Two rows need their own note: - **`catalog-pg`** is a private Postgres for the lake's catalog. It exists because until 2026-09-08 the bundled catalog stored its Iceberg metadata inside the very database that idle-stop switches off, so ArgonDB used to refuse to turn idle-stop on at all when the bundled catalog was in use. Giving the catalog its own always-on Postgres removed that refusal. - **`argondb-tail`** is always on by design. When a tenant's database is asleep there are no new changes to copy, so the tail simply waits. Its only use of the database is reading the schema, over the internal port and under an internal application name — so the tail can never wake a sleeping tenant and never resets its idle clock. The rule is: one tail per machine, never stopped for idleness, not merged into the supervisor's own process, and its cache not moved into the pageserver. Its memory is bounded: a stated limit of 6 144 MiB with an alarm at 5 632 MiB, enforced as a 6 GiB cgroup `memory.max` alarmed at 5.5 GiB, with each buffer given its own budget. If a small memory share would push the tail below a 512 MiB floor, the machine refuses to start rather than run it starved. Whether there should be one tail per tenant database, rather than one per machine, is not yet decided. #### 2.4 `compute:` — the tenant's database A compute is one running Postgres: the `compute_ctl` controller plus its Postgres server, with its own data directory, its own configuration and its own port. Stopping it is a **memory** decision, not a **durability** decision, because a compute holds no data that is not already in the shared storage layer. (ArgonDB's analyst pool deletes a compute's data directory before every start precisely to guarantee that.) Measured: one idle compute gives **133 906 kB** back to the machine when it stands down (machine memory 361 561 kB with it up, 227 655 kB with it down), and starting it again takes **619 ms** at the median (fastest 613 ms, slowest 636 ms, over ten cycles) when someone comes back. If a compute never starts accepting connections, the wake fails loudly with the end of the compute's own log, and counts as a failure against the wake's circuit breaker. After **six failures within 120 seconds** the breaker opens, and from then on every connection is refused with a message saying so, until the machine is restarted. **A company's compute starts the moment the company is created, and this is in v1.5.5.** Step 2 of `tenant create` gives the company the lowest free slot in the tenant port band and records that slot in the company's own directory, then asks the running supervisor to start the compute. The supervisor adds the row, its per-child state, its idle clock and its wake gate to the live table, publishes the port to the estate directory, and only then answers. The create command returns once that is true, so the company's first write already goes to its own compute and the primary refuses it. If the compute cannot start, the create fails at step 2 with the cause named and nothing left behind: no database, no roles, no wall, no published port. Running the command again resumes onto the same slot. The slot, not a position in a sorted list, is the port, so creating a company no longer moves any other company's port; companies that predate the record keep the port they had. An earlier edition of this guide said a company created while the server is running would only get its compute at the next restart; that was wrong about the tagged v1.5.5 release, which does carry the start-at-create behavior. What is still true for a company created under the older v1.5.4 release: its compute, and its pool entry, only arrive once the upgraded machine has run its first pass over the company list. **Rows written before the company had its own compute are copied over once, and only then does the primary stop accepting the company's writes.** The moment a company gets its own compute, the pass installs a refusal on the primary for that one company's database: any insert, update, delete or schema change to that database on the primary is refused by name, quoting the company, the compute's port and the ruling behind it. Reads on the primary still work. Only after that refusal is in place does the machine copy the rows that had already landed on the primary onto the company's own compute — once, and rolled back as a whole if anything goes wrong partway — and replay the company's database roles onto its own compute from the primary's own record, so a connection routed there logs in exactly as it would have on the primary. The one defect known in this copy is candid in section 2.7. #### 2.5 `sidecar:` — the tenant's application A tenant's own application — a web backend, say — can run beside its database on the same machine, as its own process. **A first version exists:** creating a tenant with a configured application starts that application as one process per tenant and records its process id, unix user, cgroup and command in `/tenants//sidecar.json`. The record is checked against the running system on every pass, including the process's start time, so a reused process id is treated as dead rather than mistaken for the tenant's application. What is **not built** is the per-tenant user separation: on one machine, two tenants' applications cannot yet run as two different unix users (that needs root, section 2.6), and ArgonDB refuses to start a tenant's application whose own user is missing rather than quietly running it under the supervisor's user. Since v1.5.7.2, ArgonDB creates that user itself, through a small helper that root installs once per machine with `sudo /bin/argondb box install-useradd-helper --user `. The helper lets the supervisor's user create `argon-t-` users and do nothing else. Without the helper, the refusal names that install command. This first version is not yet a row the supervisor's own process table tracks. With `[compute] per_tenant` on, the tenant's gate does sit in front of the application's recorded port (section 3.6): a request that reaches a stood-down tenant there wakes it and is told to retry. It does not yet hold the tenant awake — bytes on the application port do not reset the idle clock (section 3.7). Only the application's own connection to its database does that, because those connections pass through the compute's gate like any other client. The rest of this section, and sections 3.7, 3.8 and 4.5, describe the full design the application row is being built toward: one more row per tenant in the supervisor's table, sleeping and waking with its database, whose command is whatever the file `/tenants//run` says. The supervisor runs what it finds there. It does not build, deploy, update, version, roll back or download your application. If the command does not exist, the supervisor refuses to start the row and names the path — the same way it treats a missing `compute_ctl` today. A row is never silently missing. The application's environment starts **empty** and is filled with an explicit list. This matters more than anything else in the design, because by default a child process inherits its parent's environment, and the supervisor's holds `ARGONDB_AUTH_TOKEN`, the lake's credentials and `ARGONDB_BOX_INSTANCE`. `ARGONDB_BOX_INSTANCE` is the one variable added back on purpose: an application without it could not be identified and cleaned up by the supervisor, which would be worse than the leak the empty environment prevents. The application gets exactly one secret — its own tenant's SQL credential — and a health check that is either an HTTP GET or a plain TCP connect. It is restarted under the ordinary restart policy (section 4.7). If it keeps crashing and its breaker trips, **the tenant's database is not affected**: the supervisor stops restarting that one row, marks the machine DEGRADED, raises an alarm and keeps supervising everything else. #### 2.6 The isolation box Isolation is what stops one tenant's application from reading another tenant's files or starving the machine. ArgonDB describes it as a ladder of four rungs, each stronger and more expensive than the one below. Rung 1 is one unix user per tenant plus one cgroup with limits on memory (`MemoryMax`), CPU (`CPUQuota`) and number of processes (`TasksMax`). - The limits half needs **no privilege at all**, and has been measured working. - The user half needs **one root command per tenant when the tenant is created**, and none after that. What rung 1 does not cover, stated plainly: - the tenant's own data — the application reads it, that is its job; - an escape through the kernel — a unix user is not a virtual machine; - what the application does with data it is allowed to see. **Which rung each deployment target gets:** - An **on-premises machine** uses rung 1 as above (or rung 0, plain process separation, when no tenant application is configured). - The **one-click cloud VM** requires **rung 2, a container per tenant**, on top of rung 1. Rung 2 is designed and not built. - **Hosted multi-tenant** — our hardware running code we did not write — is not covered. A unix user is not an adequate wall between tenants who may be hostile to each other, and the right rung for that product will be chosen when that product is designed. The kernel-escape gap applies on every rung listed here. Section 4.6 has the recipe and section 6 the traps. #### 2.7 The pooler A connection pooler lets many client connections share a small number of real database connections, which is what keeps a busy web application from exhausting Postgres. **This is built and has shipped since 2026-09-20.** ArgonDB runs one PgBouncer process per machine when a working PgBouncer is present. **The pooler is optional**: if no supported binary is found, the machine boots on the direct port only and says so, instead of refusing to start. When it does run, it is in transaction mode, offering one pooled port for the whole machine; it picks the tenant from the `dbname` in each connection's startup packet. The direct port is still there beside it. Defaults: | setting | value | |---|---| | pool size | one quarter of the tenant's own connection limit | | `reserve_pool_size` | 0 | | `max_db_client_connections` | 200 | | `max_client_conn` (whole machine) | 10 000 | | `server_idle_timeout` | the tenant's idle window | | `min_pool_size` | 0 | | `max_prepared_statements` | 200 | **The pooled port is the connection string ArgonDB gives you by default.** The provisioner prints the direct port beside it, labeled for the things a transaction-mode pooler cannot carry: migrations, `LISTEN/NOTIFY`, session-level `SET`, session advisory locks, and desktop database tools. ArgonDB's own tools (`argondb`, the tail, the `--pg-url` defaults) always use the direct port. The pooled port is always the Postgres port plus 200, so it moves with the database's port. The pooler is **PgBouncer specifically**, version **1.25.2 or newer**. Pooling of prepared statements needs 1.21.0, and 1.25.2 adds the SCRAM security fixes; Ubuntu 26.04's own package is 1.25.1, which is too old, so install it from the PostgreSQL project's apt repository (PGDG) or build it from source. You choose the pooler's behavior with `[pooler] enabled`: - `on` or `auto` — use the pooler if a working PgBouncer is found, otherwise decline loudly and serve the direct port only; the two now behave the same way, and `on` is written as intent, not as a refusal switch; - `off` — no pooler. A PgBouncer that is missing, older than **1.25.2**, or unable to report its own version is treated as absent: the machine starts on the direct port, with a message naming the problem (and how to install one), rather than refusing to boot. An earlier edition of this guide said `[pooler] enabled = "on"` refused to start without a working PgBouncer; that boot-time refusal was removed. **PgBouncer comes from your platform, not from ArgonDB.** The release artifact does not include a PgBouncer binary, and never will — that split is settled. `argondb serve` looks for `pgbouncer` on the `PATH` (or at `ARGONDB_PGBOUNCER_BIN`) and checks its version against the minimum. Install it before you turn the pooler on if you want pooling; the machine runs without it either way. **With a compute per tenant turned on, the pooler sends each tenant's pool to that tenant's own compute, not to the primary.** A database with a recorded compute port gets a pool entry that dials that port; a database with no recorded port yet — one whose compute has not started — gets **no pool entry at all**, and the machine's log names it and says plainly that it is not being routed to the primary instead. Because a company's own compute now starts the moment the company is created (section 2.4), this gap is at most as long as one pass of the machine's own bookkeeping. **What this proves, and what it does not, stated plainly.** The routing itself is proven both ways: a pooled connection lands on the right tenant's own compute with the setting on, and on the primary with it off. What has **not** passed, as of this writing, on the branch everyone builds from, is the test that two tenants' data stay apart under a pooled connection with the per-tenant setting on. The attempt that came closest found a company's data missing from its own compute at the moment of the test, because the one-time copy described in section 2.4 can be skipped: if a table gets created on a company's new compute before that copy runs, the copy fails and is not retried, and that company's older rows are left behind on the primary, readable there but never moved. A machine that hits this is not repaired by a later pass. With the per-tenant setting off — the default — none of this applies, and the same test passes cleanly. #### 2.8 One database per machine today, and what many tenants per machine needs ArgonDB's tenancy model is that each tenant is its own database, running as its own Postgres instance with its own history of changes (its own **timeline** and write-ahead log), served by its own compute that stops when idle and wakes on connect. All tenants on a machine share one storage layer: one pageserver, one safekeeper and one lake. Because each tenant has its own timeline, a branch — for an upgrade rehearsal, a snapshot or a restore — branches one tenant, not the whole machine. What exists today: - **The default is one primary database per machine**, with the idle-stop and wake described in this chapter. That remains the default until the per-tenant shape is finished. - **A compute per tenant exists and is off by default.** Setting `[compute] per_tenant = true` (or `ARGONDB_COMPUTE_PER_TENANT`, which wins over the file) gives each tenant database its own `compute:` row with its own port pair, configuration, data directory, idle clock, wake gate and wake breaker, all using the same stop and wake steps as the primary. It is refused unless idle-stop is also on. On Linux and Mac there is no count cap: each new tenant compute takes the next free pair of ports at or above `[compute] tenant_port_base` (`ARGONDB_TENANT_PG_PORT`, default 25450), skipping every port the machine reserves for itself and every port some other listener already holds. The first port of the pair is the client's, owned by that tenant's own gate; the second is where its Postgres really listens. The pair is recorded in `tenants//compute-slot.json` and reused on every restart. If another listener holds a recorded port at boot, that one tenant is refused by name — naming the port and who holds it — and the machine keeps serving every other tenant. A compute that cannot start for lack of memory is refused the same way. How this works on AWS and Azure is a separate question, not yet built. - **Each tenant compute runs on its own timeline**: a branch of the machine's primary timeline, cut when the tenant is created. The pageserver will only serve a tenant's compute its own timeline. With the setting on, the pooler sends a tenant's pooled connections to that tenant's own compute, the compute carries the tenant's own roles, and once a tenant has its compute its writes go there and the primary refuses them; rows written to the primary before that are copied onto the tenant's timeline once, with every table's row count compared. - **The tail follows every tenant's timeline**, as a separate stream inside one tail process. - **Limits.** A company gets its own compute the moment it is created (section 2.4), so this is not a restart-only affair for a new company. The number of tenant computes on one machine is bounded by free ports and by memory, never by a count of its own, and a compute refused for either is refused for its tenant alone. What is still open: the cross-tenant isolation test with this setting on has not yet passed on the branch everyone builds from (section 2.7). **Per-tenant compute is always the operator's choice, never the system's default.** Whatever the per-tenant setting resolves to on a given machine, it is something an operator turns on; the machine never turns it on by itself. The full lifecycle of a tenant's application beside its database (section 2.5) builds on this and is designed, not built. The gate's second listener for the application port, with its own failure message, is built (section 3.6); the routing by port is what the design chose (section 3.9). --- ### 3. How it works This section walks through what actually happens, step by step: how the setting is read, how the machine decides a database is idle, how it stops and starts it, and what is refused along the way. ![Wake and stand-down, with every timeout](diagrams/argondb-and-the-erp/02a-wake-and-stand-down-sequence.svg) #### 3.1 Reading the setting at startup Idle-stop is controlled by one setting, and ArgonDB reads it once at startup because it decides which ports the machine binds. `[compute] idle_stop` in `argon.toml` is a **string**: `"off"` (the default) or a whole number of seconds such as `"300"`. It is a string on purpose, so that one parser handles both the file and the environment variable and the two cannot disagree. Writing `idle_stop = 300` without quotes is refused, with the expected type in the message; the fix is to add quotes. Precedence is **environment, then file, then default**: `ARGONDB_PRIMARY_IDLE_SECS` beats `argon.toml`, which beats "off". This is the same order as the machine's `port` and `memory_budget_mb` settings. The environment wins **even when it says `off`**: an operator who exports `off` has turned idle-stop off, whatever the file says. Two values are refused rather than quietly replaced with the default, because this setting decides whether your database stops: - anything that is neither `off` nor a whole number — the message names where the value came from; - `0`, which "would stand the primary down the instant it goes idle, which is a restart loop rather than a policy". Every section of `argon.toml` rejects unknown keys: a misspelled key is refused and the message lists the keys that are accepted. A setting you believe is active but that does nothing is the worst thing a configuration file can give you, so ArgonDB never ignores one. #### 3.2 How ArgonDB decides a database is idle To stop a database safely, the supervisor has to know that nobody is using it. It keeps **one** long-lived connection to the database's internal port and runs **one** query every **2 seconds**. It reuses the same connection because opening a fresh one every two seconds would itself show up as activity in `pg_stat_activity`, the very table it is reading. The query returns five numbers: - client connections currently running a query; - how long ago the most recent idle client connection last did anything; - replication senders (not counting the storage layer's own); - logical replication subscriptions with a live worker; - autovacuum workers. These are the same signals the underlying Neon compute monitor uses. Two kinds of connection are excluded in the query itself: anything whose `application_name` starts with `argondb_internal` — every connection ArgonDB makes to itself, including this monitor, which calls itself `argondb_internal_idle_probe` — and anything starting with `compute_ctl`. Without the first exclusion, the monitor would count itself as activity and keep the database awake forever. If any of four conditions holds, the database is **held awake** and its idle clock resets to now. They are checked in this order: 1. a client query is running; 2. a replication sender is connected; 3. a logical subscription is active; 4. autovacuum is running. A hold is a reason the **database** cannot be stopped. The idle clock only ever moves forward, so a stale reading cannot make the database look idle sooner. If the database reports a negative age (its clock is off), that counts as "now", which keeps the database awake. The window is inclusive: exactly `idle_stop` seconds since the last activity still counts as recent; one second more does not. The analyst pool uses the same rule. **A database the monitor cannot measure is never stopped.** If the monitor cannot connect, it logs the failure and leaves the clock alone; if a read fails, it drops the connection, reconnects on the next tick, and leaves the database running. When the monitor cannot tell "nothing is happening" from "I failed to look", it assumes the database is busy. #### 3.3 Stopping an idle database, in five steps When the idle window has passed, the supervisor stops the database in five steps. It only does this when idle-stop is on and the database is not already stopped. The order of the steps is what makes it safe. 1. **Close the monitor's own connection first.** Otherwise Postgres would wait for it during shutdown. 2. **Mark the database as down before sending any signal.** This is the step whose position matters most. Without it, a connection arriving between the stop signal and the port being released would be handed to a Postgres that is shutting down, and the client would get `FATAL: the database system is shutting down` instead of a wake. Marking it down first turns that moment into a wake request, which the supervisor handles as soon as the stop is done. This race was seen in testing; it is not theoretical. 3. **Send SIGTERM to the whole process group**, not just the controller, because Postgres runs beneath `compute_ctl` and signaling the controller alone can leave Postgres running. 4. **Wait, with two limits**: up to **60 seconds**, or until the process exits, whichever comes first. The wait also collects the exited process so it does not linger as a zombie. If 60 seconds pass, the supervisor sends SIGKILL to the group and prints the end of its log. 5. **Wait for the port to be free**: try connecting to the internal port every 100 ms for up to another **60 seconds**. A simpler implementation would skip this, but `compute_ctl` can exit before Postgres has finished its shutdown checkpoint and released the port, and a wake started in that moment would fail to bind. If the port is still taken after 60 seconds, the group is SIGKILLed with a warning naming the port. The database is then recorded as **stood down**. That is a different state from **tripped** (given up on after repeated failure): a stood-down database can be restarted at any time, has not counted against any breaker, and does not trigger the supervisor's "every process has failed, so the machine is dead" shutdown. **Measured cost:** 34 seconds against a 20-second window. The extra 14 seconds are Postgres's shutdown checkpoint plus step 5. That time is spent on stopping, never on waking. #### 3.4 Waking a database on the first packet When a client connects to a stopped database, the gate holds the connection, asks the supervisor to start the database, waits for it to accept connections, and then passes the client through. Step by step: 1. **Bytes arrive at the gate.** The trigger is bytes, not the TCP connection itself, and that matters twice over. `argondb status` checks whether the port is reachable by connecting and disconnecting; if a bare connection woke the database, the status command would keep it awake forever. And modern browsers open speculative connections that send nothing; a gate in front of an application that woke on those would keep every tenant awake forever. A timeout or an empty read returns immediately and starts nothing. 2. **The gate checks whether the database is up.** If it is not, the gate sends a wake request to the supervisor and **keeps your connection open** while it waits, for up to **180 seconds**. 3. **The supervisor starts the database.** Only the supervisor holds the process handles, so there is exactly one owner of the database process whether it is being started at boot, restarted after a crash, or woken. It uses the same command-building path as a crash restart and appends to the same log, so the previous shutdown and anything that went wrong before it are still in the log afterwards. 4. **Wait for the database to accept TCP connections**: up to 120 tries, 500 ms apart, while also watching the process. If the database process dies, the wait fails **immediately** with the end of its log instead of running out the clock. The deadline is a backstop. 5. **On success**, the database is marked up, its restart history notes the start, its idle clock is reset to now, a line reading `woken by a client connection on the public PG port` is written to its log, and the gate is told to proceed. 6. **The gate passes the connection through.** It turns off Nagle's algorithm on both sides (a 40 ms delay per round trip would otherwise show up in every measurement), forwards the bytes it was holding, and from then on only copies bytes in both directions. **Three behaviors under concurrency, each deliberate:** - A wake request that arrives when the database is already up — because another connection woke it, or it was restarted after a crash — succeeds immediately. - A wake requested during the breaker's back-off period is refused cheaply, without starting anything. A storm of client retries therefore produces one start attempt per back-off step, not one per connection. - A failed wake cleans up before it is counted as a failure: SIGTERM to the group, wait up to 10 seconds, mark the database stood down again, then record the failure. #### 3.5 Wake failures use the same restart policy as everything else Waking a database uses the same restart policy — the same back-off and the same circuit breaker — that the supervisor uses to restart any crashed process, so there is one set of rules, not two. (The analyst pool uses it too.) The numbers: | setting | value | |---|---| | first back-off | 2 s | | longest back-off | 30 s | | running this long clears the failure history | 60 s | | breaker opens after | 6 restarts within 120 s | Individual rows may use their own numbers. The agent endpoint (`argondb-mcp`) trips on its **third** exit within **sixty seconds**, because when it fails it is almost always a configuration refusal: it checks every configured surface before opening any of them, so a bad setting makes it exit within milliseconds every time, and retrying reads the same environment and reaches the same answer. With many tenants on a machine, the consequence that matters is this: **a tenant whose database cannot start uses up its own breaker and nobody else's.** When a breaker trips, the supervisor does exactly four things: it stops restarting **that row**, raises an alarm, records a DEGRADED state that `argondb status` shows, and keeps supervising everything else. An expensive automatic recovery never loops silently: after a bounded number of failed attempts it stops and says so. #### 3.6 What a failed wake says, and why it says it in Postgres terms When a wake fails, your client receives a genuine Postgres error message — not a dropped connection — so every existing Postgres driver can tell you what happened. The gate builds a standard protocol 3.0 `ErrorResponse` with severity `FATAL`, a SQLSTATE and a message. The SQLSTATE is **`57P03 cannot_connect_now`**, the same code Postgres itself sends while it is starting up, which **every driver already treats as "try again later" rather than "your password is wrong"**. There are four messages, and each points to its own fix: | condition | message | |---|---| | the supervisor is shutting down | `the ArgonDB supervisor is shutting down; the primary compute cannot be woken` | | the start failed | `the primary compute could not be started: {why}` | | the 180 s deadline passed | `the primary compute did not start within 180s; see /logs/compute.log` | | the internal port refuses the connection | `the primary compute is not accepting connections on its internal port {n}: {e}` | Two more reach you as the `{why}` of the second row. During back-off: `the primary failed to start and is in a {n}s backoff; retry shortly`. With the breaker open: `the primary's wake breaker is OPEN after repeated start failures ({n} attempts); see logs/compute.log. It will not be retried automatically — restart the box once the cause is fixed`. **The application port has the other vocabulary: HTTP `503` with `Retry-After`.** With `[compute] per_tenant` on, each tenant's gate owns a second listener, the tenant's application port, in front of its application. That port is recorded like the tenant's other ports: `tenants//compute-slot.json` carries `app_public_port` (the gate's) and `app_internal_port` (where the application listens, on loopback), each the next free port found at create — in the same pass, under the same lock, as the tenant's database ports — and reused as recorded on restart. A record written before that pair existed gets one at boot and keeps every other port it had. Which vocabulary the gate speaks is chosen by which listener accepted the connection, never by guessing the protocol from the first bytes. The first-bytes rule is the SQL port's and is unchanged: a browser's speculative pre-connect wakes nothing. What differs is that the application port does **not** hold the client through the wake. First bytes to a stood-down tenant start the wake and are answered at once: ``` HTTP/1.1 503 Service Unavailable Retry-After: 1 Content-Type: text/plain; charset=utf-8 Connection: close tenant acme: the database compute is waking (4 ms in; its last start took 640 ms); retry after 1s ``` `Retry-After` comes from a measurement, never a constant: it is what remains of the tenant's last measured start, rounded up and never below 1. That measurement is seeded from the compute's start at boot or at create (spawn to accepting TCP — the same two steps a wake runs) and overwritten by every successful wake the gate times, on either port. If a wake fails, the next `503` says why. Once the compute is up, the connection is passed through to the application's internal port exactly as the SQL port is passed through to Postgres. An application that is not listening gets `502 Bad Gateway` naming the application, with no `Retry-After`, because no wake can fix that. The SQL port's four messages above are unchanged. **Not built:** bytes on the application port do not yet reset the tenant's idle clock (the `last_sidecar_bytes` term in section 3.7), so a tenant used only through its application stands down after its window, and its next request is a `503` and a wake. The supervisor also does not yet start or stop the application with the compute (section 3.8); the provisioning command (`argondb tenant sidecar`) runs it. The application learns its internal port from its own command line; the machine prints it at boot (`tenant compute application gate listening on …`). #### 3.7 What counts as activity when there is an application The gate's application listener exists (section 3.6), but this part — application bytes keeping a tenant awake — is designed and not built. The rule is settled: **only bytes crossing the gate count as activity.** An open but silent connection does not keep a tenant awake. There is no connection-count hold, no second, longer window for an idle socket, and no flag an application can raise to say "I'm busy". An earlier recommendation to count open-but-silent application connections as activity was not adopted. The principle is that **one component sees all the traffic.** An application that kept its own idle timer would be a second control loop over the same lifecycle, and it would fail in the expensive direction: an application with a 30-second polling loop is never idle by its own reckoning, so its tenant's database would never stop and per-tenant scale-to-zero would save nothing. Because the gate fronts both ports, the rule is: ``` tenant_idle = sql_reading.hold().is_none() && now - last_sql_activity > idle_secs && now - last_sidecar_bytes > idle_secs ``` `last_sidecar_bytes` is a number the gate already has — it counts first bytes anyway — so this needs no probe, no agent inside the application and no cooperation from it. **Activity in someone else's code is measured at our socket, not inside their process.** A busy application is deliberately **not** a fifth "held awake" reason. A hold is a reason the database cannot be stopped; a busy application is a reason the tenant is not idle, which the third line of the rule already covers. What this means for you: a browser holding a WebSocket open, with nothing sent for the whole idle window, does **not** keep the tenant awake. The tenant stops underneath it; the client reconnects when it next sends something, and the gate wakes the tenant. Server push over a silent socket does not survive a stand-down. Every WebSocket client already has to handle reconnecting, and that is the reason for the rule. #### 3.8 Wake and stop order with an application Designed, not built. When a tenant has an application beside its database, the order is fixed so that **the application is never running while its database is down.** - **Waking:** database first, then the application. Once the database accepts connections, the application is started and its health check is waited on, with the same two limits as the database — a deadline, and a check that the process is still alive — failing loudly with the end of its log. - **Stopping:** the mirror image. The application first (SIGTERM to its process group, with its own grace period), then the database's five steps unchanged. **An SQL connection is let through as soon as the database is up**, not when the application is. A `psql` session does not wait for somebody's web server to start. A connection that arrived on the application port waits for the health check, because there is nothing to pass it to until the application answers. **ArgonDB does not bound the application's share of the wake time.** A Go binary answering `/healthz` in 20 ms and a JVM taking 8 seconds are both legitimate. ArgonDB will measure and report the total; it will not limit it. #### 3.9 Routing when there are many tenants The shipped gate maps one port to one database. A gate for many tenants needs one new thing: a way to tell which tenant a connection is for. There are three possible sources — a port per tenant, the `database` field in the Postgres startup packet, or the TLS server name (SNI). The design is **a port per tenant**. It needs no protocol parsing and no new format, it is what the analyst pool already does, and it extends naturally to the application port, which gets its own recorded port the same way. Reading the startup packet would put ArgonDB's code in front of the TLS handshake; SNI would mean ending TLS at the gate. The operating system already solves routing by port. #### 3.10 What has been measured, and under what conditions These are the measured numbers behind this chapter, each with its conditions, because a number without its conditions is not a measurement. | quantity | figure | conditions | |---|---|---| | wake, from connect to first row, on the client's clock | fastest 613 / median **619** / slowest 636 ms, 10 cycles | small empty database, fast NVMe machine, pageserver warm, lake off | | spread across those 10 cycles | 23 ms | the path is repeatable, which a latency budget needs more than a low number | | memory returned by one stand-down | **133 906 kB** (361 561 → 227 655 kB PSS) | `idle_stop = 20s`; the "up" reading taken while a client query was running, the "down" reading after the process was confirmed gone | | stand-down wall time | 34 s against a 20 s window | the extra 14 s is the shutdown checkpoint plus the wait for the port | | whole machine, one database | 368 205 kB PSS | lake off, pool off, idle-stop off | | shared plant a second tenant does not pay for again | 231 116 kB | same run | | per always-on compute, three computes | 80 280 kB | two of the three were read-only replicas | | per always-on compute, lone primary | 137 089 kB | being written to, with two idle client connections — **use this number** | | a machine left idle for a full hour, with one continuous view already landed | **zero** view warnings, zero refreshes, zero cuts, zero writes to the lake; the view's own worker used an average of 0.2% of one CPU core | one hour, sampled every 60 s, with no writes at all; a single write right after gave exactly one refresh, so the zeros above are a real result and not a broken measurement | Memory is measured as PSS (proportional set size) rather than RSS, because every Postgres process maps the same shared buffer memory, and adding up RSS would count it once per process. **What an idle machine still does, even doing nothing else.** The hour above is the measurement behind one plain statement: a machine with nothing to do raises no alarms and advances no continuous view. Two things do keep moving in the background, and neither is a problem. The position marking how far the machine has read the database's own change log creeps forward on its own, with no write and no view refresh behind it — watch for a view actually landing new rows, not for this number moving. And a routine background check prints a warning roughly every two minutes about a table with a landed view, which was noticed when it was measured and is a known loose end, not something broken; do not treat that one warning, by itself, as a reason to wake anyone up (section 6 has the rest of the gotchas). The figure of about 630 always-on computes on a machine with 87 GB of free memory is **arithmetic, not a measured limit**. Memory pressure stayed flat at three computes, so the point where the machine starts to struggle was never approached. Memory is also unlikely to be the first limit: 600 computes means 600 listening ports, 600 data directories, 600 timelines on one pageserver and 600 × `max_connections` file descriptors, and none of those has been measured. The shipped per-tenant table does not get close to that figure either: its own default limit is eight tenant computes on one machine, and that number is a starting product default, not something anyone has measured the machine failing past. --- ### 4. How to use it This section is the practical part: what to put in your configuration, what you will see in the logs, how your client should behave, and the refusals you can trigger. The examples are taken from the code or from real runs; the few that describe designed-but-not-built pieces say so. #### 4.1 Turn on idle-stop, and the two ways it refuses You turn idle-stop on with one key in `argon.toml`: ```toml # argon.toml. Every section rejects unknown keys: a misspelled key is # refused with the accepted keys listed, never warned about and ignored. [compute] # "off" (the default: the primary never stands down), or a whole number of # seconds as a STRING. A TOML integer is refused with the type in the # message; the fix is quotes. idle_stop = "300" [resources] memory_budget_mb = 2048 ``` The environment beats the file, which beats the default: ```bash # 1. the file's 300 s applies argondb serve --data-dir /srv/argon # 2. the environment wins: 60 s, whatever argon.toml says ARGONDB_PRIMARY_IDLE_SECS=60 argondb serve --data-dir /srv/argon # 3. the environment wins even when it says `off`. # An operator who exports `off` has turned it off. ARGONDB_PRIMARY_IDLE_SECS=off argondb serve --data-dir /srv/argon ``` The two refusals, exactly as printed. For `idle_stop = "5 minutes"`: ``` [compute] idle_stop="5 minutes" is neither `off` nor a whole number of seconds. Use `off` (the default: the primary never stands down) or a positive idle window in seconds, e.g. 300. ``` For `idle_stop = "0"`: ``` [compute] idle_stop=0 would stand the primary down the instant it goes idle, which is a restart loop rather than a policy. Use `off` to disable idle-stop, or a positive idle window in seconds. ``` Notice that the message starts with where the value came from. Set the same bad value in the environment and the message starts with `ARGONDB_PRIMARY_IDLE_SECS` instead, so you know which one to fix. **Turning on a compute per tenant** needs an idle window set beside it: ```toml [compute] idle_stop = "300" # REQUIRED alongside per_tenant: each tenant's compute # stands down after this many idle seconds per_tenant = true # default false; ARGONDB_COMPUTE_PER_TENANT=on|off wins over it tenant_port_base = 25450 # the default; each tenant takes the next free port pair at or above it, recorded and reused ``` Turning `per_tenant` on without an `idle_stop` window is refused at startup, naming both keys and both remedies. At startup the machine prints one line per tenant company it already knows about, and a summary line stating that per-tenant computes are on, how many tenant databases it found, the port band they occupy, and the idle window each one stands down after. A company created while the machine is already running gets its own slot and its own compute as part of being created (section 2.4); point that company's clients at the pooled port, or at its own public port in the band — never at the primary's port, which still answers reads for that company but refuses every write to it. #### 4.2 Watch a database stop and wake Here is what the supervisor prints, in order. The messages are exactly what the code prints; timestamps are left out because yours will differ. ```bash # Turn it on with a short window and watch the supervisor's output. ARGONDB_PRIMARY_IDLE_SECS=20 argondb serve --data-dir /srv/argon 2>&1 | ts ``` At startup, the gate announces the port split: ``` [argondb-box] primary gate listening on 127.0.0.1:5432 → postgres :5532 (idle_stop ON) ``` A client connects and runs a query. The monitor holds the database awake and says why. The line is printed only when the reason changes, so a long query does not fill the log: ``` [argondb-box] primary held awake: a client query is running (idle_stop 20s) ``` The client disconnects. Twenty seconds later the database is stopped, with one line in the supervisor's output: ``` [argondb-box] primary idle 20s — standing it down (storage stays up; the next client connection on :5432 wakes it) ``` and one in `/logs/compute.log`: ``` standing the primary down after 20s idle ([compute] idle_stop); the next client connection on :5432 wakes it ``` If the database has not stopped 60 seconds after SIGTERM, or its internal port still accepts connections 60 seconds after the stop, the supervisor says which and kills the process group: ``` [argondb-box] the primary did not stop within 60s of SIGTERM — SIGKILLing its process group. Log tail: [argondb-box] WARNING: :5532 still accepts connections 60s after the primary's stand-down — SIGKILLing its process group ``` The next connection's first byte wakes it: ``` [argondb-box] primary gate: 127.0.0.1:41922 asked for a stood-down primary — waking [argondb-box] waking the primary (attempt 1) [argondb-box] primary awake (attempt 1) ``` and the database's own log gets the line `woken by a client connection on the public PG port`. **To see the port split directly**, look at who owns which port. With idle-stop off, a `postgres` process owns the public port and the internal port is not bound at all. With it on, the supervisor owns the public port and `postgres` owns the public port plus 100: ```bash ss -tlnp | grep -E ':(5432|5532)\b' # idle_stop OFF: 127.0.0.1:5432 users:(("postgres",...)) and :5532 absent # idle_stop ON : 127.0.0.1:5432 users:(("argondb",...)) and :5532 owned by postgres ``` The measured wake, ten cycles: ``` cycle 1: 636 ms cycle 6: 616 ms cycle 2: 625 ms cycle 7: 617 ms cycle 3: 634 ms cycle 8: 613 ms cycle 4: 618 ms cycle 9: 620 ms cycle 5: 621 ms cycle 10: 619 ms wake latency over 10 cycles: min 613 ms · p50 619 ms · max 636 ms ``` #### 4.3 What your client sees while the database wakes, and how to retry **Normally your client sees nothing.** The gate holds the connection for about 619 ms and then passes it through; there is no error to handle and **you do not need to send a warm-up query first**. That is because the gate holds the first connection instead of failing it. (Some hosted Postgres platforms, whose control plane is not on the connection path, advise sending a lightweight `SELECT 1` to warm the database first. That advice does not apply here.) **If a wake fails**, your client gets SQLSTATE `57P03`. Treat it as "try again later", not as a login or configuration error. Retry no faster than the supervisor's own back-off: it doubles from 2 s to 30 s, and the breaker opens after 6 failures in 120 s. Retrying faster gains nothing, because requests inside the back-off window are refused without a start attempt. Python, `psycopg` 3 — `sqlstate` is `57P03`: ```python import time, psycopg def connect_with_wake_retry(dsn, attempts=6, base=1.0, cap=30.0): """57P03 cannot_connect_now == 'the tenant is waking, or its wake failed'. Anything else is a real error and is re-raised immediately.""" delay = base for n in range(attempts): try: return psycopg.connect(dsn) except psycopg.OperationalError as e: if getattr(e, "sqlstate", None) != "57P03": raise if n == attempts - 1: raise time.sleep(delay) delay = min(delay * 2, cap) ``` Go, `pgx` v5 — the same check on `*pgconn.PgError.Code`: ```go func connectWithWakeRetry(ctx context.Context, dsn string, attempts int) (*pgx.Conn, error) { delay := time.Second var err error for n := 0; n < attempts; n++ { var c *pgx.Conn c, err = pgx.Connect(ctx, dsn) if err == nil { return c, nil } var pgErr *pgconn.PgError if !errors.As(err, &pgErr) || pgErr.Code != "57P03" { return nil, err // not a wake failure — do not retry } select { case <-ctx.Done(): return nil, ctx.Err() case <-time.After(delay): } if delay < 30*time.Second { delay *= 2 } } return nil, err } ``` JDBC gives you the same string from `SQLException.getSQLState()`. That is the point of reusing `57P03` instead of inventing a code: every driver already knows what it means. **You will not see a bare dropped connection on a failed wake.** The error carries severity, SQLSTATE and message, so `e.diag.message_primary` (psycopg) or `pgErr.Message` (pgx) holds ArgonDB's own sentence — including the path to `logs/compute.log` when the deadline passed. **One trap you are likely to hit: a silent connection is not a wake.** A TCP health check that connects and disconnects, or a browser's speculative connection, is closed after 30 seconds having woken nothing. If you want a monitor to keep a tenant awake, it must **send** something — a real query. #### 4.4 The analyst pool: lazy mode, cap and port band The analyst pool is a set of extra read-only Postgres computes that serve analytical and agent queries. It is the one place ArgonDB already runs several computes on one machine, each with its own idle rule, so it is the working example for per-tenant computes. You configure it entirely through the environment: ```bash # mode: off | lazy | always | auto (the default). # auto — `always` on machines with 16 GiB of RAM or more, `lazy` below # lazy — computes start on the first analyst query and stop after the # analyst idle window (300 s by default) # always — the full cap of computes is started at boot # off — no replica computes, no static computes, no pin-redemption endpoint ARGONDB_ANALYST_POOL=lazy \ ARGONDB_ANALYST_PG_PORT=25440 \ argondb serve --data-dir /srv/argon ``` Ports are worked out from one base. The per-tenant compute band (section 2.8) follows the same pattern: | member | port | |---|---| | replica compute `i` | `port_base + i` | | static compute `j` | `port_base + cap + j` | | pin-redemption endpoint | `port_base + 9` | Defaults: port base **25440**, cap **2** replica computes, **2** static computes, and the pool is checked every **2 seconds**. Three refusals you can trigger, all at startup, so the machine never starts a compute on the wrong port: ``` # ARGONDB_ANALYST_POOL=lazy with a cap of 0: … ARGONDB_ANALYST_POOL=lazy asks for one — set the pool to `off`, or a cap ≥ 1 # ARGONDB_ANALYST_PG_PORT=0 or a non-port: ARGONDB_ANALYST_PG_PORT=0 is not a valid TCP port (1-65535) # a band that would overflow, or whose members would collide: ARGONDB_ANALYST_PG_PORT= + --port-offset overflows … ``` The whole band is checked before anything binds, and the rule that static computes never collide with the pin-redemption port is enforced by a test. **Two things about `lazy` that surprise people.** A lazy pool keeps its full cap of computes running while there has been recent activity, and none otherwise — that is per-compute scale-to-zero. And `argondb status` shows a lazy pool's computes as **down**, which is the normal resting state, not a fault. The status output says so: ``` the pool is `lazy` — computes start on the first analyst query and idle-stop after the analyst idle window ``` **The pool's computes are replicas of one database, not databases of their own.** That one sentence was the whole difference between the analyst pool and a per-tenant compute table; the table is now built separately, each tenant compute on its own timeline (section 2.8). **One coupling to know about:** the pool's only user is the agent endpoint, so `ARGONDB_MCP_PORT=0` turns off the pool whatever `ARGONDB_ANALYST_POOL` says, and `argondb status` tells you which setting did it. #### 4.5 A tenant application beside its database (design; a first version is built) **A first version of this runs today** (section 2.5): one process per company, recorded in `/tenants//sidecar.json`, with its command read from the estate directory rather than the `run` path below, and without the per-company unix user. The rest of this example is the contract a tenant's application is held to. The supervisor reads exactly one path per tenant: ``` /tenants//run # a command line. The supervisor runs what is there. ``` The whole contract for your application is: listen on the port you are given, answer the health path, read your database connection string from the environment, and exit on SIGTERM. A small HTTP application as a tenant's sidecar: ```bash # /srv/argon/tenants/acme/run (mode 0755, owned by argon-t-acme) #!/bin/sh exec /srv/argon/tenants/acme/bin/acme-app \ --listen "127.0.0.1:${ARGONDB_SIDECAR_PORT}" \ --health-path /healthz ``` ```python # acme-app, in ~20 lines. Nothing here is ArgonDB-specific except the two # environment variables it reads. import os, http.server, psycopg DSN = os.environ["ARGONDB_SIDECAR_DSN"] # the tenant's OWN credential, and the only secret PORT = int(os.environ["ARGONDB_SIDECAR_PORT"]) class App(http.server.BaseHTTPRequestHandler): def do_GET(self): if self.path == "/healthz": # the supervisor's health probe self.send_response(200); self.end_headers(); self.wfile.write(b"ok"); return with psycopg.connect(DSN) as c: n = c.execute("select count(*) from orders").fetchone()[0] self.send_response(200); self.end_headers() self.wfile.write(f"{n} orders\n".encode()) # SIGTERM to the process GROUP is the stop signal. Default handling (terminate) # is correct; install a handler only if you need a drain. http.server.HTTPServer(("127.0.0.1", PORT), App).serve_forever() ``` The row the supervisor will build for it: | field | value | already exists? | |---|---|---| | command | `/tenants//run` | new: read, not built by ArgonDB | | environment | empty, plus an explicit list | new | | port | from the tenant's port band, by the shared rule | exists | | health check | HTTP GET on that port, or a TCP connect | new | | log | stdout and stderr, one append-only file | exists | | ownership stamp | `ARGONDB_BOX_INSTANCE` plus the supervisor's record | exists | | restart policy | the standard one — 2 s → 30 s, trips at 6 in 120 s | exists | | isolation | its own unix user and its own cgroup | new | The environment is a closed list: the connection string, the port, and `ARGONDB_BOX_INSTANCE` added back on purpose. It does **not** get `ARGONDB_AUTH_TOKEN`, the lake's credentials, or anything else the supervisor was started with. **What the supervisor will not do for your application:** build, deploy, update, version, roll back or download it. A `run` file that does not exist is refused at start **with the path named**, never silently skipped. #### 4.6 Isolation rung 1: one root command per tenant Here is what rung 1 needs in practice. These probes were run on a real machine on 2026-09-20 as an ordinary user, with no sudo: ``` $ stat -fc %T /sys/fs/cgroup cgroup2fs $ cat /proc/self/cgroup 0::/user.slice/user-1000.slice/session-63939.scope $ cat /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers cpu memory pids $ loginctl show-user "$(id -un)" -p Linger Linger=yes $ systemd-run --user --scope -p MemoryMax=32M --unit=guide02-cap-probe \ bash -c 'cat /sys/fs/cgroup/$(awk -F: "{print \$3}" /proc/self/cgroup)/memory.max' Running as unit: guide02-cap-probe.scope; invocation ID: 3616a8fc799a4d44b15f60245641f8d5 33554432 $ systemctl --version | head -1 systemd 259 (259.5-0ubuntu3.4) ``` **How to read the limit probe.** The scope was created without privilege, and the process inside it read back its own memory limit as `33554432` — 32 MiB, exactly what was asked for. That is proof the limit was applied, not merely the absence of an error. Memory, CPU and process-count limits for a tenant's application cost **no privilege** on a cgroup v2 machine where the `cpu`, `memory` and `pids` controllers are delegated to the user. **The user half needs root once per machine, and then never again per tenant.** Root installs the user helper and its sudoers rule once: ```bash # once per machine, as root — installs /usr/local/sbin/argondb-useradd and # /etc/sudoers.d/argondb-useradd, which lets the supervisor's user run exactly # that helper without a password: sudo /bin/argondb box install-useradd-helper --user ``` After that, ArgonDB creates each tenant's user itself by running `sudo -n /usr/local/sbin/argondb-useradd `. The helper checks the key and runs the one `useradd` line below, and nothing else. The manual command stays available as the fallback for a machine without the helper: ```bash # once per tenant, at provisioning, as root: useradd --system --no-create-home --shell /usr/sbin/nologin argon-t- # once per machine, at install, as root — ONLY if the supervisor is to switch # user itself (it does not run as root and should not start as root): setcap cap_setuid,cap_setgid+ep /usr/local/bin/argondb ``` Giving the supervisor binary `cap_setuid` is a real grant of privilege, and it is made on purpose. The alternatives are a setuid-root helper (worse), or `systemd-run --user` under a user manager that already runs as the tenant's user — which just moves the question to "who creates that user" and lands back at `useradd`. **There is no way to get a per-tenant user without crossing root once.** Put together, starting a tenant's application under rung 1 looks like this: ```bash # per tenant, at run time, no privilege beyond the user switch: systemd-run --uid=argon-t-acme --gid=argon-t-acme \ --scope --unit=argon-sidecar-acme \ -p MemoryMax=512M -p CPUQuota=50% -p TasksMax=64 \ /srv/argon/tenants/acme/run ``` ![The isolation ladder, rung by rung](diagrams/argondb-and-the-erp/02c-isolation-ladder.svg) **Unprivileged user namespaces depend on the machine's kernel policy.** On 2026-09-12, on this same machine, creating one failed (`unshare -U -r` gave `write failed /proc/self/uid_map: Operation not permitted`, because `kernel.apparmor_restrict_unprivileged_userns = 1`). On 2026-09-20 it worked: ``` $ sysctl -n kernel.apparmor_restrict_unprivileged_userns 0 $ unshare -U -r /bin/sh -c 'id -u; cat /proc/self/uid_map' 0 0 1000 1 $ grep "^$(id -un):" /etc/subuid /etc/subgid # account name redacted below /etc/subuid::100000:65536 /etc/subgid::100000:65536 ``` The restriction had been lifted on that machine between the two runs. This does not change the approach. Even with the restriction lifted, a user namespace still gave no second real user there, because the `newuidmap` and `newgidmap` helpers were not installed, so the machine's subordinate id range could not be mapped. And the case against relying on user namespaces never rested only on that one failure: lifting the restriction is itself a root step, and the policy differs between Linux distributions, so `useradd` is the answer that works everywhere. #### 4.7 The restart policy, and what a DEGRADED machine looks like Every supervised process restarts under the same policy, defined once: ```rust pub(crate) const BASE_BACKOFF_SECS: u64 = 2; pub(crate) const MAX_BACKOFF_SECS: u64 = 30; pub(crate) const HEALTHY_RESET_SECS: u64 = 60; pub(crate) const BREAKER_MAX_RESTARTS: usize = 6; pub(crate) const BREAKER_WINDOW_SECS: u64 = 120; ``` So a process that keeps crashing goes: exit, wait 2 s, restart; exit, wait 4 s; 8 s; 16 s; 30 s; 30 s — and on the sixth restart within 120 s the breaker trips. A process that stays up for more than 60 seconds after its last restart has its history cleared, so a process that is mostly fine is not given up on because of one crash. One process has a tighter pair of numbers, for a reason: ```rust pub(crate) const MCP_BREAKER_MAX_RESTARTS: usize = 2; // trips on the THIRD exit pub(crate) const MCP_BREAKER_WINDOW_SECS: u64 = 60; // inside sixty seconds ``` The agent endpoint, `argondb-mcp`, is the only process whose usual failure is a configuration refusal that will never fix itself: it checks every configured surface **before** opening any of them, so a bad setting makes it exit within milliseconds, every time, and every retry reaches the same conclusion. The standard policy would take seven exits and about 90 seconds of back-off to give up — 90 seconds of a machine that looks healthy while its agent endpoint is missing. **How it looks in `argondb status`.** A database that was stopped for being idle, or a pool that is lazily cold, is shown as down with a note saying this is expected: ``` primary Postgres (internal; idle_stop ON) [down] the primary compute is stood down after `[compute] idle_stop` — the next client connection on the public PG port wakes it ``` A **tripped** process is different: the supervisor has stopped restarting it and the machine is DEGRADED. A DEGRADED machine **starts, says so everywhere, and keeps serving what it can.** The lake catalog's authorization check is a worked example: it prints its cause, how long it has been failing and its retry policy on one line, and trips its own breaker after a bounded number of attempts rather than looping: ``` [argondb-box] catalog authz: BREAKER TRIPPED — 10 recovery attempts over 300s all failed; no further automatic retry. The box stays DEGRADED … ``` When tenant applications land, the DEGRADED message will name **which tenant's** application was given up on. The mechanism already exists; only the wording changes. #### 4.8 The refusal that protects the lake This refusal stops you from choosing an idle window so short that new tables would silently stop reaching the lake. It exists because allowing idle-stop on a machine with the bundled lake catalog made a new combination possible: the lake switched on together with idle-stop. **The refusal.** With `[lake] enabled = true`, schema refresh on, and a window shorter than the tail's 30-second schema refresh, the machine refuses to start and nothing is launched: ``` idle_stop is 20s, which is SHORTER than the tail's schema-refresh cadence (30s), so this lake-ON box would silently stop enrolling newly published tables. The tail discovers a new relation by reading the PRIMARY (`refresh_tables`, once per 30s cadence) over an INTERNAL connection that deliberately cannot wake the primary and cannot hold it awake — so at a 20s window the primary is always stood down when the tick fires. Tables already tracked keep materializing (their rows come from the safekeeper and the pageserver, not the primary), which is why this looks like a healthy box. Remedy, pick one: raise `[compute] idle_stop` (or ARGONDB_PRIMARY_IDLE_SECS) to at least 30; or set `[lake] enabled = false` if this box does not publish to the lake; or set ARGONDB_SCHEMA_REFRESH=off to declare the published set STATIC, after which new tables are never discovered by your instruction rather than by accident. ``` **Three settings that do start:** ```bash # (a) exactly the refresh interval. The rule is "strictly shorter"; # 30 is not shorter than 30. ARGONDB_PRIMARY_IDLE_SECS=30 argondb serve --data-dir /srv/argon # (b) the same short window, with the set of published tables declared fixed. ARGONDB_PRIMARY_IDLE_SECS=20 ARGONDB_SCHEMA_REFRESH=off argondb serve --data-dir /srv/argon # (c) the same short window with the lake off entirely ([lake] enabled = false). ARGONDB_PRIMARY_IDLE_SECS=20 argondb serve --data-dir /srv/argon ``` Both sides of the 30-second boundary are tested, in unit tests and on a live machine. The check and the tail read the 30-second interval from one shared constant, and the same number is passed to the tail when it starts, so the check can never compare against an interval the running tail is not actually using. --- ### 5. What is built today and what is designed | capability | state | |---|---| | one primary database stops when idle and wakes on connect | **works today** (since v1.4.0), off by default | | the byte-triggered gate, the pass-through, and the Postgres error messages | **works today** | | waking governed by the same restart policy as crash restarts | **works today** | | the five-step stop, including waiting for the port to be free | **works today** | | several computes on one machine, each with its own idle rule | **works today** as the analyst pool — but its computes are replicas of one database, not separate databases | | idle-stop on a machine using the bundled lake catalog | **works today** (since 2026-09-08) — the catalog has its own always-on Postgres | | the refusal protecting the lake's schema refresh | **works today** | | both `compute_ctl` control servers listen only on 127.0.0.1 | **works today** (since 2026-09-13) | | `ARGONDB_SHARED_PRELOAD_EXTRA` accepts only extensions the build installs | **works today** (since 2026-09-13); list them with `argondb extensions --list` or `--census` | | the pooler: one PgBouncer per machine, transaction mode | **works today** (since 2026-09-20) | | PgBouncer supplied by the platform, not the release | **works today** — the release carries none, and the pooler is optional: a missing or too-old PgBouncer boots the machine on the direct port instead of stopping startup | | the tail as an always-on process with a stated memory limit | **decided; being built** | | one compute per tenant database, each on its own timeline | **works today, off by default** — a company gets its own compute the moment it is created; only a company already on the older v1.5.4 release waits for its first upgraded-machine pass | | rows written before a company had its own compute, copied over once; that company's database roles replayed onto its compute | **works today**, with one known defect in the copy (section 2.7) | | each tenant's pooled connections routed to that tenant's own compute | **works today**, routing proven both ways — but the test that two tenants' data stay apart under this setting has **not** passed on the branch everyone builds from (section 2.7) | | the tail following every tenant's timeline | **works today** | | a tenant's application as its own process, one per tenant | **works today** in a first version, started by the same command that creates the tenant; a separate unix user per tenant needs root and is not built; it is not yet a row in the supervisor's own table, and the gate in front of its port wakes the tenant but does not yet keep it awake | | the application's full lifecycle as a supervisor row — sleeping and waking with its database, health check, restart row | **designed, not built** | | wake requests that name the tenant, one gate per tenant compute | **works today** with the per-tenant table | | the gate's second listener, for the application port | **works today** (since 2026-09-29) — HTTP `503` with a measured `Retry-After` while waking, pass-through once up, `502` when the application refuses; the port is recorded in `compute-slot.json` | | HTTP `503` with `Retry-After` as the second failure vocabulary | **works today** (since 2026-09-29), on the application port only; the SQL port's `57P03` messages are unchanged | | per-tenant unix user and cgroup (isolation rung 1) | **the user half is not built** — the cgroup half needs no privilege, the user half needs one `useradd` per tenant; the user-namespace route lacks `newuidmap` (section 4.6) | | a container per tenant on the one-click cloud VM (rung 2) | **required; designed, not built** | | a tenant's application always a separate OS process | **the rule** | | which cloud the per-tenant model targets first | **AWS first**; Azure follows once its lake settings exist, which are proposed and not built | | a silent WebSocket does not keep a tenant awake | **the rule** — only bytes count | **The default shape is still one primary per machine.** A compute per tenant, each on its own timeline, is there for an operator to turn on; the machine never turns it on by itself. --- ### 6. Gotchas Every refusal, limit and remaining gap you are likely to hit, with what to do about it. **On v1.5.7, a machine that has created a tenant may refuse to start again.** Creating a tenant gives it its own timeline in the machine's storage, whether or not the tenant ever gets its own compute. On v1.5.7 the process that copies changes into the lake attached to whichever timeline the storage layer listed first, instead of the machine's own, and that order changes from one start to the next. When it picked a tenant's timeline, the machine refused to start (`argondb-tail exited before it bound Flight … refusing the boot`). Removing a tenant with a hand `DROP DATABASE` left its timeline behind and caused the same refusal. v1.5.7.1 fixes it: the machine tells that process exactly which timeline is its own, and the process names every other timeline once in its log and leaves it alone. *What to do on a v1.5.7 machine that holds a tenant:* do not restart it on v1.5.7. Stop it, install v1.5.7.1, and start it on the same data directory; nothing in the directory needs changing, even if v1.5.7 already refused to start it. *To remove a tenant,* use `argondb tenant delete `, never `DROP DATABASE`: it removes the tenant's records, its database, its timeline and its directory, and running it twice removes nothing twice. It refuses a tenant that has its own compute, and it leaves the tenant's lake tables for `argondb lake drop-orphan`. **A connection that sends nothing wakes nothing.** It is closed after 30 seconds. A TCP-connect health check, a browser's speculative connection, or `nc -z` will never wake a tenant and never keep one awake. *What to do:* make your monitor send a real query. This is deliberate — it is what stops `argondb status` from keeping every machine awake forever. **The internal port is the public port plus 100 only while idle-stop is on.** With it off there is no internal port, and Postgres owns the public port. Anything that hard-codes 5532 breaks when you turn idle-stop off; anything that assumes Postgres owns 5432 breaks when you turn it on. *What to do:* read the ports from `argondb status`, which shows both shapes, and remember that changing `ARGONDB_PG_PORT` moves both by the same 100. **A tenant compute's ports are a recorded pair, not a plus-100 split.** Each tenant takes the next free pair at or above `tenant_port_base` — the first for clients, the second internal — and keeps it in `tenants//compute-slot.json`, a different arithmetic from the primary's plus-100 split above. *What to do:* do not carry the primary's plus-100 assumption over to a per-tenant compute's ports; read them from the record or from `argondb status`. If something else holds a recorded port at boot, that tenant alone is refused, naming the port and its holder — free the port or move the other listener. **`per_tenant = true` without an `idle_stop` window does not start.** It is refused at startup, naming both keys. *What to do:* set `idle_stop` to a number of seconds, or leave `per_tenant` off. **Under `per_tenant`, a company with no known compute port yet gets no pooled entry at all — on purpose.** Its pooled connection fails, and the log names the company and says plainly that it is not being routed to the primary. The alternative — quietly sending it to the primary anyway — is exactly what this avoids. *What to do:* wait one pass after creating the company, or read the log line. **Under `per_tenant`, the one-time copy of a company's older rows can fail for good.** If a table gets created on a company's own compute before that copy has run, the copy is rolled back and every later pass refuses to retry it, naming the company. *What to do:* there is no fix on the branch everyone builds from today — do not turn `per_tenant` on for a machine whose companies already hold rows on the primary until that fix lands, and check the log for a copy that says it was not done. **`idle_stop = 300` (without quotes) is refused.** The value is a string so that one parser serves both the file and the environment. *What to do:* add quotes. **`idle_stop = "0"` is refused**, not read as "always off" or "always on". *What to do:* use `"off"` to disable it, or a positive number of seconds. **A machine with the lake on refuses a window shorter than 30 seconds.** This is not a warning; the machine does not start. *What to do:* one of the three fixes the message lists — a window of at least 30; the lake off; or the set of published tables declared fixed with `ARGONDB_SCHEMA_REFRESH=off`. Section 4.8 has examples. **Idle-stop is off by default, on purpose.** On a laptop or a single VM, stopping the database saves no money, so the setting buys nothing and adds wake time. It pays off when you run many instances. **Stopping takes longer than the window.** Measured: 34 seconds against a 20-second window. The extra is the shutdown checkpoint plus waiting for the port. *What to do:* in a test, do not assert the process is gone at "window plus a little"; assert it is gone within the window plus 60 seconds. **A stopped database is not a failed one.** Stopping for idleness counts against no breaker and does not trigger the "everything has failed" shutdown. Do not read "asleep" as healthy or unhealthy. **An open wake breaker is not retried automatically.** After six failed wakes within 120 seconds, every connection is refused with a message saying so, until the machine is restarted. *What to do:* read `logs/compute.log`, fix the cause, restart the machine. This is deliberate: an expensive automatic recovery sits behind a breaker, never in a loop. **The 619 ms wake is conditional.** It was measured on a small empty database, a fast NVMe machine, a warm pageserver and the lake off. A large database with a cold pageserver will not match it. **The application's share of the wake is neither measured nor bounded.** It is your application's number. ArgonDB will report it, not limit it. **About 630 computes per machine is arithmetic, not a limit.** Memory pressure was never approached. The first real limit is more likely the pageserver's per-timeline cost, the port band, or file descriptors — none of them measured. The shipped per-tenant table does not get anywhere near that figure either: its own default cap is eight tenant computes per machine, a starting product default rather than a measured ceiling. **An idle machine still logs one warning roughly every two minutes, and it is not a fault.** A routine background check on a landed continuous view's table prints that it is temporarily constrained, even with no writes happening at all — twenty-one times in the measured idle hour. It has been noticed and is a known loose end, not a fix that shipped. *What to do:* do not page anyone on that one line alone for an otherwise idle machine; page on an actual view warning instead, which stayed at zero the whole hour. **The gate only speaks Postgres errors.** A client of a web application behind the gate whose wake failed would receive a Postgres error on what it believes is an HTTP connection. The HTTP error path is proposed, not built. *What to do:* do not put this gate in front of a web application yet. **A busy application is not a reason to keep the database up.** Deliberately: a hold is a reason the database itself cannot be stopped. A busy application would count through the gate's byte counter instead, and that part does not exist in the shipped code. **A silent WebSocket does not keep a tenant awake — by design.** Only bytes count. A browser holding a socket open with nothing sent for the idle window lets its tenant stop like anything else; the client reconnects when it next sends something and the gate wakes the tenant, but server push over that silent socket does not survive the stop. *What to do:* make any client that depends on server push handle reconnecting. There is no setting to hold the tenant awake instead. **A PgBouncer older than 1.25.2, or one that cannot report its version, is treated as absent, not refused.** The machine falls back to serving the direct port only and says so; it does not stop startup. *What to do:* check the resolved path and version printed at startup, or `argondb status`, before assuming the pooled port is live. **ArgonDB does not ship PgBouncer, and this is settled — the platform always supplies it.** It must come from your platform — on the `PATH` or at `ARGONDB_PGBOUNCER_BIN`. *What to do:* install it as part of provisioning if you want pooling; the machine still starts on the direct port without it. **Rung 1 alone is not enough on the one-click cloud VM.** That target requires rung 2, a container per tenant, on top of rung 1 — and rung 2 is designed, not built. The rung-1 recipe in section 4.6 is for on-premises machines only. **Rung 1 does not protect against a kernel escape.** A unix user is not a VM. If tenants were ever hostile to each other, rather than one customer's own tenants, rung 1 would not be enough and stronger isolation would be needed. **Rung 1's user half needs root once per tenant.** There is no way around it. Plan one `useradd` per tenant at provisioning. **Unprivileged user namespaces may or may not work on a given machine.** The same machine refused them on 2026-09-12 and allowed them on 2026-09-20 (section 4.6). *What to do:* never build provisioning that assumes either state; check, and keep `useradd` as the portable path. **`ARGONDB_MCP_PORT=0` turns off the analyst pool.** `argondb status` says so by name, but the link surprises people: the pool's only user is the agent endpoint. **A lazy pool's computes show as down in `argondb status`.** That is their normal resting state, and the status note says so. **A tenant application must keep `ARGONDB_BOX_INSTANCE`.** Its environment starts empty, which is right, but this one variable is added back on purpose; without it the supervisor could not identify and clean up the process, which is worse than the leak the empty environment prevents. **`compute_ctl`'s internal HTTP server has no authorization of its own.** It serves `/extensions`, `/extension_server/{*filename}`, `/grants` and `/refresh_configuration`, and in testing it answered an unauthenticated `POST /extensions` with a 422 naming the field it wanted. ArgonDB closes this by binding that server to `127.0.0.1` only, with no setting that can widen it. *What remains:* the SQL port still follows `ARGONDB_LISTEN_ADDRESS`, so widening that widens the SQL port and nothing else. Check what is actually listening rather than assuming. --- ### 7. More diagrams Three more detailed pictures of parts the main diagram compresses. They use the same legend: solid outline means built and shipping, dashed means designed and measured but not built. | diagram | what it shows that the main picture cannot | |---|---| | [2a — wake and stop, with every timeout](diagrams/argondb-and-the-erp/02a-wake-and-stand-down-sequence.svg) | both sequences across four columns (client, gate, supervisor, database) with every time limit on the path — 30 s for first bytes, 180 s wake deadline, 120 × 500 ms connect wait, 60 s grace, 60 s wait for the port, 2 s monitor interval — the four "held awake" reasons in order, the two excluded application names, and each refusal message at the step that produces it | | [2b — the supervisor's process table](diagrams/argondb-and-the-erp/02b-supervisor-process-table.svg) | every always-on process that exists today with its restart rule and memory limit, the one process whose breaker is different and why, and where a tenant's two rows attach — with the four shared fields spelled out | | [2c — the isolation ladder](diagrams/argondb-and-the-erp/02c-isolation-ladder.svg) | all four rungs side by side — what each is, buys, costs, and does **not** cover — plus which rung each target uses, and why a cloud VM is rung 3 for the machine but rung 1 for the tenants inside it | --- ## Guide 3 — the lake and its process model This chapter is for a developer deciding whether to build on ArgonDB, and for whoever will later operate or debug it. It explains how rows you write to Postgres become Apache Iceberg tables (the "lake"), how a read of the lake stays consistent with the database, and which processes on the box do the work. It also states the limits: what is measured, what fails and how, and what is designed but not built yet. One naming note before we start. Our decoding process is called **the tail** (`argondb-tail`). In ArgonDB the word "sidecar" means the tenant's own application process, never one of ours. #### In short: what you type, what you stop doing, what you will be refused This summary exists so you can decide quickly whether the model fits your application; every item is explained later in the chapter. **What you type.** - `argondb lake publish ` to put a table in the lake (or set the `argon:publish` marker in the table's comment yourself). Add `--old-row-from-wal` for a heavily updated table to make decoding it much faster. - `argondb lake unpublish
` (optionally `--purge`) to stop. - `argondb lake address`, `argondb lake namespace`, `argondb lake list` and `argondb lake catalog-tables` to find out what the lake calls your table and what state it is in. - `argondb-query` with `--at-lsn` and `--scope` to read at a position; or SQL against `lake_.*` foreign tables from Postgres; or plain Iceberg SQL from DuckDB, Spark or Trino. - `ARGONDB_CHANGELOG` to turn on a per-table change log, and a handful of `ARGONDB_TAIL_*` environment variables to size memory. **What you stop doing by hand.** - No change-data-capture pipeline: no connector, no topic, no sink, no logical-replication slot, no triggers, no second copy of the schema. - No reconciliation job between the lake and the database. Both come from one decode, so they cannot drift. - No compaction, snapshot-expiry or orphan-cleanup jobs. ArgonDB runs them, always, every 120 seconds. - No bookkeeping of "how far has the lake got". Every Iceberg snapshot carries its own position, and restart resumes from it. **What you will be refused, and what it means.** - A read pinned to a position a table cannot serve exactly, a time-travel read into a recorded gap, a time-travel read below a history restart (repin forward, never back), a scope naming a table that does not exist, or a position outside the 48-hour retention window. Each refusal names the table and the fix (section 3.2). - Publishing TEMP or UNLOGGED tables (they write no WAL). - Writing to a `dv_*` table through any door but its one owner (section 2.7). - `0` as the ring memory budget (section 6.6). - A catalog that cannot commit several tables atomically: the tail refuses to start against it (section 2.5). ### 1. The diagram ![Deep dive 3 — the lake and its process model](diagrams/argondb-and-the-erp/03-the-lake-and-its-process-model.svg) The picture shows the whole path from a Postgres write to an analytical read, in four frames. Read them in order. The top frame is the **write path**. An ordinary `INSERT` commits in Postgres and its WAL (the write-ahead log) goes to the safekeepers. `argondb-tail` reads that WAL as a physical standby would. It decodes heap and catalog pages, the commit log (CLOG) and WAL records into typed row changes in Apache Arrow format, and holds them in an in-memory **ring**. On a regular cadence it lands everything the ring holds as **one atomic multi-table commit** into Iceberg. You configure none of this. There is no connector, no topic, no sink and no second copy of the schema. The left-middle frame is **what lands**: a standard Iceberg table per published relation, optionally a `__changes` change log beside it, and two system tables per branch. The right-middle frame is **who reads it**, and it is one rule. Every read names one position `L`. Every table in the read's scope is served at exactly that position, or the read is refused. The bottom frame is the **process model**: five supervised processes on one box, each with a memory cap and a restart rule. Two positions carry the design: - `H`, the **head**, is the newest commit fully decoded and held in memory. - `W`, the **watermark**, is the newest position durably committed to Iceberg. It is stamped on every snapshot a commit touches. `W ≤ H` always. The interval `(W, H]` *is* the tail: decoded rows that exist only in the tail's memory, served over Arrow Flight, and dropped as soon as `W` moves past them. A read at `L` is `Iceberg@W ∪ tail(W, L]`, merged by primary key. That is why the lake never drifts from the database. The lake copy and the not-yet-landed remainder are two halves of one decode, not two systems that have to be reconciled. Dashed outlines in the picture mean *designed and measured, but not built yet*. On this diagram two items carry that status: the Iceberg v3 writer and the "speed" trigger of the HTAP offload. Section 5 gives the evidence for each. ### 2. What it is This section introduces each part of the system on its own: what it is, why it exists, and the numbers that define it. Section 3 then shows how the parts work together over time. #### 2.1 The application and the primary Postgres Your application writes to a real Postgres, and nothing about that changes. This is the most important promise ArgonDB makes to an application developer. Any SQL, type, extension behaviour or driver behaviour that the Neon Postgres fork supports works unchanged. ArgonDB's additions must not alter the transactional write path or the OLTP read path. There is no trigger, no logical-replication slot and no publication, and by default no `REPLICA IDENTITY FULL`. The decode is **physical** — heap pages, catalog pages, CLOG and WAL records — which is exactly why your schema, your WAL settings and your writes stay untouched. The one exception is something you opt into per table: `argondb lake publish
--old-row-from-wal` sets that table's `REPLICA IDENTITY FULL` so the tail can read old row versions straight from the WAL, which makes decoding much faster (section 3.9). It is off unless you ask for it. A second, separate rule is that ArgonDB's additions must not *load* the primary either. The first rule forbids changing how writes behave; the second forbids adding work to the machine that serves them. A change can satisfy one and break the other, so ArgonDB checks them separately. #### 2.2 The safekeepers and the pageserver The tail needs two things from storage: the WAL, to learn what changed, and old page versions, to decode those changes. This section explains where each comes from, and the one mechanism here most likely to surprise you: the tail can hold WAL on disk. Each tenant database is its own Postgres instance on its **own timeline**, with its own WAL. What the tenants share is the storage plane: one pageserver, one safekeeper set and one lake serve every tenant timeline. The WAL is made durable by the safekeepers. A single box runs **one** safekeeper; the separated (cloud) topology runs three, agreeing by Paxos consensus. The pageserver serves versioned pages by `GetPage@LSN` (the page as of a given log position). The tail reads each timeline's WAL from a **safekeeper** and its pages from the **pageserver**. On the box the pageserver requires a token on both of its ports; the tail carries one scoped for that purpose, and a tenant compute's token names its one timeline and is refused on any other. The tail also holds one Postgres connection per tenant timeline it follows — to that tenant's own compute — and this is the one thing about the tail that is easy to get wrong when reasoning about isolation. That connection is used *only* for schema introspection: at boot, once per cadence, and at the write-floor flip. It uses an internal port and an internal `application_name`, so it never wakes a stood-down compute and never resets its idle clock. If a schema refresh fails while the compute is stood down, the tail logs a warning and retries next cadence. No schema can change while the compute that would change it is stopped. **The tail holds WAL on the safekeepers.** The tail is one of the terms that decides when a safekeeper may delete old WAL. It reports its resume point as the `flush_lsn`/`apply_lsn` of the standby status update it already sends, under `application_name=argondb-tail`. The safekeeper takes the minimum over all live `argondb-tail` sessions, **persists it to `argon_tail_hold.json`** beside `safekeeper.control`, and keeps WAL back to that point. The hold is capped at `ARGONDB_TAIL_WAL_HOLD_CAP_BYTES` (default 4 GiB) below `remote_consistent_lsn` (the position the pageserver has durably uploaded). Persisting the hold is the point. A box restart is exactly the moment when no tail session is live, so the file is loaded before the safekeeper's first WAL-removal pass. When the tail falls further behind than the cap, removal advances to `remote_consistent_lsn − cap`. For each WAL segment given up this way the safekeeper logs one line, `ARGON tail WAL hold OVER CAP`, naming what was forfeited. The next tail restart will then report `RESUME POINT UNSERVABLE at boot` and rebuild the affected tables from scratch, with a recorded gap (section 3.4). The hold is **not a lease**. A tail that has been removed keeps pinning up to the cap of WAL until `argon_tail_hold.json` is deleted with the box stopped. #### 2.3 `argondb-tail`: decode The decoder is the heart of the lake. It turns physical storage into logical, schema-typed row changes, and everything else in the lake is fed from its output. It produces inserts, updates and deletes in Arrow format, with MVCC visibility resolved. Uncommitted and aborted transactions never appear in any output, lake or tail. This is the **single decode path**: the ring, the change log, the lake copy and every read all come from it. There is no second derivation anywhere in the system, which is what makes "the lake cannot drift" a structural statement rather than a quality claim. **Row identity is the primary key, when there is one.** A table with a primary key gets full incremental insert/update/delete handling via an Iceberg **RowDelta** commit: data files and equality-delete files in one snapshot. A table without a primary key is supported, not a gap. Accuracy is identical. But its lake copy is maintained by a full-table **scan-replace** in every cadence in which the table changed. A missing primary key therefore costs update performance, both in time and in load on the whole system. Scan-replace is also used during catch-up, and as the automatic per-table fallback on any error. The practical advice is one line: put a primary key on every published table. #### 2.4 The ring `(W, H]` The ring is the tail's in-memory buffer. It holds rows that have been decoded and committed in Postgres but are not yet in Iceberg, so that a reader can see them immediately rather than waiting for the next lake commit. Its budget is `ARGONDB_TAIL_DRAIN_QUEUE_MB`, **2048 MiB by default**. Two things about it matter before you size anything. **First, the ring counts allocated bytes, not rows times row width.** It uses Arrow's own measure of each batch's allocated buffer capacity. That is deliberate: the problem the budget solves is process memory (RSS), and a batch of three rows whose builders rounded twenty columns up to 64-byte buffers costs the process what it allocated. The ring holds **one Arrow `RecordBatch` per (commit, table) pair**. On the ERP benchmark workload (about 36 rows per transaction, spread over ten published tables) that means mostly single-digit-row batches, and that is where the cost comes from. Measured on 2026-09-19: - one batch per commit: **364.2 bytes per row**; - the same rows coalesced into one batch: **76.7 bytes per row**; - a ratio of **4.7×**, with **78.9 %** of the ring's counted bytes being batch overhead at about 1,150 bytes fixed per batch. `ARGONDB_TAIL_RING_COALESCE_ROWS` (default **512**) exists to remove that overhead by merging small batches. **Second, counted bytes are not resident bytes.** Measured directly: a ring high-water of 2,064.4 MiB (`drain_queue_bytes_high_water`) against a tail peak resident size (`VmHWM`) of 3,217.2 MiB, a ratio of **1.56×**. Size your machine against the process bound in section 3.5, not against the ring budget. A figure of 22 KiB per ring row circulated in earlier versions of the design. It is **withdrawn**. It was an RSS peak divided by a row count sampled by a different loop; the two numbers were never read together, and nothing in the current design rests on it. **Setting the budget.** `0` is **refused**, not read as "unbounded", because unbounded is exactly what the budget exists to prevent. An unparseable value falls back to the default with a `WARN` saying so; it never turns the budget off. The value is read **once**, when the ingest loop starts, so it cannot change under a running loop. #### 2.5 The cut A cut is how rows move from the ring into Iceberg. It is one pass of the tail's cadence loop, and it is what makes the lake consistent across tables. Each cut does five things: 1. Choose a target position `W'` on a commit boundary. A Postgres transaction is never split. 2. Compute the net change per primary key and prepare a RowDelta commit for each table that has one, or a scan-replace for the rest. 3. Prepare the change-log appends. 4. Land **everything touched in `(W, W']` as one atomic multi-table commit** through the Iceberg REST catalog's `/transactions/commit` endpoint, stamped with `Argon.watermark-lsn = W'`. 5. On success, advance `W`. The tail then drops ring rows at or below `W'`. **Atomic multi-table commit is a catalog requirement, not a preference.** Without it the consistency guarantee in section 2.9 cannot hold, and no weaker mode is offered. The obvious alternative — commit each table separately and let them trickle in, which is what typical CDC pipelines do — would make consistency depend on which catalog you chose, and a guarantee that holds only on some catalogs is not a guarantee. At startup the tail probes the catalog for this capability and refuses clearly if it is missing. **One transaction per warehouse.** Under the tenant wall (see [Guide 1](#guide-1-one-database-per-tenant-and-the-wall-between-tenants)), each non-primary database has its own Lakekeeper warehouse. So the single commit buys atomicity *within one tenant*, not across tenants. **A cut on demand.** Cuts normally run once per `[lake] cadence_secs` (30 seconds by default), but a caller can ask for one sooner. A publication landing must read rows at or after the moment the landing started. If its first read falls short of that, it asks the tail for a cut. The request does not start a second kind of cut. It wakes the same loop early, the landing waits for that pass, and then it reads again at the position the cut landed. Requests that arrive together share one cut, so ten landings pay for one cut, not ten. If the cut is refused, the landing is told which database and warehouse refused it. It then waits for the next regular cut, as it did before this feature existed, and says why. When nothing asks, the loop cuts on its normal cadence. #### 2.6 The published table, and the `__changes` change log A published relation lands as an ordinary Iceberg table that any Iceberg engine can read. Next to it you can ask for a change log: a second table that records every row change with its position, which is what makes row-exact time travel and change digests possible. **Where the table lands.** The address is `branch / database / schema`, with the bare relation name as the leaf — for example `br_main/postgres/public` + `orders`. Each cut that touches the table writes one snapshot carrying three properties: `Argon.watermark-lsn`, `Argon.branch` and `Argon.txn-range`. These names — the `Argon.*` snapshot properties, the `_argon_` column prefix, the `__changes` suffix and the `argon:` comment markers — are fixed wire contracts. They will not be renamed, because foreign engines and existing deployments read them. **The change log is opt-in, per table.** You turn it on with `--changelog` or `ARGONDB_CHANGELOG` on the tail. The value is a comma-separated list of **flat lake keys** (like `postgres__public__orders`), not relation names (see example 10). The change log is an append-only table of **rows**: no row is ever modified or removed except by retention. Its schema is the source table's columns followed by four extra columns: | column | type | nullability | meaning | |---|---|---|---| | `_argon_lsn` | `long` | required | the commit position of this row change | | `_argon_op` | `string` | required | `I`, `U` or `D`; a delete tombstone carries the key | | `_argon_seq` | `long` | required | ordering within one commit, per table | | `_argon_ts` | `timestamptz` | **optional** | commit wall clock, in microseconds; optional so that older change logs can evolve cleanly, and their old rows read null | The change log comes from the same single decode and is committed in the **same** atomic, watermark-stamped transactions as its base table. A change log therefore can never disagree with its table about what happened at a position. **Retention.** The change log is kept for at least the pin window (48 hours, configurable upward). Time travel rebuilds a past instant from a retained snapshot plus change-log replay, so a change log that expired early would strand pins that are still valid. **It is not append-only at the snapshot level.** Maintenance compaction may commit an Iceberg Replace snapshot. Rows are preserved exactly and every ArgonDB reader is unaffected. But a foreign engine doing standard Iceberg incremental reads must fall back to a full re-read when it meets a replace snapshot. #### 2.7 `dv_`, and the single-writer rule Every lake table has exactly one writer. This rule exists so that no two processes can ever race to commit the same table, and it is why lake tables that ArgonDB derives (rather than copies) live in a separate namespace. There are three writers, each with its own territory: - The **committer** (inside the tail) alone writes `br_`, the copies of your Postgres tables. - The **lake writer** alone writes `dv_`: continuous views (see [Guide 4](#guide-4-continuous-materialized-views-and-a-view-across-the-tenant-boundary)), transforms run with `run_transform(landing: lake)`, and rows added with `append_rows`. - The **`argon_transform`** Postgres role alone writes the `derived` Postgres schema. Database grants enforce this. `dv_` is a separate namespace because of ownership. The tail's maintenance plane decides ownership at the namespace root. A foreign writer nesting tables under our root is still treated as foreign, and a table owned by the lake writer is never something the committer might also decide to write. **Inside `dv_` the rule goes one level finer: every table there has exactly one writer, and it is checked on every write.** A continuous view's table is written only by the view maintainer. A table that is not a view is written only by a transform landed with `run_transform(landing: lake)` or by `append_rows` — those two count as one owner, so one may append to what the other created. Any other door is refused, and nothing is written: ```text lake-writer/not-owner: is a continuous view — its one writer is the view maintainer, and the `` door is not its writer — every dv_* table has exactly ONE writer; nothing was written. Land under another name, or remove through the door that made it ``` The one exception is a table that was just created and has no snapshot and no view declaration yet: it has no owner, so any door may take it — otherwise the refusal would block the view maintainer's own first refresh. **Nothing may enter a published `br_` table from outside the WAL.** No analytics-side write ever lands in `br_*`. ArgonDB detects a foreign snapshot on a published table, refuses to treat that table as unmaterialized, and reports it. If you need to bulk-load, do it *through* Postgres so the rows get real log positions. A row with no log position and no Postgres transaction is refused; it is never re-baselined into place. Where a watermark can be recovered, the table keeps flowing. Where none can, the table is held (quarantined) with a gap record. #### 2.8 `_argon_gaps` and `_argon_value_notes` These two system tables record the two ways the lake can differ from the database: a stretch of history the tail could not decode, and a value the lake cannot represent. They exist so that neither difference is ever silent. **`_argon_gaps`** sits at the **branch** level of the namespace and records every re-baseline the tail performs. Its schema: | column | type | nullability | meaning | |---|---|---|---| | `start_lsn` | `long` | required | the gap covers `(start_lsn, end_lsn]` — half-open, like the tail | | `end_lsn` | `long` | required | | | `ts` | `timestamptz` | required | when the gap was **recorded** | | `reason` | `string` | required | the explanation; rows written before the `forfeits` column existed carry their count inside this text | | `table_name` | `string` | optional | `NULL` = every tracked table; otherwise the one lake table | | `forfeits` | `long` | optional | how many commits an undecoded-window barrier episode merged into this one row. Absent (null) is **not** the same claim as `0` | **`_argon_value_notes`** records values Postgres can hold but the lake cannot: `NaN` in a bounded `numeric(p≤38,s)`, and `±infinity` in `date`, `timestamp` or `timestamptz`. These map to NULL in the lake, with a durable record and a disclosure on reads through **both** serving paths. Floating-point columns are not affected: both sides use IEEE 754, so NaN and ±Inf pass through untouched. Both tables are included in **every** pin session, scoped or not, because a digest must see recorded gaps and a value note must reach every read it affects. #### 2.9 The query plane The query plane is what turns "Iceberg plus the ring" into one answer that matches what Postgres would have returned. It is `argondb-query`, which implements the unified read `Iceberg@W ∪ tail(W, L]`, merged by primary key, as a DataFusion table provider. **The consistency guarantee.** A read at `L` returns, for every table in the pin's scope, exactly the committed state Postgres would show at `L`. Effects of transactions that committed at or before `L` are fully visible; all others are fully invisible, across all the scoped tables. The guarantee never extends beyond the scope that was pinned. This guarantee is absolute. Two others are goals, not guarantees: - **Freshness.** The tail's head should be within about 1 second of the compute's flushed WAL, and the Iceberg watermark should trail by no more than the commit cadence (default target ≤ 60 s). - **Read-your-writes.** A reader should be able to see its own just-committed writes. That asymmetry is the contract: **correctness is absolute, timeliness is negotiable.** A slow tail makes reads less fresh, and that is visible and bounded. A read that mixes positions would break correctness, and there is no acceptable degraded form of that. #### 2.10 The pin A pin is how a reader names the position `L` it wants to read at, and carries that choice across several queries. It is a **signed value token**, not a server-side handle, so it survives restarts and needs no registry. The token carries `(branch, LSN, scope, minted_at, coordinate-provenance)`. The provenance field records whether the position was chosen by the server at the protected default or named by the caller. That field decides whether a later redemption of the token may refuse. - **Scope** is the set of tables the consistency claim covers. An explicit `pin()` defaults to `all` — the atomic-cut guarantee across every table. Reads without a pin scope themselves to the tables the query touches. A pin is never refused because of data it does not cover. - **Freshness** defaults to `latest_servable`: the newest position that will still be servable for the token's *whole validity window*, not merely the newest one servable at the moment of minting. The alternative is `freshness=head` (section 6.9). Since 2026-10-22 the proof that a chosen position is durable for every tracked table is one record per landing cut, not a stamp on every quiet table: the cut writes a single property-only commit on the lake-owned `_argon_estate` table saying "every tracked table not listed here was unchanged through this cut", the pin reads it once, and a quiet table is treated as durable through that cut from its own last stamp. A landing cut stamps every clean table — by its own stamp or by that record — and a fresh tail's first cut does too; a cut that landed nothing and answered no landing's ask is the idle cut, and a landing's cut-on-demand ask counts as a landing (2026-10-24). When the table later lands, its landing snapshot records the position its previous rows held through, so a token minted inside that window still redeems. The record applies only to the set of tables it names, and a reader on older bytes simply sees the per-table stamps and snaps to an older position — never a stale one served as durable — which is why a tail and its readers are upgraded together. - **Validity** is the retention window (default 48 hours) and does not depend on any server process staying up. A token minted at the default survives tail restarts **and** watermark advances for the whole window. Redeeming it rebuilds the session on demand; the session cache only makes that faster. - **There is no heartbeat and no live-pin registry.** Safety against garbage collection is arithmetic. Snapshot expiry keeps the newest N snapshots, plus the time-travel **anchor** (the newest snapshot at or older than the window), plus — on a table without a change log — *every* watermark-stamped snapshot inside the window, because on such a table those watermarks are the only servable positions. No valid token can point below the window, so orphan cleanup needs no registry. #### 2.11 The MCP endpoint and the analytical Postgres Besides `argondb-query`, there are two more ways to read the lake: through the agent endpoint (MCP), and from inside Postgres through foreign tables. There is also a third read surface that does not use the lake at all. - **`argondb-mcp`** on port `:8018` serves four mounts, one token each: `argondb` (read-only), `argondb-dba`, `argondb-admin` and `argondb-write`. See [Guide 7](#guide-7-agents-and-the-mcp-endpoint). - **The analytical Postgres** reads the lake through `lake_.*` foreign tables, mounted by the `argondb_lake` foreign-data-wrapper extension. These **always** read through the query plane (example 5). - **The analyst computes** read Postgres *pages*, not the lake, so whether a table is published does not constrain them. **The extension set is fixed.** The box's compute is a separate OS process whose only entry point is the SQL wire protocol. That would stop being true if a client could `CREATE EXTENSION` its own C code, so the shipped extensions are a fixed list rather than whatever the build happens to install: - core PostgreSQL (vendored); - `pgcrypto`, `btree_gin`, `btree_gist`; - ArgonDB's pure-SQL `uuid-ossp` shim; - the `neon` preload with its required `neon_rmgr` (the preload loads it internally; a tier without it starts no compute at all); - `pg_trgm`, required because Lakekeeper's own catalog migration creates it and the box will not boot without it; - the lake reader, `argondb_lake`. Ten of the eighteen extension-like components the build used to install have been removed from the build entirely, not merely from the preload allowlist. `CREATE EXTENSION` for any of them fails. #### 2.12 The five processes on the box `argondb serve` starts and supervises everything the lake needs. Knowing the five processes, and which of them may be scaled to zero, tells you what runs all the time and what it costs. | process | what it does | scale to zero? | |---|---|---| | **`argondb-tail`** | decoder, committer, the Flight ring. Resident; durable once per cut | **no** — a tail at zero is a stale lake | | **`argondb-maintenance`** | compaction, clustering, Puffin statistics, snapshot expiry, orphan cleanup. Every 120 s, no off switch | **yes**, on a schedule | | **Lakekeeper + catalog-pg + OpenFGA** | the Iceberg REST catalog; atomic multi-table commit probed at boot | **no** | | **`argondb lake-store`** | the bundled S3-compatible object store | — | | **`argondb-mcp` + the analyst pool** | the agent endpoint and the read-only analyst computes | technically yes, in practice no (cold start about 35–49 s) | **Past a single box**, the tail is **one process multiplexed across every tenant's timeline**. It is not one tail per tenant, and not one tail per box. It runs on a shared estate instance and holds N streams, one per tenant timeline, each with its own member set, its own watermarks and its own replay budget. It never runs on a tenant's own instance, because a tail co-located with one tenant could decode every other tenant's WAL. With the cloud deployment's three-member safekeeper quorum, "redundant" means **one multiplexed tail with a warm standby**: a second tail on an estate instance in another availability zone, which takes over on a fenced lease with no committed transaction lost. It does not mean one tail process per tenant. **What works today** is the multiplexing: one tail process follows every tenant timeline as its own stream on one box. Streams are added and removed while the process runs; the tenant set is read from `/tenants/*/identity.json`, which `argondb tenant create` writes. One stream stalling does not hold up the others, and a stream whose timeline is gone is named and dropped while the rest keep running. With no tenants, the tail behaves exactly as a single-timeline tail. Limits of the multiplexed tail today: - **One Flight port per stream.** The Flight ticket format carries no tenant, so stream *k* listens on the base `--listen` port plus *k* (8817, 8818, …). A stream that would need a port past 65535 is refused rather than wrapped. A single routed endpoint is not built. - **Memory grows per stream**: about +7.9 to +9.8 MB for each loaded stream and +1.4 to +2.8 MB for each branch-only stream with no writer, measured in one process. At 100 tenants that is about 1 GB, so budget for it. - **Not built yet**: the warm standby and its fencing lease, and a guard that prevents two tail processes from writing the same tenant's lake. The missing guard is harmless today, because the supervisor starts exactly one tail and restarts it only after it exits; a standby cannot be added safely until the guard exists. On one box the safekeeper member set is a list of one. ### 3. How it works This section follows the parts of section 2 through time: when a cut fires, what happens under memory pressure, how a read is assembled, what happens on a gap, and how the tail restarts. #### 3.1 The ring, the cut, and the positions over time ![3a — the ring, the cut, and the coordinates W and H over time](diagrams/argondb-and-the-erp/03a-ring-and-cut.svg) A cut can be triggered in several ways. Knowing all of them lets you predict how many Iceberg snapshots your tables will get, instead of guessing. A cut fires at **whichever comes first** of the first three conditions, subject to the fourth. 1. **The clock.** `--cadence-secs`, which the box sets to **30 s** rather than leaving it to the tail's own default. The number the box checked `[compute] idle_stop` against at startup must be the number the running tail actually sleeps on; "both defaults happen to be 30" would be a coincidence, not an agreement. 2. **The ring's high-water mark**, 50 % of its budget (1 GiB at the default). The cadence loop is woken and cuts now. Its sleep is a wait on "clock or high-water, whichever first"; nothing downstream changes. 3. **The lake writer's byte trigger**, per table: **128 MiB** of staged Parquet. The writer can decide it is due between two clock ticks. 4. **The floor**, which is the one that says *no*: `ARGONDB_TAIL_MIN_CUT_SECS`, default **5** seconds (the same value as the lake writer's own default cadence). It limits the snapshot rate. The reason is downstream: the maintenance worker's compaction thresholds are multiples of the cadence, so an unlimited byte-triggered cut every ~2 s at peak would produce 15× the expected rate of manifests. There is one more threshold, and it is **backpressure, not cadence**: the lake writer's staged ceiling of **1 GiB per table**. At the ceiling **nothing is written**, and the refusal names the ceiling, the currently staged bytes and the next cadence tick. **When the ring is full.** At 100 % of the ring budget, the decoder stops pulling from the WAL stream until the ring is back under the high-water mark. Nothing is ever dropped; the ring is only discarded by the explicit trip and re-baseline paths. The safekeeper's TCP window backs up, and the WAL stays on the safekeeper under the hold (section 2.2). **Backpressure is a named phase, not a stall.** Three parts of the tail ask "is this decoder stuck?": - the lag governor's "not advancing" verdict, which can lead to a forfeit; - the self-heal decision; - the liveness watchdog's "wedged" verdict ("no WAL event and no WAL bytes for 60 s"). A decoder that is not reading *because the ring is full* is not stuck, and none of the three may treat it as stuck. So: - the ingest loop prints `ingest phase=backpressure` on its liveness line, alongside `await-event` and the other phases; - a backpressured window is **excluded** from the "did not advance" measurement; - the watchdog's "no WAL bytes" clock does not run while backpressured. All three use one shared check. Deadlock is impossible by construction, and a test asserts it: the cadence loop that empties the ring runs on its own task and never waits on the decode loop, and the decode loop holds no lock the cut needs. #### 3.2 What is atomic, what is idempotent, what is refused This section states exactly what you can rely on when things go wrong: which steps are all-or-nothing, which can be safely repeated, and the complete list of reasons a read can be refused. **Atomic.** One cut = one multi-table commit = one Iceberg snapshot per table touched, base table and change log together, all stamped with the same `Argon.watermark-lsn`. There is no partial step, and tables never trickle in one at a time. **Idempotent.** Every stage is idempotent by log position. The durable record of progress is the Iceberg snapshot metadata itself; there is **no separate progress store**. On startup the committer reads each table's `Argon.watermark-lsn` and resumes from the **minimum** over them — not the maximum. Per-table watermarks are not guaranteed to be equal after a crash, because scan-replace fallbacks land per table, *outside* the atomic cut. Resuming from the maximum would silently skip a lagging table's unapplied changes. Each table then applies only changes above its own watermark, so the overlap is deduplicated by construction. Commits are conditional (compare-and-swap in the catalog), and a half-finished commit interrupted by a crash is retried and deduplicated by the same watermark check. **Refused, and with which message.** Only five things can refuse a read. Each refusal names the blocking table and the remedy. - A pin scope naming a table that does not exist: `pin scope names unknown lake table(s) [...]; published tables: [...]`. - Time travel whose change-log replay window overlaps a recorded gap: `time travel of to refused: the changelog replay window ( … ] intersects a recorded gap ( … ] () — the range is unreconstructable; pin at or after , or repin for a fresh view`. This one is **permanent**: those rows were never decoded, and no amount of retention brings them back. - Time travel on a table without a change log, to a position that is not one of its retained watermarks: ` has no changelog, so time travel can only land exactly on a retained snapshot watermark (nearest at or below is ); enable the changelog for row-exact history`. You can reach this **only** through `freshness=head` or an explicit `at_lsn`, because then you named the position and own its lifetime. - Time travel to a position below a **history restart**. Occasionally a whole-table rebuild cannot commit in place — a Postgres schema change that Iceberg cannot express in place is the usual cause — and takes a staging swap instead: the old table is dropped and a fresh one renamed over it. The new table keeps no snapshot below its own watermark. A read below that position is refused, naming the restart's **first servable position** as the remedy: `time travel of to refused: a whole-table materialize restarted this table's history at (), so every position from down lost its base — this is NOT retention expiry and repinning at an older position will never succeed; the new history's first servable position is : repin at `. This check runs **ahead of** the gap check, because the gap check's own "pin at or after" position can itself sit below the restart, and only the restart's message names one that still has a base. A position at or above `` is served normally. Repinning older — the usual reflex for a refusal — never helps here, and that is exactly the difference from retention expiry: the message says so. - `L` outside the retention window. **A position chosen by the server is never refused.** When `latest_servable` finds no single position that every scoped table can serve, the pin is still granted. Each table that cannot serve the chosen position is served at *its own* newest retained watermark at or below it, and the mint discloses, per table, the older position it reflects. That disclosure is what separates this from silent staleness, and the read path prints it verbatim (example 7). **Disclosed.** Reads without an explicit pin must say so when the session is serving meaningfully behind the live database. "Behind" is measured against the **pageserver's position**, not only the decoded head, because during catch-up the head sits at the watermark. The threshold is **16 MiB** of WAL (unchanged since 2026-08-02). An active catch-up always discloses. #### 3.3 The unified read, per table ![3b — the unified read: Iceberg@W ∪ the changes in (W, L]](diagrams/argondb-and-the-erp/03b-unified-read.svg) The unified read is how one answer is built from two sources: the durable Iceberg snapshot and the changes that came after it. It is the same procedure for a fresh read and for time travel. 1. Pick `L` — by default the current head `H`, or supplied by the client for read-your-writes — and a scope. 2. For each scoped table, pick the Iceberg snapshot with the **greatest `W ≤ L`**. Any *retained* snapshot qualifies, not only the current one. That is what makes time travel the same code path as a fresh read. 3. Fetch the changes in `(W, L]`: from the **ring over Arrow Flight** when `L` is above the table's current watermark, otherwise from the **`__changes` change log**, whose rows carry `_argon_lsn` and `_argon_op`. 4. Merge: apply upserts and deletes over the Iceberg scan, keyed by primary key. Append-only tables are simply concatenated. 5. Refuse only in the five cases listed in section 3.2. The key point is in step 3: **the ring and the change log are two sources for one merge**, not two read paths. The merge neither knows nor cares which one fed it, so time travel and read-your-writes have identical semantics and identical code. **Two behaviours that look like bugs but are not.** - A lake commit can move a table's `W` past a pin taken moments earlier. For an implicit "latest" pin, the read **re-pins at the fresh head and rebuilds**; it does not fail. An explicit `--at-lsn` below `W` is served by time travel. - A time-travel pin includes the tail's **current** pending value notes (section 2.8) in a historical read. A note about a value that landed *after* the pinned position may therefore be shown on an answer that does not contain that value. This over-disclosure is deliberate. The failure it prevents is an affected value served with **no** note; a note with no value is the safe direction. #### 3.4 The gap record and the rescan ![3d — the gap record and the rescan](diagrams/argondb-and-the-erp/03d-gap-and-rescan.svg) Sometimes the tail cannot decode a stretch of history — for example, the WAL it needed was removed. It then rebuilds the affected tables from their current state (a re-baseline) and writes a gap record, so that nothing downstream ever claims to know what happened inside that stretch. Four situations lead to a re-baseline: 1. The resume point is unservable at boot (the WAL hold went over its cap). 2. At startup, a table is a **laggard**: further behind the head than `ARGONDB_MAX_LAG_BYTES` (**256 MiB** by default). It re-baselines alone. 3. An undecoded-window **barrier** episode. 4. The watchdog escalates a stall on the "not advancing" verdict. **A quarantined table is not a gap**, and the two are easy to confuse. A table whose **decode** fails is **quarantined on its own** at its last good watermark. An alert is raised with the offending row's evidence and the named remedy. The cadence, the catch-up rescan and reads of every other table carry on unaffected; this containment is a hard rule. Nothing was lost — the table simply stopped advancing. `argondb lake list` shows it as `QUARANTINED @ (frozen; cause=: )`. **The ordering is the guarantee.** The gap record must land *before* any of the rescan's table commits. A crash between the two can then only leave a conservative **false positive**: a gap recorded over a window that turns out to be fine, which replay from the watermark heals. It can never leave an advanced watermark with an unrecorded gap. The reverse order could produce a lake that is silently missing history and confident about it; this ordering makes that state unreachable. It is required at every stage of the planned design for sharding decode across processes, not only in today's single pipeline. **What a reader sees.** Neither consequence is silent: - A `changes()` digest whose window **overlaps** a gap is still served, but **annotated**: its counts are declared lower bounds over the overlapping range(s). - A **time-travel read inside** a gap is refused permanently, naming the range and the reason. There is also a defensive guard in the replay path. A re-baseline resets snapshot history, so no base snapshot below a gap should survive. The replay path still checks for gaps, so that any future violation of that rule becomes a refusal instead of wrong rows. #### 3.5 The memory budget, and where it turns into lag ![3c — the tail's memory budget as a stack](diagrams/argondb-and-the-erp/03c-tail-memory-budget.svg) The tail has a hard memory limit, so that it can share a box with your database without threatening it. The limit is built up from named buffers, each with its own setting, plus slack for the allocator. ``` baseline 1,024 MiB tokio, the Flight server, the schema, the pagestream connections ring budget 2,048 MiB ARGONDB_TAIL_DRAIN_QUEUE_MB open-txn budget 512 MiB ARGONDB_MAX_OPEN_TXN_BYTES scan-buffer budget 512 MiB ARGONDB_MAX_SCAN_BUFFER_BYTES TOAST-index budget 256 MiB old-tuple cache 128 MiB ARGONDB_TAIL_OLD_TUPLE_CACHE_MB writer staging 1,024 MiB one table's ceiling ------------------------------------------------------------------ 5,504 MiB + 25 % of the 4,480 MiB of BUDGETS (not the baseline) 1,120 MiB ------------------------------------------------------------------ stated bound 6,624 MiB -> memory.max 6,144 MiB, memory.high 5,632 MiB ``` The tail runs in a cgroup with `memory.max` = 6,144 MiB (the hard cap) and `memory.high` = 5,632 MiB (the alarm line). Several facts qualify this table. **There is a floor, and the box does not refuse below it on its own.** The tail's floor is **4,704 MiB**. If a box's own memory budget would give the tail a smaller share, the share is raised **up** to the floor at boot, and the boot line names the share, the budget and the floor. The box refuses to start only when an operator explicitly sets `ARGONDB_TAIL_MEMORY_MB` below the floor. (A stricter reading — "refuse whenever the share is below the floor" — would stop every budgeted box in the estate from booting today.) **Arrow Flight scans sit outside this cap.** `ARGONDB_TAIL_FLIGHT_SCAN_MB` (default **8,192 MiB per process**) bounds Arrow Flight scan reads with its own process-wide budget. The cgroup cap above does not cover them, and both the boot line and the v1.5.4 release notes' limits line say so. Raising the 6,144 MiB cap to cover reads as well (to roughly 16.5 GiB) was considered and rejected: it would undo the published claim that the tail's previous "≥ 24 GiB" footprint is replaced by 6 GiB. Of the three options considered, this was the only one that changed none of the published limits. **Measured: the baseline.** On 2026-09-19, on a test machine, idle: box booted, data seeded, tables published and drained, then one 120 s cadence at **zero** simulated users ("clerks" — the unit of load in ArgonDB's ERP benchmark). The tail's `VmHWM` was **172.5 MiB** and `VmRSS` **72.4 MiB**, with a drain queue of 0 bytes, 0 ms of backpressure and 0 bytes of ingest lag. That is 17 % of the 1,024 MiB baseline constant. The bound is **not** restated on this reading, for three reasons: - recomputing would move 6,624 → 5,772 MiB and with it the published cap, and one reading is not enough to do that; - the reading is at `ERP_SCALE=1` with ten published tables, and the baseline grows with the tenant, so one point is not the curve; - a real baseline *below* the constant makes the bound conservative, which is the safe direction. **Measured: without the budget.** The same 10-clerk load with the old-tuple cache on and **no** ring budget reached 19,007 MiB sampled (`VmHWM` 19,013 MiB). That is what the budget removes. ArgonDB's tests refer to this reading rather than spending 19 GiB of a shared machine every night to prove it again. **Not built: the alarm log line.** The design asked for a log line `tail memory: over the alarm line`. The release that added the cap and the alarm (`memory.high`) shipped without it, and the v1.5.4 release notes say so. Do not search your logs for that line. **Memory, lag and the cap are one system.** ``` memory held by the tail ≤ the bound 6,144 MiB cap WAL the tail has not read ≤ the hold cap 4 GiB replay after any restart ≤ the same cap the replay budget is the LARGER of the hold cap and ARGONDB_MAX_LAG_BYTES ``` Backpressure moves bytes from the first line to the second. The second line is where a tenant that writes faster than its tail can decode eventually forfeits history. At 10 clerks the ERP workload writes about 2.2 KiB of WAL per transaction, so 4 GiB ≈ **1.9 million transactions**. At about 560 txn/s under backpressure, a tail whose offered rate exceeded its decode rate by 100 txn/s would take **about 5 hours** to reach the cap. **At 50 clerks the picture is different.** The tail decodes **659.6 txn/s** against **2,900.4 tps** offered and never catches up (904 s, residual 497 MB). Doubling the cap doubles the runway — 13 minutes becomes 26 — and changes nothing else, because at that load the deficit is structural. **The cap is not what forfeits the tenant; the decode rate is.** **With the multiplexed tail (section 2.12), the lag budget is per stream, not shared.** Each tenant timeline's stream owns its own `ARGONDB_MAX_LAG_BYTES` (256 MiB); it is never one 256 MiB budget divided across N companies. This is simple, and it means the Nth company cannot be starved by the first. The cost is that the estate needs N × 256 MiB of headroom in the worst case. You can watch this per stream. Three gauges carry one series per stream timeline: `argondb_tail_stream_clean_lsn_lag_bytes`, `argondb_tail_stream_replay_budget_bytes` and `argondb_tail_stream_over_replay_budget`. A stream past its budget logs `REPLAY BUDGET EXCEEDED on timeline ` and falls back to a fresh baseline **alone**; the other streams resume incrementally. In practice the headroom needed is larger than 256 MiB per stream: the replay budget actually applied is the **larger** of `ARGONDB_MAX_LAG_BYTES` and the WAL hold cap, and the hold is on by default at 4 GiB, so the budget in force per stream is 4 GiB. The estate's real worst case is therefore N × 4 GiB of headroom. Under a three-member quorum that hold is kept on all three safekeepers, tripling a lagging stream's storage cost compared with a single box. #### 3.6 Restart The tail is designed to be killed at any moment. It has no clean-shutdown path at all; every exit is treated as a crash, and every restart resumes from the last durable cut. This section explains why, and what a restart costs. `argondb-tail` installs no signal handler. A unit test asserts that the tail's main source file never mentions `SIGTERM`, `ctrl_c`, `signal_hook` or `SignalKind`. The restart point is made durable **at every cut**, never at exit. A "clean position written at shutdown" is produced only by the polite exit path, which is exactly the path missing when it is most needed. A drain-on-SIGTERM was considered and rejected: it would reopen the class of bug where a crash lands between the write and the exit. The sequence: 1. The process dies where it stands. The ring, the open-transaction buffers and the cache are gone. The resume point is the last cut's. 2. The supervisor decides: restart with a backoff from 2 s up to 30 s, or **trip after 6 exits in 120 s**. A trip marks the box DEGRADED, naming the tail, and raises an alarm; the tenant's database keeps serving. Under a cgroup cap, the exit is the kernel's kill *inside* the cap, and the DEGRADED text says so. 3. The supervisor rebuilds the same command and environment. The log is **appended**, so the panic that killed the previous run is still there. 4. On boot the tail reads the per-table resume points. Its replay budget is `max(hold cap, ARGONDB_MAX_LAG_BYTES)`. **Before** choosing, it probes the safekeeper (connecting as `argondb-resume-probe`) to prove the resume point is still servable, and the boot line names the outcome. Servable: decode from the resume point, with no re-baseline and no gap. Unservable: a fresh baseline with a named gap. 5. Replay costs at most the cap's worth of WAL at the decode rate — up to about 1.9 million transactions, about 45 minutes at about 700 txn/s. With the ring budget in place, a crash mid-window re-decodes at most about 4 s of peak decode, plus whatever lag the tail was already carrying. 6. The old-tuple cache starts cold, so its hit rate reads low for the first few minutes. Nothing else differs. **Upgrading the tail binary is the same sequence with step 1 done on purpose.** Stop the tail process (SIGTERM to its process group), replace the binary the box uses (`argondb-tail` on the path, or the file named by `ARGONDB_TAIL_BIN`), and let the supervisor start the new one from the same resume point. No state passes between the two versions except through the lake and the WAL hold, so there is nothing for a new version to migrate. **`argondb stop`** sends SIGTERM to every child's process group, with a 10 s budget for the set and a 10 s grace per group, then SIGKILL — a teardown bounded at 35 s. The tail gets the same SIGTERM as everything else and dies. At the next `argondb serve`, the hold — loaded from `argon_tail_hold.json` before the safekeeper's first removal pass — has kept the WAL back to the resume point. **On the multiplexed estate tail (section 2.12)**, restart from the watermark is still the baseline. A failure is a restart, and lake freshness pauses for every stream until the tail is back. Nothing is lost, because none of the above depends on there being only one tenant. The warm standby — a second tail in another availability zone that takes over on a fenced lease before a restart would otherwise be needed — is not built yet. **The lake writer has the same restart property.** A continuous-view refresh killed right after its Iceberg commit returns, before anything else is recorded, restarts cleanly: the restarted run reads the committed position, so the view comes back whole. Replaying the dead refresh's own token afterward answers "already landed" and writes nothing twice. This was measured with a real process kill, not a simulation, and the rows matched an oracle built from the changes exactly. **Candor:** the measurement's catalog lives in memory and is reopened from the newest committed metadata file, which stands in for a REST catalog's durable pointer — it is not the same thing as a running Lakekeeper. #### 3.7 The maintenance plane An Iceberg table that only ever receives small commits slowly gets slower to read: too many small files, too many snapshots, stale statistics. The maintenance plane is the set of background jobs that prevents this, and ArgonDB runs it for you. It does five jobs: compaction (Iceberg `RewriteFiles`), clustering by the `argon:cluster-by` marker, exact distinct-value (NDV) statistics in Puffin files, snapshot expiry under explicit retention, and orphan-file cleanup. It runs **whole** in every deployment on a fixed **120-second** cadence. That is a code constant (`MAINTENANCE_SWEEP_SECS = 120`) with no switch, no cadence setting and no off state; tests reach it only through a test-only override. Safety comes from each operation's own guards, never from being able to turn it off. **Why ArgonDB owns it, not you.** ArgonDB is the single writer and already knows each table's state from the WAL. An external maintenance job would be a second writer racing the committer, and it would hand you back the operational pipeline this product exists to remove. **Clustering defaults to the primary key.** When a published table has exactly one primary-key column and no explicit marker, it is clustered by that column. The common lake lookup is a primary-key equality, and before this default the clustering mechanism sat unused while a benchmark spent 15.90 s scanning for one row. Precedence: - an explicit `argon:cluster-by=` always wins; - `argon:cluster-by=none` opts out; - tables with a composite primary key, or none, get **no** default. A leading-column default for composite keys is deliberately not implemented, because it would optimise a prefix-equality lookup the read surface never issues. A single `uuid` key — the UUID v7 every ERP table uses — is measured, not assumed, to take this default: it lands as Iceberg's native uuid type and sorts through the same clustering path as any other single key. A unit test in the committer and a box test (`testing/box/uuid-v7-cluster-default.sh`) prove it: a declared sort order and monotone row groups for the single-key table, and no default for its composite-key twin. Because clustering is now a cost every such table pays, the scan-replace path sorts efficiently. For any table that fits in `ARGONDB_MAX_SCAN_BUFFER_BYTES`, it writes rows in key order by feeding slices to the Parquet writer rather than building a sorted copy. Above that budget, it writes budget-sized windows, each sorted by the cluster key, and **declares no sort order at all**: a global Iceberg `SortOrder` over files that are only sorted window by window would be false metadata a foreign reader might act on. **A pass that overruns the cadence backs off rather than overlapping.** On the live demo at a 120 s cadence, measured pass durations did exceed the cadence. A single pass has a deadline of 30 cadences, which is one hour. #### 3.8 Branches A branch of your database gets its own lake namespace. This section explains what that costs and why the lake does not share files between branches. A branch maps to a distinct Iceberg namespace, materialized **lazily**: the lake does no work for a branch until something reads or publishes on it. An unread branch costs the lake nothing, while the Postgres side stays Neon copy-on-write. Once materialized, a branch's lake tables share nothing at the file level. The storage cost of a materialized branch is a full copy of what it materializes. **Why no file sharing.** Copy-on-write file sharing in the lake would cut that storage cost, but it would couple every branch's maintenance. Compaction and orphan cleanup could no longer prove a file dead by looking at one table's retained snapshots. Every operation would need reference counting across branches, and a bug in that coordination would corrupt a branch you were not even touching. **The `br_` prefix is conditional, and this is the most common wrong assumption about lake addresses.** Only `main`, `temp` and `system` take it — they are the three names DuckDB reserves as schema names. Every other branch **is** its namespace, bare. A `br_$BRANCH` composed in a shell script is therefore right for three branch names and silently wrong for all others. The `dv_` sibling is unconditional. Ask `argondb lake namespace` instead of composing the name (example 8). #### 3.9 What is measured, and the numbers This section collects the performance readings in one place, so you can judge whether the tail keeps up with your write rate. "Clerks" are simulated users in ArgonDB's ERP benchmark. | reading | number | |---|---| | decode rate, 10 clerks, old-tuple cache on | **666.65 txn/s** (3.3× the 201 txn/s before the cache) | | old-row reads served from memory | **84.3 %** hit rate (87.7 % at its peak) | | drain time at that load | **578 s**, lag at end 0.0 MB, jumps 0.0 | | tail memory at idle (0 clerks, one cadence) | `VmHWM` **172.5 MiB**, `VmRSS` **72.4 MiB** | | ring counted vs resident | 2,064.4 MiB counted vs 3,217.2 MiB `VmHWM` = **1.56×** | | ring cost per row, ERP profile | **364.2 B/row** one batch per commit, **76.7 B/row** coalesced, **78.9 %** overhead | | no ring budget, same load | **19,007 MiB** sampled, `VmHWM` 19,013 MiB | | 50 clerks | decode **659.6 txn/s** against **2,900.4 tps** offered; never drains (904 s, residual 497 MB) | | WAL per transaction, ERP | **about 2.2 KiB** → the 4 GiB cap ≈ 1.9 million transactions | | decode ceiling before the cache, profiled | fetching old tuples from the pageserver (`oldpage_fetch`) is **96.5 %** of decode time and by itself limits decode to **198.0 txn/s**; the other five stages together would allow between 5,888 and 4.4 million txn/s | **Where the ceiling comes from.** The 198 txn/s ceiling and the 201 txn/s "before the cache" figure are the same fact. Fetching old tuples from the pageserver is nearly the whole cost, which is why the old-tuple cache exists; serving old-row reads from memory is what lifts 198–201 txn/s to 666.65. No change inside the tail itself moves that ceiling: every such change acts on the 3.5 % of time that is not the old-page fetch. Spreading the fetch across more pageserver connections was measured and does not help (8 connections: 113–190 txn/s against 2,114 offered, no better than one), because the pageserver's per-tenant WAL-redo process is the serial resource. Three levers have shipped against that ceiling, in the order they landed. - **The old-page read is decoupled from decode.** Changes that need no old page are encoded while the old-page fetch is in flight; changes that need one wait behind it, and commit order is preserved. It is on by default (`ARGONDB_TAIL_DECODE_DECOUPLED` is the off switch). Measured gain: only +2.0 to +2.7 %, because the fetch itself is still the bound. - **The old row can come from the WAL instead of the pageserver**, per table, if you opt in with `argondb lake publish
--old-row-from-wal`. That sets the table's `REPLICA IDENTITY FULL`, so every UPDATE and DELETE record carries the whole old row, at a WAL growth cost of **×1.047** at the ERP mix (about 31.6–31.8 bytes per UPDATE). It is **off by default**, refused by name when `wal_level` is below `logical` (Postgres would log no old row, so the opt-in would change nothing), and refused when the table's replica identity is already `NOTHING` or `USING INDEX` (someone chose that on purpose; set it to `FULL` yourself if you want this instead). A refusal publishes nothing. - **The zero-WAL lever, now the default, and usually the better answer.** Under the default replica identity, on a table with a primary key, Postgres only logs an UPDATE's old key when the key actually changed. So a record with no old key already *proves* the key did not change, and the tail can apply it without ever reading the old page. It changes no table's identity and adds no WAL byte at all. `ARGONDB_TAIL_ZERO_WAL` defaults **on**; setting it `off` restores the plain old-page read. Measured on one binary, all three arms side by side: with neither lever, decode held at **748.0–754.8 txn/s**; the old-row-from-WAL lever reached **9,695.8–10,067.7 txn/s**; the zero-WAL lever reached **10,009.7–10,388.9 txn/s**, about 3–6 % higher than the old-row lever, with zero old-page reads and no added WAL at all. (Read these as decode ceilings, not end-to-end throughput — they measure transactions decoded per second of decode time, not what a tenant sustains through the whole path.) The zero-WAL lever falls back to the ordinary old-page read, which is always correct, whenever it cannot prove the key is unchanged: no primary key; identity `NOTHING`, `FULL` or `USING INDEX`; the lever switched off; a catalog change that might affect identity not yet re-read; or a record older than the identity the tail has cached. A `DELETE` never uses this lever — under the default identity its record already carries the key, so no page was read for it before this lever either. **The old-row-from-WAL lever stays the answer for a table the zero-WAL lever cannot help**, most importantly one with no primary key. **The "56-byte drain residual".** The diagram's bottom caption reads "338 MiB drained in 578 s, zero gaps" and mentions a "56-byte drain residual". That residual is `argondb_tail_ingest_lag_bytes + argondb_tail_commit_lag_bytes`, and it is **not** a floor. A dedicated probe at 10 clerks measured `level c10 drain: drained after 31s (residual 0)`, with trajectory `shape=REACHED-ZERO`, so ArgonDB's tests keep "residual 0" as the meaning of *drained*. The 56 bytes were one standby-snapshot WAL record caught in flight — a race, not a constant. Accepting a nonzero floor in the tests would have quietly redefined "drained", which the design forbids. ### 4. How to use it This section is a set of worked examples, in the order you would meet them: publish a table, find it, read it from Postgres, from DuckDB and at a pinned position, then time travel, the change log, tuning and repair. About the examples: the outputs shown for `argondb lake address`, `argondb lake namespace` and the `--help` texts were captured from a real run of a 2026-09-18 build of `argondb`. Those commands open no Postgres connection, reach no catalog and need no running box. The other examples show exact command syntax and exact message text taken from the software; values in them such as table names and log positions are illustrative, and they were not run for this guide. Some `--help` texts are trimmed with `…`. #### Example 1 — publish a table, and the marker that does it Publishing is how you choose which tables go to the lake. Nothing is published by default, and new tables are not added automatically. Publishing is one command, with no connector configuration: ```console $ argondb lake publish public.orders published public.orders — the full table backfills at the tail's next cadence, then follows incrementally ``` What the command actually *does* is set a Postgres **table comment** containing the marker `argon:publish`: ```sql -- what `argondb lake publish` sets: COMMENT ON TABLE public.orders IS 'argon:publish'; -- an existing comment is preserved and the marker appended: COMMENT ON TABLE public.orders IS ' argon:publish'; ``` Keeping the marker in a comment, rather than in a config file, has three consequences that matter in practice. 1. **The published set lives in the database's own catalog, and branches with it.** Branch the database, and the branch starts with the same published set. 2. **Re-publishing is harmless and says so:** `public.orders is already published`. 3. **Any client can set that comment without the CLI**, so the marker can land on things that cannot be published. That is *accepted and announced*, never silently ignored and never refused at `COMMENT` time. Refusing would need an event trigger, and therefore superuser on the primary, which you do not have on a bring-your-own-Postgres deployment. See section 6.4. If the table has columns no Iceberg type can carry yet, the marker is still set and kept. The table backfills automatically once its columns become lake-eligible, with no re-publish, and the command says so explicitly instead of implying a backfill is imminent. #### Example 2 — see it appear, from both directions There are two listings because there are two questions: "which of my Postgres tables are in the lake?" and "what does the lake actually hold?". From **Postgres** — which published relations have a lake table, and at what watermark. Three tab-separated cells: name, state, lake key. ```console $ argondb lake list --tail http://localhost:8817 public.orders lake @ 0/1BD2748 postgres__public__orders public.line_items not mirrored postgres__public__line_items public.legacy_audit QUARANTINED @ 0/1A9F210 (frozen; cause=decode: numeric out of range) postgres__public__legacy_audit ``` Pass `--tail` (the tail's Flight address; `http://localhost:8817` on a box). Without it the change-log configuration is **not** checked, and the command says on stderr that it did not look, rather than letting you read silence as a clean bill of health. From the **catalog** — what the lake actually holds, including tables no Postgres relation stands behind: ```console $ argondb lake catalog-tables --branch main postgres__public__orders br_main.postgres.public.orders nested _argon_gaps br_main._argon_gaps branch-system ``` Three cells: the flat lake key, the catalog address, and the **shape** — `nested`, `old-shape`, `branch-system` or `unclassified`. Do not ignore the shape. An old-shape table's flat key is its bare leaf name, which cannot match what the Postgres side is told, so a script that ignores the third cell will read a live lake table as missing. A catalog that cannot be reached is an **error**, never an empty listing. A branch whose namespace does not exist yet **is** an empty listing. **Where a company's status rows live.** Every database carries its own publish-status and gaps tables, written by the box on a cadence. A company that has its own compute has them written on that compute, and only while it is awake: the pass never wakes a sleeping company just to refresh a status row, so a stood-down company's rows are brought current by the first pass after its next wake. The `as_of` column says how old they are. The `publish-status` command reads from the right place and says which one it read: the company's own compute when it has one, the primary when it does not, and a plain statement rather than an error when the writer has not reached a database yet. #### Example 3 — `argondb lake address`: the spellings of one table A relation has several spellings depending on who is asking, and since the three-level address (branch, database, schema) was introduced they are no longer the same string. Ask for them; do not assemble them yourself. ```console $ argondb lake address postgres.public.orders lake."br_main.postgres.public".orders $ argondb lake address --field namespace postgres.public.orders br_main.postgres.public $ argondb lake address --field lake-table postgres.public.orders postgres__public__orders $ argondb lake address --field qualified postgres.public.orders "public"."orders" $ argondb lake address --field label postgres.public.orders br_main.postgres.public.orders $ argondb lake address --field leaf postgres.public.orders orders ``` Which one you want depends on where you will use it: - **`address`** — the catalog name a DuckDB, Spark or Trino user types. It is **quoted**, and has to be: DuckDB reads a nested namespace only when the whole dotted path is one quoted identifier. - **`lake-table`** — the flat key. Use it wherever only two name parts exist: the tail's per-table settings and its Flight `tables` action, the FDW's foreign tables, the relations registered in a pin session, and the `--scope` argument. - **`label`** — how the maintenance plane names the table in its logs: namespace levels joined by `.`, with no catalog alias and no quotes. It exists because the quoted `address` is useless for matching log lines. - **`qualified`** — how the tail names the table in its logs, for example in ` left the published set`. **The change-log table** is a flag, not another `--field`, because the change log is a different relation with its own set of all these names: ```console $ argondb lake address --changes postgres.public.orders lake."br_main.postgres.public".orders__changes $ argondb lake address --changes --field lake-table postgres.public.orders postgres__public__orders__changes ``` Note what changes and what does not. The lake-side names become the change log's. `namespace` does **not** change (a change log always sits in its base table's namespace), and neither does the Postgres-side `target` (a change log has no Postgres table behind it). This keeps a `lake list | … | lake address` round trip exact. A table in a second database, and the JSON form a script should consume: ```console $ argondb lake address shop.sales.orders lake."br_main.shop.sales".orders $ argondb lake address --field lake-table shop.sales.orders shop__sales__orders $ argondb lake address --format json postgres.public.orders { "box_warehouse": "argon", "branch": "main", "catalog": "lake", "changelog": false, "primary": "postgres", "tables": [ { "address": "lake.\"br_main.postgres.public\".orders", "bucket": "argon", "changelog": false, "database": "postgres", "label": "br_main.postgres.public.orders", "lake_table": "postgres__public__orders", "leaf": "orders", "namespace": "br_main.postgres.public", "qualified": "\"public\".\"orders\"", "schema": "public", "table": "orders", "target": "public.orders", "warehouse": "argon", "warehouse_refusal": null } ], "warehouse_per_database": true } ``` `"changelog": true|false` is included so a script never has to guess which relation it was given. #### Example 4 — unpublish, and the two levels Unpublishing stops a table flowing to the lake. There are two levels, one reversible and one not. ```console $ argondb lake unpublish --help Stop publishing a table. Lake files and history are KEPT (re-publish re-baselines). With --purge, also drops the Iceberg table and its changelog and deletes their files — irreversible Usage: argondb lake unpublish [OPTIONS]
Arguments:
Options: --pg-url Omitted = this box's primary (see `lake publish --pg-url`) --purge Also delete the lake table, changelog, and all their files --yes Skip the interactive confirmation (scripts) --branch Branch whose lake table to purge [default: main] --data-dir This box's data directory (… only consulted with --purge, which is the arm that reaches the catalog). Same default as `argondb serve`/`status` [env: ARGONDB_DATA_DIR=] [default: ~/argon] --port This box's Postgres port, for the SAME port-precedence `argondb status` uses (`--port` / `ARGONDB_PG_PORT` / argon.toml `port`; presence-based …) -h, --help Print help ``` The two levels mirror `DROP TABLE` versus `DROP TABLE PURGE` exactly: - **Plain unpublish** stops capture and keeps every lake file and all history. It is instant and reversible; re-publishing re-baselines the table. - **`--purge`** also drops the Iceberg table and its change log and deletes their files, after an explicit confirmation showing what will be destroyed. It cannot be undone. `DROP TABLE` in Postgres behaves like the **plain** unpublish: the lake data is kept. Two kinds of table can never be published: TEMP and UNLOGGED. They write no WAL, so there is nothing to capture. #### Example 5 — read the lake copy from Postgres, through `lake_.*` If your application or your analysts already speak SQL to Postgres, they can read the lake from there. The `argondb_lake` foreign-data-wrapper extension mounts every table *this database* published as a foreign table in the `lake_` schema: ```sql CREATE EXTENSION argondb_lake; -- needs superuser: it creates a FOREIGN DATA WRAPPER CREATE SERVER argondb_lake_srv FOREIGN DATA WRAPPER argondb_lake OPTIONS (tail 'http://localhost:8817', branch 'main'); SELECT argondb_lake_sync(); -- one idempotent pass: CREATE FOREIGN TABLE IF NOT EXISTS SELECT argondb_lake_start_sync(); -- a background worker on the cadence, so later publishes -- mount without a reconnect SELECT count(*) FROM lake_main.postgres__public__orders; SELECT argondb_lake_freshness(); -- the "how far behind" note, as queryable JSON ``` Three properties of this mount are worth knowing. - **It always reads through the query plane.** A foreign table here is not a direct Iceberg scan. It is the unified read, so it sees `Iceberg@W ∪ tail`, not only the base table. - **It is scoped to its own database.** Before 2026-09-11 the mount asked the tail for all tracked tables, a cluster-wide set. Under the database-per-tenant topology that meant tenant A's schema held tenant B's tables as ordinary foreign tables — readable by anyone with `USAGE` on the schema, with no policy, grant or wall checked anywhere on the path. Both halves are now scoped by one check at the tail: the **list** is scoped, and **every read request carries the asking database**, so a foreign table mounted some other way is refused *by the read plane* with a named message instead of being served. - **Tables another company granted you are mounted too.** A table another tenant has explicitly granted to this database appears in the mount, read-only; ungranted tables are refused by name at read time. After a revoke takes effect within one schema-refresh cycle of the read plane (measured at 28–29 seconds); the next read is refused with the revoke's instant, and the mount itself stays in place. - **The mount cannot read `timestamptz` columns yet.** A query that streams one fails; select the other columns. - **A mount outside this database's own list is warned about**, once per sync pass, but sync's ordinary job is `CREATE FOREIGN TABLE IF NOT EXISTS`, so nothing else touches it. Such a mount can be old, hand-made, a revoked grant, or one whose source table is gone. Only that last kind — a **stale** cross-database mount — can be dropped, and only if you turn it on: `SET argondb_lake.drop_stale_mounts = on;` (superuser-only, because it is destructive) before the next `argondb_lake_sync()`. It stays off by default. Each drop is its own warning naming the mount, the lake table and its source database; a mount a view depends on, or that its owner does not own, is named and left in place instead. **A revoked grant's mount is never dropped** — its source is still alive, so it keeps its plain warning instead. #### Example 6 — read the same table from DuckDB, with nothing of ours running The lake is standard Iceberg, so you are never locked in. Any Iceberg-capable engine can read these tables with no ArgonDB component running. ArgonDB's metadata rides in `Argon.*` snapshot properties, which never break a foreign reader. ```sql -- in DuckDB, after ATTACHing the Iceberg REST catalog as `lake` SELECT count(*) FROM lake."br_main.postgres.public".orders; SELECT count(*) FROM lake."br_main.shop.sales".orders; -- database `shop`, schema `sales` ``` **The quotes are required**, and the error you get without them does not look like a quoting mistake: ```console D SELECT count(*) FROM lake.br_main.postgres.public.orders; Parser Error: NameListToString NOT IMPLEMENTED ``` **Discovery does not work, and this has been measured.** DuckDB lists only the catalog's top level and never walks into a nested namespace. A table you have not named is invisible to `SHOW ALL TABLES`, `duckdb_schemas()` and `duckdb_tables()`: ```console D SHOW ALL TABLES; ┌──────────┬─────────┬───────┬──────────────┬──────────────┬───────────┐ │ database │ schema │ name │ column_names │ column_types │ temporary │ ├──────────┴─────────┴───────┴──────────────┴──────────────┴───────────┤ │ 0 rows │ └───────────────────────────────────────────────────────────────────────┘ ``` Name the namespace once, and it becomes visible for the rest of the session: ```sql USE lake."br_main.postgres.public"; SHOW ALL TABLES; ``` To learn the namespaces without guessing, walk the REST catalog yourself, one level per call — which is exactly what DuckDB does not do for you: ```bash curl -s "$CATALOG/v1/$WH/namespaces" | jq -r '.namespaces[][]' curl -s "$CATALOG/v1/$WH/namespaces?parent=br_main" | jq -r '.namespaces[][-1]' curl -s "$CATALOG/v1/$WH/namespaces?parent=br_main%1Fpostgres" | jq -r '.namespaces[][-1]' curl -s "$CATALOG/v1/$WH/namespaces/br_main%1Fpostgres%1Fpublic/tables" \ | jq -r '.identifiers[].name' ``` `%1F` is the level separator the REST spec uses. `$WH` is the warehouse's **UUID**, not its name; given the name, the catalog answers `400 WarehouseIdIsNotUUID`. Read the UUID from `GET /catalog/v1/config?warehouse=argon`, field `defaults.prefix`. **A foreign engine sees the base table at its watermark.** No ring, no pin, no merge. That is not accidental staleness; it is a different, simpler read than the unified one, and it is why the Postgres mount in example 5 exists. **A `dv_` table — a continuous view, or a transform's output — reads the same way, but tells you which position it is at differently.** It is plain Iceberg too, so DuckDB reads it with nothing of ours running. But its snapshots carry a property called `Argon.view-current-to-lsn`, the position the view is current to, never the `Argon.watermark-lsn` a published table carries. The lake writer refuses to write a view snapshot that carries `Argon.watermark-lsn` at all, because a view did not come from the WAL, and a reader that mistook its position for a watermark would pin it wrong. So a foreign reader can tell a published table and a view apart by which property name the current snapshot carries. Two DuckDB v1.5.4 quirks matter if you read those properties yourself: `iceberg_snapshots()` returns no summary column, so read the summary from the table's `metadata.json` instead; and that same function refuses a multi-level namespace written as an identifier, so pass the address as a quoted string. #### Example 7 — a pinned read, and what the pin token carries `argondb-query` is the bundled reader for the unified read. Its flags are the three coordinates of a read: branch, position and scope. ```console $ argondb-query --help Usage: argondb-query [OPTIONS] Arguments: SQL to execute against the unified views Options: --tail The tail's Flight address. Read from `ARGONDB_TAIL_ADDR` when the flag is absent … [env: ARGONDB_TAIL_ADDR=] [default: http://localhost:8817] --branch [default: main] --at-lsn Pin LSN (X/Y). Default: current tail head (freshest consistent view) --scope Pin scope …: comma-separated lake table names the pin covers. Default: all tables (the atomic cut) --csv Print the result as CSV (header row first) … -h, --help Print help -V, --version Print version ``` ```bash # freshest consistent view, scope = all tables (the atomic cut) argondb-query "SELECT count(*) FROM postgres__public__orders" # the same question at a named position argondb-query --at-lsn 0/1BD2748 "SELECT count(*) FROM postgres__public__orders" # narrowed scope — the consistency claim now covers exactly these two tables argondb-query --scope postgres__public__orders,postgres__public__line_items \ "SELECT o.id, sum(l.qty) FROM postgres__public__orders o JOIN postgres__public__line_items l ON l.order_id = o.id GROUP BY 1" # machine-readable, for another program argondb-query --csv "SELECT * FROM postgres__public__orders LIMIT 10" ``` `--scope` takes **flat lake keys**; `argondb lake address --field lake-table` produces them. Narrowing the scope is also the *remedy* every scope-related refusal suggests, so it is worth knowing before you need it. The token itself is `(branch, LSN, scope, minted_at, coordinate-provenance)`, signed and stateless. The field that does the work is the last one: **whether the position was chosen by the server at the protected default, or named by the caller.** A server-chosen position is never refused. When no position serves every scoped table, the pin is granted anyway, and the tables that cannot serve it are served at their own newest retained watermark, with a per-table disclosure. A caller-named position can be refused, because the caller named it and owns its lifetime. The disclosure a non-atomic pin prints, exactly as the software prints it ("D9 changelog" in this message is the software's internal name for the `__changes` change log): ```text note: this pin is NOT one atomic cut — no coordinate serves every scoped table, so 2 scoped table(s) are reflected as of an OLDER coordinate than 0/1BD2748: postgres__public__audit, postgres__public__legacy. Rows committed to those tables after their own coordinate are NOT visible here; every other scoped table is at 0/1BD2748. A table is listed here for one of two reasons: it has no D9 changelog, so it can only be read exactly at a retained snapshot watermark and it retains none at 0/1BD2748; or its lake watermark has run ahead of the live decoder, so its changelog holds nothing between its own coordinate and this one and it is served base-only. Enable the D9 changelog on the first kind for row-exact history at any coordinate, narrow the scope to tables that share a coordinate, or repin — the second kind clears itself as soon as the decoder catches up. ``` #### Example 8 — branches, and the `br_` prefix that is not always there Branch names are the one piece of lake addressing that shell scripts get wrong, so there is a command that needs no running box to answer it: ```console $ argondb lake namespace br_main $ argondb lake namespace postgres public br_main.postgres.public $ argondb lake namespace --branch feature-x feature-x $ argondb lake namespace --territory lake-writer dv_main $ argondb lake address --branch feature-x postgres.public.orders lake."feature-x.postgres.public".orders ``` Look at the third result again: **`feature-x` has no `br_` prefix.** Only `main`, `temp` and `system` take it — the three names DuckDB reserves as schema names. So a `br_$BRANCH` composed in a shell is right for three branch names and silently wrong for all others. The `dv_` lake-writer namespace is prefixed **unconditionally**, which a hard-coded string cannot express either. Give levels below the branch to compose deeper names; the result is the same string `argondb lake address --field namespace` gives for a relation in that schema. Use `lake namespace` when you have no relation to name, and `lake address` when you do. In operation: branching the database branches the published set with it; the lake does no work for the branch until something reads or publishes on it; and once materialized, the branch's lake tables share no files with any other branch. #### Example 9 — time travel, and what a gap refusal prints Time travel is a read at a position below the current watermark. It is served from the newest retained snapshot with `W' ≤ L`, plus change-log replay over `(W', L]`: ```bash argondb-query --at-lsn 0/1A00000 "SELECT count(*) FROM postgres__public__orders" ``` It can be refused in four ways. Each message is shown exactly. **Inside a recorded gap** — permanent, because the rows were never decoded: ```text time travel of postgres__public__orders to 0/1A00000 refused: the changelog replay window (0/19C0000 … 0/1A00000] intersects a recorded gap (0/19F1200 … 0/1A0C440] (watchdog stall-escalation re-baseline) — the range is unreconstructable; pin at or after 0/1A0C440, or repin for a fresh view ``` **No change log, and `L` is not one of the table's retained watermarks:** ```text postgres__public__orders has no changelog, so time travel can only land exactly on a retained snapshot watermark (nearest at or below 0/1A00000 is 0/19C0000); enable the changelog for row-exact history ``` You can reach this one **only** through `freshness=head` or an explicit `at_lsn`. The default, `latest_servable`, snaps down to a durably servable position when the pin is minted, precisely so the ordinary path never lands here. **Below a history restart** — the one refusal whose remedy points *forward*: ```text time travel of postgres__public__vio_dim to 1/D54A4238 refused: a whole-table materialize restarted this table's history at 1/D54B78F0 (schema-change-not-expressible-in-place) — this is NOT retention expiry and repinning at an older position will never succeed; the new history's first servable position is 1/D54B78F0: repin at 1/D54B78F0 ``` Repinning *older* never helps here — that is exactly the difference from retention expiry, which the message calls out by name. The cause is always one of three: a schema change Iceberg could not express in place, an Iceberg format-version downgrade, or a failed eligibility check. **A scope naming something that does not exist:** ```text pin scope names unknown lake table(s) ["postgres__public__ordres"]; published tables: ["postgres__public__orders", "postgres__public__line_items"] ``` The misspelling in the example is deliberate: the refusal prints the published set, so you can correct yourself from the message alone. **Recording an old gap by hand.** The tail records its own gaps. For a gap that happened before gap recording existed — known from logs — there is an operator command that only ever **adds** rows: ```console $ argondb lake gap-record --help Record a changelog gap in `_argon_gaps` …. Ops escape hatch: the tail records its own gaps; this backfills gaps that predate gap recording (known from logs) so changes() stops being confidently wrong over those windows Usage: argondb lake gap-record [OPTIONS] --start --end --reason Options: --start Gap range is (start, end]: LSNs as X/Y --end --reason Why the range is unreconstructable (shown verbatim in digests and refusals) --table
Affected lake table; omit = every tracked table --forfeits How many commits an undecoded-window barrier episode coalesced into this ONE record. Omit for every gap class that is not a barrier episode — omitted is NOT the same claim as 0 --at When the gap happened, RFC3339 (default: now) --branch [default: main] … ``` `--reason` is shown **verbatim** in digests and refusals. Whatever you type is what a developer who hits that window six months from now will read. #### Example 10 — the change log's columns, and a `changes()` digest The change log is what gives you row-exact history and change digests. Turn it on in the tail's configuration, naming tables by **flat lake key**: ```bash export ARGONDB_CHANGELOG=postgres__public__orders,postgres__public__line_items # or, equivalently, on the tail's own command line: argondb-tail --changelog postgres__public__orders --changelog postgres__public__line_items … ``` The tail accepts any string here without checking it, so an entry that names nothing silently produces no change log. That is why `argondb lake list --tail ` reads the configuration back and names every entry that matches no published table — and why, without `--tail`, it tells you it did not check. The resulting table can be queried like any other. Its four extra columns are described in section 2.6: ```sql -- from a foreign engine SELECT _argon_lsn, _argon_op, _argon_seq, _argon_ts, id, status FROM lake."br_main.postgres.public".orders__changes WHERE _argon_lsn > 0/19C0000 ORDER BY _argon_lsn, _argon_seq; ``` The replay the query plane itself runs has exactly that shape: ```sql SELECT * FROM chg WHERE _argon_lsn > {w_base} AND _argon_lsn <= {lsn} ``` Through the agent endpoint, the `changes` digest tool takes a window in any of three forms: ```jsonc { "name": "changes", "arguments": { "since": "now-1h", // or "0/1A2B3C", or "2026-07-08T14:00:00Z" (μs resolution) "until": null, // defaults to now (head) "tables": ["postgres__public__orders"], // default: all tracked tables with history "budget_tokens": 4000 // default } } ``` If the window overlaps a recorded gap, the digest is **still served**, and its counts are marked as lower bounds over the overlapping range(s). A digest is never silently short. #### Example 11 — the tail's environment settings These are the settings you would actually change when sizing the tail for a machine or a workload. Each row says what it changes and what happens on a bad value. | variable | default | what it changes | |---|---|---| | `ARGONDB_TAIL_DRAIN_QUEUE_MB` | **2048** (MiB) | the ring's byte budget. High-water (cut now) at 50 %; backpressure at 100 %. `0` is **refused** with a WARN; an unparseable value falls back to the default with a WARN. Read once, when the ingest loop starts | | `ARGONDB_TAIL_MIN_CUT_SECS` | **5** | the minimum interval between cuts, so a byte-triggered cut cannot outrun the maintenance worker's compaction thresholds (which are multiples of the cadence). Same value as the lake writer's default cadence | | `ARGONDB_TAIL_RING_COALESCE_ROWS` | **512** | how many rows one merged ring batch may hold. `0` turns merging off. Removes about 4.7× of the ring's counted bytes on the ERP workload. Unparseable → default, **and the log says so** | | `ARGONDB_TAIL_WAL_HOLD_CAP_BYTES` | **4 GiB** | how much WAL the safekeeper keeps for the tail's resume point, *and* the tail's restart replay budget — one number, two jobs. `0` turns the whole mechanism off and restores the pre-2026-09-17 behaviour exactly. Unparseable → default, printed; never unbounded, never off | | `--cadence-secs` (set by the box) | **30 s** | how often the cadence loop cuts when no byte trigger fires. The box sets it explicitly rather than relying on the tail's default | | `ARGONDB_MAX_LAG_BYTES` | **256 MiB** | how far behind the head a table may be at startup before it re-baselines on its own; also the other term of the replay budget, which takes the **larger** of this and the hold cap | | `ARGONDB_TAIL_ZERO_WAL` | **on** | the zero-WAL lever (section 3.9): an UPDATE record with no old key, on a primary-keyed table under the default replica identity, is applied as a same-key update with no old-page read at all. `off` restores the read. Read once per process | | `ARGONDB_TAIL_DECODE_DECOUPLED` | on | whether the old-page fetch overlaps the rest of decode (section 3.9). The off switch exists for measurement and fault isolation | | `ARGONDB_TAIL_MEMORY_MB` | derived (6,144 / 5,632) | moves the cgroup cap (`memory.max`) and its alarm line (`memory.high`) together. Set below the 4,704 MiB floor, the box refuses to boot | A worked sizing pass, with the trade-off in each direction: ```bash # A memory-constrained box: halve the ring. The cut rate RISES (high-water is # hit sooner), backpressure time rises, and the tail holds less. export ARGONDB_TAIL_DRAIN_QUEUE_MB=1024 # A box with headroom and a bursty writer: raise the ring and let the clock # do more of the cutting. Snapshots get larger and less frequent; the tail # holds more, and a crash re-decodes more. export ARGONDB_TAIL_DRAIN_QUEUE_MB=4096 # Turn batch merging OFF to reproduce the old, unmerged ring cost (for a # measurement, not as a configuration): expect ~4.7x the counted bytes for # the same rows. export ARGONDB_TAIL_RING_COALESCE_ROWS=0 # Restore the pre-2026-09-17 WAL horizon exactly, hold and replay budget both. # A restart under lag then re-baselines with a gap, as it did before the hold. export ARGONDB_TAIL_WAL_HOLD_CAP_BYTES=0 ``` The ring setting's effect is checked, not assumed. ArgonDB's memory-bound test runs a second pass at `ARGONDB_TAIL_DRAIN_QUEUE_MB=512` and requires: the budget read back as 512 MiB; a high-water at or under 512 MiB; *more* backpressure time than the default pass; and the queue still drained. **What to watch while it runs.** The metrics are `drain_queue_bytes_high_water` and `drain_queue_backpressure_millis_total`. Two log lines carry them: ```text tail drain queue: bytes=1073741824 (1024 MiB) high_water=1024 MiB budget=2048 MiB rows=46355 backpressure_holds=3 backpressure_secs=1.8 backpressure_expiries=0 ingest phase=backpressure for 2s, ring_rows=46355, drain_queue=2048 MiB (budget 2048 MiB, 3 hold(s)), open_txns=1 holding 12 MiB (peak 40 MiB, budget 512 MiB), evicted_open_txns=0, toast 0 value(s) / 0 block(s) scanned + 0 hint block(s) ``` `ring_bytes_peak` and `ring_backpressure_secs_total` are **retired names** that nothing emits. They appear only in older design notes. #### Example 12 — the maintenance plane's cadence, and what it writes You do not configure maintenance, but you will see its snapshots and log lines, so it helps to know what it writes. The cadence is a code constant: ```rust pub const MAINTENANCE_SWEEP_SECS: u64 = 120; ``` There is no switch, no setting and no off state; tests reach it only through a test-only override. In the split production layout it is its own supervised process, started only when the tail is also started. A maintenance worker with no tail would be a second writer with no ingestion to coordinate with. What a pass writes, per table: - a **compaction** (`RewriteFiles`) snapshot, when the file count crosses a threshold that is a multiple of the cadence; - a **clustered** rewrite, when `argon:cluster-by` applies (explicit, or the single-primary-key-column default); - a **Puffin statistics** file with exact distinct-value counts; - a **snapshot-expiry** commit under explicit retention, which keeps the newest N snapshots, *plus* the time-travel anchor, *plus*, on a table with no change log, every watermark-stamped snapshot inside the retention window; - an **orphan-file** deletion pass, limited to data files no retained snapshot references and older than a grace period for in-flight writers. Statistics are re-computed only when a table's rows changed. Computing them reads every data file of the table, so each pass first checks whether the statistics already registered still describe the table's current rows. Each table it skips is counted as `stats_skipped_fresh=` on the pass's `maintenance pass cost:` line. The check matters because a cut gives every table a new snapshot. A cut that carried no rows for a table still records its new watermark there (the lake watermark moves), while the table's rows stay the same (the row watermark does not). Such a snapshot is marked `Argon.rows-unchanged-since`, naming the snapshot before it. Statistics count as current when they sit at the current snapshot, or at an earlier one with only marked snapshots after it. Anything else means a re-scan: a snapshot that carried rows, a compaction rewrite, another engine's snapshot without the mark, or statistics whose snapshot has already expired. Until 2026-10-05 only the first case counted, so on a quiet box every cut forced every table to be re-scanned. a customer's 385-table box re-read all 20 MB of its tables on every pass. The `profile` tool decides "current" with the same check. Its log lines use the `label` spelling from example 3: ```text maintenance br_main.postgres.public.cas_zoo__changes: changelog compaction rewrote 7 data file(s) -> 1 ``` A pass that overruns the cadence backs off rather than overlapping. On the live demo at the 120 s cadence, passes longer than the cadence were measured, and they are kept as the test case for the back-off. A pass's deadline is 30 cadences, one hour. Clustering is set with a table comment, the same way as the publish marker: ```sql COMMENT ON TABLE public.orders IS 'argon:publish argon:cluster-by=customer_id'; COMMENT ON TABLE public.events IS 'argon:publish argon:cluster-by=none'; -- opt out of the primary-key default ``` #### Example 13 — force a clean rebuild of one table The tail heals itself only when a write fails: a failed RowDelta commit triggers a scan-replace. A table left duplicated or drifted by a cause that has since been fixed never heals on its own. For that there is a repair command: ```console $ argondb lake rederive public.orders # DRY RUN by default: prints the plan, touches nothing $ argondb lake rederive public.orders --execute ``` It automates the proven path: unpublish (never with `--purge`), then publish again. That sends the table back through the tail's own mid-run join, so the tail stays the single writer. It deliberately does **not** rewrite the table from the CLI process, which would be a second writer racing the cadence on the same table. #### Example 14 — the whole path, end to end Every question about freshness comes down to this path. Across companies it is longer — the architecture site traces one price change across six hops, from one company's database to another's. Within one tenant it is five steps: ```text 1. an ordinary COMMIT in Postgres <- the application's latency; unchanged 2. the WAL reaches the safekeepers <- durability 3. the tail decodes it into the ring <- H advances; typically well under a second behind 4. a cut lands it in Iceberg <- W advances; ≤ 30 s by clock, sooner on a byte trigger 5. a reader pins L and reads <- Iceberg@W ∪ tail(W, L], merged by primary key ``` Between steps 3 and 4 the row is **already readable at full consistency**; that is what the ring is for. Step 4 is about durability and file health, not visibility. A reader that pins at the head sees a row the moment step 3 completes. A foreign engine reading the Iceberg table directly sees it only after step 4. ### 5. What works today, and what is designed but not built This table lets you check each feature before you depend on it. In the diagrams, solid outlines mean the feature works today; dashed outlines mean it is designed and measured but not built. | feature | status | detail | |---|---|---| | Physical decode to Arrow row changes, single decode path | works today | shipped since v0.1.0-alpha | | One atomic multi-table commit per cut, watermark-stamped | works today | uses the Iceberg REST `/transactions/commit` endpoint, through ArgonDB's fork of iceberg-rust, which carries this ahead of upstream pull request #2786 | | `argon:publish` marker, per table, nothing published by default | works today | | | `__changes` change log, opt-in per table | works today | | | Pins, gap records, time travel, and pins that are always granted at the default | works today | | | Maintenance plane, every 120 s, no off switch | works today | | | Mapping of lake-unrepresentable values, and per-table isolation of decode failures | works today | | | Branch → Iceberg namespace, materialized lazily | works today | | | Three-level address (`branch/database/schema` + leaf) | works today, one step left | Tables created before 2026-09-11 still use the older name shape; renaming them one by one is not done yet and waits on a rehearsal on a scratch copy. Old-shape names boot and list through a compatibility read path; that state is deliberate and tested | | One tail process following every tenant timeline as its own stream | works today | one Flight port per stream; a per-stream replay budget with its own gauges; about 8–10 MB of memory per loaded stream | | Pageserver token auth on the box, a tenant compute's token scoped to its one timeline | works today | the tail and the management path carry the box-wide token; a compute's token is refused on any other timeline (section 2.2) | | Warm standby tail and the guard that stops two tails writing one lake | **not built** | today the supervisor runs exactly one tail | | Old-page read decoupled from decode | works today, on by default | measured gain 2.0–2.7 % | | The zero-WAL lever (no old-page read for a same-key UPDATE) | works today, **on by default** | decode ceiling 10,009.7–10,388.9 txn/s, no added WAL, at P1 with neither lever at 748.0–754.8 txn/s | | Old row taken from the WAL (`--old-row-from-wal`), per table | works today, off by default | decode ceiling 9,695.8–10,067.7 txn/s at the same P1 level, for WAL growth of ×1.047; the remedy for a table the zero-WAL lever cannot help | | History-restart refusal names the first servable position | works today | the one refusal whose remedy points forward, never back | | Lake view refresh killed right after its Iceberg commit restarts whole, not reapplied | measured (a test, against an in-memory catalog standing in for the REST catalog) | see section 3.6 | | `dv_*` tables refuse a write from any door but their own owner (`not-owner`) | works today | section 2.7 | | Sync drops a **stale** cross-database mount | works today, opt-in (`argondb_lake.drop_stale_mounts`, off by default, superuser-only) | example 5 | | Sync drops a **revoked-grant** mount | **not built — open question** | its source is still alive, so the mount is kept and warned instead | | Ring byte budget and backpressure | works today | | | Ring batch merging (`ARGONDB_TAIL_RING_COALESCE_ROWS`) | works today | the per-row cost reading in section 2.4 is its measurement | | Tail WAL hold on the safekeepers, capped and persisted | works today | a restart test proves both directions (hold on, hold off) | | 6 GiB cgroup cap and 5.5 GiB alarm line | works today | | | The `tail memory: over the alarm line` log line | **not built** | the design asked for it; the cap and the alarm shipped without it, and the v1.5.4 release notes say so | | Automated box test proving the memory bound | works today | three passes, with five checks that must pass and three that must fail | | Postgres lake mount (`argondb_lake`), scoped to its own database plus tables granted to it | works today | `timestamptz` columns cannot be read through it yet | | Derived tables landing in Postgres (the `derived` schema) | works today | | | Derived tables landing in the lake (`dv_`), for transform results and for direct appends | works today (v1.4) | | | `upsert` mode for lake landing | built, opt-in only | a mode you ask for explicitly (`--mode upsert` or its MCP equivalent); never the default of any flow, which lands by `append`. Refused on a table with no key | | Iceberg v3-capable writer | decided, **not built** | today the writer always writes Iceberg format **v2**. The deployment default and the per-table `argon:format-version=3` override return when the build starts | | Transparent HTAP offload (an enrolled session's analytics-shaped query, sent to ordinary Postgres, is answered by the read plane) | shell built, V1 limited, **not shipped** | the shell passed its 4 box tests on 2026-07-24. V1 runs only the isolation trigger (offload a heavy read to protect the primary). The speed trigger only logs its decision and never executes, because in V1 the accelerator is reachable only as an MCP tool. The `offload_min_cost` threshold needs a benchmark before any deploy. The `argondb_offload` extension is no longer in the shipped extension set, so `CREATE EXTENSION argondb_offload` fails today | | Sharding decode across several processes (three planned stages) | designed, **not planned** | measured 2026-07-27: peak decode backlog 0.30 MiB against a publish backlog of 959.81 MiB. The bottleneck was downstream of anything sharding helps, and it will not be built on theory | | Deriving the WAL hold cap from free disk space | deferred; the measurements recommend against it as posed | the cap stays a constant. If it is revisited, derive it from a stated runway in minutes at the measured decode deficit, not from a share of free disk | | Restating the memory bound on the measured baseline | deliberately not done | one reading does not move a published number; `ERP_SCALE=1` is one point, not the curve; and running under the constant is the safe direction | | Maintenance on Kubernetes (Helm) | **not built** | the Helm chart ships no maintenance workload, and no tail workload either, so a stock Kubernetes install does not ingest into the lake and serves with a permanent freshness disclosure | | Capturing old row images for foreign incremental reads | not built, by choice | `GetPage@LSN` can always produce old images later; to be revisited if a CDC customer relies on incremental reads | ### 6. Gotchas These are the things that surprise people in practice. Each states the symptom, the reason, and the remedy. #### 6.1 `br_` is conditional Lake namespaces for most branches have no prefix, which breaks scripts that assume one. Only `main`, `temp` and `system` take `br_`. A `br_$BRANCH` composed in a shell is right three times and wrong every other time; `dv_` is unconditional. Remedy: `argondb lake namespace [--branch B] [--territory lake-writer]` (example 8). #### 6.2 A nested namespace needs one quoted identifier Unquoted lake names fail in DuckDB with an error that does not mention quoting. `lake.br_main.postgres.public.orders` is four SQL name parts, and DuckDB stops at the parser: `Parser Error: NameListToString NOT IMPLEMENTED`. It reads like a broken query. Remedy: `argondb lake address `, which quotes correctly every time. #### 6.3 `SHOW ALL TABLES` does not list your tables Discovery from DuckDB does not work; reading does. This is measured, not theoretical: DuckDB lists only the catalog's top level. Reading a table you *can* name works perfectly. Remedy: `USE lake."br_main.postgres.public";` once per namespace per session, or walk the REST catalog, or run `argondb lake catalog-tables`. #### 6.4 The publish marker is a comment, so it can land on things that are not tables Because any client can write a comment, the publish marker can end up on relations that cannot be captured. A view, a materialized view, a foreign table, a partitioned **parent** (it holds no storage — publish its leaf partitions instead), a sequence and an index are all uncapturable, and any client can `COMMENT` the marker onto them without the CLI. Refusing the DDL would need a Postgres event trigger, and therefore superuser on the primary, which you do not have on a bring-your-own-Postgres deployment. So the marker is **accepted and announced**: one rate-limited `WARN` per (database, relation) naming the relation and its kind, and `argondb lake list` shows it as `not-publishable ()`. The WARN always names the **database** (`postgres.pmv.v`), because publish discovery reads the whole cluster and a relation name alone no longer identifies a relation. Also: TEMP and UNLOGGED tables cannot be published at all (no WAL, nothing to capture). And **new tables are not published automatically**: publishing is per table, by design. #### 6.5 No primary key means a full rewrite every cadence the table changed A table without a primary key is exact but expensive to keep up to date. ArgonDB guarantees 100 % agreement between the database and the lake with or without a primary key. What a missing key costs is update performance, in time and in load on the whole system, because the lake copy is maintained by full-table scan-replace. Remedy: put a primary key on every published table. It also affects maintenance: tables with a composite primary key, or none, get **no** clustering default, so fast primary-key point lookups do not apply to them. #### 6.6 `0` is refused on the ring budget and on the open-transaction budget Two settings reject `0`, and one nearby setting treats `0` as "off" — so the same digit means opposite things in one process. On `ARGONDB_TAIL_DRAIN_QUEUE_MB` and on the open-transaction budget (`ARGONDB_MAX_OPEN_TXN_BYTES`), `0` is refused rather than read as "unbounded", because unbounded is what the budget exists to prevent. If you want effectively no limit, set a large number. Unparseable values fall back to the default **and the log says why**. By contrast, `ARGONDB_TAIL_WAL_HOLD_CAP_BYTES=0` **does** mean off; it is the documented way to restore the old WAL horizon exactly. Read the table in example 11 before setting either. #### 6.7 The tail's self-heal only fires when a write fails A table that went wrong for a reason that is now fixed stays wrong until you act. The tail's snapshot-replace heal is triggered by a failed RowDelta commit and nothing else. Remedy: `argondb lake rederive
--execute`, which automates unpublish-then-publish (never `--purge`), so the table goes back through the tail's own mid-run join rather than being rewritten by a second process (example 13). #### 6.8 A time-travel refusal inside a gap is permanent If a time-travel read hits a recorded gap, no amount of retention or waiting will fix it: the rows were never decoded. The refusal names the recorded range and the reason, and the remedy it prints — `pin at or after , or repin for a fresh view` — is all that is available. A `changes` digest over the same window behaves differently: it is served, marked as a lower bound. #### 6.9 Read-your-writes is lost at the default on tables without a change log On a table without a change log, a default pin can be older than your last write. `latest_servable` snaps down to the newest watermark that every scoped table without a change log retains, so a pin on such a table is as fresh as the last **lake commit**, not the last write. This cost is accepted by design. Remedies, in order of preference: turn on the change log for the table; narrow the scope; or use `freshness=head`, which means "the head or a refusal" and **discloses at mint time that the position is transient**. #### 6.10 A pin token can outlive what it names — in exactly one case Pin tokens usually survive everything within their window; one kind does not. A token minted at the `latest_servable` default survives tail restarts **and** watermark advances for its whole window. A token minted at a **caller-named** position — `freshness=head` above a change-log-less table's `W`, or an explicit `at_lsn` in that transient range — survives a restart but **not** the watermark moving past it. The mint says so when this applies, and the provenance field in the token is what later lets redemption tell the two apart. #### 6.11 Over-disclosure on a time-travel pin A historical read may carry a value note that does not apply to it. An `at_lsn` pin includes the tail's **current** pending value notes, so a note about a value that landed after the pinned position can appear on an answer that does not contain that value. This is deliberate and the safe direction: the failure being prevented is an affected value served with no note. #### 6.12 Do not reconcile a table with Iceberg's `total-records` The obvious check — compare the snapshot summary's `total-records` with the source row count — fails in a way that looks like a broken lake. That field counts rows present in data files, not rows in the table. An `UPDATE` writes the new row version to a new data file and retires the old one with a delete file, so `total-records` goes up while the row count does not; on a table that has ever been updated it reads high until compaction rewrites the files. Subtracting delete counts does not repair it, because ArgonDB writes an equality delete for every upserted key, so the result is only a lower bound. Remedy: count rows with a scan, for example `SELECT count(*) FROM lake."br_main.postgres.public".
;`. #### 6.13 Backpressure is not a stall, and the metrics say so If you write your own alerts, a naive "not advancing" alert will fire on healthy backpressure. Do not treat a non-advancing read position as stuck without excluding backpressured windows. The product already does this with one shared check; a second copy in your monitoring that skips it will misfire. Read `ingest phase=...` and `tail drain queue: ...` (example 11). #### 6.14 Two metric names you may find in older notes do not exist `ring_bytes_peak` and `ring_backpressure_secs_total` are retired names that nothing emits. The real names are `drain_queue_bytes_high_water` and `drain_queue_backpressure_millis_total`. #### 6.15 There is no clean shutdown, and that is the design The tail treats every exit as a crash, on purpose. `argondb-tail` installs no signal handler, and a unit test asserts its main source file never mentions `SIGTERM`, `ctrl_c`, `signal_hook` or `SignalKind`. The restart point is durable at every cut. Do not add a drain-on-SIGTERM: it reopens the class of bug where a crash lands between the write and the exit. #### 6.16 The decode rate, not the cap, is what forfeits a fast tenant If a tenant writes faster than the tail can decode, a bigger WAL hold only delays the forfeit. At 50 clerks the tail decodes 659.6 txn/s against 2,900.4 tps offered and never catches up. Doubling the WAL hold cap doubles the runway — 13 minutes becomes 26 — and changes nothing else, because at that load the deficit is structural. Size the decode, not the buffer. Profiled stage by stage, the pre-cache ceiling is **one cost**: fetching old tuples from the pageserver (`oldpage_fetch`) is 96.5 % of decode time and by itself limits decode to 198.0 txn/s, whatever else in the tail changes. Decoupling that fetch from decode gained only 2–3 %. What moves the ceiling is skipping the old-page read outright. On a primary-keyed table under the default replica identity, the **zero-WAL lever does this by default now** for every UPDATE that keeps its key, with no WAL cost at all. For a table it cannot help — most importantly one with no primary key — the remedy is the older, opt-in lever: `argondb lake publish
--old-row-from-wal`, at the cost of somewhat more WAL (section 3.9). #### 6.17 Snapshot expiry keeps more than you might expect, and for a reason On tables without a change log, metadata grows more than you would guess. Snapshot expiry keeps: the newest N snapshots, **plus** the time-travel anchor (the newest snapshot at or older than the retention window, or the oldest snapshot if the table is younger than the window), **plus**, on a table with **no** change log, every watermark-stamped snapshot inside the window — because there the servable positions *are* those watermarks, and replay cannot cover any distance from the anchor. If you are estimating metadata growth on tables without a change log, that third term dominates. #### 6.18 A quarantined table is not a gap A table that stops advancing may be quarantined, and that is different from losing data. A table whose decode fails is quarantined **on its own** at its last good watermark. The cadence, every other table and the read plane are unaffected, and nothing was lost. `argondb lake list` shows it as `QUARANTINED @ (frozen; cause=: )`, and a read of it is served base-only at that watermark **and says so**. (Before this state appeared in the listing, the watermark simply stopped moving and you had to read the tail's log to learn why.) #### 6.19 Two things about the change log that surprise foreign readers Foreign Iceberg engines that read the change log incrementally need to know two things. First, it is append-only at the **row** level, not the **snapshot** level: maintenance compaction may commit a Replace snapshot. Rows are preserved exactly and every ArgonDB reader is unaffected, but a foreign engine doing standard Iceberg incremental reads must fall back to a full re-read on a replace snapshot. Second, a delete is a tombstone carrying the **key**, not the old row image. Capturing old images is not built, and that is a deliberate choice. #### 6.20 A too-small memory budget is raised at boot, not refused A small box still boots; the tail simply gets more than its computed share. The tail's floor is 4,704 MiB. A box whose own memory budget would give the tail less does **not** fail to boot: the share is raised to the floor, and the boot line names the share, the budget and the floor. The box refuses only when an operator sets `ARGONDB_TAIL_MEMORY_MB` explicitly below the floor. If you read the memory rule (section 3.5) as "refuses below the floor", you will expect every small deployment to fail to boot; it does not. ### 7. Sub-diagrams Four more detailed diagrams open up parts the main picture compresses. They use the same drawing style, palette and legend as the main set, and each is generated from a Python script kept beside it in the repository. | diagram | what it opens up | |---|---| | [3a — the ring, the cut, and W/H over time](diagrams/argondb-and-the-erp/03a-ring-and-cut.svg) | what may enter the ring, the three thresholds on one budget, the four cut triggers, and the staircase of `W` under a continuous `H` | | [3b — the unified read, per table](diagrams/argondb-and-the-erp/03b-unified-read.svg) | the three cases (`L` above `W`, `L` below `W`, no change log), the merge they share, and what the pin token carries | | [3c — the tail's memory budget as a stack](diagrams/argondb-and-the-erp/03c-tail-memory-budget.svg) | the seven buffers, the 25 % slack, the cgroup cap and alarm, the measured baseline, and the three lines that make memory, lag and the cap one system | | [3d — the gap record and the rescan](diagrams/argondb-and-the-erp/03d-gap-and-rescan.svg) | the four routes to a re-baseline, quarantine as the fifth and different outcome, the required ordering, the gap row's columns, and the two things a reader is told | ### 8. Where to go next The lake is one part of a larger system; these chapters cover its neighbours. - [Guide 1 — one database per tenant, and the wall between tenants](#guide-1-one-database-per-tenant-and-the-wall-between-tenants): why each tenant has its own database and its own lake warehouse. - [Guide 2 — the per-tenant unit](#guide-2-the-per-tenant-unit-the-gate-wake-and-stand-down): how a tenant's compute stands down and wakes, which the tail's schema connection is careful never to trigger. - [Guide 4 — continuous materialized views](#guide-4-continuous-materialized-views-and-a-view-across-the-tenant-boundary): the lake writer's main job in `dv_`. - [Guide 7 — agents and the MCP endpoint](#guide-7-agents-and-the-mcp-endpoint): the `changes` digest and the other agent tools that read the lake. --- ## Guide 4 — continuous materialized views, and a view across the tenant boundary A continuous materialized view in ArgonDB is a query result that the database keeps up to date for you. You define it once, as a SELECT. From then on ArgonDB brings it forward from the stream of committed changes to its input tables, instead of re-running the whole query on a schedule or having you maintain a summary table by hand. Every copy of the result is stamped with the exact point in the write-ahead log it is current to, so a reader always knows what moment in time the view answers for. Where it can, ArgonDB also answers ordinary queries from a matching view without being asked. A view never reaches across the wall between tenants: it reads only tables of its own database. This chapter is for a developer who has to use views: define them, read them, predict what they cost, and work out what is wrong when one stops advancing. It explains how they work; it does not argue for them. **Before you build: the limits that matter most.** Each is covered in detail below. - **One kind of schema change on an input is not handled correctly by an incremental view.** The latest full correctness run fails 5 of 102 assertions, and one of them is a real, unfixed product defect in how an incremental view follows a particular schema change. If your workload makes that kind of change, check the result by hand or use a recompute view for that table (§6.1, §9). - **A view over a table you only reach through a mount grant is refused outright.** Ruled on 2026-09-27: a mounted table is never a view input, because a mount is a live read with no integrity guarantee. Have the owner publish the table to your database and build the view over the landed copy, which is your own table (§2.13, §4.9, §6.17). - **A row written while a keyed view is refreshing may be missing from the very next refresh.** Whether this is a defect is not known yet; it is being measured (§9). - **A dependent view that stops consuming holds its upstream's snapshots forever in production.** The pin-window expiry that should eventually free them is not enforced by the live sweep. Drop a dependent you no longer need rather than leaving it tripped (§3.8, §6.18). - **A Postgres-landed recompute view ignores its own refresh-interval cap.** The cap works for the lake landing; on the Postgres landing it silently never engages (§6.19). - **Some box tests have not been run.** The box test for the create step is written and has not been run. For the PostgreSQL landing, the box test for keyed lookups and a kill between `COPY` and `COMMIT` is written and passed on 2026-09-25, but the read-back through DuckDB has not been run. The chain correctness test's coverage check has not yet passed on a box (§5, §9). - **Only a schema change to a column a view reads marks it invalid.** If a column the view reads is dropped or has its type changed, the view is marked `INVALID` and keeps serving its last result. It clears at the next refresh that succeeds, once the column is back as it was. Other schema changes, and columns the view does not read, change nothing (§5). - **Not decided yet: how long view snapshots are kept** — the full pin window, or less because a view can always be rebuilt (§5). - **A recompute view is capped in how often it refreshes**, and a view under steady updates collects delete files that nothing compacts yet, so its reads slow down over time (§2.9, §6.13). **How to read the examples.** Each example is marked either **RUN** — run for this chapter, with its real output pasted — or **FROM THE CODE** — the exact signature, flag, constant or refusal text quoted from the source code or its design, and not run for this chapter. No ArgonDB box was booted while this chapter was written. The command-line help in §4.0 was built from the source and run on a development machine, and is RUN. Everything else is FROM THE CODE; where a FROM THE CODE example shows output, that output is put together from the code that produces it, and it says so. **How to read the pictures.** A solid outline is built. A dashed outline is designed, or designed and partly built; it is not a promise about a date. The main picture also marks one question that is not decided yet: how long view snapshots are retained. Where a caption in a picture has gone out of date, the text says so and the text wins. --- ### 1. The diagram ![Deep dive 4 — continuous materialized views, and a view across the tenant boundary](diagrams/argondb-and-the-erp/04-continuous-materialized-views-and-a-view-across-tenants.svg) The picture shows how one view is kept current and read, and then the three choices you make when you define one. Read it as four horizontal bands, top to bottom. Read the top band left to right as one refresh happening. **The top band is one refresh.** The tail — ArgonDB's process that reads the write-ahead log — decodes every committed change to every published table into a change row. Each change row carries `_argon_lsn` (its position in the log), `_argon_op` (`I`/`U`/`D`, for insert, update and delete) and `_argon_seq`. Change rows live in exactly two places that last. Above the lake watermark `W` (the log position the lake has been published up to) they are in the tail's in-memory ring. At and below `W` they are in the table's changelog, `__changes`. Which one a consumer uses depends on when it runs, not on preference. The view maintainer runs inside the `argondb-maintenance` process. It notices that `W` has moved past the log position a view is current to, called `L_v`, and schedules that view at `L_1 = W`. It applies the view's arithmetic to the changes in the window `(L_v, L_1]` and lands one snapshot. The rows and the stamp saying which point they are current to, `Argon.view-current-to-lsn = L_1`, are in the same commit. A later read pinned at some log position `L` may then be answered from that view. The rightmost box, where reads are rewritten to use views, is the newest part of the band; §5 says exactly how much of it is built. **The three bands below are the three choices you make.** - **The classes band** is the arithmetic. Some views can be brought up to date from what changed: keyed projections, algebraic aggregates, and joins where ArgonDB knows which side changed. Others have to be recomputed from their inputs. The classifier decides which when you create the view, and prints its answer. - **The landings band** is where the rows physically sit. Either one Iceberg table in `dv_`, read through the same foreign-data wrapper as every other lake table, or one real PostgreSQL table in `derived.*` on the primary database. A view has one landing, never both. - **The bottom band** is how views meet the wall between tenants, and its shape is the point: a view never reaches across the wall. A view that looks like it spans two companies is an ordinary join of two local tables, one of which is a reference table the hub copied into this database earlier. The "no query reaches across" box is not a special policy for views. It is the same wall that the lake mount and every scan ticket already carry, checked once more by the maintainer. The word "sidecar" in this part of the system means only the tenant's own application process. ArgonDB's own process is *the tail*. --- ### 2. What it is This section takes the picture one box at a time. For each box it gives the coordinates, states, defaults, limits and failure modes the picture had no room for. #### 2.1 The tail — the change rows, and the two things they do not carry The tail is where every view's input comes from. It reads the write-ahead log and turns it into a stream of change rows for each published table, so that nothing downstream has to read the log itself. Each change row is the table's own columns — the **new** image of the row — plus `_argon_lsn`, `_argon_op` (`I`/`U`/`D`), `_argon_seq` (the order within one commit) and `_argon_ts`. Two things are missing, and they shape everything downstream: - **A `D` row carries the key, not the old row.** - **A `U` row carries the new row, not the old one.** The tail does keep old rows while it decodes, but that cache is internal and is never sent. So any arithmetic that needs to subtract the old value — every aggregate — has to go and fetch the old row itself. §3.4 is where the cost of that fetch is worked out. A third property is easy to miss and matters a great deal: **an UPDATE that changes a primary key is not a special case.** The tail sends it as a `D` on the old key and an `I` or `U` on the new one, because the merge plan treats a key as a partition. The classifier relies on this, and a box test checks it. #### 2.2 Two delta sources, and the rule that picks between them To bring a view forward, the maintainer needs the changes to its inputs since the view was last refreshed. Those changes can be in two places, and only one of them is still there when the maintainer runs. - **Above `W`**, the change rows are in the tail's **ring**. They are served over Flight as `(from_lsn, to_lsn]` tickets, and they are **evicted the moment a cut lands**. - **At and below `W`**, the change rows exist only if the table is enrolled in the **changelog**, `__changes`. This is an append-only Iceberg table of exactly those rows. It is committed in the same atomic cut as the base table, and it is kept for at least the pin window (48 hours). The rule that follows: the ring is the right source when a *read* needs to make up the difference to a newer point (§3.10), and the changelog is the right source for *maintenance*. A refresh runs at a cut, and at a cut the ring has just been evicted below it. **So a view is maintained incrementally only over inputs whose changelog is enrolled.** A view with an input that is not enrolled is a recompute view whatever its shape. That is the first thing the classifier prints. **A view's first refresh is pinned at or after the point its inputs' changelogs start, never left floating.** The first refresh raises its pin to the latest of its inputs' changelog enrolment points. This departs from the usual default, which is to pin at the branch's latest servable point. That point could sit inside the gap before an input's changelog began, and a window starting there could be rebuilt from neither source: the changelog only starts recording at enrolment, and the ring has been evicted below the cut. The floor is read from the branch's gap record, and under per-tenant compute a tenant's maintainer reads that record through a per-database filter: every tail of the box appends to the one `_argon_gaps`, so a row that ends above the tenant tail's own watermark is treated as another timeline's re-baseline and not applied. An enrolment row is the one kind that filter was not written for. It is scoped to one table by name, it ends at that table's lake watermark — which can stand above a quiet tenant's held watermark — and it is this tail's own row whenever the table is one this tail tracks, so it passes the filter whatever its coordinate (2026-10-24). Before that, a hand `argondb lake changelog` on a quiet tenant recorded a floor the maintainer never saw, and the view's first refresh re-pinned at the cut every pass: a dead pin on exactly the shape the floor exists for. **A first refresh asks the tail for a cut naming its inputs.** The cut watermark `W` a refresh pins at advances on every cut, including a cut that landed nothing: on a quiet tenant Postgres' own WAL moves the head, the idle cut publishes the new `W`, and nothing is made durable there until the next landing or the hourly record. A first refresh — the one with no `L_0` — would pin at that `W` with no input able to serve it exactly. So it asks the tail for a cut on demand naming its inputs, the same door a landing's data pin uses: the answering pass is a landing cut, it stamps the named inputs once and covers the rest with its record, and the refresh pins at that cut, where every input is durable. The ask is not remembered as a landing's working set, so the idle cuts after it stay idle. The maintainer's log says `first refresh asked the tail for a cut on demand naming its N input(s): pass P answered at `. A refusal or a time-out is said and the pin falls back to `W` (2026-10-24). A third source exists for a view that reads another view — the upstream view's own committed snapshots — and §3.8 covers it. Some windows are **not a delta** at all: one that overlaps a recorded gap (`_argon_gaps`) on any input, or one that begins below where an input's changelog is still kept. For those, the view recomputes in full at `L_1`, and the cause is recorded. Both cases are read from the same record and named by the same rule, so a reader gets one consistent BEHIND sentence either way. And a changelog retention floor is itself recorded as a gap row *before* the changelog is actually pruned, so you never see old rows simply gone with nothing to explain them (§4.12). #### 2.3 The view maintainer — where it runs, and the three things it will not do The view maintainer is the part of ArgonDB that refreshes views. You never call it. It watches the lake advance and brings each view forward, so you stop scheduling refreshes by hand. It is a **task inside the `argondb-maintenance` process**. It sits beside the lake's maintenance sweep, never inside it, because the two run on different clocks: the sweep runs every 120 seconds, and a view refresh runs when a cut lands. The maintainer follows cuts. Every `ARGONDB_VIEWS_POLL_SECS` seconds (default 5) it reads the tail's `freshness` action. That setting is how quickly it *notices* an advance; it is not a refresh interval. It acts only when the watermark has advanced, with one exception: - **A new view refreshes once when it is created, even on an idle branch.** Creating a view does not move the watermark. Without this exception, a view created after its branch's writers had finished would never refresh at all. So while any view on the branch has never landed a refresh, the maintainer re-runs a cut that has already passed, and it stops as soon as that first refresh lands. Once every view has landed, nothing is recomputed while the watermark stays still. Its queue is ordered by staleness: never-refreshed views first, then by `W − L_v` **in bytes**, then by name. The order is total, so a test can check it. `ARGONDB_VIEWS_CONCURRENCY` (default 1) limits how many refreshes run at once. **Under per-tenant compute (`[compute] per_tenant = true`) a tenant's views are maintained by the tenant's own tail.** That tail runs this same maintainer for its database's views and cuts them at the tenant's own coordinate. The box maintainer skips a tenant database by name, one line per pass, and never crosses the wall [D-2026-10-03-f3cf87]. **It makes three promises: it holds no ring, it puts no backpressure on the tail, and it never blocks a cut.** A maintainer that falls behind simply targets a later cut. Nothing is skipped; the window `(L_v, L_1]` just grows, and the changelog holds it for the pin window. If it falls far enough behind, the views are behind, the reader is told so, and reads are still served. Two breakers sit around it: - **Each view has its own breaker, and it counts a streak, not a total.** `ARGONDB_VIEWS_BREAKER_FAILURES` (default 3) failures in a row mark that one view `FAILED ()`. It stops being scheduled, it keeps serving at its own `L_v`, and the rewrite rule stops using it. One success clears the streak. - **The maintainer as a whole has the ordinary restart policy:** backoff from 2 seconds to 30 seconds, and the breaker opens at 6 restarts in 120 seconds. A tripped maintainer means views stop advancing, and reads keep being served at `L_v`, with the reader told how far behind. It never turns into a refused read. **It has its own memory pool.** `ARGONDB_VIEWS_MEMORY_MB` bounds a pool that spills to disk under `ARGONDB_VIEWS_SPILL_DIR`. Both are read at boot, so a bad value stops the process from starting rather than failing the first refresh. The box's memory reserve is split evenly, and this pool is a fourth claimant on it. That has a cost, and here it is: the read plane's own share fell from 3,641 MB to 2,730 MB on a 32 GB box, and from 142 MB to 106 MB at the 1,280 MB floor. #### 2.4 One refresh = one snapshot, stamped in the same commit A refresh is the unit of work that brings a view forward. Each one produces exactly one new snapshot of the view, and that snapshot says what point it is current to. You never see a half-applied refresh, and you never have to guess how fresh a view is. A refresh pins at `L_1 = W`, plans the view's own definition text, computes using the views memory pool, and lands the result through `argondb_transform::land::land_view_lake`, which is the committer's `LakeWriter` and nothing else. The snapshot carries its point in time **in the same commit as the rows**, as these properties: - `Argon.view-current-to-lsn = L_1`; - `Argon.view-definition-sha256`; - `Argon.view-class`; - `Argon.view-inputs` — each input, with the `W` it was read at; - `Argon.run-token` — so that a retry after `state-unknown` finds its own commit. **A view never carries `Argon.watermark-lsn`, `Argon.replay-from-lsn` or `Argon.changelog-through-lsn`.** Those are claims about how much of the write-ahead log a table covers, and only the committer may make them. That is the "one timeline, two clocks" line in the picture. The log position (LSN) is the one timeline every piece of ArgonDB data sits on. An Iceberg snapshot id or `committed-at` time is a clock that belongs to the storage, not to the data. **A refresh killed right after it commits is left fully present, never half-applied.** Because the coordinate is written in the same commit as the rows, there is nothing left over for a crash to leave half-done. If the maintainer is killed after that commit lands, the view is already at its new coordinate; replaying the dead refresh's own attempt lands nothing a second time, and the next refresh simply continues from where the committed one left off. This is proven by killing the maintainer at that exact instant and checking the rows against an oracle built from the changes alone. The proof's own limit: it stands in for a durable catalog pointer with an in-memory one reopened from the newest committed file, not the real catalog server. A Postgres-landed view has no such gap to prove at all, because its rows and its state row commit in one transaction together — but a kill in the middle of that transaction is not yet proven on a running box (§5). #### 2.5 A read pinned at `L` — the rewrite rule The rewrite rule is how an ordinary query gets answered from a view without the query naming it. When a read's plan contains a piece that a view already holds, the rule can swap that piece for the view, so you get the speed of the view without rewriting your queries. The rule is installed by `pin()`, the function that fixes a read's point in time, so **every** caller of the read plane gets it: the lake scan on the analyst compute, the MCP `query` tool, `argondb-query` on the command line, and the transform runner's own compute. It is installed **first** among the optimizer's rules, not last. DataFusion's own filter and projection pushdown rewrite the very sub-plans the rule looks for, and a filter pushed under a table's alias would be invisible to a later match and dropped. When the rule matches, it replaces the sub-plan with a node that holds the choice. The choice is made at match time — a matched view is by construction a subset of the base scan — so nothing plans the original query twice. The rule's answer is one of four: - `Exact` — the view is exactly current; - `Compensated { from, plan }` — the view plus the changes since it; - `Disclosed { behind, age_secs }` — the view as it is, with the reader told how far behind; - `Never(reason)` — do not use the view. `Never` is not an error: the original plan runs. #### 2.6 The keyed projection A keyed projection is the simplest kind of view: some columns and rows of one table, filtered, with the table's key carried through. It is the cheapest to keep current, because each change to the table touches at most one row of the view. It is a select-project-filter over **one** table, with that table's key passed through unchanged or declared by you. A refresh applies the filter and projection to the change rows: `I` and `U` rows are upserted by key, and `D` rows are deleted by key. **It is exact, its cost grows with the number of changes, and it needs no old rows.** There is an insert-only form: the same view, on a window whose changes are all inserts, which can simply append. **This is checked on every refresh, never assumed.** The classifier never assigns the insert-only form at `create`; it is chosen at refresh time, when the window turns out to hold only inserts. The picture's caption "refuse today (oracle gate RED)" records the status of update and delete windows as measured on 2026-09-19. §6.1 has what has been fixed since, what remains, and the workaround. #### 2.7 The algebraic aggregate An algebraic aggregate is a `GROUP BY` view whose totals can be adjusted by adding and subtracting, instead of being recounted. This is what lets a running total over a large table stay current at the cost of the changes, not the table. It is a `GROUP BY g` with `count`, `sum`, `avg` (kept as the pair `sum` and `count`), `min` or `max`. For each group, a refresh adds the new value for an `I`, subtracts the old value and adds the new one for a `U`, and subtracts the old value for a `D`. A group whose count reaches zero is deleted. It **needs the old rows** for `U` and `D` changes. This is the one place a refresh reads base data beyond the changes themselves. **An aggregate with `min` or `max`** needs one more step. When a `D` or `U` removed a group's current smallest or largest value, the new extreme cannot be worked out by subtraction. So the affected groups are recomputed from the base table at `L_1` (`WHERE g IN (…)`), and the other groups are updated incrementally as usual. The view is classed this way at `create` even though the recompute is only triggered on some refreshes, because the maintainer has to carry the affected-group step for that view at all. An aggregate with **no** `GROUP BY` is keyed on the empty tuple. It has one row, and it works. **A view's key names its output columns, and an alias is a supported key.** A view that selects `s.txn_month AS month` and is keyed on `(upc, month)` works: the maintainer traces each key column back through the view's own query to the `GROUP BY` term it came from, and reads every set of groups through that same projection. It never looks for a column called `month` in the base table. #### 2.8 The join view, with a known changed side A join view joins a fact table to a dimension table on a foreign key, many fact rows to one dimension row — for example, invoice lines joined to customers. It can have a keyed projection or an algebraic aggregate on top. ArgonDB can keep it current from the changes as long as it knows which side is the fact and which is the dimension. The textbook way to compute the change to such a view is in two terms: the changed fact rows joined to the dimension at `L_1`, plus the fact rows at `L_0` joined to the changed dimension rows. As built (`view_maintainer::refresh_join`), there is **one term per input**. Each input in turn is bound to its own changed rows (collapsed to their new images) while every other input is read at `L_1`. The view's own definition text is planned, unchanged, over each binding, and the terms are combined as a set. **Every term is read at `L_1`.** This departs from the textbook form, which reads the fact side at `L_0`, and it departs in the only safe direction. An upsert must land the fact row as it is at `L_1`; reading it at `L_0` would silently undo a fact change the first term just computed. It is also what makes the two terms produce identical rows for a key both of them touched, so `UNION` (never `UNION ALL`) removes the duplicates. The rows to delete come from the old side, not from the deletion markers. A deleted dimension row leaves no deletion marker on the fact table, and its fact rows still have to leave the view. **The picture's caption on this box is out of date.** It says the join view is built, but that a real foreign key still makes it a recompute view when it is created on a box. That reading turned out to be a flaw in the test, not in the product, and the join view is now proven when created through the command line on a box. A different defect, still unfixed, is in the SQL function's own read of the foreign keys. §6.2 has both. The picture has not been redrawn; treat its caption as replaced by §6.2. #### 2.9 The recompute view A recompute view is any view ArgonDB cannot bring forward from the changes alone. Rather than refuse such a view, ArgonDB runs the whole query again and swaps in the result. It is always correct, and it costs as much as reading the inputs. It covers everything else: - `DISTINCT` on columns that are not the key; - window functions; - aggregates that cannot be adjusted by adding and subtracting (`percentile`, `string_agg`, `array_agg`); - outer joins; - semi-joins and anti-joins as subqueries in `WHERE`; - self-joins; - `HAVING` over non-algebraic terms; - projections with no key; - **any view with an input that has no changelog.** The whole SELECT is run at `L_1` and landed as one atomic swap. The cost grows with the size of the inputs and does not depend on how much changed. **A recompute view is capped in how often it refreshes.** Recompute views are allowed, and the limit was set after the full-replace write was measured at 776,000 rows per second. `create` still prints the classifier's cost estimate. It also now works out a `min_refresh_interval` from that estimate: a recompute view refreshes no more often than the smallest whole number of cut intervals in which its estimated cost is at most `ARGONDB_VIEWS_C4_COST_SHARE` percent (default 10) of one interval. `argon.views()` shows the interval it chose. Nothing refuses a refresh for being expensive. The cap limits how *often* a recompute view refreshes, never *whether* it may run (§3.6, §4.4). **The classifier errs toward recompute.** A shape it does not recognise becomes a recompute view. That is always correct, though sometimes expensive; the opposite mistake would put wrong rows in a table people read. #### 2.10 The lake landing — `dv_.` The lake landing is the default place a view's rows live: an Iceberg table in the lake, read the same way as every other lake table. Choose it unless you need the view inside PostgreSQL (§2.11). A lake-landed view is a `dv_*` Iceberg table written through the `argondb-committer` crate's `LakeWriter`. The transform runner lands through the same code, so there is only one Iceberg writer implementation. That is not the same as one writer process. The rule is one writer per table, decided by who owns the table: - the transform runner writes the `dv_` tables it owns (transform outputs); - the view maintainer writes the `dv_` tables it owns (views); - neither of them writes `br_`; - the committer is the only writer of `br_`. Each refresh uses one of three write verbs: - `upsert`, for keyed projections, aggregates and joins: one `row_delta` snapshot, with equality deletes on the view's key plus the new rows; - `append`, for the insert-only keyed projection; - `replace`, for recompute views: one `rewrite_files` snapshot, and readers see either the old table or the new one. The maintainer does not choose the verb. `land::refresh_mode(class, first_refresh, landing)` works it out from the class. **The first refresh of a lake-landed view is always an `append`** into the empty table that `create` made. A reader tells a view from a transform output by its properties. A view carries `Argon.view-current-to-lsn`. A transform output carries `Argon.origin` and `Argon.committed-at`, and no log position. You read a lake-landed view through the foreign table `lake_.`, mounted behind the same tenant wall as every other lake table. **A tool outside ArgonDB entirely sees the same table and the same coordinate.** A view's Iceberg table is not ArgonDB-private: DuckDB's own `iceberg` extension can read it like any other Iceberg table, and the snapshot it reads carries the view's `Argon.view-current-to-lsn` stamp and never a base table's `Argon.watermark-lsn` — a base-table-only property is refused rather than written onto a view snapshot. This is proven from the outside reader's own point of view: the row count matches the maintainer's own refresh line, the coordinate matches what `argondb view list` shows for the same snapshot, and the watermark key is simply absent. §4.6 has the DuckDB SQL. #### 2.11 The PostgreSQL landing — `derived.` on the primary The PostgreSQL landing puts a view's rows in a real PostgreSQL table on the primary database, `derived.`. Choose it when you need to look a view up by key at transaction speed, join it with live tables in one PostgreSQL query, or protect it with row-level security and grants. **It is built, and proven against a real PostgreSQL (§5).** The one defect on join views — a join view declared with this landing used to land in the lake instead — is now fixed: the join refresh honours the declared landing (§9). You choose the landing per view at `create`. **A view has exactly one landing.** If you want both, create a second view with the other landing, so that each has one writer and one state row. The rows are written through the landing transaction as the `argon_transform` role, with a `PRIMARY KEY` on the view's key and any secondary index you declare. Each refresh is **one transaction**: - for keyed projections, aggregates and joins: `INSERT … ON CONFLICT (key) DO UPDATE`, plus `DELETE … WHERE key IN (…)`, plus the `argon.view_state` row, plus `COMMIT`; - for recompute views: `TRUNCATE` plus `COPY`. A row cap of 10,000,000 (`DEFAULT_ROW_CAP`) applies to the size of the whole view for a recompute view, and to the size of the change for the other classes. What it gives you: index lookups at transaction-processing speed, joins with live tables in one PostgreSQL query, row-level security and grants, and — since it is marked for publishing by default — a lake copy that arrives through the tail like any other table. That copy is a `br_*` table with a real `Argon.watermark-lsn`, because its rows went through the write-ahead log. It is the one place a view's content properly appears on the committer's timeline. What it costs: heap, write-ahead log, safekeepers and pageserver for every refreshed row, **on the primary**. This is the one exception to "the analytical side never loads the transactional side", and it is bounded: the maintainer never puts load on the primary except through that one transaction. #### 2.12 The bottom band — a view over a distributor's own rows and a landed reference table The bottom band shows the case people ask about most: a view that seems to combine two companies' data. It is not special. It is a view over two tables that both sit in one tenant's own database. `public.invoice_lines` is written by the distributor's own people all day, and reaches the distributor's lake through the tail with a changelog. `derived.mfr_price_list` is a manufacturer's price list that the **hub** landed into this same database earlier. It is read-only, it is local, and it is marked for publishing, so it is in this distributor's lake copy too. A view over both is therefore an **ordinary join of two local tables** in one database. It is classified, refreshed, stamped and served exactly like any other view, and nothing about it is handled as a special case. The sharing model put the rows there; the view model reads them locally. A table this database only holds because another database **granted** it a mount is a different thing, and a view over it is refused (§2.13): a landed copy is this database's own table, while a granted table still belongs to whoever granted it. #### 2.13 The wall — "no query reaches across" The wall between tenants applies to views exactly as it applies to every other read. Another party's area cannot be reached, and rows from two parties only meet after the hub has copied them into one database. This is not a new check. A view's database is its inputs' database, worked out by the same check the pin uses, and that check counts only the database's own tables. The lake mount is limited to the database that is asking, plus any table another company has explicitly granted it, read-only (chapter 6); a granted table can be queried, but it is not one of this database's own inputs for a view. Every scan ticket carries the database. And `argon.create_view` checks it from the compute exactly as a scan ticket does. A join across databases is refused by name (`cross-database-join`). A view of another database is refused with the same words as a table of another database. **A view over a table this database only holds by mount grant is refused by name, `view-input-granted`.** A grant lets this database's mount read the grantor's table, but a lake-landed view is scoped to the OWNER of its inputs, so the view would be created and then be unreadable by the very database that created it. The door checks this before it pins or enrolls anything. Whether a grantee may ever build a view over a granted table is not decided — it is an open design question — so today the door refuses rather than record something nobody has ruled on (§4.9, §6.17). **A mounted table is never a view input.** This was ruled on 2026-09-27. A table another database granted you a mount of is visible in your database — the tail lists it so catalog sync can mount what a scan serves — but a mount is a live read with no integrity guarantee: PostgreSQL allows no foreign key onto a foreign table, and the lake copy of a view over it would belong to the owner's scope, not yours. So the create door refuses it by name, before it takes a pin: ``` mounted-input-refused: dist__public__sales is a mount grant into this database, a live read with no integrity; land it with a publication and read the landed copy ``` The remedy is the publication landing: the owner publishes the table with your database as a grantee, you subscribe, and the landed `derived.` is your own table — a view over it is admitted like any other (chapter 6, "One mechanism, both directions"). The refusal is one body shared by all four doors, fed the asking database's mount grants by the tail; the operator doors (CLI, MCP) are no database's client and have no mounts. The `all_tenants.*` views are a different thing entirely. They are the read-only reader identity's union over every walled database, used for the hub's census. They are not a tenant's views, and a tenant can never reach them. --- ### 3. How it works This part walks through the machinery end to end. It starts with the control path: how you define a view, what checks it before it is recorded, and how the maintainer decides when to refresh it. Then it follows the data: what one refresh does, how views built on other views stay consistent, how an ordinary query gets answered from a view, and what all of this was measured to cost. #### 3.1 The definition surface, and why it is a function call You define a view by calling a SQL function, not by writing `CREATE MATERIALIZED VIEW`. This section explains why: the place you define views from cannot run DDL, and a PostgreSQL materialized view would be the wrong kind of object anyway. The place you work from on the analytical side is the **analyst compute**. It is a read-only compute on Neon storage. It sees the normal PostgreSQL catalog, and it sees the published lake tables through the `argondb_lake` foreign data wrapper, mounted as `lake_.
`. Two facts about it decide the shape of the definition surface before any preference does. **A read-only compute refuses DDL.** `CREATE MATERIALIZED VIEW` on a hot standby fails with PostgreSQL's own message, `cannot execute CREATE MATERIALIZED VIEW in a read-only transaction`. So an event trigger on that statement would have nothing to fire on, in the one place definitions come from. **A PostgreSQL materialized view is the wrong object.** PostgreSQL would own its refresh. It cannot be published: ArgonDB refuses it by name, with "a materialized view cannot be the unit of publication". And its definition would be parsed by PostgreSQL's planner against foreign tables whose meaning actually lives in the read plane. So a definition is captured by three SQL functions. The `argondb_lake` extension ships them. You can call them on the analyst compute, and on the primary, where the same extension is installed: - `argon.create_view(name text, sql text, options jsonb) → text` - `argon.drop_view(name text, cascade boolean DEFAULT false) → text` - `argon.views → TABLE(name, landing, class, key_columns, current_to_lsn, behind, inputs, last_refresh, state)` The `options` parameter is `jsonb`, not `json`, and the difference matters. If you pass `json`, PostgreSQL answers `function argon.create_view(text, text, jsonb) does not exist`, because there is no implicit cast between the two types. Add `::jsonb` to your literal. **None of the three writes anything locally.** That is what lets them run on a read-only compute. Each one sends a Flight action to the read plane, the same way `argondb_lake_sync` and `argondb_lake_freshness` already do. The request travels under the key `argondb.view.v1`, and it uses a different kind of Flight call from a table scan. The two kinds are kept apart on purpose: a scan request sent to the view entry point is refused with `WrongSurface`, naming both kinds of call, and a view request sent where a scan is expected is refused the same way. **The `argon` schema must already exist; the extension never creates it.** Every ArgonDB box creates that schema before any extension is installed. If it is missing, the extension's setup step refuses loudly and names what should have created it. A final setup step then moves the three functions into the schema with `ALTER FUNCTION … SET SCHEMA`. #### 3.2 One body, four callers There are four ways to create or drop a view, and all four run the same code. That code, the view door, is what runs behind: - the Flight action itself; - `argondb view create|drop|list` on the command line; - the `create_view` tool on the `argondb-write` MCP mount; - `argon.create_view` in SQL, which reaches the door through the Flight action. The command-line version is only an adapter. It passes your request along and makes no choices of its own. So a view created from any of the four is checked, classified and recorded in exactly the same way. #### 3.3 What `create` does, in order, and what each step can refuse `create` runs seven steps, in a fixed order. Each can refuse, and the order is chosen so the cheapest mistakes are caught first. Note that `create` does not refresh the view; it only checks the definition and records it. 1. **The name, checked against the shared naming rules**, before anything is parsed. The rule is `[a-z][a-z0-9_]*`, at most 63 bytes, no `_argon_`, `br_` or `dv_` prefix, and no `__` anywhere inside. If you misspelled the name, you find out without waiting for a query plan. The refusal names the same rule a transform's target table has to meet. 2. **The qualifier rewrite.** You may write a table as `lake_main.orders` or as plain `orders`. The door rewrites the mount qualifier by parsing the statement, not by editing text. It parses with the same `sqlparser` dialect DataFusion uses, inspects the qualifier on every table reference, and prints the statement back out. `lake_main` inside a string literal, or used as a column alias, is not a qualifier, and a text search could not tell the difference. The test suite checks these cases as negative controls. 3. **The granted-input check.** Before anything is pinned or enrolled, every table the definition names is checked against the read plane's own record of which database each table actually belongs to. A table your database only reaches through a mount grant from another database is refused as `view-input-granted` (§2.13) — before the pin, because the pin does not serve the grantor's own warehouse and you would otherwise be told a confusing `plan-failed` about a table your own `EXPLAIN` just read; and before enrollment, which would otherwise start tracking changes on the GRANTOR's changelog rather than yours. This check only runs when the request names its own database, which the SQL door always does; the CLI and the MCP tool do not, and skip it. 4. **Validation in both SQL dialects.** First in PostgreSQL, before the call leaves your compute: `argon.create_view` runs `EXPLAIN` on the definition in your own session. That proves the text parses as PostgreSQL, that every table it names is a `lake_` foreign table your database is allowed to read, and that PostgreSQL knows the types of the output columns. This step connects to nothing; a plain `EXPLAIN` never opens a Flight scan. Then in DataFusion, in the read plane, where the classifier walks the plan. A definition that passes PostgreSQL but fails DataFusion is refused as `plan-failed`, naming the construct. Its remedy names the usual causes: `to_char`, `current_setting`, `txid_current`, `nextval`. **The command-line `create` runs only the DataFusion half, and says so on the first line of its output.** 5. **Classify.** Every refusal is shown to you word for word (§3.5 describes the classifier; §3.4 lists the refusals). 6. **The door's own two refusals.** The classifier deliberately does not raise these, because they depend on the options you passed. One is the self-join with no equality predicate. The other is an incremental view with no key. The no-key check is a single rule that the door applies before it records the view, and that the landing step applies again on every landing. 7. **Record.** The view's table, `dv_.`, is created empty, with the definition stored on it. Recording the view as a real table does two jobs. It means a definition can only name a view that already exists, which is what makes cycles impossible (§3.8). And it is how `drop` can hand the definition back to you. **A view that has been declared carries no coordinate.** `argon.views()` prints `NEVER REFRESHED` for it, never `0/0`. A view that has never refreshed is not current to any point in the log, and a zero would read as "behind by the whole log". #### 3.4 The refusal family When ArgonDB will not accept a view definition, it tells you why with one of sixteen named refusals. The set is closed: there are exactly sixteen, a unit test checks that count against the code, and a new kind cannot even compile until it is added to the list. Every refusal carries four things: its kind, the view, the construct that caused it, and a remedy. It prints like this: ``` refused: — view ``: . Remedy: ``` | kind | what causes it | |---|---| | `multi-statement` | the definition is not exactly one statement | | `not-a-select` | DDL, DML, `COPY`, `EXPLAIN`, `SET` | | `name-grammar` | the view name breaks the lake naming rules | | `non-deterministic` | any function DataFusion does not mark as immutable, such as `now()` or `random()`. **Refused for every class, recompute included** | | `input-not-published` | an unpublished table, or a table of another database | | `input-argon-internal` | an input named `_argon_*` | | `input-changelog` | a table's `__changes` change log named directly | | `input-transform-output` | a `dv_*` transform output. Its content is not a function of a log position, so a view over it would have no coordinate to carry | | `self-cycle` | the view's own name appears somewhere among its inputs, directly or through other views; the path, such as `v → a → b → v`, is printed | | `view-depth-exceeded` | the longest path from the view down to a base table is longer than `ARGONDB_VIEWS_MAX_DEPTH`; the path is printed | | `view-input-postgres-landed` | a view landed in PostgreSQL used as the input to another view (it runs on a different clock); the remedy names the lake-landed alternative | | `cross-database-join` | inputs from two different databases | | `top-level-order-or-limit` | `ORDER BY` or `LIMIT` at the top level | | `unrepresentable-type` | an unbounded PostgreSQL `numeric` under `sum` and similar aggregates | | `unpredicated-self-join` | the largest table joined to itself with no equality predicate; raised by the create step, not by the classifier | | `plan-failed` | a single SELECT that DataFusion still cannot plan | Two things about this list matter to you. It comes from what a view has to be, not from a history of bugs. And every refusal has a matching positive test beside it in the unit tests. So if a refusal ever starts firing on a definition it should accept, a test fails; you do not find out from a support ticket. **A table's `__changes` change log can be read in SQL, but it cannot be written and it cannot be a view's input.** A company's own SQL session (the `query` tool under that company's own MCP token) reads the change logs of its own tables by their flat lake name, which `argondb lake address --changes --field lake-table ..
` prints. The columns are the change log's own: the table's columns plus `_argon_lsn`, `_argon_op`, `_argon_seq` and `_argon_ts`. Another company's change log is refused by name under the tenant wall. A write to a change log is refused by name, because the change log is read-only. `view create` over one is still refused as `input-changelog`: ```sql -- under acme's own MCP token: every change to acme's orders since a position SELECT _argon_lsn, _argon_op, id, status FROM acme__public__orders__changes WHERE _argon_lsn > 27000000 ORDER BY _argon_lsn, _argon_seq; ``` **One more refusal sits outside this closed set, on purpose:** `view-input-granted` (§2.13, §3.3). It is not a fact about what a view must be — it waits on an open ruling — so it is raised by the create door itself, not by the classifier, and a test keeps it out of the classifier's own closed list. Like every door refusal it behaves as a refusal, not a crash: it prints as `: `, and nothing is recorded — the view does not show up in `argon.views()` afterward (§6.17). #### 3.5 The classifier The classifier looks at a view's definition and decides which kind of view it is: a keyed projection, an algebraic aggregate, a min/max aggregate, a join with a known changed side, or a recompute view. That answer decides how the view will be refreshed and what it will cost. You can ask for it without creating anything, using `argondb view classify`. At its core the classifier is a pure function of the query plan. It uses no engine, no catalog and no I/O. Anything it cannot read off the plan (Is this table published? Is its change log enrolled? Is this equi-join on a foreign key?) it asks of a set of facts supplied by the caller. Two things follow. The tenant wall stays with the caller, because the caller supplies the facts. And every rule can be tested both ways in fast unit tests rather than on a booted box. **It classifies the plan before optimisation, on purpose.** DataFusion's expression-simplifying pass folds `now()` into a fixed timestamp. If the classifier looked at the optimised plan, the very non-determinism it must refuse would already be gone, and the refusal would never fire. A test checks this. The classifier's answer carries the view, the class, the key, the inputs, a cost estimate, and notes. Three of these are worth understanding: - **The key** is one of five kinds: a key you declared; a key inherited from a source table (naming that table and its key columns); the `GROUP BY` columns; the empty tuple, for an aggregate with no `GROUP BY` (one row); or no key, with the reason. A view's identity is declared or inherited. It is **never guessed** from the output columns. "No key" does not refuse the view. It forces it to be a recompute view. - **The inputs.** For each table, the classifier records the name as you see it; its role in a join (sole input, fact, dimension, or unresolved); whether its change log is enrolled; whether the input is itself a view (there is exactly one place that answers this); and where its changes will come from. The rule in §2.2 that turns an otherwise incremental view into a recompute view reads only that last field. So a base table with no change log and a keyless recompute view used as an input cause the same demotion through the same path, just with different stated causes. - **The notes** are where the classifier explains, in your terms, every place it chose the safe, more expensive answer. When you are iterating on a definition, read the notes to see which construct cost you the class. The cost estimate describes the shape of the work. None of it pretends to be a measured number. It says: the dominant cost term; whether every refresh scans all inputs; the old-image lookups needed; whether affected groups are recomputed; whether a change to a dimension forces an unpruned scan of the fact table (and if so, the remedy); whether the plan is a cartesian product; and the full list of cost terms. The unpruned fact scan is the common case, not the rare one. By default, the lake's compaction clusters a table by its primary key, not by a foreign key, so a dimension change usually means scanning the whole fact table. **Foreign keys are an input in three places and a constraint in none.** A foreign key tells the classifier which side of a join is the fact. It lets a view inherit its identity through the fact table's primary key. And referential integrity itself plays no part: a view never refuses a row for a foreign-key reason, ordinary inner-join rules apply, and a row whose parent is missing at `L` is simply not joined. Every door — the analyst compute's SQL function, the CLI and the MCP tool — gets its foreign keys from the same one place: the tail's own schema reader, which already knows every published table's primary key and now knows its foreign keys too. On the command line you may still add `--fk child.col=parent.col` to restate an edge, or to add one the schema does not declare. With no foreign key at all, a join is classified as a recompute view and the demotion is printed. It is never lost silently (§6.2). #### 3.6 The trigger, the queue, and what backpressure means here This section explains when a view refreshes. In short: when the lake's watermark moves, not on a timer. It also explains what happens when the maintainer cannot keep up. The trigger is **the watermark advancing, not the clock**. The view refreshes up to the lake watermark `W`, not up to the live head `H` of the log. That is what makes the change log the source of its changes, and it makes a refresh repeatable: running it again to the same `L_1` reads the same rows. The tail cuts to the lake about every 30 seconds when load is light, and more often under heavy writes, as data volume forces it. So a view is at most one cut plus its own refresh time behind the head. For a **recompute view**, a minimum refresh interval (`min_refresh_interval`) skips cuts when a refresh would cost more than the changes since the last one are worth. Since 2026-09-20 that interval is derived from the cost model, instead of defaulting to "every cut" (§2.9, §5). The incremental kinds (keyed projection, algebraic and min/max aggregate, join) have no such option, and refuse it if you declare one. **The interval is only a floor; it never causes a refresh.** A view whose inputs have not changed is not refreshed at all, and nothing periodically recomputes an unchanged view. The one clock setting is `max_staleness`. It is a bound that is disclosed to readers, not a refresh trigger. **Falling behind is a visible state, not a slowdown.** A view that cannot be scheduled before the next cut simply aims at a later cut. If its window reaches back past the change log's retention, or across a recorded gap, the window no longer counts as a set of changes: the first cut that meets it reports the view **BEHIND** once, naming the input, the range and the cause, and the next cut recomputes it past the gap, after which incremental maintenance picks back up. That BEHIND report never charges the per-view breaker (§4.10). The "already reported" flag lives only in memory, so restarting the maintainer reports the same BEHIND once more before it recomputes. None of this ever slows down the tail. **A first refresh never aims below the point where its inputs' change logs were enrolled.** Its coordinate is fixed at or after the latest of those points (§2.2). So if one input enrolled its change log late, the first cut a new view is scheduled at may not be the very next one. **A view that is already refreshing, whose input gets enrolled later, goes BEHIND rather than FAILED.** This is the v1.5.6 fix: before it, a recompute view could trip its own breaker because of somebody ELSE's `create` — a later `create` over the same table can raise that table's enrolment floor above the next cut, and the view's next refresh used to be charged as a failure for landing below that floor. Now that case is recognised on its own: the view logs a BEHIND note naming the input and the point it was enrolled at, keeps its current coordinate, and picks its next refresh at or after that point — with nothing charged to its breaker. Proven at unit level both ways; the box gate covers the BEHIND-not-FAILED direction. The maintainer prints one line when it starts. That line is built from the settings actually in force, not written separately beside them. The list of classes in it comes from the code's own table of which classes this build maintains, so the line changes in the same edit that changes the list: ``` view maintainer: concurrency= (ARGONDB_VIEWS_CONCURRENCY) breaker= consecutive failure(s) \ (ARGONDB_VIEWS_BREAKER_FAILURES) poll=s (ARGONDB_VIEWS_POLL_SECS) window= key(s) \ (ARGONDB_VIEWS_WINDOW_KEYS) pass deadline=s (ARGONDB_VIEWS_PASS_DEADLINE_SECS) ring step=s \ (ARGONDB_VIEWS_RING_STEP_SECS) recompute bound=s|off (ARGONDB_VIEWS_RECOMPUTE_SECS) idle line \ every s (ARGONDB_VIEWS_IDLE_LOG_SECS) — cut-driven; \ classes ; ``` Every setting is checked strictly at startup. Unset or empty means the default. A plain positive integer means that number (one exception: `ARGONDB_VIEWS_RECOMPUTE_SECS=0` turns that bound off). Anything else stops the process with a message that names the variable, what it read, and what it expected. In the code's own words, "a view maintainer that silently defaulted this would run under a limit nobody chose". #### 3.7 One refresh, step by step A refresh takes the changes since the view's last coordinate, works out their effect on the view, and lands the result as one snapshot. The picture shows the two places changes can come from, the arithmetic, and the coordinate stamp. The six steps below are what actually runs. ![4b — one refresh: two delta sources, the arithmetic, one snapshot, one coordinate](diagrams/argondb-and-the-erp/04b-refresh-delta-sources-and-the-coordinate-stamp.svg) 1. **The window.** For each input, the window is `(L_0, L_1]`. It is applied as a plain integer filter on `_argon_lsn`, and read from the change log. For a view whose input is another view, it is read from that upstream view's snapshots instead. 2. **Collapse to one change per key.** The window is a sequence of `(lsn, seq, op, key, image)` for each table. The arithmetic runs **per key, in `(lsn, seq)` order**, which is the same total order the read plane uses when it merges changes at read time. That merge calls the same function the refresh uses, rather than repeating its logic, so the two cannot drift apart. 3. **Fetch the old images, using the window before it is collapsed.** The keys of all `U` and `D` rows are gathered and fetched in one keyed read at `L_0`. A key that was deleted and re-inserted inside one window collapses to a plain `I`, yet it did exist at `L_0`. Collapsing first would skip its `−f(old)` term. This was a real defect, found when the pieces were first combined. The fix is that the old-image keys are read from the uncollapsed window. 4. **The rows to delete are not just the tombstones.** A row that an UPDATE moved out of the view's `WHERE` clause has no tombstone and no new image in the view. Left alone, it would stay in the view for ever. So a refresh **deletes every key the window touched, by equality, and adds back the survivors, in the same snapshot**. That needed a landing call that can delete a key with no new image. A landing mode that cannot do this is refused by name, `delete-keys-unsupported-mode`, rather than dropping those deletes. 5. **The window limit is soft, and applies at commit boundaries.** `ARGONDB_VIEWS_WINDOW_KEYS` (default 1,000,000) splits a large window, but only between commits. If the first commit in the window is by itself over the limit, that commit is taken whole, and the refresh says so. A `WARN` line names the commit, its row count, the limit it went over, and both remedies: raise the setting, or recreate the view as a recompute view. The refresh's own summary line repeats the warning. This is the settled behaviour, not a stopgap (§6.15). The alternative, a hard limit that splits a commit, would publish a half-applied transaction at a coordinate no cut ever had, and the consistency contract does not allow that. The earlier behaviour, refusing the oversized commit, was measured to act as a poison pill: one large transaction froze the view for ever. 6. **Land.** One snapshot, one coordinate, one commit (§2.4). #### 3.8 Chains — a view whose input is a view You can build a view on top of another view in the same database, for example a cleaned-up line table and then a summary over it. This section explains how ArgonDB keeps such a chain consistent: no view ever runs ahead of what it reads, members refresh in dependency order, and a stalled member shows up as a named state downstream. ![4d — a chain at one cut: topological order, one L_1, and what a stalled member does downstream](diagrams/argondb-and-the-erp/04d-a-chain-at-one-cut.svg) A view may read another view **of the same database**. You name it exactly as you name a base table: `lake_.`. The views form a graph with no cycles, and **its edges are recorded**. `Argon.view-inputs` lists input views as well as input tables. The maintainer keeps the edges for each database in memory and rebuilds them from those records when it starts; it keeps no progress store of its own. This is where views differ from transforms. A transform may read another transform's output, but that link is not tracked, because a transform has no dependents to protect. A view does. **Cycles.** You cannot create a cycle, and that follows from how views are built, not from a check. A definition can only name a view that already exists. A definition cannot be changed afterwards. And dropping a view that has dependents is refused or cascades. So every edge points from a newer view to an older one. There is still a cycle check, as a safety belt. It exists for `create_view('v', 'SELECT … FROM lake_main.v')`, which the planner would otherwise refuse as "unknown table `v`", a misleading message. It also covers the day a future in-place ALTER makes the "cannot be changed" argument stop holding. **Depth.** `ARGONDB_VIEWS_MAX_DEPTH` (default 8) refuses at `create`, by name, a view whose longest path down to a base table is longer than the limit, and prints the path. Refreshes along one path run one after another. So the deepest view is behind the head by one cut plus the sum of its ancestors' refresh times. Limiting depth is what keeps that delay bounded. There is **no limit on how many views one view reads, or on how many views read one view**. A view that joins `k` views is just a join, and the join's own cost model already prices it. A view with `n` dependents pays nothing extra beyond keeping its changes around for them. **Order.** At each cut, the maintainer refreshes views in dependency order. It recomputes that order each cut, over the views whose `L_v < W`, using a standard topological sort of the recorded edges. Every member that refreshes at that cut refreshes to the same `L_1 = W`. Views that do not depend on each other run in parallel, up to `ARGONDB_VIEWS_CONCURRENCY`. A dependent never starts before every one of its inputs has committed at `L_1`. **The guarantee.** For every edge from an upstream view `U` to a downstream view `D`, `L_v(D) ≤ L_v(U)`, always. A view is never ahead of any of its inputs. This is checked after every pass. When it is at risk, the dependent takes one of two states instead of a new coordinate. Both sentences are produced by one piece of code, so `argon.views()` and the per-view behind-line always print the same text: ``` BEHIND (upstream FAILED|INVALID at ) WAITING (upstream coalescing to ) ``` **Where a dependent's changes come from.** When an incremental upstream view refreshes, it lands one snapshot of added rows and deletes. Read as a set of changes (its new data files and its equality-delete files), that snapshot is a window of keyed change rows. A key that was deleted and has no new image is a `D`. A key with a new image is an `I` or a `U`. All rows of one refresh carry `_argon_lsn = L_1`, and `_argon_seq` is their position. So the dependent's per-key collapse runs unchanged over a sequence of upstream snapshots. **No `__changes` table is created.** A recompute view that has a key lands its recompute as a difference: it compares the new result with its own state at `L_v` by key, and lands only that difference as an upsert instead of a full replace. Everything above then works unchanged, and `argon.views()` prints its class as `C4 (diffed)`. A recompute view with no key has no identity to take a difference on. So its dependents are recompute views too, and the dependent's `create` says so, with the remedy: declare a key on the upstream view. One function in the code makes this call. It answers "from the change log", "from the upstream view's snapshots", or "none", with the cause. **Retention.** Snapshot expiry does not remove a view snapshot at coordinate `L` while any dependent's `L_v < L`. It also never keeps one past the pin window (48 hours). A dependent that has not caught up within that window is past retention. It is recomputed at its next cut, with the cause `upstream delta expired`, and the upstream view is unaffected. **The hold checks every dependent's coordinate, no matter which landing it uses.** The sweep that decides whether to hold an upstream's snapshots reads lake-landed dependents off their snapshots and Postgres-landed dependents off their own state table, through one merged view of the whole estate. Before this, the sweep only looked at the lake, so a Postgres-landed dependent always looked never-refreshed and held its upstream's snapshots forever; now it is held only for as long as it is genuinely behind. If the Postgres-landed state cannot be read at all, every Postgres-landed dependent counts as unknown and its upstream is held anyway for that pass — the safe direction — with a warning; a lake catalog that cannot be read holds nothing that pass, and says so. **Stated plainly, a limit today:** the live sweep applies no age limit to a hold, so a dependent that has simply stopped consuming — a tripped one, say — holds its upstream's snapshots indefinitely, past the pin window, with nothing today that expires them on its own (§6.18). **Drop and alter.** Dropping a view that has dependents is refused by name, `view-has-dependents`, listing every dependent, direct or indirect. With `cascade => true`, the drop goes in reverse dependency order: dependents first. So there is never a moment when a view exists but one of its inputs is gone. It returns the list of views it dropped, **each with its definition text**, so you can recreate the ones you meant to keep. There is no ALTER in this version: to change a view, drop it and create it again. Definitions cannot be changed in place, and that is exactly what makes the argument that cycles are impossible hold. #### 3.9 The rewrite rule, test by test The rewrite rule lets an ordinary query be answered from a view, without the query naming the view. Before it uses a view, it runs a short ladder of tests to prove the view holds everything the query needs. This section goes through those tests in order. ![4c — the rewrite rule's test ladder](diagrams/argondb-and-the-erp/04c-rewrite-rule-test-ladder.svg) The candidates are the views whose inputs are all inside the pin's scope and whose state is not `FAILED` or `INVALID`. For each candidate, the rule asks the following in order, and stops at the first "no": 1. **Same tables.** As built, the query sub-plan must read **exactly the same set of tables** as the view, once the view is expanded. The design allows a subset; the build is narrower. A query that joins one extra table is not matched at that point in the plan. Its inner parts are still tried, so `orders ⋈ customers ⋈ regions` can be served from a view over `orders ⋈ customers`, but only when the planner has already grouped that pair under one node. Self-joins are refused. 2. **Join conditions.** Every group of columns the view joins as equal must be contained in a group the query joins as equal. 3. **Filters.** The view's range and other predicates must each contain the query's. Whatever the query filters beyond the view becomes a **compensating filter** applied on top of the view. 4. **Columns.** The query's output expressions and the compensating filters must be computable from the view's columns. A predicate on a column the view does not include refuses the match. That includes a filter on a non-group column over an aggregate view. 5. **Aggregates.** This is a step the third-party view-matching crate studied for this design does not attempt. A query that groups more coarsely than the view (`g' ⊆ g`) is rewritten as `sum(sum)`, `sum(count)`, `min(min)`, `max(max)`. **`avg` is not rolled up** and is refused as `avg-not-rolled-up`. To roll it up, the view would have to carry both the `sum` and the `count` of that argument, and the classifier keeps `avg` as that pair only when the definition spells it out (§6.4). `DISTINCT` aggregates, aggregates with `FILTER`, and ordered aggregates are refused by name. A query that groups more finely than the view never matches. A query that does not aggregate never matches an aggregate view. 6. **The coordinate**, §3.10. Each step is its own function with its own table-driven tests. All of it was written by hand; nothing is linked in or vendored. Across a chain, the comparison uses the view's **expanded** definition. An input view that is a plain select-project-join is inlined. **An aggregate view used as an input is treated as an opaque table**, so a view built on an aggregate view only matches queries that name that aggregate view directly. Candidates are tried **deepest first**, and the first match wins. A match on a downstream view implies a match on each of its ancestors, and the deepest view is by construction the smallest table that still contains the answer. `EXPLAIN` prints the rewritten node like this: ``` ViewRewrite( depth= @ [+ compensation (, ]] [disclosed]) ``` #### 3.10 The coordinate a read gets — exact, compensated, disclosed, or nothing A query reads the data as of some point in the log, `L`. A view is current to its own point, `L_v`, which is at or before `L`. This section explains the four ways the rule can handle the gap: answer exactly, answer exactly by adding the missing changes, answer from the view and tell you how far behind it is, or not use the view at all. - **Exact.** `L_v = L`. Or `L_v < L`, but for every input the in-memory ring window `(w, L]` is empty and `w ≤ L_v`, where `w` is that input's watermark. Then there is nothing to add, so the answer is exact for every class. **The comparison is made against each input's own watermark for row changes, never against the pin's `L` or a lake watermark that was stamped with no change.** That is what lets an idle view (nothing changed in any input) serve exact answers while sitting well behind the moving head of the tail. When an input has moved, the refusal names the bytes between `L_v` and that input's own watermark, never the head. This rule fixed a real regression in a shipped build. Once the maintainer stopped refreshing idle views, the old version of this rule refused every one of them, as behind a head that had nothing to do with their inputs. §6.16 has the details. - **Exact by compensation.** `L_v < L`, the ring windows hold inserts only, no input has a landed row change after `L_v` (the same per-input watermark check as Exact), and the path can be compensated: the view at the top is a keyed projection (including its insert-only form) or an algebraic aggregate, and every view in between is a keyed projection. An aggregate in the middle of a chain is not composed. The rewrite is `view@L_v ∪ definition(Δ)`. It is planned at read time in a scratch session. In that session each base table is just its insert rows from the ring, and each upstream view is its own definition over those rows, re-aggregated for an aggregate view. A view refreshed at every cut has `L_v = W`, so the rows to add are exactly the ring's `(W, L]`, and nothing new has to be fetched. - **Disclosed.** Any other case that fits within the reader's bound. The answer comes from the view, with a per-view line naming `L_v` and the distance: `served from view at , bytes behind the pin coordinate `. If `L_v < w`, the view missed a cut. That case can only be disclosed, because the missing rows are in the lake, not in the ring. - **Never.** A view with no coordinate, a view ahead of the pin, or a view that is `FAILED` or `INVALID`. The query runs against the base tables as normal. This is never an error. **The reader's bound is the setting `argondb_lake.view_staleness`.** It takes `'0' | '' | 's' | 'any'`, and **the default is `'0'`**. The value is checked when a query reads it, not when you `SET` it, and that is deliberate. The extension framework (pgrx) offers no way to validate a setting at `SET` time, so this setting follows the same pattern as `argondb_lake.pin`: accept any string at `SET`, check it at the one place it is read, and fail loudly by name. So `SET argondb_lake.view_staleness = 'nonsense'` succeeds, and the next query that reads it is refused. The seconds form is measured against `Argon.committed-at`. **A bound you name is a line the answer may not cross; it is not just disclosed.** If you set `view_staleness = 0` and the only view available is one whose class cannot compensate, the query runs on the base tables. You do not get a disclosed view. This is the same rule the read plane applies whenever a caller names a bound, and it is why a default of `0` is safe: if you set nothing, you get an exact answer or no rewrite. **You can turn the rule off in three ways.** Each is recorded in the rule's log, not in the plan text. A rule that is switched off leaves the plan untouched, because a plan node added only to say "off" would get in the way of DataFusion's own optimisation passes. - Per session: `SET argondb_lake.view_rewrite = off`; the CLI flag `--no-view-rewrite`; or the MCP tool's `view_rewrite: false`. - Per view: the `use_in_rewrite` option at `create`, available on all three doors (§6.5). - Per deployment: `ARGONDB_VIEW_REWRITE=off`. The log lines are `fired `, `refused : `, `session-off`, `view-opted-out` and `deployment-off`. `argondb-query --explain-views` prints them. **Reading a view directly is not a rewrite.** `SELECT … FROM lake_main.daily_revenue` is a scan of a lake table the foreign data wrapper mounted. The scan request carries the session's pin `L`. The read plane then treats the view's `L_v` the way it treats a table's watermark `w`, and the same answers apply. At `L_v = L` it reads the view as is. At `L > L_v` it adds the missing changes, for the classes that support that. At `L < L_v` it reads an older view snapshot, if one is still retained. Otherwise it reads the newest snapshot below `L`, and discloses the gap. #### 3.11 What is measured These are the numbers behind the cost claims in this chapter: how fast views can be written, how fast inputs can be scanned, and where a keyed lookup stops paying off. They come from one cost-measurement run on 2026-09-19. Conditions: one DataFusion 53.1 session at default parallelism, on a 20-core i9-13900H with 91 GB of memory. The test data was written with the same writer settings the committer uses. Each value is the median of 2–3 runs after a warm-up. | quantity | value | why it matters to you | |---|---|---| | lake writer, `upsert` | 655 k rows/s at 100 k rows; 668 k at 1 M | writing a 190 k-row cut takes **~0.3 s** | | lake writer, `replace` | 776 k rows/s at 100 k; 768 k at 1 M | fully replacing a 1 M-row recompute view takes **~1.5 s** | | full scan + algebraic aggregate, 3 narrow columns | 272–407 M rows/s | the cheap half of an algebraic aggregate | | full scan + aggregate over a wide text column | 55–92 M rows/s | the expensive half | | one keyed point read, key present | 5.5 ms at 10 M rows; 4.5 ms at 100 M | **the cost does not grow with the table** | | where keyed lookup stops paying | **~100 keys** | below ~10 changed keys a keyed lookup is ~1,000× cheaper than a scan; from ~100 up, a full scan wins | The measurements corrected the cost model in three ways. They change what you should worry about. 1. **For any realistic ERP window, fetching old images for an algebraic aggregate costs one full scan of each input per cut.** A busy tenant's cut (the load point the measurement calls c10) carries about 190 k change rows. That is three to four orders of magnitude past the ~100-key crossover, so the cost model, which takes the cheaper of the keyed lookup and the scan, never picks the keyed lookup. The keyed lookup matters only when things are quiet. 2. **The kind of view does not bound its memory; the number of groups and the join's build side do.** An algebraic aggregate over 10 M distinct groups peaked at **2,581.7 MB**. The same kind over 997 groups peaked at **2.3 MB**, and a keyed projection at **0.0 MB**. With a 512 MB pool, the many-groups case failed with `Resources exhausted: Additional allocation failed for GroupedHashAggregateStream`. That is the pool doing its job: it refused, rather than taking the box down. 3. **Recompute views are much cheaper than first assumed.** An earlier figure of "4–6 minutes per refresh" was really compaction's bound, and has been withdrawn. A 1 M-row recompute view takes about 2 s per refresh, and can keep up with a 30-second cut cadence. The spread between runs is reported, not averaged away. The same wide-column scan query, over the same data on the same machine, measured 180.8 ms in one run and 665.6 ms in another. No conclusion above depends on a difference smaller than a factor of four. **Many Postgres-landed views over one input, on one primary** (measured 2026-09-27 on a 20-thread, 61 GB laptop running the compose-2 stack, with the product's defaults: a 30-second cut cadence and one refresh at a time; median of 10 measured cuts, warm-up excluded). Everything above is one view at a time. This is the per-manufacturer slice shape: one 1.4 M-row `invoice_lines`-shaped input, N views each filtering one manufacturer's UPCs, declared key `(invoice_id, line_no)`, landed in `derived`, with 20 changed rows per manufacturer per cut. | N views, class | landing transactions on the primary per cut (sum; longest) | maintainer pass per cut (slowest) | WAL per cut | a write reaches all N views | fits a 30 s cadence? | |---|---|---|---|---|---| | 8, keyed projection | 3.2 s; 441 ms | 3.8 s (8.4 s) | 0.65 MiB | 29 s | yes | | 64, keyed projection | 6.9 s; 174 ms | 13.2 s (17.2 s) | 4.0 MiB | 34 s | yes | | 200, keyed projection | 20.7 s; 537 ms | 47.0 s (78.3 s) | 13.4 MiB | 80 s | **no** | | 64, recompute (the same filter as `SELECT DISTINCT`) | ~10 s; up to 749 ms | **~13–15 min** (a partial cut: the 10-cut median was not reachable) | 269 MiB | ~15 min | **no** | What it means for you. **Keyed views over one input scale to about 64 per maintainer at a 30-second cadence on this machine.** 200 does not fit. Where between 64 and 200 it stops fitting was not measured; a straight line through the medians puts it near 130, and through the slowest passes near 90. **Recompute views are not a per-manufacturer shape at this input size.** Each one re-reads the whole input at roughly 12–14 s per view, 7–100 times the cap's own estimate. Two costs in the keyed landing grow with things other than the delta: the key-matched `DELETE` grows with the view's size (about 350 ms per view at 175,000 rows), and at N = 200 creating and dropping two staging tables per refresh is about 9 s of the roughly 15 s of server time. Both are left for their own work. --- ### 4. How to use it This section is the working half of the chapter: what you type to define, inspect, read and drop a view, and what you get back. It goes class by class, then covers reading the estate, reading through a pin, chains, drops, the cross-tenant shape, the breaker and the configuration knobs. **How to read the examples.** Each example is marked either **RUN** — run for this chapter, with its real output pasted — or **FROM THE CODE** — the exact signature, flag, constant or refusal text quoted from the source code or its design, and not run for this chapter. No `argondb serve` box was booted while this chapter was written. The RUN examples were run on a development build machine against the tree this chapter describes. Where a FROM THE CODE example shows expected output, that output was composed from the code that produces it, and the example says so. #### 4.0 The CLI surface — RUN The `argondb view` command is the command-line way to work with views. It has four subcommands: `classify` tells you what a definition would become without creating anything, `create` records a view, `drop` removes one, and `list` shows every view and how far behind it is. The same operations are available as SQL functions on the analyst compute (§3.1); the CLI is useful for scripts and for iterating on a definition. The binary was built from this tree (`cargo build --release -p argondb-ctl --bin argondb`, build exit code 0) and then run. The output below is verbatim. The help text refers to the view classes by short internal codes and to section numbers of the design document; in the order the first help screen lists them, the codes stand for the keyed projection, the algebraic aggregate, the min/max aggregate, the join view and the recompute view. ```console $ argondb view --help Continuous materialized views. `classify` answers what a definition WOULD be — class, key, inputs, FK join roles and the refresh estimate — and creates nothing, so a definition can be iterated on before it is recorded Usage: argondb view Commands: classify Classify a view DEFINITION without creating anything: print its maintenance class (C1/C2/C2m/C3/C4), its key, its inputs with their FK join roles and the per-refresh cost estimate — or the typed refusal, naming the construct and the remedy create **CREATE** a continuous view: validate the definition, classify it, and RECORD it (`argon.create_view` on the CLI — the same body, design §2.2) drop **DROP** a continuous view, printing its definition text so you can recreate what you meant to keep (design §15.6) list List every continuous view with its BEHIND-NOTE: the LSN each view is current to, the tail head at the time of asking, and the difference in bytes (design §3.5). Reads only — the maintainer is a view's one writer help Print this message or the help of the given subcommand(s) Options: -h, --help Print help ``` ```console $ argondb view create --help **CREATE** a continuous view: validate the definition, classify it, and RECORD it (`argon.create_view` on the CLI — the same body, design §2.2). The DataFusion half of §2.3's two-dialect validation only, and it says so: a Postgres `EXPLAIN` of this text on the analyst compute is the other half and this command has no backend to run it on. The SQL function runs both. Exit codes: 0 = created, 2 = REFUSED. Usage: argondb view create [OPTIONS] --name --sql Options: --name The view's NAME, under the same lake grammar a transform target takes --sql The definition: ONE SELECT over published lake tables. Bare lake names or `lake_.
` — both are accepted, because the door rewrites the mount qualifier the analyst compute shows --fk A declared foreign key, `.=.`, repeatable. §2.4 reads these from the analyst compute's own `pg_constraint`; a CLI has no such backend, and WITHOUT one a join classifies C4 with the demotion PRINTED --key The view's key columns, repeatable. Omitted ⇒ inherited from a passed-through PK or `GROUP BY`, or the view is keyless and therefore recompute class — stated either way --max-refresh-seconds Accept a recompute-class refresh at the classifier's estimated cost — §2.5's explicit acceptance for the worst well-typed input (a self-join with no equality predicate) --join-role §15.4: `
=fact` or `
=dimension`, repeatable — which side of a join is the FACT, for a join on a column an upstream VIEW computed. A view is not in `pg_constraint`, so a declared role is the only way such a join is C3; a base-table join reads the catalog and never needs one. Both sides must be named --max-staleness §4.3's reader-facing disclosure bound, recorded with the view --index §5.2's declared secondary index, as a comma-separated column list; repeat the flag for a second index. POSTGRES LANDING ONLY — on the lake landing it is REFUSED by name, never ignored, because an Iceberg table is scanned and not indexed --tail-addr [default: http://127.0.0.1:8817] --branch [default: main] --data-dir [env: ARGONDB_DATA_DIR=] [default: ~/argon] --port -h, --help Print help (see a summary with '-h') ``` ```console $ argondb view drop --help **DROP** a continuous view, printing its definition text so you can recreate what you meant to keep (design §15.6). Exit codes: 0 = dropped, 2 = REFUSED. Usage: argondb view drop [OPTIONS] --name Options: --name --cascade Drop the view's dependents too (§15.6). LIVE since L13 landed view-over-view (block V8): without it a view with dependents is REFUSED `view-has-dependents`, naming the full transitive set; with it they are dropped in REVERSE TOPOLOGICAL order (dependents first, this view last) and each is returned with its definition text --tail-addr [default: http://127.0.0.1:8817] --branch [default: main] --data-dir [env: ARGONDB_DATA_DIR=] [default: ~/argon] --port -h, --help Print help (see a summary with '-h') ``` **What it does with no read plane to talk to.** This is the output you will probably meet first, if you point the command at the wrong `--tail-addr`. `classify` takes a pin before it plans, so a refused connection shows up as a chain of causes, not as a view refusal: ```console $ argondb view classify --name dg_probe \ --sql 'SELECT id, region FROM orders WHERE region = 1' \ --tail-addr http://127.0.0.1:1 Error: taking the pin Caused by: 0: connecting to tail flight 1: transport error 2: tcp connect error 3: tcp connect error 4: Connection refused (os error 111) ``` Two things follow from this output. First, **`create` and `drop` are in this build.** An `argondb` binary that offers only `classify` and `list` predates the create command, which is a quick way to tell whether the binary in front of you is current. Second, **`--fk`, `--key`, `--join-role`, `--index` and `--max-refresh-seconds` are the CLI's spelling of the view options** (`ViewOptionsV1`). The SQL function takes the same six keys as one `jsonb` document (§6.7). The help for `classify` and `list` was captured in the same run but is not reproduced here. #### 4.1 A keyed projection — FROM THE CODE A keyed projection is the simplest view: it selects some rows and columns from one table and keeps that table's key. It is the cheapest class to maintain, because each refresh only applies what changed. ```sql -- on the analyst compute (a read-only Neon compute), as any user who can read -- the lake_ mount for this database: SELECT argon.create_view( 'open_invoice_lines', $$ SELECT id, invoice_id, sku, qty, unit_price FROM lake_main.invoice_lines WHERE status = 'open' $$, '{"landing": "lake"}'::jsonb); ``` What `classify` answers, and why. The plan is one table scan, a filter and a projection. The source table's primary key `id` is passed through, so the view inherits it as its key (`ViewKey::Inherited { source: "invoice_lines", columns: ["id"] }`). The class is keyed projection (`ViewClass::C1KeyedProjection`), the landing verb is `upsert`, and it does not need old images of changed rows. The rendered answer has the shape below. The field order is the rendering function's own, line for line; the angle-bracketed parts are placeholders: ``` view: open_invoice_lines class: C1 (maintained from the delta, landing verb `upsert`) key: inherited:invoice_lines[id] inputs: - invoice_lines (role sole, changelog enrolled) estimate: · note: ``` The last two notes are always present, and `create` composes them: ``` note: estimate: — every refresh reads the window's delta note: state: DECLARED — recorded, no refresh has landed. The maintainer (lane L4) lands the first one; until then this view carries no coordinate and `argon.views()` says so ``` The extra clause about a refresh-interval cap appears only on a recompute view's estimate line (§4.4), because an incremental class like this one takes no `min_refresh_interval`. **What `create` does not do: it does not refresh.** The maintainer is the only writer of a view, so after `create` the table exists and is empty until the next cut. **What refreshes.** At the first cut after `create`, the maintainer schedules the view (views that have never refreshed go first) and lands an `append`. The first refresh of a lake-landed view is always an `append`, because `create` already made the empty table. Every later refresh is an `upsert` — unless that window happens to hold inserts only, in which case the refresh is narrowed to an insert-only keyed projection and the verb to `append`. #### 4.2 An algebraic aggregate — FROM THE CODE An algebraic aggregate is a `GROUP BY` view whose aggregates can be updated by adding and subtracting: `count`, `sum`, `avg`, and (with a fallback) `min` and `max`. It lets you keep a summary table current without rescanning everything, at the cost of reading the old values of changed rows. ```sql SELECT argon.create_view( 'revenue_by_region_month', $$ SELECT c.region, date_trunc('month', o.order_date) AS month, sum(l.qty * l.price) AS revenue, count(*) AS n FROM lake_main.orders o JOIN lake_main.order_lines l ON l.order_id = o.id JOIN lake_main.customers c ON c.id = o.customer_id GROUP BY 1, 2 $$, '{"landing": "lake", "max_staleness": "1 cut"}'::jsonb); ``` Classification: the key is the grouping, `region` and `month` (`ViewKey::GroupBy(["region", "month"])`), because an aggregate's identity is its grouping. The aggregates are only `sum` and `count`, so the class is algebraic aggregate (`C2AlgebraicAggregate`), and it needs old images. Add a `min` or `max` and the class becomes the min/max aggregate (`C2mMinMaxUnderDeletes`), which carries the ability to recompute the affected groups. The class is fixed at `create` even though whether a refresh actually needs that recompute is decided per refresh. `avg` is accepted and is kept as a `sum`,`count` pair — but see §6.4: the rewrite rule will not roll `avg` up. **What refreshes.** For each group, an insert adds the new row's contribution, an update subtracts the old and adds the new, and a delete subtracts the old. The old images are fetched at `L_0` for every key updated or deleted in the window. A group whose count reaches zero is deleted. At a busy tenant's cut of about 190,000 change rows, that old-image read is one full scan of each input: 0.18–0.67 s per 10-million-row input (§3.11). #### 4.3 A join view with a declared foreign key — FROM THE CODE A join view joins a fact table to a dimension table along a foreign key. To maintain it from changes, ArgonDB has to know which side is the fact; the foreign key tells it. Without that knowledge the join falls back to the recompute class. On the analyst compute, the foreign-key roles are read for you and you pass no flag. (§6.2 describes the read this used to depend on, and the fix that replaced it.) The CLI has no Postgres backend to read, so you declare them: ```bash argondb view create \ --name invoice_lines_with_customer \ --sql "SELECT l.id, l.sku, l.qty, c.region FROM lake_main.invoice_lines l JOIN lake_main.customers c ON c.id = l.customer_id" \ --fk invoice_lines.customer_id=customers.id \ --tail-addr http://127.0.0.1:8817 --branch main ``` The `--fk` grammar is `.=.`, with no spaces. It is repeatable, one flag per column pair for a composite key. Each malformed shape is refused by name: ``` `--fk `: expected `.=.`, with one `=` `--fk `: the child side `` must be `
.` `--fk `: the parent side `` must be `
.` `--fk `: every part of `
.` must be non-empty ``` With the foreign key present, `invoice_lines` is the fact, `customers` is the dimension, and the class is join view (`C3JoinChangedSideKnown`). **Drop the `--fk` and the same definition classifies as a recompute view, with the demotion printed.** A unit test asserts both directions, turning on the foreign key alone. The estimate carries a second cost term when it applies. When a change to the dimension forces a scan of the whole fact table, because the fact table is not clustered by the foreign key, the estimate carries a remedy string (`Estimate::unpruned_fact_scan`). By default the lake clusters a table by its primary key, not its foreign key, so this is the common case. The maintainer prints the classifier's own remedy string on each refresh that does that scan. The remedy is the existing `argon:cluster-by=` marker on the fact table, not a new mechanism. **The exit codes are the contract:** `0` means created, `2` means refused. A script should branch on the exit code and never parse the text. #### 4.4 A recompute view, and the acceptance it needs — FROM THE CODE A recompute view is any view that cannot be maintained from changes. Every refresh runs the whole query again and swaps in the result. It is always correct, and its cost grows with the size of its inputs rather than the size of the change, so ArgonDB limits how often it runs. ```bash argondb view create \ --name sku_price_spread \ --sql "SELECT sku, percentile_cont(0.5) WITHIN GROUP (ORDER BY unit_price) AS median_price FROM lake_main.invoice_lines GROUP BY sku" \ --key sku \ --tail-addr http://127.0.0.1:8817 --branch main ``` `percentile_cont` is a holistic aggregate: nothing incremental exists for it. So the class is recompute (`C4Recompute`) and the landing verb is `replace`. The `notes` say which construct cost the view its incremental class. The cost is proportional to the size of the inputs on every cut that has any change, and does not depend on the size of the change. **How often a recompute view refreshes is derived, not left unlimited.** This has been the default since 2026-09-20. `create` prints a cost estimate, built from the inputs' current row counts in the catalog statistics and the conservative end of the rates measured in §3.11. From that estimate it derives `min_refresh_interval`: the smallest whole number of the tail's cut cadences under which the estimated cost is at most `ARGONDB_VIEWS_C4_COST_SHARE` percent (default 10) of one cadence. If a recompute view's input advanced within that interval, the view is held at its previous `L_v`. It is printed as `held by min_refresh_interval (N s)`, and a view that depends on a held view is `WAITING`. You can set the per-view `min_refresh_interval_secs` option (SQL, CLI or MCP) to raise the derived floor. A value below the floor is refused, and the option is refused outright on an incremental class — it applies only to recompute views. Nothing refuses a refresh for being expensive. The only effect is that a recompute view no longer refreshes more often than its own estimated cost justifies. One recompute shape is refused rather than costed: the worst well-typed input, which is the largest table joined to itself with no equality predicate. It is admitted only when you accept recompute explicitly: ``` refused: unpredicated-self-join — view ``: `` is joined to itself with no equality predicate — a cartesian product, classified C4, estimated . Remedy: add the equality predicate the join is missing; or, if the cross product is what you meant, re-issue `create` with an explicit `max_refresh_seconds` accepting recompute at that cost ``` ```bash # the acceptance, spelled: argondb view create --name deliberate_cross --sql "..." --max-refresh-seconds 900 ... ``` #### 4.5 Reading the estate: `argon.views()` and a view's coordinate — FROM THE CODE `argon.views()` lists every view in the database with its class, its state and how far behind it is. It is how you see what the maintainer has done, and it is the first place to look when a view seems stale. ```sql SELECT name, landing, class, current_to_lsn, behind, state FROM argon.views() WHERE state = 'refreshed' ORDER BY name; ``` Unlike its two sibling functions, `argon.views()` returns a set of rows rather than text, because it is a listing and you will want to filter it. The columns are `(name, landing, class, key_columns, current_to_lsn, behind, inputs, last_refresh, state)`. **`current_to_lsn` and `behind` are NULL for a view that has been declared and never refreshed.** Such a view is not current to any LSN, and a `0/0` would read as "behind by the whole head". The note about what the listing covers arrives as a Postgres `NOTICE`, not as a column, because it describes the answer as a whole and not any one row. `state` is worked out from evidence rather than read from a stored column: a coordinate on the landed table means a refresh has landed. Two values can be produced by this build on their own — `declared` and `refreshed`. `BEHIND` and `WAITING` come from the pass that handles chains of views (§4.7). The **behind-note** is three numbers: the view's coordinate, the tail head at the time you asked, and the difference in bytes. The subtraction is done in exactly one place. It is not a subtraction that stops at zero: a separate "ahead of head" answer exists and is named rather than shown as "0 bytes behind". It means either the head was read from a different tail than the one the refresh followed, or the coordinate on the snapshot is not one this writer wrote. The CLI prints the same numbers, and **prints what it covers**: ```bash argondb view list --tail-addr http://127.0.0.1:8817 --branch main # without --pg-url this covers the LAKE landing only and says so on the first line: # LAKE LANDING ONLY — no --pg-url was given, so any view landed in `derived.*` on ... argondb view list --tail-addr http://127.0.0.1:8817 --pg-url "$ARGON_PG" --branch main ``` **A view whose table vanished mid-listing is one marked row.** If a view's lake table is dropped between the listing naming it and the listing loading it (a tenant being created or deleted), `argondb view list` and `argon.views()` print that view as `table not found (dropped since the listing began)` and list every other view, exit 0; any other load failure (catalog unreachable, refused, timed out) still fails the whole listing, because a partial answer there would look complete. That includes the probe for the lake namespace itself: a catalog that does not answer fails the listing rather than reading as "no views". **`--database ` filters the listing** to the views whose declaration names `` as an input database (never the view-name prefix), and the `scope:` line says so: `scope: … · database only (N of M views)`. A database with no views prints an empty listing with that line, exit 0. On a box without per-tenant compute every tenant's views live in the one box warehouse, so the walk is box-wide and the filter is what narrows it; under per-tenant compute the same declaration is read the same way. **Only the CLI listing shows a gap.** `argondb view list` reads the branch's recorded-gap record and prints, under any view a gap or a retention floor is holding back, the same BEHIND sentence the maintainer logs. `argon.views()` has no gap column on this build — its `state` column still reads `refreshed` for such a view — so today a SQL caller learns about a gap from the maintainer's own log or from the CLI, never from the listing itself. #### 4.6 Reading a view through the FDW, and through the query plane at a pin — FROM THE CODE There are two ways to get rows out of a view. You can read the view directly, as a table. Or you can query the base tables and let the rewrite rule answer from a view when it can prove the answer is the same. This section shows both, and the settings that control the second. ```sql -- on the analyst compute. The mount is scoped to this database already. SET argondb_lake.pin = '0/3A21B40'; -- read at a named coordinate SELECT * FROM lake_main.open_invoice_lines WHERE sku = 'X-1'; ``` That is a scan of a lake table, not a rewrite (§3.10). The pin is the session's `argondb_lake.pin` setting or, when unpinned, the tail head with a behind-note. ```sql -- the same read, letting the rewrite rule serve it from a view: SET argondb_lake.view_staleness = '0'; -- the default: exact, or no rewrite SELECT sku, sum(qty) FROM lake_main.invoice_lines WHERE status = 'open' GROUP BY sku; SET argondb_lake.view_staleness = 'any'; -- take the rewrite however far behind SET argondb_lake.view_staleness = '5000000';-- ... or within 5 MB of the pin SET argondb_lake.view_staleness = '30s'; -- ... or within 30 s (reads Argon.committed-at) SET argondb_lake.view_rewrite = off; -- turn the rule off for this session ``` `argondb_lake.view_rewrite` is a boolean setting, so Postgres's own grammar refuses a bad value at `SET`. `argondb_lake.view_staleness` is a string: any value is accepted at `SET`, and a bad one is refused when it is read (§3.10 gives the grammar and the default). Through the query plane on the CLI, the same three controls are flags: ```bash argondb-query --at-lsn 0/3A21B40 --view-staleness 30s --sql "SELECT ..." argondb-query --no-view-rewrite --sql "SELECT ..." argondb-query --explain-views --sql "SELECT ..." # the rule's log: what fired, what refused and why ``` `EXPLAIN` of a rewritten plan carries this line: `ViewRewrite( depth= @ [+ compensation (, ]] [disclosed])`. An earlier box run of the rewrite differential test, which compares every answer with the rule on and off, recorded these lines (not run for this chapter): ``` fired ViewRewrite(v_vrd_c2 depth=0 @ 0/16F3178) VIEW-REWRITE-DIFFERENTIAL: PASS (22 assertion(s), 0 failed; C1/C2/C3 rewrite-on == rewrite-off row for row) ``` The 22 assertions cover keyed projections, algebraic aggregates and join views: with the rule on, each returned the same rows as with it off. **From DuckDB, outside ArgonDB entirely** — the shape a box test runs, DuckDB with its `iceberg` extension loaded: ```sql LOAD iceberg; ATTACH '' AS lake (TYPE iceberg, ENDPOINT '', AUTHORIZATION_TYPE 'none', ACCESS_DELEGATION_MODE 'none'); SELECT count(*) FROM lake."dv_main"."region_mix"; -- the rows -- the coordinate: read out of the current snapshot's own metadata SET VARIABLE loc = (SELECT metadata_location FROM iceberg_load_table_response('lake."dv_main"."region_mix"')); CREATE TEMP TABLE md AS SELECT * FROM read_json(getvariable('loc'), compression = 'gzip', columns = {'current-snapshot-id': 'UBIGINT', 'snapshots': 'JSON[]'}); SELECT e.key, e.value ->> '$' FROM (SELECT unnest(snapshots) AS s, "current-snapshot-id" AS c FROM md) cur, json_each(cur.s -> 'summary') e WHERE (cur.s ->> 'snapshot-id')::UBIGINT = cur.c; -- expect Argon.view-current-to-lsn among the keys, and no Argon.watermark-lsn ``` Two reader-side limits, both measured by that box test: this version of DuckDB's own snapshot-listing function returns no summary column, so the coordinate has to be read from the raw metadata file instead; and its table-lookup function refuses a multi-part table address written as an identifier, so the address has to go in as a plain string. #### 4.7 A two-view chain, and the states it takes — FROM THE CODE A chain is a view that reads another view. It lets you build in layers — a cleaned-up "silver" view, and a summary "gold" view over it — while each layer is still maintained from changes. Chains add states you will see in `argon.views()` when an upstream view stalls. ```sql -- silver: a keyed projection over a base table SELECT argon.create_view( 'silver_lines', $$ SELECT id, order_id, sku, qty, qty * unit_price AS line_total FROM lake_main.invoice_lines WHERE status = 'open' $$, '{"landing": "lake"}'::jsonb); -- gold: an algebraic aggregate over the silver view, named exactly as a base table is named SELECT argon.create_view( 'gold_sku_totals', $$ SELECT sku, sum(line_total) AS total, count(*) AS n FROM lake_main.silver_lines GROUP BY sku $$, '{"landing": "lake"}'::jsonb); ``` At the next cut both refresh to the same `L_1`, silver first. Gold reads silver's own committed `row_delta` snapshot as its stream of changes; no `silver_lines__changes` table is created (§3.8). Three things to check when a chain misbehaves, each with its exact text: | symptom in `argon.views()` | what it means | what to do | |---|---|---| | `BEHIND (upstream silver_lines FAILED at )` | silver's breaker opened. Gold did not move, and it is not `INVALID` — its own definition still plans | fix silver's cause. At the next cut both advance, and gold lands one snapshot over the grown window | | `WAITING (upstream silver_lines coalescing to )` | silver has `min_refresh_interval > 0` and targets a later cut, so gold targets that cut too | nothing. The rule that a view is never ahead of its input, `L_v(gold) ≤ L_v(silver)`, is holding | | gold's class is recompute, and `create` said why | silver is a keyless recompute view, so it has no changes to hand down | declare a key on silver (`--key`, or the `key` option), then drop and recreate | The state sentences are produced in one place, so what `argon.views()` prints and what the per-view behind-line prints are always the same text. #### 4.8 `drop_view` with dependents, and CASCADE — FROM THE CODE Dropping a view that other views read would leave them with nothing to read. So ArgonDB refuses that drop unless you ask for a cascade, and a cascade hands back every definition it removed so you can recreate the ones you wanted. ```sql SELECT argon.drop_view('silver_lines'); -- refused: view-has-dependents — names every transitive dependent SELECT argon.drop_view('silver_lines', cascade => true); -- drops gold_sku_totals first, then silver_lines, and returns each with its -- definition text so you can recreate what you meant to keep ``` ```bash argondb view drop --name silver_lines --cascade --tail-addr http://127.0.0.1:8817 --branch main ``` The signature is `drop_view(name text, cascade boolean DEFAULT false)`. The request sent over the wire is `ViewDropV1 { name, database, cascade }`. A cascade drops in reverse dependency order — dependents first — so there is never a moment when a view's input is gone while the view remains. A dependent that is in the middle of a refresh when a cascade drops it is abandoned, and its uncommitted snapshot never lands. `--cascade` works on this build. A `drop` with dependents and no `--cascade` is refused `view-has-dependents`, naming the full set of dependents, direct and indirect, as in the SQL example above. #### 4.9 A view across the tenant boundary — FROM THE CODE This is the bottom band of the main diagram, written as SQL. It shows how a distributor builds a view that compares its own sales with a manufacturer's price list — without any query reaching into another tenant's data. The manufacturer's rows are already in the distributor's database, because the hub copied them there. Inside one distributor's database, after the hub has landed a manufacturer's price list there as `derived.mfr_price_list` (a landed copy, marked for publication, so it is in this distributor's lake copy): ```sql SELECT argon.create_view( 'realised_vs_list_by_month', $$ SELECT date_trunc('month', l.invoiced_at) AS month, l.sku, sum(l.qty * l.unit_price) AS realised, sum(l.qty * p.list_price) AS at_list FROM lake_main.invoice_lines l JOIN lake_main.mfr_price_list p ON p.sku = l.sku GROUP BY 1, 2 $$, '{"landing": "lake"}'::jsonb); ``` **Nothing here is a cross-tenant feature.** Both tables belong to this one database. The view's database is its inputs' database, and the maintainer checks the tenant wall once more with the same test the pin uses. What makes the view possible is that the hub already copied the rows in. What would not work — a join across two databases — and its exact refusal: ``` refused: cross-database-join — view ``: . Remedy: joins are within ONE database in v1; the `all_tenants` composition is a reader roster feature and a view over it is a different question ``` A manufacturer's own views work the same way over its `hub-` database: its slice of orders, invoices, movements and stock, joined to its own product master, and never a distributor's other rows. **What is refused instead: the same view over a GRANTED price list.** If the manufacturer had merely granted this database a mount of its own price-list table, rather than landing a copy, the table would still read fine in `EXPLAIN` — and `argon.create_view` would still refuse it, before anything is pinned, enrolled or recorded: ``` view-input-granted: view `realised_vs_list_by_month`: input .public.price_list is a mounted grant from ; views over granted tables are not ruled — [ruling pending]. A lake-landed view is scoped to its inputs' OWNER, so this one would be created and then unreadable here; define the view in , or over tables this database owns ``` Both landings meet this refusal. The remedy today is the one the message names: build the view in the grantor's own database, or have the grantor land a copy and build the view over that copy instead. This is proven with the tenant wall both on and off, alongside a positive control over an owned table. One caution specific to this shape. `mfr_price_list` reaches the lake through the tail like any other published table, so a view over it is maintained incrementally only if **its** changelog is enrolled. If it is not, this view is a recompute view, and the note says which input cost it the incremental class. Check with `argondb view classify` before you create. #### 4.10 The breaker: how a view fails and how it recovers — FROM THE CODE The per-view breaker stops the maintainer from retrying a view that keeps failing, without affecting any other view. A failed view keeps serving its last good snapshot; it just stops advancing until you fix it. The breaker counts a streak of consecutive failures, not a total (`BreakerVerdict`). `ARGONDB_VIEWS_BREAKER_FAILURES` (default 3) consecutive failures mark the view `FAILED ()`, and one success clears the streak. **Two BEHIND states are never counted as failures at all:** a window that lands across a recorded gap or below an input's changelog retention floor (§2.2, §4.12), and a cut that lands below an input's enrolment floor that a later `create` recorded (§3.6). The second one is the v1.5.6 fix — before it, a view could trip its own breaker over somebody else's `create`, because the refused pin was wrongly charged to it. Unit tests cover its behaviour under several load patterns: the incident that motivated it, a healthy steady state, and a bursty pattern with a quiet window inside it (five failures, never three in a row — the case a plain count gets wrong), plus a control showing one view's failures do not touch another. The breaker's verdict takes three values: | verdict | meaning | |---|---| | `Retrying { consecutive, of }` | counted, still below the threshold: this view will be tried again | | `Tripped { consecutive, cause }` | `FAILED ()`; the view stops being scheduled and keeps serving at its own `L_v` | | `AlreadyTripped { cause }` | the caller should not have scheduled it; this is reported rather than silently counted again | A tripped view **keeps serving at `L_v`**, with its state in the disclosure, and the rewrite rule stops using it (`Coordinate::Never`). A read of it is never refused. To recover, drop the view and create it again with the fixed definition, or restart the maintainer. The maintainer keeps no progress store of its own, so on the next start it reads each view's `L_v` back from the view's own landed table and carries on. Outside the per-view breaker is the ordinary restart policy for the maintainer process: backoff from 2 s to 30 s, and a breaker at 6 restarts in 120 s. A trip of that breaker records the maintainer as DEGRADED, and the box keeps serving. The per-view breaker exists so that one bad definition cannot trip the process breaker for every other view. #### 4.11 Setting the knobs — FROM THE CODE These environment variables configure the view maintainer. You set them to change how many refreshes run at once, how much memory they use, and when a view is given up on. `argondb-maintenance` reads every one of them at boot, and a bad value stops the process with an error rather than falling back to a default. | variable | default | what it bounds | |---|---|---| | `ARGONDB_VIEWS_CONCURRENCY` | `1` | how many refreshes run at once | | `ARGONDB_VIEWS_BREAKER_FAILURES` | `3` | consecutive failures before a view is `FAILED` | | `ARGONDB_VIEWS_POLL_SECS` | `5` | how often the maintainer asks the tail where `W` is | | `ARGONDB_VIEWS_WINDOW_KEYS` | `1000000` | the soft window bound, applied at commit boundaries (§3.7) | | `ARGONDB_VIEWS_PASS_DEADLINE_SECS` | `1800` | how long one pass may run. After that it is abandoned, with a warning that names the view and the step it was waiting on, and it counts as a failure for that view | | `ARGONDB_VIEWS_RING_STEP_SECS` | `94` | how long each step of one ring read (the connect, the `do_get`, the drain of its stream) may take. Past it that read is abandoned, which fails one view's refresh for the pass but never the pass. It must be below the pass deadline | | `ARGONDB_VIEWS_RECOMPUTE_SECS` | `600` | how long one view's whole refresh may take. Past it that view alone is abandoned for the pass, named, and counted as one failure for it, and the pass goes on. `0` turns it off, and then only the pass deadline ends a stalled view. It must be below the pass deadline, the default included | | `ARGONDB_VIEWS_IDLE_LOG_SECS` | `600` | how often a maintainer with nothing to report says it is idle, and how often a pass still running says which step it is in | | `ARGONDB_PG_CONNECT_TIMEOUT_SECS` | `30` | how long a connection to a landing database may wait for Postgres to answer | | `ARGONDB_VIEWS_MEMORY_MB` | a fourth even share of the box's memory reserve | the spilling memory pool a refresh computes under | | `ARGONDB_VIEWS_SPILL_DIR` | under the data dir | where that pool spills to disk | | `ARGONDB_VIEWS_MAX_DEPTH` | `8` | the maximum chain depth, enforced at `create` | | `ARGONDB_VIEW_REWRITE` | on | switches the rewrite rule off for the whole deployment | | `ARGONDB_TRANSFORM_PG_URL` | unset | the connection a Postgres-landed view lands through (host, port, user, credentials). Leaving it unset is normal; such a view is then skipped, with a message naming this variable | The maintainer is never silent when it is stuck. It prints a summary only when a pass refreshed, failed or held a view, so three more lines tell a quiet maintainer from a stuck one: - With no summary for `ARGONDB_VIEWS_IDLE_LOG_SECS`, it writes `views: idle — tail watermark head , view(s) current, held, last pass
`, `plan`, `execute` and `recompute`. The per-step bound `ARGONDB_VIEWS_RING_STEP_SECS` covers only step `ring`. A view is reported in `ring` only while a ring read is actually in flight. Every other step is ended only by `ARGONDB_VIEWS_PASS_DEADLINE_SECS`, and that includes `recompute`. `recompute` is the view's own work after its ring read has answered: the rest of the pin and the query plan. Before v1.5.7.13, a view that stalled there was still reported as `awaiting ring`, because nothing cleared the mark its last ring read had set. That is what the ERP saw three times in 48 hours on `demoharbor_v_program_earned`, and each time the stall ended only at 1800 s. Since 2026-10-13 the view's own work names its wait. After the ring read, a refresh is in one of four steps: `data-pin` (the rest of the pin), `lake-scan .
` (the query plan reading one input table, named), `plan` (planning the definition) and `execute` (the joins, aggregates and diff). The plan reads the lake on worker tasks, and those tasks now carry the view's step. A lake scan's step restarts at every batch, so its seconds are the wait for the next batch. A stall reads, for example, `awaiting lake-scan demoharbor.program_earned_src for view (590s in that step; …)`. `recompute` is left only for a view that entered no other step after its read. One view's whole refresh is also bounded, by `ARGONDB_VIEWS_RECOMPUTE_SECS`. A view whose refresh (pin, scans, plan and landing together) runs past it is abandoned alone. Nothing of it lands, and its worker tasks are stopped. It is named in its `refresh FAILED` line with the step it was in, and its breaker counts one failure, the same as for a pass-deadline abandon. The pass then goes on with the other views. While the view runs, its "still running" line says `its per-view recompute bound is s, ARGONDB_VIEWS_RECOMPUTE_SECS`. Unset, the bound is 600 seconds (ten minutes), so a maintainer nobody configured still marks one stalled view by name and moves on. Set it to `0` to turn it off; then only the pass deadline ends a stalled view, as before. Step `ring` is the maintainer reading a table's newest rows from the tail's ring (a Flight `do_get`), and it has a bound of its own. Before v1.5.7.12 it had none: a `do_get` the tail accepted and never answered held the whole pass for the 30-minute pass deadline, along with every other view in that pass, and neither log showed which side had stalled. Now the connect, the `do_get` and the drain of the answer are each bounded by `ARGONDB_VIEWS_RING_STEP_SECS` (94 seconds by default: five times the measured time to drain the largest ring a customer box has held, about 19 GiB). A read past its bound fails only that view's refresh for this pass. The view stays at its last LSN, the ordinary `refresh FAILED` line names the table, the step and the elapsed time, the breaker counts it, and the next pass tries again. The other views go on. Both sides log every ring read in one format, so the two logs can be matched on `table` and `lsn` with one `grep ring-read:`: ``` ring-read: side=tail table= lsn=(,] rows= answer_ms= verdict=answered|refused ring-read: side=maintainer view= table= lsn=(,] rows= answer_ms= verdict=within-bound bound_s= ring-read: side=maintainer view= table= lsn=(,] rows=0 answer_ms= verdict=abandoned step= bound_s= ``` What a stall looks like: a maintainer line with `verdict=abandoned step=do_get` and no tail line for the same table and range means the tail never got past its own state lock. A tail `verdict=answered` next to a maintainer `step=drain` means the rows left the tail and the stream stalled on the way. `step=connect` means the TCP connection itself never completed. A port that accepts the connection and then says nothing shows up as `do_get`. The tail's line is new log volume, one line per table per pin. Expect the maintainer's line to be rare. A refresh pins at the cut it is following, and the lake normally already holds that point for every input, so the read goes to the lake and never to the ring. The maintainer reads the ring only when the cut is ahead of an input's lake copy, for example on a view's first refresh. Many tail lines with no maintainer line beside them is the normal, healthy picture. If the maintainer task itself ends, a watcher logs `views: maintainer task ended: ; respawning` and starts it again, at most once a minute. When the tail's last cut failed, the tail's `freshness` answer carries `last_cut` with the reason, `argondb view list` prints `LAST CUT: FAILED at
`. Otherwise it is refused `key-missing`. Nothing lands without a key unless you pass `no_key: true`, and then the answer says so. 2. **A lake-landed table has no log position.** It is always read at its current snapshot, never at your pin; the run's pin is recorded for reference only. 3. **The SELECT runs on the lake, at a pin.** A Postgres table that is not published, or is published but not yet in the lake, is not visible to it, and the refusal is `compute-failed` — **even when `query` answers over the same name**, because `query` may be answered by Postgres. This is the most common surprise with this tool. Section 4.8 is the tool that prevents it. **`upsert` works on the lake landing only.** In one snapshot it deletes every key the run's rows carry and appends their new versions. It needs an existing **keyed** table (otherwise `target-keyless`), and on the Postgres landing it is refused `landing-option`. #### 4.6 Publishing a table: preview, then apply ```jsonc // Dry run — the DEFAULT. Shows the exact change and makes none. { "tool": "lake_publish", "arguments": { "table": "public.orders" } } ``` ```jsonc // Applied. { "tool": "lake_publish", "arguments": { "table": "public.orders", "execute": true } } ``` `lake_publish` sets the `argon:publish` marker. The whole table is copied into the lake at the next maintenance cycle, and followed from then on. Publishing a published table does nothing. The destructive opposite, and its second check: ```jsonc // Refused: purge is destructive and confirm is absent. { "tool": "lake_unpublish", "arguments": { "table": "public.orders", "purge": true, "execute": true } } ``` `purge=true` also drops the Iceberg table and its change log and deletes their files. It needs `confirm=true`, and without it refuses, naming exactly what would be lost. Unpublishing without `purge` keeps the lake files and history and can be undone. **Two admin tools refuse to apply**, and say why: - `maintenance_run` with `execute=true` refuses. Maintenance has a single owner process, and there is no way in this release to ask it to run; starting a second sweep alongside it is exactly what the refusal prevents. - `retention_set` with `execute=true` checks the value, then tells you the operator step — set `ARGONDB_PIN_RETENTION_SECS` on the maintenance owner and restart — rather than pretend to change it live. Configuration is not reloaded at runtime. **`apply_index` changes the primary database**, so it needs `confirm=true`. Without it, it refuses and shows the exact `CREATE INDEX` it would run, noting that it locks and writes the primary. It logs full arguments and results, sits behind a cost circuit breaker, uses `IF NOT EXISTS`, and takes **structured** parameters — table, columns, unique, using, name — with no free SQL; every identifier is checked and quoted. #### 4.7 `publish_status` — "I published it; why isn't it in the lake?" The same tool, from one body and one description, is on both the analyst and diagnostic mounts, so the two cannot drift apart. ```jsonc { "tool": "publish_status", "arguments": { "table": "orders", "include_gaps": true } } ``` States: `mirrored`, `awaiting-variant`, `awaiting-json-text`, `no-backfill`, `backfill-pending`, `not-publishable` — "plus catalog-unreachable / state-unknown / unknown, which mean lake presence could NOT be determined and are never a claim that a table is missing". **Why it exists.** In the tool's own words: "Publishing a table marks it, but a column the native decoder cannot handle (jsonb, json, arrays, inet, interval, ranges, user-defined enums, composites, domains) keeps the whole table out of the lake — with no SQL error and no catalog entry, so a lake-routed query simply never finds it at any pin." A marker on something that is never captured at all — a view, a materialized view, a partitioned parent, a foreign table, an `UNLOGGED` or `TEMPORARY` table — shows as `not-publishable`. Four more things that change how you call it: - Names use the **Postgres** spelling (`orders`, `sales.orders`), not the lake spelling (`sales__orders`). - It covers **every database** in the cluster. A table outside the primary database is named `database.schema.table`, and a bare name matches in every database, so two databases that each have `orders` give you two rows, not a silent pick. - Each row shows the table's **key**: `key: [id]`, or `key: none (append-only; no row-level updates)`. Check it before planning to update a table a transform created. - `include_gaps=true` adds each gap's window, when it re-baselined, how many commits were lost and whether the lake has caught up. **Zero gaps is a fact only when `gaps-reader` says `ok`**; if the lake catalog could not be read it says UNKNOWN, never zero. A gap is permanent history: `resolved` means the rows are current again, not that the missing window was recovered. #### 4.8 `transform_pin_serves` — "is it ready to transform over?" ```jsonc { "tool": "transform_pin_serves", "arguments": { "tables": ["orders", "derived__revenue_by_region"] } } ``` ``` status: ready pin: 0/15C6F08 tables-asked: 2 total-served: 37 serves: orders=yes serves: derived__revenue_by_region=yes ``` One fact per line: `status:`, `pin:`, `tables-asked:`, `total-served:`, then `serves: =yes|no` for each name in the order asked, and a `missing:` list when the status is `not-ready`. Read its description before you write a polling loop, because it names the mistake: "THIS IS THE TOOL TO POLL after publishing a table and before calling run_transform. Do **NOT** use a `query` count for that: on a hybrid box `query` routes to the Postgres arm and answers the instant the rows commit, which says nothing about the lake arm a transform computes on — that mismatch is a measured race, not a theory (on v1.4.0 the starter's `query` probe returned in 0 s and the transform was refused for 61 s)". Two spelling rules: `tables` use **lake** spellings (a published `orders` is `orders`; `derived.revenue_by_region` is `derived__revenue_by_region`), and an internal name (`__base`, `__delta`) answers `no`, because callers may not reference those. #### 4.9 What each mount cannot do **Analyst (`argondb`).** It cannot write, and Postgres is what refuses: `SELECT INTO` needs `CREATE` on the schema, `nextval` needs `USAGE` on the sequence, `COPY FROM PROGRAM` needs superuser, and a function's inner write needs table write permission. None is granted, so each fails with `42501 insufficient_privilege`. It cannot send two statements in one call: > `one statement per call — found N; send separate calls` It cannot run transaction control under a pin (4.3). And it cannot reach the primary database at all: it is given no connection to it. **Diagnostic (`argondb-dba`).** It cannot change anything: it holds an observation connection and a metrics endpoint, and no write connection. It cannot run free SQL on the observation path; no such tool exists. Its `explain_query` returns no plan and points you to the analyst mount's. **Write (`argondb-write`).** It cannot read anything except its own readiness check. Its tool list is tested for the absence of `health`, `top_queries`, `index_advice`, `query`, `sample`, `explain_query`, `execute_sql`, `lake_publish`, `lake_unpublish`, `retention_set` and `apply_index`, by name. In the mount's own words: "This mount holds no observation handle and no analyst handle: it cannot read the primary's runtime stats, workload or health internals, and it serves no free-SQL read tool". And it cannot even **spell** a reserved name. For `append_rows`, `target` is a bare name matching `[a-z][a-z0-9_]*`, at most 63 bytes, not starting with `_argon_`, `br_` or `dv_` and not containing `__` — "the server computes the namespace, so a `br_` target is unspellable here, not merely refused". **Admin (`argondb-admin`).** It cannot change anything by accident: `execute` defaults to false everywhere, and the two destructive tools also need `confirm=true`. It cannot trigger maintenance or change retention live (4.6). And it cannot be reached from another machine unless someone set its own listen variable (2.3). **From a company token (on `argondb`).** It cannot read another company. Its session's only connection is its own database as `_mcp`, so a table in another company's database is refused by Postgres, never by a filter (2.6). It cannot write, because the role holds `SELECT` and nothing else. It cannot call the two box-wide tools; `search` answers: > `search is refused under company "alpha"'s MCP token: it reads box-wide > state (every company's), not company "alpha"'s own database. Ask the box > operator, whose token keeps operator scope.` `publish_status` answers the same, with its own name. And a company token cannot open any other mount, because only the analyst listener accepts it. #### 4.10 Making a company token and connecting with it `tenant create` has already left the first token on disk, so a new company needs nothing more: ```console $ argondb tenant create alpha … [argondb] tenant create: MCP token for alpha (reads alpha only, as alpha_mcp) written to /tenants/alpha/mcp-token ``` If making or writing the token fails, the company is **not** undone. The line says `the MCP token was NOT minted` (or `minted but NOT written`) and names the re-issue command as the remedy. Re-issuing prints the token on stdout and the notice on stderr, so the token can be captured cleanly: ```console $ ALPHA_MCP_TOKEN=$(argondb tenant mcp-token alpha) [argondb] tenant mcp-token: issued for alpha; every earlier token of alpha is revoked $ argondb tenant mcp-token alpha --revoke [argondb] tenant mcp-token: every MCP token of alpha is REVOKED (record removed, alpha_mcp NOLOGIN) ``` Revoking does two things: it removes the record the MCP process checks, and it takes login away from `alpha_mcp` as a second barrier. Both commands take `--data-dir` and `--port` when the box is not at its defaults. Connecting is the analyst mount's ordinary HTTP add, with the company's token as the bearer instead of the box's: ```bash claude mcp add --transport http argondb-alpha http://localhost:8018 \ --header "Authorization: Bearer $ALPHA_MCP_TOKEN" ``` The `--header` flag is the Claude Code client's. Any MCP client that sends `Authorization: Bearer ` works, because that header is all the listener reads. Every response then carries `x-argondb-mcp-scope: company:alpha`, and a request with the box token carries `x-argondb-mcp-scope: operator`. Read that header in a client that holds more than one credential: it is the server's own statement of which wall the answer came from. The port is the analyst port the box's MCP process was given (`ARGONDB_MCP_PORT`); 8018 is the standalone default from 4.1. --- ### 5. What it does, and what it does not do #### What it does - Four MCP mounts, each with its own token, port, network default and role; tool lists read from the live routers. - A separate write mount, so data-writing applications need no diagnostic access. - A read-only analyst mount, enforced by the database role, a one-statement rule and a check at startup. - Pins as signed values, with time travel, and the five things a pin tells you about itself. - A token budget on every answer: render, measure, trim, mark. - `changes()` net-effect summaries, with gaps shown. - `describe` and `profile` from stored statistics, with provenance on every value. - One `schema_overview` across Postgres and the lake. - `run_transform` into Postgres or the lake, exactly once, in five modes. - `append_rows`, direct appends into lake tables. - `create_view`, continuous materialized views. - `transform_pin_serves` on the write and diagnostic mounts. - `execute_sql` and `explain_query` on the analyst mount, compatible with `postgres-mcp` clients. - A per-company token on the analyst mount: made at company creation, re-issued or revoked on demand, checked on every request, and read as the company's own read role on the company's own database — its `public` and `derived` schemas, nothing else. - A cross-company isolation test, with the MCP token as one of its surfaces. - The accelerator, for listed query shapes, reachable through the MCP tools only. - An optional Postgres arm for agent reads at a pinned position, controlled by `ARGONDB_ANALYST_POOL`; with it off, reads are answered by the lake. - Agent load kept off the transactional database, as fixed policy. #### What it does not do - **No query plan from the diagnostic mount.** Use the analyst mount's `explain_query`. - **No triggering maintenance from MCP.** `maintenance_run execute=true` refuses. - **No live retention change.** `retention_set execute=true` tells you the restart to do. - **No acceleration on an ordinary Postgres connection.** A query sent to Postgres directly is never switched to the accelerator for speed; that path only logs the opportunity. - **No company-scoped `search` or `publish_status`.** Both refuse by name under a company token. - **No role scoping on the lake.** Row-level security inside a company applies to Postgres, not to the lake (section 6). That half is decided and not built. --- ### 6. Things that will catch you out **A read on the Postgres arm tells you nothing about the lake.** With the Postgres arm on, `query` answers the moment a row commits; the lake a transform reads may be a minute behind. A readiness check written as a `query` count passes instantly and proves nothing. *Fix:* poll `transform_pin_serves`. It takes the same pin and reads the same list that `run_transform`'s `compute-failed` refusal prints, so the two cannot disagree. **Three tables exist only in the lake.** `_argon_timeline`, `_argon_gaps` and `_argon_value_notes` are never copied into Postgres. A statement that reads only lake-only tables is sent to the lake, so that works. *But* joining one of them to an ordinary table sends the statement to Postgres, where it fails by name. That is deliberate. **Lake names are not Postgres names.** `publish_status` takes the Postgres spelling (`sales.orders`); `transform_pin_serves` and a transform's SELECT take the **lake** spelling (`derived__revenue_by_region`). Mix them up and you get a confident wrong answer one way and a `no` the other. A table's identity is database + schema + table; the flat `____
` name can be the same for two different databases, which is why the merged map matches on all three. **A published table can silently never arrive.** One column of a type the decoder cannot handle — jsonb, json, arrays, inet, interval, ranges, user-defined enums, composites, domains — keeps the **whole table** out of the lake, with no SQL error. *Fix:* check `publish_status` before relying on a table, and treat `awaiting-variant`, `awaiting-json-text` and `not-publishable` as different situations. **"Zero" is not always a measurement.** `publish_status`'s gap count is a fact only when `gaps-reader` reports `ok`. `catalog-unreachable`, `state-unknown` and `unknown` mean the answer could not be determined, never that a table is missing. **`state-unknown` is the one status you must not retry blindly.** Every `run_transform` refusal happens before the landing opens, so `refused` means nothing landed. `state-unknown` may have committed: re-run with the **same** token. The same goes for `append_rows`. **`staged` is safe but invisible.** `append_rows` returning `staged` means the data is in object storage and no reader sees it until the next commit cycle. Pass `wait_for_commit: true` if you need `committed`, with its snapshot id, before you answer your own caller. **A lake-landed table ignores your pin.** It is read at its current snapshot; the run's pin is recorded for reference only. **`drop` on the lake landing cannot be retried safely.** A repeated drop answers `target-missing`, where the Postgres landing answers `already-landed`. **A `create` without a key is refused, not guessed.** Declare `key`, pass one source table's key straight through, or say `no_key: true`. Without a key an Iceberg table cannot take row-level updates cleanly. **On the lake, a key is a declaration, not a constraint.** Iceberg has no unique index. On Postgres the key is enforced, and a duplicate is refused `duplicate-key`. **Dry run is the default, and `execute=true` is not always enough.** The two destructive admin tools also need `confirm=true`, and two admin tools refuse `execute=true` outright (4.6). **Port 8021 is not the write mount.** It is 8017, so that it cannot collide with a slot-1 analyst mount at 8021. **Opening the analyst mount does not open admin or write.** `ARGONDB_LISTEN_ADDRESS=0.0.0.0` moves the analyst and diagnostic mounts; admin and write stay on loopback unless their **own** variables say otherwise. And a non-loopback mount without its token stops the whole process at startup. **A blank `ARGONDB_MCP_ALLOWED_HOSTS` is not "no restriction".** It falls back to the loopback set, and there is no value that disables the check. A non-empty value **replaces** the default: list every host you want, including loopback if you still need it. **`budget_tokens` never refuses; it trims and says so.** For a hard size refusal, rely on the row caps (3.5). A budget does not buy completeness: cells are clipped at 160 characters with a footer, and `read()` gets you the whole value. **Two unpinned reads may legitimately disagree.** The lake gives its newest servable position; the Postgres arm gives its current replay position. Pin them if it matters; every answer's header shows which you got. **Role restrictions inside a company do not reach the lake.** Row-level security filters Postgres. The lake holds the same company's rows with no policy layer, so an analyst tool or a transform can see rows the same person could not read through SQL. The exposure is over-broad access **within** one company, not across companies. The product has no fix for it; take it into account before giving an agent the analyst or write mount for a company whose users are restricted by row-level security. **A company token only opens the analyst mount.** The diagnostic, write and admin mounts each compare against their one operator token, so a company token there gets a plain `401 … Bearer token required`. **Re-issuing a company token revokes the old one at the next request.** The listener reads the auth store on every call, so a client still holding the previous token gets `401` with `…'s token was revoked by a re-issue` at once. Re-issue only when you can hand the new token to every client of that company. **`search` and `publish_status` refuse under a company token.** Both read box-wide state. A company session gets no text search at all, and checking a publication is the operator's job. **The box token does not read a company's lake rows on the shipped shape.** With the catalog authorizer set to `openfga` and the analyst pool off, a box-token lake read of a company table answered `pin scope names unknown lake table(s)` when the per-company token landed. The box token keeps operator scope as a credential, but its lake session does not list company tables. This was recorded at the time and not fixed. Company tokens are unaffected, because their reads never touch the lake. --- ### 7. The detailed diagrams Four more pictures of parts the main diagram compresses. Each is generated by a script in `docs/diagrams/argondb-and-the-erp/generate/`, in the same palette and with the same legend. #### 7.1 The four mounts, their tools and their refusals ![The four mounts and their refusals](diagrams/argondb-and-the-erp/07a-the-four-mounts-and-their-refusals.svg) Generator: `docs/diagrams/argondb-and-the-erp/generate/d07a_mounts.py`. The top band is section 2's table: token, port, network default, role and exact tool list per mount. The red band beneath shows **what each mount refuses**, because the refusals are what define a mount. The bottom frame shows the four checks that apply to all of them: the startup check, the per-request token, the host allowlist, and tool lists read from the routers rather than typed. #### 7.2 A multi-step session under one pin ![A session under one pin](diagrams/argondb-and-the-erp/07b-a-session-under-one-pin.svg) Generator: `docs/diagrams/argondb-and-the-erp/generate/d07b_session.py`. Left to right: taking the pin and its five disclosures; the same `pin_id` on `schema_overview`, `profile`, `query`, `read` and finally `changes(since=…)`; then the five ways a pin is refused by name. The point: step 5 means something only because steps 1–4 were at a position the agent can name. #### 7.3 The budget ![The budget contract](diagrams/argondb-and-the-erp/07c-the-budget-contract.svg) Generator: `docs/diagrams/argondb-and-the-erp/generate/d07c_budget.py`. The five stages (render → shape → estimate → trim → answer), what a budget buys (depth, with anything left out always named), and what it does **not** buy: the row caps, which refuse instead of trimming, and stored statistics, which avoid the scan rather than shorten it. #### 7.4 The Postgres arm and the lake arm ![The Postgres arm and the lake arm](diagrams/argondb-and-the-erp/07d-the-postgres-arm-and-the-lake-arm.svg) Generator: `docs/diagrams/argondb-and-the-erp/generate/d07d_hybrid.py`. Side by side: what each arm can see, what each guarantees about a position, the routing rule in the order it is applied, and two measured incidents that came from forgetting which arm answered. --- ### 8. Further reading - `docs/2026-08-23-mcp-analyst-isolation-design.md` — how the analyst mount is kept read-only, and the alternatives that were considered. - `docs/2026-08-29-mcp-hybrid-lake-only-routing.md` — the routing rule for lake-only tables. - `docs/2026-09-08-mcp-run-transform-hybrid-watermark.md` — the measured race between `query` and `run_transform`, and why a mount is not an arm. - `docs/2026-09-04-erp-spec.md` — the residual gap on role restrictions in the lake arm, and the isolation table whose MCP row is the per-company token. - `docs/2026-09-07-separated-stack-baseline.md` — how ArgonDB's agent performance is compared against a standard `postgres-mcp` server on the same database. The code is in `crates/argondb-mcp/src/`: `lib.rs` (the mounts, routing, host allowlist and startup checks), `dba.rs`, `write.rs` and `admin.rs` (one per mount), `budget.rs`, `token.rs` (pins), `delta.rs` (`changes()`), `profile.rs`, `cards.rs`, `accel.rs`, `company_token.rs` (the per-company token), and the tool descriptions `publish-status.tool.txt` and `transform-pin-serves.tool.txt`, which live in their own files so two mounts cannot drift apart. --- ## Guide 8 — building an application cloud on ArgonDBCloud: the ERP's configuration system and ERPCloud You sell a business application that runs on ArgonDB — an ERP, say. Your customer wants it running in **their own** AWS or Azure account, and they want to know what it will cost before anything is bought. You do not want to write a second installer, a second price calculator and a second set of questions just for your application. **ArgonDBCloud** is ArgonDB's installer and cost advisor for AWS and Azure. It asks a few questions, prices every sensible way of running ArgonDB for the answers, recommends one, writes a plan, and builds that plan in the customer's account. Your application plugs into it with one document, an **overlay**, that adds your own questions and your own cloud resources. The customer then gets one set of questions, one combined estimate and one install — for ArgonDB and your application together. That is what this chapter calls an **application cloud**, and ERPCloud is the ERP's. This chapter is for the engineer who has to build the application's side: the overlay, the configuration system, and the customer's flow. It says what you can use today and what is still being built, and it ends with the questions we need you to answer. **What you stop doing by hand.** You stop writing your own cloud price calculator and keeping it current. You stop asking the customer the same question twice — once for the database and once for the application. You stop writing configuration files per customer by hand; both sides are derived from one plan. --- ### 1. What ArgonDBCloud is ArgonDBCloud walks a user through seven steps: say what you need, see the alternatives with dollar figures, see which one is recommended, choose, read the plan, apply it, and get the connection details. Nothing is created in the account until the user types "apply". Two things it is not: - **Not a trial run.** The dollar figures are a model built from the cloud providers' list prices and ArgonDB's measured resource use. Nothing is deployed to measure them. - **Not a hosted service.** Everything it creates lives in the user's account, under the user's credentials. ArgonDBCloud keeps nothing. #### 1.1 The three shapes Every ArgonDB box runs the same set of processes. The three shapes differ in **where** those processes run and **where the data lives**. | | (a) one VM, local disk | (b) one VM + object storage | (c) one instance per tenant | |---|---|---|---| | where a company's database runs | on the one VM | on the one VM | on that company's own instance | | data survives the VM dying | no, unless you keep snapshots | yes, it is in the bucket | yes | | does stopping idle companies save money | no | no | yes: the instance stops | | typical customer | a few companies, a trial | a few companies, durable | many companies | - **(a)** puts everything on one virtual machine and its disk. Cheapest, and the least durable. - **(b)** is the same machine, but the database's storage, its write log backups and its analytical lake live in the customer's bucket, so the machine can be replaced without losing data. - **(c)** gives each company (each **tenant**) its own instance, running that company's database, its pooler entry, its share of the analytical work and **its copy of your application**. A small set of shared instances runs the storage, the write-log quorum across three data centers, and the control plane. #### 1.2 Why ERPCloud is shape (c) An ERP customer is a group of companies, and each company is a tenant. Shape (c) is the only one where each company's database **and its ERP process** run on that company's own machine, and the only one where an idle company costs less: its instance stops, and only its disk keeps billing. When somebody comes back to a stopped company, a small "waker" on the shared instances holds their connection, starts the company's instance, and passes the connection through. On a single box today that wake takes about **0.6 seconds**. In the cloud it is the time to start an instance, **measured at 13 seconds** (four wakes, two live runs) — and the company's database starting on that instance will add its own time once it runs there. The advisor reads the measured figure and shows it beside the saving. Idle-stop is off unless the customer turns it on. Shapes (a) and (b) are the small-customer forms: a handful of companies on one machine. Per-company instances have a floor cost per company, so the advisor rarely recommends (c) for a few companies. The rule it uses is "cheapest that fits": the cheapest shape that gives the customer everything they asked for, with room for a year of growth. #### 1.3 What the customer can install today The advisor shows every shape with its price, even one it cannot install yet; such a shape is marked **NOT YET INSTALLABLE**, with what is missing. Today the tool can price all three shapes. It cannot install any of them yet: the install half is the next stage of work (section 6). --- ### 2. The contract you build against #### 2.1 Four calls, seven documents ArgonDBCloud is a library first. The command-line tool, the later web application and your ERP installer are all thin callers of the same four calls, which pass the same versioned JSON documents: ``` needs ──▶ advise() ──▶ alternatives (every shape priced; one recommended) choice ──▶ plan() ──▶ plan (saved, reviewable; nothing created) plan ──▶ apply() ──▶ apply-result (resumable; writes the state) state ──▶ teardown() ──▶ teardown-result ``` The life of one install is **needs → advise → alternatives → choice → plan → apply → state → teardown**. | document | what it says | |---|---| | `needs/v1` | how big, how busy, which features; only the cloud and region are required, and anything left blank is estimated | | `alternatives/v1` | each shape as a priced bill, monthly and yearly, with its assumptions and limits, and which one is recommended | | `choice/v1` | which alternative the user picked, and any overrides | | `plan/v1` | every resource to create, in order, with its cost | | `apply-result/v1` | what was created, what failed, and how to resume | | `state/v1` | the record of the install: what exists, its ids and its outputs | | `overlay/v1` | your application's questions and resources (section 3) | Every document starts with its own `"schema": "/v"` field. A reader refuses a major version it does not know and ignores fields it does not know within a version it does. The JSON Schema files, each with an example, ship inside the `argondbcloud` crate under `schemas/`. #### 2.2 What you can call today - **`advise` works.** Give it a `needs` document and a price book, and it returns the priced alternatives and the recommendation for all three shapes — with your overlay's lines in every one of them when the `needs` carries one. - **`plan`, `apply` and `teardown` work for shape (a)**, the one-VM shape, on AWS and Azure. `plan` writes the ordered list of resources an install will create, each with its token; `apply` creates them and keeps a state file it can resume from; `teardown` removes exactly what that file says was created. Every one of them has been proven against a RECORDED cloud — a tape of calls and replies — and not yet against a real account: that is the next stage. Shapes (b) and (c) are priced but cannot be planned yet. #### 2.3 Two ways in 1. **Link the Rust crate** `argondbcloud`. The documents become typed values, and a schema change breaks your build instead of a customer's install. 2. **Drive the command-line tool** `argondbcloud` from any language, with `--json` so it prints documents instead of screens: ``` argondbcloud advise needs.json --json --overlay overlay.json # the alternatives, with your lines argondbcloud advise needs.json --json --argondb-version 1.5.7.17 # the same, for the ArgonDB version you name (one of `argondbcloud supported`; any other is refused by name) argondbcloud tui --overlay overlay.json # the customer's screens argondbcloud plan needs.json --shape a --json # the plan: ordered resources, tokens, saved argondbcloud plan needs.json --shape a --argondb-version 1.5.7.17 --json # the plan pins that ArgonDB release (its download and checksum); outside the supported list the tool stops with the list printed and writes nothing argondbcloud supported # the ArgonDB versions this tool installs, newest first; the first is the default argondbcloud plan --alternatives alts.json --shape a --token-seed --json # the same plan every time: tokens made from the seed, so one recorded tape fits it argondbcloud plan needs.json --shape b --json # shape (b): the bucket, the VM's role, the endpoint, a VM that boots ArgonDB against the bucket argondbcloud plan needs.json --shape b --restore-from --name # a NEW VM from an existing install's bucket (the bucket is adopted, never re-created) ARGONDBCLOUD_PROVIDER=live argondbcloud plan needs.json --shape b --json # with your account in reach, the plan first reads its service quotas (on-demand vCPUs, Elastic IPs, hosted zones, gp3 storage, buckets) and prints one `quota:` line per read — what the account allows, what is in use, what this plan asks; a plan that would not fit stops here, names the quota and the command that raises it, and writes nothing (ArgonDBCloud 1.0.3). A quota it is not allowed to read is printed NOT READ and the plan goes on; --override quota=skip prints the overrun instead of stopping. argondbcloud apply plan.json --json # build it (a recorded or a live provider) argondbcloud apply plan.json --replay tape.json --json # build it against a recorded tape (--record tape.json writes one) argondbcloud apply --resume --json # continue after a stop; nothing is made twice argondbcloud census # what the cloud holds under the install's tag right now argondbcloud requests # shape (b): the bucket's request count against the model argondbcloud destroy --confirm # shape (b): terminate the VM outside the engine (the zone-loss drill) argondbcloud backup --state --label nightly # shape (b): a named restore point — the bucket's current versions and a snapshot of the data volume, recorded without copying data; `census` lists the points and marks one older than the bucket's 7-day window as expired argondbcloud restore --from-state --point --name --dry-run # what a restore would build and bill, and roughly how long until the new install answers; nothing is created argondbcloud restore --from-state --point --name # a NEW install from the point: its own bucket (the point's view of the old one, copied), its own VM and disk (from the snapshot); the old install is never touched argondbcloud teardown --confirm # remove exactly what was built (--keep-data leaves the bucket and the disk and prints what they bill) argondbcloud upgrade --state --to 1.5.7.19 # shape (b): move a running install to a newer ArgonDB; a DRY RUN by default (the plan, the downtime estimate, the one non-reversible step named), the real swap with --yes, recorded in the state file argondbcloud validate overlay.json # check any document against its schema argondbcloud -V # the version and the build it came from ``` (Thirteen verbs and `-V`. The developer guide's list of them is now checked against the tool's own source by a test, so the two cannot drift apart again; `--help` on any verb shows every option. `backup` and `restore` arrived on 2026-10-28 and were exercised once on a real account: a point taken on a live install, more rows written after it, and a new install restored from the point that held exactly the point's rows — the later ones absent — and answered in under four minutes. A restore to an arbitrary timestamp is refused by name: the tool restores named points.) `apply`, `teardown` and `census` need a provider to talk to: a recorded tape (for tests — two ship with the tool, for shapes (a) and (b) on AWS, with the seed that makes a matching plan), or the live AWS client in a build carrying the `live-aws` feature, which has deployed shape (a) and shape (b) into a real account (2026-10-21 and 2026-10-23). With neither they stop with a message and print nothing else — no document to misread. After an apply the install's public address is in the output; after a teardown each removed resource is marked deleted, not "planned". Azure's live client is built and has not yet met an account. A failed step is undone on its own and the output names the one command to resume and the one to tear down; the data volume, the bucket and the snapshots are removed only after you type the install id. `upgrade` is the first day-2 operation (ArgonDBCloud 1.0.3): without `--yes` it only prints what it would do; with it the new release is fetched and checked on the machine, the service stopped, the release swapped (the one step that cannot be undone), the service started and the estate migrated — about half a minute of downtime measured on the October rehearsal — and the state file records every step. If your installer is written in Rust, link the crate. Otherwise drive the tool with `--json`, treat the schema files as the interface, and pin the ArgonDBCloud version you test against. --- ### 3. The overlay — where your application plugs in #### 3.1 What an overlay is An overlay is one JSON document your application hands to ArgonDBCloud inside `needs`. It has four parts: - **Questions** — what your customer must be asked, the unit, and what to assume if they skip it. They appear in the same screens as ArgonDB's own questions, and a skipped answer is shown with the assumption behind it. A question is a number, a text (`kind: string`) or a secret (`kind: secret`). **Every answer is recorded** (from ArgonDBCloud 1.0.4, 2026-10-29): the plan's `needs_resolved.overlay_answers` and the state's `overlay_answers` carry one flat object keyed by question — a number as a number (the customer's, or the assumption the install was sized with), a text as a string, and a secret as its NAME over `{"secret": true}`, never its value — and `plan.md` and the plan screen print them on one `answers:` line, a secret by its name only. Until 1.0.4 only text answers were recorded and every number was used for the price and then dropped; the 2026-10-28 rehearsal of the ERP's own document found its two answers in neither plan nor state. - **Resources** — what your application needs in the cloud. Each one is either **per tenant** (one per company) or **per install** (one for the whole customer). In shape (c) a per-tenant resource like your application process is **added to each company's instance size** rather than becoming a separate machine, because your process runs on that company's instance. - **Permissions** — what your processes need from the cloud, per cloud. - **Install lines** — what to run on an instance to install your application. Through ArgonDBCloud 1.0.3 the installer reads these lines and appends nothing to the machine (measured on 2026-10-28 with the ERP's own document); a 1.0.4 row makes it append them after its own install block and say which placeholders it fills in. ArgonDBCloud prices your resources with the same price data and folds them into every alternative, so the customer sees **one combined estimate**. The plan builds the resources whose type ArgonDBCloud knows how to build, in the same install as ArgonDB's, so the customer gets **one install**. Today that means object storage — an S3 bucket on AWS, or a Blob container on Azure — and, since 2026-10-27, the install's **front door**: a DNS record in a Route 53 zone (one the install creates, or an existing one it adopts by tag and never overwrites), a TLS certificate from ACM validated through a record the install places itself, and a Cloudflare tunnel made with an API token you hand over at apply time and that is written nowhere. Any other resource, such as an email service or a queue, appears in the plan as a named row marked "not buildable", and your installer sets it up itself; on Azure the DNS and certificate halves are such rows until ArgonDBCloud 1.1. If you give the overlay a `version`, it is recorded in the plan and in the install's state, so later you can tell exactly what was installed. **What works today.** The advisor checks the overlay against its schema, prices it, and folds it into every alternative. A few rules to write it by: - a resource's `quantity` is one of your question keys or a plain number; - a question's `infer` is a number, an ArgonDB input (`tenants`, `data_gb`, `growth_gb_per_month`, `query_rate`), or two of those multiplied; - a resource with a `demand` (CPU and memory) is added to the machine's size and costs nothing on its own line. A per-tenant one counts once per company. A per-install one counts once, on the one machine, or on the shared estate machines in shape (c); - only object storage is priced from the cloud's price list so far: an S3 bucket on AWS, a Blob container on Azure, per GB-month. For anything else you can give your own monthly price with a one-line reason, and it is shown as "HAND-PRICED". Without one it appears as "price unavailable", and the alternative is marked partial; - the permissions you name for a bucket are given to the machine's own cloud identity in shape (b), never more than that identity already has. Shape (a) machines have no cloud identity yet, so there the plan records the permission and does not grant it. What that means for the ERP's own installer, now that this part has landed (2026-10-24): on a shape (a) install the ERP's sidecar reaches its documents bucket only with credentials of its own until the 1.0 cut gives (a) machines a cloud identity. The email relay and the tunnel stay things the engine will not build, because it creates nothing it cannot probe and delete; their monthly cost is a stated gap unless the ERP's overlay gives them a price of its own, which is a change to the overlay, not to ArgonDBCloud. A queue, a relay or a tunnel would each need a new kind of thing the engine knows how to build, and none has been invented. #### 3.2 A worked ERP overlay The only ERP piece that is fully specified today is the **ERP process itself**: one process per company, under its own user, on that company's instance, started by ArgonDB from a command recorded when the company is created. A browser reaches a company's ERP through that process's port. The document store, the queue, the email relay and the web front end below are **placeholders**: they are what the ArgonDBCloud design expects an ERP to need, and we need your real list (section 7). The process size (half a CPU, 1 GiB) is an example figure, not a measurement. **Since 2026-10-27 the block also carries the four things you do by hand today on splink-server — the reference data, your schema migrations, the Splink estate tenant and the first Administrator — as `run-after-install` resources**: programs your installer ships, run on the estate machine after it answers, with the estate's connection string in their environment (section 3.3). The queue, relay and web placeholders of the earlier block were dropped; your own list replaces placeholders, not this guide. ```json { "schema": "overlay/v1", "name": "splink-erp", "version": "2026.10.29", "questions": [ { "key": "documents_gb", "text": "How many GB of scanned documents does each company keep?", "unit": "GB", "infer": "5", "basis": "5 GB of documents per company (PLACEHOLDER: Splink's number)" }, { "key": "erp_release", "kind": "string", "text": "Which ERP release does this install run?", "unit": "", "infer": "7df00239", "basis": "the ERP's current cut (PLACEHOLDER: the installer's own)" }, { "key": "estate_tenant", "kind": "string", "text": "The Splink estate tenant's name?", "unit": "", "infer": "splink", "basis": "the estate tenant the ERP's Users live in" }, { "key": "admin_email", "kind": "string", "text": "The first Administrator's email?", "unit": "", "infer": "admin@example.com", "basis": "a placeholder until you name one" }, { "key": "admin_password", "kind": "secret", "text": "The first Administrator's password?", "unit": "", "infer": "", "basis": "given to the first-Administrator step by environment on the VM" }, { "key": "cf_account", "kind": "string", "text": "The Cloudflare account id the tunnel lives in?", "unit": "", "infer": "PLACEHOLDER", "basis": "the ERP's Cloudflare account (ids only; the token is the secret below)" }, { "key": "cf_zone", "kind": "string", "text": "The Cloudflare zone id of splink.app?", "unit": "", "infer": "PLACEHOLDER", "basis": "the zone the tunnel's hostname is a record of" }, { "key": "cf_hostname", "kind": "string", "text": "The tunnel's public hostname?", "unit": "", "infer": "erp.splink.app", "basis": "the ERP's host name" }, { "key": "cf_direct_hostname", "kind": "string", "text": "The DNS-only name a client that must reach the box directly resolves?", "unit": "", "infer": "erp-db.splink.app", "basis": "PLACEHOLDER: the ERP's name for direct reach (the conformance gate, an operator's psql); an unproxied A record in the same zone at the install's address" }, { "key": "cf_token", "kind": "secret", "text": "The Cloudflare API token (Tunnel: Edit, DNS: Edit on the zone)?", "unit": "", "infer": "", "basis": "handed at apply time (argondbcloud apply --answers); written nowhere; its custody is OPEN, the project owner's" } ], "resources": [ { "id": "erp-sidecar", "per": "tenant", "provider_type": { "aws": "ec2-instance-share", "azure": "vm-share" }, "demand": { "vcpu": "0.5", "ram_gib": "1.0" }, "note": "the ERP application, one process per company on the tenant's own instance" }, { "id": "erp-documents", "per": "tenant", "provider_type": { "aws": "s3-bucket", "azure": "blob-container" }, "quantity": "documents_gb", "unit": "GB-month", "note": "PLACEHOLDER: one bucket for the install, one prefix per company (//)" }, { "id": "erp-tunnel", "per": "install", "provider_type": { "aws": "tunnel", "azure": "tunnel" }, "provider": "cloudflare", "account": "cf_account", "zone": "cf_zone", "hostname": "cf_hostname", "service": "http://localhost:8443", "token": "cf_token", "direct_record": { "name": "cf_direct_hostname", "ttl": 300 }, "price": { "monthly_cents": 0, "basis": "PLACEHOLDER: Cloudflare Tunnel is free on the plan the ERP holds; name the plan's price here" }, "note": "the ERP's front door under the zone ruling of 2026-10-28: the Cloudflare tunnel to splink-server's successor — the tunnel, its ingress and the hostname's CNAME in the Cloudflare zone (the hostname route IS the DNS record; TLS terminates at Cloudflare; no Route 53 zone, no ACM certificate) — plus the optional direct_record: a DNS-only A record at the install's address for a client that must reach the box directly (the conformance gate, an operator's psql); the connector on the VM is a run-after-install step the ERP's installer may add (§3.3.1)" }, { "id": "erp-reference-data", "per": "install", "provider_type": { "aws": "run-after-install", "azure": "run-after-install" }, "script": "#!/bin/bash\n# PLACEHOLDER (the ERP's): load the reference data the ERP's installer put on\n# this VM (user_data_append) into the estate, through the DSN this step is given.\nset -eu\nREL=\"${1:?erp release}\"\nF=/opt/splink-erp/reference-data/$REL.sql\n[ -r \"$F\" ] || { echo \"no reference data for release $REL at $F (the ERP's installer ships it)\" >&2; exit 2; }\n\"$ARGONDB_PSQL\" \"$ARGONDB_DSN\" -v ON_ERROR_STOP=1 -f \"$F\"\necho \"reference data $REL loaded into the estate ($ARGONDBCLOUD_INSTALL_ID)\"\n", "args": [ "${erp_release}" ], "idempotency": { "key": "reference-data-${erp_release}" }, "timeout": 900, "order": 10, "note": "replaces the hand reference load on splink-server" }, { "id": "erp-migrations", "per": "install", "provider_type": { "aws": "run-after-install", "azure": "run-after-install" }, "script": "#!/bin/bash\n# PLACEHOLDER (the ERP's): the ERP's OWN schema migrations against the estate's\n# DSN. ArgonDB's own migrations are the component's (`argondb estate migrate`,\n# run by the box at boot and by `upgrade`); these are the application's.\nset -eu\nREL=\"${1:?erp release}\"\n/opt/splink-erp/bin/erp-migrate --dsn \"$ARGONDB_DSN\" --to \"$REL\"\necho \"ERP schema at release $REL\"\n", "args": [ "${erp_release}" ], "idempotency": { "key": "migrations-${erp_release}" }, "timeout": 1800, "order": 20, "note": "replaces the hand migration on the box" }, { "id": "erp-estate-tenant", "per": "install", "provider_type": { "aws": "run-after-install", "azure": "run-after-install" }, "script": "#!/bin/bash\n# PLACEHOLDER (the ERP's): the Splink estate tenant, created once and listed in\n# the tenant census; the flags as the ERP's installer spells them.\nset -eu\nT=\"${1:?estate tenant}\"\nif \"$ARGONDB_BIN\" tenant list --data-dir \"$ARGONDB_DATA_DIR\" 2>/dev/null | grep -qx \"$T\"; then\n echo \"estate tenant $T already stands\"\nelse\n \"$ARGONDB_BIN\" tenant create \"$T\" --data-dir \"$ARGONDB_DATA_DIR\"\n echo \"estate tenant $T created\"\nfi\n\"$ARGONDB_BIN\" tenant list --data-dir \"$ARGONDB_DATA_DIR\"\n", "args": [ "${estate_tenant}" ], "idempotency": { "key": "estate-tenant-${estate_tenant}" }, "timeout": 600, "order": 30, "note": "replaces the hand tenant create; the tenant census lists it" }, { "id": "erp-first-administrator", "per": "install", "provider_type": { "aws": "run-after-install", "azure": "run-after-install" }, "script": "#!/bin/bash\n# PLACEHOLDER (the ERP's): the Splink estate tenant's Users and the first\n# Administrator. UNTIL the in-engine authorizer's Administrators path lands\n# (lane 3 of docs/2026-10-24-in-engine-authorizer-design.md) this step acts\n# under the BOX TOKEN path: ARGONDB_ADMIN_AUTH_TOKEN comes from\n# /etc/argondb/box.env, sourced into this environment by the engine's wrapper.\n# The password is a SECRET answer: it reaches this step as the argument the\n# wrapper expands from ARGONDBCLOUD_SECRET_ADMIN_PASSWORD on the VM, and is in\n# no plan, state, screen, tape or log.\nset -eu\nT=\"${1:?estate tenant}\"; EMAIL=\"${2:?administrator email}\"; PASSWORD=\"${3:?administrator password}\"\n/opt/splink-erp/bin/erp-admin create --dsn \"$ARGONDB_DSN\" --tenant \"$T\" --email \"$EMAIL\" --password-stdin <<<\"$PASSWORD\" --box-token \"$ARGONDB_ADMIN_AUTH_TOKEN\"\necho \"first Administrator $EMAIL of tenant $T\"\n", "args": [ "${estate_tenant}", "${admin_email}", "${admin_password}" ], "idempotency": { "key": "first-administrator-${estate_tenant}-${admin_email}" }, "timeout": 300, "order": 40, "note": "replaces the hand Users insert; written against the box token path until authorizer lane 3 lands" } ], "permissions": { "aws": [ "s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket on erp-documents" ], "azure": [ "Storage Blob Data Contributor on erp-documents" ] }, "user_data_append": "PLACEHOLDER: the lines that install the ERP's package at a pinned release under /opt/splink-erp (its binaries, its reference data, its migrator) and the sidecar at the same release" } ``` **The front-door line** (`erp-tunnel`) is the only ingress line this overlay carries: a Cloudflare tunnel whose account, zone and hostname are questions the customer answers, whose token is a secret handed on `apply`, `teardown` and `census` with `--answers needs.json`, whose `service` is the port the tunnel fronts on the machine, and — since ArgonDBCloud 1.0.4 — whose optional `direct_record` names the DNS-only A record in the same zone at the install's address, for the one client that must reach the machine directly (a conformance gate, an operator's `psql`); the worked overlay names it through the `cf_direct_hostname` question. `plan` prints the overlay's front door in one line, in the plan document, on the screen and in `--json`: ``` ingress: tunnel (cloudflare) — hostname routes are the DNS records; TLS terminates at Cloudflare; no Route 53 zone, no ACM certificate; DNS-only A record erp-db.splink.app for direct reach ``` What this overlay still does not do: install the connector on the machine — your installer may add that as a fifth post-install step. **Which of the three you write — decided 2026-10-28.** The project owner ruled that the ERP's domain stays in Cloudflare: the install's front door is the Cloudflare tunnel, its hostname routes are the DNS records, TLS ends at Cloudflare's edge, and where a client must reach the machine's address directly a DNS-only A record names it. His own reason: the ERP already creates a subdomain for every distributor and manufacturer it onboards, so that the application looks credible — "why does AWS need to be involved at all in the domain?" So if your application's front door is a tunnel, write the `erp-tunnel` line and nothing else for ingress — no hosted zone, no `dns-record`, no `tls-certificate`. Your tenant subdomains stay yours, made the way you make them today with your own Cloudflare token; the installer routes only the install hostname you put on the tunnel line. Write a record and a certificate only when your DNS lives on Route 53 and your clients reach the machine directly; then the record is the install's address and the certificate is issued by DNS validation in that zone, as built; then the plan's line reads `ingress: route53 + acm — , `. The worked overlay above carries the tunnel and nothing else since 2026-10-29 (it showed `erp-dns` and `erp-cert` from before the ruling until then), and the ERP's own document already carried neither. One shape is refused by name, both when you validate the overlay and when you plan it: a certificate whose domain is your tunnel's hostname (it would never be served, because TLS ends at Cloudflare), or a record for that name (the tunnel's route already is the record). The refusal names both resources and the ruling; an overlay that carries neither kind is accepted silently, as before. **And the three kinds you do not write stay — decided 2026-10-29.** The project owner ruled that the installer's hosted-zone, `dns-record` (Route 53) and `tls-certificate` (ACM) kinds are kept as built and documented as not for the ERP's overlay; a customer whose DNS is on Azure or Route 53 is the row that exercises them. Nothing that works and is taped is removed, and nothing is built for a customer who has not asked; the Azure halves of those kinds wait for such a customer. He also ruled who makes the DNS-only A record for direct reach: the installer's tunnel kind, as an optional arm under the same apply-time token — planned, probed, counted and torn down with the tunnel, tape-only until production — so the record's lifetime is the install's and nothing is left for a hand or a second automation. It was built on 2026-10-29 as the tunnel's `direct_record` arm, proven on recorded tapes with and without it (nothing has run against a live Cloudflare account yet), and ships in ArgonDBCloud 1.0.4 with the rows that make the install lines reach the machine and record your answers (below). We ran the 2026-10-04 form of this overlay (the five placeholder resources, before the four `run-after-install` steps, which add no-charge lines and change no figure, and before the three front-door lines, which add a hosted zone at $0.50 a month when the install creates one, a record's queries at $0.40 per million, a certificate at $0.00 with its basis — ACM issues public certificates at no charge — and the tunnel at the price you name) through `argondbcloud advise`, offline, with recorded AWS prices, for 20 companies in us-east-1 and nothing else given: - **The ERP process moves the estimate most.** On one machine — shapes (a) and (b) — 20 companies × half a CPU adds 10 CPUs, which takes shape (a) from about $168 to about $611 a month. In shape (c) it fits inside each company's own instance. That half-CPU figure is the number we most need you to measure. - **The documents store is priced:** 5 GB × 20 companies = 100 GB, $2.30 a month. - **The queue, the email relay and the web front end are not priced** yet, so every alternative is marked partial. - **Twenty companies turn on "keep running through a failure"** by default (it switches on from 10 companies), which rules out shapes (a) and (b); the recommendation was (c), at about $3,220 a month for the priced lines. A customer who does not need that says no on the "What you need" screen. - **One company, with backups, reads (b)** — and (b) is the production shape for the first migration (ruled 2026-10-27; (c) is the growth path, and every recommendation now says so in one sentence). The same overlay for one company priced (a) at about $155, (b) at about $159 and (c) at about $777 a month; (b) is within 5 % of (a) and wins as the more durable shape. Its plan carries a `capabilities` list — `lake-in-bucket` once the install's ArgonDB is 1.5.7.19 or later, meaning the analytical copy of the data lives in the install's bucket and comes back with the database on a replacement machine — and a daily snapshot of the data disk beside the bucket's versioning. - **Read each alternative's "partial" flag**, not the document's top-level "incomplete" flag, which read false in this run. We also checked the document against the overlay schema with an independent validator, alone and inside a `needs` document: both pass, and a resource with `"per": "company"` is refused because only `tenant` and `install` are allowed. The block above, with its four steps, was checked the same way on 2026-10-27 and with `argondbcloud validate` (`overlay/v1: valid`); a `timeout` above an hour is refused. #### 3.3 Steps that run after the estate is up Until now an overlay could only declare things the plan prices and the apply creates — a bucket, a container, a line it cannot build. Nothing in it could RUN after the estate was up, so loading reference data, running your migrations, creating the estate tenant and the first Administrator were hand scripts on a box. A **`run-after-install` resource** is that hand step as a declared part of the plan: - **What you write.** `"provider_type": { "aws": "run-after-install", "azure": "run-after-install" }`, `"per": "install"`, the program's text inline in `script` (at most 16 KiB — the overlay is one document, so the plan carries the whole program), `args` (plain strings, or `${question key}` to pass an answer), `idempotency` with a `key`, a `timeout` in seconds (an hour at most; ten minutes when absent) and an `order` (lowest first; two steps with the same order are refused by name). It costs nothing and shows as a no-charge line naming the script by size and checksum. - **What happens.** The plan places every step after the estate answers on its port (`box-ready`), after any bucket the overlay declares, and after the step before it — and BEFORE the overlay's front door (the hosted zone, the record, the certificate and its issuance wait, the tunnel and its direct-reach record), so a certificate that is slow or refused never leaves your steps unrun: the reference load and the ingress are independent work on a migration night (ArgonDBCloud 1.0.3; the 1.0.2 planner put the steps after the certificate wait, and the October rehearsal measured all four left waiting behind a refused issuance). A step that genuinely needs the certificate says so with `"after": [""]`; that step, and every step ordered behind it, then waits for the issuance. `after` may name only an ingress resource of the same overlay, and is refused by name otherwise. `apply` runs each step ON the estate machine, as root, through the cloud's own command channel (AWS Systems Manager on AWS; the Azure half is written but not yet run) — never an ssh key the installer makes. The program finds the estate's connection string in `ARGONDB_DSN`, the estate's own `psql` and CLI in `ARGONDB_PSQL` and `ARGONDB_BIN`, the install id, and every number or text answer as `ARGONDBCLOUD_ANSWER_`. - **Secrets.** An answer of kind `secret` (a password, a token) that a step's `args` name reaches the step ONLY as an environment variable on the machine, read from the root-only file the first boot wrote; it is in no plan, state, screen, tape or log. What the program itself prints is its own: a script that echoes its secret puts it in its log. - **The record.** `state/v1` keeps, per step, the key it ran under, the exit code, when it started and finished, and the path of its captured output (`~/.argondbcloud/installs//post-install/.log`; AWS returns at most 24,000 characters per stream, the whole log stays on the machine) — never the arguments. Exit 0 is `created`. A non-zero exit or a timeout stops the apply at that step, names it and the log, and `apply --resume` runs ONLY that step again; a step already created under the same key is not run again. Tearing the install down marks every step `deleted` without touching the cloud. - **Write every step so it can run twice.** The engine guarantees a step is not run twice under one key; it cannot guarantee a run that was killed between the program and its record is not run again. `create table if not exists`, `on conflict do nothing`, an "already there" check — the four steps in the block are written that way. - **What you provide, what you drop.** Provide the four programs (the block's are the component's worked example, not your code), each reading the connection string from `ARGONDB_DSN` and never a box path; your package on the machine by `user_data_append`; the `secret` question for the first Administrator's password, answered through `apply --answers`. The first-Administrator step acts under the box's own token until the in-engine authorizer's Administrators path lands, and says so. Drop the four hand scripts on splink-server: each is a resource of the plan now, run by `apply` and readable from `state/v1`. #### 3.4 The lines that install your application: `user_data_append` The overlay's `user_data_append` is shell text the machine's FIRST BOOT runs, as root, AFTER the engine's own block has installed ArgonDB, started it and seen it answer, and after your overlay's secret answers have been written to `/etc/argondbcloud/secrets/`. It is where your application puts its own package on the machine — the ERP's six lines: make a directory, unpack the tarball at a pinned release, run the render step, install the systemd unit, start it — which the steps of §3.3 then call through the connection string. When your document declares no step, this field is your whole install path. **Until ArgonDBCloud 1.0.4 (2026-10-29) the engine validated this field and appended nothing.** The production rehearsal of 2026-10-28 measured the ERP's six lines reaching neither machine. From 1.0.4 `apply` appends them, and this is the contract: - **Exactly two placeholders are substituted:** `${install.id}` (the install id, `-<6 hex>`, what every resource is tagged with) and `${install.name}` (the name you chose with `--name`). Any other `${...}` is refused by name when you validate the overlay. So ArgonDB's version is not substituted (it is `argondb_version` in `state/v1`; pin your own release in your own text), the estate's connection string is not substituted (its password is minted on the box — read `/etc/argondb/pg-password` there, as a step's wrapper does), and your answers are not substituted (a step's `args` carry those). Write a shell variable as `$NAME`; a braced form such as `${HOME}` reads as a placeholder and is refused. - **How it runs.** The text is written to `/opt/argondbcloud/overlay-user-data.sh` through a quoted here-document (nothing in it is expanded; a line equal to the delimiter `ARGONDBCLOUD_OVERLAY_USER_DATA_EOF` is refused), run with `bash -e` (the first failing line ends it), its output kept in `/var/log/argondbcloud/overlay-user-data.log`, its exit and moment in `/var/lib/argondbcloud/overlay-user-data.exit`, and the log's last 20 lines plus one marker line printed to cloud-init's output, which reaches the machine's console. `box-ready` is unchanged: it watches ArgonDB's port, which opens before your lines run, so a failing append never leaves it false for the engine's own reasons. - **How it is read back.** The plan carries one more step, `user-data-append`, right after `box-ready`; every post-install step waits on it, so your steps run only once your package is on the machine. On AWS the engine reads the marker from the instance's console output — no agent, no command channel, no extra permission on the machine's role, so a document with no step is read the same way as one with four; on Azure the reader runs through Run Command. It waits up to 900 seconds and refuses by name past that (a resume reads again). The result is recorded twice: as the step's own record and, under the install, as `overlay_user_data` in `state/v1` — the exit, the moments, the engine's copy of the read, the machine's own log path and the last 20 lines — and in the apply result's outputs. A non-zero exit fails the apply by name with the resume command. `plan.md` says in one line how many lines the overlay appends. - **The size bound.** EC2 takes 16 KiB of user data in all, and the engine's own shape (b) block is about 12.7 KB of it, so roughly 3 KiB remain for your lines on (b); Azure's custom data takes 64 KiB. The whole user data is checked against the bound when you plan and again at the instance call, and refused by name past it — fetch a script from a place the machine reaches (the install's own bucket) instead of inlining it. - **An overlay without the field** gets byte-identical user data, no `user-data-append` step and no `overlay_user_data` record. **What the ERP's six lines may rely on, and what they may not.** They reach the machine and run after ArgonDB is up, as root, with `${install.id}` and `${install.name}` filled in, and their exit is recorded. They may NOT rely on `~/.argondbcloud/installs//state.json` being ON the machine: the state file lives on the operator's machine, in the installer's own `~/.argondbcloud`, and nothing copies it to the instance — the render line as the ERP's 2026-10-28 cut wrote it reads a path that does not exist there. Where the render reads the install's answers from on the machine is an open question for the ERP coordinator and the program agent. --- ### 4. Two configuration systems, one source An ERPCloud install configures two things: **ArgonDB** on every instance, and **the ERP** — where its documents, queue and relay are, and which company runs where. #### 4.1 ArgonDB's side Each ArgonDB box reads one file, `argon.toml` in its data directory, and refuses keys it does not know. **`argondb config check --data-dir `** reads the file and prints what the box would do with it, without starting anything; it fails with the same message the box would. An installer should run it on every file it writes. The settings an ArgonDBCloud install writes: | setting | what it does | default | |---|---|---| | `[storage] remote` | puts the database's storage in a bucket: `s3://bucket/prefix` or `azure://container/prefix`; a bad value stops the box with a message | off: local disk | | write-log backup | when `remote` is set, the write log is backed up to the same bucket too; if the bucket cannot be reached at startup, the box refuses to start. It follows `remote` and has no switch of its own | off, because `remote` is off | | `[storage] sole_owner` | the installer's statement that this box alone writes its bucket prefix, which lets it delete old storage files there; checked at every start against an owner marker in the bucket, and refused without a bucket | `false` | | `[lake] scheme = "azure"` and the `adls_*` settings | puts the analytical lake in Azure Data Lake Storage instead of S3 | the S3 lake | | `[compute] per_tenant` | one database process per company, each with its own port and idle clock | off | | `[compute] idle_stop` | stop a database process after this many idle seconds, and wake it on the next connection; written as a string, `"300"` | `"off"` | **Which release has them.** Every ArgonDB Linux release since v1.5.7.4 has all five — the write-log backup, `sole_owner`, the Azure lake settings, `idle_stop` and `per_tenant` (the current download is v1.5.7.18). All of these are off unless set. **ArgonDBCloud itself ships from its own channel since 1.0.0 (2026-10-24):** `https://releases.argondb.ai/argondbcloud/`, whose `LATEST` names the ArgonDB version the installer installs by default (`supported_argondb`, pinned by checksum — v1.5.7.16 at the 1.0.0 cut, v1.5.7.18 at the 1.0.1 cut of 2026-10-27, which also installs v1.5.7.17 and v1.5.7.16 when you name one with `--argondb-version`); a copy still rides inside every ArgonDB Linux download. **What is not a setting yet.** Running each company on its own cloud instance, waking a stopped instance on connect, and creating an instance per company are still to be built. ArgonDB's directory of companies already has the empty columns for them: which instance, which zone, which address. #### 4.2 The ERP's side The ERP's configuration needs at least: 1. **Which company runs where.** ArgonDB already keeps this, in its estate directory: one row per company with its database, its connection entry, its ERP command, its status and, in the cloud, its instance, zone and address. **Read it; do not keep a copy.** No service should work out a company's location from a naming convention. 2. **The ERP process's own settings.** Its start command and user live in the same row; its connection string is the company's pooled connection string. 3. **Where each company's documents go.** A bucket and a prefix per company — a placeholder until you name the store. 4. **The queue and the email relay.** Their addresses — placeholders too. #### 4.3 The rule we propose: both sides come from the plan **Proposed:** neither side has a configuration file the customer fills in. The customer answers ArgonDBCloud's questions and your overlay's questions once. Those answers become one `needs`, one `plan` and, after the install, one `state`. Then: - **ArgonDB's `argon.toml`** on each instance is written from the plan and the state: the bucket the install created becomes `[storage] remote`; the installer creating that bucket for one box sets `sole_owner = true`; the cloud chooses the lake setting. - **The ERP's configuration** is written from the same state: the documents bucket, the queue address and the relay address are outputs of your overlay's resources. Which company runs where is read from ArgonDB's directory when the ERP needs it, never from a file. - **The overlay's install lines** are where that writing happens on each instance. - **Your own answers, by path** (built 2026-10-29, ArgonDBCloud 1.0.4): the render reads `state.overlay_answers.` — for the ERP's document, `state.overlay_answers.documents_gb` is the number `5` and `state.overlay_answers.emails_per_day` is `50`; a text question's key holds its text; a secret's key holds `{"secret": true}` (the value is on the machine under `/etc/argondbcloud/secrets/` and in no document). The same object is `plan.needs_resolved.overlay_answers`, and the plan's `overlay.questions[]` says which answer was given and which assumed. So a customer never configures anything twice, and reinstalling from the same plan gives the same configuration on both sides. **Where this stands.** The `state` document is written by `apply` for shape (a): one record per resource — planned, creating, created or failed — at `~/.argondbcloud/installs//state.json`, rewritten after every step. The copy in the install's bucket comes with shape (b). Every resource is tagged with the install id, so a lost state file can be rebuilt and nothing untagged is ever touched. The first deploy into a real account is the next stage of work. #### 4.4 Versions and upgrades - **ArgonDBCloud has its own version number**, 1.0.0 since 2026-10-24 and 1.0.1 since 2026-10-27, installing ArgonDB 1.5.7.18 by default (the Linux release current at the cut, pinned by checksum). The documents have their own versions, which change separately from the tool's. - **Each ArgonDBCloud release lists the ArgonDB versions it can install and refuses others by name.** 1.0.1 installs 1.5.7.18, 1.5.7.17 and 1.5.7.16; `argondbcloud supported` prints the list, and `plan --argondb-version` picks one. Asking for any other version stops the tool with the list printed and no plan written, so a mismatch is caught before anything is built. A plan that names no version gets the default. - **Your overlay's version goes into the plan.** We suggest versioning the overlay with the ERP release it installs, so a plan names exactly which ERP it built. - **Upgrades, backups and restores, resizing and monitoring** are the next ArgonDBCloud release, 1.1. Until then, an ArgonDB upgrade is ArgonDB's own job (`argondb estate migrate`), and an ERP upgrade is the ERP's. - **Read documents the way ArgonDBCloud does:** ignore fields you do not know, refuse a major version you do not know. --- ### 5. What the ERPCloud customer sees This is the flow a customer walks, screen by screen, with the places your questions and resources appear. `argondbcloud tui --overlay overlay.json` walks screens 1 to 7 today, your questions included; screens 8 and 9 wait for the install half, so a session ends with the plan saved and nothing built. 1. **Welcome.** "Install ArgonDB in your own AWS or Azure account … Nothing is created in your account until you type "apply"." Pick the cloud. An ERPCloud front end would name the ERP here. 2. **Where.** Pick the region. 3. **How big.** Companies, data today, people at once, hours a day, growth, requests per second; Enter skips any of them. **Your number questions go here** — for example, "How many GB of scanned documents does each company keep?" 4. **What you need.** Backups (yes by default), keep running through a failure, private network only, single sign-on, a throwaway trial. 5. **What I assumed.** Every number, with the reason behind each estimate. Your skipped answers appear here too, with their assumptions, and any line can be changed. 6. **The alternatives.** One column per shape: monthly and yearly cost, what survives a failure, "first query after idle: 13 s (measured)" for (c), the recommended one and any not yet installable. **Each total includes your resources** — one combined estimate. 7. **The plan.** Every resource in order with its monthly cost, the total, and the credentials it will use. Your resources are lines in the same list. "Type "apply" to build it." 8. **Apply.** One line per resource as it is created. On a failure: the cloud's own error, and the commands to resume or tear down. 9. **Done.** The connection string, the state file, the cost and the teardown command. An ERPCloud version would add the ERP's own address. **After the install.** With idle-stop on, a company nobody has used for a while has its instance stopped. The next connection — a SQL client, or a browser going to the ERP — reaches the waker, which starts the instance and passes the connection through. A failed wake answers a SQL client with a "cannot connect now" error, and is planned to answer a browser with "503, retry after". The customer's first page takes about 13 seconds for the instance (measured), plus the database's own start once it runs there — not a second. **Credentials.** The customer signs in with their own cloud tools. On AWS the installer takes on a dedicated role with only the permissions the plan needs; on Azure it creates a service principal limited to the install's resource group. It never stores a long-lived key. --- ### 6. What exists today, and what does not **You can build against now:** - the `needs` and `overlay` documents and their schemas; - `advise` with your overlay, linked as a crate or through `argondbcloud advise --overlay … --json` — object storage priced, everything else shown as unpriced; - the ArgonDB settings in section 4.1 — all five in today's release (v1.5.7.18). **Design against, but do not depend on yet:** - a deploy into an Azure account: `plan`, `apply-result` and `state` are real documents, proven against a recorded cloud and, on AWS, against a live one (shape (a) on 2026-10-21; shape (b) — the VM destroyed and a new VM restored from the bucket — on 2026-10-23); Azure waits for its account; - everything in shape (c) on a real cloud — instance per company, the cloud waker, the three-data-center storage — which is the longest stretch of the remaining work; - the ArgonDBCloud 1.0 release itself. The price model, the overlay, the screens up to the plan, the ArgonDB storage changes for shape (b), and the design for health monitoring are done. ArgonDBCloud creates shapes (a) and (b) in an AWS account today and nothing in an Azure one yet. We are not promising dates here beyond the ones the design already carries. --- ### 7. What we need from you One note answering these, numbered so your answers can be too: 1. **Your resources.** The real list beyond the ERP process — documents, queue, email, web front end, anything else — each per company or per install, with its size or quantity and its unit. 2. **Your questions.** The exact wording a customer sees, the unit, and what to assume when they skip it. 3. **Your permissions.** What the ERP's processes need from each cloud, kept to the minimum. 4. **Your install lines.** What runs on a company's instance to install the ERP: where the binary comes from, how its version is pinned, and how it gets its configuration. 5. **Clouds and regions.** AWS, Azure, or both, and which regions. Shape (c) comes to AWS first. 6. **Whose credentials.** Does the customer run the installer under their own sign-in, or does Splink act inside the customer's account — and with what role? 7. **One install or two.** Should the ERP's resources be created and removed together with ArgonDB's, or as a second, linked install with its own lifecycle? 8. **Which way in.** Will your installer link the crate, drive the tool with `--json`, or wait for the web application? 9. **What else ERPCloud must show.** Any screen, question, total or final line your customer needs that section 5 does not have. ---