# Aeshnidae.Mule — the mule vault Account-wide item storage with no client window. Items go in and out by command (`/mule`), and the whole vault is listed on the website under My Characters. One vault per **account**, so every character on it reaches the same storage, from anywhere. **Not deployed to live.** Built and tested on staging on 2026-10-09. The live steps are at the bottom. --- ## The one rule **Items stay real WorldObjects.** A deposit changes which container an item is in and nothing else. Nothing here reads an item's properties into a row, destroys the object and rebuilds it later — so an item keeps its guid, its spells, its enchantment rows, its tinks, its imbues, its workmanship, its item experience, its stack size and our 9042/9043 stamps, because none of those are copied: the object simply moves. Proved, not asserted: 1,000 real looted items were put in and taken out, and the 56,995 biota property rows behind them were byte-identical before and after (see **What was tested**). ## Where the vault lives, and how it stays there A vault is a **chain of hidden container WorldObjects** of our own weenie **3000136** `aelrynthmulevault` (ledger `2026-10-09_001`), with dynamic guids, saved in the **shard** database. They are never placed in the world, never put in anyone's inventory and never sent to a client. `aeshnidae_mule` maps an account to them, one row per link. Three choices in that sentence are load-bearing: * **A chain, not one container.** ACE exposes `ItemsCapacity` through `WorldObject.ItemCapacity`, which is a `byte?` cast of the int property, and `Container.TryAddToInventory` compares against that cast. A weenie row saying 1000 reads back as `(byte)1000 = 232`, silently. So one logical vault is four containers of 250 (`SlotsPerContainer`), grown one at a time as they fill. The player is never shown the seam: `/mule` counts and lists across the whole chain, and the website reports every item as being in the first link so the page groups them as one. * **Each link is top-level** (`container_Id` NULL), not nested inside the first. ACE loads a *nested* container's contents down two paths that race each other (`Container.SortWorldObjectsIntoInventory` marks a side pack loaded while that pack's own constructor callback is still reading it). A vault is not the place to find out what that does, so the vault is flat — and that is also why packs and foci are refused (`AllowPacksAndFoci` false, and `ContainersCapacity` 0 on the weenie). * **WeenieType Storage (57), not Container (21).** `Container.Reset` calls `ClearUnmanagedInventory`, which **destroys every item whose `GeneratorId` is null** — which is every item a player ever deposited. That method's only exemption is `if (this is Storage) return false`. `Storage` also pins `Default_ChestResetInterval` at infinity and saves its own row on every add and remove. The weenie carries no `ResetInterval` and no `ItemUseable`, so there is no path to open it either. ### Nothing in ACE saves or ticks it, so this mod does both jobs ACE's upkeep of an object runs off its landblock: `Landblock.Tick` walks a heartbeat list an object only joins through `AddWorldObject`, and `Landblock.SaveDB` only ever walks its own `worldObjects`. A vault container is in no landblock, so: * **Saving is ours.** Every movement forces the moved item's biota to disk and waits for the write before the player is told it worked; a timer (`SaveIntervalSeconds`, 60) sweeps for retries and writes the containers' own rows; and `Dispose` saves **synchronously** at shutdown. That last one is not a style choice — ACE runs `ModManager.Shutdown()` *after* `WorldManager.StopWorld()`, so anything queued onto the world thread at shutdown is dropped, while the shard's own write queue is still drained afterwards. * **Loading is ACE's.** `Container`'s biota constructor asks the shard for everything whose `container_Id` is the container and queues the sort onto `WorldManager`'s action queue, which the world thread drains every tick whether or not any landblock is awake. `WorldObject.EnqueueAction` has an explicit allowance for exactly this case. The mod starts that load when a character of the account logs in (`Patches.AfterEnterWorld`), so the first `/mule` finds it there. A vault that has not finished loading is **never treated as empty** — every verb refuses instead. * **The heartbeat we deliberately do not replace.** Enchantment durations on a stored item do not tick down, and an item with a `Lifespan` cannot expire in storage. The second is a protection; the first is a quirk worth knowing. Both are listed below. ### ACE would delete the vault, and `Patches` stops it `ShardDatabaseOfflineTools.PurgeOrphanedBiotasInParallel` deletes every biota with no parent Container, no Wielder and no Location, exempting only `WeenieType.Allegiance`. A vault container is exactly that. On a **second** run of the same tool, every item that had been in it would go too, by the "Parent container not found" pass. The tool only runs when `Offline.PurgeOrphanedBiotas` is set in `Config.js`, at startup, and is documented as a once-every-few-months job — which is precisely the switch someone flips a year from now having never heard of the vault. So the protection is in code: a Harmony prefix on `ShardDatabaseOfflineTools.PurgeBiota` refuses any guid in `aeshnidae_mule`. `ModManager.Initialize` runs at `Program.cs:158` and the purge at `:173`, and `MuleDb` reads the container ids with its own MySqlConnector connection (it does not need `DatabaseManager`, which starts at `:259`), so the guard is loaded in time. Giving the containers a dummy Location row would also satisfy the tool and was rejected: a biota with a Location in a landblock's range is what `GetDynamicObjectsByLandblock` loads when that landblock wakes, and a vault appearing on the ground is worse than the problem. **This was tested for real** — see below. ACE purged an identical *unprotected* container in the same run it refused the four real ones. ## The sequence, and what a crash at each step leaves behind In ACE, **which container holds an item is written in exactly one place: the item's own biota**, as `PropertyInstanceId.Container` (`biota_properties_i_i_d`, type 2). A player's biota does not list what it carries; a container's biota does not list its contents. So a movement is **one row changing value** — there is no two-phase commit to get wrong, and no instant at which the database says the item is in two places or in neither. A deposit, in order (`Movement.cs` carries this comment too): | | step | a crash here leaves | |---|---|---| | 0 | every refusal checked, before anything moves | nothing touched | | 1 | `player.TryRemoveFromInventory` — **memory only** | the item in the player's pack. ACE's `...WithNetworking` variant is deliberately **not** used: its `DeepSave` would write `container_Id = NULL`, the one state nothing in the game can find again | | 2 | `link.TryAddToInventory` — **memory only** | the item in the player's pack. Steps 1 and 2 are one uninterrupted block on the world thread, and a player's own periodic save runs on that same thread, so it cannot land between them and write the NULL. If step 2 fails the item goes straight back (also memory only) and nothing was written | | 3 | **the commit**: the item's biota, one row, waited on | before the write drains: in the pack. After it lands: in the vault. Nothing in between | | 4 | the ledger row | a correctly placed item with no audit line — the only durable inconsistency the sequence can produce, and one that cannot lose or duplicate anything | | 5 | the client is told, and the player's own biota is forced | nothing; neither is load-bearing (a player's biota does not record what it holds) and both are done anyway so nothing about the player is stale at the moment the item changes hands | | 6 | **only now** is the player told it worked, from the write's own callback | — | A withdraw is the same shape with the ends swapped. **What is genuinely not transactional**, said plainly: the pair (item's biota row, ledger row). The ledger can be missing a line for a movement that happened. Neither direction can lose or duplicate an item — the biota row is the truth, the ledger is the audit, and `mule-admin verify` compares them. There is no transaction spanning "leaves the player" and "arrives in the vault" because in ACE those are not two writes; they are one. ## The lock Every mutation, and every read that must agree with one (the count behind the limit, the save sweep's walk of `Inventory`), runs inside `lock (Vault.Gate(accountId))`. Two clients on one account is normal here — FairPlay's per-address caps are off for the beta — so two simultaneous deposits are the obvious race. Player commands happen to arrive on the world thread (`InboundMessageManager`'s queue is drained in `WorldManager.UpdateWorld`), which would serialise them by itself. The **console does not**: `Aeshnidae.RemoteConsole` invokes a handler straight off its own watcher thread. Nor does the save sweep. So the lock is load-bearing, and `mule-test race` proves it holds and reports how many times it was contended — a run with no contention is reported as proving nothing. One sliver the lock alone does not close: `Vault.Get` **cannot** take the lock, because it is called from the world thread and the load it would wait for finishes on that thread. So between a caller getting a handle and taking the lock, the sweep could have released that vault. Nothing on disk could go wrong — the item's own row is still the truth and the sweep saves before letting go — but the released handle's in-memory count would diverge from a freshly loaded one, and the limit is counted from that count. Every mutation therefore calls `Vault.IsCurrent(handle)` as its first act inside the lock and refuses if it lost the race, rather than working from a count that may be stale. ## What it will not take Checked before anything moves, in `Movement.RefuseDeposit` — ACE's own `HandleActionPutItemInContainer_Verify` list in the same order, minus the parts that only make sense for a container on the ground, plus the vault's own rules: * anything ACE itself will not move (`!Guid.IsDynamic()`, a Creature, `Stuck`) * a busy or teleporting player * an item in a trade window * **anything worn or wielded** — take it off first; ACE's dequip is a separate path with its own networking and the vault does not drive it * anything not in the player's own packs (the search is `SearchLocations.MyInventory` only, so a corpse, the ground, a vendor's stock and another open container are all out) * **a pack with things in it** — store what is inside it, not the pack * **packs and foci themselves** (see above; `AllowPacksAndFoci`) * anything in `RefuseWcids` **Attuned items ARE allowed** (`AllowAttuned` true). It is safe *today* only because a vault belongs to one account and nothing crosses an ownership line: what goes in comes back out to the same account's characters. **If shared access or a market is ever added this must go back to false**, or attuned gear becomes tradeable through the vault. There is a check waiting for that in `RefuseDeposit`. ## The seams left for later No shared access and no marketplace in this version, but the data does not need reshaping to add them: a vault is identified by **account**, the ledger records **which character** moved each item, and `aeshnidae_mule` is keyed `(account_Id, seq)` so the chain can grow. Grants (another account allowed at this vault) and a market (an item offered out of this vault) are new tables beside these two. ## Tables | | | |---|---| | `aeshnidae_mule` | account → the object ids of its vault containers, one row per link. **The authority for which container belongs to whom** — nothing else records it, and the containers carry no account number | | `aeshnidae_mule_ledger` | append-only, INSERT only: one row per movement and per refusal, with account, character, object id, wcid, name, stack size, direction, result, detail and time to the millisecond. Feeds `/mule log` and the website | | `aeshnidae_mule_test`, `_test_item` | the console harness's scaffolding (staging only; created on first use of `mule-test`) | ## Commands See `Mods/COMMANDS.md` for the player's half. Operator and harness: ``` mule-admin status, counts, lock contention, purges refused mule-admin reload re-read Settings.json mule-admin verify [account] memory against the shard, one account or all mule-admin chain the container chain and what is in each link mule-admin where where the shard says one object is mule-admin log [n] the ledger mule-admin tally ledger totals by direction and result mule-admin orphans rows with no container, and containers with no row mule-admin release save and let go of a loaded vault mule-admin save save every loaded vault now mule-test pack|fill|deposit|withdraw|race|count|audit|pause|clear STAGING ONLY ``` `mule-test` refuses unless `AllowTestHarness` is true, which it is not by default and must never be on live: those verbs mint and destroy real items. It drives the **same** core the `/mule` command uses, with a **stand-in pack** — a real Container holding real WorldObjects with real biotas — in place of a player's inventory, and keeps a durable record of every item it ever created so a crash can be audited afterwards. What it does **not** exercise is the Player half: the refusals that need a live session and the client messages. `Movement.DebugPauseAfterRemoveMs` / `...AfterAddMs` (set by `mule-test pause`, zero otherwise) widen the gaps inside a movement so a `SIGKILL` can be aimed at them. They are left in the shipped path on purpose: a test that runs against different code from the one players use proves nothing about the one players use. ## Settings Every number is documented at the top of `Settings.cs`. The ones worth knowing: `MaxItems` 1000, `SlotsPerContainer` 250 (must be ≤ 255 — see above), `AllowAttuned` true, `AllowPacksAndFoci` false, `SaveIntervalSeconds` 60, `ReleaseAfterSeconds` 300, `LoadTimeoutSeconds` 20, `AllowTestHarness` **false**. ## What was tested, on staging, 2026-10-09 All of it with real items (`LootGenerationFactory`, treasure profile 31), real biotas, and counts taken from the shard rather than from memory. Account 2, 1,001 items minted. 1. **Volume.** 200 in and 200 out in batches, then 1,000 in as 4 × 250. Memory and shard agreed at every step, and the chain grew by itself: 1 → 2 → 3 → 4 links. 2. **The limit.** With 1,000 in, the 1,001st was refused `refused:full` ("1000 of 1000") and stayed in the pack; the vault stayed at exactly 1,000. 3. **Crash safety**, `sudo systemctl kill -s SIGKILL ace-staging`, four times: * mid-batch, commits batched to the end of the loop → nothing written, everything exactly where it had been. 1000 + 1 = 1001. * **deliberately in the gap between the remove and the add** (`pause 400 0`): 7 items committed, the 8th caught in flight. After the restart: vault 993, pack 8, **993 + 8 = 1001**, 7 ledger rows, 0 orphans. The in-flight item's row still said the vault, so it rolled back there. * **deliberately in the gap between the add and the commit** (`pause 0 400`): vault 900 → 907, pack 101 → 94, **907 + 94 = 1001**, 0 orphans. The in-flight item's row still said the pack, so it rolled back there. * while a vault was loading: nothing changed. After every one: *"EVERY ITEM ACCOUNTED FOR: none lost, none orphaned, each in exactly one place"*, and the 56,995-row biota fingerprint of all 1,001 items **byte-identical** to before the kill. 4. **The lock.** Two threads, unmarshalled, depositing on one account: 40 each → vault +80, pack −80, shard agreed, contended once. Then both racing for the **last 169 slots**: one got 100, the other 69 and then `refused:full`; the vault landed on exactly 1,000, never 1,001, and 169 left the pack as 169 arrived. 5. **Fidelity.** A full 200-item round trip, then the 1,001-item set across three kills: the fingerprint of every `biota`, `_int`, `_int64`, `_float`, `_bool`, `_string`, `_d_i_d`, `_i_i_d`, `_spell_book` and `_enchantment_registry` row was **identical**, excluding only the three things a move is meant to change (`PlacementPosition`, Owner, Container). The set carried 200 × workmanship, 200 × `NumTimesTinkered`, 200 × `ImbuedEffect`, 200 × item XP, 200 × 9042, 200 × 9043, 606 spellbook rows and 25 stacks. 6. **The purge guard, for real.** `Offline.PurgeOrphanedBiotas` was turned on on staging and the server restarted. ACE tried to purge all four vault containers and the guard refused each by name; all 1,000 items survived; and the **identical unprotected container** (the harness's stand-in pack, same weenie, same shape) **was purged in the same run** — which is the proof that the threat is real and the guard is what stops it. The flag was put back to false (backup `~/ace-staging/bin/Config.js.bak-2026-10-09-purgetest`, and the file is otherwise identical to it). 7. **Graceful restart.** `systemctl restart ace-staging` with 1,000 items stored: counts preserved, and the mod's synchronous shutdown save ran (`shut down - 1 vault row(s) written`). 8. **Hot reload.** The mod was rebuilt in place eight times with the vault populated; it reloaded and re-read all four links and 1,000 items from the shard each time. 8a. **The shipped build was re-tested after the `IsCurrent` guard was added**: 150 out, 150 back in through two racing threads (consistent, contended), then one more SIGKILL in the remove/add gap — vault 993, pack 7, **993 + 7 = 1000**, 0 orphans, 7 ledger rows, and the 56,934-row fingerprint identical. The vault was then put back to 1,000. 9. **The website.** Verified end to end against the staging databases through a temporary second AeshPlayer instance on 127.0.0.1:8093 with its own SELECT-only MariaDB user (`aeshwebstaging`, since dropped), reached over an SSH tunnel and driven in a browser. The page showed **`Account vault (1,000)`** as its own group of 1,000 items beside `Worn and wielded` (3) and `Main pack` (8), as its own option in the **Where** list, with the same appraisal rendering as everything else (`Acid Assagai · Awakened 7 · workmanship 6 · Silver · 3 tinkers · 26 damage · 3,234 pyreal`) and the same search — "blood drinker" matched 269 of the 1,011. An account with no vault got no vault entry; a session for one account asking for another account's character got 404; no cookie and a tampered cookie got 401. 10. **The website still cannot write.** As the live `aeshweb` user: `UPDATE` on `biota_properties_i_i_d` (the dupe), `INSERT`/`DELETE` on `biota`, `CREATE TABLE`, `DROP TABLE`, `GRANT` to itself, reading the staging databases and reading `mysql.user` were all refused (1142/1044), with `SELECT` on `biota` succeeding as the control. Its grants are still exactly `SELECT` on the three live databases. ### Not verified * **Everything that needs a player in a client.** No one was online on staging, so `/mule` itself, `/mule deposit` on a selected item, `/mule deposit pack N`, `/mule withdraw`, `/mule list`, `/mule log`, the refusals that need a live session (trade window, wielded, too busy) and every client message were exercised only as code, not in a client. `mule-test` drives the identical core through a stand-in container, so the ordering, the commit, the lock, the limit and the accounting are all tested — the Player half is not. * Enchantment durations freezing while an item is stored is by design and was not measured. * Behaviour with more than one account's vault loaded at once (staging had one). * `mule-admin repair` does not exist yet. A link whose biota is missing is refused loudly and by hand — see `Vault.LoadNow`. ## Live steps that remain 1. Apply `Mods/Content/sql/changes/2026-10-09_001_aelrynth-mule-vault.sql` to the live world database (`push-content.sql` / `push-content.sh`), **after Tom's go-ahead**, then `clearweenie 3000136`. Check `SELECT MAX(class_Id) FROM weenie WHERE class_Id >= 3000001` on live first: it was 3000135 on both live and staging on 2026-10-09. 2. Deploy and build the mod on the VPS, then `ace-cmd.sh 'mod-load Aeshnidae.Mule'`. Deploy a `Settings.json` with `AllowTestHarness` false and `Trace` false **first**. 3. `publish-source.sh` — the licence requires the running source. 4. `publish-note.sh Mods/patchnotes/2026-10-09_mule-vault.md`, and a card on the front page. 5. Add the row to the site's In-force page (`C:\Aeshnidae\site\current.html`). 6. `publish-player-site.sh` for the website change. 7. Post the enable line to `#commits` by hand (enabling is not a commit). 8. Then watch the first real deposits: `mule-admin verify`, `mule-admin tally`.