View Source Persistence

By default ProcessHub keeps its per-coordinator process registry in an in-memory ETS table, rebuilt from peers on restart via the configured synchronization strategy. A node that restarts as a sole survivor has no record of the children it was hosting.

For restart-survival on a single node, ProcessHub supports opt-in on-disk registry backends, configured via the :registry_backend field on %ProcessHub{}.

Backends

:registry_backend accepts:

• :ets — (default) in-memory ETS. Unchanged prior behaviour.
• {:dets, opts} — on-disk persistence via :dets.
• {:durable_ets, opts} — hybrid: ETS for reads, mirrored synchronously to DETS for restart-survival (see Hybrid backend).
• {Module, opts} — a custom module implementing ProcessHub.Service.Storage.Behaviour.

Both disk backends require path: String.t(); a hub configured without one refuses to start, because ProcessHub has no directory of its own that your application can reach. They share the same on-disk format, so a hub can switch between them against the same path.

ProcessHub.child_spec(%ProcessHub{
  hub_id: :my_hub,
  registry_backend: {:dets, path: "/var/lib/myapp/hub.dets"}
})

Operational profile

Each DETS mutation calls :dets.sync/1 before returning, so any operation observed as :ok is durable on disk (~1–5 ms per write on local SSD, more on slower disks). Workloads with very high registry mutation rates should keep the default :ets backend.

DETS does not compact automatically — deleted entries leave gaps, but the registry is bounded by child count, so this is usually negligible. Entries with a :ttl are stored as {key, value, expire_ms} and filtered on read; there is no background sweeper for expired rows.

On open the file is opened with repair: true. If it is still unrecoverable, the corrupt file is rotated to <path>.corrupt-<system_monotonic>, an error is logged, and a fresh empty file is opened — the hub starts empty and rebuilds cluster state from peers rather than fail-stopping.

Custom backends

Implement ProcessHub.Service.Storage.Behaviour and pass registry_backend: {MyModule, opts}. Mutating callbacks may return {:error, reason}, so a future replicated backend (Raft, etc.) can plug in without breaking existing call sites.

Fast-restart stale-binding reaping

If a node restarts within :net_ticktime, peers never see a :nodedown and can be left holding registry entries that point at the node's now-dead pids (the rejoin sync only appends). This is independent of the registry backend — the stale rows live on the peer.

Every node broadcasts a per-boot token on startup. A peer purges another node's bindings only when that node's token changes, which distinguishes a genuine restart (new token → reap the dead pids) from a transient network flap (same token → keep the live bindings). This runs for all backends, including the default :ets, and requires no :auto_recovery. Purging only withdraws the bindings: the :auto_recovery reconcile round restarts the children left running nowhere, and without it nothing does. The one exception is a node held in its :nodedown_grace_period: without :auto_recovery its restart ends the grace, so its children move as after any failure and back to it as it rejoins.

Observability

ProcessHub reports through its hook system, not telemetry. The recovery lifecycle is observable via the recovery_state_changed hook (see below); registry-file corruption is logged at ERROR. There are no per-mutation events on the hot path.

Declared children and orphan recovery

Experimental

The :auto_recovery lifecycle is experimental and may change. The persistence backends above are not affected by this notice.

With :auto_recovery enabled, a hub maintains a declared list: a versioned, durable list of the children that SHALL exist. start_child/3 with durable: true adds the child's spec to it; a deliberate stop removes it; no other code path writes it. Each node then periodically reconciles the cluster toward the list:

orphans = declared children − children observed running anywhere

The remainder is started through the normal start path with check_existing: true, and a running child whose declared entry was removed (a stop that crashed halfway) is stopped. The same difference covers both directions: after a whole-cluster outage the live registry is empty and every declared child returns; after a single-node rejoin the peers already hold the children and the difference is empty; a child stopped during the absence is absent from the list and stays dead — list absence is the stop record, and it never expires, so a node may be away arbitrarily long without resurrecting a stopped child on return.

A restored child keeps the metadata of the registry row it left behind: the row in the live registry while there is one, bound or not, and otherwise its copy in the durable medium, which is all a restarted hub has. A child_data_alter hook may rewrite any key; every key it leaves alone survives.

The declared list

Mutations are serialized through the hub's leader node. The leader is chosen by :elector when it is available, and is otherwise the first hub member by name; either way every node picks the same one. The leader increments one monotonic version per mutation and persists before acknowledging, so "which copy is newer" is a single integer comparison and adoption replaces the whole list. The list mutation always commits before the process action — add before start, remove before terminate — so either half of a crashed command heals in the next round.

Every node keeps its adopted copy in a store beside the registry file, one row per declared child plus a version row, so a mutation writes only the rows it changes. After each write the leader announces the new version, and a peer behind it pulls the list and adopts it. On boot a node adopts the highest version among its local copy, its peers, and the remote manifest. The list is exactly as durable as the registry: a backend with a :path gets a DETS-backed list beside that file, while an :ets hub keeps the list in memory and starts from an empty one after a restart. To keep a list across restarts without putting the registry on disk, use :remote_manifest. With no leader reachable, durable: true starts and stops of declared children return {:error, :no_leader}; everything else stays leader-free.

durable: true requires a :permanent restart type (the default): a :transient or :temporary child may finish on its own node, and no other node could distinguish "finished" from "lost".

ProcessHub.start_child(:my_hub, child_spec, durable: true)
ProcessHub.Service.DeclaredChildren.declared_children(:my_hub)
#=> %{version: 3, children: [%{id: :my_child, ...}]}

Remote manifest

An optional off-cluster copy protects the list against the loss of every cluster disk. Configure remote_manifest: {module, opts} inside :auto_recovery with a module implementing ProcessHub.Storage.RemoteManifest — built-in adapters are ProcessHub.Storage.RemoteManifest.LocalPath (dependency-free; point it at storage that lives outside the cluster) and ProcessHub.Storage.RemoteManifest.S3 (behind the optional :ex_aws_s3 dependency, using conditional writes).

The leader ships every mutation asynchronously — retried with backoff, coalescing superseded versions — and boot fetches the remote copy before the first reconcile round: the higher version wins, whichever side holds it. A failing or slow adapter never affects a command; failures emit the manifest_ship_failed hook.

A missing list is never empty truth

A missing or corrupt local list while durable evidence exists (a seeded marker beside the list file, or durable registry rows) is not read as "nothing declared": the hub restores from the remote manifest, or parks its reconcile — starting and stopping nothing, emitting the alarm-grade declared_parked hook — until an operator intervenes. The explicit operator call is ProcessHub.Service.DeclaredChildren.clear/1.

On the first boot with the feature enabled and no stored list, the hub seeds version 1 from its durable registry rows, once — so an existing deployment's children carry over. The stored list has a format marker; a newer marker than the running release understands refuses to open.

Row bookkeeping

Every row carries hub-owned state under the reserved metadata key :__process_hub__: epoch, changed_by, changed_at (diagnostics only), and durable: true for declared children.

Every write that authors a row increments epoch. Merges resolve by higher epoch, ties by the lexicographically lower changed_by, and adopt the winner verbatim — so every node converges on the same value in any order. The epoch is a counter, never a wall clock: hardware without a battery-backed clock boots with a bogus time exactly when it rejoins and merges. Caller metadata cannot set the key.

node_pids is not part of that resolution. It is a set of per-node observations, each owned by the node it names — a payload from node N only touches the {N, pid} entry, and a child missing from N's payload removes nothing else. No absence observation deletes durable state.

A deliberate stop deletes the registry row (stop memory lives in the list); a :temporary or :transient child the supervisor declines to restart gets the same treatment. Placement churn leaves a short-lived stub (30 seconds) for the child's new copy to register into. If nothing registers, the janitor removes a durable child's stub (the reconcile restarts the child while it is declared) and re-places any other child through the distribution strategy, at most five times with a growing back-off, before it removes the stub.

Rounds and states

:recovery_state is :recovering until the first round completes, then :normal (terminal). With auto_recovery: false it is :normal from init/1.

The first round opens on evidence: once cluster_settle_ms has passed since start and every connected peer the hub knows about has delivered its registry data, which counts once it has been merged into the local registry. reconcile_grace_ms caps that wait — it opens the round whatever the evidence says, so a peer that is connected but never answers cannot hold a node in :recovering, and :normal is always reached. Later rounds follow completed synchronisation rounds, rate-limited to one per reconcile_interval_ms. When a node joins the hub, a round follows within a couple of seconds instead, so a child that could not start for want of a node is not left waiting for the full interval. A host that knows of another reason a refused start can now succeed (its placement target gained a member, say) calls ProcessHub.Service.Recovery.request_round/1 for the same.

Every node submits from its adopted copy of the list. Duplicates are prevented by check_existing: true and by the ring routing concurrent submissions to the same owner, where the supervisor rejects the second. A round also reduces a child observed on more than one node to its ring owner's instance, and removes rows marked durable that are undeclared and observed running nowhere (a stale rejoining peer's leftovers).

Configuration

%ProcessHub{
  hub_id: :my_hub,
  auto_recovery: [
    reconcile_grace_ms: 30_000,
    cluster_settle_ms: 2_000,
    reconcile_interval_ms: 15_000,
    remote_manifest: {ProcessHub.Storage.RemoteManifest.LocalPath, path: "/mnt/off-cluster"}
  ]
}

• :reconcile_grace_ms — the cap on the wait for the first round, not the normal path to it. Default 30_000, range [50, 600_000]. A value below :cluster_settle_ms wins, so a hub tuned down keeps its timing.
• :cluster_settle_ms — how long silence from the cluster counts as "this node is alone" before the first round may open. Default 2_000, range [0, 60_000]. Raise it when your cluster layer is slow to connect peers; set 0 when the host forms its cluster before starting its hubs.
• :reconcile_interval_ms — minimum spacing between rounds, and the per-handler budget for the blocking pre_recovery_replay hook. Default 15_000, range [1_000, 600_000].
• :remote_manifest — {module, opts} implementing ProcessHub.Storage.RemoteManifest. Default nil (disabled).

A value outside these ranges refuses the hub: ProcessHub.start_link/1 returns {:error, {:invalid_auto_recovery, reason}} before anything is opened. A value that is not false, true or a keyword list starts the hub with recovery disabled and logs a warning.

The superseded keys :marker_path, :replay_timeout_ms, :recovery_timeout_ms and :stopped_row_ttl_ms are deprecated: accepted with a WARN, ignored, and rejected in a later release. So are Recovery.prepare_recovery/1 and prepare_recovery_cluster/1, now no-ops. See migration-guide.md.

An opted-in hub opens its backend with recovery_replay: false: restoration flows through the reconcile, never through the backend open, so a returning node cannot republish its stale view as fact.

Hooks

• recovery_state_changed — the :recovering → :normal transition (async).
• pre_recovery_replay — once, before the first round issues any start. Synchronous: use it to wait on prerequisite services.
• post_recovery_replay — once, after the first round completes (async).
• reconcile_round — every round, including quiet ones, with candidates, orphans, started, skipped_pending, duplicates, stopped_undeclared, removed_stale, elapsed_ms.
• reconcile_duplicate — emitted by the node stopping its own duplicate instance.
• declared_tiebreak — a version tie with differing content was resolved.
• declared_parked — alarm-grade: the list is lost and the reconcile is parked.
• manifest_ship_failed — a remote ship attempt failed; the shipper retries.

Introspection and failure modes

Recovery.recovery_state(:my_hub)         # :recovering | :normal
Recovery.await_normal(:my_hub, 60_000)   # :ok | {:error, :timeout}
DeclaredChildren.declared_children(:my_hub)  # %{version: v, children: [...]}

await_normal/2 returns once the first round has completed. Size the timeout above reconcile_grace_ms.

• Parked list — the round performs no starts and no stops; a lost list is never read as "nothing declared".
• A child mid-migration — a declared child with a registered but unbound row must be an orphan in two consecutive rounds before it is started.
• Divergence partitions — both sides mutating the list under a non-locking partition strategy resolve on heal by version, ties deterministically to the lowest mutating node (declared_tiebreak fires). Quorum strategies refuse the situation outright.

Hybrid backend (:durable_ets)

{:durable_ets, opts} keeps an in-memory ETS table as the source of truth for reads and writes, mirroring every mutation synchronously to a DETS file. On open the DETS file is replayed into ETS so reads are immediately authoritative.

• Reads (get, exists?, foldl, match, export_all) hit ETS only — hot-path callers pay no DETS cost.
• Writes (insert, remove, clear_all) write ETS, then DETS, then :dets.sync/1 before returning; once :ok is observed the row is durable. A failed DETS write rolls back the ETS write and returns {:error, reason}.
• Crash window — a write in memory but not yet synced is lost on restart, identical to {:dets, _}.

Pick {:durable_ets, _} when reads dominate and restart-survival matters; {:dets, _} when writes dominate and RAM is constrained; :ets when neither restart-survival nor disk durability is required.