View Source ProcessHub.Service.Recovery (ProcessHub v0.8.0)
Orphan reconcile: every round converges the cluster toward the hub's declared
list (ProcessHub.Service.DeclaredChildren) — it starts
declared − observed running anywhere through the normal start path with
check_existing: true, and stops running children whose declared entry was
removed. The same difference covers a whole-cluster restart and a single-node
rejoin; stop knowledge is list absence and never expires.
Experimental
The orphan reconcile (the
:auto_recoverylifecycle) is experimental and may change in future releases. Use in production at your own discretion.
The first round opens as soon as the cluster has settled (cluster_settle_ms)
and every connected peer the hub knows about has delivered its registry data;
reconcile_grace_ms caps that wait so a peer that never answers cannot hold
the hub in :recovering. Later rounds follow completed synchronisation rounds,
rate-limited to one per reconcile_interval_ms. A node joining, or a host
calling request_round/1, gets a round within a couple of seconds instead.
This module owns the :auto_recovery config, the scheduling, and the
recovery lifecycle; the round itself lives in ProcessHub.Service.Recovery.Round and the coordinator stays
the GenServer. See guides/Persistence.md for the model.
Summary
Types
What is asking for a round: a completed synchronisation round, evidence (the
settle window or a peer's registry data, which a joining node sends), the
grace cap, or the host (request_round/1).
Functions
Blocks until the coordinator reaches :normal or timeout_ms elapses.
Records that the cluster settle window has passed.
Completes a round: clears what spawn_round/1 set and stamps the rate limit.
The first round also moves the coordinator to :normal, dispatches the
transition hook, and fires the async post_recovery_replay.
Parses a settings struct's :auto_recovery for the hub start: an unknown shape
falls back to the disabled config with a WARN, an out-of-range value is an error.
Returns the disabled (default) config.
Parses the :auto_recovery config field into a normalized map.
Deprecated. Armed the next boot for marker-driven replay by deleting the local marker file.
Deprecated. Fanned prepare_recovery/1 out to every hub member.
Returns the coordinator's current :recovery_state.
Records that node has delivered its registry data. Only the first round
waits on this evidence, so nothing is kept once the hub is :normal.
Asks the hub for a reconcile round soon, for a host that knows a start the last round refused can now succeed — a placement target gaining a member, for example. A node joining the hub already does this by itself.
Runs the round a waiting request was put off for.
Returns whether trigger may run a round now — the single answer for the first
round as well as for later ones.
Arms the two first-round timers: the settle window, after which silence from
the cluster counts as being alone, and reconcile_grace_ms, which opens the
round whatever the evidence says.
Runs a round in a separate process; replies to the coordinator with
{:reconcile_done, result}, which it hands to complete_round/2 —
Round.run_safe/2 guarantees a reply whatever happened.
Spawns a round when round_due?/2 allows trigger one. Evidence or a demand
that has to wait, because a round is running or the last one was too recent,
is tried again a couple of seconds later.
Types
@type trigger() :: :sync | :evidence | :grace | :demand
What is asking for a round: a completed synchronisation round, evidence (the
settle window or a peer's registry data, which a joining node sends), the
grace cap, or the host (request_round/1).
Functions
@spec await_normal(ProcessHub.hub_id(), non_neg_integer()) :: :ok | {:error, :timeout}
Blocks until the coordinator reaches :normal or timeout_ms elapses.
Returns :ok on reaching :normal (immediately when the hub does not exist
or has no recovery), or {:error, :timeout} otherwise. :normal means the
first reconcile round has completed, so callers SHOULD size the timeout above
reconcile_grace_ms.
@spec cluster_settled(ProcessHub.Coordinator.State.t()) :: ProcessHub.Coordinator.State.t()
Records that the cluster settle window has passed.
@spec complete_round( ProcessHub.Coordinator.State.t(), ProcessHub.Service.Recovery.Round.result() ) :: ProcessHub.Coordinator.State.t()
Completes a round: clears what spawn_round/1 set and stamps the rate limit.
The first round also moves the coordinator to :normal, dispatches the
transition hook, and fires the async post_recovery_replay.
@spec config_or_disabled(map() | struct()) :: {:ok, ProcessHub.Hub.recovery_config()} | {:error, {:invalid_auto_recovery, term()}}
Parses a settings struct's :auto_recovery for the hub start: an unknown shape
falls back to the disabled config with a WARN, an out-of-range value is an error.
@spec disabled_config() :: ProcessHub.Hub.recovery_config()
Returns the disabled (default) config.
@spec parse_config(false | true | keyword() | term()) :: {:ok, ProcessHub.Hub.recovery_config()} | {:error, :invalid_auto_recovery | {:invalid_auto_recovery, atom()}}
Parses the :auto_recovery config field into a normalized map.
Accepts the documented shapes:
false— disabled (the default).true— enabled with defaults.keyword()—:reconcile_grace_ms,:reconcile_interval_ms,:cluster_settle_ms, and:remote_manifest({module, opts}implementingProcessHub.Storage.RemoteManifest, defaultnil).
The superseded keys :marker_path, :replay_timeout_ms,
:recovery_timeout_ms, and :stopped_row_ttl_ms are deprecated: they are
accepted with a WARN and ignored, and will be rejected in a future release.
Returns {:ok, recovery_config}, or {:error, {:invalid_auto_recovery, reason}}
for out-of-range values. Unknown shapes return {:error, :invalid_auto_recovery}
so the caller can decide whether to fall back to disabled or to refuse to start.
@spec prepare_recovery(ProcessHub.hub_id()) :: :ok
Deprecated. Armed the next boot for marker-driven replay by deleting the local marker file.
There is no marker any more: every node reconciles its durable registry against
the cluster continuously, so recovery after an outage needs no pre-boot step.
The function is kept so existing operator tooling keeps running — it logs a
warning and returns :ok without touching the filesystem.
Scheduled for removal in a future release.
@spec prepare_recovery_cluster(ProcessHub.hub_id()) :: {:ok, [node()]} | {:error, :not_alive}
Deprecated. Fanned prepare_recovery/1 out to every hub member.
A no-op for the same reason as prepare_recovery/1; it still reports the hub's
members so existing callers keep matching on {:ok, nodes}. Returns
{:error, :not_alive} when the hub is not running, as before.
Scheduled for removal in a future release.
@spec recovery_state(ProcessHub.hub_id()) :: :recovering | :normal
Returns the coordinator's current :recovery_state.
Returns :normal when the hub does not exist or was started without
:auto_recovery.
@spec registry_delivered(ProcessHub.Coordinator.State.t(), node()) :: ProcessHub.Coordinator.State.t()
Records that node has delivered its registry data. Only the first round
waits on this evidence, so nothing is kept once the hub is :normal.
@spec request_round(ProcessHub.hub_id()) :: :ok
Asks the hub for a reconcile round soon, for a host that knows a start the last round refused can now succeed — a placement target gaining a member, for example. A node joining the hub already does this by itself.
The round does not wait for reconcile_interval_ms, only until a couple of
seconds have passed since the last round, and never overlaps a running one.
Before the first round it does nothing, since that round comes anyway. Returns
:ok; a hub that is not running, or runs without :auto_recovery, does nothing.
@spec retry_round(ProcessHub.Coordinator.State.t()) :: ProcessHub.Coordinator.State.t()
Runs the round a waiting request was put off for.
@spec round_due?(ProcessHub.Coordinator.State.t(), trigger()) :: boolean()
Returns whether trigger may run a round now — the single answer for the first
round as well as for later ones.
Triggers are :sync (a completed synchronisation round, the default),
:evidence (the settle window closing or a peer's registry data arriving) and
:grace (reconcile_grace_ms) and :demand (request_round/1). Rounds
never overlap. The first round waits for the evidence gate — the settle window
passed and every connected peer the hub knows about heard from — except under
:grace, which is the cap and opens it regardless. After that, the sync tick
runs at most one round per reconcile_interval_ms, while evidence and demand
only wait until a couple of seconds have passed since the last round.
@spec schedule_first_round(ProcessHub.Hub.t()) :: :ok
Arms the two first-round timers: the settle window, after which silence from
the cluster counts as being alone, and reconcile_grace_ms, which opens the
round whatever the evidence says.
The grace fires whether or not any peer answered, so :normal is reached in
bounded time on every boot. Disabled hubs schedule nothing.
@spec spawn_round(ProcessHub.Coordinator.State.t()) :: ProcessHub.Coordinator.State.t()
Runs a round in a separate process; replies to the coordinator with
{:reconcile_done, result}, which it hands to complete_round/2 —
Round.run_safe/2 guarantees a reply whatever happened.
@spec trigger_round(ProcessHub.Coordinator.State.t(), trigger()) :: ProcessHub.Coordinator.State.t()
Spawns a round when round_due?/2 allows trigger one. Evidence or a demand
that has to wait, because a round is running or the last one was too recent,
is tried again a couple of seconds later.