View Source ProcessHub.Service.Recovery (ProcessHub v0.8.0)

Orphan reconcile: every round converges the cluster toward the hub's declared list (ProcessHub.Service.DeclaredChildren) — it starts declared − observed running anywhere through the normal start path with check_existing: true, and stops running children whose declared entry was removed. The same difference covers a whole-cluster restart and a single-node rejoin; stop knowledge is list absence and never expires.

Experimental

The orphan reconcile (the :auto_recovery lifecycle) is experimental and may change in future releases. Use in production at your own discretion.

The first round opens as soon as the cluster has settled (cluster_settle_ms) and every connected peer the hub knows about has delivered its registry data; reconcile_grace_ms caps that wait so a peer that never answers cannot hold the hub in :recovering. Later rounds follow completed synchronisation rounds, rate-limited to one per reconcile_interval_ms. A node joining, or a host calling request_round/1, gets a round within a couple of seconds instead. This module owns the :auto_recovery config, the scheduling, and the recovery lifecycle; the round itself lives in ProcessHub.Service.Recovery.Round and the coordinator stays the GenServer. See guides/Persistence.md for the model.

Summary

Types

What is asking for a round: a completed synchronisation round, evidence (the settle window or a peer's registry data, which a joining node sends), the grace cap, or the host (request_round/1).

Functions

Blocks until the coordinator reaches :normal or timeout_ms elapses.

Records that the cluster settle window has passed.

Completes a round: clears what spawn_round/1 set and stamps the rate limit. The first round also moves the coordinator to :normal, dispatches the transition hook, and fires the async post_recovery_replay.

Parses a settings struct's :auto_recovery for the hub start: an unknown shape falls back to the disabled config with a WARN, an out-of-range value is an error.

Returns the disabled (default) config.

Parses the :auto_recovery config field into a normalized map.

Deprecated. Armed the next boot for marker-driven replay by deleting the local marker file.

Deprecated. Fanned prepare_recovery/1 out to every hub member.

Returns the coordinator's current :recovery_state.

Records that node has delivered its registry data. Only the first round waits on this evidence, so nothing is kept once the hub is :normal.

Asks the hub for a reconcile round soon, for a host that knows a start the last round refused can now succeed — a placement target gaining a member, for example. A node joining the hub already does this by itself.

Runs the round a waiting request was put off for.

Returns whether trigger may run a round now — the single answer for the first round as well as for later ones.

Arms the two first-round timers: the settle window, after which silence from the cluster counts as being alone, and reconcile_grace_ms, which opens the round whatever the evidence says.

Runs a round in a separate process; replies to the coordinator with {:reconcile_done, result}, which it hands to complete_round/2 — Round.run_safe/2 guarantees a reply whatever happened.

Spawns a round when round_due?/2 allows trigger one. Evidence or a demand that has to wait, because a round is running or the last one was too recent, is tried again a couple of seconds later.

Types

@type trigger() :: :sync | :evidence | :grace | :demand

What is asking for a round: a completed synchronisation round, evidence (the settle window or a peer's registry data, which a joining node sends), the grace cap, or the host (request_round/1).

Functions

Link to this function

await_normal(hub_id, timeout_ms \\ 60000)

View Source
@spec await_normal(ProcessHub.hub_id(), non_neg_integer()) :: :ok | {:error, :timeout}

Blocks until the coordinator reaches :normal or timeout_ms elapses.

Returns :ok on reaching :normal (immediately when the hub does not exist or has no recovery), or {:error, :timeout} otherwise. :normal means the first reconcile round has completed, so callers SHOULD size the timeout above reconcile_grace_ms.

Records that the cluster settle window has passed.

Link to this function

complete_round(state, result)

View Source

Completes a round: clears what spawn_round/1 set and stamps the rate limit. The first round also moves the coordinator to :normal, dispatches the transition hook, and fires the async post_recovery_replay.

Link to this function

config_or_disabled(hub_conf)

View Source
@spec config_or_disabled(map() | struct()) ::
  {:ok, ProcessHub.Hub.recovery_config()}
  | {:error, {:invalid_auto_recovery, term()}}

Parses a settings struct's :auto_recovery for the hub start: an unknown shape falls back to the disabled config with a WARN, an out-of-range value is an error.

@spec disabled_config() :: ProcessHub.Hub.recovery_config()

Returns the disabled (default) config.

@spec parse_config(false | true | keyword() | term()) ::
  {:ok, ProcessHub.Hub.recovery_config()}
  | {:error, :invalid_auto_recovery | {:invalid_auto_recovery, atom()}}

Parses the :auto_recovery config field into a normalized map.

Accepts the documented shapes:

  • false — disabled (the default).
  • true — enabled with defaults.
  • keyword() — :reconcile_grace_ms, :reconcile_interval_ms, :cluster_settle_ms, and :remote_manifest ({module, opts} implementing ProcessHub.Storage.RemoteManifest, default nil).

The superseded keys :marker_path, :replay_timeout_ms, :recovery_timeout_ms, and :stopped_row_ttl_ms are deprecated: they are accepted with a WARN and ignored, and will be rejected in a future release.

Returns {:ok, recovery_config}, or {:error, {:invalid_auto_recovery, reason}} for out-of-range values. Unknown shapes return {:error, :invalid_auto_recovery} so the caller can decide whether to fall back to disabled or to refuse to start.

Link to this function

prepare_recovery(hub_id)

View Source
This function is deprecated. The recovery marker is no longer used; this is a no-op. See migration-guide.md.
@spec prepare_recovery(ProcessHub.hub_id()) :: :ok

Deprecated. Armed the next boot for marker-driven replay by deleting the local marker file.

There is no marker any more: every node reconciles its durable registry against the cluster continuously, so recovery after an outage needs no pre-boot step. The function is kept so existing operator tooling keeps running — it logs a warning and returns :ok without touching the filesystem.

Scheduled for removal in a future release.

Link to this function

prepare_recovery_cluster(hub_id)

View Source
This function is deprecated. The recovery marker is no longer used; this is a no-op. See migration-guide.md.
@spec prepare_recovery_cluster(ProcessHub.hub_id()) ::
  {:ok, [node()]} | {:error, :not_alive}

Deprecated. Fanned prepare_recovery/1 out to every hub member.

A no-op for the same reason as prepare_recovery/1; it still reports the hub's members so existing callers keep matching on {:ok, nodes}. Returns {:error, :not_alive} when the hub is not running, as before.

Scheduled for removal in a future release.

@spec recovery_state(ProcessHub.hub_id()) :: :recovering | :normal

Returns the coordinator's current :recovery_state.

Returns :normal when the hub does not exist or was started without :auto_recovery.

Link to this function

registry_delivered(state, node)

View Source

Records that node has delivered its registry data. Only the first round waits on this evidence, so nothing is kept once the hub is :normal.

@spec request_round(ProcessHub.hub_id()) :: :ok

Asks the hub for a reconcile round soon, for a host that knows a start the last round refused can now succeed — a placement target gaining a member, for example. A node joining the hub already does this by itself.

The round does not wait for reconcile_interval_ms, only until a couple of seconds have passed since the last round, and never overlaps a running one. Before the first round it does nothing, since that round comes anyway. Returns :ok; a hub that is not running, or runs without :auto_recovery, does nothing.

Runs the round a waiting request was put off for.

Link to this function

round_due?(state, trigger \\ :sync)

View Source
@spec round_due?(ProcessHub.Coordinator.State.t(), trigger()) :: boolean()

Returns whether trigger may run a round now — the single answer for the first round as well as for later ones.

Triggers are :sync (a completed synchronisation round, the default), :evidence (the settle window closing or a peer's registry data arriving) and :grace (reconcile_grace_ms) and :demand (request_round/1). Rounds never overlap. The first round waits for the evidence gate — the settle window passed and every connected peer the hub knows about heard from — except under :grace, which is the cap and opens it regardless. After that, the sync tick runs at most one round per reconcile_interval_ms, while evidence and demand only wait until a couple of seconds have passed since the last round.

Link to this function

schedule_first_round(hub)

View Source
@spec schedule_first_round(ProcessHub.Hub.t()) :: :ok

Arms the two first-round timers: the settle window, after which silence from the cluster counts as being alone, and reconcile_grace_ms, which opens the round whatever the evidence says.

The grace fires whether or not any peer answered, so :normal is reached in bounded time on every boot. Disabled hubs schedule nothing.

Runs a round in a separate process; replies to the coordinator with {:reconcile_done, result}, which it hands to complete_round/2 — Round.run_safe/2 guarantees a reply whatever happened.

Link to this function

trigger_round(state, trigger \\ :sync)

View Source

Spawns a round when round_due?/2 allows trigger one. Evidence or a demand that has to wait, because a round is running or the last one was too recent, is tried again a couple of seconds later.