@spec await_normal(ProcessHub.hub_id(), non_neg_integer()) :: :ok | {:error, :timeout}

Blocks until the coordinator reaches :normal or timeout_ms elapses.

Returns :ok on reaching :normal (immediately when the hub does not exist or has no recovery), or {:error, :timeout} otherwise.

@spec cspec_count(ProcessHub.hub_id()) :: non_neg_integer()

Reports the cspec_count currently persisted, used for telemetry measurements at recovery boot.

@spec delete_marker(nil | String.t()) :: :ok | {:error, term()}

Deletes the marker file at path. Returns :ok if the marker is absent (no-op). Returns {:error, posix()} on permission/IO failure.

@spec disabled_config() :: ProcessHub.Hub.recovery_config()

Returns the disabled (default) config.

@spec dispatch_blocking_hook(
  :ets.tid(),
  ProcessHub.Service.HookManager.hook_key(),
  term(),
  pos_integer()
) :: :ok

Dispatches a hook synchronously, awaiting each handler's reply. Each handler is wrapped in try/catch so a misbehaving handler can neither crash the coordinator nor (via the per-handler timeout) hang it past the overall recovery_timeout_ms.

Handlers are executed in registered order; per-handler timeouts are computed as the remaining slice of the total budget. A handler raising is logged at WARN and the dispatch continues to the next handler.

@spec dispatch_state_changed_hook(ProcessHub.Hub.t(), atom(), atom(), atom(), map()) ::
  :ok

Dispatches the recovery_state_changed hook.

Fires on every lifecycle moment: :init → :recovering (:marker_absent), :init → :normal (:marker_present/:disabled), and :recovering → :normal (:replay_complete/:recovery_timeout). measurements carries the per-moment counts (e.g. cspec_count, succeeded, elapsed_ms).

@spec enqueue(ProcessHub.Hub.t(), term()) :: ProcessHub.Hub.t()

Appends a deferred cluster event to the queue.

@spec gate_closed?(ProcessHub.Hub.t()) :: boolean()

Returns true when cluster events must be deferred (gate closed).

@spec init_marker(atom(), nil | String.t(), boolean()) ::
  {ProcessHub.Hub.recovery_marker(), ProcessHub.Hub.recovery_state(),
   :normal | :recovery}

Builds the marker config for a hub: resolves the absolute path and computes the initial recovery_state from the marker + filesystem.

Returns {marker, initial_state, resolved_mode}. The caller (the coordinator's init/1) stamps the marker onto %Hub{} and decides what to do next via start/2.

@spec marker_exists?(nil | String.t()) :: boolean()

Returns whether the marker file at path exists.

nil paths and unreadable parents return false (selecting recovery mode is the safe direction).

@spec maybe_inject_replay_flag(keyword(), atom(), map() | struct()) :: keyword()

Injects recovery_replay: bool into backend opts when :auto_recovery is enabled. Honours an explicitly set value if the caller already provided one.

@spec open_gate(ProcessHub.Hub.t(), atom(), map()) :: ProcessHub.Hub.t()

Opens the recovery gate, dispatches the transition hook, and drains queued events.

@spec parse_config(false | true | keyword() | term()) ::
  {:ok, ProcessHub.Hub.recovery_config()}
  | {:error, :invalid_auto_recovery | {:invalid_auto_recovery, atom()}}

Parses the :auto_recovery config field into a normalized map.

Accepts the documented shapes:

• false — disabled.
• true — enabled with defaults.
• keyword() — explicit :recovery_timeout_ms and the optional :marker_path override.

Returns {:ok, recovery_config} or {:error, reason} for out-of-range values. Unknown shapes return {:error, :invalid_auto_recovery} so the caller can decide whether to fall back to disabled or to refuse to start.

@spec prepare_recovery(ProcessHub.hub_id()) :: :ok | {:error, term()}

Deletes the recovery marker on the local node so the next coordinator boot selects recovery mode.

Safe to call on a running hub — only the marker file is touched. Idempotent (returns :ok even when the marker is absent). Hubs started without :auto_recovery are no-ops.

See prepare_recovery_cluster/1 for the RPC fan-out variant.

@spec prepare_recovery_cluster(ProcessHub.hub_id()) ::
  {:ok, [node()]} | {:partial, [node()], [node()]} | {:error, term()}

Fans out prepare_recovery/1 to every member of the hub via :rpc.multicall/4.

Returns:

• {:ok, [node]} — every member acked.
• {:partial, [acked], [unreachable]} — at least one peer failed.
• {:error, reason} — cluster API itself failed (hub not running).

@spec recovery_state(ProcessHub.hub_id()) :: :recovering | :normal

Returns the coordinator's current :recovery_state.

Returns :normal when the hub does not exist or was started without :auto_recovery.

@spec replay(ProcessHub.Hub.t(), ProcessHub.Hub.recovery_config()) :: replay_result()

Runs the persisted-registry replay synchronously inside the calling process (the coordinator).

Sequence:

1. dispatch the pre_recovery_replay hook synchronously (blocking),
2. iterate ProcessRegistry.dump/1,
3. call Distributor.compose_start_operation/3 once with all child specs,
4. wait up to recovery_timeout_ms for completion (best effort),
5. dispatch post_recovery_replay (async).

Returns a replay_result/0 summary that the coordinator uses to update state and dispatch the recovery_state_changed hook.

Per-child failures during replay are logged at WARN and surface in the failed count; they never abort the replay loop. If recovery_timeout_ms elapses before completion, the function returns with reason: :replay_timeout. Replay continues in the background.

@spec replay_measurements(map()) :: map()

Builds the recovering → :normal hook measurements from a replay result.

@spec resolve_marker_path(atom(), nil | String.t()) :: String.t()

Resolves the absolute marker path for a hub.

If path is non-nil, returns it as-is. Otherwise resolves to <:filename.basedir(:user_data, "process_hub")>/<hub_id>/cluster.healthy.

@spec resolve_mode(boolean(), boolean()) :: :normal | :recovery

Resolves the effective recovery mode at coordinator init from the marker.

• marker_enabled? is false → :normal
• marker_exists? is true → :normal (trust peers)
• otherwise → :recovery (rebuild from disk)

@spec spawn_replay(ProcessHub.Hub.t()) :: ProcessHub.Hub.t()

Runs the replay in a separate process; replies with {:marker_replay_done, result}.

@spec start(ProcessHub.Hub.t(), :normal | :recovery) :: ProcessHub.Hub.t()

Drives the marker-gated lifecycle from coordinator init.

• marker disabled → no-op.
• mode :normal → emit :skipped, write the marker.
• mode :recovery → emit :started, schedule the timeout, and send self() :start_marker_replay so the replay runs after init returns.

Returns the updated state.

@spec timeout_measurements(ProcessHub.Hub.t()) :: map()

Builds the recovering → :normal hook measurements for the timeout ceiling.

@spec write_marker(nil | String.t()) :: :ok | {:error, term()}

Writes a zero-byte marker file at path, creating parent directories. Idempotent on success; returns {:error, posix()} on IO failure.