# `Slap.Cluster`
[🔗](https://github.com/mjrusso/slap/blob/slap_cluster-v0.4.0/slap_cluster/lib/slap/cluster.ex#L1)

Runs N SlateDB databases ("shards") in one bucket: opens and closes them,
turns SlateDB's writer fencing into supervised events, fans out durability
notifications, and finds the shard for a key. It knows nothing about what
the application stores; the application puts its own per-shard processes
on top.

Define one module per cluster. `use` takes `:otp_app` and optionally
`:defaults`, options that the application environment and `start_link/1`
can override (for example `shard_children` in a library that builds on
this one). `:settings` maps are merged by key, in this order: defaults,
application environment, then `start_link/1` options. Each source
overrides only the settings it contains.

    defmodule MyApp.Cluster do
      use Slap.Cluster, otp_app: :my_app
    end

    # config/runtime.exs
    config :my_app, MyApp.Cluster,
      store: {:url, "s3://bucket/prefix", aws_endpoint: "https://rustfs:9000"},
      shards: 64,
      settings: %{flush_interval: "10ms"},
      cache: [capacity_bytes: 2 * 1024 * 1024 * 1024],
      shard_children: {MyApp.ShardChildren, :child_specs, []}

and add `MyApp.Cluster` to the application's supervision tree. On start
it checks that the store honours conditional writes (see
`Slap.SlateDB.probe_store/3`), then the strategy opens the shards. With the
default `Slap.Cluster.Strategy.Local`, startup attempts every shard before
`start_link/1` returns; failed opens retry in the background. With
`probe: :background`, it can return before any shard opens. Use
`await_ready/2` before boot work that needs every shard.

## Options

  * `:store` - required. A store, as for `Slap.SlateDB.open/2`. Shard `n` is the
    database `<path>/shard-00n` in it.
  * `:shards` - required. The number of shards. Fixed for the life of the
    data: `shard_for/1` depends on it.
  * `:path` - the path of the shards in the store (default `""`).
  * `:settings` - SlateDB settings for every shard, as for
    `Slap.SlateDB.open/2`. SlateDB's `flush_interval` defaults to 100 ms,
    which can add up to 100 ms before a durable write's WAL flush begins.
  * `:cache` - `[capacity_bytes: n]` for one block cache shared by this
    node's shards, `:disabled`, or `nil` (default) for SlateDB's own cache
    per shard.
  * `:merge_operator` - as for `Slap.SlateDB.open/2`.
  * `:strategy` - `module` or `{module, opts}` (default
    `Slap.Cluster.Strategy.Local`). See `Slap.Cluster.Strategy`.
  * `:shard_children` - `{mod, fun, args}` or a 1-arity function, called
    with the shard's `Slap.Cluster.Shard` context (prepended to `args`),
    returning child specs to start for each shard. See "Shard children".
  * `:child_options` - application settings in the shard context (default
    `[]`). The cluster does not interpret them. When `:shard_children` is
    `{module, function, args}`, the module may define
    `c:Slap.Cluster.ShardChildren.validate_child_options!/1` to reject invalid
    settings before startup.
  * `:probe` - `true` (default) runs the conditional write probe before
    startup completes and fails startup if the probe fails. `:background`
    starts without shards and retries after 1 s, backing off to at most
    30 s. Calls return `{:error, :unassigned}` until the probe passes and
    the shards open. `false` skips the probe.
  * `:close_timeout` - ms to wait for a shard's database to close on stop
    (default 30,000).
  * `:lag_interval` - ms between durability lag telemetry samples per
    shard (default 1,000).

## Shard children

When a shard opens on a node, its children start under a supervisor, after
the database is open. They get a `Slap.Cluster.Shard` context with the
database handle, and name themselves with `via/2`.

When the shard stops (the cluster shuts down, the strategy moves it, or
it is fenced), the children stop first, while the database is still open,
then the database is closed. Children that need to fail in-flight work
should trap exits and check `shard_status/1` in `terminate/2`: `:stopping`
or `:fenced`. This works for processes anywhere under the children, not
only for the direct ones, which is why it is not the exit reason.

If the database process crashes, the shard stops, and the strategy
decides whether to reopen it (`handle_shard_down/3` with `:crashed`).

## Durability

Writes go straight to the context's `db`. `notify_when_durable/3` sends a
message once a write's `seq` is durable. All notifications for a shard
come from one process, which releases waiters lowest seq first, so a
process that asks in seq order (as a single writer does) receives them in
seq order. One SlateDB subscription per shard serves every waiter.

## Telemetry

Events are sent under `[:slap, :cluster, ...]`: `[:shard, :start | :stop | :fenced]`,
`[:durability, :lag]` (`%{lag: last_write_seq - durable_seq}`, every
`:lag_interval`), and `[:probe]`. Metadata includes `:cluster` and
`:shard`.

Invalid configuration and call arguments raise. Routing failures return
`{:error, reason}` from `call/4` and `lookup/2`.

# `assignments`

```elixir
@spec assignments(module()) :: %{required(non_neg_integer()) =&gt; term()}
```

Every shard's location as this node sees it: `{:local, pid}` (the shard's
database process), `{:remote, node}` or `:unassigned`.

# `await_ready`

```elixir
@spec await_ready(module(), timeout()) :: :ok | {:error, :timeout}
```

Waits until every configured shard has completed startup on its owner,
including its direct application children, with the inspection scope
described in `status/2`. Returns `:ok` or `{:error, :timeout}`.

The timeout covers the whole wait, including remote checks and retries.
`:infinity` keeps waiting and rediscovering owners without a deadline.
This checks lifecycle state; use `Slap.SlateDB.probe_store/3` to check
storage reachability.

# `call`

```elixir
@spec call(module(), non_neg_integer(), {module(), atom(), list()}, keyword()) ::
  {:ok, term()} | {:error, :unassigned | :not_owner | {:erpc, term()}}
```

Runs `mod.fun(ctx, ...args)` with the context of `shard` on the node that
owns it: here when the shard is open on this node, otherwise over `:erpc`
on the owner (multi-node strategies need distributed Erlang). Returns
`{:ok, result}` for the application's result, or a routing error:

  * `{:error, :unassigned}` - no node owns the shard (for example during
    a failover), after the retries.
  * `{:error, :not_owner}` - the owner the strategy named no longer has
    the shard, after the retries.
  * `{:error, {:erpc, reason}}` - the owner could not be reached
    (`:noconnection`) or the call timed out (`:timeout`).

On `:not_owner`, `:unassigned`, or an owner that cannot be connected to
(the call was not sent), the strategy's view is
refreshed and the call retried, up to `:retries` times (default 3), after
a jittered delay of about `:retry_delay` ms (default 100) times the
attempt. `:timeout` (ms, default 30,000; `:infinity` for none) bounds
the remote route, including retries and connecting. A local `mod.fun`
runs in the caller and must enforce its own timeout. `mod` must be
loaded on the owner. Connecting to an owner that is not connected yet
waits at most a second.

Telemetry: `[:slap, :cluster, :call, :remote]` for each remote call and
`[:slap, :cluster, :call, :not_owner]` for each stale owner.

# `durable_seq`

```elixir
@spec durable_seq(Slap.Cluster.Shard.t()) :: non_neg_integer()
```

The shard's durable sequence number. See `Slap.SlateDB.durable_seq/1`.

# `local_shards`

```elixir
@spec local_shards(module()) :: [non_neg_integer()]
```

The shards open on this node, in order.

# `lookup`

```elixir
@spec lookup(module(), non_neg_integer()) ::
  {:ok, {:local, Slap.Cluster.Shard.t()} | {:remote, node()}}
  | {:error, :unassigned}
```

Where `shard` is: `{:ok, {:local, ctx}}` with its context when it is open
on this node, `{:ok, {:remote, node}}` when the strategy places it on
another node, or `{:error, :unassigned}`.

# `notify_when_durable`

```elixir
@spec notify_when_durable(
  Slap.Cluster.Shard.t(),
  non_neg_integer(),
  term(),
  keyword()
) :: :ok
```

Sends `{:slap_cluster_durable, tag}` to the calling process (or
`:to`) once the shard's durable sequence number reaches `seq`, the value
a write returned. If it already has, the message is sent at once.

A process that asks in seq order gets the notifications in seq order, and
waiters that become durable together are released lowest seq first.
Notifications are dropped if the shard stops first; check
`shard_status/1` then.

# `shard_for`

```elixir
@spec shard_for(module(), binary()) :: non_neg_integer()
```

The shard for `key`: `xxh64(key) mod shards`. The hash is pinned; see
`Slap.Cluster.Hash`.

# `shard_status`

```elixir
@spec shard_status(Slap.Cluster.Shard.t()) :: :running | :stopping | :fenced
```

`:running`, `:stopping` (the shard is being closed or moved) or `:fenced`
(another writer opened its database).

# `status`

```elixir
@spec status(module(), timeout()) :: {:ok, map()} | {:error, :timeout | :unavailable}
```

Reports current lifecycle state without accessing storage.

Returns `{:ok, %{ready?: boolean, probe: probe, shards: shards}}`. `shards`
maps each configured shard number to `%{node: node, status: status,
last_error: reason}`. Status is `:running`, `:starting`, `:stopping`,
`:fenced`, `:crashed`, `:unassigned`, or `:unavailable`. A running shard has
completed startup, including its direct application children. Child restarts
are checked too. An application-provided supervisor is checked for a live
process; its descendants are outside the readiness check. Remote placements
are confirmed with their owning hosts, which are queried concurrently.

`probe` reports this node's startup probe, with `:status` (`:pending`,
`:passed`, `:failed`, or `:skipped`) and `:last_error`. Hosts retain probe
and shard failures after recovery, until the host restarts. An unavailable
shard reports the current owner-query error in `:last_error`.
Readiness is rechecked on every call and may change immediately afterward.
Use `Slap.SlateDB.probe_store/3` to check current storage reachability.

`timeout` bounds the entire query, including remote checks. Returns
`{:error, :timeout | :unavailable}` if the query cannot complete locally.

# `via`

```elixir
@spec via(module(), non_neg_integer(), term()) :: {:via, Registry, {atom(), term()}}
```

A `:via` name for a per-shard process, registered in the cluster's
registry on the node that owns the shard.

---

*Consult [api-reference.md](api-reference.md) for complete listing*