Cluster Usage

Use this page when you want to configure cluster membership and start nodes. For runtime internals, routing, wire format, and operational boundaries, see Cluster Architecture.

What You Must Provide

Cluster does not discover peers on its own. Before enabling components.clusterEnabled, prepare the pieces that identify the local node and describe the cluster.

Item	Why it matters
Stable node id	Maps the local process to exactly one `NodeInfo` entry.
Member list	Defines every known node, host, data/API port, replication port, and capability.
Placement mode	Chooses local-only, mirror, or striped key ownership.
Startup role	Explicitly starts the process as Primary or Replica.
Primary endpoint	Lets a Replica find the Primary replication listener.
ACK policy	Decides how much replica confirmation a Primary write waits for.
Transport mode	Chooses plain local/private networking or secure replication.

Treat Cluster as a replication runtime and placement description. Service discovery, client routing, automated failover, and snapshot orchestration still belong to your server, control plane, or deployment tooling.

Choose A Mode

ReplicationMode controls placement and is stored in ClusterConfig.

Mode	Meaning
`STANDALONE`	Local-only operation. Replication is not required.
`MIRROR`	Writes are routed to every data-bearing node.
`STRIPE`	Each key is routed to one data-bearing node using deterministic rendezvous hashing.

STRIPE does not move existing data when the node set changes. Treat membership changes in striped deployments as an operational migration.

Mode Selection Guide

Goal	Start with	Notes
Single process or local tests	`STANDALONE`	Keeps Cluster disabled or local-only.
Keep the same write stream on more than one node	`MIRROR`	Primary ships entries and blob payloads to Replica nodes.
Assign each key to one data-bearing owner	`STRIPE`	Routing is deterministic, but migration is external.
Automatic scale-out or rebalancing	External control plane	Cluster currently exposes primitives, not an autoscaling system.

For the current runtime, MIRROR is the most direct replicated deployment. STRIPE is useful when you are building an upper layer that can route clients, control membership changes, and handle data movement deliberately.

Choose Startup Roles

NodeStartupRole is explicit for non-standalone modes.

Startup role	Runtime role	Notes
`PRIMARY`	`PRIMARY`	The local node must exist in config and have `COORDINATOR_ELIGIBLE`.
`REPLICA`	`REPLICA`	The runtime must resolve a primary node id, host, and replication port.
`AUTO`	rejected for `MIRROR` / `STRIPE`	Automatic role selection is not implemented for clustered modes.

Primary and replica are runtime roles. They are separate from placement mode: a mirrored cluster still has a primary process shipping writes and one or more replica processes consuming them.

Minimal Two-Node Rollout

For a first replicated setup, keep the shape simple and explicit.

Step	Node 1	Node 2
Create `node.id`	`1`	`2`
Write `cluster.akcc`	Same two-node membership	Same two-node membership
Startup role	`PRIMARY`	`REPLICA`
Primary id	`1`	`1`
Replication port	Listen on node 1 `replPort`	Connect to node 1 `replPort`
Data directory	Dedicated node 1 directory	Dedicated node 2 directory

Keep each process on its own dataDir. Do not point two nodes at the same WAL, SST, blob, or VersionLog files.

Prepare Node Identity

Every engine instance has a stable numeric node id. AkkEngine::open() loads it from paths.nodeIdPath; if the file is missing, it creates a random non-zero uint64_t and writes it to disk.

When paths.dataDir is set, the default path is:

<dataDir>/node.id

For non-standalone modes, that id must match one NodeInfo::nodeId in ClusterConfig. In practice, create or copy node.id before first clustered startup, or read the generated id and add it to the config before enabling components.clusterEnabled.

node.id is local identity, not a generated cluster membership service. If you delete it and let the engine create another id, the process may no longer match the node entry that other peers expect.

Define Cluster Config

Cluster configuration is represented by cluster::ClusterConfig. It stores node ids, peer hosts, data/API ports, replication ports, node capabilities, placement mode, and acknowledgement policy.

#include "akk/engine/AkkEngine.hpp"
#include "akk/engine/cluster/ClusterConfig.hpp"

#include <utility>
#include <vector>

namespace engine = akkaradb::engine;
namespace cluster = akkaradb::engine::cluster;

cluster::ClusterConfig makeConfig() {
    std::vector<cluster::NodeInfo> nodes{
        {
            .nodeId = 1,
            .host = "127.0.0.1",
            .dataPort = 7070,
            .replPort = 7170,
            .capabilities = cluster::COORDINATOR_ELIGIBLE | cluster::DATA_BEARING,
        },
        {
            .nodeId = 2,
            .host = "127.0.0.1",
            .dataPort = 7071,
            .replPort = 7171,
            .capabilities = cluster::DATA_BEARING,
        },
    };

    cluster::AckPolicy ack{};
    ack.mode = cluster::AckPolicyMode::ALL_TARGETS;
    ack.stage = cluster::AckStage::APPLIED;

    return cluster::ClusterConfig{
        std::move(nodes),
        cluster::ReplicationMode::MIRROR,
        ack,
    };
}

NodeInfo::host is the address peer nodes use. replPort is the replication listener port. dataPort is recorded as the public data/API port for that node; it does not start the API server by itself.

Save A Durable Config

ClusterConfig can be saved and loaded independently from EngineOptions. Store the same membership view on every node, then select the local role at runtime.

cluster::ClusterConfig cfg = /* build or load from your own config source */;
cfg.save("data/node-1/cluster.akcc");
cfg.save("data/node-2/cluster.akcc");

Then point each process at its local copy:

EngineOptions opts;
opts.components.clusterEnabled = true;
opts.paths.dataDir = "data/node-2";
opts.paths.clusterConfigPath = "data/node-2/cluster.akcc";

cluster.akcc stores the durable membership, placement, capabilities, and ACK policy. Runtime-only values such as local startup role, secure pins, primary overrides, and bind host stay in EngineOptions.

Start A Primary

Enable the cluster component before AkkEngine::open(). The engine uses opts.cluster.config when provided; otherwise it loads paths.clusterConfigPath.

engine::AkkEngineOptions opts;
opts.paths.dataDir = "data/node-1";
opts.paths.nodeIdPath = "data/node-1/node.id";
opts.components.clusterEnabled = true;

opts.cluster.config = makeConfig();
opts.cluster.runtime.startupRole = cluster::NodeStartupRole::PRIMARY;
opts.cluster.runtime.transportMode = cluster::TransportMode::PLAIN;
opts.cluster.runtime.replBindHost = "127.0.0.1";

auto db = engine::AkkEngine::open(std::move(opts));

Primary startup fails if the local node id is missing from config or the local node is not COORDINATOR_ELIGIBLE. The replication listener uses the local node's configured replPort.

Only one process should be started as Primary for a given write stream. The current Cluster runtime does not provide leader election or fencing, so preventing split-brain is the responsibility of the deployment layer.

Start A Replica

Replica startup needs a primary id. If primaryNodeId exists in ClusterConfig, the runtime can fill primaryHost and primaryReplPort from that config entry.

engine::AkkEngineOptions opts;
opts.paths.dataDir = "data/node-2";
opts.paths.nodeIdPath = "data/node-2/node.id";
opts.components.clusterEnabled = true;

opts.cluster.config = makeConfig();
opts.cluster.runtime.startupRole = cluster::NodeStartupRole::REPLICA;
opts.cluster.runtime.primaryNodeId = 1;
opts.cluster.runtime.transportMode = cluster::TransportMode::PLAIN;

auto db = engine::AkkEngine::open(std::move(opts));

You can override primary resolution directly:

opts.cluster.runtime.primaryNodeId = 1;
opts.cluster.runtime.primaryHost = "10.0.0.10";
opts.cluster.runtime.primaryReplPort = 7170;

Replica startup fails when the primary id is missing, points at the local node, is not coordinator-eligible, or cannot resolve to a host and replication port.

The replica client reconnects in the background. If a connection attempt or handshake fails, it waits about 200 ms and tries again while the engine remains open.

Add Or Restart A Replica

Use this path when the Replica is new or has been down only briefly enough for an external snapshot or the in-memory catch-up window to cover the gap.

Stop writes or take a consistent source copy if the Replica needs a fresh snapshot.
Create a dedicated dataDir and stable node.id for the Replica.
Place a cluster.akcc that includes both the Primary and the Replica.
Start the process with NodeStartupRole::REPLICA and the expected primaryNodeId.
Watch logs and stats until the Replica has connected and applied new entries.

The Primary keeps only a short in-memory entry buffer. A Replica that missed more history than that buffer contains needs an external data copy before reconnecting.

Stop Or Remove A Replica

For a planned removal:

Stop the Replica process.
Update the durable cluster config used by the remaining nodes.
Restart or reload the processes according to your deployment model.
Review the Primary ACK policy. ALL_TARGETS and QUORUM depend on live replica counts.
Remove or archive the Replica data directory only after you no longer need it for recovery.

Current membership changes are configuration changes. Cluster does not rebalance striped data or migrate files automatically.

Configure ACK Policy

AckPolicy controls how long primary writes wait for replica confirmation.

Policy	Meaning
`NONE`	Do not wait for replica acknowledgements.
`ALL_TARGETS`	Wait until every currently live replica has acknowledged the sequence.
`QUORUM`	Wait until at least `quorum` live replicas have acknowledged the sequence.

AckStage controls what the acknowledgement means.

Stage	Replica behavior
`RECEIVED`	ACK after receiving and decoding the entry bytes.
`APPLIED`	ACK after applying the entry to the local engine.
`DURABLE`	Force local durability sync, then ACK.

Higher stages give stronger confirmation and higher write latency. Blob frames are sent to replicas but are not waited on by the entry acknowledgement policy.

ACK Policy Examples

cluster::AckPolicy fireAndForget;
fireAndForget.mode = cluster::AckPolicyMode::NONE;

cluster::AckPolicy appliedOnAll;
appliedOnAll.mode = cluster::AckPolicyMode::ALL_TARGETS;
appliedOnAll.stage = cluster::AckStage::APPLIED;

cluster::AckPolicy oneDurableReplica;
oneDurableReplica.mode = cluster::AckPolicyMode::QUORUM;
oneDurableReplica.quorum = 1;
oneDurableReplica.stage = cluster::AckStage::DURABLE;

DURABLE asks the Replica to force its local durability path before ACKing. It is the strongest built-in stage, but it is also the most expensive. Choose it for flows where recovery semantics matter more than write latency.

Configure Secure Transport

TransportMode::SECURE uses the native secure channel before replication frames are exchanged. If secure.identitySeedPath is empty and paths.dataDir is set, the runtime uses:

<dataDir>/cluster.identity

opts.cluster.runtime.transportMode = cluster::TransportMode::SECURE;
opts.cluster.runtime.secure.identitySeedPath = "data/node-1/cluster.identity";
opts.cluster.runtime.secure.expectedPrimaryNodeId = 1;

TransportMode::PLAIN is allowed only when every advertised node host is loopback, link-local, unique-local IPv6, or private IPv4. Public/WAN hosts are rejected and should use SECURE.

Pin Primary Identity

For secure replication, a Replica can require the Primary to match the expected node id:

opts.cluster.runtime.primaryNodeId = 1;
opts.cluster.runtime.secure.expectedPrimaryNodeId = 1;

For stricter peer validation, populate pinnedPeers with the expected public key for each cluster node id. The key is derived from that node's cluster.identity seed. Keep the identity seed durable and backed up; replacing it changes the node's secure identity.

Durable Files

When paths.dataDir is set, cluster-related paths default to:

Path	Default
`paths.clusterConfigPath`	`<dataDir>/cluster.akcc`
`paths.nodeIdPath`	`<dataDir>/node.id`
Cluster manifest	`<dataDir>/cluster.akmf`
Secure identity seed	`<dataDir>/cluster.identity` when secure mode needs a default

cluster.akcc stores membership and policy. Runtime-only values such as replBindHost, primary overrides, secure pins, and transport mode are not serialized into the cluster config.

Operational Playbooks

Planned Replica Restart

Stop the Replica, keep its data directory intact, and start it again with the same node.id and config. If the outage is short, the Primary may replay buffered entries during handshake. If it was long, refresh the Replica from a snapshot before reconnecting it.

Primary Process Restart

Stop writes before restarting the Primary. Start it again with the same node.id, config, and PRIMARY role. Replicas reconnect in the background and resume from their last local sequence within the available catch-up window.

Manual Failover

Promoting another node is an operational procedure, not automatic runtime behavior. Choose the target, ensure its data is current enough for your application, update the config and client routing, then start exactly one process as PRIMARY.

Startup Failure Checklist

Symptom	Check
`selfNodeId not found`	The binary `node.id` value does not match any `NodeInfo::nodeId`.
Primary startup is rejected	The local node does not have `COORDINATOR_ELIGIBLE`.
Replica startup is rejected	`primaryNodeId` is missing, equals local node id, or resolves to a non-coordinator node.
Config load fails	`cluster.akcc` is missing, truncated, wrong version, or has a CRC mismatch.
Plain transport is rejected	One or more `NodeInfo::host` values are public/WAN addresses.
Cluster backend is unavailable	The cluster runtime backend was not linked or could not be loaded from `runtimeBackendPath`.

Runtime Checklist

Symptom	Check
Replica keeps reconnecting	Primary host, `replPort`, transport mode, and secure identity expectations.
Writes do not wait for replicas	`AckPolicyMode` may be `NONE`, or no live replicas satisfy the policy.
`QUORUM` never completes	`quorum` may be larger than the number of connected replicas that can ACK the chosen stage.
Blob data is missing after reconnect	Blob frames are not retained in the Primary catch-up buffer; use snapshot/copy for long gaps.
Striped reads miss data after membership change	`STRIPE` ownership changed without an external migration.