VersionLog Architecture

Use this page when you need to understand how VersionLog works internally. For configuration and API examples, see VersionLog Usage. For production monitoring and maintenance commands, see VersionLog Operations.

Write Path

When VersionLog is enabled, each engine mutation flows through the same sequence as the live storage path:

1. Reserve a new engine sequence number.
2. Append to WAL when WAL is enabled.
3. Append a VersionLog entry.
4. Apply the mutation to MemTable.
5. Publish the VersionLog commit frontier for COMMIT_ORDER visibility.
6. Ship the mutation through the cluster runtime when cluster mode is enabled.

COMMIT_ORDER history and getAt() never expose a VersionLog entry ahead of the engine mutation that committed it. APPLIED is the lower-latency option that can expose appended history earlier.

Entry Format

Each .akvlog segment starts with a 32-byte AKV5 v1 header. Entries store a packed header, raw key bytes, stored value bytes, and a trailing CRC32C.

Field	Meaning
seq	Engine sequence number.
sourceNodeId	Local node, replica source node, or rollback sentinel.
timestampNs	Append timestamp.
flags	Storage flags, including compression, rollback, and retention-base markers.
keyFp64	Key fingerprint for indexing.
key	Raw key bytes.
value	Raw value, compressed value payload, or blob reference bytes.
crc32c	Entry checksum.

The current entry size limit is 32 MiB. Keys larger than uint16_t and values larger than uint32_t are rejected by the serializer.

Compression

With codec = ZSTD, VersionLog attempts to compress each value. The stored value begins with a four-byte uncompressed size followed by the Zstd payload. If compression is not smaller than the raw value plus the prefix, the entry is stored uncompressed.

Public VersionEntry values are always exposed as decompressed bytes, and the internal compression flag is stripped before returning to callers.

Segments And Sidecars

The base logPath is segment 0; later segments insert -seg-N before the extension. For example:

history.akvlog
history-seg-1.akvlog
history-seg-2.akvlog

Each segment may have a derived .akvidx sidecar:

history.akvidx
history-seg-1.akvidx

Sidecars store a Bloom filter and a sorted mapping from key fingerprint to (seq, VLog byte offset) records. history() uses the Bloom filter to skip impossible segments and reads only matching key offsets. getAt() can binary-search per-key sequences and read the selected record directly.

The .akvlog segment is authoritative. A missing, stale, malformed, or unwritable .akvidx sidecar causes a safe segment scan and never makes valid VLog data unreadable. Recovery, rotation, clean close, retention-base creation, and the maintenance tool attempt to regenerate sidecars.

Active Index Memory

VersionLog does not keep all historical values in memory. Closed segments are represented by compact segment metadata and sidecar indexes. The active segment keeps only key-to-offset metadata while it is mutable, bounded by segmentBytes.

If segmentBytes = 0, active metadata is disabled for the single-file mode, and reads safely fall back to file scans rather than accumulating unbounded in-memory history.

Async writes that have not reached disk are retained as a small resident overlay so readers can see the configured visibility boundary.

Parallel Lane Storage

PARALLEL write admission assigns each key to a lane by stable fingerprint. Each lane owns its active segment, queue, worker, active index, and durable tail. Different lanes can persist concurrently, while the same key remains ordered through one lane.

PARALLEL mode writes an .akvtail file beside each active lane segment. The tail records the durable contiguous byte length. Recovery scans only that prefix and ignores bytes after it, so interrupted lane appends cannot become visible history.

Recovery

Recovery discovers all segment files, validates file headers, validates each entry length and CRC32C, reconstructs segment metadata, rebuilds sidecars, and restores the active index for the latest segment.

EAGER recovery performs this before open returns. BACKGROUND recovery returns after opening the active file, then VersionLog and AkkEngine operations wait for the recovery barrier before they observe or mutate engine state.

Corruption is never accepted. Recovery errors include VersionLog: context with the file path and, when available, entry offset and sequence number.

Retention

Retention is segment-granular and disabled by default. retentionDays and retentionMinCommitSeq are ORed: a closed segment can be pruned when its file age reaches the age boundary or its highest sequence is below the sequence boundary.

Before deleting expired segments, VersionLog writes synthetic VLOG_FLAG_RETENTION_BASE entries for keys whose latest value at the retained boundary would otherwise disappear. This preserves the state needed by getAt(), history(), and rollback from the retained window forward.

Queries before the base boundary remain unavailable. Retention never deletes the active segment and keeps segments that reach beyond the current commit frontier.

Blob Interaction

Blob GC is rejected while VersionLog is enabled. Historical entries may still refer to previous blob values, and deleting those blobs would break point-in-time reads. VersionLog retention reduces history segments; it does not by itself prove that a blob is unreferenced.

Rollback Entries

Rollback operations use sourceNodeId = ROLLBACK_NODE and set VLOG_FLAG_ROLLBACK. Stats expose those entries through vlog.rollbackEntries.

Rollback restores state by writing new records. If the target sequence has no prior value for a key, rollback writes a tombstone. If the target sequence has a value, rollback writes that value again as the current state.

Test Coverage

Current smoke coverage checks:

Test behavior	Covered behavior
Admission and visibility	COMMIT_ORDER, APPLIED, serial admission, parallel admission, background recovery.
Segment index fallback	Missing or corrupt .akvidx falls back to authoritative segment scans.
Recovery corruption	Corrupt CRC and truncated committed entries fail recovery.
Parallel durable tail	Recovery ignores bytes after the .akvtail durable prefix.
Concurrent reads/writes	Parallel writers and readers preserve sorted histories and values.
Maintenance tool	validate and rebuild-indexes rebuild derived sidecars and refuse missing logs.