Aof durability

[jjuleslasarte]

AOF-based Durability (Sync Replication)

Summary

This PR adds an AOF-based durability mode where Valkey blocks client responses in the output buffer until the underlying durability provider (AOF fsync) acknowledges the write. It is milestone one in the durability plan (here)

When durability yes and appendfsync always are both enabled (looking for feedback on these configurations, and whether we want two, or not), a client writing SET foo bar won't receive +OK until the data is fsynced to disk — giving zero-data-loss guarantees without requiring application-level WAITAOF.

The design is "provider-pluggable", as the same building block of reply tracking/blocking will be used to implement sync-replication w.r.t replicas. The durability code accepts multiple providers (AOF, replicas, etc.) and computes consensus as the MIN of all enabled providers' acknowledged offsets (AND semantics). This PR ships only the built-in AOF provider; replica-based providers will follow in other milestones.

Design decisions

• Response blocking vs. command blocking (or, WAL vs WBL): Clients are not paused before executing — the command runs immediately and the response is buffered in the client output buffer (COB). A "disallowed byte offset" marker prevents the networking layer from flushing beyond that point until the provider catches up. This keeps the event loop responsive.
• Uncommitted key tracking: When durability is enabled, modified keys are tracked per-database with the replication offset they were "dirtied" at. Read commands accessing uncommitted keys are response-blocked same as the writes: their replies are held in the client output buffer until the durability providers acknowledge the offset at which the key was last modified. This ensures clients only see durable data without rejecting the command outright. The same blocking mechanism used for write responses applies to reads of dirty keys.
• Deferred side-effects: Keyspace notifications and tracking invalidation messages are deferred until the write is acknowledged, so pub/sub subscribers and client-side caches don't learn about changes before they're durable.
• AOF fsync offloaded to IO threads: To avoid blocking the main thread on every fsync with appendfsync always, the write+fsync is offloaded to IO threads when available.

How to Review

I've split the code into commits that can be more or less reviewed alone. Following the order is probably best. Reviewers are also welcome to review the whole thing at once if that is preferred.

#	Commit	What to look for
1	aof: offload appendfsync=always flush+fsync to IO threads	Standalone performance improvement. Review the atomic state machine in aof.c, the generic trySendJobToIOThreads() in io_threads.c, and the has_background_work parameter.
2	durability: add pluggable durability provider interface	Self-contained abstraction. Review durability_provider.h for the interface, the consensus calculation (MIN/AND), and the built-in AOF provider.
3	durability: add deferred task system for post-ack execution	Review the task type registry pattern, lifetime management of client pointers in key invalidation tasks, and the pending→official task list promotion flow.
4	durability: add uncommitted key tracking per database	Review the per-DB hashtable, lazy purge on read, incremental cleanup in serverCron, and MULTI/EXEC buffering of pending keys.
5	durability: add reply blocking and wire into server subsystems	This is the main integration commit. Review: (a) the COB blocking mechanism in reply_blocking.c, (b) the pre/post command hooks in server.c, (c) integration points in db.c, networking.c, notify.c, script.c, module.c.
6	build: add durability source files to Makefile and CMake	Mechanical build stuff
7	tests: unit tests for reply blocking	gtest-based. Covers blocking/unblocking mechanics, offset tracking, provider consensus.
8	tests: integration tests for durability	Tcl-based end-to-end. Covers AOF blocking, provider pause/resume, uncommitted key rejection, MULTI/EXEC, Lua scripts, function store.

Configuration

durability yes          # Enable durability mode (default: no)
appendonly yes          # AOF must be on
appendfsync always      # Required for the AOF provider to actually block

New INFO section

INFO durability reports blocking/unblocking stats, per-type counters (read/write/other), cumulative block times, and uncommitted key counts.

New DEBUG commands

• DEBUG durability-provider-pause <name> — Freeze a provider's offset (for testing)
• DEBUG durability-provider-resume <name> — Resume a frozen provider

[jjuleslasarte]

@murphyjacob4 @zuiderkwast @hpatro @madolson

[murphyjacob4]

durability yes

Do you think we need a separate config for this? If you set up fsync always, can we imply that durability yes makes sense? I would prefer to kill the durability no and appendfsync always combination if it doesn't have any value

[yairgott]

Response blocking vs. command blocking (or, WAL vs WBL): Clients are not paused before executing — the command runs immediately and the response is buffered in the client output buffer (COB).

Should we factor in available memory before executing the command to avoid the over-buffering which may introduce OOM risk?

[jjuleslasarte]

durability yes

Do you think we need a separate config for this? If you set up fsync always, can we imply that durability yes makes sense? I would prefer to kill the durability no and appendfsync always combination if it doesn't have any value

Yeah, I went back and forth on this. I had the flag left over from the initial draft and figured it might be useful to not enable this since I wasn't sure whether we'd do a major version or minor with this change. I can remove

[jjuleslasarte]

Response blocking vs. command blocking (or, WAL vs WBL): Clients are not paused before executing — the command runs immediately and the response is buffered in the client output buffer (COB).

Should we factor in available memory before executing the command to avoid the over-buffering which may introduce OOM risk?

Good point, we should have a mechanism for this. Let me think through the options -- a proactive one might be harder (as we need to estimate the output before execution) but we can probably track the ammount of pending responses (or pending writes to the durability providers) and start throttling (rejecting) writes after a certain threshold?

[zuiderkwast]

Regarding the durability config, the way I see it is that appendfsync always already gives durability (client won't receive +OK until the data is fsynced to disk). This feature just optimizes it, so a new config is not necessary IMO.

[yairgott]

Good point, we should have a mechanism for this. Let me think through the options -- a proactive one might be harder (as we need to estimate the output before execution) but we can probably track the ammount of pending responses (or pending writes to the durability providers) and start throttling (rejecting) writes after a certain threshold?

Yeah, proactive would be challenging but a reactive approach might be good enough. We could track the total consumed output buffer and initiate throttling once a predefined threshold is reached.

Valkey’s existing client-output-buffer-limit supports hard and soft limits, but its semantics, which lead to closing the connection, might be too disruptive in this context. Ideally, once a threshold is breached, we would suspend command ingestion for that specific client and only resume once the output buffer consumption falls back below the threshold.

In addition, pointing out that this client suspension should be conditioned on the ability to zero-copy responses (e.g., the requested key is not robj based).

[jjuleslasarte]

Regarding the durability config, the way I see it is that appendfsync always already gives durability (client won't receive +OK until the data is fsynced to disk). This feature just optimizes it, so a new config is not necessary IMO.

Yeah, makes sense to me. I will remove it in the next commit, along with other feedback!