Failure Handling and Edge Case Design

This document defines how Redis Stream Coordinator handles failure modes rather than only listing individual incidents. The design rule is that every coordinator workflow must be recoverable from the Redis group metadata key and client retries within their leases.

The coordinator does not own Redis Stream message processing. It owns group metadata, member epochs, target assignment, current assignment reports, resharding metadata, and producer routing metadata.

Design Goals

• Short coordinator outages must not immediately stop healthy consumers or producers.
• Long coordinator outages must fail closed before split ownership or indefinitely stale producer routing can happen.
• A new coordinator pod must continue from Redis-recorded metadata, not from the old pod's memory.
• Heartbeat responses must be repeatable views of the latest Redis-recorded assignment state, not one-shot commands that are lost when a coordinator dies.
• Redis metadata can lose recent writes. If clients report a higher metadata version than Redis currently stores, the coordinator must drive a metadata sync round back to the current Redis version.
• Unsupported Redis commands must fail fast or use a documented fallback.

Core Model

Redis-Recorded State Machine

Every coordinator workflow is a Redis-recorded state machine. A response is derived only from the current group metadata key.

request or tick
  -> acquire Redis mutex
  -> load group metadata hash
  -> validate input
  -> compute next state
  -> write metadata with storeRevision CAS
  -> return response derived from the written metadata

The response must not ask a client to do something that was not first written to Redis metadata. If the coordinator dies before the write, the client retry sees the old state and retries safely. If the coordinator dies after the write but before the response reaches the client, the next heartbeat or routing refresh returns the same recorded decision while Redis still has that version.

Metadata Version Correction

A metadata version returned to a client may later disappear from Redis. For example, the coordinator can write metadata version 10, return it in a heartbeat response, and then Redis can fail over or restore to version 9. In that case the client has observed a newer state than Redis currently stores.

Redis remains the source of truth. The higher client version is treated as evidence that the client may hold a stale local view, not as authority to rebuild Redis state:

• Coordinator responses include the current metadataVersion, assignmentEpoch, and relevant member epoch.
• Consumers send their current local metadataVersion on every heartbeat.
• If request.metadataVersion > group.metadataVersion, the coordinator starts a metadata correction round and responds with SYNC_METADATA.
• SYNC_METADATA carries the current Redis metadataVersion, assignmentEpoch, keepable assigned shards, and pending target shards.
• The consumer applies the response as the current metadata truth, keeps only shards it was already reading and that still appear in assignedShards, revokes the rest, and retries heartbeat with the corrected version.
• The coordinator sends SYNC_METADATA to every known consumer whose heartbeat version does not match the current Redis metadata version while the correction round is active.
• If the consumer is now on the current metadata version but any local or remote revoke is still draining, the coordinator responds with REVOKE_PENDING.
• REVOKE_PENDING is also idempotent. The consumer keeps draining and does not start newly assigned shards until a later OK.
• OK is the only status that lets the consumer start newly assigned shards.
• The coordinator does not rebuild Redis metadata from consumer reports and does not auto-increment the rolled-back version just to match a client.

Correction sequence:

higher-version heartbeat
  -> SYNC_METADATA repeated until the member heartbeats with the Redis metadata version
  -> REVOKE_PENDING while local revoke/drain or another member's revoke blocks target shards
  -> OK after all live members are synchronized and conflicting previous owners are released, expired, or fenced

Client Leases

Consumers and producers keep operating only within bounded local leases:

The consumer assignment lease must be shorter than or equal to the coordinator member lease TTL. This prevents a consumer from reading after the coordinator has expired it and reassigned its shards.

Idempotent Retry

All coordinator interactions must tolerate retries:

• Heartbeat is idempotent for the same memberId, memberEpoch, assignmentEpoch, owned shards, and revoking shards.
• Revoke acknowledgements are idempotent. Duplicate REVOKED reports for a released shard are ignored.
• Producer routing refresh is idempotent by metadataVersion.
• Admin mutations use store revision checks and must not silently overwrite newer metadata.

Generic Failure Checkpoints

Every workflow should be analyzed at these checkpoints.

This checkpoint model is the main tool for handling new edge cases. New features must define their Redis-recorded handoff point, client retry behavior, lease expiry behavior, and repair path.

Before-write failure must always name the write boundary. "Before metadata write" is not one state. It means the coordinator failed before recording the next state transition for that workflow.

Rebalance Drain Failure Scenario

This is the critical rebalance flow:

member A owns shard S
member C joins or capacity changes
coordinator computes target owner B/C for shard S
coordinator asks member A to drain shard S through heartbeat response
member A stops new reads, drains in-flight work, then reports REVOKED
coordinator assigns shard S to the new target only after release is accepted

The DRAIN instruction is not an ephemeral message. It is a response derived from Redis-recorded metadata:

• target assignment says shard S should no longer belong to member A;
• current assignment still records member A as the current or revoking owner;
• member B or C receives shard S as pending until release is accepted;
• assignment epoch identifies the decision.

Sequence

sequenceDiagram
    autonumber
    participant A as Consumer A
    participant C as Coordinator
    participant R as Redis Metadata
    participant B as Consumer B

    A->>C: heartbeat owned=[S]
    C->>R: write target owner B, A must revoke S
    C-->>A: heartbeat response assigned=[], pending=[], revoke/drain S
    A->>A: stop new reads for S, drain in-flight work
    B->>C: heartbeat
    C-->>B: pending=[S], do not read yet
    A->>C: heartbeat revoking=[S:REVOKED]
    C->>R: write S released from A
    C-->>B: next heartbeat assigned=[S]
    B->>B: start reads for S

Coordinator Down After DRAIN Response

If the coordinator goes down after member A receives the DRAIN instruction:

Failure subcases:

Recorded state by write boundary:

Consumer rule: once a shard enters local revoking state, the consumer must not resume reads for that shard just because heartbeat is temporarily failing. Only a later coordinator assignment can make the shard readable again.

Coordinator rule: a shard must not move from pending to assigned for the new owner until one of these is true:

• previous owner reported REVOKED with no in-flight work;
• previous owner expired by member lease TTL;
• previous owner exceeded rebalance timeout and was fenced.

Coordinator Rolling Update and Temporary Coordinator Loss

This section covers replicas=1 with rolling update temporarily producing:

1 coordinator -> 2 coordinators -> 1 coordinator

It also covers temporary coordinator loss:

1 coordinator -> 0 reachable coordinators -> 1 coordinator

The coordinator cannot control Kubernetes readiness, service endpoint propagation, or external load balancers. It only owns process-local terminating state and Redis critical sections.

Implementation guardrails:

• CoordinatorShutdownGate enters terminating mode on Spring context shutdown.
• @CriticalSection service methods acquire the local monitor, then pass the shutdown gate, then acquire the Redis mutex.
• New heartbeat/admin mutations after terminating mode receive 503 COORDINATOR_TERMINATING and should retry against another coordinator instance.
• Already-entered short critical sections are allowed to finish so their Redis write and mutex release can complete.
• Event-loop ticks stop scheduling state transitions after terminating mode starts.

The new coordinator does not resume old stack frames. It resumes from Redis-recorded metadata:

Consumer Join, Rejoin, Leave, and Expiration

Producer Routing During Failures

Producer does not heartbeat. It uses routing metadata refresh.

Because producer has no heartbeat channel, shard-count changes are propagated only through routing metadata refresh. The producer module must therefore refresh routing metadata periodically even if publishes are succeeding. The refresh interval and routing cache TTL define the maximum time a producer can keep using an older shard count after the coordinator adds shards.

Refresh rules:

• Refresh periodically before the local routing cache expires.
• Refresh immediately when a publish path detects stale routing or a metadata-version mismatch.
• Replace the cache only when the refreshed metadataVersion is newer.
• Keep the current cache when refresh fails and the cache lease is still valid.
• Fail publish when refresh fails after the cache lease has expired.

Redis Metadata Volatility and Corruption

Coordinator metadata is stored in one Redis hash key per group:

redis-stream:coord:{streamPrefix:consumerGroup}:metadata

Recommended hash fields:

aggregate      -> GroupMetadata JSON
revision       -> storeRevision
schemaVersion  -> metadata JSON schema version
layoutVersion  -> Redis metadata layout version
updatedAt      -> last successful metadata write time

Required behavior:

Operational controls:

• Metadata keys must not have TTL.
• Redis should enable snapshots/AOF or managed backups. These reduce recovery loss; they do not remove the recent-write-loss assumption.
• Coordinator ACL users should not have FLUSHDB, FLUSHALL, or broad key deletion permissions.
• Monitoring must alert when consumers report higher metadata versions than the current Redis metadata.

Indexes are rebuildable only:

redis-stream:coord:groups

Additional index behavior:

Unsupported Redis Version or Command Set

Design Checklist for New Edge Cases

Every new coordinator feature must answer these questions before implementation:

1. What is the Redis-recorded state machine state?
2. Which metadata fields are written before the response?
3. Is the response repeatable if it is lost?
4. What does the client do while coordinator is unavailable?
5. What local lease or cache bounds client behavior?
6. What happens after the lease expires?
7. Which epoch, revision, or version rejects stale reports?
8. Can a new coordinator rebuild state from Redis metadata plus client retries?
9. What happens if the metadata key is missing?
10. What happens if a consumer reports a higher metadataVersion than Redis currently stores?
11. Which client request carries the highest version the client has observed?
12. Which metrics and runbook steps reveal this failure?

Test Matrix

The implementation should include scenario tests at three levels.

Priority tests:

• DRAIN response state written to Redis, response lost, next heartbeat returns same DRAIN.
• DRAIN delivered, coordinator unavailable, consumer keeps shard revoking and does not resume reads.
• REVOKED ack written to Redis, response lost, new owner receives assignment on next heartbeat.
• Previous owner expires while draining, new owner receives assignment after expiration.
• Metadata key deleted, coordinator fails closed and does not reconstruct from heartbeat.
• Client receives metadata version N, Redis falls back to N-1, and the next heartbeat receives SYNC_METADATA for N-1.
• SYNC_METADATA response is lost, the next high-version heartbeat receives the same SYNC_METADATA again.
• Consumer receives SYNC_METADATA, starts draining removed shards, and receives REVOKE_PENDING until it reports REVOKED and no other owner blocks its target shards.
• During a metadata correction round, multiple consumers report stale revokes concurrently and the coordinator accepts only revokes that match current Redis target/current ownership.
• Producer cache expires during coordinator outage, publish fails closed.
• New coordinator resumes rebalance from Redis-recorded metadata after old coordinator crash.

Monitoring and Runbook Requirements

Required signals:

• coordinator terminating state,
• Redis mutex latency and revision conflict count,
• heartbeat success/failure by status,
• member heartbeat age,
• metadata version regression count,
• client-reported max metadata version higher than stored coordinator metadata,
• consumer assignment lease remaining time,
• member revoking shard count,
• producer routing cache age and refresh failures,
• metadata revision conflict count,
• metadata schema/layout error count,
• Redis command compatibility failure count.

Runbooks should cover:

• DRAIN issued but revoke ack not progressing,
• coordinator unreachable during heartbeat,
• rolling update handoff stall,
• member lease expiration and rejoin,
• Redis metadata key loss or corruption,
• unsupported Redis command configuration,
• producer routing cache expiry during coordinator outage.