Module refreshing_connection 

A ConnectionLike wrapper that gives Redis connections a bounded lifetime, so they are periodically dropped and reopened.

Why this exists

A long-lived Redis connection resolves the endpoint’s DNS once, at connect time, and then keeps talking to whatever address it first resolved. If the DNS behind the endpoint later changes — due to failover, scaling, node replacement, blue/green cutover, or maintenance — the connection stays pinned to the OLD, now-stale node instead of following the endpoint to its new target (e.g. after an ElastiCache failover repoints the endpoint to a new node). A stale connection pinned to a demoted node keeps failing writes as one symptom, but the underlying problem is the stale DNS resolution, not any particular error. ConnectionManager only re-resolves DNS when it reconnects, and it reconnects on a broken socket — NOT on a valid error reply over a healthy socket, so it never notices the endpoint moved.

RefreshingConnection fixes this by giving each connection a bounded lifetime: once a connection exceeds a jittered max age it is dropped and reopened before the next command, and a fresh connect re-resolves the endpoint’s DNS onto whatever node it currently points to. Recovery from any DNS change is therefore bounded by the configured max age.

Reactive rebuild (accelerator)

Age-based recycling is the guaranteed backstop, but recovery is only as fast as the configured max age. On top of it, connections are ALSO rebuilt reactively when a command reply indicates the connection is pinned to a read-only node — the tell-tale symptom of talking to a stale/demoted node after an endpoint change. When such a reply is observed the wrapper rebuilds the connection immediately (rate-limited by [MIN_RECONNECT_GAP]), so the very next command re-resolves DNS onto the current node instead of waiting out the age window. This detection intentionally covers BOTH shapes Redis uses for a write-on-replica failure: a top-level Err(ReadOnly) (direct write commands) AND an inline Value::ServerError embedded in an otherwise Ok(..) reply (apalis runs queue ops as Lua EVALSHA scripts, and the read-only failure is returned inside the script reply, not as a top-level error). Reactive rebuild is an accelerator only; it does not replace the age-based backstop, and it is not keyed on any specific error string beyond the read-only marker.

Shared healing state

RedisStorage is cloned into every worker AND cloned per enqueue on the producer path. All clones of a given queue’s connection therefore share ONE healing connection and ONE age clock via Arc<Mutex<ConnState>>. A rebuild performed by any clone is immediately visible to all other clones. This is what makes the fix correct in steady state: without shared state, the template living in the Queue struct would have a created_at fixed at setup that never advances, so every producer clone would be born already-expired once uptime exceeds max_age, opening a brand-new connection per enqueue forever.

Locking discipline

The std::sync::Mutex guard is NEVER held across an .await; it guards only tiny synchronous critical sections (read/clone the current conn, swap in a rebuilt conn). The actual command is delegated with no lock held, and the inner ConnectionManager is a multiplexed connection, so concurrent commands from different clones stay concurrent. Cloning C is cheap and shares the underlying multiplexed pipe.

Structs

RefreshingConnection

A ConnectionLike wrapper that periodically rebuilds its inner connection (bounded lifetime), sharing healing state across all clones.