Why Rendezvous Hashing Fits Distributed Cron Better Than a Hash Ring

Abstract

Running a cron process is easy. Replicating it for availability is where the distributed-systems problem begins: every replica knows the same schedule, so every replica wakes up for the same tick. The cluster therefore needs a deterministic answer to a small but important question: given a job and the current live scheduler nodes, which node owns that job?

A ring-based consistent-hashing implementation can provide stable ownership, but it also introduces a topology structure, virtual-node tuning, and successor-oriented failover behavior that are usually unnecessary in a small scheduler control plane. Rendezvous hashing—also called Highest Random Weight, or HRW—expresses ownership directly. It computes one deterministic score for each job-node pair and chooses the node with the highest score.

This article explains why HRW is often the better engineering fit for distributed cron, while being explicit about its limits. The implementation uses TypeScript and Effect to combine Redis-backed membership, HRW ownership, scoped schedule loops, a per-tick lock and claim marker, and idempotent queue dispatch. The central principle is that HRW reduces coordination; it does not create end-to-end exactly-once execution by itself.

Thesis: For a scheduler cluster with a handful of replicas, HRW’s O(N) lookup cost is usually negligible. In return, the design avoids a ring and virtual nodes, provides a natural ordered failover candidate for every job, and keeps every scheduler replica behaviorally symmetric.

The Distributed Cron Problem

Suppose a service contains three identical scheduler replicas. Each replica registers the same cron expression for a job named check-orders. At 12:00, all three processes calculate that the job is due. Without an ownership mechanism, they all enqueue the same logical trigger.

The scheduling problem is therefore not merely “when should this cron expression fire?” It is also:

Ownership question: Given job J and a set of live scheduler nodes N, which single node should take responsibility for J?

A useful ownership function for distributed cron should satisfy several requirements. Some are algorithmic, while others are operational safeguards that no hashing scheme can provide alone.

Leader election, database row locks, Redis locks, a hash ring, and HRW can all participate in a solution. The interesting comparison is between the two deterministic mapping approaches: a ring-based topology and direct job-node scoring.

A Hash Ring Solves a Different-Shaped Problem

Ring-based consistent hashing maps both nodes and keys into a circular hash space. A key belongs to the first node encountered while moving clockwise from the key’s position. When a node joins or leaves, only a neighboring portion of the ring changes ownership, rather than nearly every key moving as it would with hash(key) modulo nodeCount.

The hash ring is a strong design for distributed caches, storage shards, and request-routing paths. In those systems, the ring is a precomputed topology that makes repeated lookups cheap. A typical implementation stores V virtual positions for each of N physical nodes, sorts the resulting V×N entries, and performs a binary search for each key.

Why virtual nodes appear

With one random ring position per physical node, the arcs between nodes are rarely equal. One server may inherit a much larger range than another. Production systems commonly add many virtual nodes per physical node to smooth the distribution and to spread the ranges of a failed physical node across several successors.

Virtual nodes are not inherently bad. They are an effective answer to the balance and failover behavior of a ring. They do, however, introduce additional state and tuning: how many virtual nodes should exist, how are weights represented, how is the ring rebuilt, and how is deterministic ordering maintained across implementations?

Why this is often unnecessary for cron

A cron scheduler does not place gigabytes of data and does not route millions of requests per second. It assigns responsibility for a comparatively small set of job definitions to a comparatively small set of scheduler replicas. There is no data migration when ownership changes; a different process simply becomes responsible for emitting the next trigger.

That changes the engineering trade-off. The ring’s O(log(VN)) lookup is attractive when lookups dominate and N is large. In a control-plane scheduler with three, five, or ten replicas, the topology-free simplicity of HRW is usually worth more than saving a few hash evaluations.

Rendezvous Hashing Expresses Ownership Directly

Rendezvous hashing was introduced as a name-based mapping scheme and is also known as Highest Random Weight hashing. For a job key j and a live-node set S, it computes a deterministic score for every candidate node and chooses the maximum:

Definition: owner(j, S) = arg max over n in S of H(j, n)

Every replica that sees the same set of live nodes calculates the same winner. No ring positions, successor search, or virtual-node table are required. The algorithm also creates an implicit ranking: the highest score is the owner, the second-highest score is the first failover candidate, and so on.

A minimal TypeScript implementation

The hash function must be deterministic across processes, architectures, and deployments. The following example uses 64-bit FNV-1a for clarity. It is not a cryptographic hash; its purpose is stable distribution. A production implementation may choose another stable, well-tested 64-bit hash.

const encoder = new TextEncoder()

const FNV_OFFSET_BASIS_64 = 0xcbf29ce484222325n
const FNV_PRIME_64 = 0x100000001b3n
const UINT64_MASK = 0xffffffffffffffffn

export const fnv1a64 = (input: string): bigint => {
  let hash = FNV_OFFSET_BASIS_64

  for (const byte of encoder.encode(input)) {
    hash ^= BigInt(byte)
    hash = (hash * FNV_PRIME_64) & UINT64_MASK
  }

  return hash
}

export const hrwScore = (
  jobName: string,
  nodeId: string,
): bigint => fnv1a64(`${jobName}\u0000${nodeId}`)

export const selectHrwOwner = (
  jobName: string,
  nodeIds: ReadonlyArray<string>,
): string | undefined => {
  let selectedNodeId: string | undefined
  let selectedScore = -1n

  for (const nodeId of nodeIds) {
    const score = hrwScore(jobName, nodeId)
    const winsTie =
      score === selectedScore &&
      (selectedNodeId === undefined || nodeId < selectedNodeId)

    if (score > selectedScore || winsTie) {
      selectedNodeId = nodeId
      selectedScore = score
    }
  }

  return selectedNodeId
}

The explicit lexical tie-break is important. Hash collisions should be rare with a 64-bit score, but a deterministic system must still define what happens when two scores are equal. The input separator also prevents ambiguous concatenations such as ab+c and a+bc from becoming the same byte sequence.

Why membership changes cause minimal disruption

Assume node a currently wins job j. Removing an unrelated node b does not change any remaining score, so a still wins. Removing a exposes the next-highest score. Adding a new node x affects only the jobs for which H(j, x) exceeds the previous maximum. This is the same minimal-remapping goal that motivates consistent hashing, achieved without maintaining a ring.

This point deserves precision: minimal remapping is not an exclusive advantage of HRW. Both ring-based consistent hashing and HRW provide it. HRW’s advantage in this workload is the route by which it obtains the property: direct comparison over a small candidate set, rather than a maintained topology with virtual positions.

Why HRW Fits Scheduled Tasks Particularly Well

Scheduler clusters are normally small

Classic HRW performs O(N) score calculations for each ownership lookup. That is a poor trade in some data-plane paths with thousands of nodes and millions of lookups. It is usually an excellent trade for a scheduler control plane. Even 100 schedules evaluated across 10 replicas require only 1,000 simple score calculations for an ownership pass.

The relevant optimization target is not the theoretical lookup bound in isolation. It is the total complexity of the system: data structures, configuration, failure behavior, test surface, and the number of invariants operators must understand.

Failure redistribution is naturally spread

When a node disappears, each job it owned moves to that job’s own second-highest candidate. Different jobs generally have different rankings, so the failed node’s responsibilities spread across the surviving cluster. A basic single-token ring instead transfers a contiguous failed range to the next clockwise successor. Virtual nodes improve that ring behavior, but HRW receives the distribution directly from independent job-node scores.

Every replica remains symmetric

There is no leader process and no standby mode. Every replica registers the same schedules, calculates the same due time, reads the live membership set, and computes the same ownership function. A replica either continues to the claim boundary or stops because another node owns the job. Deployment and recovery are therefore ordinary horizontal scaling operations rather than leadership transitions.

The comparison in one table

Neither algorithm guarantees a perfectly even split when there are only a few jobs. If three nodes share seven schedules, random variation may still produce a 4–2–1 allocation. When individual schedules have very different costs, capacity-aware filtering, correct weighted HRW, or explicit placement constraints may be more important than the base hashing choice.

Keep Three Identities Separate

A clean implementation distinguishes schedule ownership from a particular execution. Conflating these identities creates unnecessary churn or weakens deduplication.

The HRW key should normally be the schedule name, not schedule name plus due time. Including the timestamp would reshuffle ownership on every tick, eliminating stable affinity and making observability noisier. The timestamp belongs in the tick identity because two occurrences of the same schedule must never share a lock or deduplication key.

Architecture: Membership, Ownership, and Safety

A robust distributed scheduler separates three layers that are often incorrectly treated as one feature:

1. Membership answers which scheduler nodes are currently eligible.
2. HRW ownership answers which eligible node should take responsibility for a schedule.
3. The per-tick safety boundary handles temporarily inconsistent membership views and retries.

Membership through expiring heartbeats

Redis sorted sets provide a simple membership registry. Each scheduler periodically writes its stable node ID with the current timestamp as the score. Reading live members removes entries older than a configured TTL and returns the remaining IDs. For example, a five-second heartbeat and a fifteen-second live TTL tolerate brief delays while bounding failover time.

Membership is still eventually consistent. Two schedulers can read the registry at different instants and calculate different live sets. HRW guarantees agreement only when its inputs agree. That is why ownership must be followed by a final claim boundary.

A per-tick lock is a safety net, not the allocator

The lock key should contain the tick identity. If two replicas temporarily believe they own check-orders, only one should enter the critical section for check-orders:dueAt. The lock implementation must use a unique token and compare-and-delete release semantics; a plain SET NX followed by an unconditional DEL can delete another holder’s lock after a TTL rollover.

The lock should cover only the short trigger publication, not the duration of the actual business job. Long-running work belongs in a worker queue. Keeping the scheduler as a trigger process reduces lock TTL requirements, limits failure ambiguity, and prevents one slow job from blocking schedule evaluation.

Claim markers let non-owners stop retrying

A claim marker keyed by job name and due time records that the logical tick has already been dispatched. Non-owners can observe the marker and stop retrying. The marker TTL must outlive the schedule interval plus the allowed lateness window; otherwise an old tick can become eligible again while replicas are still recovering.

Critical limitation: HRW is not an exactly-once protocol. A lock narrows concurrency, but a process can still publish a message and crash before writing its claim marker. End-to-end correctness requires an idempotent tick identity, broker deduplication, an idempotent consumer, or a transactional outbox.

Implementing the Design with TypeScript and Effect

The following code is a production-oriented skeleton rather than a complete Redis client. It focuses on the boundaries that matter: pure deterministic ownership, typed service dependencies, interruptible loops, scoped fibers, and bounded retry. The Redis and queue adapters are represented as Effect services.

Domain model and service contracts

import {
  Clock,
  Context,
  Cron,
  Effect,
  Fiber,
  Result,
} from "effect"

export class RegistryError extends Error {
  readonly _tag = "RegistryError"
}

export class TickStoreError extends Error {
  readonly _tag = "TickStoreError"
}

export class CronActionError extends Error {
  readonly _tag = "CronActionError"
}

export class SchedulerError extends Error {
  readonly _tag = "SchedulerError"
}

export interface TickIdentity {
  readonly jobName: string
  readonly dueAtMs: number
  readonly key: string
}

export interface CronJob {
  readonly name: string
  readonly expression: string
  readonly timeZone?: string
  readonly onDue: (
    tick: TickIdentity,
  ) => Effect.Effect<void, CronActionError>
}

export interface MembershipRegistryShape {
  readonly nodeId: string
  readonly keyPrefix: string
  readonly register: Effect.Effect<void, RegistryError>
  readonly unregister: Effect.Effect<void, RegistryError>
  readonly liveNodeIds: Effect.Effect<
    ReadonlyArray<string>,
    RegistryError
  >
  readonly heartbeatLoop: Effect.Effect<never, RegistryError>
}

export class MembershipRegistry extends Context.Service<
  MembershipRegistry,
  MembershipRegistryShape
>()("@app/MembershipRegistry") {}

The service exposes effects rather than promises. Errors remain in the Effect error channel, and the scheduler process can decide whether to retry, fail closed, or terminate. The heartbeat loop is modeled as Effect<never, RegistryError>: under normal operation it runs forever, but a terminal registry failure can bring down the scheduler scope rather than letting it execute with stale membership.

Redis-backed membership

Assume a Redis service exposing zAdd, zRemRangeByScore, zRangeByScore, and zRem. A sorted-set registry can be implemented as follows:

export interface RedisMembershipOptions {
  readonly nodeId: string
  readonly keyPrefix: string
  readonly heartbeatIntervalMs: number
  readonly liveTtlMs: number
}

export const makeRedisMembershipRegistry = (
  options: RedisMembershipOptions,
) =>
  Effect.gen(function* () {
    const redis = yield* Redis
    const registryKey = `${options.keyPrefix}:members`

    const heartbeat = Effect.gen(function* () {
      const nowMs = yield* Clock.currentTimeMillis
      yield* redis.zAdd(registryKey, nowMs, options.nodeId)
    })

    const liveNodeIds = Effect.gen(function* () {
      const nowMs = yield* Clock.currentTimeMillis
      const cutoffMs = nowMs - options.liveTtlMs

      yield* redis.zRemRangeByScore(
        registryKey,
        "-inf",
        cutoffMs - 1,
      )

      return yield* redis.zRangeByScore(
        registryKey,
        cutoffMs,
        "+inf",
      )
    })

    const heartbeatLoop: Effect.Effect<never, RegistryError> =
      Effect.gen(function* () {
        yield* heartbeat

        while (true) {
          yield* Effect.sleep(options.heartbeatIntervalMs)
          yield* heartbeat
        }
      })

    return {
      nodeId: options.nodeId,
      keyPrefix: options.keyPrefix,
      register: heartbeat,
      unregister: redis
        .zRem(registryKey, options.nodeId)
        .pipe(Effect.asVoid),
      liveNodeIds,
      heartbeatLoop,
    } satisfies MembershipRegistryShape
  })

A real implementation should retry transient heartbeat failures with bounded exponential backoff and emit metrics for heartbeat age, live-member count, and registry errors. If the registry remains unavailable, failing closed is safer than allowing every replica to assume it is the only live node.

Modeling the tick store

The tick store owns the lock and claim-marker primitives. Keeping this contract separate makes the scheduler algorithm testable with an in-memory implementation.

export type LockResult<A> =
  | { readonly acquired: false }
  | { readonly acquired: true; readonly value: A }

export interface TickStoreShape {
  readonly exists: (
    key: string,
  ) => Effect.Effect<boolean, TickStoreError>

  readonly set: (
    key: string,
    value: string,
    ttlMs: number,
  ) => Effect.Effect<void, TickStoreError>

  readonly tryWithLock: <A, E, R>(
    key: string,
    ttlMs: number,
    body: Effect.Effect<A, E, R>,
  ) => Effect.Effect<
    LockResult<A>,
    E | TickStoreError,
    R
  >
}

export class TickStore extends Context.Service<
  TickStore,
  TickStoreShape
>()("@app/TickStore") {}

Evaluating one due tick

The evaluator checks the terminal marker, computes the current HRW owner, and lets only the owner attempt the lock. The marker is checked again inside the lock because another replica may have completed the tick between the first read and lock acquisition.

export type TickStatus =
  | "started"
  | "already-claimed"
  | "not-owner"
  | "no-live-nodes"
  | "lock-busy"
  | "abandoned"

export interface TickResult {
  readonly status: TickStatus
  readonly ownerNodeId?: string
}

export interface TickPolicy {
  readonly lockTtlMs: number
  readonly claimTtlMs: number
}

const tickIdentity = (
  jobName: string,
  dueAtMs: number,
): TickIdentity => ({
  jobName,
  dueAtMs,
  key: `${jobName}:${dueAtMs}`,
})

export const makeRunTick = (policy: TickPolicy) =>
  Effect.fn("CronScheduler.runTick")(function* (
    job: CronJob,
    dueAtMs: number,
  ) {
    const registry = yield* MembershipRegistry
    const store = yield* TickStore
    const tick = tickIdentity(job.name, dueAtMs)
    const claimKey = `${registry.keyPrefix}:claimed:${tick.key}`
    const lockKey = `${registry.keyPrefix}:lock:${tick.key}`

    if (yield* store.exists(claimKey)) {
      return { status: "already-claimed" } satisfies TickResult
    }

    const liveNodeIds = yield* registry.liveNodeIds
    const ownerNodeId = selectHrwOwner(job.name, liveNodeIds)

    if (ownerNodeId === undefined) {
      return { status: "no-live-nodes" } satisfies TickResult
    }

    if (ownerNodeId !== registry.nodeId) {
      return {
        status: "not-owner",
        ownerNodeId,
      } satisfies TickResult
    }

    const guarded = yield* store.tryWithLock(
      lockKey,
      policy.lockTtlMs,
      Effect.gen(function* () {
        if (yield* store.exists(claimKey)) {
          return {
            status: "already-claimed",
            ownerNodeId,
          } satisfies TickResult
        }

        yield* job.onDue(tick)
        yield* store.set(
          claimKey,
          registry.nodeId,
          policy.claimTtlMs,
        )

        return {
          status: "started",
          ownerNodeId,
        } satisfies TickResult
      }),
    )

    return guarded.acquired
      ? guarded.value
      : ({ status: "lock-busy", ownerNodeId } satisfies TickResult)
  })

The code deliberately sends the stable tick identity to onDue. A queue adapter should use tick.key as a message ID or deduplication key. If publication succeeds but the marker write fails, a retry may call onDue again; the repeated call must be harmless at the queue or consumer boundary.

Bounded retry enables failover

A non-owner cannot immediately take over from a failed owner because the failed node may still be present in the live set. The replica retries until one of three terminal events occurs: the claim marker appears, membership changes and the local node becomes owner, or the maximum lateness deadline is exceeded.

export interface RetryPolicy {
  readonly maxTickLatenessMs: number
  readonly retryIntervalMs: number
}

export const makeRunDueTick = (
  tickPolicy: TickPolicy,
  retryPolicy: RetryPolicy,
) => {
  const runTick = makeRunTick(tickPolicy)

  return Effect.fn("CronScheduler.runDueTick")(function* (
    job: CronJob,
    dueAtMs: number,
  ) {
    const deadlineMs =
      dueAtMs + retryPolicy.maxTickLatenessMs

    while ((yield* Clock.currentTimeMillis) <= deadlineMs) {
      const attempt = yield* runTick(job, dueAtMs).pipe(
        Effect.result,
      )

      if (Result.isSuccess(attempt)) {
        const result = attempt.success

        if (
          result.status === "started" ||
          result.status === "already-claimed"
        ) {
          return result
        }
      } else {
        yield* Effect.logWarning(
          `Cron tick attempt failed: ${job.name}:${dueAtMs}`,
        )
      }

      yield* Effect.sleep(retryPolicy.retryIntervalMs)
    }

    return { status: "abandoned" } satisfies TickResult
  })
}

The lateness window should be longer than the membership TTL plus expected Redis and broker recovery time, but short enough that an obsolete tick cannot surprise the business system much later. Different jobs may require different policies: a billing reconciliation can tolerate a longer delay than a minute-by-minute cache refresh.

Running schedule loops with scoped fibers

Effect’s structured concurrency is useful for process lifecycle management. Each schedule loop and the membership heartbeat run in scoped fibers. Interrupting the outer scope interrupts the child loops, then unregisters the node.

const parseJobCron = (
  job: CronJob,
): Effect.Effect<Cron.Cron, SchedulerError> => {
  const parsed = Cron.parse(job.expression, job.timeZone)

  if (Result.isFailure(parsed)) {
    return Effect.fail(
      new SchedulerError(parsed.failure.message),
    )
  }

  return Effect.succeed(parsed.success)
}

export const makeScheduleLoop = (
  tickPolicy: TickPolicy,
  retryPolicy: RetryPolicy,
) => {
  const runDueTick = makeRunDueTick(
    tickPolicy,
    retryPolicy,
  )

  return (job: CronJob): Effect.Effect<never, SchedulerError> =>
    Effect.gen(function* () {
      const cron = yield* parseJobCron(job)
      let dueAt = Cron.next(
        cron,
        new Date(yield* Clock.currentTimeMillis),
      )

      while (true) {
        const nowMs = yield* Clock.currentTimeMillis
        yield* Effect.sleep(
          Math.max(0, dueAt.getTime() - nowMs),
        )

        const dueAtMs = dueAt.getTime()
        yield* runDueTick(job, dueAtMs)

        dueAt = Cron.next(
          cron,
          new Date(dueAtMs + 1),
        )
      }
    })
}

export const startScheduler = (
  jobs: ReadonlyArray<CronJob>,
  tickPolicy: TickPolicy,
  retryPolicy: RetryPolicy,
) =>
  Effect.scoped(
    Effect.gen(function* () {
      const registry = yield* MembershipRegistry
      const runScheduleLoop = makeScheduleLoop(
        tickPolicy,
        retryPolicy,
      )

      yield* registry.register
      const heartbeatFiber = yield* registry.heartbeatLoop.pipe(
        Effect.forkScoped,
      )

      for (const job of jobs) {
        yield* runScheduleLoop(job).pipe(Effect.forkScoped)
      }

      return yield* Fiber.join(heartbeatFiber)
    }).pipe(
      Effect.ensuring(
        Effect.gen(function* () {
          const registry = yield* MembershipRegistry
          yield* registry.unregister.pipe(
            Effect.catchAll(() => Effect.void),
          )
        }),
      ),
    ),
  )

Joining the heartbeat fiber is intentional. If membership maintenance fails terminally, the scheduler scope fails and all schedule fibers are interrupted. Continuing to emit ticks without a trustworthy membership view would convert a registry outage into duplicate execution.

Keep the trigger idempotent and the work asynchronous

A schedule definition should normally enqueue a small trigger rather than execute the business operation in the scheduler process:

const checkOrders: CronJob = {
  name: "check-orders",
  expression: "* * * * *",
  timeZone: "UTC",
  onDue: (tick) =>
    JobQueue.enqueue({
      queue: "check-orders",
      messageId: tick.key,
      payload: {
        triggeredAt: tick.dueAtMs,
      },
    }),
}

The queue and worker boundary is where delivery semantics should be made explicit. Depending on the broker, messageId may drive native deduplication, an inbox table, or an idempotency record in the worker’s database. The scheduler’s responsibility is to produce a stable tick identity and avoid unnecessary concurrent publication.

Failure Semantics and Operational Controls

Distributed cron correctness is mostly determined by failure windows rather than the happy-path owner calculation. The following table summarizes the important cases.

Exactly-once is a system property, not a hashing property

There is an unavoidable ordering choice when the claim store and message broker are different systems. Publishing before marking can duplicate after a crash. Marking before publishing can lose a tick after a crash. A lock reduces simultaneous execution but cannot atomically commit across independent services.

Systems that require stronger guarantees should use one of three patterns: a transactional outbox in the same database as the claim, a broker with reliable deduplication keyed by the tick identity, or an idempotent consumer that records processed tick IDs in the same transaction as its business update. In many operational workloads, at-least-once dispatch plus idempotent handling is simpler and safer than pretending the scheduler can provide exactly-once by itself.

Observability should describe decisions, not just executions

Useful telemetry includes live scheduler count, heartbeat age, selected owner, tick lateness, retry count, lock contention, claim status, abandoned ticks, enqueue latency, and ownership changes. A trace for the scheduler should end after the trigger is claimed or abandoned; the worker execution belongs to a separate trace linked through the tick or message ID.

• cron_tick_total{status="started|already_claimed|not_owner|abandoned"}
• cron_tick_lateness_ms and cron_tick_attempts
• scheduler_live_members and scheduler_heartbeat_age_ms
• cron_lock_contention_total and cron_claim_write_failures_total
• cron_owner_changes_total{job="..."}

When a Hash Ring Is the Better Choice

HRW is not universally superior. The ring becomes more attractive when ownership lookup is a high-volume data-plane operation, the candidate set contains hundreds or thousands of nodes, the topology is reused for an enormous number of keys, or a mature ring implementation already provides bounded-load and weighted-placement behavior required by the system.

The decision can be summarized as follows:

Other algorithms may also be better for specific scales and constraints: Jump Consistent Hash for very compact bucket mapping, Maglev for fast network load-balancer lookup tables, or bounded-load rendezvous variants for tighter capacity limits. The correct question is not “which hashing algorithm wins in general?” but “which ownership model matches this workload and its failure assumptions?”

Production Checklist

Conclusion

Ring-based consistent hashing and rendezvous hashing share the most important stability property: changing one node does not reshuffle every job. The reason to prefer HRW for distributed cron is therefore not that rings are incorrect, nor that HRW owns minimal remapping exclusively. It is that HRW more directly matches the shape of the problem.

A scheduled job is a stable responsibility, not a block of data that must move around a topology. Scheduler clusters are usually small. Each job naturally has an ordered list of candidate owners. Removing a node simply exposes the next score. No ring, virtual nodes, or topology rebuild is necessary.

The complete design is still larger than the hash function: expiring membership tells the replicas who is eligible, HRW picks the preferred owner, a per-tick lock protects against divergent views, a claim marker terminates retries, and an idempotent queue or consumer closes the crash window. With those boundaries kept explicit, HRW provides a compact and understandable foundation for highly available distributed cron.