Skip to content
POLARISAI ACADEMY
R03-301 · Navigator

Resilience, Queues, CI/CD and Safe Releases

You can charge one idempotency key a single time, add the half-draw backoff sleeps through a successful try, decide whether a circuit breaker sends the next call, and ack or nack the in-flight job when SIGTERM arrives.

Level
Navigator
Time
24 h (estimate)
Prerequisites
R03-201
Course 3 of 4
Jump to

After this course you can

The student can build an idempotent worker queue with circuit breakers, exponential backoff, and dead-letter handling that processes 500 jobs with zero duplicates or data loss when downstream dependencies fail 40% of the time.

One key, one charge

A worker that moves money can see the same delivery twice. The queue hands the message over again when the first copy never got an ack. An idempotency key is the string stored with the outcome. Cash starts at 200, and the key table starts empty. A delivery names a key and an amount. When the key is absent, one commit subtracts the amount and inserts the row (key, amount, receipt). When the key is already there and the amount matches, the worker adds 0 and returns the stored receipt. When the key is already there and the amount differs, the worker adds 0, leaves the stored receipt where it is, and returns a refusal.

Walk four deliveries. K7 for 40 finds no row, so cash becomes 200 − 40 = 160 and the row is (K7, 40, R1). K7 for 40 finds that row, so cash stays 160 and the reply is R1. K8 for 25 finds no row, so cash becomes 160 − 25 = 135 and the row is (K8, 25, R2). K7 for 99 finds K7 stored at 40. 99 differs from 40, so cash stays 135, receipt R1 stays stored, and the reply is a refusal. Two deliveries applied, one duplicate matched, one mismatch was refused, and the money that moved is 40 + 25 = 65. A crash before that commit leaves cash at 200 with the key absent, so the redelivery applies once and cash becomes 160. A crash after the commit and before the ack leaves cash at 160 with row K7 in place, so the redelivery adds 0 and cash stays 160.

Backoff sleeps you can add

A failed try waits, then tries again. This lesson uses exponential backoff with a fixed jitter draw, so the sleeps add up on paper. Jitter here is a stated fraction of the ceiling, which keeps two workers from waking on the same millisecond. The ceiling after failed try k is the minimum of 800 milliseconds and 100 × 2^(k-1) milliseconds. The worked job draws one half, so the sleep is that ceiling divided by 2. For k from 1 through 5 the ceilings are 100, 200, 400, 800 and 800. The fifth uncapped value is 100 × 16 = 1600, and the cap of 800 cuts it to 800. The five half-draw sleeps are 50, 100, 200, 400 and 400 milliseconds. A second worker draws one quarter of the same ceilings and sleeps 25, 50, 100, 200 and 200. After the first failure those two waits are 50 and 25, so the next tries land 25 milliseconds apart.

The charge job fails try 1 and try 2, then succeeds on try 3. The sleeps that run are 50 + 100 = 150 milliseconds, and the try count is 3. Success stores the key and stops the ladder. A dead-letter queue, the DLQ, is the side queue for a body that has used every try. This lesson allows 6 tries. A poison message, a body that fails every try, sleeps through all five gaps: 50 + 100 + 200 + 400 + 400 = 1150 milliseconds. The sixth failure writes a DLQ row that stores the message id, and only then does the main queue get its ack. A crash before that DLQ row leaves the body on the main queue and leaves DLQ depth at 0. A redelivery that finds the id adds 0 to the DLQ and then acks. The charge job succeeds on try 3, so its body stays off the DLQ.

The breaker and the bulkhead

A circuit breaker counts consecutive failures on one dependency. The threshold here is 3, and the open period is 2000 milliseconds. Failures at t = 0, t = 10 and t = 20 are all sent. The third failure sets the state to open until t = 2020, and the consecutive count is 3. A call at t = 30 finds that open state. The worker answers 503 and leaves the call unsent, and the consecutive count stays 3. At t = 2020 the state becomes half-open for one trial, and that trial is sent. A success sets the state to closed and the consecutive count to 0. The chaos list is ten other calls against a closed breaker: fail, succeed, fail, succeed, fail, succeed, fail, succeed, succeed, succeed. That is 4 failures out of 10. The longest run of failures is 1, which sits under the threshold of 3, so the breaker stays closed and all 10 calls are sent. Three failures in a row are what open it.

A bulkhead is a separate slot count for each kind of work. Payments have 2 slots and reports have 4. In the snapshot both payment slots are taken and 1 report slot is taken, so free report slots are 4 − 1 = 3. Three more payments arrive and one report arrives. The three payments find 0 free payment slots and are refused at the bulkhead. The report takes one free slot, reports running become 2, and free report slots become 2. A bulkhead refusal leaves the breaker count unchanged. While the printer breaker is open, a print call gets 503 and a charge call still runs, because the charge breaker is closed. That split is the graceful degradation in this lesson: the broken path answers 503, and the healthy path still does the work.

A full queue and the SIGTERM clock

The main queue holds at most 5 messages, and depth starts at 0. Publishing another message when depth is already 5 is backpressure. The publish is refused, depth stays 5, and the refused body has no slot on the main queue and no slot on the DLQ. Of 6 publish attempts, 5 are accepted and 1 is refused. Checkout then takes one poison message off the visible queue: depth drops from 5 to 4, and in-flight rises from 0 to 1. That body fails all 6 tries, the DLQ row is stored, and the ack follows. In-flight returns to 0, depth stays 4, and DLQ depth becomes 1. The other four accepted bodies are still on the main queue. M2, M3, M4 and M5 account for that depth of 4.

SIGTERM tells the process to wind down. The grace in this lesson is 3000 milliseconds from the signal, and the process takes no new checkout after the signal. A job checked out at t = 0 with length 1000 milliseconds receives SIGTERM at t = 400. The job ends at t = 1000. The grace ends at 400 + 3000 = 3400. Because 1000 falls before 3400, the worker acks, in-flight ends at 0, and the message stays off the queue. Two consumers were running. The signal stops one, so 1 consumer remains and checkout continues on the other. A fleet of one consumer is a SPOF, a single point of failure. When that process exits, 0 consumers remain, and the depth it left behind waits until a new process starts.

An example

Four deliveries, one open breaker, one full queue

Cash starts at 200. K7 for 40 commits cash 160 and receipt R1. The second K7 for 40 adds 0 and returns R1, with cash still 160. K8 for 25 commits cash 135 and receipt R2. K7 for 99 differs from the stored 40, so cash stays 135, R1 stays stored, and the reply is a refusal. Money moved is 65. Separately, a crash before the K7 commit leaves cash at 200, and the redelivery applies once to reach 160. A crash after that commit and before the ack leaves the redelivery adding 0, with cash staying 160.

Ceilings after failed tries are 100, 200, 400, 800 and 800 milliseconds. The half draw sleeps 50, 100, 200, 400 and 400. The charge job fails try 1 and try 2, succeeds on try 3, and sleeps 50 + 100 = 150 milliseconds. It stays off the DLQ. A poison body uses all five sleeps, 1150 milliseconds, fails try 6, stores the DLQ row, and only then is acked. The quarter draw on the first ceiling is 25, so after one shared failure the half-draw worker and the quarter-draw worker wake 25 milliseconds apart.

Printer failures at t = 0, t = 10 and t = 20 are sent, and the breaker stays open until t = 2020. The call at t = 30 gets 503 and stays unsent. The trial at t = 2020 is sent, succeeds, and leaves the state closed with consecutive count 0. The chaos list of 10 calls has 4 failures and a longest run of 1, so that breaker stays closed and all 10 are sent. With 2 payment slots taken and 1 of 4 report slots taken, three further payments are refused and the one report is accepted, leaving 2 free report slots.

Six publishes hit a max depth of 5: 5 accepted, 1 refused, depth 5. Checkout of the poison body sets depth to 4 and in-flight to 1. After the DLQ row and the ack, depth is 4, in-flight is 0, and DLQ depth is 1. The 1000 millisecond job checked out at t = 0 gets SIGTERM at t = 400 and ends at t = 1000, inside the grace that ends at 3400, so the worker acks. Of 2 consumers, 1 remains.

Exercise

A second ledger, a longer ladder, a late signal

Cash starts at 200 with an empty key table. Apply K1 for 30, then K1 for 30, then K2 for 50, then K1 for 15. Write the cash after each delivery, the receipt stored for K1, and the money that moved. On the same ceilings, with the half draw, a job fails tries 1 through 4 and succeeds on try 5. Write those four sleeps, their sum, and whether the body reaches the DLQ. The printer failed at t = 0, t = 10 and t = 20. A call arrives at t = 500. Write the status and whether the call is sent. For the ten-call chaos list, write the longest failure run and whether that breaker opens. Payment slots are 2, both taken. Report slots are 4, with 1 taken. Three payments and one report arrive. Write how many payments are refused and how many report slots are free after the report is accepted.

Seven publishes arrive on an empty queue whose max depth is 5. Write how many are refused and the depth. One accepted poison body is then checked out and fails all 6 tries. Write the main depth and the DLQ depth after the ack. A different job has length 5000 milliseconds, checkout at t = 0, and SIGTERM at t = 400, with grace 3000 and depth 4 while it is in flight. This process is the only consumer. Write ack or nack, the time the grace ends, the depth after the nack, and how many consumers remain. Cash after the four deliveries is 170, then 170, then 120, then 120. The stored K1 receipt is R1, and the money moved is 80. The four sleeps are 50, 100, 200 and 400, the sum is 750, and the body stays off the DLQ. At t = 500 the status is 503 and the call stays unsent. The chaos run is 1 and that breaker stays closed. Three payments are refused and 2 report slots are free. Two of the seven publishes are refused and the depth is 5. After the poison ack the main depth is 4 and the DLQ depth is 1. The long job gets a nack. The grace ends at 3400, the nack leaves depth 5, and 0 consumers remain.

Check questions

Answer on your own first. Then open a question to see the answer.

Cash starts at 200. The deliveries are K7 for 40, K7 for 40, K8 for 25, and K7 for 99. What is the cash after the fourth delivery, and what does the second delivery return?

Cash is 135. The second delivery returns R1. The fourth delivery is a refusal, and R1 stays the stored receipt.

The half-draw sleeps are 50, 100, 200, 400 and 400 milliseconds. Tries 1 and 2 fail, and try 3 succeeds. What is the total sleep, and does the body move to the DLQ?

The total sleep is 150 milliseconds. The body stays off the DLQ, because success on try 3 stops the ladder.

Failures at t = 0, t = 10 and t = 20 open the breaker until t = 2020. Is the call at t = 30 sent, and what are the state and the consecutive count after a successful trial at t = 2020?

The call at t = 30 stays unsent and the worker answers 503. After the successful trial the state is closed and the consecutive count is 0.

Max depth is 5. Six publishes arrive on an empty queue, then one accepted poison body fails all 6 tries and is acked onto the DLQ. How many publishes were refused, what is the main depth after that ack, and what is the DLQ depth?

1 publish was refused, the main depth after the ack is 4, and the DLQ depth is 1.

Take with you

You store the key in the same commit as the charge, you add the 50 ms and 100 ms sleeps before the successful third try, you leave the call at t = 30 unsent while the breaker is open, and you ack the 1000 ms job because it ends inside the 3000 ms grace.

Final task

Submit a worker service and deployment rollback script that pass automated chaos-injection tests (40% downstream error rate, duplicate message delivery, SIGTERM mid-job, and automated canary rollback trigger).

The final task is marked against fixed answer keys and automated tests once the tests open.

Terms in this course

43 terms in the glossary belong to this course.

More lessons coming

The first lesson is open to read. The rest of the course is being written, and tests and diplomas come later. Join the waitlist