Card delivery tracking: from a next-day batch to real-time events
How I replaced a once-a-day status update with an event-driven flow on Kafka, and built recovery in so a failed update is retried instead of lost.
The problem
Card delivery status was updated by a batch job that ran once a day. A card could be delivered in the morning and still show as in transit until the next run, so the status was often up to a day out of date.
The goal was to show each change as it happened, without losing updates when a service downstream was slow or unavailable.
Before and after
Before
Delivery changes collect all day
Batch job runs once a day
Status updated next day (T+1)
After
Delivery event published
The moment the status changes
Kafka topic, then a Node.js consumer
Retries up to 3+ times on failure
Status updated in real time
Key decisions
Events instead of a schedule
Each status change is published to Kafka as it happens. Consumers process it straight away, so nothing waits for the next batch run.
Automatic retries per event
If a downstream call fails, the consumer retries the same event 3+ times. Most failures are short-lived, so this clears them without anyone stepping in.
A cron job as the safety net
Anything still failing after its retries is picked up by a scheduled job and processed again, so an outage delays an update instead of losing it.
The result
Delivery status moved from a next-day update to real time. The same pattern was then used to make credit card bill payments reflect on the card instantly.
What I would add next
Ideas I'm exploring for the next version of this pattern:
- Idempotency keys, so a retried event can never be applied twice.
- A dead-letter queue, to keep failed events visible and separate from live traffic.
- Reconciliation and alerts, comparing against the source of truth and alerting on consumer lag in Grafana.