Designing a Distributed Payments System

A payment system is a compact test of senior engineering judgment. It has ordinary distributed-system failures—timeouts, duplicates, reordering, partial outages—but an incorrect retry can move real money twice. The design goal is not mythical exactly-once delivery. It is effectively-once business behavior built from idempotent commands, immutable accounting, explicit state transitions, and reconciliation.

This lesson designs card authorization and capture for a marketplace. The platform accepts a payment request, calls an external payment service provider (PSP), records what money movement means internally, and eventually settles merchants.

1. Start with invariants, not boxes

Before choosing a database or queue, state what must remain true:

1. A client retry must not create a second logical payment.
2. A processor retry must not create a second external charge.
3. Every committed money movement must be represented by balanced ledger entries.
4. Payment status must move only through legal transitions; late messages cannot move it backward.
5. The system must retain enough evidence to explain any balance to support, finance, and auditors.
6. Internal records and processor settlement must eventually agree—or produce an actionable discrepancy.

Those invariants produce the architecture. They are also the standard against which every failure scenario is judged.

2. Requirements and estimates

Functional scope

• Create a payment intent for an order.
• Authorize a card, then capture immediately or later.
• Refund a captured payment.
• Receive signed processor webhooks.
• Expose current status and an audit history.
• Reconcile internal transactions against processor settlement files.

Non-functional requirements

Assume 10 million payments/day with a 10× peak factor:

average creates/sec = 10,000,000 / 86,400 ≈ 116
peak creates/sec    ≈ 1,160
write events        ≈ 8 per payment → ~9,300 writes/sec peak

ledger entries:
10M payments × 4 entries across auth/capture/refund lifecycle
≈ 40M entries/day

The raw QPS is manageable for a partitioned relational system. Correct transactions, operational isolation, and auditability matter more than choosing a fashionable high-scale datastore.

3. The architecture and its failure paths

Use the scenarios below as a design review. Do not only inspect the happy path: switch to each failure and explain which invariant prevents money from moving twice.

The components have deliberately separate jobs:

4. API design: identify the command

POST /v1/payments
Idempotency-Key: checkout_84d9_attempt_1
Content-Type: application/json

{
  "order_id": "ord_84d9",
  "amount": 12900,
  "currency": "USD",
  "payment_method_token": "pm_tok_...",
  "capture": false
}

The server stores:

(merchant_id, idempotency_key)
request_hash
payment_id
status: processing | completed | failed
response_status + response_body
expires_at

The key is scoped to the authenticated merchant or tenant. The request hash matters: if the same key arrives with a different amount, returning the original response would hide a client bug. Reject the mismatch.

The API can return:

• 201 Created when the operation completed synchronously.
• 202 Accepted with a payment ID when the processor result is still pending.
• The original saved response for a completed duplicate.
• 409 Conflict when a key is reused with a different canonical request.

5. Payment state is a monotonic state machine

created
  ├─> authorizing ─> authorized ─> capturing ─> captured ─> refunding ─> refunded
  │         │               │             │
  │         └─> failed      └─> voided    └─> failed_or_unknown
  └─> canceled

Persist a state version and transition with compare-and-swap semantics:

UPDATE payments
SET status = 'captured', version = version + 1
WHERE id = :payment_id
  AND status IN ('authorized', 'capturing')
  AND version = :expected_version;

A late authorized webhook cannot overwrite captured, because that transition is illegal. A duplicate captured event becomes a no-op because its processor event ID and ledger transaction reference have already been recorded.

Do not compress every unknown into failed. A timeout means the caller does not know. The processor may have committed before the response disappeared. Preserve pending or unknown so retry and reconciliation can resolve it.

6. The ledger is not the payment table

payments.status = "captured" is workflow state. It is not an auditable model of who owns money.

A double-entry ledger records immutable transactions whose debits and credits balance:

Useful tables:

ledger_transactions(
  id, reference_type, reference_id, event_type,
  effective_at, created_at
)

ledger_entries(
  transaction_id, account_id, direction,
  amount_minor, currency
)

Enforce in one database transaction:

• Entries use integer minor units or fixed-precision decimals, never binary floating point.
• All entries share one currency unless an explicit FX transaction bridges accounts.
• Sum(debits) equals sum(credits).
• (reference_type, reference_id, event_type) is unique for idempotent posting.
• Posted entries are immutable; corrections use reversing and replacement transactions.

7. Atomic state and event publication

After committing payment = authorized, downstream capture, notification, and analytics consumers need an event. Writing the database and publishing to a queue as two independent operations creates a dual-write bug:

database commit succeeds
process crashes before queue publish
payment is authorized, but capture never starts

Write an outbox row in the same local transaction as the payment transition. A relay publishes unpublished rows and marks them sent. Publication can repeat, so consumers dedupe by event ID or process idempotently.

BEGIN;
  UPDATE payments ...;
  INSERT INTO outbox(event_id, aggregate_id, event_type, payload)
  VALUES (..., 'payment.authorized', ...);
COMMIT;

The outbox closes the local atomicity gap. It does not make consumers exactly once.

8. Webhooks are untrusted, duplicated, and reordered

A robust webhook path:

1. Verify the signature over the raw body and enforce a timestamp tolerance.
2. Persist the provider event ID in an inbox table with a unique constraint.
3. Acknowledge quickly after durable acceptance.
4. Process asynchronously.
5. Fetch processor state when the event is incomplete or suspicious.
6. Apply only a legal versioned transition.
7. Post ledger entries with a unique business reference.

Do not rely on webhook arrival order. A capture event can race the synchronous authorization response; a refund can be delivered twice; an old authorization update can arrive after settlement.

9. Reconciliation: the repair loop

Idempotency prevents many duplicates, but no request path proves that two independent financial systems agree forever. Reconciliation compares:

internal payment operations
internal ledger transactions
processor transaction export
processor settlement and fee report
bank deposit

Classify mismatches:

• Present at processor, absent internally.
• Present internally, absent at processor.
• Amount or currency mismatch.
• Duplicate processor operation.
• Status/timing mismatch.
• Fee or settlement mismatch.

The reconciler should create a discrepancy record, attach evidence, and route by policy. Safe automated repairs post explicit compensating transactions. They never edit old ledger rows to make the mismatch disappear.

10. What to memorize vs. what to derive

Memorize these invariants

• Same logical command → same idempotency key.
• Same external operation → same processor key on retry.
• Ledger transaction → balanced immutable entries.
• Unknown external result → pending plus retry/reconcile, not guessed failure.
• State transitions → monotonic, versioned, and idempotent.
• State change plus event → transactional outbox.

Derive these from requirements

• Synchronous vs. asynchronous authorization response.
• Database partition key and regional ownership.
• How long idempotency records live.
• Whether capture is immediate, delayed, or partial.
• Which discrepancies can be repaired automatically.
• Availability policy when the processor, ledger, or queue is unavailable.

Worked design questions

transition capturing -> captured if legal/version matches
insert ledger transaction with unique (payment_id, "capture")
insert payment.captured outbox event

payment pay_1
authorization auth_1: 200
capture cap_1: 120
capture cap_2: 50
refund ref_1 against cap_1: 20

authorized       = 200
captured gross   = 170
refunded         = 20
captured net     = 150
remaining auth   = 30