Saba Janelidze · labBook a call

On call · Incident 1 · Payments · about 2 minutes · TEST MODE, no real money

The bank paid out twice

“I had 100. I tapped Withdraw 80 twice by accident. The bank paid me 160, and my balance still shows 20.”

Customer #42 · support chat · a made-up ticket

Find the bugRead the code and the tests →

Made by Saba Janelidze, principal engineer. I build payment and ledger systems for fintech and crypto teams. Hiring, or need help with payments? Book a call →

The code is real and runs in your browser; the customer and the bank are simulated. Build v0.1 is the withdrawal code as it ran this morning; build v1 is the fix you will pick.

  1. Reproduce
  2. Count
  3. Fix

Reproduce it

Make the bank pay twice. A and B are the customer's two taps, at the same time. Tap a step on lane A: A gets slow there, B runs past, A finishes.

build v0.1 · withdraw 80 · real code · simulated timingv0.1 · real code · simulated timing

The books our ledger vs the bank

Customer had
100.00100
Balance shows
100.00100
On hold
0.000
Bank paid out
0.000
Money lost
…

How it works: the code, the tests, and how it was built
Build v0.1 (naive)
36 of 54 ways to run lose money
Build v1 (the fix)
0 of 24 ways to run lose money
Real Postgres database, 20 withdrawals at once
build v1 pays out once
Automatic checks (CI) on this build
see the result

Each build has its own steps, so its own number of orders. The first two numbers are computed from the code when the site is built; the third is what the real-database test checks on every push.

Run the tests yourself: the code on GitHub

pnpm install
pnpm --filter @lab/core test
cd go && go test -race ./...

Needs Node 24 with pnpm, and Go 1.25 for the last line. The real-database tests also run when DATABASE_URL points at a Postgres server.

What happened, and the fix

Build v0.1 reads, checks and saves the balance in separate steps, so two requests can both pass the check. Build v1 makes the check and the move one atomic step: it happens whole or not at all, so no second request can slip in between.

The game runs the TypeScript version. The Go tab shows the same fix in Go.

Build v0.1 (naive)packages/core/src/payments/teaching/build-v01.tsbuild-v01.ts:10–23
/** * Build v0.1, the naive withdrawal. It checks the balance, so it looks safe. * But two copies running at once can both read the old balance before either saves. */export const withdrawV01: Withdraw = async ({ store, bank }, { customerId, amount, payoutId }) => {  const amountIsValid = isValidAmount(amount)  if (!amountIsValid) return refused(REFUSAL.INVALID_AMOUNT)  await store.loadCustomer(customerId)  const available = await store.readAvailable(customerId)  if (available < amount) return refused(REFUSAL.INSUFFICIENT_FUNDS)  await store.writeAvailable(customerId, available - amount)  await bank.pay(payoutId, customerId, amount)  return PAID}
Build v1 (reserve, then settle) · TypeScriptpackages/core/src/payments/withdraw.tswithdraw.ts:31–63
/** * Build v1: reserve, then settle. The check and the move are one atomic step, * so a second request can never reserve money the first one already holds. * A repeated payout ID never moves money again: it gets the stored answer * (UNKNOWN while the first payout is still on hold). */export const withdraw: Withdraw = async ({ store, bank }, { customerId, amount, payoutId }) => {  const amountIsValid = isValidAmount(amount)  if (!amountIsValid) return refused(REFUSAL.INVALID_AMOUNT)  await store.loadCustomer(customerId)  const reserved = await store.reserve(customerId, amount, payoutId)  if (reserved === RESERVE_RESULT.DUPLICATE) {    const record = await store.payout(payoutId)    return answerFor(record, customerId, amount)  }  if (reserved === RESERVE_RESULT.INSUFFICIENT_FUNDS) return refused(REFUSAL.INSUFFICIENT_FUNDS)   let answer: Awaited<ReturnType<typeof bank.pay>>  try {    answer = await bank.pay(payoutId, customerId, amount)  } catch (error) {    // Never retry blindly: the bank may already have paid. Keep the hold and resolve it later.    if (error instanceof BankTimeout) return UNKNOWN    throw error  }   if (answer === BANK_ANSWER.PAID) {    await store.settle(payoutId)    return PAID  }  await store.release(payoutId)  return refused(REFUSAL.BANK_REFUSED)}
Build v1 (reserve, then settle) · Gogo/payments/withdraw.gowithdraw.go:10–52
// Withdraw is build v1 in Go: reserve, then settle. The check and the move are one atomic step,// so a second request can never reserve money the first one already holds.// A repeated payout ID never moves money again: it gets the stored answer// (UNKNOWN while the first payout is still on hold).func Withdraw(ctx context.Context, store Store, bank Bank, req Request) (Result, error) {	if req.Amount <= 0 {		return refused(RefusalInvalidAmount), nil	}	if err := store.LoadCustomer(ctx, req.CustomerID); err != nil {		return Result{}, err	}	reserved, err := store.Reserve(ctx, req.CustomerID, req.Amount, req.PayoutID)	if err != nil {		return Result{}, err	}	switch reserved {	case ReserveDuplicate:		record, err := store.Payout(ctx, req.PayoutID)		if err != nil {			return Result{}, err		}		return answerFor(record, req), nil	case ReserveInsufficientFunds:		return refused(RefusalInsufficientFunds), nil	} 	answer, err := bank.Pay(ctx, req.PayoutID, req.CustomerID, req.Amount)	timedOut := errors.Is(err, ErrBankTimeout)	if timedOut {		// Never retry blindly: the bank may already have paid. Keep the hold and resolve it later.		return Result{Status: StatusUnknown}, nil	}	if err != nil {		return Result{}, err	} 	if answer == BankPaid {		err = store.Settle(ctx, req.PayoutID)		return Result{Status: StatusPaid}, err	}	err = store.Release(ctx, req.PayoutID)	return refused(RefusalBankRefused), err}
The stored answer for a repeated payout ID: Show the code (9 lines)
The stored answer for a repeated payout ID · TypeScriptpackages/core/src/payments/withdraw.tswithdraw.ts:67–75
/** The stored answer for a repeated payout ID. The same ID with a different request is refused. */function answerFor(record: PayoutRecord | null, customerId: string, amount: Cents): WithdrawResult {  if (!record || record.customerId !== customerId || record.amount !== amount) {    return refused(REFUSAL.PAYOUT_ID_REUSED)  }  if (record.status === PAYOUT_STATUS.PAID) return PAID  if (record.status === PAYOUT_STATUS.RELEASED) return refused(REFUSAL.BANK_REFUSED)  return UNKNOWN}
The stored answer for a repeated payout ID · Gogo/payments/withdraw.gowithdraw.go:57–70
// answerFor is the stored answer for a repeated payout ID. The same ID with a different request is refused.func answerFor(record *PayoutRecord, req Request) Result {	if record == nil || record.CustomerID != req.CustomerID || record.Amount != req.Amount {		return refused(RefusalPayoutIDReused)	}	switch record.Status {	case PayoutPaid:		return Result{Status: StatusPaid}	case PayoutReleased:		return refused(RefusalBankRefused)	default:		return Result{Status: StatusUnknown}	}}

The tests

  • Every run, counted: all 54 ways build v0.1 can run (36 lose money), and every patch on this page. The numbers in the game are the numbers in this test.
  • Random schedules: 2 to 5 withdrawals of random amounts, run in hundreds of random orders. With build v1, none loses money.
  • The defence check: the same test runs with the atomic reserve switched off, and it must fail.fails when the defence is off ✓
  • The unknown outcome: a timeout keeps the money on hold; it is resolved by asking the bank, never by paying again.
  • The Go version: the same fix in Go, with the same codes and the same reserve SQL. Its tests send 20 withdrawals at once, in memory and on a real Postgres server, under Go's race detector. (That tool fails a test when two parts of the program touch the same memory at the same time with nothing keeping them in order.) Build v1 pays out once; build v0.1 pays out more than once.

The trade-off

A lock around both the read and the save also holds: 0 of its 50 ways to run lose money (its steps differ from build v0.1, so it has 50 ways to run, not 54). The reserve is still the better step, for two reasons. It is one statement, so there is no lock to forget to release on an early return (look at the extra unlock the lock version needs). And money in flight gets its own state, "on hold" (it has left the balance but not yet the exchange), so a timeout or a refused payout can be settled or released later from the books.

Lock around the read and the save (also safe): Show the code (19 lines)
Lock around the read and the save (also safe)packages/core/src/payments/teaching/patches.tspatches.ts:43–61
/** "Lock around the read and the save": safe too. The proof card's trade-off says why the reserve still wins. */export const withdrawLockReadAndSave: Withdraw = async (  { store, bank },  { customerId, amount, payoutId },) => {  const amountIsValid = isValidAmount(amount)  if (!amountIsValid) return refused(REFUSAL.INVALID_AMOUNT)  await store.loadCustomer(customerId)  await store.lock(customerId)  const available = await store.readAvailable(customerId)  if (available < amount) {    await store.unlock(customerId)    return refused(REFUSAL.INSUFFICIENT_FUNDS)  }  await store.writeAvailable(customerId, available - amount)  await store.unlock(customerId)  await bank.pay(payoutId, customerId, amount)  return PAID}

On a real database

On a real database the reserve is one SQL statement: it moves the 80 to "on hold" only if the balance covers it, and writes the payout record in the same statement. The CHECK (available >= 0) on the table is a backstop (a last line of defence), not the fix.

The reserve on Postgres · TypeScriptpackages/core/src/payments/adapters/postgres-store.tspostgres-store.ts:68–81
      const { rows } = await this.#sql.query(        `WITH moved AS (           UPDATE balances              SET available = available - $2, held = held + $2            WHERE customer_id = $1 AND available >= $2              AND NOT EXISTS (SELECT 1 FROM payouts WHERE payout_id = $3)           RETURNING customer_id         )         INSERT INTO payouts (payout_id, customer_id, amount, status)         SELECT $3, customer_id, $2, 'HELD' FROM moved         RETURNING payout_id`,        [customerId, amount, payoutId],      )      if (rows.length === 1) return RESERVE_RESULT.RESERVED
The reserve on Postgres · Gogo/payments/postgres.gopostgres.go:60–97
// Reserve is one SQL statement: the balance check, the move to on hold and the payout record// happen together or not at all, so two requests can never both pass the check.func (s *PostgresStore) Reserve(ctx context.Context, customerID string, amount Cents, payoutID string) (ReserveResult, error) {	if amount <= 0 {		return "", fmt.Errorf("not a valid amount: %d", amount)	}	var id string	err := s.pool.QueryRow(ctx, `		WITH moved AS (		  UPDATE balances		     SET available = available - $2, held = held + $2		   WHERE customer_id = $1 AND available >= $2		     AND NOT EXISTS (SELECT 1 FROM payouts WHERE payout_id = $3)		  RETURNING customer_id		)		INSERT INTO payouts (payout_id, customer_id, amount, status)		SELECT $3, customer_id, $2, 'HELD' FROM moved		RETURNING payout_id`, customerID, amount, payoutID).Scan(&id)	if err == nil {		return Reserved, nil	}	var pgErr *pgconn.PgError	isPgErr := errors.As(err, &pgErr)	if isPgErr && pgErr.Code == "23505" {		// The same new ID at the same moment: the primary key stops the second statement, all of it.		return ReserveDuplicate, nil	}	noRows := errors.Is(err, pgx.ErrNoRows)	if !noRows {		return "", err	}	if record, err := s.Payout(ctx, payoutID); err != nil {		return "", err	} else if record != nil {		return ReserveDuplicate, nil	}	return ReserveInsufficientFunds, nil}

The real-database race: in CI, 20 withdrawals of 80 hit a real Postgres server at once. Build v0.1's requests each wait 20 ms after reading the balance, like a slow network, so its gap is always hit: it pays out more than once. Build v1 has no separate read; it pays out exactly once.

That test also caught a bug the in-memory model can't show: a deadlock. A request repeating a payout ID held the balance row and waited for the payout row, while the settle of the first request held the payout row and waited for the balance row. The fix is the usual one: settle and release now lock the balance row first, in the same order as the reserve.

The lock order (found by the real-database test): Show the code (9 lines)
The lock order (found by the real-database test)packages/core/src/payments/adapters/postgres-store.tspostgres-store.ts:153–161
      // Lock order: the balance row first, then the payout row, the same order as the reserve.      // Payout first here would deadlock with a reserve that repeats this payout ID      // (the real-database test in CI found it).      await tx.query(        `SELECT 1 FROM balances          WHERE customer_id = (SELECT customer_id FROM payouts WHERE payout_id = $1)            FOR UPDATE`,        [payoutId],      )

The clean-up code

When the bank doesn't answer, the money stays on hold, and two pieces of code settle it later. One asks the bank about a single payout. The end-of-day job checks every open hold and the bank's statement for the day. Both work on either store, and the end-of-day job has a Postgres test of its own.

Resolve an unknown payout: Show the code (19 lines)
Resolve an unknown payoutpackages/core/src/payments/clean-up.tsclean-up.ts:16–34
/** * Clean-up for an unknown outcome: ask the bank what happened, never pay again to find out. * NOT_FOUND is not final (the bank may still be processing it), so the money stays on hold. */export async function resolveUnknownPayout(  { store, bank }: PaymentPorts,  payoutId: string,): Promise<Resolution> {  const status = await bank.status(payoutId)  if (status === BANK_STATUS.PAID) {    await store.settle(payoutId)    return RESOLUTION.PAID  }  if (status === BANK_STATUS.REFUSED) {    await store.release(payoutId)    return RESOLUTION.RELEASED  }  return RESOLUTION.STILL_UNKNOWN}
The end-of-day reconciliation: Show the code (59 lines)
The end-of-day reconciliationpackages/core/src/payments/clean-up.tsclean-up.ts:48–106
/** * The end-of-day job, in two passes. * 1. Every payout still on hold is checked with the bank: paid → settle, refused → release, *    not found → leave on hold, unless the bank's statement for the day is final. * 2. Every payout on the bank's statement must match one of our payout records. *    Anything else is flagged for a person: money left that our books can't explain. * It runs on either store (memory or Postgres) through two small ports. */export async function reconcile(  store: ReconcileStore,  bank: BankStatement,  options: { readonly statementIsFinal: boolean } = { statementIsFinal: false },): Promise<ReconcileLine[]> {  const lines: ReconcileLine[] = []   const holds = await store.openHolds()  for (const hold of holds) {    const found = await bank.status(hold.payoutId)    const line = { payoutId: hold.payoutId, amount: hold.amount, bank: found }    if (found === BANK_STATUS.PAID) {      await store.settle(hold.payoutId)      lines.push({ ...line, action: RECONCILE_ACTION.SETTLED })    } else if (found === BANK_STATUS.REFUSED || options.statementIsFinal) {      await store.release(hold.payoutId)      lines.push({ ...line, action: RECONCILE_ACTION.RELEASED })    } else {      lines.push({ ...line, action: RECONCILE_ACTION.LEFT_ON_HOLD })    }  }   const ours = await store.payoutRecords()  const records = new Map(ours.map((record) => [record.payoutId, record]))  const statement = await bank.statement()  const seen = new Map<string, number>()  for (const payout of statement) {    const times = (seen.get(payout.payoutId) ?? 0) + 1    seen.set(payout.payoutId, times)    const record = records.get(payout.payoutId)    const flag = !record      ? RECONCILE_FLAG.NOT_IN_OUR_BOOKS      : times > 1        ? RECONCILE_FLAG.PAID_MORE_THAN_ONCE        : record.amount !== payout.amount || record.customerId !== payout.customerId          ? RECONCILE_FLAG.AMOUNT_OR_CUSTOMER_DIFFERS          : record.status !== PAYOUT_STATUS.PAID            ? RECONCILE_FLAG.STATUS_DIFFERS            : null    if (flag) {      lines.push({        payoutId: payout.payoutId,        amount: payout.amount,        bank: BANK_STATUS.PAID,        action: RECONCILE_ACTION.FLAGGED,        flag,      })    }  }  return lines}

How this code was built

Saba chose what to build, set the rules and approved every step. An AI agent (Claude) wrote all 13 commits on this code. The local checks before a commit ran 13 times on this code and refused 0; on every push, CI runs the tests, the real-database race and Go's race detector. Codex, OpenAI's AI for code, reviewed this incident twice as a second opinion: eight findings on the TypeScript and its page, four on the Go version. All were fixed, with tests for the money cases. The reviews, word for word: incident 1, the Go version. Result of the automatic checks (CI) for this build. Last change: cc9de33.

I build this for teams that handle money. Book a call

Full text version (contains the answers)

The whole incident, as text

The code: build v0.1

Two copies of this withdrawal run at the same moment, each for 80, on a balance of 100:

  1. load the customer
  2. read the balance and check it is at least 80
  3. save the new balance
  4. pay out

1. Reproduce

Pick the step after which request A gets slow; B then runs to the end, then A finishes. Only one of the 4 places loses money: after step 2. A has read 100; B reads 100 too, saves 20 and pays 80; A saves 20 and pays 80. The bank paid out 160.00, the balance shows 20.00, and the exchange is 80.00 short.

2. Count

The two requests can run in 54 different ways (different orders of their steps). 36 of them (67%) lose money: build v0.1 loses whenever both requests read the balance before either one saves.

3. Fix

  • Check the balance twice: still loses in 120 of 182 ways to run (66%). The gap moves; it doesn't close.
  • Lock the read: still loses in 100 of 180 ways to run (56%). The lock is released before the save.
  • Reserve, then settle: loses in 0 of 24 ways to run. This is build v1.

Go deeper: the bank stops answering

With build v1 live, a payout of 80 times out at the bank. The money stays on hold and the payout is marked "unknown". Sending a new payout (a new payout ID) pays twice if the bank had already paid. Asking the bank about this payout settles the hold at once if the bank paid; "not found" is not final, so the money stays on hold. Tonight's reconciliation checks every open hold with the bank, and every payout on the bank's statement against our own records, so it also catches a payout our books can't explain.