Saba Janelidze · labBook a call

On call · Incident 2 · Identity checks · about 2 minutes · TEST MODE, no real money

Approved, then “pending” again

“I was approved yesterday. This morning I can’t trade: my status says ‘pending’ again.”

Anna · support chat · a made-up ticket

Find the bugRead the code and the tests →

Made by Saba Janelidze, principal engineer. I build payment and ledger systems for fintech and crypto teams. Hiring, or need help with identity checks or payments? Book a call →

The code is real and runs in your browser; Anna, Boris (a second applicant), the identity provider and the network are simulated. The provider numbers each applicant’s messages: #1, #2, #3. Build v0.1 is the code that handled them last night; build v1 is the fix you will pick.

  1. Reproduce
  2. Count
  3. Fix

Reproduce it

Lock Anna out again. Make the network lose one of our answers: the provider then sends that message again.

build v0.1 · last message wins · real code · simulated networkv0.1 · real code
What the screen shows
Which of our answers the network loses

Anna’s record the provider vs our server

Provider’s record
approved · v2approved
Our record
—
Can trade
—

A normal night first: two messages, two answers.

How it works: the code, the tests, and how it was built
Build v0.1 (last message wins)
7 of 14 delivery orders lock Anna out
Build v1 (the fix)
0 of 14 delivery orders lock Anna out
Real Postgres database, messages at once
build v1 ends approved
Automatic checks (CI) on this build
see the result

The first two numbers are computed from the code when the site is built; the third is what the real-database test checks on every push.

Run the tests yourself: the code on GitHub

pnpm install
pnpm --filter @lab/core test
cd go && go test -race ./...

Needs Node 24 with pnpm, and Go 1.25 for the last line. The real-database tests also run when DATABASE_URL points at a Postgres server.

What happened, and the fix

The identity provider tells us about each change as a signed message (it carries a code made with a key only the provider and we hold, so a forged message can be told apart), numbered per applicant: 1, 2, 3. When our answer to a message gets lost, it sends the same message again, so a message can arrive twice, and late. Build v0.1 let the last message to arrive win, so a late copy of “pending” undid “approved”. Build v1 checks the signature first, keeps each message once by its ID, and applies them in version order: an old version changes nothing, and an early one waits (it is parked) until the one before it arrives. Messages for one applicant are handled one at a time, under a lock on that applicant’s row.

The game runs the TypeScript version. The Go tab shows the same fix in Go.

Build v0.1 (last message wins)packages/core/src/kyc/teaching/build-v01.tsbuild-v01.ts:5–19
/** * Build v0.1, the naive handler: the last message received wins. It works while messages arrive once * and in order. Teaching code only. */export const handleEventV01: HandleEvent = async ({ store }, event) =>  store.forApplicant(event.applicantId, async (tx) => {    const applicant = await tx.loadApplicant()    await tx.saveApplicant({      ...applicant,      status: event.status,      version: event.version,      lastSentAt: event.sentAt,    })    return EVENT_RESULT.APPLIED  })
Build v1 (keep once, apply in order) · TypeScriptpackages/core/src/kyc/handle-event.tshandle-event.ts:5–53
/** * Build v1: check the signature first, store each message once, and apply messages in the provider's * version order. A message that arrives before an earlier version is parked, and applied as soon as * the gap closes. One applicant's messages never interleave (a row lock in Postgres). A refused * message is logged as evidence, apart from the stored ones (so it can never use up a real message's ID) * and outside the applicant's lock. */export const handleEvent: HandleEvent = async ({ store, signatures, refusals }, event) => {  const signatureValid = await signatures.verify(event)  if (!signatureValid) {    await refusals.record(event, EVENT_RESULT.BAD_SIGNATURE)    return EVENT_RESULT.BAD_SIGNATURE  }  const result = await store.forApplicant(event.applicantId, (tx) => applyInOrder(tx, event))  if (result === EVENT_RESULT.VERSION_CONFLICT) await refusals.record(event, result)  return result} /** Inside the applicant's lock: keep the message once, then apply it and every parked one that follows on. */async function applyInOrder(tx: ApplicantTx, event: ProviderEvent): Promise<EventResult> {  const recorded = await tx.recordEvent(event)  if (recorded === RECORD_RESULT.SAME_ID) return EVENT_RESULT.DUPLICATE  if (recorded === RECORD_RESULT.SAME_VERSION) {    // The same version under a new ID: a copy if it says the same, a conflict if it says something else.    const kept = await tx.storedEvent(event.version)    const sameContent = kept?.status === event.status    if (!sameContent) return EVENT_RESULT.VERSION_CONFLICT  }  const applicant = await tx.loadApplicant()  if (event.version <= applicant.version) return EVENT_RESULT.STALE  if (event.version > applicant.version + 1) return EVENT_RESULT.PARKED   // Apply this message, then every parked one that now follows on.  const applied = { ...applicant, status: event.status, version: event.version, lastSentAt: event.sentAt }  const caughtUp = await applyFollowing(tx, applied)  await tx.saveApplicant(caughtUp)  return EVENT_RESULT.APPLIED} /** From this state on, applies every kept message that follows on, one version after another (not saved yet). */export async function applyFollowing(tx: ApplicantTx, from: Applicant): Promise<Applicant> {  let applicant = from  let next = await tx.storedEvent(applicant.version + 1)  while (next) {    applicant = { ...applicant, status: next.status, version: next.version, lastSentAt: next.sentAt }    next = await tx.storedEvent(applicant.version + 1)  }  return applicant}
Build v1 (keep once, apply in order) · Gogo/kyc/handle_event.gohandle_event.go:7–72
// HandleEvent is build v1 in Go: check the signature first, keep each message once, and apply messages in the// provider's version order. A message that arrives before an earlier version is parked, and applied as soon as the// gap closes. One applicant's messages never interleave (a row lock in Postgres). A refused message is logged as// evidence, apart from the kept ones (so it can never use up a real message's ID) and outside the applicant's lock.func HandleEvent(ctx context.Context, store Store, signatures SignatureCheck, refusals RefusalLog, event ProviderEvent) (EventResult, error) {	signatureValid := signatures.Verify(event)	if !signatureValid {		err := refusals.Record(ctx, event, BadSignature)		return BadSignature, err	}	result, err := store.ForApplicant(ctx, event.ApplicantID, func(tx Tx) (EventResult, error) {		return applyInOrder(ctx, tx, event)	})	if err != nil {		return "", err	}	if result == VersionConflict {		err = refusals.Record(ctx, event, result)	}	return result, err} // applyInOrder runs inside the applicant's lock: keep the message once, then apply it and every parked one that// follows on.func applyInOrder(ctx context.Context, tx Tx, event ProviderEvent) (EventResult, error) {	recorded, err := tx.RecordEvent(ctx, event)	if err != nil {		return "", err	}	if recorded == RecordSameID {		return Duplicate, nil	}	if recorded == RecordSameVersion {		// The same version under a new ID: a copy if it says the same, a conflict if it says something else.		kept, err := tx.StoredEvent(ctx, event.Version)		if err != nil {			return "", err		}		sameContent := kept != nil && kept.Status == event.Status		if !sameContent {			return VersionConflict, nil		}	}	applicant, err := tx.LoadApplicant(ctx)	if err != nil {		return "", err	}	if event.Version <= applicant.Version {		return Stale, nil	}	if event.Version > applicant.Version+1 {		return Parked, nil	} 	// Apply this message, then every parked one that now follows on.	next := &event	for next != nil {		applicant.Status, applicant.Version, applicant.LastSentAt = next.Status, next.Version, next.SentAt		next, err = tx.StoredEvent(ctx, applicant.Version+1)		if err != nil {			return "", err		}	}	err = tx.SaveApplicant(ctx, applicant)	return Applied, err}

The tests

  • Every order, counted: all 14 orders Anna’s two messages can arrive in, on build v0.1 (7 lock her out) and on every patch on this page. The numbers in the game are the numbers in this test.
  • Random orders: 2 to 5 messages, each arriving up to three times, delivered in 300 random orders. With build v1, the applicant always ends at the latest version.
  • The defence check: the same test runs on build v0.1, without version order, and it must fail.fails when the defence is off ✓
  • The lock: #2 arrives while #1 is still being saved. With the lock, Anna ends approved at version 2; without it, the same two messages leave her stuck at #1.
  • A forged message with a real ID: it is refused before anything is kept, so it cannot use up the real message’s ID.
  • The Go version: the same handler in Go, with the same codes and the same signature (a test checks it matches the TypeScript one). Its lock test runs under Go’s race detector. (That tool fails a test when two parts of the program touch the same memory at the same time with nothing keeping them in order.)

The trade-off

Two ways hold. Where the provider numbers its messages (this one does), apply them in version order, as here. Where it does not, treat each message as a nudge: ask the provider’s API for the applicant’s current status and save that. Saba uses both, depending on the provider. Asking costs one call per message and needs the provider’s API to answer; version order needs no call, but an early message waits until the one before it arrives.

Not fixes: ignoring older timestamps (the provider stamps its messages to the second, so two can carry the same time), and refusing moves that are not allowed (it throws away a message that came early; unless a copy comes later, it is lost).

Why not answer with an error and let the provider send it again later? That leans on the provider’s retry schedule, whose waits grow to hours, and providers ask for a fast answer. Parking is the same idea, done on our side.

On a real database

On Postgres, each message first locks its applicant’s row (SELECT … FOR UPDATE), so two messages for one applicant are handled one after the other. Each message is kept once: its ID is the key, so a copy adds nothing (ON CONFLICT DO NOTHING). A message with a new ID that claims a version already kept, and says something else, is refused and logged, because each applicant can hold only one message per version (UNIQUE (applicant, version)); one that says the same is a copy.

The row lock on Postgres · TypeScriptpackages/core/src/kyc/adapters/postgres-store.tspostgres-store.ts:80–90
      // The applicant's row exists from the start of the check; a first message creates it if not. Then the row      // is locked, before the message is kept: every message for this applicant waits here until the one before      // it has committed, so two never interleave (and a copy waits on the lock, not on a half-written row).      await tx.query(        `INSERT INTO kyc_applicants (applicant_id, status) VALUES ($1, 'NOT_STARTED')         ON CONFLICT (applicant_id) DO NOTHING`,        [applicantId],      )      if (this.#lockRow) {        await tx.query('SELECT 1 FROM kyc_applicants WHERE applicant_id = $1 FOR UPDATE', [applicantId])      }
The row lock on Postgres · Gogo/kyc/postgres.gopostgres.go:66–78
	// The applicant's row exists from the start of the check; a first message creates it if not. Then the row is	// locked, before the message is kept: every message for this applicant waits here until the one before it has	// committed (and a copy waits on the lock, not on a half-written row).	_, err = tx.Exec(ctx, `INSERT INTO kyc_applicants (applicant_id, status) VALUES ($1, 'NOT_STARTED')		ON CONFLICT (applicant_id) DO NOTHING`, applicantID)	if err != nil {		return "", err	}	if s.lockRow {		if _, err := tx.Exec(ctx, `SELECT 1 FROM kyc_applicants WHERE applicant_id = $1 FOR UPDATE`, applicantID); err != nil {			return "", err		}	}
Keeping each message oncepackages/core/src/kyc/adapters/postgres-store.tspostgres-store.ts:121–127
        const { rows } = await sql.query(          `INSERT INTO kyc_provider_events (event_id, applicant_id, version, status, sent_at, signature)           VALUES ($1, $2, $3, $4, $5, $6)           ON CONFLICT DO NOTHING           RETURNING event_id`,          [event.eventId, applicantId, event.version, event.status, event.sentAt, event.signature],        )

The real-database race: in CI, on a real Postgres server, #2 arrives while #1 is being saved (each look for the next version waits 40 ms). With the row lock, Anna ends approved at version 2; without it, she is stuck at #1. Another test sends #1, #2 and their copies, 20 deliveries at once: approved at version 2, each message kept once.

The clean-up code

After a night on build v0.1, ask the provider’s API for its own record of each applicant (a connection where we log in, so no one else can answer; simulated here) and compare. Behind: catch up to the provider’s latest, then apply any message we parked. A status the provider never sent: hold trading. The record is not rolled back: it stays as the night left it, as evidence, even the time it says it was sent, which only the message claimed. A person lifts the hold after the review, never a message. And if our record is newer than the provider’s answer (a real message can land while it answers), the provider is asked again before anything is called “not from the provider”. Its test replays last night on build v0.1, then runs the clean-up.

Our record against the provider’s: Show the code (19 lines)
Our record against the provider’spackages/core/src/kyc/provider-check.tsprovider-check.ts:19–37
/** * Our record of an applicant against the provider's own record of them. Every status we hold must be one the * provider sent at that version; if it is, we are either up to date or behind; if it is not, it came from * somewhere else (a message the provider never sent). */export function compareWithProvider(applicant: Applicant, record: ProviderRecord): ProviderComparison {  const ours = { status: applicant.status, version: applicant.version }  const latest = record.history.reduce<Position>(    (best, entry) => (entry.version > best.version ? { status: entry.status, version: entry.version } : best),    { status: KYC_STATUS.NOT_STARTED, version: 0 },  )  const untouched = applicant.version === 0 && applicant.status === KYC_STATUS.NOT_STARTED  const sent = record.history.some((entry) => entry.version === ours.version && entry.status === ours.status)  const fromProvider = untouched || sent  if (!fromProvider) return { result: COMPARISON.NOT_FROM_PROVIDER, ours, provider: latest }  const behind = ours.version < latest.version  if (behind) return { result: COMPARISON.BEHIND, ours, provider: latest }  return { result: COMPARISON.MATCHES, ours, provider: latest }}
The clean-up: Show the code (50 lines)
The clean-uppackages/core/src/kyc/provider-check.tsprovider-check.ts:41–90
/** How many times the clean-up asks the provider before it decides that a status did not come from it. */const ASKS = 2 /** * The clean-up: ask the provider (over our authenticated connection), compare, and act. Behind → catch up to the * provider's latest, then apply any message we parked that now follows on. Not from the provider → hold trading and * keep the record as it is, as evidence (it is not rolled back; a person lifts the hold after the review, never a * message). The provider is asked outside the applicant's lock, so that a slow answer holds up none of their * messages; a real message can therefore be applied after the provider answered. So when our record is newer than * the answer, the provider is asked again before anything is called "not from the provider". */export async function reconcileWithProvider(  store: KycStore,  provider: ProviderApi,  applicantId: string,): Promise<ProviderComparison> {  for (let ask = 1; ; ask++) {    const record = await provider.record(applicantId)    const lastAsk = ask >= ASKS    const comparison = await store.forApplicant(applicantId, (tx) => settle(tx, record, lastAsk))    if (comparison) return comparison  }} /** Inside the applicant's lock: compare and act; or do nothing (null) while the provider's answer may be out of date. */async function settle(  tx: ApplicantTx,  record: ProviderRecord,  lastAsk: boolean,): Promise<ProviderComparison | null> {  const applicant = await tx.loadApplicant()  const comparison = compareWithProvider(applicant, record)  const ahead = applicant.version > comparison.provider.version  if (comparison.result === COMPARISON.NOT_FROM_PROVIDER && ahead && !lastAsk) return null  if (comparison.result === COMPARISON.BEHIND) {    const latest = record.history.find((entry) => entry.version === comparison.provider.version)    const caughtUp = {      ...applicant,      status: comparison.provider.status,      version: comparison.provider.version,      lastSentAt: latest?.sentAt ?? applicant.lastSentAt,    }    const withParked = await applyFollowing(tx, caughtUp)    await tx.saveApplicant(withParked)  }  if (comparison.result === COMPARISON.NOT_FROM_PROVIDER) {    await tx.saveApplicant({ ...applicant, hold: HOLD_REASON.NOT_FROM_PROVIDER })  }  return comparison}

How this code was built

Saba chose what to build, set the rules and approved every step. An AI agent (Claude) wrote all 10 commits on this code. The local checks before a commit ran 11 times on this code and refused 1; on every push, CI runs the tests, the real-database race and Go’s race detector. Codex, OpenAI’s AI for code, reviewed this incident before it was published, as a second opinion: six findings. Five were fixed, three of them with a test that would fail without the fix; one is kept on purpose, with the reason. The review, word for word: incident 2. Result of the automatic checks (CI) for this build. Last change: e22830d.

I build this for teams that onboard customers and handle their money. Book a call

Full text version (contains the answers)

The whole incident, as text

The setup

The identity provider sends us two signed messages about Anna, in the same second: #1 “pending” (version 1), then #2 “approved” (version 2). Our server answers each one with 200 OK. When an answer gets lost, the provider sends that message again. Build v0.1 saves whatever message arrives, so the last one to arrive wins.

1. Reproduce

Lose our answer to #1: the provider sends #1 again after #2, and Anna ends “pending”: locked out. Lose our answer to #2: the copy of #2 says the same as #2, and she ends “approved”. Lose both: the copies come in their order, #2 last, and she ends “approved”.

2. Count

Each message can arrive once or twice, in any order: 14 delivery orders (both once: 2 orders; one of them twice: 3 + 3; both twice: 6). 7 of them (50%) lock Anna out on build v0.1: every one where #1 arrives last. Every order, in the order the messages arrived:

  1. 1 2: approved
  2. 2 1: locked out
  3. 1 1 2: approved
  4. 1 2 1: locked out
  5. 2 1 1: locked out
  6. 1 2 2: approved
  7. 2 1 2: approved
  8. 2 2 1: locked out
  9. 1 1 2 2: approved
  10. 1 2 1 2: approved
  11. 1 2 2 1: locked out
  12. 2 1 1 2: approved
  13. 2 1 2 1: locked out
  14. 2 2 1 1: locked out

3. Fix

  • Ignore older messages: still locks Anna out in 7 of 14 orders. Both messages left the provider in the same second, so “older” can’t order them.
  • Refuse moves that aren’t allowed: still locks Anna out in 4 of 14 orders. When #2 comes first, “approved” before “pending” is not an allowed move, so that arrival is thrown away. Unless a copy of #2 comes later, Anna stays “pending”.
  • Keep once, apply in order: locks Anna out in 0 of 14 orders. This is build v1.

All three patches check the signature first, so a forged “approved” for Boris gets in with none of them; it gets in only with build v0.1.

Go deeper: the regulator asks

Build v1 is live, but last night ran build v0.1, and the regulator asks for Boris’s file. Our record says “approved” at version 3, sent at 03:12; but that time, too, is only what the message said. Build v0.1 kept only the latest status, so our record only repeats what that message said, and no message is on file to search. Asking the provider for its record shows it never sent Boris a version 3: that approval did not come from the provider. So Boris’s trading is held until a person lifts it, his record is kept as evidence, what he did while “approved” is reviewed, and it is reported as a security incident. Anna, who was only behind, catches up to “approved” and can trade.