DataCheck backend architecture

Stack

Guardrails

• Money only moves through the append-only, double-entry ledger, in one transaction with a row lock.
• Every money and check request carries an idempotency key.
• No personal data in logs, caches or the AI assistant. Mask it everywhere.
• Every permission is checked on the server, never only in the frontend.
• Every agency call has a timeout, retries with backoff and a circuit breaker.
• Every sensitive action is written to the audit log.
• No change ships without passing the acceptance checks.

Database

PostgreSQL. These are the test platform's tables today, from db/migrations/001 to 004. Tables marked new in the Python backend do not exist yet; section 3 explains them.

A --< B   one A has many B        A --- B   one to one

orgs --< orgs                     (parent_id: platform > reseller > client)
orgs --< users --< sessions, otp_challenges, login_tickets,
                   recovery_codes, known_devices, notifications
orgs --- wallets --< ledger_entries          (today: one signed entry per wallet)
orgs --< prices >-- products                 (seller_id, client_id, product_code)
orgs --< checks >-- products
orgs --< bulk_jobs --< bulk_rows >-- checks
orgs --< transfers                           (from_org_id, to_org_id)
orgs --< topup_requests                      (org_id, approver_org_id)
orgs --< api_keys
orgs --< audit_log                           (org_id, actor_org_id, actor_id)

New in the Python backend:
orgs --- ledger_accounts --< journal_lines >-- journal --> check, top-up, transfer
ledger_accounts --< holds        ledger_accounts --< trial_balances
outbox, idempotency_keys, daily_usage

accounts

ledger

pricing

checks

topups and transfers

api_access, notifications, audit, reports

A simple summary

• One backend service, split into clear modules (a modular monolith). Split a module out only when it has a measured reason.
• One public API, with an endpoint per product: POST /v1/checks/{product} and POST /v1/checks/bulk.
• Money is an append-only, double-entry ledger. Every journal sums to zero. Every charge happens in one database transaction with a row lock. Real agency calls use hold, then settle or release.
• Slow work goes to background workers (bulk checks, webhooks, emails, SMS), through a queue.
• Every agency sits behind its own adapter with timeouts, retries, a circuit breaker and its own limits, so one slow agency cannot take the others down.
• Stateless app servers, several copies behind a load balancer, on a managed PostgreSQL with a standby.
• The 123 smoke checks of the test platform are the acceptance test for the new backend.

1. Service layout

Modular monolith first

One FastAPI application, one database, one deploy. Inside it, each module owns its tables and exposes plain Python functions to the others. Modules never read each other's tables directly.

                  Website and dashboard (Next.js)
                               |
                         HTTPS, JSON
                               |
                  Load balancer (nginx or cloud LB)
                     |                       |
               API replica 1           API replica 2      (FastAPI, stateless)
                     \                       /
                      \                     /
   PostgreSQL primary --- standby      Redis (queue, rate limits, short cache)
                                              |
                                      Worker processes
                              (bulk checks, webhooks, email, SMS)
                                              |
                          Agency adapters: Gava Connect, KRA, BRS, NTSA

When to split a module out

Split only when one of these is true and measured:

Until then, keep one deploy. Splitting early costs more in network calls, deploys and debugging than it saves.

2. API design

One API or one per product

One API, with an endpoint per product.

POST /v1/checks/kyc-serial      body: { "id_number": "...", "serial_number": "..." }
POST /v1/checks/kyc-phone       body: { "id_number": "...", "phone": "..." }
POST /v1/checks/brs-business    body: { "registration_number": "..." }
POST /v1/checks/ntsa-vehicle    body: { "plate": "..." }
POST /v1/checks/kra-pin         body: { "pin": "..." }
POST /v1/checks/bulk            body: { "product": "...", "rows": [ ... ] }
GET  /v1/checks/{id}            one result
GET  /v1/checks/bulk/{job_id}   bulk progress and results
GET  /v1/balance

Why:

• One key, one sign-in, one contract for the bank. A Sacco integrates once and switches products by changing the path. Separate APIs per product would mean separate keys, docs and error formats.
• Each product still has its own clear input. A path per product gives a precise request schema per product in OpenAPI, which is easier to validate and document than one generic body.
• Shared rules stay in one place. Pricing, the ledger charge, masking, audit, rate limits and idempotency are the same for every product. One API means one implementation.
• Per-product control is still possible. API key scopes are per product (checks:kyc-serial, checks:kra-pin), prices are per product, and rate limits and circuit breakers are per agency.
• New products are a new path, not a new service. Adding another check or product is one adapter, one schema and one price.

Versioning

• The version is in the path: /v1/.... Add fields freely; never remove or rename a field within a version.
• A breaking change means /v2/..., run side by side with /v1 for at least 6 months, with the date announced to API clients.

Idempotency

Every request that charges money or starts a check takes an Idempotency-Key header (a UUID the client makes).

• The server stores the key, the request hash and the response for 24 hours.
• The same key with the same body returns the stored response and charges nothing again.
• The same key with a different body returns 409 idempotency_conflict.
• The dashboard sends one automatically for each Run, Transfer, Top-up decision and Bulk start.

Pagination

Cursor-based, newest first: GET /v1/checks?limit=50&cursor=... returns { "data": [...], "next_cursor": "..." }. Cursors are opaque (an encoded created_at and id). No page numbers on large tables.

Error format

Every error has the same shape, with a stable code the client can switch on:

{ "error": { "code": "insufficient_credit", "message": "Not enough credit for this check.", "request_id": "req_8f2c..." } }

Contract first

1. Write or change the OpenAPI file in openapi/ in a pull request.
2. Review it with the frontend.
3. Generate the TypeScript client for the dashboard from it.
4. Build the endpoint. A CI step fails if the running API and the file differ.

3. Data consistency

The ledger

The Python backend keeps the books in true double entry. Every shilling that moves leaves the books balanced.

What the test platform does today. Each organisation has one row in wallets and a list of signed rows in ledger_entries. The rows are append-only, a trigger ties each balance to its latest entry, and a balance can never go below zero. That is safe, but it is single entry. A transfer writes two rows that share only a ref_id. A top-up issued by the platform and a direct client's check charge have no other side at all, so the whole book does not sum to zero and there is nowhere to see DataCheck's revenue or the credit issued. The Python backend replaces this with the design below.

Accounts. Every wallet and every system account is a row in ledger_accounts.

Journals. One row in journal per business transaction. It records the kind, the idempotency key, the actor (user or API key, and their organisation), a reason, and the source row (source_type, source_id). Its journal_lines hold the debits and credits, in integer cents. A wallet balance is credits minus debits.

The margin is lines inside the check charge journal, not a journal of its own. A charge and its margin can never be posted apart.

Schema.

create table ledger_accounts (
  id bigserial primary key,
  code text not null unique,
  kind text not null check (kind in ('wallet', 'system')),
  org_id uuid unique references orgs(id),
  name text not null,
  balance_cents bigint not null default 0,   -- cache of credits minus debits
  version bigint not null default 0,         -- goes up with every line, for cache keys
  check ((kind = 'wallet') = (org_id is not null)),
  check (kind <> 'wallet' or balance_cents >= 0)
);

create table journal (
  id bigserial primary key,
  kind text not null check (kind in ('topup', 'transfer', 'check_charge', 'refund', 'adjustment', 'reversal')),
  idempotency_key text not null,
  actor_org_id uuid not null references orgs(id),
  actor_user_id uuid references users(id),
  actor_api_key_id uuid references api_keys(id),
  reason text not null default '',
  source_type text not null,                 -- 'check', 'topup_request', 'transfer', 'adjustment'
  source_id text not null,
  reverses_id bigint unique references journal(id),
  posted_at timestamptz not null default now(),
  unique (actor_org_id, idempotency_key)
);
create index journal_source on journal (source_type, source_id);

create table journal_lines (
  id bigserial primary key,
  journal_id bigint not null references journal(id),
  account_id bigint not null references ledger_accounts(id),
  debit_cents bigint not null default 0 check (debit_cents >= 0),
  credit_cents bigint not null default 0 check (credit_cents >= 0),
  balance_after_cents bigint not null,       -- set by the trigger below
  check ((debit_cents = 0) <> (credit_cents = 0))
);
create index lines_account on journal_lines (account_id, id desc);
create index lines_journal on journal_lines (journal_id);

Rules the database enforces.

• Every journal sums to zero. A deferred constraint trigger runs at commit, after all lines are in. A journal that does not balance, or has fewer than two lines, rolls back the whole transaction.

create function journal_must_balance() returns trigger language plpgsql as $$
declare
  off bigint;
  n int;
begin
  select coalesce(sum(debit_cents - credit_cents), 0), count(*) into off, n
    from journal_lines where journal_id = new.journal_id;
  if off <> 0 or n < 2 then
    raise exception 'journal % does not balance (off by % cents, % lines)', new.journal_id, off, n;
  end if;
  return null;
end $$;

create constraint trigger journal_balances after insert on journal_lines
  deferrable initially deferred for each row execute function journal_must_balance();

• The cached balance moves with its lines. A before insert trigger on journal_lines adds the line to ledger_accounts.balance_cents, bumps version and writes balance_after_cents on the line, in the same transaction. A second trigger refuses any other change to balance_cents. The wallet check constraint refuses a negative balance.
• Lock in a fixed order. The service locks the accounts of a journal with SELECT ... FOR UPDATE in sorted id order before it inserts lines, so two transfers cannot deadlock.
• Idempotent. The unique (actor_org_id, idempotency_key) means a retried request finds its journal instead of posting a second one.
• Immutable. journal and journal_lines refuse update and delete (the same forbid_change trigger as today), and the app's database role has no update or delete grant on them. A mistake is fixed with a reversing journal (reverses_id set, lines swapped), then a new correct journal. Nothing is ever edited in place.
• Integer cents only. Never floats.

Holds are not journal lines, because no money has moved yet. A hold row lowers the available balance (balance_cents minus open holds). Settle posts the check charge journal and closes the hold; release closes the hold and posts nothing.

Audit trail

Any shilling can be traced end to end:

1. Account to lines. A wallet's journal_lines, newest first, each with its balance_after_cents.
2. Line to journal. The journal says what happened, when, why, and who did it.
3. Journal to source. source_type and source_id point to the check, top-up request, transfer or adjustment.
4. Source to actor. The audit_log row written in the same transaction carries detail.journal_id, the person or API key, their organisation and a hashed IP.

Checks that keep the books honest:

• Daily trial balance. A worker job at 00:30 Nairobi time sums every account's debits and credits for the day before. All accounts together must sum to zero. The totals and closing balance per account go into trial_balances. Any difference pages on-call.
• Reconciliation. The same job compares every ledger_accounts.balance_cents with the sum of its lines. Any difference pages on-call and freezes that wallet until finance clears it.
• Source checks. Every charged check, approved top-up and transfer has exactly one journal, and every journal has a source row.
• Statements. Any organisation can download a statement for a period: opening balance, every line with its journal kind, reason and counterparty, and closing balance. The opening balance plus the lines must equal the closing balance.
• Top-up clearing. Its balance is credit issued but not yet matched to money received. Finance clears it against bank and M-Pesa statements.

Example queries for running a trail:

-- 1. A wallet's statement for October, with a running balance
select l.id, j.posted_at, j.kind, j.reason, l.debit_cents, l.credit_cents, l.balance_after_cents
  from journal_lines l
  join journal j on j.id = l.journal_id
  join ledger_accounts a on a.id = l.account_id
 where a.org_id = :org_id
   and j.posted_at >= '2026-10-01' and j.posted_at < '2026-11-01'
 order by l.id;

-- 2. One journal: every line, the source and who did it
select j.id, j.kind, j.posted_at, j.source_type, j.source_id, j.reason,
       a.code as account, l.debit_cents, l.credit_cents,
       al.actor_name, al.actor_org_id, al.action, al.at
  from journal j
  join journal_lines l on l.journal_id = j.id
  join ledger_accounts a on a.id = l.account_id
  left join audit_log al on (al.detail->>'journal_id')::bigint = j.id
 where j.id = :journal_id;

-- 3. Everything that happened to one check: the charge and any refund
select j.id, j.kind, j.posted_at, j.reverses_id
  from journal j
 where (j.source_type = 'check' and j.source_id = :check_id)
    or j.reverses_id in (select id from journal where source_type = 'check' and source_id = :check_id)
 order by j.id;

-- 4. Trial balance: must return 0
select sum(debit_cents) - sum(credit_cents) as off_by from journal_lines;

-- 5. Journals that do not balance: must return no rows
select journal_id, sum(debit_cents - credit_cents) as off_by
  from journal_lines group by journal_id having sum(debit_cents - credit_cents) <> 0;

-- 6. Reconciliation: cached balance against the sum of lines, must return no rows
select a.code, a.balance_cents, coalesce(sum(l.credit_cents - l.debit_cents), 0) as from_lines
  from ledger_accounts a left join journal_lines l on l.account_id = a.id
 group by a.id
having a.balance_cents <> coalesce(sum(l.credit_cents - l.debit_cents), 0);

-- 7. Charged checks with no journal: must return no rows
select c.id from checks c
 where c.status = 'completed' and c.price_cents > 0
   and not exists (select 1 from journal j where j.source_type = 'check' and j.source_id = c.id::text);

Moving from the test platform. Port ledger_entries into journals, one per business transaction (grouped by ref_type and ref_id). Add the side that is missing today: topup_clearing for platform-issued top-ups, bidbox_revenue for check charges, adjustments for hand adjustments. Then every wallet balance must match the old one to the cent, and the trial balance must be zero, before the old tables go read-only.

Exactly-once charging for real agency calls

Sample checks charge and answer in one transaction. Real agency calls take seconds and can fail, so they use three steps:

• The hold carries the idempotency key, so a retry finds the existing hold instead of making a new one.
• A sweeper job releases holds older than 10 minutes that never settled, and logs them for review.

The outbox pattern

Never call an outside system (webhook, email, SMS) inside a database transaction, and never lose the message if the process dies.

1. In the same transaction as the business change, insert a row into outbox (topic, payload, created_at).
2. A worker reads unsent rows (FOR UPDATE SKIP LOCKED), sends them, and marks them sent.
3. Failures retry with backoff; after 10 tries the row is marked failed and shows on the admin page.

Webhooks to banks are delivered at least once. Each carries an event_id so the receiver can drop duplicates.

Other rules

• One request, one transaction, unless the work has outside calls (then hold, call, settle).
• Read your own writes: after a change, read from the primary, not the standby.
• Background jobs are idempotent: running one twice gives the same result.

4. High availability

Zero-downtime migrations

Use expand, then contract:

1. Expand: add new columns or tables, nullable or with defaults. Deploy code that writes both old and new.
2. Backfill in batches.
3. Switch reads to the new shape.
4. Contract in a later release: drop the old column.

Never rename or drop a column in the same release as the code change. Create indexes with CREATE INDEX CONCURRENTLY. Alembic migrations run before the new code goes live.

5. Performance

• Indexes. Every list screen has a matching index (for example checks (org_id, created_at desc)). Check slow queries weekly with pg_stat_statements.
• Connection pooling. PgBouncer in transaction mode in front of Postgres. Each API replica keeps a small pool (10 to 20).
• Rate limits in Redis, per API key, per user and per IP, with Retry-After.
• Background workers. Redis plus a job library (Arq fits FastAPI's async style; RQ or Celery also work). Bulk checks, webhooks, emails, SMS, statement exports and report rollups run there, never in the request.
• Bulk checks. Split a job into rows; each row is its own small job with its own hold and settle. Limit concurrency per agency so bulk never starves single checks.
• Reports. Pre-compute daily totals per organisation and product in a daily_usage table; charts read from it, not from raw checks.
• Payloads. Paginate everything; compress responses (gzip at the load balancer).

Caching

A TTL (time to live) is a timer on a cached item: when it runs out, the item is thrown away and the next read goes to the database.

• Never cached: balances, holds and transfers. Always read them from the Postgres primary. A stale balance could let someone spend credit they do not have, or make a transfer look lost.
• Transaction history and statements may be cached, with write-through invalidation. The cache key includes the account's version, for example history:<account id>:v<version>:<cursor>. The trigger that posts a line bumps version in the same transaction, so the next read builds a new key and the old one is never read again. An outbox event ledger.posted then deletes the old keys to free memory. Reading version is one primary-key lookup in Postgres.
• Closed periods are cached until changed. Once a month's trial balance has passed, its statement does not change, because corrections are new reversing journals dated today. Cache it with no expiry. The key includes the month and the account's version at close.
• Reference data (product list, base prices, permission tables, report totals for past days) is cached and invalidated the same way when it changes.
• TTL is only a safety net. Every key gets a long TTL (24 hours) so forgotten keys are cleaned up and a missed invalidation cannot last forever. Freshness never depends on it.
• No personal data in the cache. Cached history holds masked inputs only. Full results are never cached.
• If Redis is down, read from Postgres. Nothing breaks, it is only slower.

6. Resilience with agencies

Each agency (Gava Connect, KRA, BRS, NTSA) gets its own adapter with:

Graceful degradation:

• If one agency is down, only its products show "Temporarily unavailable" on the dashboard and in the API. The rest keep working.
• Bulk jobs pause the affected rows and resume when the circuit closes.
• A status page lists each agency's state.

7. Observability

• Structured logs in JSON: timestamp, level, request_id, org_id, user_id or api_key_id, route, status, duration_ms. Never log ID numbers, phone numbers, names or results.
• Metrics (Prometheus or the cloud's equivalent): request rate, error rate and latency per route; checks per product and outcome; agency latency and errors; queue depth; ledger holds open.
• Tracing with OpenTelemetry, from the load balancer through the API, the database and the agency call, with request_id as the link.
• Alerts go to the on-call phone, not only email.

8. Security

• OWASP Top 10 as the review checklist for every pull request that touches an endpoint.
• Secrets in the platform's secret store or environment, never in the repo. Rotate agency credentials and signing keys yearly and when someone leaves.
• Encryption in transit (TLS 1.2 or later everywhere, including to Postgres and Redis) and at rest (disk encryption on the database and backups). Authenticator secrets and webhook secrets are also encrypted in the database.
• Personal data. Mask ID, serial and phone numbers in history, logs and the assistant. Full results are shown once to the person who ran the check, then only the masked copy is kept.
• Audit. Every sensitive action is written to the append-only audit log in the same transaction as the change.
• Retention. Full results: delete after 30 days (draft, confirm with each agency's terms). Masked check records and the audit log: 7 years. A nightly job enforces it.
• Access. Every endpoint declares its permission; a test fails if one does not. Least privilege for staff database access; production access is logged.
• Compliance. Each organisation accepts the lawful-purpose and consent declaration before running checks; the acceptance is stored with who and when.

9. Testing strategy

10. Environments and CI/CD

Pipeline on every push: lint (ruff), type check (mypy), unit and integration tests, OpenAPI diff, build image, acceptance smoke against a throwaway stack. On main: deploy to test automatically. Production: a tagged release, manual approval, rolling deploy, migrations first.

11. Python service conventions

bidbox-api/
  app/
    main.py                FastAPI app, routers mounted
    core/                  settings, db session, security, errors, logging
    modules/
      accounts/            router.py, service.py, models.py, schemas.py, tests/
      ledger/
      pricing/
      checks/
      agencies/            one file per agency adapter
      topups/
      transfers/
      api_access/
      notifications/
      audit/
      reports/
  workers/                 job definitions (bulk, outbox, notifications, retention)
  migrations/              Alembic
  openapi/                 the contract
  tests/acceptance/        the smoke script

• Layers: router (HTTP only) calls service (rules and transactions) which uses models (SQLAlchemy). Routers never touch the database directly.
• Schemas: Pydantic models for every request and response; no raw dicts across module boundaries.
• Money: int cents, named *_cents.
• Time: store UTC, show Nairobi time.
• Style: ruff and black, mypy strict on service.py files, type hints everywhere.
• Reviews: every pull request has a test; money and permission changes need two approvals.

12. Things to do

Pick from the top. Each line maps to a module from the module list.