Transaction autocategorization at SPRK

Problem

SPRK builds a personal-finance app for the Spanish market on top of Open Banking (PSD2). Every feature the product sells — spend analysis, subscription detection, fixed-cost forecasting — depends on one data layer: every bank transaction must be correctly categorized. A transaction arrives as short, noisy, bank-mangled text — RECIBO LUZ ENDESA, UBER *EATS MADRID, BIZUM DE JOAN P. — and must leave as a category from a 31-value taxonomy, with a confidence, a human-review flag, and reason codes. Cheaply, fast, and auditably.

Two domain facts shape everything. First, the long tail: a large fraction of volume concentrates in well-known merchants that rules resolve for free, but roughly half the corpus names small local merchants that no general-purpose model knows anything about. Second, PSD2 standardizes access to bank data, not its semantics: measurement per issuer showed the ISO 20022 transaction codes are nearly constant per bank — some issuers stamp the same “transfer” family on card purchases, Bizums, and true transfers alike. The engineering answer was a canonical two-level transaction-type contract with first-class UNKNOWN (never an error, never a null), per-issuer decoders, and a rule that direction derives from the amount’s sign, never from a code that can’t be trusted.

The business north star was explicit: maximize the auto-categorization rate while minimizing marginal LLM cost. Every change was measured against it.

I joined as the sole engineer on the AI service — the inherited prototype was an 830-line monolithic orchestrator chaining 12 decision layers, including an XGBoost model and a three-provider LLM consensus cascade. Six months later the repository held 1,517 commits (98.2% mine) and no module of the original had survived unrewritten.

Approach

Rule-first cascade, 11 steps. Deterministic layers always run before any model: user overrides (absolute), structural signals, global learned rules, a merchant knowledge base, transfer descriptors, MCC ranges, keywords — then embeddings, then exactly one grounded LLM call, then an honest fallback to human review with confidence 0. The rules live as data, not code: 458 canonical merchants, 40 knowledge-base profiles, 36 descriptor groups and 97 MCC ranges in versioned YAML, validated at import time — an orphaned alias fails startup, not a production categorization.

One grounded LLM call, with the right to abstain. When no rule resolves, a single model call receives the sanitized descriptor, the transaction signals, and — when it exists — the merchant’s knowledge-base profile. The model may answer unknown_merchant instead of being forced to guess; on provider failure or low confidence the system fails fast to review, never blocks, never invents. The call sits behind circuit breakers, a Redis result cache, and a per-call cost ledger.

That design is the result of two argued reversals of the inherited system, both recorded as RFCs. The multi-LLM consensus cascade died in July: model agreement is only evidence when errors are independent, and on the long tail they are not — general-purpose models share exactly the same gap in world knowledge, so unanimity among ignorant voters isn’t knowledge, and the arbiter can’t tell which voter is the ignorant one. The online XGBoost layer died the same month, by arithmetic: its documented activation threshold was on the order of 100,000 labeled transactions; the real corpus held ~7,500. Training moved to an offline library, ready to reactivate when the data volume justifies it.

Engineering the whole system, not just the pipeline. Four-package Python monorepo with a strict dependency DAG; infrastructure as code (AWS CDK), ECS Fargate, 27 versioned migrations; hermetic CI (outbound network banned in unit tests, LLM providers doubled) with a merge gate against real PostgreSQL and Redis; ~60k lines of production code held 1:1 with test code under TDD. GDPR shaped the architecture: PII encrypted at rest with key rotation, fingerprint-only logs, per-user isolation enforced in queries and verified against an HMAC-signed claim with row-level security as depth, and erasure implemented as an immediate processing block. Decisions live in a register of 39 RFCs with formal supersession — errors aren’t deleted, they’re documented and superseded.

Result

The result I value most is the measurement machinery, because it produced the period’s most important (and least flattering) number. On a distilled 532-case corpus the pipeline scored 0.93 agreement. On the full judged corpus of 7,584 rows: 0.523. Not a regression — the small corpus giving way to the long tail (fallback volume jumps from 3.6% to 37.8%). Without the full harness, the system would have looked like a 0.93 that didn’t exist.

Step	Volume	Agreement
Merchant knowledge base	2,380	0.816
Structural signals	1,493	0.756
Transfer descriptors	391	0.798
Keywords	358	0.729
Grounded LLM	6	1.000
Fallback (the long tail)	2,865 · 37.8%	0.103

The decomposition of the review load is the actionable finding: 75% of it is absence of merchant knowledge, not model or calibration failure. The ceiling arithmetic assigns ~22 points to a merchant-enrichment loop, ~10 to confidence calibration, ~9 to structurally suppressed personal transfers. Confirmation came from the model itself: given the right to abstain, it abstained on 99.2% of questions (1,611 of 1,624), and enriching the prompt with more transaction context moved that number not at all. The bottleneck is knowledge, not model capacity.

The replay infrastructure met the strictest bar: a live recording of 1,624 model verdicts (62 minutes, zero provider failures) reproduced byte-for-byte identically from fixtures at zero cost — after its first run failed loudly with 73 missing fixtures and was reconciled arithmetically before the gate was trusted. On that foundation, swapping the decision semantics from first-match-wins to evaluate-then-decide (Jeffreys-calibrated confidences) was validated as a 9-rows-in-7,584 change (0.12%), deployed behind a runtime parameter — reversible without a redeploy.

A provider-portability measurement closed the period: the same open-weights model scored 48.2% F1 on a host that silently ignored guided decoding and 59.8% on one with grammar-constrained sampling — same weights, same price. Migrating the serving raised the available ceiling from 53% to the 74–76% range and made an entire production failure class (15.7% of grounded calls lost to token truncation) unrepresentable by construction.