Use Scheduled Jobs with Manual Upload for Data Ingestion Over Live Partner APIs

Historical decision text is preserved. Accepted describes the decision, not deployment of every capability discussed. Consult the Architecture and Infrastructure tabs for the verified running system.

Successors or amendments: ADR-018

Date: 2026-05-10
Status: Superseded by ADR-018 (2026-10-20)
Deciders: Daniel Meza

Context and Problem Statement

Urza's Wallet must display current financial product data: credit card interest rates, pension fund historical returns, and home loan rates. This data changes periodically (daily to monthly, depending on product type) and must be sourced from Costa Rican banks and financial operadoras.

The NerdWallet reference architecture uses a dedicated Offer Ingestion Service that "polls partner feeds and APIs on a scheduled basis, validates freshness, transforms raw data into canonical schema." This presupposes that partner institutions expose structured data feeds (REST APIs, SFTP (SSH File Transfer Protocol) CSV files, or XML feeds). The question is whether this assumption holds for the Costa Rican financial market in 2026.

We need a data ingestion strategy that is:

1. Practical given the actual state of data availability from CR banks and operadoras.
2. Reliable — stale or missing data must be detected and surfaced to operators.
3. Safe — a bad ingestion run should not corrupt existing product data.
4. Extensible — the strategy must accommodate structured API feeds if banks begin publishing them.
5. Low-maintenance — a small team cannot maintain dozens of custom scraper adapters continuously.

Decision Drivers

• Costa Rican bank API reality: As of 2026, no major Costa Rican bank (BAC, BCR, BN, Davivienda, Scotiabank, Promerica, BPDC) publishes a public machine-readable API for product rates and fees. SUGEF mandates hojas de tarifas (rate sheets) as PDF publications, not as structured data feeds. This is fundamentally different from the US market where NerdWallet partners with banks that expose standardized SFTP or REST feeds.
• SUPEN and SUGEVAL regulatory publications: Pension fund returns are published quarterly by the operadoras to SUPEN (Superintendencia de Pensiones). SUPEN's website exposes downloadable Excel/PDF reports, not an API. The data is public but unstructured.
• Manual upload as editorial control: Financial data accuracy is a trust signal. Editors should be able to override scraped data (e.g., when a bank updates rates mid-cycle and the scraper hasn't run) without waiting for the next scheduled ingestion.
• Canonical schema isolation: The NerdWallet ADR-003 principle applies: "The Offer Ingestion Service is the sole owner of this mapping logic." Regardless of whether data arrives via scraper, PDF parser, or manual upload, it must pass through the same canonical mapper before reaching the product database.
• Freshness requirements: Credit card rates are typically published monthly. Pension fund returns are published quarterly. Home loan rates can change weekly. The ingestion schedule must be configurable per source, not a single global cadence.
• Cost of web scraping at scale: Maintaining 20+ scraper adapters is a significant engineering investment. A scraper breaks every time a bank updates its website layout. Mitigation: the scraper adapter model uses CSS (Cascading Style Sheets) selector configurations stored as JSON (not hardcoded selectors), so selector updates don't require code deployments.

Considered Options

1. Live API calls to partner systems at render time — Call bank APIs directly when a user requests a comparison page.
2. Partner-push model (webhooks) — Banks send data updates to Urza's Wallet via webhooks when rates change.
3. SFTP/batch file ingestion — Banks deposit standardized CSV/XML files on an SFTP server on a schedule.
4. Scheduled web scraping + PDF parsing + manual upload (hybrid) — A scheduled Railway cron job runs source adapters per vertical; editors can override via CSV/JSON upload.
5. Manual entry only (admin UI forms) — All product data entered manually by editors, no automation.

Decision Outcome

Option 4: Scheduled web scraping + PDF parsing + manual upload (hybrid) is adopted.

The three source types and their trust hierarchy:

Trust priority means: If a manual upload and a scraper run both update the same product within the same ingestion cycle, the manual upload takes precedence. The source_type field on the ingestion_runs log records which source produced the current product data.

Ingestion pipeline design:

1. All three source adapters output a validated raw source object (the same interface).
2. The Schema Validator (zod) checks the object against the vertical's canonical field list.
3. The Canonical Mapper normalizes values: fees to CRC, rates to annual percentage, issuer names to canonical issuer_id.
4. The Canonical Mapper upserts the product row using INSERT ... ON CONFLICT (id) DO UPDATE.
5. The Freshness Tracker updates updated_at on success; logs the run in ingestion_runs.
6. On success, the ingestion service calls POST /api/revalidate to trigger on-demand ISR (Incremental Static Regeneration) invalidation.

Scraper configuration: Each source adapter reads its CSS selectors and target URLs from a JSON configuration file (config/scrapers/{issuer-slug}.json). Updating a selector requires only a config file change (no code change), which can be deployed without a full service rebuild.

Failure handling:

• A failed scraper run increments ingestion_failure_count on the affected products without updating updated_at. After 3 consecutive failures, the is_stale flag is set and the product page displays a "Datos pendientes de actualización" notice to users.
• All failures are logged to ingestion_runs with error_summary. Editors can see the failure log in the admin UI.
• The ingestion service never deletes product records — it only upserts. A bad scraper run cannot remove a product from the platform.

Path to structured API integration: When a CR bank publishes a structured API (REST or SFTP), adding support requires:

1. A new source adapter implementing the same SourceAdapter interface.
2. A new entry in the ingestion scheduler configuration.
3. No changes to the Canonical Mapper, Schema Validator, or database.

The canonical schema is the integration point, not the source adapters. This is the key principle from the NerdWallet ADR-003: "Partner API changes are isolated to the ingestion service; downstream consumers are unaffected."

Positive Consequences

• Practical for the CR market: Works with the actual data availability reality (PDFs and web pages) rather than requiring banks to build APIs that don't exist.
• Editorial control: Manual uploads give editors a reliable override mechanism when scrapers fail or when a bank updates rates out of cycle.
• Canonical schema isolation: All downstream consumers (API, frontend, comparison engine) work against a single, validated schema regardless of source type.
• Resilient to scraper failures: is_stale flag and ingestion_failure_count surface data quality issues to users and editors without corrupting the existing product data.
• Extensible to APIs: The SourceAdapter interface makes adding a structured API adapter a self-contained change.
• Kleppmann alignment: DDIA Ch. 11 notes that "derived data" (the canonical product records) should be separated from source data (scraped raw objects). The ingestion pipeline is a one-way transformation: source → validated → canonical. This makes the system's data flow explicit and auditable.

Negative Consequences / Trade-offs

• Scraper maintenance burden: Bank websites change. Scraper adapters require maintenance when bank pages update their layout. Mitigation: CSS selector configuration files (not hardcoded), monitoring via ingestion_failure_count, and an admin UI that alerts editors when a scraper has failed 3 consecutive times.
• Ingestion lag: There is a delay between a bank publishing a rate change and Urza's Wallet reflecting it. For scheduled scrapers running every 6–24 hours, this lag can be up to 24 hours. For SUGEF mandate publications (hojas de tarifas), the lag is bounded by the upload schedule. This lag is displayed to users via the "última actualización" timestamp on every product.
• Legal risk of web scraping: Scraping bank websites may conflict with their Terms of Service. Mitigation: (1) robots.txt is respected; (2) rate limiting is applied to avoid server stress; (3) only publicly available data (same as what any user can see) is scraped; (4) legal review of scraping legality under Costa Rican law before launch; (5) official partnerships (and API access) are pursued as a business objective.
• No real-time rate updates: If a bank updates mortgage rates in response to a BCCR (Banco Central de Costa Rica) rate decision, Urza's Wallet may show outdated rates for up to 12 hours. This is disclosed to users. Real-time updates require a bank partnership with API access — a Phase 2 commercial objective.

Pros and Cons of the Options

References

• NerdWallet reference architecture — ADR-003 (Canonical Product Schema with Centralized Offer Ingestion)
• Kleppmann, Designing Data-Intensive Applications, Ch. 11 (Stream Processing — derived data, one-way dataflows)
• SUGEF: Reglamento sobre Transparencia de Información de Productos Financieros, which mandates hojas de tarifas publication
• Architecture reference §4c (Data Ingestion Service Component Diagram), §5.2 (Data Flow)