ETL pipelines, data integrations and connectors for sources without an API

Data that arrives on time, in one shape, from sources that were never designed to cooperate.

WHAT YOU GET

- Connectors and ingestion: APIs, files, feeds, databases, and sources with no public API at all (browser-based extraction where it is permitted)
- Normalisation: one schema out of many, deduplication, validation, and quarantine for records that do not fit
- Scheduling and reliability: incremental loads, backfills, backoff and retries, idempotent runs you can safely re-execute
- LLM processing layers: extraction, classification and enrichment on top of raw records, with evaluation rather than guesswork
- Monitoring: freshness and volume checks, tracing, alerts before your dashboard quietly goes stale

STACK

Go, Python, Ruby on Rails, Node.js. PostgreSQL, Kafka, AWS Lambda and S3, Airflow-style orchestration, OpenTelemetry.

BACKGROUND

I spent four years building a petabyte-scale ETL platform with hundreds of connectors and LLM layers, and I run my own production pipeline for sources without an API.

HOW WE WORK

I start with the smallest pipeline that produces correct data, then harden it. Weekly written updates, tested code, and a runbook so your team can operate it without me.