Create scrapers and inspect executions
Scratch
Web-scraping operations for clean, trustworthy data
Scratch turns fragile scraping scripts into observable runs, structured outputs, and an operating base for datasets that can be sold, monitored, and trusted.
RustAPI and workers
NATSdurable scrape flow
Postgresexecution history
Execution pipelineobservable
DashboardAPINATSWorkerData
run queued scrape request
log 24 items extracted
save persisted with provenance
The problem
Scraping is easy. Trustworthy data operations are not.
One-off scripts can fetch a page. The harder part is knowing what ran, what failed, what changed, and whether the resulting data is fresh enough to power a product.
Cron jobs hide failure states until a downstream user notices.
Raw files and logs make provenance expensive to recover.
Schema drift, duplicates, and stale captures erode confidence.
Proxy and browser behavior needs policy, observability, and review.
Platform
The pieces that make scraping operational.
Scratch separates the interface, control plane, workers, data store, proxy policy, and CI runner so each part stays explicit.
scratch-ui
Operations dashboard
A Next.js console for managing scrapers, reviewing executions, and inspecting structured outputs without tailing logs.
scratch-api
Rust control plane
An Axum API backed by PostgreSQL for scraper definitions, queued runs, health checks, and persisted execution state.
scratch-scraper
Worker runtime
A standalone Rust scraping worker driven by NATS JetStream so scrape execution is decoupled from the API process.
scratch-residential
Opt-in network
An opt-in residential client and gateway designed for consent-based routing, quotas, allowlists, and kill-switch behavior.
Dagger CI
Validation pipeline
A Dagger-based CI runner keeps API, worker, residential proxy, frontend, and end-to-end checks explicit.
Workflow
From scrape request to data product foundation.
Every run should leave behind usable state: what was requested, what happened, what changed, and what data can move downstream.
- 01
Define
Create scraper definitions with target URLs, extraction rules, and the constraints needed to keep runs reviewable.
- 02
Run
Queue executions through the API and let NATS JetStream distribute work to scraper workers.
- 03
Observe
Track status, errors, logs, and item counts from the dashboard instead of rebuilding state from scattered output.
- 04
Package
Persist structured results in PostgreSQL as the foundation for normalized, deduplicated, and freshness-aware datasets.
Architecture
A decoupled system for visible execution.
The API controls state, JetStream carries work, workers perform extraction, and PostgreSQL keeps the operational record.
Persist definitions and publish requests
Buffer requests and responses
Fetch pages and extract structured items
Store history, logs, and data
Apply opt-in residential routing policy
Reliability
Built for reviewable data collection.
Scratch treats scraping as infrastructure: failures are visible, network paths are policy-bound, and validation runs across the actual stack.
Visible failures
Executions carry status, timestamps, logs, and item counts so a failed scrape becomes an operational event, not a mystery.
Policy-aware networking
The residential path is explicit opt-in residential infrastructure with allowlists, quotas, and a kill switch.
Local parity
PostgreSQL, NATS JetStream, API, worker, and frontend run locally so development and CI exercise the real flow.
Reviewable extraction
Scratch favors observable extraction quality and reviewable changes over automatic behavior that hides scraper drift.
Build the data operations layer behind your scrapers.
Scratch is early, technical, and built in the open. Clone it, run the stack, and help shape the platform for clean aggregated data.