Cloudflare Logpush Transformers run SQL on log batches with embedded DataFusion
- Sources: primary
- Summary: Transformers are SQL queries over a Logpush dataset, versioned and attachable to NDJSON jobs. Cloudflare embedded Apache DataFusion in its Go delivery service via cgo and Arrow IPC instead of calling Workers, avoiding a network hop. It reports about 3 ms per 5,000-row batch on one goroutine.
- Why it matters: Teams can cut downstream ingest cost and strip sensitive fields at the source without running a separate log-processing tier.