Cloudflare Pipelines Now Available as Logpush Destination
Key Points
- Pipelines as Logpush destination for efficient log storage
- In-flight SQL transformations before R2 storage
- Parquet and Apache Iceberg table support
Summary
Cloudflare Logpush now supports Pipelines as a destination, enabling direct log ingestion, transformation, and storage in R2 as Parquet files or Apache Iceberg tables. This replaces traditional JSON delivery with a more storage-efficient and query-optimized approach.
Key Points
- Send logs directly to Pipelines for in-flight transformation before storage in R2
- Store logs as Parquet files or Apache Iceberg tables managed by R2 Data Catalog
- Query logs instantly using R2 SQL or any Apache Iceberg/Parquet-compatible query engine
- Transform logs using Pipelines SQL with support for string functions, regex, hashing, JSON extraction, timestamp conversion, and conditional expressions
- Example use cases: drop noisy fields, redact sensitive values, derive new columns
- Reduces data footprint and improves query efficiency compared to JSON format