Fail over to S3 when ClickHouse backs up
A ClickHouse gateway that fails each signal over to an S3 overflow bucket when inserts back up and fails back on its own once ClickHouse recovers, so an ingest stall never becomes data loss.
The gateway design from ClickHouse's own LogHouse pipeline: ClickHouse stays in the hot path, S3 is a release valve that only opens under backpressure, and a separate catch-up collector replays the overflow from SQS notifications so live traffic never waits behind backlog.
Per signal, a failover connector routes to <signal>/clickhouse first and <signal>/s3 second. The queue placement is the whole trick: the connector owns the sending queue, the clickhouse exporter has no queue but inline retries, and the awss3 exporters queue and retry forever, so an S3 hiccup never looks like a second outage. Each signal gets its own prefix in the bucket, which is what lets the catch-up collector replay one signal at a time.
Set the CLICKHOUSE_* variables plus AWS_REGION and OVERFLOW_S3_BUCKET, and create the bucket with an event notification and SQS queue per prefix. The catch-up collector, an awss3 receiver draining those notifications into the same clickhouse exporter, is a separate deployment and not part of this flow.
Use this flow
Open the config in Telflo and it becomes a working pipeline on the canvas: adapt what's specific to you, test it against recorded traffic, and push it to your fleet over OpAMP. Free account, no card.
Components
What's in it, and why
Entry point for all three signals on gRPC :4317 and HTTP :4318.
First in every ingest pipeline so overload becomes backpressure at the receiver rather than an OOM kill of the gateway.
Builds the large, infrequent inserts ClickHouse wants (10,000 items or 5 seconds) before the data reaches the connector, so the ClickHouse and S3 legs see the same batch shape.
Owns the sending queue for logs and tries logs/clickhouse first; when that exporter reports failure it switches to logs/s3 and probes ClickHouse again every 30 s, failing back as soon as an insert succeeds.
The same two-level failover for traces: traces/clickhouse first, traces/s3 under backpressure, with its own 10,000-item queue.
The same two-level failover for metrics: metrics/clickhouse first, metrics/s3 under backpressure, with its own 10,000-item queue.
Writes the otel_* tables with create_schema off (the tables are managed elsewhere), async inserts, no queue of its own, and a bounded retry (5 s to 30 s, giving up after 300 s) so the connector can fail over instead of buffering forever.
Overflow for logs: OTLP protobuf objects under the logs/ prefix of the bucket, behind a 10,000-item queue and retries that never expire, so S3 flakiness stalls the overflow rather than dropping it.
Overflow for traces under the traces/ prefix, same queue and endless retry.
Overflow for metrics under the metrics/ prefix, same queue and endless retry.
Liveness endpoint on :13133 for the orchestrator.
Notes
Gotchas
- 1
The
failoverconnector only switches levels when the exporter behind a level returns an error, so theclickhouseexporter must be allowed to fail: keep itssending_queueoff and its retry bounded (max_elapsed_time: 300s), because a queued or endlessly retrying exporter never hands back the error the connector is waiting for. - 2
Fail-back is automatic: every
retry_interval(30 s) the connector offers the next batch to the higher level and stays there if it succeeds. Nothing replays what already went to S3, which is why the catch-up collector exists. - 3
The connector's current semantics date from contrib v0.157.0, which dropped
retry_gapandmax_retries(a level used to be abandoned after a fixed number of retries); this config validates on older releases but fails over differently there, so keep the 0.159.0 pin. - 4
Overflow objects are OTLP protobuf (
marshaler: otlp_proto), one per flush, partitioned by minute underlogs/,traces/, andmetrics/; the catch-up collector'sawss3receiver needs the same prefix layout, and the bucket's event notifications need one SQS queue per prefix. - 5
The
awss3exporters retry forever, so if S3 is also unreachable the connector's 10,000-item in-memory queue fills and backpressure reaches the receiver; that is the intended failure mode, since the alternative is silent loss, but sizequeue_sizefor the outage you actually want to absorb. - 6
wait_for_async_insert: '1'makes each insert block until ClickHouse has durably buffered it, so a stalled ClickHouse shows up as a slow insert, then a 5 s timeout, then retries, and only after 300 s as the failure that trips the failover; expect up to that long before S3 takes over. - 7
The
clickhouseexporter runs withcreate_schema: false: theotel_*tables must already exist (ClickStack or the Ship OTel data to ClickHouse flow creates them), and the 720httlonly applies to tables the exporter creates itself. - 8
A full outage writes every signal to S3 at line rate and replaying it later re-inserts all of it, so the catch-up collector's insert rate, not the outage length, decides how long ClickHouse runs behind after recovery.
Sources and references (12)ShowHide
Everything consulted while researching and fact-checking this flow, including the README of every component it uses.
- clickhouse.com/blog/reliable-opentelemetry-ingestion-at-scale
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/connector/failoverconnector/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/exporter/awss3exporter/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/receiver/awss3receiver/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/exporter/clickhouseexporter/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/CHANGELOG.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/exporter/exporterhelper/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/processor/memorylimiterprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/processor/batchprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/receiver/otlpreceiver/README.md
- clickhouse.com/docs/optimize/asynchronous-inserts
- clickhouse.com/docs/use-cases/observability/clickstack/ingesting-data/schemas
More flows
Related flows
Survive backend outages with a persistent queue
Replaces the default in-memory exporter queue with a disk-backed queue that survives collector restarts, sized in items instead of opaque requests.
Replace vendor agents on your VM fleet
Scrapes CPU, memory, load, disk, filesystem, and network metrics plus system and application logs on every VM.
Redact PII before telemetry leaves the network
Masks emails, card numbers, SSNs, bearer tokens, AWS keys, and IPs in log bodies and span attributes at a central gateway, before telemetry leaves your network.
Test it before your fleet runs it
Free account, no card. Open this flow in the editor, adapt it, and see what it does to real data before anything ships.