Survive backend outages with a persistent queue
Replaces the default in-memory exporter queue with a disk-backed queue that survives collector restarts, sized in items instead of opaque requests.
Replaces the default in-memory exporter queue with a disk-backed queue that survives collector restarts, sized in items instead of opaque requests. Retries never expire, so a recovered backend receives everything that was buffered. When the disk queue fills, backpressure reaches the SDKs instead of data being dropped silently.
Use this flow
Open the config in Telflo and it becomes a working pipeline on the canvas: adapt what's specific to you, test it against recorded traffic, and push it to your fleet over OpAMP. Free account, no card.
Components
What's in it, and why
Ingests traces, metrics, and logs over gRPC (4317) and HTTP (4318) so one collector fronts all SDK traffic that needs outage protection.
Refuses data at the front of the pipeline when heap crosses the limit, which is what turns a full disk queue into backpressure to SDKs instead of an OOM kill.
Groups records into 8192-item batches before export so the persistent queue stores and retries a few large requests instead of thousands of tiny ones.
Sends OTLP to the backend with the disk-backed sending_queue and never-expiring retry policy that this flow exists for.
bbolt-backed disk persistence that the exporter queue writes through, so queued telemetry survives collector restarts and OOM kills.
HTTP liveness endpoint on 13133 so Kubernetes can probe the collector without touching the pipeline.
Notes
Gotchas
- 1
The file_storage directory must exist and be writable before startup (create_directory: true handles first boot, but on Kubernetes you still need a PVC or hostPath mount; an emptyDir volume is wiped on pod rescheduling, which defeats the whole point).
- 2
With max_elapsed_time: 0 nothing is ever aged out of the queue, so a long outage grows the bbolt file until the volume fills; size the volume for your outage budget and alert on the collector's own otelcol_exporter_queue_size metric.
- 3
bbolt never returns disk space by itself; without compaction.on_rebound and on_start the database file stays at its post-outage high-water mark forever.
- 4
This is backpressure, not zero loss: once the disk queue fills, block_on_overflow and memory_limiter push refusal back to the SDKs, and whether data survives from there depends on SDK-side retry and buffering; the OTel resiliency docs are explicit that these mechanisms reduce rather than eliminate loss.
- 5
Use the otlp_grpc exporter id, not otlp: core v0.144.0 renamed the exporter and the old id is a deprecated alias. Also note that setting sending_queue::storage removes the in-memory queue entirely, so queue throughput becomes disk-bound; leave fsync off unless you need integrity over throughput.
Sources and references (9)ShowHide
Everything consulted while researching and fact-checking this flow, including the README of every component it uses.
- github.com/open-telemetry/opentelemetry-collector/blob/v0.146.0/exporter/exporterhelper/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.146.0/extension/storage/filestorage/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.146.0/processor/memorylimiterprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.146.0/extension/healthcheckextension/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.146.0/receiver/otlpreceiver/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.146.0/exporter/otlpexporter/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.146.0/processor/batchprocessor/README.md
- opentelemetry.io/docs/collector/resiliency/
- opentelemetry.io/docs/collector/internal-telemetry/
More flows
Related flows
Fail over to S3 when ClickHouse backs up
A ClickHouse gateway that fails each signal over to an S3 overflow bucket when inserts back up and fails back on its own once ClickHouse recovers, so an ingest stall never becomes data loss.
Replace vendor agents on your VM fleet
Scrapes CPU, memory, load, disk, filesystem, and network metrics plus system and application logs on every VM.
Redact PII before telemetry leaves the network
Masks emails, card numbers, SSNs, bearer tokens, AWS keys, and IPs in log bodies and span attributes at a central gateway, before telemetry leaves your network.
Test it before your fleet runs it
Free account, no card. Open this flow in the editor, adapt it, and see what it does to real data before anything ships.