Skip to content
All flows

Survive backend outages with a persistent queue

Replaces the default in-memory exporter queue with a disk-backed queue that survives collector restarts, sized in items instead of opaque requests.

Replaces the default in-memory exporter queue with a disk-backed queue that survives collector restarts, sized in items instead of opaque requests. Retries never expire, so a recovered backend receives everything that was buffered. When the disk queue fills, backpressure reaches the SDKs instead of data being dropped silently.

How the data moves
receiverprocessorexporterextension
traces
otlpmemory_limiterbatchotlp_grpc
metrics
otlpmemory_limiterbatchotlp_grpc
logs
otlpmemory_limiterbatchotlp_grpc

Use this flow

Open the config in Telflo and it becomes a working pipeline on the canvas: adapt what's specific to you, test it against recorded traffic, and push it to your fleet over OpAMP. Free account, no card.

Components

What's in it, and why

otlp

Ingests traces, metrics, and logs over gRPC (4317) and HTTP (4318) so one collector fronts all SDK traffic that needs outage protection.

memory_limiter

Refuses data at the front of the pipeline when heap crosses the limit, which is what turns a full disk queue into backpressure to SDKs instead of an OOM kill.

batch

Groups records into 8192-item batches before export so the persistent queue stores and retries a few large requests instead of thousands of tiny ones.

otlp_grpc

Sends OTLP to the backend with the disk-backed sending_queue and never-expiring retry policy that this flow exists for.

file_storage

bbolt-backed disk persistence that the exporter queue writes through, so queued telemetry survives collector restarts and OOM kills.

health_check

HTTP liveness endpoint on 13133 so Kubernetes can probe the collector without touching the pipeline.

Notes

Gotchas

  • 1

    The file_storage directory must exist and be writable before startup (create_directory: true handles first boot, but on Kubernetes you still need a PVC or hostPath mount; an emptyDir volume is wiped on pod rescheduling, which defeats the whole point).

  • 2

    With max_elapsed_time: 0 nothing is ever aged out of the queue, so a long outage grows the bbolt file until the volume fills; size the volume for your outage budget and alert on the collector's own otelcol_exporter_queue_size metric.

  • 3

    bbolt never returns disk space by itself; without compaction.on_rebound and on_start the database file stays at its post-outage high-water mark forever.

  • 4

    This is backpressure, not zero loss: once the disk queue fills, block_on_overflow and memory_limiter push refusal back to the SDKs, and whether data survives from there depends on SDK-side retry and buffering; the OTel resiliency docs are explicit that these mechanisms reduce rather than eliminate loss.

  • 5

    Use the otlp_grpc exporter id, not otlp: core v0.144.0 renamed the exporter and the old id is a deprecated alias. Also note that setting sending_queue::storage removes the in-memory queue entirely, so queue throughput becomes disk-bound; leave fsync off unless you need integrity over throughput.

Test it before your fleet runs it

Free account, no card. Open this flow in the editor, adapt it, and see what it does to real data before anything ships.