Outage-Proof Pipeline
Replaces the default in-memory exporter queue with a disk-backed queue that survives collector restarts, sized in items instead of opaque requests.
By default the collector's sending queue lives in memory and is sized in requests, which means a restart loses whatever was buffered and the queue size doesn't map to anything you can reason about. This flow points the queue at a file_storage extension so it survives restarts and OOM kills, sizes it in items so the number means something, and extends retry so a backend outage measured in minutes doesn't turn into dropped data. It is not zero-loss, and the config says so: if the disk fills or the volume dies, the buffered data is gone. Size the volume against your throughput and outage tolerance before deploying this — a persistent queue with nowhere to persist is just a slower in-memory queue.
Before you use this
Sends data to
You'll need to set these before it runs
BACKEND_API_KEYBACKEND_OTLP_ENDPOINTThe configuration
# Outage-Proof Pipeline: disk-backed exporter queue + never-expiring retries.
# Survives backend outages, collector restarts, and OOM kills. Not zero-loss:
# if the disk fills or the volume dies, data is lost. Size the volume first.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
# First in every pipeline (per README) so refusal reaches the receiver and
# becomes backpressure to SDKs instead of an OOM kill. Pair with
# GOMEMLIMIT set to ~80% of limit_mib in the deployment env.
memory_limiter:
check_interval: 1s
limit_mib: 1600
spike_limit_mib: 400
# Last before export: fewer, larger requests in the persistent queue means
# fewer disk writes and fewer retry round-trips.
batch:
send_batch_size: 8192
timeout: 200ms
exporters:
otlp_grpc:
endpoint: ${env:BACKEND_OTLP_ENDPOINT} # e.g. otlp.vendor.example:4317
headers:
api-key: ${env:BACKEND_API_KEY} # rename header to your vendor's
# tls: { insecure: true } # only for local/plaintext targets
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 0 # default 300s gives up after 5 min; 0 = retry until
# delivered. The queue absorbs the backlog on disk.
sending_queue:
enabled: true
storage: file_storage # replaces the in-memory queue entirely; queued
# data survives restarts and OOM kills
sizer: items # default 'requests' makes queue_size unpredictable
# (a request can be 1 span or 10k); items = records
queue_size: 2000000 # ~2M spans/datapoints/records. Size it: sustained
# items/sec x outage budget in seconds.
num_consumers: 10
block_on_overflow: true # full queue blocks the pipeline (backpressure to
# SDKs) instead of silently dropping new data
extensions:
file_storage:
directory: /var/lib/otelcol/file_storage
create_directory: true # dir must exist and be writable; on K8s mount a
# PVC here (emptyDir defeats restart persistence)
timeout: 1s
compaction:
on_start: true # bbolt never shrinks its file on its own; compact
on_rebound: true # after drain or disk stays at high-water mark
directory: /var/lib/otelcol/file_storage
health_check:
endpoint: 0.0.0.0:13133
# do not use check_collector_pipeline: README marks it broken
service:
extensions: [file_storage, health_check]
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp_grpc]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp_grpc]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp_grpc]
Validated against otelcol-contrib v0.147.0. Fill in the ${env:…} placeholders before running it.