Use this flow if you run Gigapipe, or if you're on Loki, Mimir, and Tempo and want one store instead of three. Gigapipe keeps logs, metrics, and traces in ClickHouse and answers the Loki, Prometheus, and Tempo APIs, so you point Grafana's built-in datasources at it and your dashboards keep working.
Gigapipe serves the same OTLP paths the otlphttp exporter uses and maps the data into its own storage, so the collector only needs memory_limiter and batch in front of the exporter.
Set GIGAPIPE_ENDPOINT to the base URL (port 3100 by default) and GIGAPIPE_USERNAME and GIGAPIPE_PASSWORD to the server's login. This pipeline runs in our demo environment, shipping the OpenTelemetry demo app into a self-hosted Gigapipe.
Use this flow
Open the config in Telflo and it becomes a working pipeline on the canvas: adapt what's specific to you, test it against recorded traffic, and push it to your fleet over OpAMP. Free account, no card.
Components
What's in it, and why
Single entry point for all three signals from SDKs and downstream collectors, on gRPC :4317 and HTTP :4318.
Caps collector memory and pushes back on the receivers instead of falling over during a burst.
Groups records into fewer, larger exports than the 200ms default, sized to stay under Gigapipe's 64 MiB request cap.
Writes all three signals to Gigapipe's OTLP routes from one base URL, in protobuf with gzip, behind a disk-backed queue.
Adds your Gigapipe username and password to every request the exporter sends, since Gigapipe checks HTTP Basic auth on writes.
Keeps the exporter queue on disk, so queued data survives a collector restart and waits out a Gigapipe outage.
Notes
Gotchas
- 1
Metrics need to arrive cumulative, which is the OTel SDK default. If a source sends delta (StatsD, Datadog re-exports, or an SDK set to delta), add the
delta_to_cumulativeprocessor, because Gigapipe rejects delta points while still returning HTTP 200 and the exporter never retries them. - 2
Gigapipe builds a metric's
jobandinstancelabels fromservice.nameandservice.instance.id, and a log'slevellabel fromseverity_text. Data missing those fields still lands, it just can't be selected by that label. - 3
OTLP metrics need Gigapipe 5.3.0 or later. The server's
GIGAPIPE_LOGINandGIGAPIPE_PASSWORDvariables only work on 5.5.2 and up; older releases read theQRYN_names, and if you set the new ones there the server comes up with auth off. - 4
An OTLP metrics request is capped at 64 MiB decompressed. If you hit the 413, lower
send_batch_max_sizebefore raisingGIGAPIPE_SYSTEM_SETTINGS_OTLP_MAX_MESSAGE_SIZEon the server. - 5
Every log attribute becomes a stream label, with dots turned into underscores, plus
trace_idandspan_idfrom the trace context. That's how logs link to traces, but it means logs from different spans land in different streams. - 6
Gigapipe's ruler only evaluates recording rules, so alerts living in the Loki or Mimir ruler need to become Grafana-managed alerts.
topk,quantile_over_time,stddev, andstdvararen't supported in LogQL yet, so check the panels that use them before cutting over.
Sources and references (14)ShowHide
Everything consulted while researching and fact-checking this flow, including the README of every component it uses.
- github.com/metrico/gigapipe
- github.com/metrico/gigapipe/blob/v5.5.2/docs/otlp-metrics.md
- github.com/metrico/gigapipe/blob/v5.5.2/docs/otlp-grpc.md
- github.com/metrico/gigapipe/blob/v5.5.2/docs/configuration.md
- gigapipe.com/
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/receiver/otlpreceiver/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/processor/memorylimiterprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/processor/batchprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/exporter/otlphttpexporter/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/exporter/exporterhelper/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/extension/basicauthextension/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/extension/storage/filestorage/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/processor/deltatocumulativeprocessor/README.md
- opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/
More flows
Related flows
Dual-ship to two backends during a migration
Fans every trace, metric, and log out to both the legacy agent and the new OTLP/HTTP backend so the two can be compared side by side during a migration.
Move from Datadog to Elastic without a monitoring gap
Accepts traffic from the Datadog Agents you already run and dual-ships every trace, metric, and log to both Datadog and Elastic, so the two can be compared on identical data before you cut over.
Move from Datadog to ClickStack without a monitoring gap
Accepts traffic from the Datadog Agents you already run and dual-ships every trace, metric, and log to both Datadog and ClickStack, so the two can be compared on identical data before you cut over.
Test it before your fleet runs it
Free account, no card. Open this flow in the editor, adapt it, and see what it does to real data before anything ships.