Use this flow to try SereneDB without migrating anything. The collector sends everything to two places. Your current backend keeps getting the full stream, and SereneDB gets a copy. When the trial is over, delete the exporter you no longer need and publish.
SereneDB only accepts OTLP over HTTP, and it writes each export in one transaction with a fixed cost. It gets through a few large requests faster than many small ones, so its docs recommend putting a collector in front as a batching proxy. This flow is that collector. It takes OTLP over gRPC or HTTP and batches at the sizes SereneDB documents, then authenticates as a SereneDB role over HTTP Basic auth.
Run it as a gateway that your apps and agent collectors send to. A collector that sees a lot of traffic fills its batches faster, so SereneDB gets fewer and larger requests.
SereneDB itself only listens for Postgres connections by default. Start it with an HTTP listener for the otel API as well, for example --listen 'postgres://0.0.0.0:7890,http://0.0.0.0:4318?api=otel', and map that port too if SereneDB runs in Docker. The server creates its OTel tables at startup. After that, run the statements below as the postgres superuser to give the collector its own role.
Set SERENEDB_OTLP_ENDPOINT to the listener's base URL, like http://serenedb:4318, and SERENEDB_USERNAME and SERENEDB_PASSWORD to the collector's role. Point OTLP_ENDPOINT at your current backend's OTLP/HTTP endpoint and set OTLP_AUTH_HEADER to the Authorization header value it expects.
CREATE ROLE otel_collector LOGIN PASSWORD '<password>';
GRANT INSERT ON otel_logs TO otel_collector;
GRANT INSERT ON otel_traces TO otel_collector;
GRANT INSERT ON otel_metrics_gauge TO otel_collector;
GRANT INSERT ON otel_metrics_sum TO otel_collector;
GRANT INSERT ON otel_metrics_histogram TO otel_collector;
GRANT INSERT ON otel_metrics_exponential_histogram TO otel_collector;
GRANT INSERT ON otel_metrics_summary TO otel_collector;Use this flow
Open the config in Telflo and it becomes a working pipeline on the canvas: adapt what's specific to you, test it against recorded traffic, and push it to your fleet. Free account, no card.
Components
What's in it, and why
Takes all three signals on gRPC :4317 and HTTP :4318, so apps that export over gRPC can still reach SereneDB, which only accepts OTLP over HTTP.
Caps the collector's memory and refuses new data when it runs high, so senders back off and retry instead of the collector crashing in a burst.
Uses the sizes from SereneDB's docs: a batch goes out at 50,000 items or after 5 seconds, and none is bigger than 100,000.
Writes to SereneDB's /v1/logs, /v1/traces, and /v1/metrics endpoints as gzipped protobuf and retries temporary failures for up to five minutes.
The other half of the dual-write: everything the collector receives also goes to the backend you run today.
Adds the role's username and password to every request as HTTP Basic auth, which SereneDB checks against its database roles.
Keeps both exporters' queues on disk, so data waiting to be sent survives a collector restart.
Notes
Gotchas
- 1
SereneDB writes each export in one transaction. If it rejects one record, the whole batch fails with a 400, and the collector doesn't retry a 400, so that batch is lost. Temporary failures, like a 503 or SereneDB being down, are retried for up to five minutes before the batch is dropped. Watch the collector's
otelcol_exporter_send_failed_*metrics for the first few days. - 2
SereneDB caps a request at 100 MiB after decompression and answers anything bigger with a 413, which the collector also drops without retrying. At 100,000 items per request you hit that cap once items average about 1 KiB, so lower
send_batch_max_sizeif your log records or spans run larger. - 3
Your current backend gets the same large batches. If it limits request size, split the two exporters into separate pipelines, each with its own
batchprocessor, before you publish. - 4
SereneDB's docs say large exports go fastest with about one in flight per server CPU core. Each collector here sends up to 4 at a time per signal (
num_consumers), so raise it if a single gateway carries all your traffic and SereneDB has cores to spare. - 5
Run SereneDB v26.10.3 or newer. OTLP ingest arrived in v26.10.1, scheduled jobs (which retention uses) in v26.10.2, and the
otel_*metric functions in v26.10.3. - 6
The OTel tables keep every row until you delete it. Before real volume arrives, schedule a
CREATE JOBfor each table that deletes rows past your retention window. SereneDB's OpenTelemetry docs have the statements. - 7
SereneDB's docs put the OTLP listener on port 4318, the same port this flow's receiver uses. If the collector and SereneDB share a host, move one of them.
Sources and references (13)ShowHide
Everything consulted while researching and fact-checking this flow, including the README of every component it uses.
- serenedb.com/docs/data_import_and_export/opentelemetry
- serenedb.com/docs/configuration/cli
- serenedb.com/docs/installation/docker
- serenedb.com/docs/security/privileges
- serenedb.com/docs/sql/statements/grant
- github.com/serenedb/serenedb/releases
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/receiver/otlpreceiver/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/processor/memorylimiterprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/processor/batchprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/exporter/otlphttpexporter/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/exporter/exporterhelper/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/extension/basicauthextension/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/extension/storage/filestorage/README.md
More flows
Related flows
Dual-ship to two backends during a migration
Fans every trace, metric, and log out to both the legacy agent and the new OTLP/HTTP backend so the two can be compared side by side during a migration.
Move from Datadog to Elastic without a monitoring gap
Accepts traffic from the Datadog Agents you already run and dual-ships every trace, metric, and log to both Datadog and Elastic, so the two can be compared on identical data before you cut over.
Move from Datadog to ClickStack without a monitoring gap
Accepts traffic from the Datadog Agents you already run and dual-ships every trace, metric, and log to both Datadog and ClickStack, so the two can be compared on identical data before you cut over.
Test it before your fleet runs it
Free account, no card. Open this flow in the editor, adapt it, and see what it does to real data before anything ships.