Ship OTel data to ClickHouse
The ClickStack (HyperDX) ingestion collector as upstream ships it: parses JSON log bodies into attributes, infers severity from level fields or the body text, and routes browser session-replay events to their own ClickHouse table.
This is the collector configuration the ClickStack distribution runs, merged from HyperDX's base and standalone halves (docker/otel-collector) so it works on a stock contrib build. The otlp/hyperdx receiver takes all three signals on gRPC :4317 and HTTP :4318 with CORS open for the browser SDK, memory_limiter and a 10,000-item, 5-second batch shape inserts the way ClickHouse wants them, and the clickhouse exporter writes the otel_* tables HyperDX reads, creating them on first start.
Logs take a detour. The routing/logs connector sends records carrying an rr-web.event attribute (HyperDX session replay) to logs/out-rrweb, where a second clickhouse/rrweb exporter writes them to the hyperdx_sessions table. Everything else goes through logs/out-default, where one transform processor parses JSON bodies into attributes, fills severity_text and severity_number from whichever level/severity spelling a library used or from the first severity word in the body, and lowercases the result so HyperDX's severity filter matches.
Set CLICKHOUSE_ENDPOINT, CLICKHOUSE_DATABASE, CLICKHOUSE_USER, and CLICKHOUSE_PASSWORD. The exporter timeout and the JSON column switch read the same HYPERDX_OTEL_* variables as upstream, with upstream's defaults, so a ClickStack .env drops in unchanged. Nothing here masks or drops data: put the PII Redaction Gateway flow in front of it if you need that.
Use this flow
Open the config in Telflo and it becomes a working pipeline on the canvas: adapt what's specific to you, test it against recorded traffic, and push it to your fleet over OpAMP. Free account, no card.
Components
What's in it, and why
Single entry point for all three signals on gRPC :4317 and HTTP :4318, with include_metadata on and every origin and header allowed over CORS so HyperDX's browser SDK can post straight from a page.
Caps the collector heap (1500 MiB, 512 MiB spike) and turns overload into backpressure at the receiver before ClickHouse-sized batches pile up in memory.
Parses JSON log bodies into attributes, infers severity_text and severity_number from level/severity/log.level attributes or the first severity keyword in the body, then lowercases severity_text so HyperDX's severity filter matches.
Builds the large, infrequent inserts ClickHouse wants: 10,000 items or 5 seconds, whichever comes first, instead of the 200 ms default flush.
Splits logs by content: records with an rr-web.event attribute go to the session-replay pipeline, everything else to the default one, so replay payloads never run through the parsing transform.
Writes otel_logs, otel_traces, and otel_metrics_* in the database HyperDX reads, with a 720h table TTL, create_schema on, and the JSON column type behind HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE.
Same connection, but logs_table_name is hyperdx_sessions: the table HyperDX's session replay view reads browser recording events from.
Liveness endpoint on :13133 for Docker and Kubernetes probes.
Notes
Gotchas
- 1
create_schema: trueruns CREATE DATABASE and CREATE TABLE on every start, which is how a fresh ClickStack comes up with its tables; once they exist, set it to false so a TTL or engine change is a deliberate migration rather than whatever the next collector start applies. - 2
The
jsonswitch (HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE) makes the exporter create tables with the JSON column type instead of Map, and the exporter never alters tables that already exist, so decide before the first start against a database, not after. - 3
The
routing/logsmatch isIsMatch(attributes["rr-web.event"], ".*"), so any log record that carries anrr-web.eventattribute, whatever its value, is treated as a session-replay event and skips the parsing transform. - 4
Severity inference only runs on records that arrive with
severity_number0 and an emptyseverity_text, and the body scan only reads the first 256 characters, so a record whose only level marker sits deep in a long line is filed asinfo. - 5
The JSON body parse merges with the
upsertstrategy, so a key inside the body overwrites an attribute the SDK already set with the same name, andflattenturns nested objects into dotted keys. - 6
There is no
file_storageextension and theclickhouseexporter runs on the default in-memory sending queue, exactly as upstream ships it; a ClickHouse outage longer than the retry window (max_elapsed_time: 300s) drops data unless you add a persistent queue or put the Fail over to S3 flow in front of it. - 7
The
otlp/hyperdxreceiver allows every origin and header over HTTP because HyperDX's browser SDK posts straight to it; if only servers send to this collector, tightencorsbefore exposing :4318. - 8
Metrics support in the
clickhouseexporter is still alpha (traces and logs are beta), so treat theotel_metrics_*table schemas as subject to change between collector upgrades.
Sources and references (16)ShowHide
Everything consulted while researching and fact-checking this flow, including the README of every component it uses.
- github.com/hyperdxio/hyperdx/blob/main/docker/otel-collector/config.yaml
- github.com/hyperdxio/hyperdx/blob/main/docker/otel-collector/config.standalone.yaml
- clickhouse.com/docs/use-cases/observability/clickstack/ingesting-data/otel-collector
- clickhouse.com/docs/use-cases/observability/clickstack/ingesting-data/schemas
- clickhouse.com/docs/use-cases/observability/clickstack/config
- clickhouse.com/docs/use-cases/observability/clickstack/sdks/browser
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/exporter/clickhouseexporter/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/connector/routingconnector/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/processor/transformprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/pkg/ottl/ottlfuncs/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/pkg/ottl/contexts/ottllog/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.159.0/extension/healthcheckextension/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/receiver/otlpreceiver/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/processor/memorylimiterprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/processor/batchprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.159.0/exporter/exporterhelper/README.md
More flows
Related flows
Replace vendor agents on your VM fleet
Scrapes CPU, memory, load, disk, filesystem, and network metrics plus system and application logs on every VM.
Collect per-node telemetry with a DaemonSet
Per-node DaemonSet that scrapes kubelet and host metrics, tails pod logs with the container parser and file_storage checkpoints, enriches everything with k8sattributes, and forwards to the in-cluster gateway.
Redact PII before telemetry leaves the network
Masks emails, card numbers, SSNs, bearer tokens, AWS keys, and IPs in log bodies and span attributes at a central gateway, before telemetry leaves your network.
Test it before your fleet runs it
Free account, no card. Open this flow in the editor, adapt it, and see what it does to real data before anything ships.