Ship OTel data to GreptimeDB

A standalone gateway that fills in missing service names, timestamps, and environment, then ships logs, metrics, and traces to GreptimeDB over OTLP/HTTP, with hot attributes as indexed columns.

Logs Metrics Traces Collection

Use this flow if you run GreptimeDB and want the collector, not the database, to own how your telemetry is shaped. GreptimeDB stores metrics, logs, and traces in one columnar engine on object storage, and you query all of them with SQL, plus PromQL for metrics.

Run it as a standalone gateway and point your apps, SDKs, and agents at it on :4317 (gRPC) or :4318 (HTTP). The transform/defaults processor sets a missing service.name to unknown_service, gives logs with no timestamp their observed time, and stamps deployment.environment.name. A GreptimeDB pipeline could make the same fixes, but here they live in a config you can version and test. The gateway adds no host attributes of its own, so it never stamps its own hostname on someone else's data.

Each signal gets its own exporter, because GreptimeDB reads its routing from request headers. Logs go to the otel_logs table, with X-Greptime-Log-Extract-Keys promoting service, environment, host, region, and Kubernetes attributes to their own columns. Traces go through GreptimeDB's built-in greptime_trace_v1 pipeline into opentelemetry_traces, the table its Jaeger-compatible API reads. Metrics land in one table per metric, with the same hot attributes kept as labels.

Create otel_logs yourself before the first write (the statement is below). Then set GREPTIMEDB_ENDPOINT to the HTTP base URL (port 4000 by default), GREPTIMEDB_DATABASE (public on a fresh install), GREPTIMEDB_USERNAME and GREPTIMEDB_PASSWORD, and optionally DEPLOYMENT_ENVIRONMENT.

Run once in GreptimeDB, before the first write

CREATE TABLE IF NOT EXISTS otel_logs (
  "timestamp"                   TIMESTAMP(9) TIME INDEX,
  "deployment.environment.name" STRING INVERTED INDEX,
  "service.namespace"           STRING INVERTED INDEX,
  "service.name"                STRING INVERTED INDEX,
  "service.version"             STRING INVERTED INDEX,
  "host.name"                   STRING INVERTED INDEX,
  "cloud.region"                STRING INVERTED INDEX,
  "cloud.availability_zone"     STRING INVERTED INDEX,
  "k8s.namespace.name"          STRING INVERTED INDEX,
  "k8s.deployment.name"         STRING INVERTED INDEX,
  "k8s.node.name"               STRING INVERTED INDEX,
  "k8s.pod.name"                STRING SKIPPING INDEX,
  "service.instance.id"         STRING SKIPPING INDEX,
  "container.id"                STRING SKIPPING INDEX,
  severity_text                 STRING INVERTED INDEX,
  trace_id                      STRING SKIPPING INDEX,
  body                          STRING FULLTEXT INDEX,
  PRIMARY KEY ("deployment.environment.name", "service.namespace", "service.name")
) WITH (append_mode = 'true', ttl = '30d');

How the data moves

receiverprocessorexporterextension

logs

otlpmemory_limitertransform/defaultsbatchotlphttp/greptimedb_logs

traces

otlpmemory_limitertransform/defaultsbatchotlphttp/greptimedb_traces

metrics

otlpmemory_limitertransform/defaultsbatchotlphttp/greptimedb_metrics

Use this flow

Open the config in Telflo and it becomes a working pipeline on the canvas: adapt what's specific to you, test it against recorded traffic, and push it to your fleet. Free account, no card.

Use this flow in Telflo Start free

Components

What's in it, and why

otlp

Single entry point for all three signals from apps, SDKs, and agents, on gRPC :4317 and HTTP :4318.

memory_limiter

Holds the gateway under a fixed 1600 MiB with 400 MiB of headroom and pushes back on senders instead of running out of memory. The limits are sized for a 2 GiB container, so raise both if yours is bigger.

transform/defaults

Sets service.name to unknown_service when it's missing, stamps deployment.environment.name from DEPLOYMENT_ENVIRONMENT when that's set, and gives logs with no timestamp their observed time, or the current time if that's missing too.

batch

Sends 8192 items at a time (16384 at most) or whatever has arrived after 5 seconds, so GreptimeDB gets fewer, larger writes.

otlphttp/greptimedb_logs

Writes logs to the otel_logs table and promotes 13 resource attributes to their own columns through the X-Greptime-Log-Extract-Keys header.

otlphttp/greptimedb_traces

Writes spans through GreptimeDB's built-in greptime_trace_v1 pipeline, chosen with the X-Greptime-Pipeline-Name header.

otlphttp/greptimedb_metrics

Writes each metric to its own table, and through the x-greptime-otlp-metric-promote-resource-attrs header adds eight resource attributes to the ones GreptimeDB already keeps as labels.

health_check

Serves the liveness endpoint on :13133 for the load balancer or orchestrator in front of the gateway.

basicauth/greptimedb

Adds the GreptimeDB username and password to every export as HTTP Basic auth.

file_storage

Keeps all three export queues on disk under /var/lib/otelcol/file_storage, so queued data survives a collector restart and waits out a GreptimeDB outage.

Notes

Gotchas

1
If GreptimeDB creates otel_logs itself on the first insert, every promoted key joins the primary key, including high-churn ids like service.instance.id that start a new series each time an instance restarts, and none of those columns get an index. On GreptimeDB 1.1 or later, a table you create first keeps its own primary key and indexes.
2
To promote another log attribute, add it to the extract-keys header and add a matching column to the table. Use an inverted index for values you filter or group by and a skipping index for ids you look up one at a time. Don't put spaces after the commas: GreptimeDB doesn't trim them, so the key won't match. A key that's in the header but not in the table is still stored, as a plain column with no index.
3
Attributes you don't promote stay in the resource_attributes JSON column. Quote the dotted key when you query it, as in json_get_string(resource_attributes, '["os.type"]').
4
Metric names and labels are stored Prometheus-style: dots become underscores and units and types become suffixes, so a memory.usage gauge in bytes lands in memory_usage_bytes with a service_name label. GreptimeDB already keeps 19 common resource attributes as labels, and the promote header only adds to that list.
5
As of GreptimeDB 1.2, exponential histograms are dropped without an error, so have your SDKs send explicit-bucket histograms. Delta sums and histograms are stored as sent, not turned into running totals, so add the delta_to_cumulative processor if a source sends delta and you query it with PromQL.
6
GreptimeDB only accepts OTLP over HTTP as protobuf (it dropped OTLP over gRPC in 0.13), which is why the exporters are otlphttp. A standalone server listens on 127.0.0.1:4000 by default, so start it with --http-addr 0.0.0.0:4000 when the gateway runs on another host or container.
7
If DEPLOYMENT_ENVIRONMENT is unset, the environment stamp is skipped and the collector still starts, so data without its own deployment.environment.name lands with that column empty.
8
Keep /var/lib/otelcol/file_storage on a persistent volume. On an ephemeral one, a restart of the gateway throws away whatever was queued while GreptimeDB was down.

Sources and references (21)Show

Everything consulted while researching and fact-checking this flow, including the README of every component it uses.

More flows

Related flows

Replace vendor agents on your VM fleet

Scrapes CPU, memory, load, disk, filesystem, and network metrics plus system and application logs on every VM.

LogsMetricsCollectionCost control

Ship OTel data to ClickHouse

The ClickStack (HyperDX) ingestion collector as upstream ships it: parses JSON log bodies into attributes, infers severity from level fields or the body text, and routes browser session-replay events to their own ClickHouse table.

LogsMetricsTracesCollectionCost control

Collect per-node telemetry with a DaemonSet

Per-node DaemonSet that scrapes kubelet and host metrics, tails pod logs with the container parser and file_storage checkpoints, enriches everything with k8sattributes, and forwards to the in-cluster gateway.

LogsMetricsKubernetesCollection

Test it before your fleet runs it

Free account, no card. Open this flow in the editor, adapt it, and see what it does to real data before anything ships.

Get started Book a demo