Collect per-node telemetry with a DaemonSet
Per-node DaemonSet that scrapes kubelet and host metrics, tails pod logs with the container parser and file_storage checkpoints, enriches everything with k8sattributes, and forwards to the in-cluster gateway.
The per-node half of a Kubernetes setup, meant to run as a DaemonSet: kubeletstats scrapes container, pod, node, and volume metrics from the local kubelet; hostmetrics reads CPU, memory, load, disk, network, and filesystem from the node under /hostfs; and filelog tails pod logs with the container operator handling docker, containerd, and CRI-O formats. k8sattributes enriches everything with namespace, deployment, pod, and container metadata, associating by connection first and pod UID as the fallback.
It forwards over OTLP to an in-cluster gateway at otel-gateway.observability.svc.cluster.local. Change that to wherever your gateway Service lives. File offsets checkpoint to file_storage so a pod restart doesn't replay logs.
Use this flow
Open the config in Telflo and it becomes a working pipeline on the canvas: adapt what's specific to you, test it against recorded traffic, and push it to your fleet over OpAMP. Free account, no card.
Components
What's in it, and why
Scrapes container, pod, node, and volume metrics from the local kubelet every 30 seconds.
Reads CPU, memory, load, disk, network, and filesystem metrics from the node through the /hostfs mount.
Tails pod logs with the container operator, which handles docker, containerd, and CRI-O formats without a per-runtime regex.
Runs first so backpressure reaches the scrapers before the agent is OOM-killed on a busy node.
Adds namespace, deployment, node, pod, and container metadata, associating by connection first and pod UID as the fallback.
Batches on a 10 second timeout before forwarding to the gateway.
Forwards to the in-cluster gateway Service over OTLP gRPC.
Checkpoints filelog offsets to disk so a pod restart does not replay logs already sent.
Notes
Gotchas
- 1
It needs the usual DaemonSet privileges: a service account that can read the kubelet API, and hostPath mounts for
/var/logand/hostfs. - 2
insecure_skip_verifyis on because kubelets commonly serve self-signed certificates; the tradeoff is that the kubelet's identity is not verified. - 3
The exporter points at
otel-gateway.observability.svc.cluster.local, a placeholder. Point it at your own gateway Service or the agent has nowhere to send. - 4
The file_storage directory has to survive pod restarts to be worth anything; an emptyDir volume is wiped on rescheduling, which defeats the checkpoint.
More flows
Related flows
Replace vendor agents on your VM fleet
Scrapes CPU, memory, load, disk, filesystem, and network metrics plus system and application logs on every VM.
Ship OTel data to ClickHouse
The ClickStack (HyperDX) ingestion collector as upstream ships it: parses JSON log bodies into attributes, infers severity from level fields or the body text, and routes browser session-replay events to their own ClickHouse table.
Run a cluster gateway for Kubernetes
Cluster-level gateway that receives OTLP from apps and node agents, tails pod logs, stamps deployment.environment, deletes a PII attribute, and ships everything to your OTLP/HTTP backend with a debug tap on logs.
Test it before your fleet runs it
Free account, no card. Open this flow in the editor, adapt it, and see what it does to real data before anything ships.