Redact PII before telemetry leaves the network
Masks emails, card numbers, SSNs, bearer tokens, AWS keys, and IPs in log bodies and span attributes at a central gateway, before telemetry leaves your network.
Masks emails, card numbers, SSNs, bearer tokens, AWS keys, and IPs in log bodies and span attributes at a central gateway, before telemetry leaves your network. Identifiers you still need for correlation are hashed instead of dropped. Runs the redaction, transform, and attributes processors in the documented order, with a persistent sending queue.
Use this flow
Open the config in Telflo and it becomes a working pipeline on the canvas: adapt what's specific to you, test it against recorded traffic, and push it to your fleet over OpAMP. Free account, no card.
Components
What's in it, and why
Accepts OTLP over gRPC and HTTP from agents and SDKs as the single entry point to the gateway.
Runs first in each pipeline so the gateway applies backpressure to receivers instead of OOMing under log bursts.
Masks span and log attribute values matching the blocked regexes (cards, SSNs, AWS keys, emails), any attribute whose key looks like a secret, and blocked_values matches in log bodies (string and map).
Applies OTTL replace_pattern statements to string log bodies, adding explicit placeholders and body-specific patterns (bearer tokens, JWTs, IPs) beyond the redaction processor's blocked_values.
Hashes user identifiers so cross-signal correlation still works, and deletes fields that should never leave the network.
Batches records last in the chain, after all masking and drops, per the batch README placement guidance.
Ships the scrubbed telemetry to any OTLP backend; renamed from otlp in core v0.144.0.
Exposes the liveness endpoint Kubernetes probes need for a gateway deployment.
Backs the exporter sending queue with disk so queued data survives gateway restarts.
Notes
Gotchas
- 1
The redaction processor's body redaction only masks blocked_values matches; bearer tokens, JWTs, and bare IPs are not in that list, so the transform stage is what catches those in bodies, and it swaps in readable placeholders like *TOKEN* instead of generic masking.
- 2
The credit-card regex matches any 13-16 digit run including spaces and dashes, which also hits order IDs and millisecond timestamps; run planted-PII sample data through the config and check for mangled fields before rolling it out.
- 3
replace_pattern only works on string bodies. Structured (map) log bodies make the statement error, and error_mode: ignore skips it while logging the error, so key/value bodies are not scrubbed by the transform stage; they are still covered by the redaction processor, which recursively walks map bodies and redacts blocked keys and masks blocked_values in place.
- 4
The attributes processor hash action is SHA-1 pseudonymization, not anonymization; hashed user IDs are still personal data under GDPR, and identical inputs always produce identical hashes across your whole estate.
- 5
redaction is alpha for logs at v0.146.0 (beta for traces), and summary: debug adds redaction.* attributes (e.g. redaction.masked.count, redaction.redacted.keys) to every affected record, which costs money on per-attribute-billed backends; switch to summary: silent once you trust the rules.
Sources and references (11)ShowHide
Everything consulted while researching and fact-checking this flow, including the README of every component it uses.
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.146.0/processor/redactionprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.146.0/processor/redactionprocessor/processor.go
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.146.0/processor/transformprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.146.0/processor/attributesprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.146.0/processor/memorylimiterprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.146.0/processor/batchprocessor/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.146.0/receiver/otlpreceiver/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.146.0/exporter/otlpexporter/README.md
- github.com/open-telemetry/opentelemetry-collector/blob/v0.146.0/exporter/exporterhelper/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.146.0/extension/storage/filestorage/README.md
- github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.146.0/extension/healthcheckextension/README.md
More flows
Related flows
Replace vendor agents on your VM fleet
Scrapes CPU, memory, load, disk, filesystem, and network metrics plus system and application logs on every VM.
Survive backend outages with a persistent queue
Replaces the default in-memory exporter queue with a disk-backed queue that survives collector restarts, sized in items instead of opaque requests.
Ship OTel data to ClickHouse
The ClickStack (HyperDX) ingestion collector as upstream ships it: parses JSON log bodies into attributes, infers severity from level fields or the body text, and routes browser session-replay events to their own ClickHouse table.
Test it before your fleet runs it
Free account, no card. Open this flow in the editor, adapt it, and see what it does to real data before anything ships.