Skip to content
All flows

Redact PII before telemetry leaves the network

Masks emails, card numbers, SSNs, bearer tokens, AWS keys, and IPs in log bodies and span attributes at a central gateway, before telemetry leaves your network.

Masks emails, card numbers, SSNs, bearer tokens, AWS keys, and IPs in log bodies and span attributes at a central gateway, before telemetry leaves your network. Identifiers you still need for correlation are hashed instead of dropped. Runs the redaction, transform, and attributes processors in the documented order, with a persistent sending queue.

How the data moves
receiverprocessorexporterextension
logs
otlpmemory_limiterredactiontransform/scrub_bodiesattributes/piibatchotlp_grpc
traces
otlpmemory_limiterredactionattributes/piibatchotlp_grpc

Use this flow

Open the config in Telflo and it becomes a working pipeline on the canvas: adapt what's specific to you, test it against recorded traffic, and push it to your fleet over OpAMP. Free account, no card.

Components

What's in it, and why

otlp

Accepts OTLP over gRPC and HTTP from agents and SDKs as the single entry point to the gateway.

memory_limiter

Runs first in each pipeline so the gateway applies backpressure to receivers instead of OOMing under log bursts.

redaction

Masks span and log attribute values matching the blocked regexes (cards, SSNs, AWS keys, emails), any attribute whose key looks like a secret, and blocked_values matches in log bodies (string and map).

transform/scrub_bodies

Applies OTTL replace_pattern statements to string log bodies, adding explicit placeholders and body-specific patterns (bearer tokens, JWTs, IPs) beyond the redaction processor's blocked_values.

attributes/pii

Hashes user identifiers so cross-signal correlation still works, and deletes fields that should never leave the network.

batch

Batches records last in the chain, after all masking and drops, per the batch README placement guidance.

otlp_grpc

Ships the scrubbed telemetry to any OTLP backend; renamed from otlp in core v0.144.0.

health_check

Exposes the liveness endpoint Kubernetes probes need for a gateway deployment.

file_storage

Backs the exporter sending queue with disk so queued data survives gateway restarts.

Notes

Gotchas

  • 1

    The redaction processor's body redaction only masks blocked_values matches; bearer tokens, JWTs, and bare IPs are not in that list, so the transform stage is what catches those in bodies, and it swaps in readable placeholders like *TOKEN* instead of generic masking.

  • 2

    The credit-card regex matches any 13-16 digit run including spaces and dashes, which also hits order IDs and millisecond timestamps; run planted-PII sample data through the config and check for mangled fields before rolling it out.

  • 3

    replace_pattern only works on string bodies. Structured (map) log bodies make the statement error, and error_mode: ignore skips it while logging the error, so key/value bodies are not scrubbed by the transform stage; they are still covered by the redaction processor, which recursively walks map bodies and redacts blocked keys and masks blocked_values in place.

  • 4

    The attributes processor hash action is SHA-1 pseudonymization, not anonymization; hashed user IDs are still personal data under GDPR, and identical inputs always produce identical hashes across your whole estate.

  • 5

    redaction is alpha for logs at v0.146.0 (beta for traces), and summary: debug adds redaction.* attributes (e.g. redaction.masked.count, redaction.redacted.keys) to every affected record, which costs money on per-attribute-billed backends; switch to summary: silent once you trust the rules.

Test it before your fleet runs it

Free account, no card. Open this flow in the editor, adapt it, and see what it does to real data before anything ships.