PII Redaction Gateway

Masks emails, card numbers, SSNs, bearer tokens, AWS keys, and IPs in log bodies and span attributes at a central gateway, before telemetry leaves your network.

Redaction is a compliance control, so this flow is written for a central gateway tier that all egress telemetry transits — not for per-node agents, where the policy would be enforced in as many places as you have hosts and drift immediately. It masks the usual categories (email addresses, card numbers, SSNs, bearer tokens, AWS access keys, IP addresses) across both log bodies and span attributes, combining OTTL replacement patterns with the redaction processor as a backstop for values that pattern-match but weren't anticipated. Review the patterns against your own data before trusting them: redaction that silently misses a field is worse than no redaction, because it reads as a control that passed.

Before you use this

Sends data to

otlp_grpc

You'll need to set these before it runs

OTLP_BACKEND_ENDPOINT

The configuration

collector v0.147.0

# PII Redaction Gateway
# Run this on a central gateway tier that all egress telemetry transits,
# not on per-node agents: redaction is a compliance control and must be
# enforced in one auditable place.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # First processor in every pipeline so backpressure reaches receivers
  # (memory_limiter README best practice).
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20

  # Masks attribute values on spans and logs, and also scans log bodies
  # (string and map) against blocked_values. transform/scrub_bodies below
  # adds body-specific patterns and readable placeholders.
  redaction:
    allow_all_keys: true   # blocked-pattern mode; set false + allowed_keys for a strict allowlist
    blocked_key_patterns:
      - '(?i).*password.*'
      - '(?i).*token.*'
      - '(?i).*api_key.*'
      - '(?i).*secret.*'
      - '(?i)authorization'
    blocked_values:
      - '4[0-9]{12}(?:[0-9]{3})?'                          # Visa
      - '5[1-5][0-9]{14}'                                  # Mastercard
      - '\b\d{3}-\d{2}-\d{4}\b'                            # US SSN
      - 'AKIA[0-9A-Z]{16}'                                 # AWS access key id
      - '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'   # email
    summary: debug   # stamps redaction.* attributes (e.g. redaction.masked.count); handy in test runs, set silent in prod

  # Scrubs string LOG BODIES with OTTL regexes: readable placeholders plus
  # token/JWT/IP patterns that are not in redaction's blocked_values.
  transform/scrub_bodies:
    error_mode: ignore   # replace_pattern errors on map bodies; skip the statement, keep the record
    log_statements:
      - replace_pattern(log.body, "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}", "***EMAIL***")
      - replace_pattern(log.body, "(?i)bearer +[a-z0-9._\\-]+", "***TOKEN***")
      - replace_pattern(log.body, "eyJ[A-Za-z0-9_-]{8,}\\.[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+", "***JWT***")
      - replace_pattern(log.body, "AKIA[0-9A-Z]{16}", "***AWS_KEY***")
      # 13-16 digit runs incl. spaces/dashes; also matches order ids and ms timestamps. Test first.
      - replace_pattern(log.body, "\\b(?:\\d[ -]?){13,16}\\b", "***CARD***")
      - replace_pattern(log.body, "\\b(?:\\d{1,3}\\.){3}\\d{1,3}\\b", "***IP***")

  # Hash (SHA-1) identifiers you still need for correlation; delete the rest.
  attributes/pii:
    actions:
      - key: user.email
        action: hash
      - key: user.id
        action: hash
      - key: enduser.id
        action: hash
      - key: user.full_name
        action: delete
      - key: user.phone_number
        action: delete

  # Last before export, after all masking and drops (batch README).
  batch:
    send_batch_size: 8192
    timeout: 200ms

exporters:
  otlp_grpc:   # renamed from `otlp` in core v0.144.0; old id is a deprecated alias
    endpoint: ${env:OTLP_BACKEND_ENDPOINT}   # e.g. backend.example.com:4317
    headers:
      authorization: <YOUR_API_KEY>
    sending_queue:
      storage: file_storage   # persistent queue; survives gateway restarts
    # retry_on_failure is enabled by default (5s initial, 30s max, 300s max elapsed)

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  file_storage:
    directory: /var/lib/otelcol/file_storage   # must exist and be writable by the collector

service:
  extensions: [health_check, file_storage]
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, redaction, transform/scrub_bodies, attributes/pii, batch]
      exporters: [otlp_grpc]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, redaction, attributes/pii, batch]
      exporters: [otlp_grpc]

Validated against otelcol-contrib v0.147.0. Fill in the ${env:…} placeholders before running it.

security

compliance

redaction