Host Fleet Monitoring

Scrapes CPU, memory, load, disk, filesystem, and network metrics plus system and application logs on every VM.

A baseline agent config for a fleet of VMs, meant to be rolled out unchanged over OpAMP and specialized with environment variables. hostmetrics covers CPU, memory, load, disk, network, and filesystem, with container and snap mount points excluded so you aren't paying for series about the overlay filesystem. The process scraper is present but commented out — it is per-PID, high cardinality, and needs elevated permissions, so it should be enabled deliberately with include filters. Logs come from two sources: filelog for application logs and journald for a named set of system units. `resourcedetection` runs env first so `OTEL_RESOURCE_ATTRIBUTES` can pin values that would otherwise be guessed, then falls through to system and the cloud detectors. The resource processor carries the fleet tags (environment, team, cost center) you'll want to templatize per host group before rollout.

Before you use this

Sends data to

otlp_grpc

You'll need to set these before it runs

DEPLOY_ENVOTLP_ENDPOINT

The configuration

collector v0.147.0

# Host Fleet Monitoring: one config for every VM, rolled out over OpAMP.
# Target: otelcol-contrib v0.146.0

extensions:
  health_check:
    endpoint: localhost:13133      # liveness probe; leave check_collector_pipeline off (README: broken)
  file_storage:
    directory: /var/lib/otelcol/file_storage
    create_directory: true         # extension fails to start if the directory is missing

receivers:
  hostmetrics:
    collection_interval: 60s
    scrapers:
      cpu:
      memory:
      load:
      disk:
      network:
      filesystem:
        exclude_mount_points:
          mount_points: ["^/var/lib/docker/.*", "^/run/.*", "^/snap/.*"]
          match_type: regexp
        exclude_fs_types:
          fs_types: [tmpfs, squashfs, overlay, autofs, devtmpfs]
          match_type: strict
      # process:                   # per-PID CPU/mem/disk series: high cardinality and needs
      #                            # elevated permissions. Enable deliberately with include filters.

  filelog:
    include: ["/var/log/app/*.log"]   # application logs; set your paths
    exclude: ["/var/log/app/*.gz"]
    start_at: end
    include_file_path: true
    storage: file_storage             # checkpoint offsets across restarts

  journald:                           # Linux-only; needs journalctl and journal read access
    priority: info
    units: [sshd, docker, containerd] # system units only; app logs come via filelog
    storage: file_storage             # cursor survives restarts

processors:
  memory_limiter:                  # first in every pipeline so backpressure reaches receivers
    check_interval: 1s
    limit_mib: 400                 # pair with GOMEMLIMIT at ~80% of this value
    spike_limit_mib: 100

  resourcedetection:
    detectors: [env, system, ec2, gcp, azure]  # first detector to set an attribute wins;
    timeout: 2s                                # env first so OTEL_RESOURCE_ATTRIBUTES can pin values
    override: false
    system:
      hostname_sources: [os]       # kernel hostname; the dns default can stall or mislabel VMs

  resource:                        # fleet tags: templatize per host group before rollout
    attributes:
      - key: deployment.environment
        value: ${env:DEPLOY_ENV}
        action: upsert
      - key: team
        value: <YOUR_TEAM>
        action: upsert
      - key: cost.center
        value: <YOUR_COST_CENTER>
        action: upsert

  batch: {}                        # defaults (8192 items / 200ms); keep last before the exporter

exporters:
  otlp_grpc:
    endpoint: ${env:OTLP_ENDPOINT} # host:port of any OTLP backend or gateway collector
    tls:
      insecure: false
    sending_queue:
      sizer: items                 # predictable sizing vs the default "requests"
      queue_size: 10000
      storage: file_storage        # persistent queue: survives reboots and backend outages
    retry_on_failure:
      enabled: true                # defaults: 5s initial, 30s max, 300s max_elapsed

service:
  extensions: [health_check, file_storage]
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [memory_limiter, resourcedetection, resource, batch]
      exporters: [otlp_grpc]
    logs:
      receivers: [filelog, journald]
      processors: [memory_limiter, resourcedetection, resource, batch]
      exporters: [otlp_grpc]

Validated against otelcol-contrib v0.147.0. Fill in the ${env:…} placeholders before running it.

hosts

collection

fleet