Prometheus: Service Discovery

Prometheus provides a variety of service discovery mechanisms to automatically find and monitor endpoints within modern and dynamic environments. Below, we break down the primary categories and their roles, step by step:

1. Cloud Platform Integrations
   • AWS EC2: Discovers EC2 instances to monitor automatically based on filter criteria.
   • Google Cloud Platform: Identifies compute resources across projects and regions.
   • Microsoft Azure: Detects virtual machines and scale sets for scraping metrics.
   • Others: DigitalOcean, OpenStack, Linode, Hetzner.
2. Container and Orchestration Systems
   • Kubernetes: Automatically finds pods, services, and nodes for metric collection in dynamic clusters.
   • Docker & Swarm: Discovers running containers and swarm tasks as scrape targets.
   • Mesos, Consul, Eureka: Integrates with other orchestrators and service registries, enabling flexible target tracking.
3. Generic Discovery Methods
   • DNS-based Discovery: Leverages SRV or A records to dynamically track available endpoints.
   • File-based Discovery: Reads endpoints and metadata from local files (YAML or JSON), supporting automation and CI/CD workflows.
   • HTTP-based Discovery: Periodically queries remote HTTP endpoints for lists of targets, adapting to external changes in real-time.

By supporting these diverse mechanisms, Prometheus enables efficient and reliable monitoring—even as infrastructure scales, shifts, and evolves.

Setting up Prometheus for automated service discovery involves specifying the appropriate mechanism in the configuration file. Here are practical examples for different environments and approaches, explained step by step:

1. AWS EC2 Service Discovery
   1. Create a new job in your prometheus.yml:

        - job_name: "node_exporter"
          ec2_sd_configs:
            - region: us-east-1
              filters:
                - name: "tag:Monitor"
                  values: ["true"]
          relabel_configs:
            - source_labels: [__meta_ec2_tag_Name]
              target_label: instance
            - source_labels: [__meta_ec2_public_ip]
              target_label: ip
      

   2. This configuration finds EC2 instances in us-east-1 with the Monitor:true tag and assigns labels for easier metric identification.
2. Kubernetes Service Discovery
   1. Add a Kubernetes job to scrape pods with specific annotations:

        - job_name: 'kubernetes-pods'
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: 'true'
      

   2. Prometheus now automatically discovers and scrapes all pods in your Kubernetes cluster that have the annotation prometheus.io/scrape=true.
3. File-Based Service Discovery
   1. Reference a targets file within your config:

        - job_name: "services"
          file_sd_configs:
            - files:
              - targets.yml
      

   2. Example targets.yml file:

      - targets:
        - "localhost:11111"
        - "localhost:22222"
        labels:
          job: "services"
          env: "production"
      - targets:
        - "localhost:44444"
        labels:
          job: "services"
          env: "development"
      

   3. Updating the targets.yml file allows you to add or remove endpoints without restarting Prometheus.
4. HTTP-based Service Discovery
   1. Point Prometheus to a dynamic HTTP endpoint that returns the targets:

        - job_name: "dynamic-http"
          http_sd_configs:
            - url: 'http://example.com/discovery'
              refresh_interval: 60s
              http_headers:
                Purpose: Prometheus-scraper
      

   2. Prometheus will refresh the list every 60 seconds, allowing external automation to control the available targets.

Each snippet above is tailored for a different scenario, helping Prometheus to discover and monitor new endpoints automatically as your infrastructure changes.

To make service discovery in Prometheus both flexible and reliable, a few core parameters and configuration keys are used across all supported mechanisms. Here’s a step-by-step breakdown of the essential parameters, what they do, and how to adjust them:

1. job_name
   • Acts as a unique identifier for each scrape job. All targets discovered via a configuration share this name, making it easy to group and filter metrics.
   • Example: job_name: "kubernetes-pods"
2. refresh_interval
   • Controls how often Prometheus refreshes its target list from the discovery method. The interval can be set per job.
   • Defaults: Typically 1 minute for HTTP-based, 5 minutes for file-based service discovery. Can be customized.
   • Example: refresh_interval: 60s
3. relabel_configs
   • Defines rules to filter, rename, or enrich metric labels based on metadata from the discovered targets.
   • Allows fine-grained control over which endpoints are scraped and how they are labeled.
   • Example:

     - source_labels: [__meta_kubernetes_pod_label_app]
       target_label: app
     

4. files
   • Specifies the list of local files from which static targets are read for file-based discovery.
   • Files can be in YAML or JSON format and patterns can be used for multiple files.
   • Example: files: ["targets.yml"]
5. url
   • For HTTP-based discovery, this is the endpoint Prometheus queries to obtain the latest targets.
   • The endpoint must return a JSON-encoded list of targets according to the specification.
   • Example: url: "http://example.com/discovery"
6. http_headers (optional)
   • Allows you to define custom HTTP headers when Prometheus calls an HTTP-based discovery endpoint. Useful for authentication or identifying requests.
   • Example:

     http_headers:
       Authorization: Bearer $TOKEN

Adjusting these parameters lets you fine-tune how Prometheus finds, filters, and labels its metrics sources, making it well-suited for a wide range of infrastructure and security needs.

Securing your Prometheus setup is essential to protect sensitive metrics and ensure only trusted entities can access or modify monitoring data. Below is a step-by-step guide to best practices for authentication and security:

1. Limit Network Exposure
   • Deploy Prometheus and exporters within private networks or VPNs. Avoid exposing endpoints to the public internet whenever possible.
   • Restrict access to trusted IP ranges using firewalls, security groups, or Kubernetes NetworkPolicies.
2. Enable HTTPS/TLS Encryption
   • Encrypt all communication channels by enabling HTTPS on Prometheus, exporters, and any service discovery endpoints.
   • Use valid TLS certificates and store certificate files securely.
3. Add Authentication Layers
   • Utilize Basic Authentication, OAuth2, or Authorization headers on service discovery and metrics endpoints.
   • Protect the Prometheus UI and APIs using authentication, such as a reverse proxy (e.g., NGINX) with required credentials or integration with your corporate Identity Provider.
   • For HTTP-based service discovery, pass secrets only with secure mechanisms (headers, OAuth2).
4. Use Role-Based Access Control (RBAC)
   • Apply RBAC in platforms like Kubernetes to tightly control what users and applications can access in Prometheus and its exporters.
   • Limit write access and administrative privileges to only essential personnel and systems.
5. Leverage Secret Management
   • Never embed passwords, tokens, or certificates directly in configuration files.
   • Store sensitive data in centralized secrets management solutions (e.g., HashiCorp Vault, Kubernetes Secrets, AWS Secrets Manager).
6. Regularly Rotate Credentials
   • Rotate API tokens, passwords, and certificates on a regular schedule to reduce risk from potential leaks.
   • Establish alerts for expiring credentials and automate the renewal process when possible.
7. Harden Exporters and Endpoints
   • Configure exporters to require authentication and run with minimum privileges.
   • Avoid exporting excessive or unnecessary metrics.
8. Audit & Monitor
   • Regularly audit Prometheus configurations and access logs for unauthorized access attempts.
   • Monitor the /targets and /metrics endpoints for unexpected changes or exposures.

Applying these security best practices ensures your Prometheus deployment remains resilient against unauthorized access and data leaks, making your monitoring infrastructure trustworthy and robust.

If service discovery in Prometheus isn’t working as expected, these step-by-step troubleshooting tips and practical advice can help resolve issues and streamline your monitoring setup:

1. Check Prometheus Targets UI
   • Visit /targets in the Prometheus web interface to verify if all intended targets are discovered. Targets marked as "down" may indicate issues with connectivity, permissions, or target endpoint health.
2. Validate Configuration Files
   • Run your prometheus.yml through a YAML linter or validator to catch formatting and syntax errors.
   • Double-check that all parameters, especially those under service discovery configuration sections, are correct and match your environment.
3. Review Log Files
   • Inspect Prometheus logs for error messages related to service discovery. Look for logs about failed scrapes, unreachable endpoints, or misconfigured plugins.
4. Test Connectivity & Permissions
   • Use tools like curl or ping from the Prometheus server to check access to metrics endpoints.
   • Make sure cloud and orchestration platform credentials are valid and have adequate permissions.
5. Examine Label and Relabel Configurations
   • Misconfigured relabel_configs can filter out or incorrectly label targets. Simplify relabel rules to isolate issues and reintroduce them incrementally.
6. Monitor Refresh Intervals
   • Ensure the refresh_interval is appropriate; too infrequent and new targets will not appear quickly, too frequent may increase system load.
7. Handle Missing or Dead Targets
   • If targets are not being picked up or remain after deletion, verify external service discovery endpoints (HTTP/file) or synchronization with your orchestration platform.
   • For file-based discovery, confirm that discovery files exist in the correct path and conform to the required format.
8. Common Mistakes to Avoid
   • Avoid high cardinality labels that can strain memory and slow queries.
   • Monitor exporter availability and keep ports/firewalls open as needed.
   • Regularly update configurations as the monitored environment changes to catch legacy or obsolete settings.

Keeping these troubleshooting practices in mind will make your Prometheus service discovery setup more reliable and responsive to changes in your environment.