Skip to content

Virtual Appliance Observability

SailPoint supports the export of virtual appliance (VA) metrics and logging information to third-party monitoring tools.

Exporting VA Metrics

You can configure an Identity Security Cloud Virtual Appliance (VA) to export metrics to an observability tool of your choice. These include priority metrics, such as CPU usage, disk space, memory usage, and network traffic, and secondary metrics from SailPoint applications, such as aggregation times, CCG health status, and VA cluster uptime.

SailPoint uses an OpenTelemetry agent (otel_agent) to send metrics from the VA to your preferred observability tool. The list of tools that support OpenTelemetry Protocol (OTLP) is extensive, and each requires a separate OTLP configuration from the OTEL agent to send metrics.

To address this, SailPoint has selected the following widely-used tools that share a common set of OTLP agent configurations and support standard observability backends and pipelines:

  • New Relic
  • Dynatrace
  • Grafana Labs
  • Honeycomb

Configuring VA Metrics Export

To enable metrics to move from the VA, you need to provide the endpoint URL and API keys to authenticate the otel_agent to the observability tool. Then, run the otel_agent within the VA to load its configuration from /etc/otel-agent-config.yaml and transmit metrics to your OTLP-supported backend.

  1. Enable the otel_agent.

    • On the VA, locate the /home/sailpoint/config.yaml file.
    • Add the line enableOTELAgent: true.

  2. Start the otel_agent service by executing the command sudo systemctl start otel_agent.

  3. When the agent successful starts, it creates an otel.env file in the /home/sailpoint directory. This is where you configure the OTLP endpoint and API key for the otel_agent.

    • Set values for:

      • OTLP_ENDPOINT="" to define the endpoint URL.
      • OTLP_API_KEY="" to define the API key.

    Note

    The VA encrypts the value inside the otel.env file for the OTLP_ENDPOINT and OTLP_API_KEY.

    If you do not define these values in the otel.env file, the otel_agent can still run, but it will send metrics to your SailPoint tenant rather than sending them to your observability tool.

  4. After defining these values, restart the otel_agent service using the command sudo systemctl restart otel_agent.

  5. The otel-agent automatically detects the OTLP backend and loads the appropriate configuration.

  6. Check the agent’s log files at /home/sailpoint/log/otel_agent.log. If there are no errors and the otel_agent has detected the backend, then the configuration is successful.

  7. Go to your observability tool and search for the available VA metrics. You can create a dashboard to monitor them more easily.

Tip

You can switch between observability tools by changing the endpoint and API key in the otel.env file and restarting the otel_agent.

Available VA Metrics

Once configured, the following system metrics will be available to your OTLP-supported observability tool:

    system_cpu_load_average_15m
    system_cpu_load_average_1m
    system_cpu_load_average_5m
    system_filesystem_usage{state="free", type="ext4"}
    system_filesystem_usage{state="reserved", type="ext4"}
    system_filesystem_usage{state="used", type="ext4"}
    system_memory_utilization{state="buffered"}
    system_memory_utilization{state="cached"}
    system_memory_utilization{state="free"}
    system_memory_utilization{state="slab_reclaimable"}
    system_memory_utilization{state="slab_unreclaimable"}
    system_memory_utilization{state="used"} 
    system.network.io

Available CCG metrics are:

    ccg_healthy
    ccg_queue_healthy
    ccg_sufficient_uptime{"type":"gauge","count":1,"sum":1.0,"min":1.0,"max":1.0,"latest":1.0}

Troubleshooting VA Metrics Export Configuration

The following list describes some common issues with exporting VA metrics to your observability tool and their solutions.

  1. If otel_agent failed to start, check to be sure you've added enableOTELAgent: true at the bottom of the /home/sailpoint/config.yaml file. You can also check sudo docker logs otel_agent for errors.

  2. Error logs in /home/sailpoint/log/otel_agent.log may indicate that the API key or endpoint URL is wrong or that it may have extraneous details.

    For example, OTLP_API_KEY should only mention the API key. Grafana Lab may provide additional information with the API key, such as:

        OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic nlJam9pY0hKdlpDMWhjQzF643434sbjE5".

    In this case, OTLP_HEADERS="Authorization=Basic is already managed in the otel-agent-config.yaml, so there is no need for that information here.

    Remove the excess information in the otel.env file so that it only includes OTLP_API_KEY="nlJam9pY0hKdlpDMWhjQzF643434sbjE", then restart the otel_agent service using the command sudo systemctl restart otel_agent.

  3. If sudo docker logs otel_agent has an unknown OTLP backend and no information for the Detected backend, then the otel_agent is not able to detect the observability tool because the endpoint URL is wrong. Check and correct the endpoint URL, then restart otel_agent.

  4. The otel_agent specifically looks for the keywords OTLP_ENDPOINT and OTLP_API_KEY in the otel.env file. If these variables are missing or if you erase all contents from the otel.env file and restart the otel_agent, it will load an empty configuration, as shown below.

    ##############################
# OTLP Configuration
##############################
# Set these variables to export metrics to an OTLP-compatible backend.
OTLP_ENDPOINT=""      
OTLP_API_KEY="" 
# Example: OTLP_ENDPOINT="https://otlp.nr-data.net:4318" 
# Example: OTLP_API_KEY="ABCD1234567890"
    Running the agent without these values will send metrics to your SailPoint tenant rather than to your observability tool.

Exporting VA Logs

You can configure a virtual appliance to send systemd logs to the following tools:

  • Splunk HTTP event collector (HEC)
  • CloudWatch Logs
  • Datadog

Configuring VA Log Exports

To configure VA log exports:

  1. Ensure network connectivity between the VA and the monitoring tool.
  2. Create or gather any tokens, URLs, or keys required to configure your specific monitoring tool.
  3. Create a host-logging.yaml file in the VA's /home/sailpoint directory.
  4. Enter the key information for your tool as shown in the following example.
  5. Wait for the VA to detect the configured host-logging.yaml.
  6. Confirm in the monitoring tool that the logs are appearing.

Example host-logging.yaml

---
config:
  enableSailPointLogging: true

loggers:
  - name: splunk_hec
    type: splunk_hec
    hec_host: 172.31.31.31
    hec_port: 8383
    hec_token: quzkajkfkdjafk
    insecure_ssl: true
  - name: cwl
    type: cwlogs
    log_group_name: sailpoint
    region: us-east-1
  - name: thedog
    type: datadog
    api_key: 1234-1234-1234
    tags:
      customer: foobar

where:

  • name is a valid YAML string.
  • type is one of the following:
    • splunk_hec
    • cwlogs
    • datadog

Global Config Keys

Key Required Description
enableSailPointLogging No Enables export of the SailPoint container logs in addition to the Flatcar OS system journal

Monitoring Tool Keys

Splunk HEC

Type: splunk_hec

Key Required Description
hec_host Yes Hostname of HEC host
hec_port Yes TCP port HEC is listening on
hec_token Yes HEC token (from Splunk)
index No Index targeted
insecure_ssl No Bypasses certificate validity check (for development/test purposes)

CloudWatch Logs

Type: cwlogs

Key Required Description
log_group_name Yes CloudWatch Logs Group Name
log_stream_name Yes CloudWatch Logs Stream Name
region Yes AWS region CloudWatch Logs Group resides in

Datadog

Type: datadog

Key Required Description
api_key Yes Datadog API key
host No Datadog API host target (defaults to US/Datadog Standard)
dd_source No Sets the dd_source field
dd_sourcecategory No Sets the dd_sourcecategory field
tags No Sets additional tags (should be in key: value form)

Documentation Feedback

Feedback is provided as an informational resource only and does not form part of SailPoint’s official product documentation. SailPoint does not warrant or make any guarantees about the feedback (including without limitation as to its accuracy, relevance, or reliability). All feedback is subject to the terms set forth at https://developer.sailpoint.com/discuss/tos.