Migrate from Prometheus/Grafana to OpenTelemetry#

The Prometheus/Grafana based monitoring setup will soon be deprecated in favor of the OpenTelemetry setup. This section provides the details required to update/migrate your Prometheus configuration and Grafana dashboard to continue monitoring with OpenTelemetry. Refer to Opentelemetry setup for the new setup before proceeding further.

Hint

Refer to Prometheus/Grafana-only section for the soon to be deprecated setup.

Update Prometheus configuration#

With a Prometheus-only setup, you need to set up a scrape_configs configuration or service discovery plugin to specify the targets for pulling metrics data. In the OpenTelemetry setup, each Pod pushes metrics to the OpenTelemetry Collector. The Prometheus configuration now only needs to scrape from the OpenTelemetry Collector to get all the data from OpenTelemetry-instrumented applications.

The new Prometheus configuration for the otel-collector Collector hostname is:

scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 500ms
    static_configs:
      - targets: ['otel-collector:8888'] # metrics from the collector itself
      - targets: ['otel-collector:8889'] # metrics collected from other applications

Update Grafana dashboard#

The OpenTelemetry Histogram provides quantile window buckets automatically (unlike the Prometheus Summary instrument). You need to manually configure the required quantile window. The quatile window metric will then be available as a separate time series metric.

In addition, the OpenTelemetry Counter/UpDownCounter instruments do not add the _total suffix to the base metric name.

To adapt Prometheus queries in Grafana:

  • Use the histogram_quantile function to query the average or desired quantile window time series data from Prometheus. For example, to view the 0.99 quantile of the jina_receiving_request_seconds metric over the last 10 minutes, use query histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[10m])).

  • Remove the _total prefix from the Counter/UpDownCounter metric names.

You can download a sample Grafana dashboard JSON file and import it into Grafana to get started with some pre-built graphs.

Hint

A list of available metrics is in the Flow Instrumentation section.