LiteLLM

Plugin: go.d.plugin Module: prometheus

Overview

Monitor LiteLLM gateway workload, routing, provider performance, usage, cost, governance, and internal dependencies.

The built-in profile follows requests through proxy admission, in-flight work and queueing, guardrails and LiteLLM overhead, routing and fallback, deployment selection, and the provider API response. Side branches organize response and provider caches, tokens and media, spend and budgets, rate limits, MCP tools, managed batches and files, callback failures, Redis/PostgreSQL operations, spend-update queues, and the single-process Python/process runtime. Optional charts appear when their callbacks, features, or runtime mode export metrics.

Each context represents one monitored entity type. Labels identifying that entity form chart instances; bounded aspects such as status, stream mode, service tier, and validated rate-limit classification become dimensions; ownership and descriptive details remain filterable chart metadata. When LiteLLM configuration omits a supported optional dimension label, the same context and instance remain available with an unclassified dimension.

Additive counters and histogram components always produce complete service or capability views. Selected additional views break them down by deployment, provider/model, route, accounting entity, guardrail/hook, MCP server/tool, managed batch, or callback when LiteLLM exports the required identity labels. Point-in-time gauges retain their complete exported identity, including multiprocess pid, so distinct states, limits, budgets, queues, and last-seen values are not summed together. Deployment-configured custom metadata and tag_* labels remain part of complete gauge identity when LiteLLM emits them. Additive service views intentionally roll those deployment-controlled labels up unless a dedicated entity breakdown is defined.

Netdata periodically scrapes the authenticated LiteLLM Prometheus /metrics endpoint and applies the built-in litellm profile.

This collector is supported on all platforms.

This collector supports collecting metrics from multiple instances of this integration, including remote instances.

Default Behavior

Auto-Detection

By default, it detects instances running on the local host by trying to connect to known ports that are allocated to exporters.

Limits

The default configuration for this integration does not impose any limits on data collection.

Performance Impact

The default configuration for this integration is not expected to impose a significant performance impact on the system.

Setup

You can configure the prometheus collector in two ways:

Method	Best for	How to
UI	Fast setup without editing files	Go to Nodes → Configure this node → Collectors → Jobs, search for prometheus, then click + to add a job.
File	If you prefer configuring via file, or need to automate deployments (e.g., with Ansible)	Edit go.d/prometheus.conf and add a job.

important

UI configuration requires paid Netdata Cloud plan.

Prerequisites

Enable LiteLLM Prometheus metrics

Enable the LiteLLM Prometheus callback and configure the Netdata job with access to its /metrics endpoint. LiteLLM's prometheus_metrics_config controls which metrics and labels the endpoint exports. Service views remain complete when optional dimension labels are omitted, and optional entity views appear only when their identity labels are available. Gauge identity follows every non-redundant label required to distinguish the emitted state, including configured custom metadata, custom tags, and the opt-in user email/alias budget labels. Arbitrary configured label names are not turned into dimensions because they are deployment-defined, unbounded classifications rather than one stable bounded aspect. Size max_time_series and max_time_series_per_metric for the exported surface.

Configuration

Options

The following options can be defined globally: update_every, autodetection_retry.

Config options

Group	Option	Description	Default	Required
Collection	update_every	Data collection interval (seconds).	10	no
	autodetection_retry	Autodetection retry interval (seconds). Set 0 to disable.	0	no
Target	url	Target endpoint URL.		yes
	timeout	HTTP request timeout (seconds).	10	no
	expected_prefix	If set, the job's check passes only when at least one post-job, pre-profile metric name starts with this prefix. Guards against scraping an unexpected endpoint; profile-owned relabeling cannot satisfy it.		no
Customization	app	Application name used as the app segment of chart contexts (prometheus.<app>.<metric>). When unset, it is taken from a matched profile, otherwise it falls back to the job name.		no
Filters	selector	Time series selector (filter).		no
Limits	max_time_series	Global time series limit applied after job and profile relabeling. If the final output exceeds it, the data is not processed.	2000	no
	max_time_series_per_metric	Per-metric time series limit applied to final metric families. Metrics exceeding it are skipped.	200	no
Customization	fallback_type	Job-level fallback type overrides for untyped metrics.		no
	relabeling	Job-owned Prometheus-compatible metric relabeling, applied before profile selection.		no
	profiles	Curated, exporter-specific chart profiles with optional untyped classification, profile-owned normalization, and scoped fallback-chart policy. User profiles may constrain unmatched fallback charts; stock profiles preserve unknown future families. Disable profiles with mode none.	auto	no
HTTP Auth	username	Username for Basic HTTP authentication.		no
	password	Password for Basic HTTP authentication.		no
	bearer_token_file	Path to a file containing a bearer token (used for Authorization: Bearer).		no
TLS	tls_skip_verify	Skip TLS certificate and hostname verification (insecure).	no	no
	tls_ca	Path to CA bundle used to validate the server certificate.		no
	tls_cert	Path to client TLS certificate (for mTLS).		no
	tls_key	Path to client TLS private key (for mTLS).		no
Proxy	proxy_url	HTTP proxy URL.		no
	proxy_username	Username for proxy Basic HTTP authentication.		no
	proxy_password	Password for proxy Basic HTTP authentication.		no
Request	method	HTTP method to use.	GET	no
	body	Request body (e.g., for POST/PUT).		no
	headers	Additional HTTP headers (one per line as key: value).		no
	not_follow_redirects	Do not follow HTTP redirects.	no	no
	force_http2	Force HTTP/2 (including h2c over TCP).	no	no
Virtual Node	vnode	Associates this data collection job with a Virtual Node.		no

selector

This option allows you to filter out unwanted time series. Only metrics matching the selector will be collected.

• Logic: (pattern1 OR pattern2) AND !(pattern3 or pattern4)
• Pattern syntax: selector.
• Option syntax:

selector:
  allow:
    - pattern1
    - pattern2
  deny:
    - pattern3
    - pattern4

fallback_type

This job option allows you to process untyped metrics as Counter or Gauge instead of ignoring them. Classification uses the post-job, pre-profile metric name. Profile relabeling preserves the selected type but cannot create or change it by renaming the final metric.

Selected profiles may provide exporter-owned fallback_type defaults inside their own match scope. Job gauge rules take precedence over job counter rules, and both job rule sets take precedence over every profile rule. Use them for deployment-specific overrides rather than exporter behavior that belongs in a profile. Keep patterns narrow: a broad job rule such as gauge: ['*'] overrides profile counter classifications. Blank patterns and patterns with leading or trailing whitespace are rejected.

• Metric name pattern syntax: shell file name pattern.
• Option syntax:

fallback_type:
  counter:
    - metric_name_pattern1
    - metric_name_pattern2
  gauge:
    - metric_name_pattern3
    - metric_name_pattern4

relabeling

A list of job-owned relabeling blocks, applied after selector and before profile selection. Each block applies a list of Prometheus metric_relabel_configs rules to the metrics whose name matches match. Profiles may own the same block format for exporter normalization after selection. See the relabeling reference for the full action set and more examples.

• match: Netdata simple patterns matched against the full metric name — including any _bucket/_sum/_count suffix, so prefer globs like app_lat* over an exact app_lat (space-separated; * matches any sequence, ? any character, a leading ! negates). Use * to target every metric. Required.
• metric_relabel_configs: Prometheus relabel rules (source_labels, separator, regex, modulus, target_label, replacement, action), applied in order to the scraped samples before charts are built.

Relabeling that would corrupt a histogram or summary — splitting it, dropping a component, mutating the le/quantile label, or merging two families — is rejected.

relabeling:
  - match: 'http_*'
    metric_relabel_configs:
      - source_labels: [code]
        regex: '(\d)\d\d'
        target_label: code_class
        replacement: '${1}xx'

profiles

Profiles ship curated charts for recognized exporters -- see the profile format for the file format and how to author your own. profiles.mode selects them:

• auto (default): every profile whose match hits at least one scraped metric.
• exact: only the profiles named in mode_exact.entries (each must match, or the job fails its check).
• combined: auto plus the profiles named in mode_combined.entries.
• none: no profiles — generic autogen charts only (the pre-profile behavior).

Selection uses post-job, pre-profile family names. A selected profile may carry fallback_type rules that classify untyped scalar families inside its match scope and relabeling blocks that normalize matching source families automatically before chart routing. Job fallback policy takes precedence; conflicting profile rules use the same ordering as normalization. Each original family is processed only by the first applicable profile normalizer: profile-name order in auto, configured entry order in exact, and configured entries followed by remaining auto profiles in name order in combined. Later profile pipelines do not see the family. All selected templates consume the same final names and labels; the collector does not create a private metric stream per profile.

Only the block matching the selected mode (mode_exact or mode_combined) is read; entries under the other block are ignored. Metrics not covered by an authored profile chart keep their generic autogen charts unless an applicable profile autogen.selector rejects them. Every selector is limited to its profile's match scope; when scopes overlap, every applicable selector must accept the series. This changes fallback charts only; use selector or a relabeling drop rule to discard samples. Stock profiles leave unknown future families eligible for generic fallback; closed fallback selectors are a user-owned deployment policy, not a stock-profile authoring pattern.

profiles:
  mode: exact
  mode_exact:
    entries:
      - name: haproxy

via UI

Configure the prometheus collector from the Netdata web interface:

1. Go to Nodes.
2. Select the node where you want the prometheus data-collection job to run and click the ⚙ (Configure this node). That node will run the data collection.
3. The Collectors → Jobs view opens by default.
4. In the Search box, type prometheus (or scroll the list) to locate the prometheus collector.
5. Click the + next to the prometheus collector to add a new job.
6. Fill in the job fields, then click Test to verify the configuration and Submit to save.
   • Test runs the job with the provided settings and shows whether data can be collected.
   • If it fails, an error message appears with details (for example, connection refused, timeout, or command execution errors), so you can adjust and retest.

via File

The configuration file name for this integration is go.d/prometheus.conf.

The file format is YAML. Generally, the structure is:

update_every: 1
autodetection_retry: 0
jobs:
  - name: some_name1
  - name: some_name2

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config go.d/prometheus.conf

Examples

LiteLLM

Collect the complete LiteLLM surface with automatically matched application, process-runtime, and Python-GC profiles.

jobs:
  - name: litellm
    url: http://127.0.0.1:4000/metrics
    expected_prefix: litellm_

Alerts

There are no alerts configured by default for this integration.

Metrics

The built-in Prometheus profiles on this page map Prometheus metrics into 109 curated Netdata charts across the primary and applicable supporting profiles. The tables are generated from the same profile design and runtime chart contracts used by the Agent.

Eligible metrics that are not covered by a curated chart, including future exporter metrics, can still be collected through the generic Prometheus autogeneration behavior. This catalogue describes curated profile coverage; it is not an allowlist of every metric that the collector can render.

LiteLLM

Curated gateway, routing, provider, usage, cost, governance, dependency, and runtime metrics from LiteLLM.

Gateway

Prometheus metric	Netdata chart	Dimension	Unit	Scope
litellm_proxy_total_requests_metric_total	Gateway / Client Traffic — Client Request Outcomes	requests	requests/s	route, status_class
litellm_proxy_failed_requests_metric_total	Gateway / Client Traffic — Client Failure Causes	failures	failures/s	status_class, exception_class
litellm_proxy_failed_requests_metric_total{rate_limit_category=".+",rate_limit_type=".+"}	Gateway / Client Traffic — Client Rate-Limit Attribution	failures	failures/s	rate_limit_category, rate_limit_type
litellm_in_flight_requests	Gateway / Client Traffic — In-Flight Requests	requests	requests	collector job service
litellm_request_queue_time_seconds_bucket	Gateway / Admission Queue — Request Queue-Time Distribution	matching series	observations/s	model_id
litellm_request_queue_time_seconds_count	Gateway / Admission Queue — Queued Request Measurements	requests	requests/s	model_id
litellm_request_queue_time_seconds_sum	Gateway / Admission Queue — Accumulated Request Queue Time	time	seconds/s	model_id
litellm_request_total_latency_metric_bucket	Gateway / Request Latency — End-to-End Request-Latency Distribution	matching series	observations/s	model_id, service_tier
litellm_request_total_latency_metric_count	Gateway / Request Latency — End-to-End Request Measurements	requests	requests/s	model_id, service_tier
litellm_request_total_latency_metric_sum	Gateway / Request Latency — Accumulated End-to-End Request Time	time	seconds/s	model_id, service_tier
litellm_overhead_latency_metric_bucket	Gateway / Processing Overhead — LiteLLM Overhead-Latency Distribution	matching series	observations/s	model_id
litellm_overhead_latency_metric_count	Gateway / Processing Overhead — LiteLLM Overhead Measurements	requests	requests/s	model_id
litellm_overhead_latency_metric_sum	Gateway / Processing Overhead — Accumulated LiteLLM Overhead	time	seconds/s	model_id
litellm_overhead_with_guardrails_latency_metric_bucket	Gateway / Processing Overhead — Guardrail-Inclusive Overhead Distribution	matching series	observations/s	model_id
litellm_overhead_with_guardrails_latency_metric_count	Gateway / Processing Overhead — Guardrail-Inclusive Overhead Measurements	requests	requests/s	model_id
litellm_overhead_with_guardrails_latency_metric_sum	Gateway / Processing Overhead — Accumulated Guardrail-Inclusive Overhead	time	seconds/s	model_id

Routing and Deployments

Prometheus metric	Netdata chart	Dimension	Unit	Scope
litellm_deployment_total_requests_total	Routing and Deployments / Deployment Requests — Deployment Request Workload	requests	requests/s	model_id
litellm_deployment_success_responses_total	Routing and Deployments / Deployment Requests — Classified Deployment Outcomes	successful	requests/s	model_id
litellm_deployment_failure_responses_total	Routing and Deployments / Deployment Requests — Classified Deployment Outcomes	failed	requests/s	model_id
litellm_deployment_failure_responses_total	Routing and Deployments / Deployment Requests — Deployment Failure Causes	failures	failures/s	model_id, status_class, exception_class
litellm_deployment_cooled_down_total	Routing and Deployments / Deployment Health — Deployment Cooldowns	cooldowns	events/s	model_id, status_class
litellm_deployment_state	Routing and Deployments / Deployment Health — Deployment State	state	{status}	model_id
litellm_deployment_rpm_limit	Routing and Deployments / Deployment Capacity — Deployment Request Limit	limit	requests/min	model_id
litellm_deployment_tpm_limit	Routing and Deployments / Deployment Capacity — Deployment Token Limit	limit	tokens/min	model_id
litellm_remaining_requests_metric	Routing and Deployments / Deployment Capacity — Provider Remaining Requests	remaining	requests	model_id
litellm_remaining_tokens_metric	Routing and Deployments / Deployment Capacity — Provider Remaining Tokens	remaining	tokens	model_id
litellm_deployment_latency_per_output_token_bucket	Routing and Deployments / Output-Token Latency — Deployment Output-Token Latency Distribution	matching series	observations/s	model_id
litellm_deployment_latency_per_output_token_count	Routing and Deployments / Output-Token Latency — Deployment Output-Token Latency Measurements	requests	requests/s	model_id
litellm_deployment_latency_per_output_token_sum	Routing and Deployments / Output-Token Latency — Accumulated Deployment Output-Token Latency	time	seconds/s	model_id
litellm_deployment_successful_fallbacks_total	Routing and Deployments / Fallbacks — Fallback Outcomes	successful	fallbacks/s	requested_model, fallback_model
litellm_deployment_failed_fallbacks_total	Routing and Deployments / Fallbacks — Fallback Outcomes	failed	fallbacks/s	requested_model, fallback_model

Provider API

Prometheus metric	Netdata chart	Dimension	Unit	Scope
litellm_llm_api_latency_metric_bucket	Provider API — Provider API Latency Distribution	matching series	observations/s	model_id, service_tier
litellm_llm_api_latency_metric_count	Provider API — Provider API Latency Measurements	requests	requests/s	model_id, service_tier
litellm_llm_api_latency_metric_sum	Provider API — Accumulated Provider API Latency	time	seconds/s	model_id, service_tier
litellm_llm_api_time_to_first_token_metric_bucket	Provider API — Provider Time-to-First-Token Distribution	matching series	observations/s	model_id, service_tier
litellm_llm_api_time_to_first_token_metric_count	Provider API — Provider Time-to-First-Token Measurements	requests	requests/s	model_id, service_tier
litellm_llm_api_time_to_first_token_metric_sum	Provider API — Accumulated Provider Time-to-First-Token	time	seconds/s	model_id, service_tier

Usage and Cost

Prometheus metric	Netdata chart	Dimension	Unit	Scope
litellm_input_tokens_metric_total	Usage and Cost / By Deployment — Request Token Throughput	input	tokens/s	model_id
litellm_output_tokens_metric_total	Usage and Cost / By Deployment — Request Token Throughput	output	tokens/s	model_id
litellm_total_tokens_metric_total	Usage and Cost / By Deployment — Reported Total Token Throughput	total	tokens/s	model_id
litellm_spend_metric_total	Usage and Cost / By Deployment — LLM Spend	spend	USD/s	model_id, service_tier
litellm_input_audio_tokens_metric_total	Usage and Cost / By Deployment — Audio and Reasoning Token Throughput	input_audio	tokens/s	model_id
litellm_output_audio_tokens_metric_total	Usage and Cost / By Deployment — Audio and Reasoning Token Throughput	output_audio	tokens/s	model_id
litellm_output_reasoning_tokens_metric_total	Usage and Cost / By Deployment — Audio and Reasoning Token Throughput	output_reasoning	tokens/s	model_id
litellm_images_generated_metric_total	Usage and Cost / By Deployment — Generated Images	images	images/s	model_id
litellm_video_duration_seconds_metric_total	Usage and Cost / By Deployment — Generated Video Duration	duration	seconds/s	model_id
litellm_input_tokens_metric_total	Usage and Cost / By API Key — Request Token Throughput by API Key	input	tokens/s	hashed_api_key
litellm_output_tokens_metric_total	Usage and Cost / By API Key — Request Token Throughput by API Key	output	tokens/s	hashed_api_key
litellm_total_tokens_metric_total	Usage and Cost / By API Key — Reported Total Token Throughput by API Key	total	tokens/s	hashed_api_key
litellm_spend_metric_total	Usage and Cost / By API Key — LLM Spend by API Key	spend	USD/s	hashed_api_key
litellm_input_tokens_metric_total	Usage and Cost / By Team — Request Token Throughput by Team	input	tokens/s	team
litellm_output_tokens_metric_total	Usage and Cost / By Team — Request Token Throughput by Team	output	tokens/s	team
litellm_total_tokens_metric_total	Usage and Cost / By Team — Reported Total Token Throughput by Team	total	tokens/s	team
litellm_spend_metric_total	Usage and Cost / By Team — LLM Spend by Team	spend	USD/s	team
litellm_input_tokens_metric_total	Usage and Cost / By User — Request Token Throughput by User	input	tokens/s	user
litellm_output_tokens_metric_total	Usage and Cost / By User — Request Token Throughput by User	output	tokens/s	user
litellm_total_tokens_metric_total	Usage and Cost / By User — Reported Total Token Throughput by User	total	tokens/s	user
litellm_spend_metric_total	Usage and Cost / By User — LLM Spend by User	spend	USD/s	user
litellm_input_tokens_metric_total	Usage and Cost / By Organization — Request Token Throughput by Organization	input	tokens/s	org_id
litellm_output_tokens_metric_total	Usage and Cost / By Organization — Request Token Throughput by Organization	output	tokens/s	org_id
litellm_total_tokens_metric_total	Usage and Cost / By Organization — Reported Total Token Throughput by Organization	total	tokens/s	org_id
litellm_spend_metric_total	Usage and Cost / By Organization — LLM Spend by Organization	spend	USD/s	org_id
litellm_input_tokens_metric_total	Usage and Cost / By End User — Request Token Throughput by End User	input	tokens/s	end_user
litellm_output_tokens_metric_total	Usage and Cost / By End User — Request Token Throughput by End User	output	tokens/s	end_user
litellm_total_tokens_metric_total	Usage and Cost / By End User — Reported Total Token Throughput by End User	total	tokens/s	end_user
litellm_spend_metric_total	Usage and Cost / By End User — LLM Spend by End User	spend	USD/s	end_user
litellm_input_tokens_metric_total	Usage and Cost / By Requested Model — Request Token Throughput by Requested Model	input	tokens/s	requested_model
litellm_output_tokens_metric_total	Usage and Cost / By Requested Model — Request Token Throughput by Requested Model	output	tokens/s	requested_model
litellm_total_tokens_metric_total	Usage and Cost / By Requested Model — Reported Total Token Throughput by Requested Model	total	tokens/s	requested_model
litellm_spend_metric_total	Usage and Cost / By Requested Model — LLM Spend by Requested Model	spend	USD/s	requested_model

Caching

Prometheus metric	Netdata chart	Dimension	Unit	Scope
litellm_cache_hits_metric_total	Caching — Cache Outcomes	hits	events/s	model_id
litellm_cache_misses_metric_total	Caching — Cache Outcomes	misses	events/s	model_id
litellm_cached_tokens_metric_total	Caching — LiteLLM Cached Token Throughput	cached	tokens/s	model_id
litellm_input_cached_tokens_metric_total	Caching — Request Input-Cache Token Throughput	read	tokens/s	model_id
litellm_input_cache_creation_tokens_metric_total	Caching — Request Input-Cache Token Throughput	creation	tokens/s	model_id
litellm_provider_cache_read_input_tokens_metric_total	Caching — Provider Cache Token Throughput	read	tokens/s	model_id
litellm_provider_cache_creation_input_tokens_metric_total	Caching — Provider Cache Token Throughput	creation	tokens/s	model_id

Governance

Prometheus metric	Netdata chart	Dimension	Unit	Scope
litellm_remaining_api_key_budget_metric	Governance / API Keys — API-Key Remaining Budget	remaining	USD	hashed_api_key
litellm_api_key_max_budget_metric	Governance / API Keys — API-Key Budget Limit	limit	USD	hashed_api_key
litellm_api_key_budget_remaining_hours_metric	Governance / API Keys — API-Key Budget Reset Window	remaining	hours	hashed_api_key
litellm_remaining_api_key_requests_for_model	Governance / API Keys — API-Key Model Request Headroom	remaining	requests	hashed_api_key, model
litellm_remaining_api_key_tokens_for_model	Governance / API Keys — API-Key Model Token Headroom	remaining	tokens	hashed_api_key, model
litellm_remaining_org_budget_metric	Governance / Organizations — Organization Remaining Budget	remaining	USD	org_id
litellm_org_max_budget_metric	Governance / Organizations — Organization Budget Limit	limit	USD	org_id
litellm_org_budget_remaining_hours_metric	Governance / Organizations — Organization Budget Reset Window	remaining	hours	org_id
litellm_remaining_team_budget_metric	Governance / Teams — Team Remaining Budget	remaining	USD	team
litellm_team_max_budget_metric	Governance / Teams — Team Budget Limit	limit	USD	team
litellm_team_budget_remaining_hours_metric	Governance / Teams — Team Budget Reset Window	remaining	hours	team
litellm_team_members_metric	Governance / Teams — Team Membership	members	members	team
litellm_remaining_user_budget_metric	Governance / Users — User Remaining Budget	remaining	USD	user
litellm_user_max_budget_metric	Governance / Users — User Budget Limit	limit	USD	user
litellm_user_budget_remaining_hours_metric	Governance / Users — User Budget Reset Window	remaining	hours	user
litellm_provider_remaining_budget_metric	Governance / Providers — Provider Remaining Budget	remaining	USD	api_provider

Guardrails

Prometheus metric	Netdata chart	Dimension	Unit	Scope
litellm_guardrail_latency_seconds_count	Guardrails — Guardrail Invocations	invocations	invocations/s	guardrail_name, hook_type, status
litellm_guardrail_errors_total	Guardrails — Guardrail Errors	errors	errors/s	guardrail_name, hook_type, error_type
litellm_guardrail_latency_seconds_bucket	Guardrails — Guardrail Latency Distribution	matching series	observations/s	guardrail_name, hook_type
litellm_guardrail_latency_seconds_sum	Guardrails — Accumulated Guardrail Latency	time	seconds/s	guardrail_name, hook_type

MCP Gateway

Prometheus metric	Netdata chart	Dimension	Unit	Scope
litellm_mcp_tool_calls_total	MCP Gateway — MCP Tool Calls	calls	calls/s	mcp_server_name, mcp_tool_name
litellm_mcp_tool_call_spend_metric_total	MCP Gateway — MCP Tool-Call Spend	spend	USD/s	mcp_server_name, mcp_tool_name

Managed Batches and Files

Prometheus metric	Netdata chart	Dimension	Unit	Scope
litellm_managed_batch_created_total	Managed Batches and Files — Managed Batches Created	created	batches/s	api_provider, model
litellm_check_batch_cost_jobs_processed_total	Managed Batches and Files — Successfully Cost-Tracked Batches	cost_tracked	batches/s	api_provider, model
litellm_check_batch_cost_errors_total	Managed Batches and Files — Batch-Cost Processing Errors	errors	errors/s	collector job service
litellm_managed_file_created_total	Managed Batches and Files — Managed Files Created	created	files/s	api_provider, model
litellm_managed_file_deleted_total	Managed Batches and Files — Managed File Deletions	deletions	files/s	result
litellm_managed_batch_duration_seconds_bucket	Managed Batches and Files — Managed Batch Duration Distribution	matching series	observations/s	api_provider, model
litellm_managed_batch_duration_seconds_count	Managed Batches and Files — Completed Managed Batches	completed	batches/s	api_provider, model
litellm_managed_batch_duration_seconds_sum	Managed Batches and Files — Accumulated Managed Batch Duration	time	seconds/s	api_provider, model

Inventory and Callbacks

Prometheus metric	Netdata chart	Dimension	Unit	Scope
litellm_total_users	Inventory and Callbacks — Total Users	users	users	collector job
litellm_active_users	Inventory and Callbacks — Billable Users	users	users	collector job
litellm_teams_count	Inventory and Callbacks — Teams	teams	teams	collector job
litellm_callback_logging_failures_metric_total	Inventory and Callbacks — Callback Logging Failures	failures	failures/s	callback_name

Internal Services

Prometheus metric	Netdata chart	Dimension	Unit	Scope
litellm_auth_latency_count	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_batch_write_to_db_latency_count	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_postgres_latency_count	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_proxy_pre_call_latency_count	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_redis_latency_count	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_redis_daily_org_spend_update_queue_latency_count	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_redis_daily_tag_spend_update_queue_latency_count	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_redis_daily_team_spend_update_queue_latency_count	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_reset_budget_job_latency_count	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_router_latency_count	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_self_latency_count	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_auth_failed_requests_total	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_batch_write_to_db_failed_requests_total	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_postgres_failed_requests_total	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_proxy_pre_call_failed_requests_total	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_redis_failed_requests_total	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_redis_daily_org_spend_update_queue_failed_requests_total	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_redis_daily_tag_spend_update_queue_failed_requests_total	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_redis_daily_team_spend_update_queue_failed_requests_total	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_reset_budget_job_failed_requests_total	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_router_failed_requests_total	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_self_failed_requests_total	Internal Services — Internal Service Request Outcomes	values of label outcome	requests/s	service
litellm_auth_failed_requests_total	Internal Services — Internal Service Failure Causes	values of label service	failures/s	function_name, error_class
litellm_batch_write_to_db_failed_requests_total	Internal Services — Internal Service Failure Causes	values of label service	failures/s	function_name, error_class
litellm_postgres_failed_requests_total	Internal Services — Internal Service Failure Causes	values of label service	failures/s	function_name, error_class
litellm_proxy_pre_call_failed_requests_total	Internal Services — Internal Service Failure Causes	values of label service	failures/s	function_name, error_class
litellm_redis_failed_requests_total	Internal Services — Internal Service Failure Causes	values of label service	failures/s	function_name, error_class
litellm_redis_daily_org_spend_update_queue_failed_requests_total	Internal Services — Internal Service Failure Causes	values of label service	failures/s	function_name, error_class
litellm_redis_daily_tag_spend_update_queue_failed_requests_total	Internal Services — Internal Service Failure Causes	values of label service	failures/s	function_name, error_class
litellm_redis_daily_team_spend_update_queue_failed_requests_total	Internal Services — Internal Service Failure Causes	values of label service	failures/s	function_name, error_class
litellm_reset_budget_job_failed_requests_total	Internal Services — Internal Service Failure Causes	values of label service	failures/s	function_name, error_class
litellm_router_failed_requests_total	Internal Services — Internal Service Failure Causes	values of label service	failures/s	function_name, error_class
litellm_self_failed_requests_total	Internal Services — Internal Service Failure Causes	values of label service	failures/s	function_name, error_class
litellm_auth_latency_bucket	Internal Services — Internal Service Latency Distribution	matching series	observations/s	service
litellm_batch_write_to_db_latency_bucket	Internal Services — Internal Service Latency Distribution	matching series	observations/s	service
litellm_postgres_latency_bucket	Internal Services — Internal Service Latency Distribution	matching series	observations/s	service
litellm_proxy_pre_call_latency_bucket	Internal Services — Internal Service Latency Distribution	matching series	observations/s	service
litellm_redis_latency_bucket	Internal Services — Internal Service Latency Distribution	matching series	observations/s	service
litellm_redis_daily_org_spend_update_queue_latency_bucket	Internal Services — Internal Service Latency Distribution	matching series	observations/s	service
litellm_redis_daily_tag_spend_update_queue_latency_bucket	Internal Services — Internal Service Latency Distribution	matching series	observations/s	service
litellm_redis_daily_team_spend_update_queue_latency_bucket	Internal Services — Internal Service Latency Distribution	matching series	observations/s	service
litellm_reset_budget_job_latency_bucket	Internal Services — Internal Service Latency Distribution	matching series	observations/s	service
litellm_router_latency_bucket	Internal Services — Internal Service Latency Distribution	matching series	observations/s	service
litellm_self_latency_bucket	Internal Services — Internal Service Latency Distribution	matching series	observations/s	service
litellm_auth_latency_sum	Internal Services — Accumulated Internal Service Latency	values of label measurement	seconds/s	service
litellm_batch_write_to_db_latency_sum	Internal Services — Accumulated Internal Service Latency	values of label measurement	seconds/s	service
litellm_postgres_latency_sum	Internal Services — Accumulated Internal Service Latency	values of label measurement	seconds/s	service
litellm_proxy_pre_call_latency_sum	Internal Services — Accumulated Internal Service Latency	values of label measurement	seconds/s	service
litellm_redis_latency_sum	Internal Services — Accumulated Internal Service Latency	values of label measurement	seconds/s	service
litellm_redis_daily_org_spend_update_queue_latency_sum	Internal Services — Accumulated Internal Service Latency	values of label measurement	seconds/s	service
litellm_redis_daily_tag_spend_update_queue_latency_sum	Internal Services — Accumulated Internal Service Latency	values of label measurement	seconds/s	service
litellm_redis_daily_team_spend_update_queue_latency_sum	Internal Services — Accumulated Internal Service Latency	values of label measurement	seconds/s	service
litellm_reset_budget_job_latency_sum	Internal Services — Accumulated Internal Service Latency	values of label measurement	seconds/s	service
litellm_router_latency_sum	Internal Services — Accumulated Internal Service Latency	values of label measurement	seconds/s	service
litellm_self_latency_sum	Internal Services — Accumulated Internal Service Latency	values of label measurement	seconds/s	service

Process runtime

Curated CPU, memory, file-descriptor, and lifecycle metrics exported by the monitored process.

Supporting profile for LiteLLM. Included whenever the LiteLLM profile is active and the endpoint exports process runtime metrics.

Process Runtime

Prometheus metric	Netdata chart	Dimension	Unit	Scope
process_cpu_seconds_total	Process Runtime — Process CPU Usage	used	cores	collector job process
process_resident_memory_bytes	Process Runtime — Process Resident Memory	resident	bytes	collector job process
process_virtual_memory_bytes	Process Runtime — Process Virtual Memory	virtual	bytes	collector job process
process_open_fds	Process Runtime — Open File Descriptors	open	fds	collector job process
process_max_fds	Process Runtime — File Descriptor Limit	limit	fds	collector job process

Python garbage collection

Curated collection, uncollectable-object, and collection-run metrics for each Python garbage-collector generation.

Supporting profile for LiteLLM. Included whenever the LiteLLM profile is active and the endpoint exports Python garbage-collection metrics.

Process Runtime

Prometheus metric	Netdata chart	Dimension	Unit	Scope
python_gc_objects_collected_total	Process Runtime / Python GC — Collected Objects	collected	objects/s	Python garbage-collector generation
python_gc_objects_uncollectable_total	Process Runtime / Python GC — Uncollectable Objects	uncollectable	objects/s	Python garbage-collector generation
python_gc_collections_total	Process Runtime / Python GC — Collections	collections	collections/s	Python garbage-collector generation

Troubleshooting

Debug Mode

Important: Debug mode is not supported for data collection jobs created via the UI using the Dyncfg feature.

To troubleshoot issues with the prometheus collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn't working.

• Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your system, open netdata.conf and look for the plugins setting under [directories].

  cd /usr/libexec/netdata/plugins.d/

• Switch to the netdata user.

  sudo -u netdata -s

• Run the go.d.plugin to debug the collector:

  ./go.d.plugin -d -m prometheus

  To debug a specific job:

  ./go.d.plugin -d -m prometheus -j jobName

Getting Logs

If you're encountering problems with the prometheus collector, follow these steps to retrieve logs and identify potential issues:

• Run the command specific to your system (systemd, non-systemd, or Docker container).
• Examine the output for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem.

System with systemd

Use the following command to view logs generated since the last Netdata service restart:

journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep prometheus

System without systemd

Locate the collector log file, typically at /var/log/netdata/collector.log, and use grep to filter for collector's name:

grep prometheus /var/log/netdata/collector.log

Note: This method shows logs from all restarts. Focus on the latest entries for troubleshooting current issues.

Docker Container

If your Netdata runs in a Docker container named "netdata" (replace if different), use this command:

docker logs netdata 2>&1 | grep prometheus

Disappearing or sparse metrics not clearing alerts

The Prometheus collector detects metrics that disappear from a successful scrape response. Generated charts and individual dimensions expire after their configured successful-cycle lifetime. An expired chart or dimension makes its alerts REMOVED; this is not a normal CLEAR transition and does not send a recovery notification. Export an explicit normal value (for example 0) whenever an alert needs a reliable recovery transition. A failed scrape does not advance the expiry lifetime; use the generic collector collection-failure alert to detect that separate condition.