vLLM

Plugin: go.d.plugin Module: prometheus

Overview

Monitor vLLM inference-serving workload, latency, scheduling, token throughput, cache behavior, connector activity, and runtime health.

The built-in profiles follow requests through scheduling, prefill, decode, engine execution, and HTTP service boundaries. They separate model-engine, endpoint, parser, service-runtime, and Ray replica-worker identities, and organize KV cache residency and offloading, connector operations, speculative or diffusion decoding, and WebSocket activity under their causal owners. Optional charts appear when the corresponding vLLM features export metrics.

Netdata periodically scrapes either the vLLM server Prometheus /metrics endpoint or the Ray node metrics endpoint. Auto-selection applies one built-in vllm profile to native vllm:* and Ray ray_vllm_* metrics. The profile normalizes Ray metric names into the native vLLM namespace, preserves replica and worker identity, and suppresses Ray 2.48's deprecated unsuffixed counter aliases so each counter is presented once.

This collector is supported on all platforms.

This collector supports collecting metrics from multiple instances of this integration, including remote instances.

Default Behavior

Auto-Detection

By default, it detects instances running on the local host by trying to connect to known ports that are allocated to exporters.

Limits

The default configuration for this integration does not impose any limits on data collection.

Performance Impact

The default configuration for this integration is not expected to impose a significant performance impact on the system.

Setup

You can configure the prometheus collector in two ways:

Method	Best for	How to
UI	Fast setup without editing files	Go to Nodes → Configure this node → Collectors → Jobs, search for prometheus, then click + to add a job.
File	If you prefer configuring via file, or need to automate deployments (e.g., with Ansible)	Edit go.d/prometheus.conf and add a job.

important

UI configuration requires paid Netdata Cloud plan.

Prerequisites

Expose the vLLM metrics endpoint

Run a vLLM server with a reachable Prometheus /metrics endpoint, or configure a Ray deployment whose node metrics endpoint exposes ray_vllm_* families. For Ray, configure or discover each node's metrics export port; Ray commonly uses port 8080 when --metrics-export-port=8080 is set. Ray 2.48 exports an unsuffixed compatibility gauge beside most canonical _total counters unless RAY_EXPORT_COUNTER_AS_GAUGE=0 is set.

Configuration

Options

The following options can be defined globally: update_every, autodetection_retry.

Config options

Group	Option	Description	Default	Required
Collection	update_every	Data collection interval (seconds).	10	no
	autodetection_retry	Autodetection retry interval (seconds). Set 0 to disable.	0	no
Target	url	Target endpoint URL.		yes
	timeout	HTTP request timeout (seconds).	10	no
	expected_prefix	If set, the job's check passes only when at least one post-job, pre-profile metric name starts with this prefix. Guards against scraping an unexpected endpoint; profile-owned relabeling cannot satisfy it.		no
Customization	app	Application name used as the app segment of chart contexts (prometheus.<app>.<metric>). When unset, it is taken from a matched profile, otherwise it falls back to the job name.		no
Filters	selector	Time series selector (filter).		no
Limits	max_time_series	Global time series limit applied after job and profile relabeling. If the final output exceeds it, the data is not processed.	2000	no
	max_time_series_per_metric	Per-metric time series limit applied to final metric families. Metrics exceeding it are skipped.	200	no
Customization	fallback_type	Job-level fallback type overrides for untyped metrics.		no
	relabeling	Job-owned Prometheus-compatible metric relabeling, applied before profile selection.		no
	profiles	Curated, exporter-specific chart profiles with optional untyped classification, profile-owned normalization, and scoped fallback-chart policy. User profiles may constrain unmatched fallback charts; stock profiles preserve unknown future families. Disable profiles with mode none.	auto	no
HTTP Auth	username	Username for Basic HTTP authentication.		no
	password	Password for Basic HTTP authentication.		no
	bearer_token_file	Path to a file containing a bearer token (used for Authorization: Bearer).		no
TLS	tls_skip_verify	Skip TLS certificate and hostname verification (insecure).	no	no
	tls_ca	Path to CA bundle used to validate the server certificate.		no
	tls_cert	Path to client TLS certificate (for mTLS).		no
	tls_key	Path to client TLS private key (for mTLS).		no
Proxy	proxy_url	HTTP proxy URL.		no
	proxy_username	Username for proxy Basic HTTP authentication.		no
	proxy_password	Password for proxy Basic HTTP authentication.		no
Request	method	HTTP method to use.	GET	no
	body	Request body (e.g., for POST/PUT).		no
	headers	Additional HTTP headers (one per line as key: value).		no
	not_follow_redirects	Do not follow HTTP redirects.	no	no
	force_http2	Force HTTP/2 (including h2c over TCP).	no	no
Virtual Node	vnode	Associates this data collection job with a Virtual Node.		no

selector

This option allows you to filter out unwanted time series. Only metrics matching the selector will be collected.

• Logic: (pattern1 OR pattern2) AND !(pattern3 or pattern4)
• Pattern syntax: selector.
• Option syntax:

selector:
  allow:
    - pattern1
    - pattern2
  deny:
    - pattern3
    - pattern4

fallback_type

This job option allows you to process untyped metrics as Counter or Gauge instead of ignoring them. Classification uses the post-job, pre-profile metric name. Profile relabeling preserves the selected type but cannot create or change it by renaming the final metric.

Selected profiles may provide exporter-owned fallback_type defaults inside their own match scope. Job gauge rules take precedence over job counter rules, and both job rule sets take precedence over every profile rule. Use them for deployment-specific overrides rather than exporter behavior that belongs in a profile. Keep patterns narrow: a broad job rule such as gauge: ['*'] overrides profile counter classifications. Blank patterns and patterns with leading or trailing whitespace are rejected.

• Metric name pattern syntax: shell file name pattern.
• Option syntax:

fallback_type:
  counter:
    - metric_name_pattern1
    - metric_name_pattern2
  gauge:
    - metric_name_pattern3
    - metric_name_pattern4

relabeling

A list of job-owned relabeling blocks, applied after selector and before profile selection. Each block applies a list of Prometheus metric_relabel_configs rules to the metrics whose name matches match. Profiles may own the same block format for exporter normalization after selection. See the relabeling reference for the full action set and more examples.

• match: Netdata simple patterns matched against the full metric name — including any _bucket/_sum/_count suffix, so prefer globs like app_lat* over an exact app_lat (space-separated; * matches any sequence, ? any character, a leading ! negates). Use * to target every metric. Required.
• metric_relabel_configs: Prometheus relabel rules (source_labels, separator, regex, modulus, target_label, replacement, action), applied in order to the scraped samples before charts are built.

Relabeling that would corrupt a histogram or summary — splitting it, dropping a component, mutating the le/quantile label, or merging two families — is rejected.

relabeling:
  - match: 'http_*'
    metric_relabel_configs:
      - source_labels: [code]
        regex: '(\d)\d\d'
        target_label: code_class
        replacement: '${1}xx'

profiles

Profiles ship curated charts for recognized exporters -- see the profile format for the file format and how to author your own. profiles.mode selects them:

• auto (default): every profile whose match hits at least one scraped metric.
• exact: only the profiles named in mode_exact.entries (each must match, or the job fails its check).
• combined: auto plus the profiles named in mode_combined.entries.
• none: no profiles — generic autogen charts only (the pre-profile behavior).

Selection uses post-job, pre-profile family names. A selected profile may carry fallback_type rules that classify untyped scalar families inside its match scope and relabeling blocks that normalize matching source families automatically before chart routing. Job fallback policy takes precedence; conflicting profile rules use the same ordering as normalization. Each original family is processed only by the first applicable profile normalizer: profile-name order in auto, configured entry order in exact, and configured entries followed by remaining auto profiles in name order in combined. Later profile pipelines do not see the family. All selected templates consume the same final names and labels; the collector does not create a private metric stream per profile.

Only the block matching the selected mode (mode_exact or mode_combined) is read; entries under the other block are ignored. Metrics not covered by an authored profile chart keep their generic autogen charts unless an applicable profile autogen.selector rejects them. Every selector is limited to its profile's match scope; when scopes overlap, every applicable selector must accept the series. This changes fallback charts only; use selector or a relabeling drop rule to discard samples. Stock profiles leave unknown future families eligible for generic fallback; closed fallback selectors are a user-owned deployment policy, not a stock-profile authoring pattern.

profiles:
  mode: exact
  mode_exact:
    entries:
      - name: haproxy

via UI

Configure the prometheus collector from the Netdata web interface:

1. Go to Nodes.
2. Select the node where you want the prometheus data-collection job to run and click the ⚙ (Configure this node). That node will run the data collection.
3. The Collectors → Jobs view opens by default.
4. In the Search box, type prometheus (or scroll the list) to locate the prometheus collector.
5. Click the + next to the prometheus collector to add a new job.
6. Fill in the job fields, then click Test to verify the configuration and Submit to save.
   • Test runs the job with the provided settings and shows whether data can be collected.
   • If it fails, an error message appears with details (for example, connection refused, timeout, or command execution errors), so you can adjust and retest.

via File

The configuration file name for this integration is go.d/prometheus.conf.

The file format is YAML. Generally, the structure is:

update_every: 1
autodetection_retry: 0
jobs:
  - name: some_name1
  - name: some_name2

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config go.d/prometheus.conf

Examples

Native vLLM

Collect native vLLM server metrics with automatically matched application, FastAPI, process-runtime, and Python-GC profiles.

jobs:
  - name: vllm
    url: http://127.0.0.1:8000/metrics
    expected_prefix: 'vllm:'

vLLM on Ray

Collect vLLM families from a Ray node metrics endpoint. The automatically matched unified profile normalizes them into the native vLLM namespace, removes Ray 2.48's deprecated unsuffixed compatibility gauges, and removes vLLM's pre-canonical KV-offload duplicates. Other Ray system families remain available for generic collection or another matching profile.

Config

jobs:
  - name: vllm-ray
    url: http://127.0.0.1:8080
    expected_prefix: ray_vllm_

Alerts

There are no alerts configured by default for this integration.

Metrics

The built-in Prometheus profiles on this page map Prometheus metrics into 120 curated Netdata charts across the primary and applicable supporting profiles. The tables are generated from the same profile design and runtime chart contracts used by the Agent.

Eligible metrics that are not covered by a curated chart, including future exporter metrics, can still be collected through the generic Prometheus autogeneration behavior. This catalogue describes curated profile coverage; it is not an allowlist of every metric that the collector can render.

vLLM

Curated inference, scheduling, latency, token, cache, connector, engine, HTTP, and runtime metrics from vLLM and Ray.

Request Lifecycle

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:corrupted_requests_total	Request Lifecycle — Corrupted Requests	corrupted	requests/s	model_name and engine
vllm:time_to_first_token_seconds_bucket	Request Lifecycle — Time to First Token	matching series	observations/s	model_name and engine
vllm:e2e_request_latency_seconds_bucket	Request Lifecycle — End-to-End Request Latency	matching series	observations/s	model_name and engine
vllm:time_to_first_token_seconds_count	Request Lifecycle — Requests Reaching First Token	requests	requests/s	model_name and engine
vllm:time_to_first_token_seconds_sum	Request Lifecycle — Completed Pre-Response Time	time	seconds/s	model_name and engine
vllm:e2e_request_latency_seconds_sum	Request Lifecycle — Completed End-to-End Request Time	time	seconds/s	model_name and engine
vllm:request_params_n_bucket	Request Lifecycle — Requested Sequences	matching series	observations/s	model_name and engine
vllm:request_params_n_sum	Request Lifecycle — Requested Sequence Volume	sequences	sequences/s	model_name and engine
vllm:request_params_max_tokens_bucket	Request Lifecycle — Requested Token Limit	matching series	observations/s	model_name and engine
vllm:request_params_max_tokens_sum	Request Lifecycle — Requested Token Limits Sum	token_limits	tokens/s	model_name and engine
vllm:request_params_max_tokens_count	Request Lifecycle — Requests with Explicit Token Limit	requests	requests/s	model_name and engine
vllm:request_max_num_generation_tokens_bucket	Request Lifecycle — Maximum Generated Tokens	matching series	observations/s	model_name and engine
vllm:request_max_num_generation_tokens_sum	Request Lifecycle — Maximum Generated Tokens Sum	maximums	tokens/s	model_name and engine
vllm:request_params_n_count	Request Lifecycle — Parent Requests	requests	requests/s	model_name and engine
vllm:request_success_total	Request Lifecycle / Outcomes — Request Outcomes	requests	requests/s	model_name and engine and finished_reason

Scheduler

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:num_requests_running	Scheduler — Request State	running	requests	model_name and engine
vllm:num_requests_waiting_by_reason{reason="capacity"}	Scheduler — Request State	waiting_capacity	requests	model_name and engine
vllm:num_requests_waiting_by_reason{reason="deferred"}	Scheduler — Request State	waiting_deferred	requests	model_name and engine
vllm:num_preemptions_total	Scheduler — Request Preemptions	preemptions	preemptions/s	model_name and engine
vllm:engine_sleep_state	Scheduler — Engine Sleep State	values of label sleep_state	{status}	model_name and engine
vllm:request_queue_time_seconds_bucket	Scheduler — Request Queue Time	matching series	observations/s	model_name and engine
vllm:request_queue_time_seconds_sum	Scheduler — Completed Request Queue Time	time	seconds/s	model_name and engine

Prefill

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:prompt_tokens_by_source_total	Prefill — Prompt Tokens by Source	values of label source	tokens/s	model_name and engine
vllm:request_prompt_tokens_bucket	Prefill — Prompt Size	matching series	observations/s	model_name and engine
vllm:request_prefill_time_seconds_bucket	Prefill — Prefill Time	matching series	observations/s	model_name and engine
vllm:request_prefill_kv_computed_tokens_bucket	Prefill — Computed KV Tokens	matching series	observations/s	model_name and engine
vllm:request_prompt_tokens_sum	Prefill — Completed Request Prompt Volume	prompt_tokens	tokens/s	model_name and engine
vllm:request_prefill_kv_computed_tokens_sum	Prefill — Computed KV Token Volume	computed_tokens	tokens/s	model_name and engine
vllm:request_prefill_time_seconds_sum	Prefill — Completed Request Prefill Time	time	seconds/s	model_name and engine

Decode

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:generation_tokens_total	Decode — Generated Tokens	generated	tokens/s	model_name and engine
vllm:request_generation_tokens_bucket	Decode — Output Size	matching series	observations/s	model_name and engine
vllm:request_decode_time_seconds_bucket	Decode — Decode Time	matching series	observations/s	model_name and engine
vllm:inter_token_latency_seconds_bucket	Decode — Inter-Token Latency	matching series	observations/s	model_name and engine
vllm:request_time_per_output_token_seconds_bucket	Decode — Mean Output-Token Time	matching series	observations/s	model_name and engine
vllm:request_generation_tokens_sum	Decode — Output Token Volume	output_tokens	tokens/s	model_name and engine
vllm:request_decode_time_seconds_sum	Decode — Completed Request Decode Time	time	seconds/s	model_name and engine
vllm:inter_token_latency_seconds_count	Decode — Inter-Token Intervals	intervals	intervals/s	model_name and engine
vllm:inter_token_latency_seconds_sum	Decode — Inter-Token Interval Time	intervals	seconds/s	model_name and engine
vllm:request_time_per_output_token_seconds_sum	Decode — Accumulated Mean Output-Token Time	request_means	seconds/s	model_name and engine

Engine Execution

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:request_inference_time_seconds_bucket	Engine Execution — Inference Time	matching series	observations/s	model_name and engine
vllm:request_inference_time_seconds_sum	Engine Execution — Completed Request Inference Time	time	seconds/s	model_name and engine
vllm:iteration_tokens_total_bucket	Engine Execution — Tokens per Engine Step	matching series	observations/s	model_name and engine
vllm:iteration_tokens_total_count	Engine Execution — Engine Steps	steps	steps/s	model_name and engine
vllm:iteration_tokens_total_sum	Engine Execution — Engine Step Token Volume	tokens	tokens/s	model_name and engine
vllm:estimated_flops_per_gpu_total	Engine Execution — Estimated Compute per GPU	compute	GFLOP/s/GPU	model_name and engine
vllm:estimated_read_bytes_per_gpu_total	Engine Execution — Estimated Memory Bandwidth per GPU	read	GB/s/GPU	model_name and engine
vllm:estimated_write_bytes_per_gpu_total	Engine Execution — Estimated Memory Bandwidth per GPU	write	GB/s/GPU	model_name and engine

KV Cache

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:kv_cache_usage_perc	KV Cache — KV Cache Usage	used	percentage	model_name and engine
vllm:prefix_cache_queries_total	KV Cache — Local Prefix Cache	queries	tokens/s	model_name and engine
vllm:prefix_cache_hits_total	KV Cache — Local Prefix Cache	hits	tokens/s	model_name and engine
vllm:external_prefix_cache_queries_total	KV Cache — External Prefix Cache	queries	tokens/s	model_name and engine
vllm:external_prefix_cache_hits_total	KV Cache — External Prefix Cache	hits	tokens/s	model_name and engine
vllm:mm_cache_queries_total	KV Cache — Multimodal Cache	queries	items/s	model_name and engine
vllm:mm_cache_hits_total	KV Cache — Multimodal Cache	hits	items/s	model_name and engine

KV Cache Residency

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:kv_block_lifetime_seconds_bucket	KV Cache Residency — KV Block Lifetime	matching series	observations/s	model_name and engine
vllm:kv_block_idle_before_evict_seconds_bucket	KV Cache Residency — KV Block Idle Time Before Eviction	matching series	observations/s	model_name and engine
vllm:kv_block_reuse_gap_seconds_bucket	KV Cache Residency — KV Block Reuse Gap	matching series	observations/s	model_name and engine
vllm:kv_block_lifetime_seconds_count	KV Cache Residency — KV Cache Events	evictions	blocks/s	model_name and engine
vllm:kv_block_reuse_gap_seconds_count	KV Cache Residency — KV Cache Events	reuse_gaps	blocks/s	model_name and engine
vllm:kv_block_lifetime_seconds_sum	KV Cache Residency — Accumulated KV Residency Time	lifetime	seconds/s	model_name and engine
vllm:kv_block_idle_before_evict_seconds_sum	KV Cache Residency — Accumulated KV Residency Time	idle_before_eviction	seconds/s	model_name and engine
vllm:kv_block_reuse_gap_seconds_sum	KV Cache Residency — Accumulated KV Residency Time	reuse_gap	seconds/s	model_name and engine

KV Offloading

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:kv_offload_load_size_bucket	KV Offloading — KV Offload Load Size	matching series	observations/s	model_name and engine
vllm:kv_offload_store_size_bucket	KV Offloading — KV Offload Store Size	matching series	observations/s	model_name and engine
vllm:kv_offload_load_size_count	KV Offloading — KV Offload Transfer Operations	loads	operations/s	model_name and engine
vllm:kv_offload_store_size_count	KV Offloading — KV Offload Transfer Operations	stores	operations/s	model_name and engine
vllm:kv_offload_load_bytes_total	KV Offloading — KV Offload Transfer Throughput	loaded	bytes/s	model_name and engine
vllm:kv_offload_store_bytes_total	KV Offloading — KV Offload Transfer Throughput	stored	bytes/s	model_name and engine
vllm:kv_offload_load_time_total	KV Offloading — Completed KV Offload Transfer Time	load	seconds/s	model_name and engine
vllm:kv_offload_store_time_total	KV Offloading — Completed KV Offload Transfer Time	store	seconds/s	model_name and engine
vllm:kv_offload_lookup_sync_delay_seconds_bucket	KV Offloading — KV Offload Synchronous Lookup Delay	matching series	observations/s	model_name and engine
vllm:kv_offload_lookup_async_delay_seconds_bucket	KV Offloading — KV Offload Asynchronous Lookup Delay	matching series	observations/s	model_name and engine
vllm:kv_offload_lookup_sync_delay_seconds_count	KV Offloading — KV Offload Lookup Measurements	synchronous	lookups/s	model_name and engine
vllm:kv_offload_lookup_async_delay_seconds_count	KV Offloading — KV Offload Lookup Measurements	asynchronous	lookups/s	model_name and engine
vllm:kv_offload_lookup_sync_delay_seconds_sum	KV Offloading — KV Offload Lookup Delay Accumulation	synchronous	seconds/s	model_name and engine
vllm:kv_offload_lookup_async_delay_seconds_sum	KV Offloading — KV Offload Lookup Delay Accumulation	asynchronous	seconds/s	model_name and engine
vllm:kv_offload_cpu_cache_usage_perc	KV Offloading — CPU KV Cache Usage	total	percentage	model_name and engine
vllm:kv_offload_cpu_cache_write_usage_perc	KV Offloading — CPU KV Cache Usage	writes	percentage	model_name and engine
vllm:kv_offload_cpu_cache_read_usage_perc	KV Offloading — CPU KV Cache Usage	reads	percentage	model_name and engine
vllm:kv_offload_cpu_allocation_size_bucket	KV Offloading — CPU KV Allocation Size	matching series	observations/s	model_name and engine
vllm:kv_offload_cpu_allocation_size_count	KV Offloading — CPU KV Allocation Measurements	allocations	allocations/s	model_name and engine
vllm:kv_offload_cpu_allocation_size_sum	KV Offloading — CPU KV Allocation Volume	blocks	blocks/s	model_name and engine
vllm:kv_offload_allocation_failure_total	KV Offloading — KV Offload Admission Outcomes	allocation_failures	events/s	model_name and engine
vllm:kv_offload_stores_skipped_total	KV Offloading — KV Offload Admission Outcomes	stores_skipped	events/s	model_name and engine
vllm:kv_offload_tiering_lookup_sync_delay_seconds_bucket	KV Offloading — Tiered KV Offload Synchronous Lookup Delay	matching series	observations/s	model_name and engine
vllm:kv_offload_tiering_lookup_async_delay_seconds_bucket	KV Offloading — Tiered KV Offload Asynchronous Lookup Delay	matching series	observations/s	model_name and engine
vllm:kv_offload_tiering_lookup_sync_delay_seconds_count	KV Offloading — Tiered KV Offload Lookup Measurements	synchronous	lookups/s	model_name and engine
vllm:kv_offload_tiering_lookup_async_delay_seconds_count	KV Offloading — Tiered KV Offload Lookup Measurements	asynchronous	lookups/s	model_name and engine
vllm:kv_offload_tiering_lookup_sync_delay_seconds_sum	KV Offloading — Tiered KV Offload Lookup Delay Accumulation	synchronous	seconds/s	model_name and engine
vllm:kv_offload_tiering_lookup_async_delay_seconds_sum	KV Offloading — Tiered KV Offload Lookup Delay Accumulation	asynchronous	seconds/s	model_name and engine

NIXL Connector

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:nixl_xfer_time_seconds_bucket	NIXL Connector — NIXL Transfer Duration	matching series	observations/s	model_name and engine
vllm:nixl_post_time_seconds_bucket	NIXL Connector — NIXL Transfer Post Time	matching series	observations/s	model_name and engine
vllm:nixl_bytes_transferred_bucket	NIXL Connector — NIXL Transfer Size	matching series	observations/s	model_name and engine
vllm:nixl_num_descriptors_bucket	NIXL Connector — NIXL Transfer Descriptors	matching series	observations/s	model_name and engine
vllm:nixl_xfer_time_seconds_count	NIXL Connector — Successful NIXL Transfers	transfers	transfers/s	model_name and engine
vllm:nixl_xfer_time_seconds_sum	NIXL Connector — Completed NIXL Transfer Time	transfer	seconds/s	model_name and engine
vllm:nixl_post_time_seconds_sum	NIXL Connector — Completed NIXL Transfer Time	post	seconds/s	model_name and engine
vllm:nixl_bytes_transferred_sum	NIXL Connector — NIXL Transfer Throughput	transferred	bytes/s	model_name and engine
vllm:nixl_num_descriptors_sum	NIXL Connector — NIXL Descriptor Throughput	descriptors	descriptors/s	model_name and engine
vllm:nixl_num_failed_transfers_total	NIXL Connector — NIXL Failures	transfers	failures/s	model_name and engine
vllm:nixl_num_failed_notifications_total	NIXL Connector — NIXL Failures	notifications	failures/s	model_name and engine
vllm:nixl_num_kv_expired_reqs_total	NIXL Connector — NIXL Requests with Expired KV	expired	requests/s	model_name and engine

HF3FS Connector

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:hf3fs_save_duration_seconds_bucket	HF3FS Connector — HF3FS Save Duration	matching series	observations/s	model_name and engine
vllm:hf3fs_load_duration_seconds_bucket	HF3FS Connector — HF3FS Load Duration	matching series	observations/s	model_name and engine
vllm:hf3fs_save_duration_seconds_count	HF3FS Connector — HF3FS Transfer Measurements	saves	operations/s	model_name and engine
vllm:hf3fs_load_duration_seconds_count	HF3FS Connector — HF3FS Transfer Measurements	loads	operations/s	model_name and engine
vllm:hf3fs_save_duration_seconds_sum	HF3FS Connector — Completed HF3FS Transfer Time	saves	seconds/s	model_name and engine
vllm:hf3fs_load_duration_seconds_sum	HF3FS Connector — Completed HF3FS Transfer Time	loads	seconds/s	model_name and engine
vllm:hf3fs_num_failed_save_total	HF3FS Connector — HF3FS Transfer Failures	saves	failures/s	model_name and engine
vllm:hf3fs_num_failed_load_total	HF3FS Connector — HF3FS Transfer Failures	loads	failures/s	model_name and engine

Mooncake Connector

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:mooncake_store_operation_time_seconds_bucket	Mooncake Connector / Operation Timing — Mooncake Store Operation Time	matching series	observations/s	model_name and engine and operation and status
vllm:mooncake_store_operation_total	Mooncake Connector / Operation Timing — Mooncake Store Operations	operations	operations/s	model_name and engine and operation and status
vllm:mooncake_store_operation_time_seconds_sum	Mooncake Connector / Operation Timing — Completed Mooncake Store Operation Time	time	seconds/s	model_name and engine and operation and status
vllm:mooncake_store_operation_keys_total	Mooncake Connector / Operation Volume — Mooncake Store Key Throughput	keys	keys/s	model_name and engine and operation and status
vllm:mooncake_store_operation_bytes_total	Mooncake Connector / Operation Volume — Mooncake Store Byte Throughput	bytes	bytes/s	model_name and engine and operation and status
vllm:mooncake_store_operation_failed_keys_total	Mooncake Connector / Operation Volume — Mooncake Store Failed Keys	failed	keys/s	model_name and engine and operation and status

Speculative Decoding

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:spec_decode_num_drafts_total	Speculative Decoding — Drafts	drafts	drafts/s	model_name and engine
vllm:spec_decode_num_draft_tokens_total	Speculative Decoding — Draft Token Outcomes	proposed	tokens/s	model_name and engine
vllm:spec_decode_num_accepted_tokens_total	Speculative Decoding — Draft Token Outcomes	accepted	tokens/s	model_name and engine
vllm:spec_decode_num_accepted_tokens_per_pos_total	Speculative Decoding — Accepted Tokens by Position	values of label position	tokens/s	model_name and engine

Diffusion Decoding

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:diffusion_num_denoising_steps_total	Diffusion Decoding — Diffusion Denoising Steps	steps	steps/s	model_name and engine
vllm:diffusion_num_canvas_positions_total	Diffusion Decoding — Diffusion Canvas Positions	positions	positions/s	model_name and engine
vllm:diffusion_num_committed_tokens_total	Diffusion Decoding — Diffusion Committed Tokens	committed	tokens/s	model_name and engine

WebSocket Service

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:websocket_connections_active	WebSocket Service — Active WebSocket Connections	active	connections	collector job service
vllm:websocket_connections_total	WebSocket Service — WebSocket Connection Lifecycle	opened	connections/s	collector job service
vllm:websocket_connection_duration_seconds_count	WebSocket Service — WebSocket Connection Lifecycle	closed	connections/s	collector job service
vllm:websocket_connection_duration_seconds_bucket	WebSocket Service — WebSocket Connection Duration	matching series	observations/s	collector job service
vllm:websocket_connection_duration_seconds_sum	WebSocket Service — Completed WebSocket Connection Time	time	seconds/s	collector job service

Tool Parsing

Prometheus metric	Netdata chart	Dimension	Unit	Scope
vllm:tool_call_parser_invocations_total	Tool Parsing — Tool Parser Invocations	invocations	invocations/s	model_name and request_type and mode and outcome

FastAPI HTTP instrumentation

Curated request outcomes, latency, in-progress work, measurements, and body traffic from instrumented FastAPI services.

Supporting profile for vLLM. Included when the endpoint exposes vLLM's native HTTP transport metrics; Ray-only metrics do not activate it.

FastAPI

Prometheus metric	Netdata chart	Dimension	Unit	Scope
http_requests_total	FastAPI / HTTP Endpoints — HTTP Request Outcomes	values of label status	requests/s	FastAPI HTTP endpoint and method
http_request_duration_seconds_bucket	FastAPI / HTTP Endpoints — HTTP Request Duration	matching series	observations/s	FastAPI HTTP endpoint and method
http_request_duration_seconds_count	FastAPI / HTTP Endpoints — HTTP Request Measurements	requests	requests/s	FastAPI HTTP endpoint and method
http_request_duration_seconds_sum	FastAPI / HTTP Endpoints — HTTP Completed Request Time	time	seconds/s	FastAPI HTTP endpoint and method
http_request_duration_highr_seconds_bucket	FastAPI / HTTP Service — High-Resolution HTTP Request Duration	matching series	observations/s	instrumented FastAPI service
http_request_duration_highr_seconds_count	FastAPI / HTTP Service — HTTP Request Measurements	requests	requests/s	instrumented FastAPI service
http_request_duration_highr_seconds_sum	FastAPI / HTTP Service — HTTP Completed Request Time	time	seconds/s	instrumented FastAPI service
http_requests_inprogress	FastAPI / HTTP In Progress — HTTP Requests In Progress	requests	requests	FastAPI service, refined to HTTP endpoint and method when in-progress labels are enabled
http_request_size_bytes_sum	FastAPI / HTTP Body Traffic — HTTP Body Throughput	request	bytes/s	FastAPI HTTP route handler
http_response_size_bytes_sum	FastAPI / HTTP Body Traffic — HTTP Body Throughput	response	bytes/s	FastAPI HTTP route handler

Process runtime

Curated CPU, memory, file-descriptor, and lifecycle metrics exported by the monitored process.

Supporting profile for vLLM. Included when the endpoint exposes vLLM's native process metrics; Ray-only metrics do not activate it.

Process Runtime

Prometheus metric	Netdata chart	Dimension	Unit	Scope
process_cpu_seconds_total	Process Runtime — Process CPU Usage	used	cores	collector job process
process_resident_memory_bytes	Process Runtime — Process Resident Memory	resident	bytes	collector job process
process_virtual_memory_bytes	Process Runtime — Process Virtual Memory	virtual	bytes	collector job process
process_open_fds	Process Runtime — Open File Descriptors	open	fds	collector job process
process_max_fds	Process Runtime — File Descriptor Limit	limit	fds	collector job process

Python garbage collection

Curated collection, uncollectable-object, and collection-run metrics for each Python garbage-collector generation.

Supporting profile for vLLM. Included when the endpoint exposes vLLM's native Python garbage-collection metrics; Ray-only metrics do not activate it.

Process Runtime

Prometheus metric	Netdata chart	Dimension	Unit	Scope
python_gc_objects_collected_total	Process Runtime / Python GC — Collected Objects	collected	objects/s	Python garbage-collector generation
python_gc_objects_uncollectable_total	Process Runtime / Python GC — Uncollectable Objects	uncollectable	objects/s	Python garbage-collector generation
python_gc_collections_total	Process Runtime / Python GC — Collections	collections	collections/s	Python garbage-collector generation

Troubleshooting

Debug Mode

Important: Debug mode is not supported for data collection jobs created via the UI using the Dyncfg feature.

To troubleshoot issues with the prometheus collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn't working.

• Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your system, open netdata.conf and look for the plugins setting under [directories].

  cd /usr/libexec/netdata/plugins.d/

• Switch to the netdata user.

  sudo -u netdata -s

• Run the go.d.plugin to debug the collector:

  ./go.d.plugin -d -m prometheus

  To debug a specific job:

  ./go.d.plugin -d -m prometheus -j jobName

Getting Logs

If you're encountering problems with the prometheus collector, follow these steps to retrieve logs and identify potential issues:

• Run the command specific to your system (systemd, non-systemd, or Docker container).
• Examine the output for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem.

System with systemd

Use the following command to view logs generated since the last Netdata service restart:

journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep prometheus

System without systemd

Locate the collector log file, typically at /var/log/netdata/collector.log, and use grep to filter for collector's name:

grep prometheus /var/log/netdata/collector.log

Note: This method shows logs from all restarts. Focus on the latest entries for troubleshooting current issues.

Docker Container

If your Netdata runs in a Docker container named "netdata" (replace if different), use this command:

docker logs netdata 2>&1 | grep prometheus

Disappearing or sparse metrics not clearing alerts

The Prometheus collector detects metrics that disappear from a successful scrape response. Generated charts and individual dimensions expire after their configured successful-cycle lifetime. An expired chart or dimension makes its alerts REMOVED; this is not a normal CLEAR transition and does not send a recovery notification. Export an explicit normal value (for example 0) whenever an alert needs a reliable recovery transition. A failed scrape does not advance the expiry lifetime; use the generic collector collection-failure alert to detect that separate condition.