Prometheus MCP Server: Complete Setup Guide for AI Monitoring

Prometheus MCP Server: Complete Setup Guide for AI Monitoring

Prometheus is the backbone of modern infrastructure observability. But querying it still requires knowing PromQL, understanding label structures, and manually navigating dashboards during an incident at 3 AM. The Prometheus MCP Server changes that equation — it exposes Prometheus data as structured tools that AI agents can discover, call, and reason over in natural language.

This guide covers everything you need to get a Prometheus MCP Server running in production: architecture, which implementation to pick, step-by-step configuration, Claude Desktop and Cursor integration, real-world monitoring workflows, security hardening, and troubleshooting.

Before going further: multiple independent Prometheus MCP Server implementations exist. They have different maintainers, authentication models, target environments, and capabilities. This guide evaluates the two most relevant ones — pab1it0/prometheus-mcp-server and the AWS Labs Prometheus MCP Server — and clearly separates their configuration, tools, and deployment guidance. Do not mix instructions between implementations.

What a Prometheus MCP Server Actually Does

The Model Context Protocol (MCP) is an open standard that lets AI agents — like Claude — discover and call external tools in a structured, protocol-compliant way. An MCP server exposes a set of named tools with defined input schemas, and the AI agent decides when and how to call them based on user intent.

A Prometheus MCP Server wraps the Prometheus HTTP API behind an MCP-compliant interface. Instead of the AI agent constructing raw HTTP requests to Prometheus, it calls named tools like execute_query or list_metrics, and the MCP server handles authentication, request formatting, and response parsing.

This matters for several reasons:

• Tool discovery: The AI agent learns what's available at runtime without hardcoding API knowledge.
• Natural language to PromQL: The agent can translate "show me the error rate for the checkout service over the last hour" into a valid PromQL range query.
• Safe, scoped access: The MCP server exposes only the operations it implements — typically read-only Prometheus endpoints — so the agent cannot accidentally modify alerting rules or delete data.
• Unified interface: A single MCP client configuration works across different Prometheus environments (self-hosted, AMP, Mimir) depending on which implementation you choose.

Architecture and Data Flow

┌─────────────────────────────────────────────────────────────────┐
│                        MCP Client Layer                          │
│                                                                   │
│   ┌──────────────────┐        ┌──────────────────────────────┐  │
│   │   Claude Desktop  │        │   Cursor / Claude Code       │  │
│   │   (stdio client)  │        │   (stdio client)             │  │
│   └────────┬─────────┘        └──────────────┬───────────────┘  │
└────────────┼──────────────────────────────────┼──────────────────┘
             │ MCP (stdio / JSON-RPC 2.0)        │
             ▼                                  ▼
┌─────────────────────────────────────────────────────────────────┐
│                   Prometheus MCP Server Process                  │
│                                                                   │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │  Tool Registry                                           │   │
│   │  - execute_query (instant PromQL)                        │   │
│   │  - execute_range_query (range PromQL)                    │   │
│   │  - list_metrics (metric name discovery)                  │   │
│   │  - get_metric_metadata (metric type/help)                │   │
│   │  - list_labels / get_label_values                        │   │
│   │  - get_targets (scrape target status)         [pab1it0]  │   │
│   │  - get_alerts (active alerts)                 [pab1it0]  │   │
│   └─────────────────────────────────────────────────────────┘   │
│                          │                                        │
│              HTTP / SigV4-signed HTTP                            │
└──────────────────────────┼──────────────────────────────────────┘
                           │
          ┌────────────────┴────────────────┐
          ▼                                 ▼
┌──────────────────┐              ┌──────────────────────┐
│  Self-Hosted     │              │  Amazon Managed       │
│  Prometheus      │              │  Service for          │
│  (HTTP API)      │              │  Prometheus (AMP)     │
└──────────────────┘              └──────────────────────┘

The MCP server runs as a subprocess of the MCP client (stdio transport). It receives JSON-RPC 2.0 requests from the client, translates them into Prometheus HTTP API calls, and returns structured JSON responses. No persistent connection to Prometheus is maintained between calls — each tool invocation makes fresh HTTP requests.

Available Implementations

pab1it0/prometheus-mcp-server

Repository: github.com/pab1it0/prometheus-mcp-server

A community-maintained Python implementation targeting self-hosted Prometheus and any Prometheus-compatible endpoint. It uses the standard Prometheus HTTP API (/api/v1/) and supports basic authentication and bearer token authentication. The project is actively maintained, has PyPI distribution, and supports Docker deployment.

AWS Labs Prometheus MCP Server

Repository: github.com/awslabs/mcp/tree/main/src/prometheus-mcp-server

An officially vendor-backed implementation from AWS Labs, part of the broader awslabs/mcp monorepo. It is designed specifically for Amazon Managed Service for Prometheus (AMP) and uses AWS SigV4 request signing via the standard AWS credential chain (environment variables, instance profiles, IAM roles). This implementation is the correct choice for AMP environments.

Implementation Comparison

Which Implementation Should You Choose?

Choose pab1it0/prometheus-mcp-server if:

• You run self-hosted Prometheus (bare metal, Kubernetes, Docker)
• Your Prometheus endpoint is accessible via a plain URL, possibly with basic auth or a bearer token
• You use Grafana Mimir, Cortex, or any Prometheus-compatible remote API
• You need get_targets and get_alerts tool coverage
• You want to avoid AWS dependency

Choose the AWS Labs Prometheus MCP Server if:

• You use Amazon Managed Service for Prometheus (AMP)
• Your authentication is handled through AWS IAM roles, instance profiles, or environment-based AWS credentials
• You need an officially supported implementation with AWS's backing and security review process
• You operate inside an AWS environment and want native SigV4 signing without managing tokens manually

The rest of this guide uses pab1it0/prometheus-mcp-server as the primary implementation for the detailed tutorial, because it covers the broader self-hosted use case and includes the most complete tool set. AWS Labs configuration is covered separately in the comparison section.

Prerequisites

Before installing the Prometheus MCP Server, verify the following:

• Python 3.10+ installed (python3 --version)
• pip or uv package manager available
• A running Prometheus instance accessible at an HTTP endpoint (e.g., http://localhost:9090)
• Network access from the machine running the MCP server to the Prometheus endpoint
• Claude Desktop, Cursor, or Claude Code installed if you plan to use a desktop MCP client
• Optional: Docker if you prefer a containerized deployment
• Optional: Bearer token or basic auth credentials if your Prometheus endpoint is protected

Installing pab1it0/prometheus-mcp-server

Option 1: Install from PyPI (Recommended)

pip install prometheus-mcp-server

Verify the installation:

prometheus-mcp-server --help

Option 2: Install with uv (Faster)

uv pip install prometheus-mcp-server

Option 3: Run with uvx (No persistent install)

If you prefer not to install globally, uvx can run the package directly. This is particularly useful for Claude Desktop configurations:

uvx prometheus-mcp-server

Option 4: Docker

docker pull ghcr.io/pab1it0/prometheus-mcp-server:latest

Run with environment variables:

docker run -e PROMETHEUS_URL=http://your-prometheus:9090 \
  ghcr.io/pab1it0/prometheus-mcp-server:latest

Option 5: Install from Source

git clone https://github.com/pab1it0/prometheus-mcp-server.git
cd prometheus-mcp-server
pip install -e .

Configuration

pab1it0/prometheus-mcp-server is configured entirely through environment variables. There is no separate config file — this makes it straightforward to inject configuration in Docker, Kubernetes, or MCP client configuration blocks.

Environment Variables

Authentication Priority

If both basic auth credentials and a bearer token are supplied, the token takes precedence. For most self-hosted Prometheus deployments behind a reverse proxy with basic auth (nginx, Traefik), use PROMETHEUS_USERNAME and PROMETHEUS_PASSWORD. For Grafana Mimir or Prometheus deployments behind an OAuth proxy emitting bearer tokens, use PROMETHEUS_TOKEN.

Connecting to a Prometheus Instance

Local Prometheus (No Auth)

The simplest case — Prometheus running locally without authentication:

export PROMETHEUS_URL=http://localhost:9090
prometheus-mcp-server

Remote Prometheus with Basic Auth

export PROMETHEUS_URL=https://prometheus.internal.example.com
export PROMETHEUS_USERNAME=monitoring-reader
export PROMETHEUS_PASSWORD=s3cur3p@ssword
prometheus-mcp-server

Remote Prometheus with Bearer Token

export PROMETHEUS_URL=https://prometheus.example.com
export PROMETHEUS_TOKEN=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...
prometheus-mcp-server

Grafana Mimir or Thanos

Both Grafana Mimir and Thanos expose a Prometheus-compatible HTTP API. Point PROMETHEUS_URL at the query-frontend or Mimir's HTTP address:

export PROMETHEUS_URL=http://mimir-query-frontend:8080/prometheus
prometheus-mcp-server

For Mimir's multi-tenant setup, you may need to add the X-Scope-OrgID header. This is not natively supported by the current pab1it0 implementation's environment variables, so you would need to route through a reverse proxy that injects the header.

MCP Client Configuration

Claude Desktop

Add the following to your Claude Desktop configuration file.

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

Using uvx (recommended for isolation):

{
  "mcpServers": {
    "prometheus": {
      "command": "uvx",
      "args": ["prometheus-mcp-server"],
      "env": {
        "PROMETHEUS_URL": "http://localhost:9090"
      }
    }
  }
}

With basic auth:

{
  "mcpServers": {
    "prometheus": {
      "command": "uvx",
      "args": ["prometheus-mcp-server"],
      "env": {
        "PROMETHEUS_URL": "https://prometheus.internal.example.com",
        "PROMETHEUS_USERNAME": "monitoring-reader",
        "PROMETHEUS_PASSWORD": "s3cur3p@ssword"
      }
    }
  }
}

With bearer token:

{
  "mcpServers": {
    "prometheus": {
      "command": "uvx",
      "args": ["prometheus-mcp-server"],
      "env": {
        "PROMETHEUS_URL": "https://prometheus.example.com",
        "PROMETHEUS_TOKEN": "your-bearer-token-here"
      }
    }
  }
}

After saving, restart Claude Desktop. The Prometheus tools will appear in the tool panel.

Claude Code

Claude Code supports MCP server configuration via the claude mcp add command or by directly editing its configuration file. For stdio servers:

claude mcp add prometheus uvx prometheus-mcp-server \
  --env PROMETHEUS_URL=http://localhost:9090

Or edit ~/.claude/mcp_servers.json:

{
  "prometheus": {
    "command": "uvx",
    "args": ["prometheus-mcp-server"],
    "env": {
      "PROMETHEUS_URL": "http://localhost:9090"
    }
  }
}

Cursor

Cursor MCP configuration lives in .cursor/mcp.json in your project directory, or globally in ~/.cursor/mcp.json:

{
  "mcpServers": {
    "prometheus": {
      "command": "uvx",
      "args": ["prometheus-mcp-server"],
      "env": {
        "PROMETHEUS_URL": "http://localhost:9090"
      }
    }
  }
}

After saving, reload Cursor's MCP configuration. You can verify tool discovery worked by checking Cursor's MCP status in the settings panel.

AWS Labs Prometheus MCP Server: Setup for AMP

This section covers the AWS Labs implementation separately. Do not mix these instructions with the pab1it0 setup above.

Installation

pip install awslabs.prometheus-mcp-server

AWS Credentials

The AWS Labs implementation uses the standard boto3 credential chain — environment variables, ~/.aws/credentials, instance profiles, or IAM roles for service accounts. No explicit token configuration is needed if your environment is already authenticated:

export AWS_DEFAULT_REGION=us-east-1
export AWS_ACCESS_KEY_ID=AKIA...
export AWS_SECRET_ACCESS_KEY=...

For production EKS deployments, use IAM Roles for Service Accounts (IRSA) instead of static credentials.

Required IAM Permissions

The IAM role or user running the MCP server needs at minimum:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "aps:QueryMetrics",
        "aps:GetSeries",
        "aps:GetLabels",
        "aps:GetMetricMetadata"
      ],
      "Resource": "arn:aws:aps:us-east-1:123456789012:workspace/ws-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    }
  ]
}

Claude Desktop Configuration for AMP

{
  "mcpServers": {
    "prometheus-amp": {
      "command": "uvx",
      "args": ["awslabs.prometheus-mcp-server"],
      "env": {
        "PROMETHEUS_URL": "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        "AWS_DEFAULT_REGION": "us-east-1"
      }
    }
  }
}

The AMP workspace URL is found in the AWS Console under Amazon Managed Service for Prometheus → your workspace → Endpoint.

Available MCP Tools

pab1it0/prometheus-mcp-server — Documented Tools

execute_query

Executes an instant PromQL query against the Prometheus /api/v1/query endpoint.

Parameters:

• query (string, required): The PromQL expression
• time (string, optional): RFC3339 or Unix timestamp for the evaluation time

Example use: "What is the current CPU usage for node exporter on host prod-web-01?"

execute_range_query

Executes a PromQL range query against /api/v1/query_range, returning time-series data.

Parameters:

• query (string, required): The PromQL expression
• start (string, required): Start time (RFC3339 or Unix)
• end (string, required): End time (RFC3339 or Unix)
• step (string, required): Query resolution step (e.g., 60s, 5m)

Example use: "Show me the HTTP request rate for the payment service over the last 2 hours."

list_metrics

Returns all metric names from /api/v1/label/__name__/values — effectively a full metric catalog.

Parameters: None required (may accept label matchers in some versions)

Example use: "What metrics are available for the checkout service?"

get_metric_metadata

Returns metric type (counter, gauge, histogram, summary) and HELP text from /api/v1/metadata.

Parameters:

• metric (string, optional): Specific metric name; omit to return all metadata

Example use: "What type is http_requests_total and what does it measure?"

list_labels

Returns all label names from /api/v1/labels.

Parameters: None required

Example use: "What labels are present across all metrics?"

get_label_values

Returns all values for a specific label from /api/v1/label/{name}/values.

Parameters:

• label (string, required): The label name

Example use: "What are all the job labels in Prometheus? What Kubernetes namespaces are being monitored?"

get_targets

Returns active and dropped scrape targets from /api/v1/targets.

Parameters: None required

Example use: "Which scrape targets are currently down?"

get_alerts

Returns currently active alerts from /api/v1/alerts.

Parameters: None required

Example use: "Are there any active alerts right now? What's firing?"

Community Prometheus MCP Server vs AWS Labs Prometheus MCP Server

This section compares the two implementations in depth to help you make an informed architectural decision.

Architecture

pab1it0: Single Python process, reads from environment variables, connects to any Prometheus-compatible HTTP endpoint. No AWS SDK dependency. Uses httpx for HTTP calls. Lightweight and portable.

AWS Labs: Python process with boto3 dependency for SigV4 request signing. Designed to authenticate against AMP's AWS-managed endpoint. Part of a larger monorepo (awslabs/mcp) with shared tooling and governance.

Target Environments

pab1it0: Any Prometheus deployment — bare metal, Docker, Kubernetes, VMs, Grafana Mimir, Cortex, Thanos. Works wherever you can reach the Prometheus HTTP API.

AWS Labs: Amazon Managed Service for Prometheus exclusively in its designed configuration. While the underlying PromQL API is Prometheus-compatible, the authentication layer is AWS-specific.

Authentication

pab1it0: None, HTTP basic auth, or bearer token. Simple and broadly compatible. You manage credential rotation externally (vault, Kubernetes secrets, environment injection).

AWS Labs: AWS SigV4 signing through boto3. Credential management is handled by AWS's standard chain — IAM roles, instance profiles, environment variables. This is significantly more secure for AWS environments because temporary credentials via IRSA or instance profiles require no long-lived secrets.

Tool Coverage

pab1it0: Eight documented tools including get_targets and get_alerts — providing operational visibility beyond just metrics queries.

AWS Labs: Six documented tools (query, range query, list metrics, metadata, labels, label values). No documented get_targets or get_alerts as of current repository state.

Installation Complexity

pab1it0: pip install prometheus-mcp-server. Single dependency group. No cloud credentials required.

AWS Labs: pip install awslabs.prometheus-mcp-server. Requires AWS credentials configured correctly in the environment. IAM policy setup adds operational overhead.

Maintenance Model

pab1it0: Community-maintained. Responsive to issues and PRs based on repository activity. No SLA or corporate backing.

AWS Labs: Backed by AWS. Part of an officially governed open-source project. More likely to stay current with AMP API changes. AWS Labs projects have formal review processes.

Decision Summary

Prometheus MCP Server vs Prometheus HTTP API

Developers sometimes ask whether they need an MCP server at all — can't Claude just call the Prometheus HTTP API directly? Here's the honest comparison:

When to Use the MCP Server

Use an MCP server when:

• You want Claude Desktop or Cursor to query Prometheus interactively without writing custom code
• You're building an SRE copilot or AI-assisted monitoring workflow
• You want a consistent security boundary where credentials stay in the MCP server process, not in prompts
• You want tool discovery to work automatically as Prometheus capabilities evolve

When to Use the Prometheus HTTP API Directly

Use the HTTP API directly when:

• You're building a programmatic agent pipeline (not interactive)
• You need access to endpoints the MCP server doesn't expose (e.g., /api/v1/rules, /api/v1/status/config)
• Latency is critical and every millisecond of the JSON-RPC hop matters
• You have an existing agent framework with its own tool management

Using Both Together

A practical production architecture might use the MCP server for interactive investigation (Claude Desktop for the SRE team) while a separate programmatic pipeline queries the HTTP API directly for automated alert enrichment. Both can coexist safely against the same Prometheus instance, especially if configured with read-only credentials.

Real-World Prometheus MCP Workflows

Investigating a Production Incident

Scenario: The on-call engineer gets paged. Error rate for the payment service spiked. They open Claude Desktop with the Prometheus MCP server connected.

Workflow:

1. Triage active alerts:

   "Are there any active alerts? What's firing right now?"

   Claude calls get_alerts → returns active alert labels, state, and annotation.

2. Understand the error spike:

   "Show me the HTTP 5xx error rate for the payment service over the last 30 minutes, broken down by endpoint."

   Claude constructs and calls execute_range_query with:

   promqlCopy

   sum by (handler) (
     rate(http_requests_total{job="payment-service", status=~"5.."}[5m])
   )
   

3. Correlate with dependencies:

   "Is the database connection pool exhausted? Show saturation metrics for the last 30 minutes."

   Claude calls list_metrics to discover available database metrics, then execute_range_query with appropriate PromQL.

4. Check target health:

   "Are all payment service instances healthy and being scraped?"

   Claude calls get_targets → returns scrape targets with health status.

This entire workflow — which would have taken 10-15 minutes of manual dashboard navigation — completes in 2-3 conversational exchanges.

Analyzing Application Latency

"What's the p99 latency for the checkout service? Has it been above 500ms in the last hour?"

Claude translates this into a histogram quantile query:

histogram_quantile(0.99, 
  sum by (le) (
    rate(http_request_duration_seconds_bucket{job="checkout-service"}[5m])
  )
)

It calls get_metric_metadata first to confirm the metric is a histogram type, then execute_range_query with a 1-hour window.

Diagnosing Error-Rate Spikes

"The error rate spiked at 14:32 UTC. What changed around that time? Show me error rates for all services, not just checkout."

Claude uses get_label_values with label job to discover all service names, then constructs a range query covering the spike window across all discovered job values.

Kubernetes Workload Investigation

"Which Kubernetes pods are consuming more than 80% of their memory limits?"

Claude calls list_metrics to identify container_memory_working_set_bytes and kube_pod_container_resource_limits, then constructs:

(
  container_memory_working_set_bytes{container!=""}
  / 
  kube_pod_container_resource_limits{resource="memory", container!=""}
) > 0.8

PromQL Assistance

One of the most practical uses: developers who know what they want but aren't fluent in PromQL.

"I want to see the rate of increase in GC pause time for the Java services. I'm not sure what metric to use."

Claude calls list_metrics filtering for JVM-related names, then get_metric_metadata to understand jvm_gc_pause_seconds_* metrics, and finally constructs an appropriate rate query.

AI-Assisted SRE Workflows

The Prometheus MCP Server enables what's emerging as the AI SRE copilot pattern:

• Pre-shift briefing: "Summarize the last 8 hours of alerts and any anomalies in our top 5 services."
• Capacity planning: "Based on the current growth rate of disk usage, when will we hit 80% on prod-db-01?"
• Runbook assistance: "The DatabaseHighConnectionCount alert is firing. What are the current connection counts and which services are contributing most?"
• Post-incident review data gathering: "Pull the error rates, latency percentiles, and pod restart counts for all services between 14:00 and 16:00 UTC yesterday."

Security Considerations

Read-Only Access and Least Privilege

Both implementations are read-only by design — they only access Prometheus query endpoints, not write, admin, or lifecycle endpoints. However, you should still apply least-privilege principles:

• Create a dedicated Prometheus user (if using authentication) with access only to the query API
• Do not expose the Prometheus admin API (--web.enable-admin-api) on the same endpoint the MCP server connects to
• Block /api/v1/admin/ at the reverse proxy level if you're exposing Prometheus through nginx or Traefik

Credential Management

Environment variables in MCP client config files (like claude_desktop_config.json) are stored in plaintext on disk. For production:

• Use a secrets manager (HashiCorp Vault, AWS Secrets Manager, Doppler) to inject credentials at runtime
• For Kubernetes deployments, use Kubernetes Secrets with restricted RBAC rather than embedding credentials in pod specs
• Rotate bearer tokens regularly — set short expiry and use a token refresh mechanism
• For the AWS Labs implementation, prefer IRSA (IAM Roles for Service Accounts) over static access keys

Network Security

• Do not expose Prometheus directly to the internet. The MCP server should connect over an internal network, VPN, or private link.
• Use TLS for all connections between the MCP server and Prometheus. Set VERIFY_SSL=true (the default) and ensure your Prometheus TLS certificate is valid.
• Firewall rules: Restrict access to Prometheus's port (default 9090) to only the hosts that need it — including the host running the MCP server.

Query Safety

AI agents can generate PromQL queries that are expensive to evaluate. Mitigate this:

• Configure --query.max-samples on your Prometheus server (default 50,000,000)
• Set --query.timeout to limit long-running queries (e.g., --query.timeout=30s)
• Consider running the MCP server against a Prometheus replica or Thanos query layer rather than your primary Prometheus instance
• For AMP, IAM policies provide an additional constraint layer

High-Cardinality Query Risk

When Claude calls list_metrics, it returns all metric names. In large environments with thousands of metrics, this can be a heavy response. The real risk is when the AI then generates queries like {__name__=~".+"} without label matchers — always test your Prometheus's response to broad queries before enabling the MCP server in production.

Common Errors and Troubleshooting

Error: PROMETHEUS_URL not set

Symptom: MCP server starts but immediately fails with a configuration error.

Cause: The PROMETHEUS_URL environment variable is missing from the MCP client config.

Fix: Verify your MCP client configuration includes the env block with PROMETHEUS_URL.

"env": {
  "PROMETHEUS_URL": "http://localhost:9090"
}

Error: Connection refused or Connection timed out

Symptom: Tool calls fail with network errors.

Cause: The Prometheus endpoint is not reachable from the machine running the MCP server.

Fix:

1. Verify Prometheus is running: curl http://localhost:9090/-/healthy
2. Check firewall rules between the MCP server host and Prometheus
3. Verify the PROMETHEUS_URL includes the correct port and scheme
4. If using Docker, remember that localhost inside a container refers to the container — use the host network or the host's IP address

Error: 401 Unauthorized

Symptom: Tool calls return HTTP 401.

Cause: Authentication credentials are missing or incorrect.

Fix:

• For basic auth: verify PROMETHEUS_USERNAME and PROMETHEUS_PASSWORD
• For bearer tokens: verify PROMETHEUS_TOKEN is current and not expired
• Test credentials directly: curl -u user:pass http://prometheus:9090/api/v1/query?query=up

Error: 403 Forbidden

Symptom: Tool calls return HTTP 403.

Cause (pab1it0): Reverse proxy or Prometheus is rejecting the request for reasons beyond authentication — possibly path-based ACLs.

Cause (AWS Labs): IAM policy does not include the required aps:QueryMetrics or related actions.

Fix: Review IAM policies (AWS Labs) or reverse proxy ACL rules (pab1it0).

Error: SSL Certificate Verification Failed

Symptom: TLS handshake errors when connecting to a Prometheus endpoint with a self-signed certificate.

Cause: VERIFY_SSL=true (default) rejects untrusted certificates.

Fix for development: Set VERIFY_SSL=false. Do not use this in production. Instead, add your CA certificate to the system trust store or configure the MCP server to use a proper TLS certificate.

MCP Tools Not Appearing in Claude Desktop

Symptom: Claude doesn't show Prometheus tools after configuration.

Cause: JSON syntax error in claude_desktop_config.json, or the MCP server process failed to start.

Fix:

1. Validate JSON: python3 -m json.tool ~/Library/Application\ Support/Claude/claude_desktop_config.json
2. Check Claude Desktop logs for MCP server startup errors
3. Run the server manually: PROMETHEUS_URL=http://localhost:9090 prometheus-mcp-server and check for errors
4. Ensure uvx or pip-installed binary is in PATH

AWS Labs: NoCredentialProviders Error

Symptom: AWS Labs MCP server fails with boto3 credential errors.

Cause: No AWS credentials found in the environment.

Fix: Ensure AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION are set, or that the process is running with an instance profile or IAM role attached.

Production Deployment

Running on Kubernetes

For a Kubernetes-based deployment (e.g., as a sidecar to an AI agent pod or as a standalone service):

apiVersion: v1
kind: Secret
metadata:
  name: prometheus-mcp-secret
  namespace: monitoring
stringData:
  PROMETHEUS_URL: "http://prometheus.monitoring.svc.cluster.local:9090"
  PROMETHEUS_USERNAME: "mcp-reader"
  PROMETHEUS_PASSWORD: "s3cur3p@ssword"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-mcp-server
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-mcp-server
  template:
    metadata:
      labels:
        app: prometheus-mcp-server
    spec:
      containers:
      - name: prometheus-mcp-server
        image: ghcr.io/pab1it0/prometheus-mcp-server:latest
        envFrom:
        - secretRef:
            name: prometheus-mcp-secret
        resources:
          requests:
            cpu: "50m"
            memory: "64Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"

Monitoring the MCP Server Itself

The Prometheus MCP server is a lightweight Python process. Monitor it with:

• Process health: Use your container orchestration health checks
• Error rate: Parse MCP server logs for JSON-RPC error responses
• Latency: Tool call latency appears in MCP client logs (Claude Desktop logs) — high latency typically indicates Prometheus query performance issues, not MCP server issues
• Prometheus query load: Monitor prometheus_engine_queries_concurrent_max and prometheus_engine_query_duration_seconds on the Prometheus server to identify load caused by MCP-driven queries

Versioning and Updates

Pin the prometheus-mcp-server version in production:

"args": ["prometheus-mcp-server==0.x.y"]

Or in Docker:

docker pull ghcr.io/pab1it0/prometheus-mcp-server:v0.x.y

Review the GitHub releases before upgrading — tool schema changes can break existing Claude Desktop configurations.

Validating Your MCP Server

Before connecting an AI agent, validate that your Prometheus MCP server is reachable and protocol-compliant. If you've deployed the MCP server with an HTTP/SSE transport (rather than stdio), you can use MCPForge Verify to check MCP compliance, tool discovery, and endpoint health — useful for remote deployments or shared team instances.

For self-hosted MCP servers in production environments, it's worth reviewing the guidance in Running MCP in Production for operational best practices around process management, logging, and security hardening.

You can also explore the MCPForge Verified Directory to discover other MCP server implementations for observability, databases, and infrastructure tools that can complement your Prometheus setup.

Limitations

Being clear about what the Prometheus MCP Server cannot do is important for production planning:

• No write operations: Cannot create, modify, or delete recording rules, alerting rules, or silences. For rule management, use the Prometheus HTTP API or Alertmanager API directly.
• No admin endpoints: Cannot access /api/v1/admin/tsdb/* for TSDB management or snapshot operations.
• No streaming: Prometheus's HTTP API is request-response, not streaming. The MCP server cannot subscribe to metric streams or push alerts in real-time.
• No dashboard creation: The MCP server queries data; it doesn't interact with Grafana or create visualizations.
• No Alertmanager integration: get_alerts returns Prometheus-level alerts (firing rules), not Alertmanager silences, inhibitions, or routing. For Alertmanager interaction, a separate MCP server would be needed.
• PromQL complexity: The AI agent's ability to generate correct PromQL depends on the model's training. For complex histogram quantile queries or multi-join operations, human review of generated PromQL is advisable before relying on results.
• No metric ingestion: The MCP server is read-only. You cannot push metrics through it.

Best Practices

Architecture:

• Run the MCP server against a Prometheus replica or Thanos query layer in high-traffic environments
• Use separate MCP server instances for different Prometheus environments (prod, staging) — configure them as separate entries in your MCP client config
• For AMP, always use IRSA instead of static AWS credentials

Security:

• Create a dedicated read-only Prometheus credential for the MCP server
• Store credentials in a secrets manager, not in config files on disk
• Enable TLS between the MCP server and Prometheus in all production deployments
• Block Prometheus admin endpoints at the proxy level

Operations:

• Pin MCP server versions in production
• Configure Prometheus-side query limits to protect against expensive AI-generated queries
• Monitor Prometheus query load after enabling MCP server access
• Test MCP server connectivity and tool discovery after every Prometheus configuration change

User guidance:

• Educate AI agent users that PromQL results are point-in-time snapshots — correlate with deployment events, not just metric values
• For incident response, use the AI agent to gather data but apply human judgment to root cause conclusions
• Regularly review what queries the AI agent is generating — log MCP tool calls where possible

Key Takeaways

• Multiple Prometheus MCP Server implementations exist — pab1it0/prometheus-mcp-server for self-hosted environments, AWS Labs for AMP. Never mix their configurations.
• Both implementations are read-only and use stdio transport by default.
• The pab1it0 implementation provides eight tools including get_targets and get_alerts, making it more complete for operational monitoring workflows.
• The AWS Labs implementation provides IAM-native authentication for AMP — the correct choice for AWS environments.
• AI agents can dramatically accelerate incident investigation, PromQL assistance, and SRE workflows — but apply human judgment to conclusions.
• Protect production Prometheus instances with query limits, dedicated read-only credentials, and network access controls.
• Validate your MCP server deployment with MCPForge Verify before connecting AI agents.