> ## Documentation Index
> Fetch the complete documentation index at: https://docs.arc.io/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Before reading these docs, install the Circle Skill for Arc to get pre-loaded context: use-arc (https://github.com/circlefin/skills/blob/master/plugins/circle/skills/use-arc/SKILL.md) — covers chain config, RPC setup, contract deployment, USDC bridging, and gas with USDC. In Claude Code: run /plugin marketplace add circlefin/skills then /plugin install circle-skills@circle. With Vercel Skills CLI: run npx skills add circlefin/skills.
> USDC is the gas token. Arc uses USDC for gas fees, not ETH. Configure gas payment in USDC when submitting transactions.
> Read EVM differences before writing Arc code. Arc targets the Osaka EVM baseline. Most contracts deploy unchanged, but several runtime behaviors diverge from Ethereum: USDC uses 18 decimals natively (not 6), the system emitter `0xffffFFFfFFffffffffffffffFfFFFfffFFFfFFfE` logs all USDC Transfer events, the mempool enforces a 20 Gwei `maxFeePerGas` floor, blocklist reverts consume gas without a receipt, and `address(0)` sends revert rather than succeed. https://docs.arc.io/arc/references/evm-differences.md is the canonical reference for all Arc-specific runtime behaviors. Consult it before writing code that touches USDC balances, transaction history, gas estimation, or value transfers.
> Sub-second finality. Transactions are final in under 1 second. No need to wait for multiple block confirmations.
> Use App Kits for multichain payment and liquidity capabilities. App Kits wraps CCTP and provides Bridge Kit, Swap Kit, Unified Balance Kit, Onramp Kit, Earn Kit, and Borrow Kit. Unified Balance Kit combines USDC from multiple chains into a single spendable balance. Earn Kit integrates earn opportunities into your app. Borrow Kit lets users borrow USDC against cirBTC collateral on Arc.
> Use Arc Foundry for contract development. Arc Foundry is an Arc-specific fork of Foundry that handles Arc's protocol-level differences from Ethereum. It provides arc-forge, arc-cast, and arc-anvil. Install it before deploying contracts: https://docs.arc.io/arc/tutorials/install-arc-foundry.md.
> Arc is available on both Testnet and Mainnet. See https://docs.arc.io/arc/references/connect-to-arc.md for RPC endpoints and https://faucet.circle.com for testnet tokens.
> Always check Contract Addresses: https://docs.arc.io/arc/references/contract-addresses.md
> Building beyond Arc? Circle offers skills for the full platform: use-usdc (https://github.com/circlefin/skills/blob/master/plugins/circle/skills/use-usdc/SKILL.md), use-circle-wallets (https://github.com/circlefin/skills/blob/master/plugins/circle/skills/use-circle-wallets/SKILL.md), use-developer-controlled-wallets (https://github.com/circlefin/skills/blob/master/plugins/circle/skills/use-developer-controlled-wallets/SKILL.md), use-user-controlled-wallets (https://github.com/circlefin/skills/blob/master/plugins/circle/skills/use-user-controlled-wallets/SKILL.md), use-modular-wallets (https://github.com/circlefin/skills/blob/master/plugins/circle/skills/use-modular-wallets/SKILL.md), use-gateway (https://github.com/circlefin/skills/blob/master/plugins/circle/skills/use-gateway/SKILL.md), use-smart-contract-platform (https://github.com/circlefin/skills/blob/master/plugins/circle/skills/use-smart-contract-platform/SKILL.md). Full Circle developer docs: https://developers.circle.com/llms.txt.

# Set up Prometheus and Grafana for an Arc node

> Deploy a local Prometheus and Grafana stack on the same host as your Arc node to scrape metrics and view the pre-built dashboards.

Deploy Prometheus and Grafana in Docker on the same host as your Arc node,
scrape metrics from the Execution Layer (EL) and Consensus Layer (CL), and load
the pre-built Arc dashboards. The stack binds to `127.0.0.1` only and is reached
over SSH port forwarding from a remote operator workstation.

## Prerequisites

* You have a running Arc node that meets the
  [node requirements](/arc/references/node-requirements), set up either from the
  [Run an Arc node](/arc/tutorials/run-an-arc-node) tutorial or
  [deployed as a systemd service](/arc/tutorials/deploy-node-as-service)
* You have installed
  [Docker Engine 24+ with BuildKit](https://docs.docker.com/engine/install/) and
  [Docker Compose v2](https://docs.docker.com/compose/install/) on the node host
* Your node is started with `--metrics 127.0.0.1:9001` on the EL and
  `--metrics 127.0.0.1:29000` on the CL (the default in
  [Run an Arc node](/arc/tutorials/run-an-arc-node))
* You have cloned the [`arc-node`](https://github.com/circlefin/arc-node)
  repository on the host, to reuse the pre-built Grafana dashboards

## Step 1: Verify the metrics endpoints

Confirm both metrics endpoints are reachable on the host before deploying the
stack. The Execution Layer serves metrics at the root path, not at `/metrics`:

```shell theme={null}
curl -s http://127.0.0.1:9001 | head
curl -s http://127.0.0.1:29000/metrics | head
```

Both commands return Prometheus-formatted text. If either fails, fix the node
startup or `--metrics` flag before continuing. See
[Monitor a node](/arc/tutorials/monitor-a-node) for troubleshooting.

## Step 2: Create the monitoring directory

This guide stores all monitoring files under `$ARC_HOME/monitoring`. Create the
directory layout:

```shell theme={null}
ARC_HOME="${ARC_HOME:-$HOME/.arc}"
ARC_MONITORING="${ARC_MONITORING:-$ARC_HOME/monitoring}"

mkdir -p "$ARC_MONITORING"/grafana-provisioning/datasources
mkdir -p "$ARC_MONITORING"/grafana-provisioning/dashboards
mkdir -p "$ARC_MONITORING"/prometheus-data
mkdir -p "$ARC_MONITORING"/dashboards
```

## Step 3: Copy the pre-built Arc dashboards

The `arc-node` repository ships ready-to-use Grafana dashboards. From the
`arc-node` repository root, copy them into the monitoring directory:

```shell theme={null}
cp -r deployments/monitoring/config-grafana/provisioning/dashboards-data/* \
  "$ARC_MONITORING"/dashboards/
```

## Step 4: Configure Prometheus

Write the following file to `$ARC_MONITORING/prometheus.yml`. The EL job scrapes
the root path; the CL job scrapes `/metrics`:

```yaml theme={null}
global:
  scrape_interval: 1s

scrape_configs:
  - job_name: "arc_execution"
    metrics_path: "/"
    scrape_interval: 1s
    static_configs:
      - targets: ["127.0.0.1:9001"]
        labels:
          client_name: "reth"
          client_type: "execution"

  - job_name: "arc_consensus"
    metrics_path: "/metrics"
    scrape_interval: 1s
    static_configs:
      - targets: ["127.0.0.1:29000"]
        labels:
          client_name: "malachite"
          client_type: "consensus"
```

## Step 5: Configure the Grafana datasource

Write the following file to
`$ARC_MONITORING/grafana-provisioning/datasources/prometheus.yml`:

```yaml theme={null}
apiVersion: 1

datasources:
  - name: prometheus
    uid: prometheus
    type: prometheus
    url: http://127.0.0.1:9090
    isDefault: true
    editable: true
```

## Step 6: Configure dashboard provisioning

Write the following file to
`$ARC_MONITORING/grafana-provisioning/dashboards/default.yml`:

```yaml theme={null}
apiVersion: 1

providers:
  - name: "arc"
    orgId: 1
    folder: ""
    type: file
    disableDeletion: false
    editable: true
    updateIntervalSeconds: 10
    options:
      path: /var/lib/grafana/dashboards
```

## Step 7: Create the Docker Compose file

Write the following file to `$ARC_MONITORING/compose.yaml`:

```yaml theme={null}
services:
  prometheus:
    image: prom/prometheus
    user: "0"
    network_mode: host
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.listen-address=127.0.0.1:9090
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus-data:/prometheus
    restart: unless-stopped

  grafana:
    image: grafana/grafana-oss
    network_mode: host
    environment:
      GF_SERVER_HTTP_ADDR: 127.0.0.1
      GF_SERVER_HTTP_PORT: 3000
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - ./grafana-provisioning:/etc/grafana/provisioning:ro
      - ./dashboards:/var/lib/grafana/dashboards:ro
    restart: unless-stopped
```

<Note>
  The Arc metrics endpoints in this guide bind to `127.0.0.1` only.
  `network_mode: host` is the simplest way for Prometheus to scrape them without
  changing the Arc node configuration or exposing metrics publicly.
</Note>

## Step 8: Start the stack

```shell theme={null}
cd "$ARC_MONITORING"
docker compose up -d
```

Confirm both containers are running:

```shell theme={null}
docker compose ps
```

Check that Grafana and Prometheus are healthy:

```shell theme={null}
curl -s http://127.0.0.1:3000/api/health
curl -s http://127.0.0.1:9090/-/ready
```

## Step 9: Access the dashboards

The stack binds Grafana and Prometheus to `127.0.0.1` only. They are not exposed
to the public internet.

If you are operating on the same machine, open:

* `http://127.0.0.1:3000` for Grafana
* `http://127.0.0.1:9090` for Prometheus

If you are connecting from a remote workstation, use SSH port forwarding:

```shell theme={null}
ssh -N \
  -L 3000:127.0.0.1:3000 \
  -L 9090:127.0.0.1:9090 \
  user@YOUR_SERVER_IP
```

Then open on your local machine:

* `http://localhost:3000`
* `http://localhost:9090`

Grafana default credentials from the Compose file above are username `admin` and
password `admin`. Change them before exposing Grafana through any reverse proxy
or shared access setup.

After Prometheus scrapes both endpoints and Grafana loads the provisioned
dashboards, you see metrics for execution layer activity, consensus layer
activity, block height, validator telemetry, and connected peers. Newly started
Prometheus instances need a short time to collect enough samples; if some panels
are empty immediately after startup, wait a minute and refresh.

## Troubleshooting

### Browser shows `ERR_CONNECTION_REFUSED`

If you are connecting to a remote server, confirm you are opening
`http://localhost:3000` and `http://localhost:9090` on your local machine, not
`http://YOUR_SERVER_IP:3000`. Also confirm the SSH tunnel is still running.

### Prometheus restarts with a permissions error

If Prometheus logs include an error about `queries.active` or write permission
under `/prometheus`, fix ownership of the local data directory:

```shell theme={null}
sudo chown -R 65534:65534 "$ARC_MONITORING"/prometheus-data
docker compose restart prometheus
```

### Grafana is healthy but dashboards show no data

Confirm Prometheus targets are up:

```shell theme={null}
curl -s http://127.0.0.1:9090/api/v1/targets
```

Re-check the Arc metrics endpoints:

```shell theme={null}
curl -s http://127.0.0.1:9001 | head
curl -s http://127.0.0.1:29000/metrics | head
```

If Prometheus reaches the targets but some panels remain empty, wait for more
samples to accumulate and refresh.

### Check Arc node health directly

You can verify the Arc node independently of Grafana:

```shell theme={null}
cast block-number --rpc-url http://127.0.0.1:8545
sudo journalctl -u arc-execution -f
sudo journalctl -u arc-consensus -f
```