Prometheus Install & Configuration

Install Prometheus

Prometheus is a time series database used to store matrices. These matrices are then used by Grafana to represent in Graphs.

Download Prometheus RPM
# wget https://github.com/prometheus/prometheus/releases/download/v2.28.1/prometheus-2.28.1.linux-amd64.tar.gz --2021-08-07 18:48:54-- https://github.com/prometheus/prometheus/releases/download/v2.28.1/prometheus-2.28.1.linux-amd64.tar.gz Resolving github.com (github.com)... 140.82.114.3 Connecting to github.com (github.com)|140.82.114.3|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://github-releases.githubusercontent.com/6838921/4063cb00-da9b-11eb-8735-e43628dda2bc?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210807%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210807T224854Z&X-Amz-Expires=300&X-Amz-Signature=2422da2a81b75153eafc9d16362af43fd2d0d9384daa4d53f1c3e6ff5afd6327&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=6838921&response-content-disposition=attachment%3B%20filename%3Dprometheus-2.28.1.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following] --2021-08-07 18:48:55-- https://github-releases.githubusercontent.com/6838921/4063cb00-da9b-11eb-8735-e43628dda2bc?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210807%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210807T224854Z&X-Amz-Expires=300&X-Amz-Signature=2422da2a81b75153eafc9d16362af43fd2d0d9384daa4d53f1c3e6ff5afd6327&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=6838921&response-content-disposition=attachment%3B%20filename%3Dprometheus-2.28.1.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.108.154, 185.199.109.154, 185.199.110.154, ... Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.108.154|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 71109475 (68M) [application/octet-stream] Saving to: prometheus-2.28.1.linux-amd64.tar.gz 100%[====================================================================================================================>] 71,109,475 5.38MB/s in 14s 2021-08-07 18:49:09 (4.77 MB/s) - prometheus-2.28.1.linux-amd64.tar.gz saved [71109475/71109475]
Extract Prometheus Tarball to /opt
# tar xfz prometheus-2.28.1.linux-amd64.tar.gz -C /opt/
Check files are extracted
# ls -l /opt/prometheus-2.28.1.linux-amd64/ total 181676 drwxr-xr-x 2 3434 3434 38 Jul 1 12:29 console_libraries drwxr-xr-x 2 3434 3434 173 Jul 1 12:29 consoles -rw-r--r-- 1 3434 3434 11357 Jul 1 12:29 LICENSE -rw-r--r-- 1 3434 3434 3646 Jul 1 12:29 NOTICE -rwxr-xr-x 1 3434 3434 98503354 Jul 1 11:22 prometheus -rw-r--r-- 1 3434 3434 926 Jul 1 12:29 prometheus.yml -rwxr-xr-x 1 3434 3434 87509137 Jul 1 11:25 promtool

Scrapping Metrics from Node Exporters

Configure Prometheus to scrap metrics.

Edit prometheus.yml to add server names or IPs where Node Exporter is running and we want to scrap the metrics. Multiple groups can be created containing multiple servers.

# vi /opt/prometheus-2.28.1.linux-amd64/prometheus.yml # my global config global: scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global default (10s). # Alertmanager configuration alerting: alertmanagers: - static_configs: - targets: # - alertmanager:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: # - "first_rules.yml" # - "second_rules.yml" # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=` to any timeseries scraped from this config. - job_name: 'node' # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ['192.168.50.35:9100'] labels: group: 'first-group' - targets: ['192.168.50.141:9100','192.168.50.142:9100'] labels: group: 'second-group'
Run Prometheus with the updated configuration
# /opt/prometheus-2.28.1.linux-amd64/prometheus --config.file=/opt/prometheus-2.28.1.linux-amd64/prometheus.yml .... level=info ts=2021-08-07T22:56:06.990Z caller=main.go:981 msg="Loading configuration file" filename=/opt/prometheus-2.28.1.linux-amd64/prometheus.yml level=info ts=2021-08-07T22:56:07.124Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/opt/prometheus-2.28.1.linux-amd64/prometheus.yml level=info ts=2021-08-07T22:56:07.124Z caller=main.go:796 msg="Server is ready to receive web requests."
Prometheus has been configured to scrap metrics from clients running Node Exporter. These Metrics can now be visualized in Grafana Dashboards.