Editing Prometheus

[[wikipedia:Prometheus (software)|Prometheus]] is an open-source systems [[monitoring]] and [[alerting]] toolkit <ref>https://prometheus.io/docs/introduction/overview/</ref> released in [[2012]]. Prometheus design is focused to [[collect]] and process metrics, not as an event logging system for [[Prometheus logs|logs]].<ref>https://prometheus.io/docs/introduction/faq/#how-to-feed-logs-into-prometheus?</ref>. [[Grafana]] is usually used to produce [[dashboards]].
*  https://prometheus.io/

== [[Prometheus Installation]]==
* http://localhost:9090
* [[Node exporter]]: [[curl -sSL http://localhost:9100/metrics]]

[[Kubernetes]]
* <code>[[helm repo add prometheus-community https://prometheus-community.github.io/helm-charts]]</code>
* <code>[[helm install myprometheus  prometheus-community/prometheus]]</code>

== Configuration files ==
* Linux: <code>[[/etc/prometheus/prometheus.yml]]</code>
* macOS: <code>[[/usr/local/etc/prometheus.args]]</code>
* [[Prometheus storage]]: <code>[[/var/lib/prometheus/]]</code>

== Binaries ==
* <code>[[prometheus]]</code>
* <code>[[promtool]]</code>
* <code>[[tsdb]]</code>

== Alternatives ==
* [[Thanos]]
* [[VictoriaMetrics]]
* [[Grafana Mimir]] (Mar 2022)

== Configuration ==

* Configuration file: <code>[[/etc/prometheus/prometheus.yml]]</code>

* In the '''global''' settings, define the default interval for scraping metrics. Note that Prometheus will apply these settings to every exporter unless an individual exporter's own settings override the global.

* This '''[[scrape_interval]]''' value tells Prometheus to collect metrics from its exporters every 15 seconds, which is long enough for most exporters. Now, add Prometheus itself to the list of exporters to scrape from with the following s'''crape_configs''' directive:

* Prometheus uses the '''job_name''' to label exporters in queries and on graphs, so be sure to pick something descriptive here. And, as Prometheus exports important data about itself that you can use for monitoring performance and debugging, we've overridden the global '''scrape_interval''' directive from 15 seconds to 5 seconds for more frequent updates. Lastly, Prometheus uses the '''static_configs''' and '''targets''' directives to determine where exporters are running. Since this particular exporter is running on the same server as Prometheus itself, we can use localhost instead of an IP address along with the default port, '''[[9090]]'''.


Your configuration file should now look like this:

<pre>
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
</pre>

== Linux Basic Prometheus Operations ==
* '''Start''' Prometheus: :<code>[[sudo systemctl start prometheus]]</code>
* '''Reload''' systemd:  :<code>[[sudo systemctl daemon-reload prometheus]]</code>
* '''Verify''' the service's status: :<code>sudo [[systemctl status prometheus]]</code>

* Delete all data:
Enable <code>--web.enable-admin-api</code> and execute:
:: http://prometheus.domain.com:9090/api/v2/admin/tsdb/delete_series
:: http://prometheus.domain.com:9090/api/v2/admin/tsdb/clean_tombstones

==Running Node Exporter==

The steps for running Node Exporter are similar to those for running Prometheus itself. Start by creating the Systemd service file for Node Exporter and copy the following content into the service file:

<pre>sudo vi /etc/systemd/system/node_exporter.service</pre>

<pre>[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target</pre>


* '''Reload''' systemd to use the newly created service: :<code>sudo systemctl daemon-reload</code>

* '''Run''' Node Exporter using the following command: :<code>sudo systemctl start node_exporter</code>

* '''Verify''' that Node Exporter's running correctly: :<code>sudo systemctl status node_exporter</code>


* Lastly, enable Node Exporter to start on boot.

<pre>
sudo systemctl enable node_exporter
</pre>


===Configuring Prometheus to Scrape Node Exporter===

Because Prometheus only scrapes exporters which are defined in the '''scrape_configs''' portion of its configuration file, we'll need to add an entry for Node Exporter, just like we did for Prometheus itself.

At the end of the '''scrape_configs''' block, add a new entry called '''node_exporter.'''

<pre>sudo nano /etc/prometheus/prometheus.yml</pre>

<pre>global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']</pre>

Finally, restart Prometheus to put the changes into effect and verify status

==Activities==
# [[/How To Install Prometheus in Linux/|Install Prometheus in Linux]], [[/Run Prometheus in Linux/]] and connect to graphical interface http://localhost:9090/graph
# Read [[Prometheus changelog]]: https://github.com/prometheus/prometheus/blob/master/CHANGELOG.md
# Read Stackoverflow prometheus questions: https://stackoverflow.com/questions/tagged/prometheus?tab=Frequent
# Export <code>[[dockerd]]</code> metrics to Prometheus: https://docs.docker.com/config/daemon/prometheus/, modify <code>[[/etc/docker/daemon.json]]</code>

== Related terms ==
* [[PromQL]]
* [[Elasticsearch]]
* Exporters: [[Nginx]] ([https://github.com/nginxinc/nginx-prometheus-exporter nginx-prometheus-exporter]), [[Node exporter]]
* [[Cortex]], Prometheus as a service
* [[Loki]]
* [[CKA]]: [[Understand how to monitor applications in Kubernetes]]
* <code>[[.tpl]]</code>
* <code>[[alert:]]</code>
* [[AWS CloudWatch Container Insights]]
* [[Amazon Managed Service for Prometheus]]
* [[Kubernetes Metrics Server]]
* [[Fluent Bit]]
* [[GitLab 12.5]] Automatically close GitLab issues with recovery alerts from [[Prometheus]]
* [[Prometheus-node-exporter Debian configuration file]]
* <code>[[/metrics]]</code>
* [[Amazon Managed Prometheus]]
* [[Google Cloud Managed Prometheus]]
* <code>[[level=error]] ... [[no space left on device]]</code>
* [[Prometheus API]]
* [[Logz.io]]

== See also ==
* {{Prometheus alternatives}}
* {{Prometheus}}
* {{monitoring software}}
* {{TSDB}}

[[Category:Monitoring]]
[[Category:Prometheus]]
[[Category:CNCF]]


{{CC license}}
Source: https://en.wikiversity.org/wiki/Prometheus_monitoring