Starting: 1st of Every Month 
+91 8409492687 
Contact@DevOpsSchool.com
Step 1 – Install and Configure Alert Manager
Step 2 – Add Alert Rules in prometheus config file(prometheus.yml)
step 3 – Add Alert managers target in prometheus config file(prometheus.yml)
Step 4 – Modify alertmanager.yml to add Notificaion Channel
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Follwing Code you Add in alert.yaml | |
| ========================================================================= | |
| groups: | |
| - name: custom_rules | |
| rules: | |
| - alert: DiskSpaceFree10Percent | |
| expr: node_filesystem_avail_bytes{mountpoint="/", fstype="xfs"} <= 6519140359 | |
| labels: | |
| severity: warning | |
| annotations: | |
| summary: "Instance [{{ $labels.instance }}] has 10% or less Free disk space" | |
| description: "[{{ $labels.instance }}] has only {{ $value }}% or less free." | |
| ========================================================================= | |
| # Follwing Code you Add in prometheus.yml | |
| ========================================================================= | |
| # Alertmanager configuration | |
| alerting: | |
| alertmanagers: | |
| - static_configs: | |
| - targets: | |
| - 3.110.33.54:9093 | |
| # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. | |
| rule_files: | |
| # - "first_rules.yml" | |
| # - "second_rules.yml" | |
| - "prometheus_rules.yml" | |
| - "alert.yaml" | |
| ========================================================================= | |
| ========================================================================= | |
| Step 1 - Download and Install Prometheus Alert Manager | |
| ========================================================================= | |
| wget https://github.com/prometheus/alertmanager/releases/download/v0.19.0/alertmanager-0.19.0.linux-amd64.tar.gz | |
| tar xvfz alertmanager-0.19.0.linux-amd64.tar.gz | |
| cd alertmanager-0.19.0.linux-amd64 | |
| ls -lh | |
| ========================================================================= | |
| Step 2 - Run Prometheus Alert Manager without service | |
| ========================================================================= | |
| ./alertmanager --config.file=alertmanager.yml | |
| # Note that it's running on port 9093 | |
| ========================================================================= | |
| Step 3 - Run Prometheus Alert Manager with service | |
| ========================================================================= | |
| # Stop existing runing Prometheus Alert Manager | |
| $ cp -r . /usr/local/bin/alertmanager | |
| $ sudo vi /etc/systemd/system/alertmanager.service | |
| ------------------------------------------------- | |
| [Unit] | |
| Description=Prometheus Alert Manager Service | |
| After=network.target | |
| [Service] | |
| Type=simple | |
| ExecStart=/usr/local/bin/alertmanager/alertmanager \ | |
| --config.file=/usr/local/bin/alertmanager/alertmanager.yml \ | |
| --web.external-url=https://[your domain name]/alertmanager \ | |
| --web.route-prefix=/ | |
| [Install] | |
| WantedBy=multi-user.target | |
| ========================================================================= | |
| Step 4 - Now start and check the service is running. | |
| ========================================================================= | |
| $ sudo service alertmanager start | |
| $ sudo service alertmanager status | |
| $ sudo service alertmanager stop | |
| ========================================================================= | |
| Step 5 - How to access Prometheus Alert Manager | |
| ========================================================================= | |
| Try http://[your domain name]:9093/ | |
| ========================================================================= | |
| Step 6 - Install and Configure "Send Only SMTP Server" in REHL or Ubuntu? | |
| ========================================================================= | |
| RHEL/CENTOS | |
| --------------- | |
| Ubuntu | |
| --------------- | |
| sudo apt install mailutils | |
| sudo vi /etc/postfix/main.cf | |
| # scroll to the bottom, and set | |
| inet_interfaces = loopback-only | |
| inet_protocols = ipv4 | |
| # restart postfix | |
| sudo systemctl restart postfix | |
| # try sending an email from the command line, | |
| echo "This is the body" | mail -s "This is the subject" -a "FROM:[senders address]" [your personal email] | |
| ========================================================================= | |
| Step 7 - Configure Alert Manager to Send Alerts from Prometheus | |
| ========================================================================= | |
| We now configure the Prometheus and Alert Manager processes to communicate with each other, and to send alerts when the alerting | |
| rules fire. | |
| # Edit Alert Manager Configuration | |
| $ sudo vi /usr/local/bin/alertmanager/alertmanager.yml | |
| --------------------- | |
| route: | |
| group_by: ['alertname'] | |
| receiver: smtp-local | |
| receivers: | |
| - name: 'smtp-local' | |
| email_configs: | |
| - to: 'your email address' | |
| from: 'your email from address' | |
| require_tls: false | |
| smarthost: localhost:25 | |
| send_resolved: true | |
| ----------------------------- | |
| # Check your configuration with the supplied amtool | |
| /usr/local/bin/alertmanager/amtool check-config /usr/local/bin/alertmanager/alertmanager.yml | |
| # If all is ok, restart the Alert Manager service. | |
| sudo service alertmanager restart | |
| sudo service alertmanager status | |
| ========================================================================= | |
| Step 8 - Edit Prometheus Configuration | |
| ========================================================================= | |
| Open the prometheus.yml configuration file and add the text below to the alerting section | |
| $ sudo vi /usr/local/bin/prometheus/prometheus.yml | |
| # Add - localhost:9093 to your Alertmanager configuration section | |
| alerting: | |
| alertmanagers: | |
| - static_configs: | |
| - targets: | |
| # alertmanager:9093 | |
| - localhost:9093 | |
| # Check the prometheus configuration with the supplied promtool. | |
| /usr/local/bin/prometheus/promtool check config /usr/local/bin/prometheus/prometheus.yml | |
| sudo service prometheus restart | |
| sudo service prometheus status | |
| ========================================================================= | |
| Step 9 - Check the Prometheus UI | |
| ========================================================================= | |
| [Status]-->[Runtime and Build Information] | |
| Copy the url in the Alertmanagers section | |
| http://localhost:9093/api/v1/alerts | |
| Test it using curl |
by 
Alert Manager templaate: Slack, SMTP, Team, Pagerduty
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| route: | |
| group_by: [cluster] | |
| receiver: **DEFAULT-RECEIVER** | |
| group_interval: 5m | |
| routes: | |
| - match: | |
| service: database | |
| receiver: **DATABASE-RECEIVER** | |
| receivers: | |
| - name: **DEFAULT-RECEIVER** | |
| pagerduty_configs: | |
| - service_key: **PRIMARY-INTEGRATION-KEY** | |
| - name: **DATABASE-RECEIVER** | |
| pagerduty_configs: | |
| - service_key: **DATABASE-INTEGRATION-KEY** |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| global: | |
| resolve_timeout: 1m | |
| slack_api_url: 'https://hooks.slack.com/services/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' | |
| route: | |
| receiver: 'slack-notifications' | |
| receivers: | |
| - name: 'slack-notifications' | |
| slack_configs: | |
| - channel: '#monitoring-instances' | |
| send_resolved: true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| config: | |
| global: | |
| resolve_timeout: 5m | |
| route: | |
| group_by: ['job'] | |
| group_wait: 30s | |
| group_interval: 5m | |
| repeat_interval: 1h | |
| receiver: 'tech-email' | |
| routes: | |
| - match: | |
| alertname: Watchdog | |
| receiver: 'null' | |
| receivers: | |
| - name: 'tech-email' | |
| email_configs: | |
| - to: 'to-email@example.com' | |
| from: 'from-email@outlook.com' | |
| auth_username: ********** | |
| auth_password: ********** | |
| require_tls: yes | |
| smarthost: ********** | |
| send_resolved: true | |
| - name: 'null' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| route: | |
| group_by: [ alertname ] | |
| receiver: 'mail' # default receiver | |
| repeat_interval: 24h | |
| routes: | |
| - receiver: 'teams' | |
| repeat_interval: 12h | |
| matchers: | |
| - severity="medium" | |
| - receiver: 'teams' | |
| repeat_interval: 4h | |
| matchers: | |
| - severity="high" | |
| receivers: | |
| - name: 'mail' | |
| email_configs: | |
| - smarthost: 'yourmailhost.com:465' | |
| auth_username: 'yourmail@yourmailhost.com' | |
| auth_password: "your mail password" | |
| from: 'yourmail@yourmailhost.com' | |
| to: 'someonesmail@yourmailhost.com' | |
| require_tls: false # if you have problems with tls | |
| - name: 'teams' | |
| webhook_configs: | |
| - url: "http://prom2teams:8089" | |
| send_resolved: true |
I’m a DevOps/SRE/DevSecOps/Cloud Expert passionate about sharing knowledge and experiences. I am working at Cotocus. I blog tech insights at DevOps School, travel stories at Holiday Landmark, stock market tips at Stocks Mantra, health and fitness guidance at My Medic Plus, product reviews at I reviewed , and SEO strategies at Wizbrand.
Please find my social handles as below;
Rajesh Kumar Personal Website
Rajesh Kumar at YOUTUBE
Rajesh Kumar at INSTAGRAM
Rajesh Kumar at X
Rajesh Kumar at FACEBOOK
Rajesh Kumar at LINKEDIN
Rajesh Kumar at PINTEREST
Rajesh Kumar at QUORA
Rajesh Kumar at WIZBRAND
