Limited Time Offer!

For Less Than the Cost of a Starbucks Coffee, Access All DevOpsSchool Videos on YouTube Unlimitedly.
Master DevOps, SRE, DevSecOps Skills!

Enroll Now
Icinga

icinga tutorials: How to monitor Kubernetes using icinga

May 15, 2024 Leave a Comment

Step 1: Prerequisites

  • Ensure you have Icinga 2 and Icinga Web 2 installed on your Icinga server.
  • Kubernetes cluster where you want to implement monitoring.
  • Administrative access to both the Kubernetes cluster and the Icinga server.

Step 2: Install and Configure kubectl on Icinga Server

kubectl is the Kubernetes command-line tool that allows you to run commands against Kubernetes clusters.

Install kubectl:

$ sudo apt-get update && sudo apt-get install -y apt-transport-https gnupg2
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
$ sudo apt-get update
$ sudo apt-get install -y kubectl

Configure kubectl:

$ Copy your Kubernetes cluster's kubeconfig file to your Icinga server.
$ Set the KUBECONFIG environment variable or use the default location (~/.kube/config).

Step 3: Install Kubernetes Monitoring Plugin

There are various plugins available for monitoring Kubernetes, such as check_kube, which can be installed on the Icinga server.

Download and install the monitoring plugin:

$ cd /usr/lib/nagios/plugins
$ sudo wget https://example.com/path/to/check_kube  # Replace with actual URL
$ sudo chmod +x check_kube

Step 4: Configure Icinga 2

You need to define commands, services, and hosts in Icinga 2 for Kubernetes monitoring.

Define Command in Icinga:
Create or edit a configuration file:

$ sudo nano /etc/icinga2/conf.d/commands.conf

Add a command definition for Kubernetes checks:

object CheckCommand "check_kubernetes" {
    command = [ "/usr/lib/nagios/plugins/check_kube" ]
    arguments = {
        "-K" = "$kubeconfig$",
        "-N" = "$namespace$",
        "-D" = "$deployment$"
    }
}

Define Host for Kubernetes Cluster:
Add a host object for your Kubernetes cluster:

object Host "kubernetes-cluster" {
    import "generic-host"
    address = "127.0.0.1"  // Use an appropriate address
    check_command = "hostalive"
}

Define Service for Kubernetes Monitoring:
Create services to monitor various components:

apply Service "check-deployment" {
    import "generic-service"
    check_command = "check_kubernetes"
    vars.kubeconfig = "/path/to/kubeconfig"
    vars.namespace = "default"
    vars.deployment = "example-deployment"
    assign where host.name == "kubernetes-cluster"
}

Step 5: Reload Icinga 2

After configuring your commands and services, reload Icinga 2 to apply changes:

sudo systemctl reload icinga2

Step 6: Verify and Adjust

  • Verify in Icinga Web 2: Log into Icinga Web 2 and check the newly configured services for your Kubernetes cluster. Ensure they are reporting correctly.Adjustments: Depending on your specific monitoring needs, you may need to add additional service checks or refine existing ones to cover different Kubernetes objects like nodes, pods, and services.

Alternate Method

Step 1: Install and Configure the Icinga2 Kubernetes Plugin
Before you start, ensure that you have kubectl configured on the same server where your Icinga2 agent is running. This will allow the Icinga2 agent to use kubectl to gather metrics from Kubernetes.

Install the Kubernetes plugin for Icinga2: You might need to write custom check scripts or find existing plugins that can use kubectl to fetch data from Kubernetes. There are community plugins available, or you can create simple bash scripts that wrap kubectl commands.

Example script: Here's a basic example of a script that checks the number of running pods in a specific namespace:


#!/bin/bash
NAMESPACE=$1
MIN_PODS=$2

# Get the number of running pods
RUNNING_PODS=$(kubectl get pods --namespace $NAMESPACE --field-selector=status.phase=Running --no-headers | wc -l)

# Check if the number of running pods is less than the minimum required
if [ "$RUNNING_PODS" -lt "$MIN_PODS" ]; then
  echo "CRITICAL - Only $RUNNING_PODS running pods in $NAMESPACE, minimum required is $MIN_PODS"
  exit 2
else
  echo "OK - $RUNNING_PODS running pods in $NAMESPACE"
  exit 0
fi
Save this script on the Icinga2 agent machine, and make sure it's executable (chmod +x check_kube_pods.sh).

Step 2: Configure Icinga2 Agent
You need to configure your Icinga2 agent to execute the Kubernetes monitoring script.

Define the check command:


object CheckCommand "check_kube_pods" {
  command = [ "/path/to/check_kube_pods.sh" ]
  arguments = {
    "--namespace" = "$namespace$"
    "--min-pods" = "$min_pods$"
  }
}
Add the service definition to Icinga2:


apply Service "kube-pods" {
  import "generic-service"
  check_command = "check_kube_pods"
  vars.namespace = "default"
  vars.min_pods = 10
  assign where host.vars.kubernetes == true
}
Step 3: Define Host and Service in Icinga2
You need to ensure that your Kubernetes cluster or specific nodes are defined as hosts in your Icinga2 configuration.

Define a host for your Kubernetes cluster:


object Host "kubernetes-cluster" {
  import "generic-host"
  address = "127.0.0.1"  // Use an internal or external IP address
  vars.kubernetes = true
}
Step 4: Restart Icinga2
After setting up the configurations, restart Icinga2 to apply the changes.


sudo systemctl restart icinga2
Step 5: Validate the Setup
Check the Icinga2 dashboard or use the Icinga2 CLI to ensure that the new checks are being executed and that they return the expected results.


icinga2 object list --type Service
This setup will allow you to start monitoring basic Kubernetes metrics. You can expand this by adding more complex checks, such as node resource usage, pod status by deployment, or custom metrics exposed by your applications running in Kubernetes.
Rajesh Kumar
Follow me
Latest posts by Rajesh Kumar (see all)
Subscribe
Notify of
guest
0 Comments
Newest
Oldest Most Voted
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x