$ kubectl api-resources | grep monitoring
alertmanagerconfigs                                  monitoring.coreos.com/v1alpha1                 true         AlertmanagerConfig
alertmanagers                                        monitoring.coreos.com/v1                       true         Alertmanager
podmonitors                                          monitoring.coreos.com/v1                       true         PodMonitor
probes                                               monitoring.coreos.com/v1                       true         Probe
prometheuses                                         monitoring.coreos.com/v1                       true         Prometheus
prometheusrules                                      monitoring.coreos.com/v1                       true         PrometheusRule
servicemonitors                                      monitoring.coreos.com/v1                       true         ServiceMonitor
thanosrulers                                         monitoring.coreos.com/v1                       true         ThanosRuler

The list is of Kubernetes Custom Resource Definitions (CRDs) related to the Prometheus Operator provided by CoreOS. The Prometheus Operator simplifies the deployment and operation of Prometheus, Alertmanager, and related monitoring components in Kubernetes environments.

Here’s a brief overview of each CRD:

1. AlertmanagerConfig (v1alpha1): This CRD allows for defining alert routing and receivers for Alertmanager in a more modular fashion.
2. Alertmanager (v1): Represents an Alertmanager cluster. You can define the version, replicas, and other specifications for running Alertmanager in a cluster.
3. PodMonitor (v1): Allows you to define how Prometheus should monitor specific pods based on labels and annotations.
4. Probe (v1): Defines how Prometheus should perform “blackbox” probes using the Blackbox Exporter.
5. Prometheus (v1): Represents a Prometheus instance in your cluster. You can define specifications like the version of Prometheus, storage configurations, etc.
6. PrometheusRule (v1): Represents a set of Prometheus alerting and/or recording rules.
7. ServiceMonitor (v1): A higher-level way to define how Prometheus instances should monitor Kubernetes services. Rather than writing raw Prometheus scrape configs, you can use a ServiceMonitor to automatically generate them based on service labels and annotations.
8. ThanosRuler (v1): Represents a Thanos Ruler instance, which can evaluate recording and alerting rules against Thanos Query data, especially useful in multi-cluster Thanos deployments.

These CRDs allow Kubernetes administrators and users to define monitoring configurations natively in Kubernetes manifests. Once these resources are applied, the Prometheus Operator ensures that the actual services (Prometheus, Alertmanager, etc.) are correctly configured and running as expected.

What is ServiceMonitor?

ServiceMonitor is a Custom Resource Definition (CRD) introduced by the Prometheus Operator, which simplifies the deployment and configuration of Prometheus on Kubernetes. It’s designed to help in automatically discovering and scraping metrics from Kubernetes services without having to manually write and update Prometheus scrape configurations.

You do not need Prometheus to scrape your service by modifying prometheus.yaml file.

ServiceMonitor  is A higher-level way to define how Prometheus instances should monitor Kubernetes services. Rather than writing raw Prometheus scrape configs, you can use a ServiceMonitor to automatically generate them based on service labels and annotations.

How ServiceMonitor works:

1. Service Discovery:
   • ServiceMonitor works by selecting Kubernetes services based on label selectors. This allows Prometheus to automatically discover the services and start scraping metrics without explicit configurations.
2. Configuration:
   • Once a Kubernetes service is selected, ServiceMonitor provides details on how to scrape metrics from the service’s pods. This includes the port, path (/metrics by default), interval, and other scrape configurations.
3. Integration with Prometheus:
   • When you create a Prometheus custom resource, you specify which ServiceMonitors it should look at, usually by specifying label selectors. This Prometheus instance will then automatically generate appropriate scrape configs based on the ServiceMonitors it’s observing.

ServiceMonitor works in the following way:

1. When a ServiceMonitor is created, the Prometheus Operator adds the ServiceMonitor configuration to the Prometheus scrape configuration.
2. Prometheus starts scraping the service endpoints specified in the ServiceMonitor configuration.
3. If the Prometheus Operator detects that the service has changed, it updates the Prometheus scrape configuration accordingly.
4. Prometheus continues to scrape the service endpoints specified in the Prometheus scrape configuration.

	$ kubectl get servicemonitors --all-namespaces
	NAMESPACE NAME AGE
	management kube-prometheus-stack-alertmanager 25d
	management kube-prometheus-stack-apiserver 25d
	management kube-prometheus-stack-coredns 25d
	management kube-prometheus-stack-grafana 25d
	management kube-prometheus-stack-kube-etcd 25d
	management kube-prometheus-stack-kube-proxy 25d
	management kube-prometheus-stack-kube-state-metrics 25d
	management kube-prometheus-stack-kubelet 25d
	management kube-prometheus-stack-node-exporter 25d
	management kube-prometheus-stack-operator 25d
	management kube-prometheus-stack-prometheus 25d
	management management-nginx-ingress-ingress-nginx-controller 25d
	A Basic Example:
	Imagine you have a Kubernetes service named my-app-service that exposes Prometheus metrics on port 8080 at the /metrics path. Instead of writing a raw Prometheus scrape configuration, you would create a ServiceMonitor resource that looks something like this:
	apiVersion: monitoring.coreos.com/v1
	kind: ServiceMonitor
	metadata:
	name: my-app-monitor
	labels:
	team: my-team
	spec:
	selector:
	matchLabels:
	app: my-app # This label should match the labels on your service
	endpoints:
	- port: metrics-port
	interval: 30s
	path: /metrics
	In the above, you're specifying:
	Which services to monitor by using matchLabels.
	The port (named metrics-port in the service definition) from which to scrape metrics.
	The interval (30s) at which Prometheus should scrape.
	The path (/metrics) at which the metrics are exposed.
	apiVersion: monitoring.coreos.com/v1
	kind: ServiceMonitor
	metadata:
	name: my-app-service-monitor
	spec:
	selector:
	matchLabels:
	app: my-app
	$ kubectl describe servicemonitors kube-prometheus-stack-apiserver -n=management
	Name: kube-prometheus-stack-apiserver
	Namespace: management
	Labels: app=kube-prometheus-stack-apiserver
	app.kubernetes.io/instance=kube-prometheus-stack
	app.kubernetes.io/managed-by=Helm
	app.kubernetes.io/part-of=kube-prometheus-stack
	app.kubernetes.io/version=16.7.1
	chart=kube-prometheus-stack-16.7.1
	heritage=Helm
	release=kube-prometheus-stack
	Annotations: <none>
	API Version: monitoring.coreos.com/v1
	Kind: ServiceMonitor
	Metadata:
	Creation Timestamp: 2023-08-25T13:02:20Z
	Generation: 1
	Resource Version: 176360792
	UID: a175f110-d863-4d20-a880-280d98210f5a
	Spec:
	Endpoints:
	Bearer Token File: /var/run/secrets/kubernetes.io/serviceaccount/token
	Port: https
	Scheme: https
	Tls Config:
	Ca File: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
	Insecure Skip Verify: false
	Server Name: kubernetes
	Job Label: component
	Namespace Selector:
	Match Names:
	default
	Selector:
	Match Labels:
	Component: apiserver
	Provider: kubernetes
	Events: <none>

view raw gistfile1.txt hosted with by GitHub

Rajesh Kumar

I’m a DevOps/SRE/DevSecOps/Cloud Expert passionate about sharing knowledge and experiences. I am working at Cotocus. I blog tech insights at DevOps School, travel stories at Holiday Landmark, stock market tips at Stocks Mantra, health and fitness guidance at My Medic Plus, product reviews at I reviewed , and SEO strategies at Wizbrand. 

Please find my social handles as below;

Rajesh Kumar Personal Website
Rajesh Kumar at YOUTUBE
Rajesh Kumar at INSTAGRAM
Rajesh Kumar at X
Rajesh Kumar at FACEBOOK
Rajesh Kumar at LINKEDIN
Rajesh Kumar at PINTEREST
Rajesh Kumar at QUORA
Rajesh Kumar at WIZBRAND