At Auryc, we leverage prometheus for metrics-based monitoring for our system. The adoption for prometheus was long before Kubernetes became a part of our tech stack. As the Kubernetes gains its popularity, the community found the cloud native nature of Prometheus makes it a perfect fit for Kubernetes, that they used to jokingly refer it as P8s in comparison to "k8s" (Kubernetes).
In this article, I'll share our journey running Prometheus on our Kubernetes environment with prometheus-operator and kube-prometheus stack. Provide a real life experience how we monitor our applications and infrastructure effectively with prometheus stack.
In the early days of Auryc, we were on the lookout for an effective solution for monitoring and alerting. Back then, Prometheus wasn't as prominent as it is now. There were several other options on the table, including StatsD, JMX, Datadog, Zabbix, and Pingdom. However, after considering various factors, I settled on Prometheus. Its open-source nature, simplicity, and compatibility with Grafana for visualization were the key deciding factors.
Our initial setup with Prometheus was fairly straightforward. We utilized a mix of Prometheus SDKs, various exporters, and built-in metrics endpoints to gather metrics from our applications and infrastructure. To monitor our virtual machines, we used the node_exporter. For the Prometheus configuration, we manually set up all the available endpoints and crafted the necessary alerting rules. We also integrated Alertmanager with PagerDuty and email for alerts and chose Grafana for visualization.
Simple as it might have been, this configuration provided us with a comprehensive end-to-end monitoring solution. It served us effectively until we began our journey with Kubernetes.
As our journey to Kubernetes it’s also a chance to upgrade the Prometheus stack. So we explore the options of prometheus-operator and kube-prometheus.
The prometheus is an project to manage Prometheus on top of Kubernetes as an operator.
The operator simplifies several aspects of managing a Prometheus setup. Firstly, it automates the deployment of Prometheus instances, ensuring they are configured correctly for the Kubernetes environment. This includes setting up Prometheus to dynamically discover targets within the Kubernetes cluster, such as services and pods, which are crucial for comprehensive monitoring.
Moreover, Prometheus-Operator introduces several custom resource definitions (CRDs), like ServiceMonitor and PrometheusRule. These CRDs allow users to define and manage Prometheus components in a Kubernetes-native way. Some of the core CRDs are:
Prometheus, which defines a desired Prometheus deployment.ServiceMonitor, which declaratively specifies how groups of Kubernetes services should be monitored. The Operator automatically generates Prometheus scrape configuration based on the current state of the objects in the API server.PodMonitor, which declaratively specifies how group of pods should be monitored. The Operator automatically generates Prometheus scrape configuration based on the current state of the objects in the API server.PrometheusRule, which defines a desired set of Prometheus alerting and/or recording rules. The Operator generates a rule file, which can be used by Prometheus instances.