Search the Community

Amazon Managed Service for Prometheus collector, a fully-managed agentless collector for Prometheus metrics from Amazon EKS workloads, now supports AWS CloudFormation. Starting today, you can easily create, configure, and manage Amazon Managed Service for Prometheus collectors using CloudFormation templates. With AWS CloudFormation, you can use a programming language or simple text file to automatically configure collectors for Prometheus metrics from Amazon EKS infrastructure and applications. You can also continue utilizing the Amazon Managed Service for Prometheus collector using the AWS Management Console, Command Line Interface (CLI) or API. View the full article

This article will lead you through installing and configuring Prometheus, a popular open-source monitoring and alerting toolset, in a Kubernetes context. Prometheus is extensively used for cloud-native applications since it is built to monitor and gather metrics from many services and systems. This post will walk you through setting up Prometheus to successfully monitor your Kubernetes cluster. Prerequisites Before you begin, ensure you have the following prerequisites in place: View the full article

In the vibrant atmosphere of PromCon during the last week of September, attendees were treated to a plethora of exciting updates from the Prometheus universe. A significant highlight of the event has been the unveiling of the Perses project. With its innovative approach of dashboard as code, GitOps, and Kubernetes native features, Perses promises a […]View the full article

Imagine you’re piloting a spaceship through the cosmos, embarking on a thrilling journey to explore the far reaches of the universe. As the captain of this ship, you need a dashboard that displays critical information about your vessel, such as fuel levels, navigation data, and life support systems. This dashboard is your lifeline, providing you […]View the full article

30 Best DevOps Tools to Learn and Master In 2023: Git, Docker ... https://www.simplilearn.com/tutorials/devops-tutorial/devops-tools

Amazon Managed Service for Prometheus now provides Alert Manager & Ruler logs to help customers troubleshoot their alerting pipeline and configuration in Amazon CloudWatch Logs. Amazon Managed Service for Prometheus is a fully managed Prometheus-compatible monitoring service that makes it easy to monitor and alarm on operational metrics at scale. Prometheus is a popular Cloud Native Computing Foundation open source project for monitoring and alerting that is optimized for container environments. The Alert Manager allows customers to group, route, deduplicate, and silence alarms before routing them to end users via Amazon Simple Notification Service (Amazon SNS). The Ruler allows customers to define recording and alerting rules, which are queries that are evaluated at regular intervals. With Alert Manager and Ruler logs, customers can troubleshoot issues in their alerting pipelines including missing Amazon SNS topic permissions, misconfigured alert manager routes, and rules that fail to execute. View the full article

It’s every on-call’s nightmare—awakened by a text at 3 a.m. from your alert system that says there’s a problem with the cluster. You need to quickly determine if the issue is with the Amazon EKS managed control plane or the new custom application you just rolled out last week. Even though you installed the default dashboards the blogs recommended, you’re still having difficulty understanding the meaning of the metrics you are looking at. If only you had a dashboard that was focused on the most common problems seen in the field—one where you understood what everything means right away, letting you quickly scan for even obscure issues efficiently… View the full article

At its GrafanaCONline event, Grafana Labs today announced an update to the open source Grafana dashboard. The update adds visual query tools to make it easier for IT professionals of any skill level to launch queries against the Prometheus monitoring platform or the company’s Grafana Loki log aggregations framework. In addition, Grafana said the open […] View the full article

New associate certification exam from CNCF and The Linux Foundation will test foundational knowledge and skills using Prometheus, the open source systems monitoring and alerting toolkit Valencia, SPAIN, KubeCon + CloudNativeCon Europe – May 18, 2022 – The Cloud Native Computing Foundation® (CNCF®), which builds sustainable ecosystems for cloud native software, and The Linux Foundation, […] The post Prometheus Associate Certification will Demonstrate Ability to Monitor Infrastructure appeared first on DevOps.com. View the full article

Charlotte, NC, May 17, 2022 – NetFoundry is celebrating Prometheus Day with native secure networking connectivity for the leading open-source application monitoring tool. The company has embedded OpenZiti directly into Prometheus, the de facto standard for monitoring application performance in day one and day two operations. Prometheus is used in 86% of all cloud projects, […] The post NetFoundry Embeds Zero Trust Into Prometheus for Secure Monitoring Anywhere appeared first on DevOps.com. View the full article

Amazon Managed Service for Prometheus usage metrics are now available in Amazon CloudWatch at no additional charge. Amazon Managed Service for Prometheus is a fully managed Prometheus-compatible monitoring service that makes it easy to monitor and alarm on operational metrics at scale. Prometheus is a popular Cloud Native Computing Foundation open-source project for monitoring and alerting that is optimized for container environments. With Amazon CloudWatch usage metrics, you can check your Amazon Managed Service for Prometheus workspace usage, and can start to proactively manage your quotas. View the full article

Prometheus and Grafana can serve the needs of both on-premises or cloud-based companies, and Hosted Prometheus and Grafana by MetricFire can also be set up on-premises or on cloud. View the full article

This is a guest post from Viktor Petersson (@vpetersson) who discusses how Screenly uses Prometheus to monitor the thousands of Raspberry Pis powering their digital signage network. At Screenly, we are long-time Kubernetes fans, and we use Prometheus to monitor our infrastructure. Many, if not most, Kubernetes teams also use Prometheus to monitor and troubleshoot our infrastructure. Over the years, we have found Prometheus to be extremely versatile, and we have expanded our use of Prometheus to include business intelligence metrics. The one problem we have experienced is that it is painful to use Prometheus for the long-term storage of metrics. Weaveworks however solves that pain point with its hosted Prometheus as a Service (Cortex) within Weave Cloud. At Screenly we use Weave Cloud to store business metrics and use them as part of our troubleshooting toolkit. View the full article

Graphite and Prometheus are both great tools for monitoring networks, servers, other infrastructure, and applications. Both Graphite and Prometheus are what we call time-series monitoring systems, meaning they both focus on monitoring metrics that record data points over time. View the full article

This article will focus on the popular monitoring tool Prometheus, and how to use PromQL. Prometheus uses Golang and allows simultaneous monitoring of many services and systems. View the full article

Prometheus is a very popular open source monitoring and alerting toolkit originally built in 2012. Its main focus is to provide valid insight into system performance by providing a way for certain variables of that system to be monitored. View the full article

Founded in 2015, the CNCF (Cloud Native Computing Foundation) is a part of the nonprofit Linux foundation project. It serves as the home for several open-source projects like the Kubernetes, Envoy, and Prometheus. The CNCF has recently announced that Rook has now joined its family of graduated projects. View the full article

Do you wish you could use CloudWatch, but don't want to go all-in on AWS products? There's AWS Lambda, EKS, ECS, CloudWatch and more. View the full article

This is a tale with many twists and turns, a tale of observation, analysis and optimisation, of elation and disappointment. It starts with disk space. Wind back four years to get the background: Weaveworks created the Cortex project, which the CNCF have recently graduated to "incubating" status. Cortex is a time-series database system based on Prometheus. We run Cortex in production as part of Weave Cloud, ingesting billions of metrics from clusters all over the world and serving insight and analysis to their devops owners. I spend one week out of four on SRE duties for Weave Cloud, responding to alerts and looking for ways to make the system run better. Lessons learned from this then feed into our Weave Kubernetes commercial product. Coming on shift September 10th, I noticed that disk consumption by Cortex was higher than I remembered. We expect the product to grow over time, and thus to use more resources, but looking at the data there had been a marked jump a couple of weeks earlier, and consumption by all customers had jumped at the same time. It had to be caused by something at our end. Bit more background: Cortex doesn’t write every sample to the store as it comes in; instead it compresses hours of data into “chunks” which are much more efficient to save and retrieve. But machines sometimes crash, and if we lost a server we wouldn’t want that to impact our users, so we replicate the data to three different servers. Distribution and segmentation of time-series are very carefully arranged so that, when it comes time to flush each chunk to the store, the three copies are identical and we only store the data once. Reason I’m telling you this is, by looking at statistics about the store, I could see this was where the increased disk consumption was coming from: the copies very often did not match, so data was stored more than once. This chart shows the percentage of chunks detected as identical: on the left is from a month earlier, and on the right is the day when I started to look into the problem. OK, what causes Cortex chunks to be non-identical? Over to Jaeger to see inside a single ‘push’ operation: The Cortex distributor replicates incoming data, sending it to ingesters which compress and eventually store the chunks. Somehow, calls to ingesters were not being served within the two second deadline that the distributor imposes. Well that was a surprise, because we pay a lot of attention to latency, particularly the “p99 latency” that tells you the one-in-a-hundred situation. P99 is a good proxy for what customers may occasionally experience, and particularly notable if it’s trending worse. Here’s the chart for September 10th - not bad, eh? But, salutary lesson: Histograms Can Hide Stuff. Let’s see what the 99.9th centile looks like: So one in a thousand operations take over ten times as long as the p99 case! By the way, this is the “tail latency” in the title of this blog post: as we look further and further out into the tail of the distribution, we can find nasty surprises. That’s latency reported on the serving side; from the calling side it’s clearer we have a problem, but unfortunately the histogram buckets here only go up to 1 second: Here’s a chart showing the rate of deadline-exceeded events that day: for each one of these the data samples don’t reach one of the replicas, leading to the chunks-not-identical issue: It’s a very small fraction of the overall throughput, but enough to drive up our disk consumption by 50%. OK, what was causing these slow response times? I love a good mystery, so I threw myself into finding the answer. I looked at: Overloading. I added extra CPUs and RAM to our cloud deployment, but still the occasional delays continued. Locking. Go has a mutex profile, and after staring at it for long enough I figured it just wasn’t showing me any hundred-millisecond delays that would account for the behaviour. Blocking. Go has this kind of profile too, which shows when one part of the program is hanging around waiting for something like IO, but it turns out this describes most of Cortex. Nothing learned here. I looked for long-running operations which could be chewing up resources inside the ingester; one in particular from our Weave Cloud dashboard service was easily cached, so I did that, but still no great improvement. One of my rules of thumb when trying to improve software performance is “It’s always memory”. (Perhaps cribbed from Richard Stiles’ “It's the Memory, Stupid!”, but he was talking about microprocessor design). Anyway, looking at heap profiles threw up one candidate: the buffers used to stream data for queries could be re-used. I implemented that and the results looked good in the staging area, so I rolled it out to production. Here’s what I saw in the dashboard; rollout started at 10:36GMT: I was ecstatic. Problem solved! But. Let’s just open out that timescale a little. A couple of hours after the symptom went away, it was back again! Maybe only half as bad, but I wanted it fixed, not half-fixed. OK, what do we do when we can’t solve a performance problem? We stare at the screen for hours and hours until inspiration strikes. It had been great for a couple of hours. What changed? Maybe some customer behaviour - maybe someone started looking at a particular page around 12:30? Suddenly it hit me. The times when performance was good lined up with the times that DynamoDB was throttling Cortex. What the? That can’t possibly be right. About throttling: AWS charges for DynamoDB both by storage and by IO operations per second, and it’s most cost-effective if you can match the IO provision to demand. If you try to go faster than what you’re paying for, DynamoDB will throttle your requests, but because Cortex is already holding a lot of data in memory we don’t mind going slowly for a bit. The peaks and troughs even out and we get everything stored over time. So that last chart above shows the peaks, when DynamoDB was throttling, and the troughs, when it wasn’t, and those different regions match up exactly to periods of high latency and low latency. Still doesn’t make sense. The DB storage side of Cortex runs completely asynchronously to the input side, which is where the latency was. Well, no matter how impossible it seemed, there had to be some connection. What happens inside Cortex when DynamoDB throttles a write? Cortex waits for a bit then retries the operation. And it hit me: when there is no throttling, there is no waiting. Cortex will fire chunks into DynamoDB as fast as it will take them, and that can be pretty darn fast. Cortex triggers those writes from a timer - we cut chunks at maximum 8 hours - and that timer runs once a minute. In the non-throttled case there would be a burst of intense activity at the start of every minute, followed by a long period where things were relatively quiet. If we zoom right in to a single ingester we can see this in the metrics, going into a throttled period around 10:48: Proposed solution: add some delays to spread out the work when DynamoDB isn’t throttling. We already use a rate-limiter from Google elsewhere in Cortex, so all I had to do was compute a rate which would allow all queued chunks to be written in exactly a minute. The code for that still needs a little tweaking as I write this post. That new rate-limiting code rolled out September 16th, and I was very pleased to see that the latency went down and this time it stayed down: And the rate at which chunks are found identical, which brings down disk consumption, doesn’t recover until 8 hours after a rollout, but it’s now pretty much nailed at 66% where it should be: View the full article

The Linux Foundation has launched an advanced cloud engineer Bootcamp to take your career to the next level by enabling IT administrators to learn the most sought after cloud skills and get certified in six months. This Bootcamp covers the whole Kubernetes ecosystem from essential topics like containers, Kubernetes deployments, logging, Prometheus monitoring to advanced topics like service mesh. Basically all the skills required to work in a Kubernetes based project. And here is the best part. With this Bootcamp, you can take the Kubernetes CKA certification exam. It comes with one-year validity and a free retake. Here is the list of courses covered in the Bootcamp. Containers Fundamentals (LFS253) Kubernetes Fundamentals (LFS258) Service Mesh Fundamentals (LFS243) Monitoring Systems and Services with Prometheus (LFS241) Cloud-Native Logging with Fluentd (LFS242) Managing Kubernetes Applications with Helm (LFS244) Certified Kubernetes Administrator Exam (CKA) Advanced Cloud Engineer Bootcamp is priced at $2300 (List Price) but if you join before 31st July, you can get it for $599 (saves you $1700). You may also use the DCUBEOFFER coupon code at check out to get an additional 15% discount on total cart value (Applicable for CKA & CKAD certifications as well). Access Advanced Cloud Engineer Bootcamp Note*: It comes with a 30 days money back guarantee How The Cloud Engineer Bootcamp Work? The whole Bootcamp is designed for six months. All the courses in the Bootcamp are self-paced. Ideally, you should spend 10 hours per week for six months to complete all the courses in the Bootcamp. Even though the courses are self-paced, you will get access to interactive forums and live chat within course instructors. Every course is associated with hands-on labs and assignments to improve your practical knowledge. At the end of the Bootcamp, you can appear for the CKA exam completely free with one-year validity a free retake You will earn a valid advanced cloudeningeer bootcamp badge and CKA certification badge. Is Cloud Engineer Bootcamp Worth It? If you are an IT administrator or someone who wants to learn the latest cloud-native technologies, this is one of the best options as it focuses more on the practical aspects. If you look at the price, it’s worth it as you will have to spend $2300 if you buy those courses individually. Even the much sought after CKA certification will cost you $300. With an additional $300, you get access to all the other courses plus support for dedicated forums and live instructor sessions. So it is entirely on you how you make use of this Bootcamp. Like learning any technology, you have to put in your work using these resources. View the full article