Monitoring & Observability

Platform Engineering is a hot topic these days. We’ve seen the hype around it in 2023, and I expect we shall see it becoming production-grade as we move into 2024. I wanted to look into this topic, and learn from those who’ve already implemented it at scale: the e-commerce hyperscaler Shopify. In the latest episode […]View the full article

At Logz.io, we recently announced the release of App 360, a new solution that aims to shift the paradigm around application performance monitoring (APM) systems. To better give our customers a look at the new solution within the Logz.io Open 360™ platform for essential observability, we recently hosted a webinar explaining App 360 in greater […]View the full article

Amazon CloudWatch Logs is excited to announce support for creating account-level subscription filters using the put-account-policy API. This new capability enables you to deliver real-time log events that are ingested into Amazon CloudWatch Logs to an Amazon Kinesis Data Stream, Amazon Kinesis Data Firehose, or AWS Lambda for custom processing, analysis, or delivery to other destinations using a single account level subscription filter. View the full article

OpenSearch Service 2.11 now supports hybrid query score normalization. It is now easier than ever for search practitioners to leverage a combination of lexical and semantic search to improve their search relevance with OpenSearch. View the full article

In the ever-evolving landscape of software development and IT operations, monitoring tools play a pivotal role in ensuring the performance, reliability, and availability of your applications. Two key disciplines in this domain are observability and Application Performance Management (APM). This post will help you understand the nuances between observability and APM, exploring their unique characteristics, […]View the full article

Amazon OpenSearch Service now offers support for the Amazon Graviton2 instance family in six additional regions- Africa (Cape Town), Asia Pacific (Osaka), Europe (Zurich), Middle East (Bahrain). Israel (Tel Aviv), and AWS GovCloud (US-West). Graviton-based instances (C6g/M6g/R6g) in OpenSearch Service provide up to 30% better price-performance than comparable x86-based (C5/M5/R5) Amazon Elastic Compute Cloud instances. View the full article

Amazon OpenSearch Service adds support for Transport Layer Security (TLS) version 1.3 amongst its transport security options for domain endpoint security. TLS 1.3 offers customers enhanced security and performance as compared to older TLS versions. In addition, we now support perfect forward secrecy, which provides additional safeguards against eavesdropping of encrypted data, through the use of a unique random session key. View the full article

Introduction We have observed a growing adoption of container services among both startups and established companies. This trend is driven by the ease of deploying applications and migrating from on-premises environments to the cloud. One platform of choice for many of our customers is Amazon Elastic Container Service (Amazon ECS). The powerful simplicity of Amazon ECS allows customers to scale from managing a single task to overseeing their entire enterprise application portfolio and to reach thousands of tasks. Amazon ECS eliminates the management overhead associated with running your own container orchestration service. When working with customers, we have observed …

With 2023 drawing to a close, the final OpenObservability Talks of the year focused on what happened this year in open source, DevOps, observability and more, with an eye towards the future. I was delighted to be joined by a special guest, Kelsey Hightower, a renowned figure in the tech community, especially known for his […]View the full article

For businesses reliant on customers’ positive digital experiences to achieve their goals, the seamless operation of cloud applications and infrastructure is paramount for financial success. Observability holds a pivotal role in modern enterprises, offering critical insights into your IT system’s health and performance. However, persistent issues of complexity and high costs have plagued the observability […]View the full article

Amazon OpenSearch Service adds multimodal support on Neural Search for OpenSearch 2.11 deployments. This empowers builders to create and operationalize multimodal search applications with significantly reduced undifferentiated heavy-lifting. For years, customers have been building vector search applications on OpenSearch k-NN, but they’ve been burdened with building middleware to integrate text embedding models into search and ingest pipelines. OpenSearch builders can now power multimodal search through out-of-the-box integrations with Amazon Bedrock text and image multimodal APIs to power search pipelines that run on-cluster. View the full article

Post co-written by Shahar Azulay, CEO and Co-Founder at GroundCover Introduction The abstraction introduced by Kubernetes allows teams to easily run applications at varying scale without worrying about resource allocation, autoscaling, or self-healing. However, abstraction isn’t without cost and adds complexity and difficulty tracking down the root cause of problems that Kubernetes users experience. To mitigate these issues, detailed observability into each application’s state is key but can be challenging. Users have to ensure they’re exposing the right metrics, emitting actionable logs, and instrumenting their application’s code with specific client-side libraries …

As we continue to navigate the ongoing evolution of the observability landscape, Logz.io is constantly striving to provide our customers with the advanced platform capabilities needed to make sense of their increasingly complex environments. Sometimes that means taking a new approach to long-standing practices. For years, organizations have relied on application performance monitoring (APM) tools […]View the full article

The Migration Assistant for Amazon OpenSearch Service is an open-source AWS solution designed for the effortless migration of self-managed OpenSearch and Elasticsearch clusters to Amazon OpenSearch Service (Managed clusters and Serverless collections). View the full article

Years before founding Logz.io, I was a software engineer, working with various tools to ensure my products and services performed correctly. There were few tools I dreaded using more than application performance management (APM), and I know that I’m not alone. I hated traditional APM. It’s heavy. It’s hard to implement. It’s expensive. It takes […]View the full article

As applications in the cloud become more distributed and complex, the Mean Time To Resolution (MTTR) for production issues is getting longer. Modern systems are built with hundreds of distinct, ephemeral, and interconnected cloud components, which can make it exceptionally hard for engineers to understand the current state of their applications, what problems are impacting […]View the full article

We are excited to announce regular expression support for Amazon CloudWatch Logs filter pattern syntax, making it easier to search and match relevant logs in the AWS GovCloud (US) Regions. Customers use filter pattern syntax today to search logs, extract metrics using metric filters, and send specific logs to other destinations with subscription filters. With today’s launch, customers will be able to further customize these operations to meet their needs with flexible and powerful regular expressions within filter patterns. Now customers can define one filter to match multiple IP subnets or HTTP status codes using a regular expression such as ‘{ $.statusCode=%4[0-9]{2}% }…

Amazon OpenSearch Service introduces OR1, the OpenSearch Optimized Instance family, that delivers up to 30% price-performance improvement over existing instances in internal benchmarks and uses Amazon S3 to provide 11 9s of durability. The new OR1 instances are best suited for indexing-heavy workloads, and offers better indexing performance compared to the existing memory optimized instances available on OpenSearch Service. View the full article

You can use the new OR1 instances to create Amazon OpenSearch Service clusters that use Amazon Simple Storage Service (Amazon S3) for primary storage. You can ingest, store, index, and access just about any imaginable amount of data, while also enjoying a 30% price/performance improvement over existing instance types, eleven nines of data durability, and a zero-time Recovery Point Objective (RPO). You can use this to perform interactive log analytics, monitor application in real time, and more. New OR1 Instances These benefits are all made possible by the new OR1 instances, which are available in eight sizes and used for the data nodes of the cluster: Ins…

Amazon OpenSearch Service zero-ETL integration with Amazon S3, a new way for customers to query operational logs in Amazon S3 and S3-based data lakes without needing to switch between tools to analyze operational data, is available for customer preview. Customers can boost the performance of their queries and build fast-loading dashboards using the built-in query acceleration capabilities of Amazon OpenSearch Service zero-ETL integration with Amazon S3. View the full article

Today, AWS announces the general availability of vector engine for Amazon OpenSearch Serverless. Vector engine for OpenSearch Serverless is a simple, scalable, and high-performing vector database which makes it easier for developers to build machine learning (ML)–augmented search experiences and generative artificial intelligence (AI) applications without having to manage the underlying vector database infrastructure. Developers can rely on the vector engine's cost-efficient, secure, and mature serverless platform to seamlessly transition from application prototyping to production. View the full article

Today we are announcing a preview of Amazon OpenSearch Service zero-ETL integration with Amazon S3, a new way to query operational logs in Amazon S3 and S3-based data lakes without needing to switch between services. You can now analyze infrequently queried data in cloud object stores and simultaneously use the operational analytics and visualization capabilities of OpenSearch Service. Amazon OpenSearch Service direct queries with Amazon S3 provides a zero-ETL integration to reduce the operational complexity of duplicating data or managing multiple analytics tools by enabling customers to directly query their operational data, reducing costs and time to action. This zer…

Today we are announcing the general availability of the vector engine for Amazon OpenSearch Serverless with new features. In July 2023, we introduced the preview release of the vector engine for Amazon OpenSearch Serverless, a simple, scalable, and high-performing similarity search capability. The vector engine makes it easy for you to build modern machine learning (ML) augmented search experiences and generative artificial intelligence (generative AI) applications without needing to manage the underlying vector database infrastructure. You can now store, update, and search billions of vector embeddings with thousands of dimensions in milliseconds. The highly performant…

Amazon DynamoDB zero-ETL integration with Amazon OpenSearch Service provides customers advanced search capabilities, such as full-text and vector search, on their Amazon DynamoDB data. With a few button clicks in the AWS console, customers can now seamlessly synchronize their data from Amazon DynamoDB to Amazon OpenSearch Service, eliminating the need to write any custom code to extract, transform, and load the data. Amazon DynamoDB zero-ETL integration with Amazon OpenSearch Service is now available for both Amazon OpenSearch Service managed clusters and serverless collections. View the full article

Today, we are announcing the general availability of Amazon DynamoDB zero-ETL integration with Amazon OpenSearch Service, which lets you perform a search on your DynamoDB data by automatically replicating and transforming it without custom code or infrastructure. This zero-ETL integration reduces the operational burden and cost involved in writing code for a data pipeline architecture, keeping the data in sync, and updating code with frequent application changes, enabling you to focus on your application. With this zero-ETL integration, Amazon DynamoDB customers can now use the powerful search features of Amazon OpenSearch Service, such as full-text search, fuzzy search…

Logz.io is excited to announce Service Map, a new way to visualize the data flow, dependencies, and critical performance metrics throughout your microservices architecture, which makes it easy to gather critical troubleshooting context as you investigate production issues. After sending your trace data to Logz.io Open 360™, Service Map automatically discovers and maps your services […]View the full article

To make it easy to interact with your operational data, Amazon CloudWatch is introducing today natural language query generation for Logs and Metrics Insights. With this capability, powered by generative artificial intelligence (AI), you can describe in English the insights you are looking for, and a Logs or Metrics Insights query will be automatically generated. This feature provides three main capabilities for CloudWatch Logs and Metrics Insights: Generate new queries from a description or a question to help you get started easily. Query explanation to help you learn the language including more advanced features. Refine existing queries using guided iteration…

Searching through log data to find operational or business insights often feels like looking for a needle in a haystack. It usually requires you to manually filter and review individual log records. To help you with that, Amazon CloudWatch has added new capabilities to automatically recognize and cluster patterns among log records, extract noteworthy content and trends, and notify you of anomalies using advanced machine learning (ML) algorithms trained using decades of Amazon and AWS operational data. Specifically, CloudWatch now offers the following: The Patterns tab on the Logs Insights page finds recurring patterns in your query results and lets you analyze them…

Here at Logz.io, we realize Kubernetes is the most common infrastructure component that organizations are running on to keep their applications going. In return, we’ve made a big investment to support Kubernetes properly and give customers the tools they need to investigate and troubleshoot any issues that arise. Check out the replay of our webinar […]View the full article

OpenSearch Service 2.11 now comes with OpenSearch Neural Sparse Retrieval. Search practitioners now have an additional search method to use for their search applications with improved semantic understanding, while keeping computational cost and computational latency low, more in line with lexical search. View the full article

Amazon OpenSearch Service now supports Neural Search on OpenSearch 2.9, enabling builders to create and operationalize semantic search applications with reduced undifferentiated heavy-lifting. For years, customers have been building semantic search applications on OpenSearch k-NN, but they’ve been burdened with building middleware to integrate text embedding models into search and ingest pipelines. Amazon OpenSearch Service customers can power Neural Search through integrations with Amazon SageMaker and Amazon Bedrock enabling semantic search pipelines that run on-cluster. View the full article

When things go wrong, we’d all love the ability to go back in time, return things to the way they were, and fix whatever issues pop up at the start so they never happen in the first place. This is no different when maintaining complex microservices-based architectures. With any complex system, things are bound to […]View the full article

Do you find yourself lying awake late at night, worried that your greatest observability fears will materialize as one of the most horrific specters of Kubernetes-driven chaos reaches up through your mattress to consume your very soul? Even as your mind races and you wonder just who that creepy character sneaking around the metaphysical boiler […]View the full article

In the first part of our 2023 PromCom recap, we spent OpenObservability Talks exploring the Perses open source project. We found heavy users of open source Grafana who found themselves grappling with issues arising from managing a vast number of dashboards, and the need to manage dashboards as code in a GitOps fashion. In this […]View the full article

Amazon OpenSearch Service adds support for four new language analyzer plugins- Nori (Korean), Sudachi (Japanese), Pinyin (Chinese), and STConvert Analysis (Chinese) plugins. These are available as optional plugins that you can associate with your OpenSearch Service clusters. View the full article

Amazon CloudWatch announces out-of-the box, best practice alarm recommendations for AWS service-vended metrics. It provides alarm recommendations and alarm configurations for key vended metrics, along with the ability to download pre-filled infrastructure-as-code templates for these alarms. Furthermore, you can now see in-line descriptions for AWS service metrics across the AWS console, which enables you to easily see metric details to help you troubleshoot or assess system health. View the full article

Search Pipelines, a new feature in OpenSearch 2.9, make it easy to build query and result processing pipelines. This lets you build search query and result processing as a composition of modular processing steps without complicating your application software. View the full article

Starting today, we are introducing a new Amazon CloudWatch metric called Attached EBS Status Check to monitor if one or more Amazon EBS volumes attached to your EC2 instances are reachable and able to complete I/O operations. With this new metric, you can now quickly detect and respond to any EBS impairments that may potentially be impacting the performance of your applications running on Amazon EC2 instances. View the full article

You can now launch Amazon CloudWatch Internet Monitor directly from the Amazon Virtual Private Cloud (VPC) console. Internet Monitor provides visibility into how internet issues impact the performance and availability for your application hosted on AWS. To use Internet Monitor, you create a monitor and associate it with one or more resources: VPCs, Network Load Balancers, Amazon CloudFront distributions, or Amazon WorkSpaces directories. View the full article

OpenSearch Service 2.9 now supports the ability for customers to manage and overlay alerts and anomalies onto dashboard visualization line charts. Customers can create new, or associate existing, alerting monitors and anomaly detectors from dashboard line charts. For example, if a customer opts to create a new monitor or detector, the new monitor will inherit the setting of the line chart and prepopulate the creation form. If customers have existing monitors or detectors present that they want to use, they can associate them with the line chart visualization on the dashboard. View the full article

OpenSearch Service 2.9 now comes with OpenSearch Service Integrations, where customers can take advantage of new schema standards such as Open Telemetry and build dashboards based on an agreed up on schema between your ingestion pipeline and OpenSearch Service. View the full article

You can now run OpenSearch version 2.9 in Amazon OpenSearch Service. With OpenSearch 2.9, we have made several improvements to Search, Observability, Security analytics, and Machine Learning (ML) capabilities in OpenSearch Service. View the full article

We are excited to announce that the Amazon OpenSearch Service has expanded its geospatial capacities. With new aggregation support in version 2.9, you can do more statistical analysis on your data, making it simpler to draw conclusions and interpret them. View the full article

Amazon OpenSearch Service now provides new Auto-Tune metrics and improved Auto-Tune events that give you better visibility into the cluster performance optimizations made by Auto-Tune. View the full article

Customers can now monitor and troubleshoot application environments that span multiple accounts (within a region) using Amazon CloudWatch Application Insights. CloudWatch Application Insights helps customers gain actionable insights for their application environments by making it easier for them to set up and monitor, recognize problems, and use data to make decisions. With this release, customers can analyze and correlate cross-account telemetry data and problems through a centralized view of the monitoring results across their accounts. View the full article

Amazon CloudWatch announces support of a new Metric Math function called DB_PERF_INSIGHTS() to create CloudWatch alarms and dashboards on Amazon RDS Performance Insights metrics. View the full article

Logz.io is thrilled to have earned over 20 Fall 2023 G2 Badges for our Logz.io Open 360™ essential observability platform! G2 Research is a tech marketplace where people can discover, review, and manage the software they need to reach their potential. We’ve earned the following Fall 2023 G2 Badges for Application Performance Monitoring (APM) and […]View the full article

Amazon CloudWatch Synthetics announces a new update to Synthetics Python runtime version syn-python-selenium-2.0 and recommends that customers migrate Synthetics canaries to the latest runtime version. Runtime version syn-python-selenium-2.0 includes updates to third-party dependency packages (Selenium v4.10.0 and Chromium v111.0.5563.146). View the full article

We are excited to announce that Amazon OpenSearch Serverless has expanded its auto-scaling capabilities to efficiently handle tens of thousands of query transactions per minute. With this new feature, you can rely on OpenSearch Serverless to help handle unpredictable surges in your search and query traffic. View the full article

We are excited to announce regular expression support for Amazon CloudWatch Logs filter pattern syntax, making it easier to search and match relevant logs. Customers use filter pattern syntax today to search logs, extract metrics using metric filters, and send specific logs to other destinations with subscription filters. With today’s launch, customers will be able to further customize these operations to meet their needs with flexible and powerful regular expressions within filter patterns. Now customers can define one filter to match multiple IP subnets or HTTP status codes using a regular expression such as ‘{ $.statusCode=%4[0-9]{2}% }’ rather than having to define mu…

We are excited to announce the integration of Amazon OpenSearch Serverless with AWS User Notifications. OpenSearch Serverless is the serverless option for Amazon OpenSearch Service that makes it simple for you to run search and analytics workloads without having to think about infrastructure management. View the full article

We are excited to announce that Amazon OpenSearch Serverless can now scan and search up to 6TB of time series data which includes one or more indexes within a collection. OpenSearch Serverless is a serverless deployment option for Amazon OpenSearch Service that makes it simple for you to run search and analytics workloads without having to think about infrastructure management. With the support for larger datasets, you can unlock valuable operational insights and make data driven decisions to troubleshoot application downtime, improve system performance, or identify fraudulent activities. View the full article

Amazon Personalize launches a new integration with self-managed OpenSearch that enables customers to personalize search results for each user and assists in predicting their search needs. The Amazon Personalize Search Ranking plugin within OpenSearch helps customers to leverage the deep learning capabilities offered by Amazon Personalize and add personalization to OpenSearch search results, without any ML expertise. View the full article

You can now run OpenSearch version 2.7 in Amazon OpenSearch Service. With OpenSearch 2.7, we have made several improvements to observability, security analytics, index management, and geospatial capabilities in OpenSearch Service. View the full article

Amazon OpenSearch Service now lets you provision up to 16,000 IOPS and 1000 MiB/s throughput for every 3 TiB gp3 volume size provisioned per data node, enabling you to achieve better search and indexing performance. View the full article

Amazon OpenSearch Service now lets you update cluster manager (master node) instance type or instance count without requiring a blue/green deployment, helping you complete the updates faster with the least potential disruption to your cluster operations and without involving any data movement. View the full article

Amazon OpenSearch Ingestion now allows you to ingest events from Amazon Security Lake in real-time, reducing the time taken to index your security data in Amazon OpenSearch Service and uncover valuable insights into potential security issues. Amazon Security Lake automatically centralizes security data from AWS environments, SaaS providers and on- premises into a purpose-built data lake. With this integration, customers can now use the extensive security analytics capabilities and rich dashboard visualizations of Amazon OpenSearch Service to quickly make sense of all their security data. View the full article

Amazon Managed Grafana now supports Trace Analytics with the OpenSearch Grafana data source plugin, in addition to the existing support for Log Analytics. Amazon Managed Grafana is a fully managed service for Grafana, a popular open-source analytics platform that enables you to query, visualize, and alert on your metrics, logs, and traces. OpenSearch is an open-source search and analytics engine for use cases such as log analytics, observability, real-time application monitoring, and clickstream analysis that is also offered as a fully managed Amazon OpenSearch Service. View the full article

Amazon CloudWatch Logs is excited to announce a new Logs Insights command, dedup, which enables customers to eliminate duplicate results when analyzing logs. Customers frequently want to query their logs and view only unique results based on one or more fields. You can now use the new dedup command in your Amazon CloudWatch Logs Insights queries to view unique results based on one or more fields. For example, you can view the most recent error message for each hostname by executing the dedup command on the hostname field. View the full article

Amazon OpenSearch Service now supports a new ‘skip unavailable’ setting for cross cluster search connections. If skip unavailable is enabled on connections, cross-cluster search ignores any remote cluster that might not available during the search. View the full article

Amazon CloudWatch Logs is excited to announce support for account level data protection policy configuration, you can now create a data protection policy that will be applied to all existing and future log groups within your AWS account. View the full article

We are excited to announce Amazon CloudWatch Logs Live Tail, a new interactive log analytics experience feature that helps you detect and debug anomalies in applications. You can now view your logs interactively in real-time as they’re ingested, which helps you to analyze and resolve issues across your systems and applications. View the full article

Amazon Kinesis Data Firehose customers can now send data to Amazon OpenSearch Service using OpenSearch Service auto-generated document ID option. This configuration option enables write-heavy operations, such as log analytics and observability, to consume fewer CPU resources at the OpenSearch domain, resulting in improved performance. View the full article

Today, Amazon OpenSearch Service announces Multi-AZ with Standby, a new deployment option that enables 99.99% availability and consistent performance for business-critical workloads. With Multi-AZ with Standby, OpenSearch Service domains are resilient to potential infrastructure failures, such as a node or an Availability Zone (AZ) failure. Multi-AZ with Standby also ensures OpenSearch Service domains follow recommended best practices, simplifying configuration and management. View the full article

We are excited to announce Amazon CloudWatch Logs data protection is now available in Middle East (UAE), Asia Pacific (Hyderabad), Europe (Spain), Europe (Zurich), and Asia Pacific (Melbourne). Data protection is a feature that leverages pattern matching and machine learning capabilities to detect and protect sensitive log data-in-transit. Amazon CloudWatch Logs enables you to centralize the logs from all of your systems, applications, and AWS services, in a single, highly scalable service. With log data protection in Amazon CloudWatch Logs, you can now detect and protect sensitive log data in-transit such as, Credit Card Numbers or Government ID’s logged by your systems,…

Today Amazon OpenSearch Service announces Amazon OpenSearch Ingestion, a new fully managed data ingestion tier that allows you to ingest and process petabyte-scale data before indexing it in OpenSearch-managed clusters or serverless collections. OpenSearch Ingestion provides a no-code capability to filter, transform, redact, and route data prior to indexing it in OpenSearch. OpenSearch Ingestion automatically provisions and scales the underlying resources for even the most demanding and unpredictable workloads. View the full article

Today we launched new observability features including log patterns, metrics analytics and support for Jaeger traces with OpenSearch 2.5 in Amazon OpenSearch Service. View the full article

Amazon CloudWatch Logs now supports ingesting enriched metadata introduced in Amazon Virtual Private Cloud (Amazon VPC) flow logs as part of versions 3 to 5 additional to the default fields. This launch includes metadata fields that provide more insights about the network interface, traffic type, and the path of egress traffic to the destination. View the full article

Amazon OpenSearch Service announces security analytics that provides new threat monitoring, detection, and alerting features. These capabilities help you to detect and investigate potential security threats that may disrupt your business operations or pose a threat to sensitive organizational data. View the full article

You can now run OpenSearch and OpenSearch Dashboards version 2.5 in Amazon OpenSearch Service. With OpenSearch 2.5, OpenSearch Service adds several new features and enhancements such as support for Security Analytics, support for Point in Time Search, improvements to observability and geospatial functionality. View the full article

We are excited to announce support for the Amazon Graviton2 instance family in four additional regions. Supported instance types include compute optimized (C6g), general purpose (M6g), and memory optimized (R6g) instances. Support for C6g, M6g, and R6g is available in the Asia Pacific (Hyderabad), Asia Pacific (Jakarta), Asia Pacific (Melbourne), and Europe (Milan) regions. View the full article

Amazon OpenSearch Service now lets you schedule service software and auto-tune updates during off-peak hours, helping you plan deployments to your domain better. In addition, with improved notifications through EventBridge events, and notifications on the OpenSearch Service console, you have better visibility of scheduled updates, when the updates start, and complete. View the full article

Amazon OpenSearch Service now supports enabling Security Assertion Markup Language (SAML) authentication for OpenSearch Dashboards during domain creation. SAML authentication for OpenSearch Dashboards enables users to integrate directly with identity providers (IDPs) such as Okta, Ping Identity, OneLogin, Auth0, Active Directory Federation Services (ADFS) and Azure Active Directory. View the full article

AWS IoT Core announces General Availability of the capability to send device logs from Internet of Things (IoT) devices to Amazon CloudWatch Logs in batches, enabling you to optimize the cost of using CloudWatch Log Action in IoT Rules. View the full article

Amazon OpenSearch Service adds a new connection mode for cross-cluster connection, simplifying the setup required to remote reindex between a local domain and remote VPC domains. Remote reindex enables you to migrate data from a source domain to a target domain. Remote reindex is also useful when you have to upgrade your clusters across multiple major versions. View the full article

Now generally available, Amazon OpenSearch Serverless is a new serverless option for Amazon OpenSearch Service. OpenSearch Serverless streamlines the process of running petabyte-scale search and analytics workloads without having to configure, manage, or scale OpenSearch clusters. OpenSearch Serverless automatically provisions and scales the underlying resources to deliver fast data ingestion and query responses for even the most demanding and unpredictable workloads. With OpenSearch Serverless, you pay only for the resources consumed. View the full article

Amazon OpenSearch Service now lets you validate configuration changes before applying them to your clusters. With the enhanced dry run option, Amazon OpenSearch Service checks for validation errors that might occur when deploying your configuration changes and provides a summary of these errors, if any. The dry run feature will also indicate whether a blue/green deployment will be required to apply a change, so that you can plan for these changes accordingly. View the full article

Starting today, Amazon CloudWatch Logs is removing the 5 requests per second log stream quota when calling Amazon CloudWatch Logs PutLogEvents API. There will be no new per log stream quota. With this change we have removed the need for splitting your log ingestion across multiple log streams to prevent log stream throttling. View the full article

We are excited to announce support for the Amazon Graviton2 instance family in four additional regions. Supported instance types include general purpose (M6g), compute optimized (C6g), and memory optimized (R6g and R6GD) instances. Support for C6g, M6g, R6g and R6gd is available in the Europe (Paris) region. In addition, the Asia Pacific (Mumbai), South America (Sao Paulo), and Canada (Central) regions already supported C6g, M6g and R6g instance families, and we have now added support for R6gd in these regions. View the full article

Amazon Kinesis Data Firehose can now deliver streaming data to an Amazon OpenSearch Serverless. With few clicks, you can easily ingest, transform, and reliably deliver streaming data into an Amazon OpenSearch Serverless without building and managing your own data ingestion and delivery infrastructure. Kinesis Data Firehose is a fully managed service that automatically scales to match the throughput of your data and without ongoing administration. View the full article

Amazon OpenSearch Service now offers a new serverless option, Amazon OpenSearch Serverless. This option simplifies the process of running petabyte-scale search and analytics workloads without having to configure, manage, or scale OpenSearch clusters. OpenSearch Serverless automatically provisions and scales the underlying resources to deliver fast data ingestion and query responses for even the most demanding and unpredictable workloads. With OpenSearch Serverless, you pay only for the resources consumed. View the full article

Most AWS analytics services have compelling serverless offerings that make it even easier for customers to analyze vast amounts of data without having to configure, scale, or manage the underlying infrastructure. Along with other serverless analytics, such as Amazon QuickSight for business intelligence and AWS Glue for data integration, we have introduced Amazon EMR Serverless, Amazon MSK Serverless, and Amazon Redshift Serverless this year. Today, we announce the preview release of a new serverless option for Amazon OpenSearch Service that makes it easy for customers to run large-scale search and analytics workloads without managing clusters. It automatically provisi…

Today we are announcing Amazon CloudWatch Logs data protection, a new set of capabilities for Amazon CloudWatch Logs that leverage pattern matching and machine learning (ML) to detect and protect sensitive log data in transit. While developers try to prevent logging sensitive information such as Social Security numbers, credit card details, email addresses, and passwords, sometimes it gets logged. Until today, customers relied on manual investigation or third-party solutions to detect and mitigate sensitive information from being logged. If sensitive data is not redacted during ingestion, it will be visible in plain text in the logs and in any downstream system that con…

We are excited to announce data protection in Amazon CloudWatch Logs, a new set of capabilities that leverage pattern matching and machine learning capabilities to detect and protect sensitive log data-in-transit. Amazon CloudWatch Logs enables you to centralize the logs from all of your systems, applications, and AWS services, in a single, highly scalable service. With log data protection in Amazon CloudWatch Logs, you can now detect and protect sensitive log data-in-transit logged by your systems, and applications. View the full article

Amazon OpenSearch Service now supports OpenSearch and OpenSearch Dashboards version 2.3. With this version, Amazon OpenSearch Service adds several features such as new algorithms to the machine learning (ML) commons library, improvements to aggregations, improvements to map visualizations, alerting, anomaly detection, and more. View the full article

Amazon CloudWatch Logs now supports exporting logs to Amazon Simple Storage Service (S3) buckets encrypted using Server side encryption with KMS (SSE-KMS) keys. View the full article

Amazon OpenSearch Service now supports managed VPC endpoints (powered by AWS PrivateLink) to connect to your Amazon OpenSearch Service VPC-enabled domain in a Virtual Private Cloud (VPC). With an Amazon OpenSearch Service managed endpoint, you can now privately access your OpenSearch Service domain within your VPC from your client applications in other VPCs, within the same or across AWS accounts, without using public IPs or requiring traffic to traverse the Internet. View the full article

Amazon Web Services (AWS) is pleased to announce the new Amazon OpenSearch Service Delivery specialization for AWS Partners that help customers perform interactive log analytics, real-time application monitoring, website search, and more. Amazon OpenSearch Service manages software installation, upgrades, patching, scaling (up to 3 PB), and cross-region replication with no downtime. Amazon OpenSearch Service is also bundled with a dashboard visualization tool, OpenSearch Dashboards. This tool helps visualize not only log and trace data, but also machine-learning powered results for anomaly detection and search relevance ranking. View the full article

Amazon Managed Service for Prometheus now provides Alert Manager & Ruler logs to help customers troubleshoot their alerting pipeline and configuration in Amazon CloudWatch Logs. Amazon Managed Service for Prometheus is a fully managed Prometheus-compatible monitoring service that makes it easy to monitor and alarm on operational metrics at scale. Prometheus is a popular Cloud Native Computing Foundation open source project for monitoring and alerting that is optimized for container environments. The Alert Manager allows customers to group, route, deduplicate, and silence alarms before routing them to end users via Amazon Simple Notification Service (Amazon SNS). The R…

Amazon OpenSearch Service now provides improved visibility into validation failures during domain updates. You can monitor the progress of a domain update, which could involve a blue/green deployment, from the OpenSearch Service console, or through the configuration APIs. OpenSearch Service will publish any validation failure events to Amazon EventBridge. You can also view these validation events in the Notifications tab of the OpenSearch Service console. View the full article

Starting today, customers of AWS Cost Anomaly Detection will see a new interface in the console, where they view and analyze anomalies and their root causes. AWS Cost Anomaly Detection monitors customers’ spending patterns to detect and alert on anomalous (increased) spend, and to provide root cause analyses. View the full article

Amazon CloudWatch Synthetics, an outside-in monitoring capability to continually verify your customer experience even when you don’t have any customer traffic on your applications, introduced a new capability to create custom groups of canaries. By creating a group of canaries, you can track success/failure status at a group or application level yet with an easy drill down to the failing canary, making it easier to pinpoint the canary failures in the context of the group or application. When groups consist of canaries across multiple AWS regions, this new capability allows you to more easily isolate region-specific issues. View the full article

Amazon DocumentDB (with MongoDB compatibility) is a database service that is purpose-built for JSON data management at scale, fully managed and integrated with AWS, and enterprise-ready with high durability. View the full article

Amazon OpenSearch Service, with the availability of OpenSearch 1.3., now gives customers the ability to organize their logs, traces and visualizations in an application-centric view. Customers can also benefit from enhanced log monitoring support with live tailing of logs, the ability to see surrounding log data, and the ability to do powerful ad-hoc analysis of unformatted log data at query time. View the full article

Today, we are announcing the general availability of a new feature, Log Anomaly Detection and Recommendations for Amazon DevOps Guru. As part of this feature, DevOps Guru will ingest Amazon CloudWatch Logs for AWS resources that make up your application, with Lambda being first. Logs will provide new enrichment data in an insight to enable more accurate understanding of the root cause behind an application issue, and provide more precise remediation steps. View the full article

EC2 Auto Scaling now publishes predictive scaling policy’s forecasts as a CloudWatch metric, enabling you to analyze, monitor, and set alarms on the accuracy of predictive scaling. Predictive Scaling is a scaling policy that proactively increases the capacity of your Auto Scaling group ahead of predicted demand, improving the availability of your application while reducing the need to stay overprovisioned that otherwise would have increased your EC2 bill. As predictive scaling only increases the capacity for your Auto Scaling groups, applying it to your current scaling configurations strictly enhances your application availability. However, an inaccurate prediction can po…

Amazon OpenSearch Service now allows users to view default quota and applied quota information through Service Quotas. Quotas, also referred to as limits in AWS services, are the maximum values for the resources, actions, and items in your AWS account. Each AWS service defines its quotas and establishes default values for those quotas. Depending on your business needs, you might need to increase your service quota values. Service Quotas enables you to look up your service quotas and to request quota increase. AWS Support might approve, deny, or partially approve your requests. View the full article

Developers can now access Amazon CloudWatch Logs within Visual Studio using the AWS Toolkit for Visual Studio. Directly from the IDE, it is now possible to search and filter log groups, log streams, and events. Additionally, log groups can be accessed from their associated resources, and log events can be downloaded to a file. View the full article

Amazon QuickSight now supports monitoring of QuickSight assets by sending metrics to Amazon CloudWatch. QuickSight developers and administrators can use these metrics to observe and respond to the availability and performance of their QuickSight ecosystem in near real time. They can monitor dataset ingestions, dashboards, and visuals to provide their readers with a consistent, performant, and uninterrupted experience on QuickSight. For more information, visit here. View the full article

It’s every on-call’s nightmare—awakened by a text at 3 a.m. from your alert system that says there’s a problem with the cluster. You need to quickly determine if the issue is with the Amazon EKS managed control plane or the new custom application you just rolled out last week. Even though you installed the default dashboards the blogs recommended, you’re still having difficulty understanding the meaning of the metrics you are looking at. If only you had a dashboard that was focused on the most common problems seen in the field—one where you understood what everything means right away, letting you quickly scan for even obscure issues efficiently… View the full ar…