Skip to main content
Welcome! In this lesson, we’ll explore how to create and configure metric filters in CloudWatch Logs to extract actionable metrics from your log data. These metrics can be used to trigger alarms, set thresholds, and automate various remediation processes. Metric filters in CloudWatch enable you to scan logs from your systems for specific patterns, phrases, or numerical data. When CloudWatch detects these patterns, it generates corresponding metrics that can automatically trigger alarms, start remediation actions, send notifications, or update dashboards.
The image is a flowchart illustrating the process of using a metric filter with AWS CloudWatch, starting from an Amazon Elastic Compute Cloud (EC2) instance with CloudWatch Agent, moving through a CloudWatch Log Group, Metric Filter, Alarm, and ending with SNS.

​
How Metric Filters Work

The process starts by selecting a log group where you want to search for specific patterns. Here’s the typical workflow:
  1. Select a Log Group: Choose the group of logs where you want to search for a particular pattern.
  2. Define a Filter Pattern: For example, to monitor error messages, you might use the keyword “error”.
  3. Assign a Metric Value: Every log event that matches the pattern is assigned a metric value (e.g., incrementing an “ErrorCount” metric).
Once the pattern is detected, CloudWatch creates a metric that you can use for setting thresholds, triggering alarms, or visualizing data on dashboards. This conversion of log data to metrics is the cornerstone of automated monitoring and remediation.
The image is a step-by-step guide for creating a metric filter, consisting of five steps: choosing a log group, defining a filter pattern, assigning a metric, setting the metric value, and saving and monitoring.
Think of metric filters as checkpoints that scan your logs for important information. Once a matching piece of data is found, it is translated into a metric, opening up options for monitoring, alarming, and even automated issue resolution.

​
Defining Filter Patterns

One of the most critical aspects of metric filters is the accuracy of your filter patterns. For example, consider a scenario where you want to filter Amazon Simple Storage Service (Amazon S3) logs. You might use a filter pattern such as: 
Filter pattern="aws:s3"
In this setup, the filter searches for events related to S3. You can further validate this pattern by testing it against your log data (e.g., using CloudTrail logs to find S3 bucket access control events). For logs in JSON format, you can target specific fields. If you need to monitor events where the “bytesTransferredOut” field exceeds 500, your filter pattern might look like this: 
Filter pattern:
{ ($.additionalEventData.bytesTransferredOut > 500) }

Select log data to test:
605134445133_CloudTrail_us-east-1_4

Log event messages:
{"SignatureVersion":"SigV4","CipherSuite":"TLS_AES_128_GCM_SHA256","bytesTransferredIn":0,"AuthenticationMethod":"AuthHead","x-amz-id-..."}

​
Monitoring HTTP 404 Errors

Let’s consider a practical example: monitoring HTTP 404 errors. Since a 404 status code indicates a failed resource request, it is essential to keep an eye on such occurrences. Given the following log entries: 
2024-09-10 12:34:21 GET /home 200 OK
2024-09-10 12:34:22 GET /login 404 Not Found
2024-09-10 12:34:23 POST /register 500 Server Err
2024-09-10 12:34:25 GET /product/1234 404 Not Found
You would define your filter like this: 
Filter Pattern: "404"
Metric Value: 1
This configuration creates a metric (for example, “404ErrorCount”) that increments by one for every 404 error detected. This metric can then be used to establish thresholds and alarms. For instance, if the count of 404 errors exceeds a specific limit within a defined period, you can trigger an alarm to notify you immediately. Here is an example configuration in YAML format: 
Metric Filter:
  Filter Pattern: "404"
  Metric Value: 1

Metric Name: 
  404ErrorCount
Once the metric filter is in place, you can configure a CloudWatch alarm to monitor the “404ErrorCount” metric. The alarm will trigger whenever the error count exceeds your set threshold—ensuring that any issues affecting your users are promptly addressed.
The image is a flowchart illustrating the process of using a CloudWatch metric to create an alarm, which triggers an SNS notification when the metric value meets a specified condition.

​
Best Practices for Creating Metric Filters

To ensure your metric filters are both effective and efficient, consider the following best practices:
  • Use clear and simple patterns that focus on log data with a high impact on user experience.
  • For JSON-formatted logs, leverage specific fields to narrow down your search efficiently.
  • Regularly test and refine your filter patterns, especially after any updates to your application.
  • Combine metric filters with CloudWatch alarms to establish a robust system for monitoring, notifications, and automated remediation.
The image outlines best practices for metric filters, including using clear patterns, focusing on high-impact data, leveraging JSON fields, regularly updating filters, and combining filters with CloudWatch Alarms.
For more detailed information on CloudWatch metric filters and alarms, please refer to the AWS CloudWatch Documentation.
We’ll catch you in the next lesson. Happy monitoring!

Watch Video