Introduction
CloudWatch Logs is a great place to aggregate your logs from your EC2 instances so you can monitor and set alerts on errors in your applications. This is especially helpful if you are using Auto Scaling, where EC2 instances will come and go; unless you ship them to CloudWatch from your instances, you will lose log events. In addition to being a place to aggregate application log events, you can use CloudWatch to set up alerts and build dashboards. In this post I’ll show you a couple of tips and tricks to collect logs from your instances and ship them to CloudWatch.
Install the Unified CloudWatch Agent
In this article, we’ll be talking about the newer unified CloudWatch Agent, which has replaced the older CloudWatch Logs agent. It allows you to collect EC2 instance logs and metrics and ship them to CloudWatch.
I recommend following AWS documentation to install and configure the unified CloudWatch agent. Here are the four steps:
- Create an IAM role your instances will use to export logs to CloudWatch
- Install the agent
- Create the configuration file
- Start the agent
The hardest part is getting the agent configuration file correct so that log events show up in the right Log Group and CloudWatch timestamps match individual log event timestamps.
Creating the Agent Configuration File
The CloudWatch agent gets its log and metric collection configuration from a file in the CloudWatch agent directory.
I have found that the CloudWatch Agent configuration file wizard does a pretty good job of generating the initial configuration and I highly recommend you run it. However, if you are using the CloudWatch agent primarily for log forwarding, you’ll need to edit the config file by hand.
Here’s an example CloudWatch agent logs section from a Couchbase server. For a real Couchbase server, you’d need to monitor most of the logs in the couchbase/logs/ directory.
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/messages",
"log_group_name": "myapp/db/messages",
"timestamp_format": "%b %d %H:%M:%S"
},
{
"file_path": "/opt/couchbase/var/lib/couchbase/logs/http_access.log",
"log_group_name": "myapp/db/couchbase/http_access",
"timestamp_format": "%d/%b/%Y:%H:%M:%S %z"
}
]
}
}
},
There are two potentially tricky settings: the log_group_name and the timestamp_format.
Log Group Name
The Log Group name is a way of grouping logs from related EC2 instances. For example, you might group your app’s web server http_accesslogs together into a single log group called myapp/web/http_access. You should aim for one log group per app/tier/log, e.g. myapp/web/http_access vs myapp/web/messages.
Timestamp_format
The timestamp_format field is listed as optional in the CloudWatch Agent Configuration but it should be considered mandatory. In the documentation it says, “If you omit this field, the current time is used,” but it’s not immediately clear why that’s a problem. If don’t specify a timestamp_format for your log file, when CloudWatch ingests your log, each log line is time-stamped with the time that log line was ingested, not the actual log event time. For example, if the first line in the log file has a timestamp of 2018-06-03, and today is 2018-06-07, that line will be timestamped 2018-06-07 since that’s the date/time the log was ingested into CloudWatch. This makes it very hard to do accurate event correlation across logs, or accurately get alerts on errors in logs when they happen.
When you correctly specify the timestamp_format, the Time column and the timestamp in the Message column will match.
How to specify a timestamp_format
Specifying a timestamp format is pretty straightforward once you’ve done it a couple of times. The important thing to note is that it’s not a regexp. You should copy a timestamp from your log file and replace the year, month, day, time fields with the special symbols listed in the CloudWatch Agent Configuration File: Logs Section, e.g. %Y, %b, etc. See the CloudWatch agent configuration file snippet above for two examples.
Summary
The CloudWatch agent is a good tool to ship logs and metrics from the EC2 instances which make up your application to CloudWatch. This allows you to aggregate logs across instances and app tiers, set up alerts when you reach certain error thresholds (e.g. too many 500 errors in a 5 min period), and create monitoring dashboards for business applications.
I showed you how to create the agent configuration file, including how to specify log group names, and why the timestamp format should be a mandatory field in your agent config file.
I hope this helps you get started with the CloudWatch agent for your EC2-based apps.
Do you need help setting up CloudWatch logs? 1Strategy can help! Please feel free to contact us or shoot us an email at info@1Strategy.com to get more information.
© 2018
