How to: Set-up and configure AWS CloudWatch Logs

A lesser known feature of the Amazon Web Services platform is the ability to stream server logs into shared storage in CloudWatch.

Storing server logs in CloudWatch is useful as it provides a means to retain logs long after an instance has been and gone. Furthermore, it allows us to configure log metrics, which can trigger alarms based on the volume of certain HTTP status codes or other log events within a fixed period of time. So for example an email notification can be triggered if a certain number of 404 or 500 HTTP status codes occur within a set period of time.

The AWS documentation covers the simple steps to install the AWS Logs Agent on your server instances. Setting up each log file to be streamed to CloudWatch Logs is very simple and at the end of the Agent installation you can configure one or more log files to stream.

As a starting point we often stream the following log files:

/var/log/syslog
/var/log/auth.log

In addition, we have our Nginx and PHP error logs streaming and it's even possible to configure Varnish logs to be streamed too.

The only part that needs a little thought is configuring the timestamp format, so here are a few helpers, it does depend on your log configuration of course but here's what worked for us:

Nginx log timestamp format:

Example timestamp: 2015/01/11 12:53:19
Timestamp format: %Y/%m/%d %H:%M:%S

System / auth log timestamp format:

Example timestamp: Jan 23 20:59:29
Timestamp format: %b %d %H:%M:%S

PHP error log timestamp format:

Example timestamp: 27-Jan-2015 10:00:32 UTC
Timestamp format: %d-%b-Y %H:%M:%S

As the above also includes the time zone, this requires an added log configuration parameter to be included in the AWS Logs config file (which is usually found here: /var/awslogs/etc/awslogs.conf)

time_zone = UTC

It's also easy to add further logs after the initial agent installation "wizard" has finished, simply by adding another entry to the bottom of the configuration file, for example:

[/logs/error.log]
datetime_format = %Y/%m/%d %H:%M:%S
file = /logs/error.log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = end_of_file
log_group_name = /logs/error.log

After the agent installation AWS Logs does display some helper notes:

Configuration file successfully saved at: /var/awslogs/etc/awslogs.conf
You can begin accessing new log events after a few moments at...
You can use 'sudo service awslogs start|stop|status|restart' to control the daemon.
To see diagnostic information for the CloudWatch Logs Agent, see /var/log/awslogs.log
You can rerun interactive setup using 'sudo ./awslogs-agent-setup.py --region eu-west-1 --only-generate-config'

Tagged with:

Why Hosting Matters

by Dan Walsh