Forward systemd service logs to AWS Cloudwatch

systemd-cloudwatch-logs-tutorial.md

Introduction

I often find myself ssh'ing into my servers and checking my systemd service logs with $ journalctl -f -u {name}.service. One day I got tired of this and wanted all of my important logs in once place (Amazon AWS Cloudwatch). To my dismay, there weren't any real good tutorials on how to do so. So, voilà.

Steps

Overall, it's a fairly simple process consisting of the following few steps.

1. Modify the service file

Open the service file with $ sudo vi /lib/systemd/system/{name}.service

Modify the [Service] section:

[Service]
...
StandardOutput=file:/var/log/{name}/logs.log
StandardError=file:/var/log/{name}/logs.log

Next, create the directory $ sudo mkdir /var/log/{name}

Finally, restart the service:

sudo systemctl daemon-reload
sudo systemctl stop {name}.service
sudo systemctl start {name}.service

Wait a little bit, and confirm logs are being written $ cat /var/log/{name}/logs.log

2. Install the cloudwatch agent

$ mkdir /tmp/cloudwatch-logs && cd /tmp/cloudwatch-logs
$ wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
$ sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

Be sure to download the appropriate agent for your OS

3. Create an IAM role

See, here (copied below for convenience).

1. Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.
2. In the navigation pane on the left, choose Roles and then Create role.
3. For Choose the service that will use this role, choose EC2 Allows EC2 instances to call AWS services on your behalf. Choose Next: Permissions.
4. In the list of policies, select the check box next to CloudWatchAgentServerPolicy. If necessary, use the search box to find the policy.
5. Choose Next: Review.
6. Confirm that CloudWatchAgentServerPolicy appears next to Policies. In Role name, enter a name for the role, such as CloudWatchAgentServerRole. Optionally give it a description. Then choose Create role.

4. Attach the IAM role to your EC2 instance

See, here (copied below for convenience)

1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
2. In the navigation pane, choose Instances.
3. Select the instance, choose Actions, Instance Settings, Attach/Replace IAM role.
4. Select the IAM role to attach to your instance, and choose Apply.

5. Create the log stream in cloudwatch

1. Navigate to https://console.aws.amazon.com/cloudwatch
2. Click Logs from the left menu. Then click Actions > Create log group. Name it /{service}/.
3. Click on the newly greated log group. Then click Create Log Stream. Name it logs.

6. Create a config file and start cloudwatch

$ mkdir ~/cloudwatch && cd ~/cloudwatch
$ vi config.json

Copy/paste the below json being sure to change settings as appropriate:

{
        "agent": {
                "metrics_collection_interval": 60,
                "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"
        },
        "logs": {
                "logs_collected": {
                        "files": {
                                "collect_list": [
                                        {
                                                "file_path": "/var/log/{service}/logs.log",
                                                "log_group_name": "/{service}/",
                                                "log_stream_name": "logs",
                                                "timezone": "UTC"
                                        }
                                ]
                        }
                },
                "log_stream_name": "logs",
                "force_flush_interval" : 60
        }
}

Finally, start the cloudwatch agent $ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/home/ubuntu/cloudwatch/config.json -s

file: output specifier available only from systemd 236 (comment in accepted answer)

So I follow this tutorial to setup my cloud watch:

the file:<filename> is problematic here because systemd will not append your log to your file (source.
replacing it with the append instruction would work, but it is somewhat not available on my Ubuntu 18.04 EC2 instances for reasons.

I mitigate this by rotating the file before restarting the service in my deployment script

(Otherwise: Great doc 👍 )

This is good, but there is some room for improvement:

1. If you have multiple servers to manage, you may be better off installing the agent on your fleet using Amazon Cloudwatch Agent Ansible role: https://galaxy.ansible.com/christiangda/amazon_cloudwatch_agent
2. This HOWTO doesn't cover log rotation. As written, systemd would create a service log of infinite since on your server, eventually filling up your disk and crashing your server. Amazon Cloudwatch Agent doesn't solve this for you. Using logrotate with the copytruncate option is one option to manage the log size so it's doesn't grow forever: https://unix.stackexchange.com/questions/605121/how-to-properly-logrotate-logs-of-service-managed-by-systemd-via-file-config (Normally this isn't a problem for system services, which log to the systemd journal by default. The journal has its own configuration options to manage disk space usage).
3. Demonstrate that multiple logs can be sent to different log groups and streams:

{
"logs_collected": {
           "files": {
               "collect_list": [
                   {
                       "file_path": "/var/log/amazon-cloudwatch-agent.log",
                       "log_group_name": "amazon-cloudwatch-agent.log",
                       "log_stream_name": "my_log_stream_name_1",
                       "timestamp_format": "%H: %M: %S%y%b%-d"
                   },
                   {
                       "file_path": "/var/log/test.log",
                       "log_group_name": "test.log",
                       "log_stream_name": "my_log_stream_name_2"
                   }
               ]
           },
       },
       "log_stream_name": "my_log_stream_name"
}

Couple notes:

1. If you can't find the service under /lib/systemd/system/ also check /etc/systemd/system/.

2. There's no need to modify the (original) service-file, systemd supports extending the configuration through the .d-pattern:

For example:

In /etc/systemd/system/{name}.service.d/override.conf put:

[Service]
StandardOutput=file:/var/log/{name}/logs.log
StandardError=file:/var/log/{name}/logs.log