Syslog to S3 via Fluentd

Overview

This guide provides a method to deliver syslog messages to S3 using Fluentd. There are two different pipeline flows: via an AWS Firehose delivery stream and directly to an AWS S3 bucket.

Prerequisites

This guide assumes that an S3 bucket or Firehose has already been created. If you need to create either of these resources, please see the Getting Started with Fluentd guide. If you have already provisioned the resources, you can adapt the guide below to fit your needs.

Setup Fluentd

Step 1. Install Fluentd

Follow the Fluentd install guide for the environment of the server from which you want to collect syslog messages.

Step 2. Edit Fluentd Configuration

You have two options when configuring Fluentd: using the Firehouse plugin or S3 plugin. Below are the configuration files for both options.
We recommend the Firehose plugin option as it is the more performant of the two, however both will deliver the logs to S3. Two different authentication types are shown in the configuration: assume role and access keys. Use the authentication type that best suits your environment.
Install the following Fluentd plugin:
td-agent-gem install fluent-plugin-kinesis
Edit the Fluentd configuration/etc/td-agent/td-agent.conf with the below config. This allows Fluentd to listen for syslog events over udp port 5140 and output to Kinesis Firehose. Update the region, delivery_stream_name and role_arn in the configuration below:
<source>
@type syslog
port 5140
bind 0.0.0.0
tag syslog
<parse>
message_format auto
</parse>
</source>
<filter syslog.** >
@type record_transformer
<record>
tag ${tag}
time ${time}
</record>
</filter>
<match syslog.**>
@type kinesis_firehose
region <FIREHOSE-REGION>
delivery_stream_name <FIREHOSE-NAME>
<assume_role_credentials>
duration_seconds 3600
role_arn <FIREHOSE-ROLE-ARN>
role_session_name "#{Socket.gethostname}-panther-audit"
</assume_role_credentials>
<format>
@type json
</format>
</match>

Via S3 Plugin

Edit the Fluentd configuration /etc/td-agent/td-agent.conf with the below config. This will allow Fluentd to listen for syslog events over udp port 5140 and output to a S3 bucket. Update the s3_bucket, s3_region, aws_key_id, and aws_sec_key in the configuration below:
<source>
@type syslog
port 5140
bind 0.0.0.0
tag syslog
<parse>
message_format auto
</parse>
</source>
<filter syslog.** >
@type record_transformer
<record>
tag ${tag}
time ${time}
</record>
</filter>
<match syslog.**>
@type s3
aws_key_id <ACCESS-KEY-ID>
aws_sec_key <SECRET-KEY>
s3_bucket <BUCKET-NAME>
s3_region <BUCKET-REGION>
path syslog/%Y/%m/%d/
store_as gzip
<buffer tag,time>
@type file
path /var/log/td-agent/buffer/s3
timekey 300 # 5 min partition
timekey_wait 2m
timekey_use_utc true # use utc
chunk_limit_size 256m
</buffer>
<format>
@type json
</format>
</match>

Step 3: Start Fluentd

After configuring Fluentd, start it by running the below command:
$ sudo systemctl start td-agent.service
Verify that Fluentd is running:
$ sudo systemctl status td-agent.service
See the Fluentd install guide on starting Fluentd in your environment if systemctl is not available.

Configure rsyslog

Configure rsyslog to forward to local Fluentd

Configure rsyslog to forward messages to the local Fluentd daemon by adding these two lines to the bottom of /etc/rsyslog.d/50-default.conf or /etc/rsyslog.conf in some environments:
# Send log messages to Fluentd
*.* @127.0.0.1:5140
Restart rsyslog with the below command:
$ sudo systemctl restart rsyslog.service
This step can be duplicated for other servers to forward syslog to the Fluentd server that was previously configured in this guide. Just replace the local address *.* @127.0.0.1:5140 with the IP address of the Fluentd server. You may need to update security groups or host-based firewalls to allow sending udp/5140 traffic to the server.

Verify Logging

After 5-10 minutes have passed, verify that syslog messages are being logged to the S3 bucket. Logs should be showing up under the syslog/ prefix within the bucket.
You can now onboard the data in the Panther UI by onboarding the S3 bucket and using the Fluentd.Syslog3164 log type.
Copy link
On this page
Overview
Prerequisites
Setup Fluentd
Step 1. Install Fluentd
Step 2. Edit Fluentd Configuration
Step 3: Start Fluentd
Configure rsyslog
Configure rsyslog to forward to local Fluentd
Verify Logging