MacOS System Logs to S3 via Fluentd
Consider using Fluent Bit instead of Fluentd to forward logs to Panther. Fluent Bit is easier to set up and less resource intensive than Fluentd.
This guide provides a method to deliver MacOS System Logs to S3 using Fluentd. There are two different pipeline flows: via an AWS Firehose delivery stream and directly to an AWS S3 bucket
This guide assumes that an S3 bucket or Firehose has already been created. If you need to create either of these resources, please see the Getting Started with Fluentd guide. If you have already provisioned the resources, you can adapt the guide below to fit your needs.
Follow the Fluentd installation instructions for the machine from which you want to collect MacOS System Logs. This guide will specifically cover using td-agent as the service to collect logs.
Use the command below to install the Fluentd MacOS plugin.
sudo /opt/td-agent/bin/fluent-gem install fluent-plugin-macos-log
The configuration information that is included by default can be removed if not in use. Use the Fluentd configuration below and add your
aws_key_id
, aws_sec_key
, s3_bucket
, and s3_region
information.Fluentd and td-agent will attempt to run services on conflicting ports. If this is a new installation you will need to change the ports in the configuration file or remove the default configuration from the file.
/etc/td-agent/td-agent.conf
<source>
@type macoslog
style ndjson
tag macos
pos_file last-starttime.log
run_interval 10s
<parse>
@type json
time_type string
time_key timestamp
time_format %Y-%m-%d %H:%M:%S.%L%z
</parse>
</source>
​
<match **>
@type s3
aws_key_id <Key ID>
aws_sec_key <Key>
s3_bucket <Bucket>
s3_region <Region>
path macoslog/%Y/%m/%d/
store_as gzip
<buffer tag,time>
@type file
path /var/log/fluent/s3
timekey 300 # 5 min partition to post to S3
timekey_wait 2m
timekey_use_utc true # use utc
chunk_limit_size 256m
</buffer>
<format>
@type json
</format>
</match>
# Point fluentd to configuration file
fluentd -c /etc/td-agent/td-agent.conf
​
# Validate configuration
/opt/td-agent/usr/sbin/td-agent --dry-run
After a few minutes have passed, verify that events are being logged to the S3 bucket. Logs should be showing up under the
macos/
prefix within the bucket.Go to Configure > Schemas. Click +New Schema then enter the values below into the schema fields:
Name: Custom.MacOSSystemLogs
Description: MacOS System Logs for Application, Security, System
version: 0
fields:
- name: pid
type: bigint
- name: ppid
type: bigint
- name: message
type: string
- name: worker
type: bigint
- name: creatorActivityID
type: float
- name: messageType
type: string
- name: activityIdentifier
type: bigint
- name: backtrace
type: object
fields:
- name: frames
required: true
type: array
element:
type: object
fields:
- name: imageOffset
required: true
type: bigint
- name: imageUUID
required: true
type: string
- name: bootUUID
type: string
- name: category
type: string
- name: eventMessage
type: string
- name: eventType
type: string
- name: formatString
type: string
- name: machTimestamp
type: bigint
- name: parentActivityIdentifier
type: bigint
- name: processID
type: bigint
- name: processImagePath
type: string
- name: processImageUUID
type: string
- name: senderImagePath
type: string
- name: senderImageUUID
type: string
- name: senderProgramCounter
type: bigint
- name: subsystem
type: string
- name: threadID
type: bigint
- name: timezoneName
type: string
- name: traceID
type: float
Select the log type
Custom.MacOSSystemLogs
and prefix macos/
in the onboarding steps. After completing the bucket onboarding, data should now be flowing into Panther!​
Last modified 2mo ago