# Fluent Bit Configuration Examples

The examples on this page provide common methods to receive data with Fluent Bit and send logs to Panther via an [HTTP Source](/~/changes/2402/data-onboarding/data-transports/http.md) or via an [Amazon S3 Source](/~/changes/2402/data-onboarding/data-transports/aws/s3.md). 

{% hint style="warning" %}
In the examples below, `log_level trace` and output `stdout` are used to test and debug the configurations. These should be removed once the Fluent Bit configuration is working as expected. 
{% endhint %}

## Dummy to a Panther HTTP source

This example uses [Fluent Bit's Dummy input](https://docs.fluentbit.io/manual/pipeline/inputs/dummy) to spawn one event per second. This is useful for testing output configurations and getting started with Fluent Bit. 

Input: [Dummy](https://docs.fluentbit.io/manual/pipeline/inputs/dummy) 

Output: [HTTP](https://docs.fluentbit.io/manual/pipeline/inputs/http)

```yaml
[SERVICE]
    log_level trace

[INPUT]
    Name dummy
    Dummy {"message": "sample json message", "type": "json"}

[OUTPUT]
    Name       http
    Match      *
    Host       logs.{REDACTED}.runpanther.net
    Port       443
    URI        /http/{REDACTED}
    Header     x-sender-header {REDACTED}
    Format     json_lines
    TLS        On
    TLS.Verify On
    Json_Date_Key false

[OUTPUT]
    Name   stdout
    Match  *
```

This configuration results in the following:

```
# Input configuration:
Dummy {"message": "sample json message", "type": "json"}
    
# Ouput of raw event pre Panther parsing:
{"message": "sample json message", "type": "json"}
```

## Tail local file to Amazon S3

This example uses the Tail input to ingest a file locally sent to S3. Multiple files can be provided. See the `path` setting in the [Fluent Bit Tail documentation](https://docs.fluentbit.io/manual/pipeline/inputs/tail) for more information. 

Input: [Tail](https://docs.fluentbit.io/manual/pipeline/inputs/tail)

Output: [S3](https://docs.fluentbit.io/manual/pipeline/outputs/s3)

{% hint style="info" %}
In the OUTPUT plugin configuration:

* Use`json_date_key false` to disable the appended date key. 
* Use  `log_key log` to specify Fluent Bit to only send the raw log. 

With these two settings, the raw input from the log file is sent without Fluent Bit's appended JSON fields.
{% endhint %}

```yaml
[SERVICE]
    log_level trace

[INPUT]
    Name       tail
    Tag        wifi_log
    Path       /var/log/wifi.log

[OUTPUT]
    Name       s3
    Match      *
    Region     {REGION}
    Bucket     {BUCKET_NAME}
    Compression gzip
    json_date_key false
    upload_timeout 5m
    log_key log

    # Retrieving AWS Creds - https://github.com/fluent/fluent-bit-docs/blob/43c4fe134611da471e706b0edb2f9acd7cdfdbc3/administration/aws-credentials.md

[OUTPUT]
    Name   stdout
    Match  *
```

This configuration results in the following:

```
# Input result from tailing file:
Mon Feb  5 16:17:04.165 Usb Host Notification hostNotificationUSBDeviceInserted USB Billboard Device    isApple N seqNum 454 Total 4
Mon Feb  5 16:17:04.176 Usb Host Notification Apple80211Set: seqNum 454 Total 4 chg 1 en0
Mon Feb  5 16:17:28.841 Usb Host Notification hostNotificationUSBDeviceInserted USB MICROPHONE isApple N seqNum 455 Total 5
Mon Feb  5 16:17:28.846 Usb Host Notification Apple80211Set: seqNum 455 Total 5 chg 1 en0
    
# Output Result in AWS S3:
Mon Feb  5 16:17:04.165 Usb Host Notification hostNotificationUSBDeviceInserted USB Billboard Device    isApple N seqNum 454 Total 4
Mon Feb  5 16:17:04.176 Usb Host Notification Apple80211Set: seqNum 454 Total 4 chg 1 en0
Mon Feb  5 16:17:28.841 Usb Host Notification hostNotificationUSBDeviceInserted USB MICROPHONE isApple N seqNum 455 Total 5
Mon Feb  5 16:17:28.846 Usb Host Notification Apple80211Set: seqNum 455 Total 5 chg 1 en0
```

## TCP to Amazon S3

This example uses the TCP input plugin. This plugin is useful if you need to ship syslog or JSON events to Fluent Bit over the network. The TCP plugin takes the raw payload it receives and forwards it to the Output configuration.

Input: [TCP](https://docs.fluentbit.io/manual/pipeline/inputs/tcp)

Output: [S3](https://docs.fluentbit.io/manual/pipeline/outputs/s3)

```yaml
[SERVICE]
    log_level trace

[INPUT]
    Name       tcp
    Tag        tcp_log
    Listen     0.0.0.0
    Port       5140
    Format     none

[OUTPUT]
    Name       s3
    Match      *
    Region     {REGION}
    Bucket     {BUCKET_NAME}
    Compression gzip
    json_date_key false
    upload_timeout 5m
    log_key log

    # Retrieving AWS Creds - https://github.com/fluent/fluent-bit-docs/blob/43c4fe134611da471e706b0edb2f9acd7cdfdbc3/administration/aws-credentials.md

[OUTPUT]
    Name   stdout
    Match  *
```

This configuration results in the following:

```
# Input command:
%echo "message from local echo" | nc 127.0.0.1 5140
%echo "message from local echo" | nc 127.0.0.1 5140

# Output in AWS S3 with prefix tcp_log/2024/02/06/02/55/generated_filename:
message from local echo
message from local echo
```

## TCP to HTTP (Panther)

This example configuration demonstrates receiving logs using the TCP input plugin and sending directly to Panther's [HTTP ingest](/~/changes/2402/data-onboarding/data-transports/http.md) using Fluent Bit's HTTP output plugin. 

Input: [TCP](https://docs.fluentbit.io/manual/pipeline/inputs/tcp)

Output: [HTTP](https://docs.fluentbit.io/manual/pipeline/outputs/http)

{% hint style="info" %}
The use of filters in the configuration below is required in order to keep raw payload as-is when sending the log to the HTTP destination. See the [Fluent Bit HTTP output documentation](https://docs.fluentbit.io/manual/pipeline/outputs/http) for more information. 
{% endhint %}

```yaml
[SERVICE]
    log_level trace

[INPUT]
    Name       tcp
    Tag        tcp_log
    Listen     0.0.0.0
    Port       5140
    Format     none

# https://stackoverflow.com/questions/75291515/how-to-disable-json-format-and-send-only-the-log-message-to-sumologic-with-fluen
[FILTER]
    Name record_modifier
    Match *
    Record headers.content-type text/plain

# https://stackoverflow.com/questions/75291515/how-to-disable-json-format-and-send-only-the-log-message-to-sumologic-with-fluen
[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard headers.*
    Nest_under headers
    Remove_prefix headers.


[OUTPUT]
    Name       http
    Match      *
    Host       logs.{REDACTED}.runpanther.net
    Port       443
    URI        /http/{REDACTED}
    Header     x-sender-header {REDACTED}
    Format     json_lines
    TLS        On
    TLS.Verify On
    Json_Date_Key false
    body_key   $log
    headers_key $headers

[OUTPUT]
    Name   stdout
    Match  *
```

This configuration results in the following:

```
# Input command: 
% echo "message from local echo `date`" | nc 127.0.0.1 5140
% echo "message from local echo `date`" | nc 127.0.0.1 5140
% echo "message from local echo `date`" | nc 127.0.0.1 5140

# Ouput of raw event pre Panther parsing:
message from local echo Mon Feb 5 19:27:40 PST 2024
message from local echo Mon Feb 5 19:27:52 PST 2024
message from local echo Mon Feb 5 19:27:53 PST 2024
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.panther.com/~/changes/2402/data-onboarding/data-pipeline-tools/fluent-bit-onboarding-guide/fluent-bit-configuration-examples.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.