> For the complete documentation index, see [llms.txt](https://docs.panther.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.panther.com/data-onboarding/panther-log-forwarder.md). # Panther Log Forwarder (Beta) ## Overview {% hint style="info" %} The Panther Log Forwarder is in open beta starting with Panther version 1.121, and is available to all customers. Please share any bug reports and feature requests with your Panther support team. {% endhint %} Panther Log Forwarder is a lightweight tool that collects logs from files or syslog and sends them to Panther. The easiest way to get started is to create a [Log Forwarder source](/data-onboarding/data-transports/log-forwarder.md) in the Panther Console, which provisions a Panther-managed S3 bucket and generates a ready-to-use configuration for you. Alternatively, you can send logs to your own S3 bucket. ## Download Download the latest release for your platform: | Platform | Download Link | Checksum | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | Linux (x86\_64) | [panther-log-forwarder\_linux\_amd64.tar.gz](https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder/latest/panther-log-forwarder_linux_amd64.tar.gz) | [SHA256](https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder/latest/panther-log-forwarder_linux_amd64.tar.gz.sha256) | | Linux (ARM64) | [panther-log-forwarder\_linux\_arm64.tar.gz](https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder/latest/panther-log-forwarder_linux_arm64.tar.gz) | [SHA256](https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder/latest/panther-log-forwarder_linux_arm64.tar.gz.sha256) | To pin to a specific version for reproducible installs or controlled upgrades, replace `latest` in the download URL with a released version tag. The `latest/` path always resolves to the newest release, while versioned paths follow this pattern: ``` https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder//panther-log-forwarder_linux_.tar.gz ``` * `` — a released version tag, e.g., `v0.1.3` (the current latest release) * `` — `amd64` (x86\_64) or `arm64` (ARM64) Each version directory also contains the matching `.tar.gz.sha256` checksum file. For example, to download v0.1.3 for Linux (x86\_64): ```bash wget https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder/v0.1.3/panther-log-forwarder_linux_amd64.tar.gz wget https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder/v0.1.3/panther-log-forwarder_linux_amd64.tar.gz.sha256 ``` ## Get Started {% hint style="success" %} **Recommended:** Create a [Log Forwarder source](/data-onboarding/data-transports/log-forwarder.md) in the Panther Console. It automatically provisions a Panther-managed S3 bucket to send logs to and generates a ready-to-use YAML configuration for your PLF agent, no additional infrastructure required. {% endhint %} Follow these steps to install the log forwarder and start sending logs in minutes. This example sends logs from a file to a Panther-managed S3 bucket using a Log Forwarder source. To send logs to your own S3 bucket, or for more configuration options, see below. ### Prerequisites * Linux with systemd (Ubuntu 20.04+, Debian 11+, RHEL 8+, Rocky Linux 8+) * Sudo/root access * A [Log Forwarder source](/data-onboarding/data-transports/log-forwarder.md) created in the Panther Console — this provisions the S3 bucket and generates the YAML configuration you'll use below. (To send logs to your own S3 bucket instead, see [S3 Output](#s3-output).) ### Step 1: Install ```bash # Download (x86_64 — for ARM64, replace "amd64" with "arm64") wget https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder/latest/panther-log-forwarder_linux_amd64.tar.gz # Extract and install tar -xzf panther-log-forwarder_linux_amd64.tar.gz cd panther-log-forwarder sudo ./install.sh ``` {% hint style="info" %} **Optional: Verify checksum before installing** ```bash wget https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder/latest/panther-log-forwarder_linux_amd64.tar.gz.sha256 sha256sum -c panther-log-forwarder_linux_amd64.tar.gz.sha256 ``` {% endhint %} ### Step 2: Configure a Forwarder Save the YAML configuration generated by your [Log Forwarder source](/data-onboarding/data-transports/log-forwarder.md) to a file in `/etc/panther-log-forwarder/forwarders.d/`. It already contains the S3 `bucket`, `region`, and credentials for your Panther-managed bucket, so no edits are needed. The generated configuration looks like this: ```yaml forwarders: my_app_logs: input: type: file path: /var/log/testapp output: type: s3 bucket: YOUR_PANTHER_BUCKET region: YOUR_REGION auth: access_key_id: YOUR_ACCESS_KEY_ID secret_access_key: YOUR_SECRET_ACCESS_KEY ``` {% hint style="warning" %} The generated configuration file contains AWS credentials, so store it securely. {% endhint %} ### Step 3: Validate and Start ```bash # Validate your configuration sudo panther-log-forwarder validate # Test connectivity to Panther and existence of S3 bucket sudo panther-log-forwarder check-connectivity # Start the service sudo systemctl start panther-log-forwarder sudo systemctl enable panther-log-forwarder # Verify it's running sudo panther-log-forwarder status ``` {% hint style="success" %} Logs should appear in Panther's Data Explorer within a few minutes. {% endhint %} *** ## Commands | Command | Description | | ----------------------------------------------------------- | --------------------------------------------- | | `panther-log-forwarder validate` | Validate configuration syntax | | `panther-log-forwarder check-connectivity` | Test connectivity to all destinations | | `panther-log-forwarder check-connectivity --forwarder NAME` | Test a specific forwarder | | `panther-log-forwarder start` | Run in foreground (for debugging) | | `panther-log-forwarder status` | Show health, metrics, and per-forwarder stats | | `panther-log-forwarder --version` | Show version | For production, use systemd: `sudo systemctl start panther-log-forwarder` *** ## Configuration Reference The log forwarder uses two types of configuration files: 1. **Global settings** in `/etc/panther-log-forwarder/panther.conf` — created automatically during installation with sensible defaults 2. **Forwarder definitions** in `/etc/panther-log-forwarder/forwarders.d/*.yaml` — you create these to define what logs to collect and where to send them ### Global Configuration The installer creates `/etc/panther-log-forwarder/panther.conf` with default values. Edit it only if you need to change defaults: ```yaml # /etc/panther-log-forwarder/panther.conf default_source_buffer_size: 3GB # Disk buffer per forwarder (default: 3GB) log_level: info # debug, info, warn, error (default: info) log_format: text # text or json (default: text) ``` ### Forwarder Configuration Each forwarder is added with a unique name in under `forwarders` and has an `input` (what to collect) and `output` (where to send) section. Forwarder names must start with a letter, contain only alphanumeric characters, underscores, or hyphens, and be 1-64 characters long. You can organize forwarders across multiple files or put them all in one, any `.yaml` file in `forwarders.d/` is loaded. ### Input Types #### File Input Tail log files using glob patterns: ```yaml input: type: file path: /var/log/app/*.log # Supports globs: *.log, **/*.log excludes: ["*.gz", "*.zip"] # Optional: patterns to skip read_from: end # Optional: "end" (default) or "beginning" ``` | Field | Required | Default | Description | | ----------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------ | | `type` | Yes | - | Must be `file` | | `path` | Yes | - | File path or glob pattern | | `excludes` | No | - | Array of glob patterns to exclude | | `read_from` | No | `end` | `end` = new lines only; `beginning` = read existing content | | `format` | No | - | Content format hint. Set to `syslog` when tailing syslog-formatted files (use with `parse: json`, see below) | | `parse` | No | - | Set to `json` to parse syslog lines into structured JSON (requires `format: syslog`) | {% hint style="warning" %} Using `read_from: beginning` will send all existing log lines in the file. For large files, this may take considerable time and disk buffer space. {% endhint %} **Notes:** * Only newline-delimited logs are supported (one log entry per line) * Multi-line logs are not currently supported * On startup, the log forwarder validates that file paths are accessible and reports permission errors before starting #### Syslog Input Receive syslog messages over the network: ```yaml input: type: syslog mode: tcp # "tcp" or "udp" address: 0.0.0.0:1514 # Listen address ``` | Field | Required | Description | | --------- | -------- | ----------------------------------------------------------- | | `type` | Yes | Must be `syslog` | | `mode` | Yes | `tcp` or `udp` | | `address` | Yes | Host and port to listen on | | `parse` | No | Set to `json` to parse syslog messages into structured JSON | **Notes:** * Messages will be sent in raw format to Panther. If your messages adhere to RFC 3164 or RFC 5424 format, you can use the built in corresponding syslog schema in Panther ([`Syslog.RFC5424`](https://docs.panther.com/data-onboarding/pages/-MXJ6kVt9vlEsqLUXPBi#syslog.rfc5424)/[`Syslog.RFC3164`](https://docs.panther.com/data-onboarding/pages/-MXJ6kVt9vlEsqLUXPBi#syslog.rfc3164)) for proper parsing and classification. * Use `parse: json` to send the message parsed as json to your Panther source. Parsing is best effort and covers most common syslog formats, additionally to RFC compliant ones. ### S3 Output The log forwarder sends logs to an Amazon S3 bucket. When you create a [Log Forwarder source](/data-onboarding/data-transports/log-forwarder.md) in the Panther Console, Panther provisions the bucket and generates this configuration for you. To send logs to your own S3 bucket, use either of two authentication paths: 1. **Inline static credentials** - set `auth.access_key_id` and `auth.secret_access_key`. This is what a Log Forwarder source generates for you, and it's also useful when the forwarder runs outside AWS or you prefer keeping credentials in the `yaml` file rather than providing them to the systemd unit. 2. **AWS default credential chain** (on EC2 / ECS / EKS) - omit the `auth` block. The AWS SDK resolves credentials from the instance profile / task role or environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) automatically. File-based credentials (`~/.aws/credentials`) are not supported by default, unless you explicitly grant access to the `panther-log-forwarder` user to read those files. ```yaml output: type: s3 bucket: my-log-bucket region: us-east-1 # key_prefix: "custom/prefix/%Y/%m/%d/" # Optional auth: # Optional — omit to use the AWS default credential chain access_key_id: ${AWS_ACCESS_KEY_ID} secret_access_key: ${AWS_SECRET_ACCESS_KEY} ``` | Field | Required | Default | Description | | ------------- | -------- | --------------------------------- | ----------------------------------------------------------------------------------- | | `type` | Yes | - | Must be `s3` | | `bucket` | Yes | - | S3 bucket name | | `region` | Yes | - | AWS region (e.g., `us-east-1`) | | `key_prefix` | No | `logs/{forwarder_name}/%Y/%m/%d/` | S3 key prefix (supports strftime tokens, must end with `/`) | | `buffer_size` | No | `3GB` | Per-forwarder disk buffer size override | | `auth` | No | - | Inline static AWS credentials (see [Credential Management](#credential-management)) | **Minimum IAM permissions:** {% hint style="info" %} This applies only when sending to your own S3 bucket. When you use a Log Forwarder source, Panther provisions the bucket and configures these permissions for you. {% endhint %} The IAM identity used must have `s3:PutObject` and `s3:ListBucket` permissions on the target bucket. If the bucket uses **SSE-KMS** with a customer-managed key, also grant `kms:GenerateDataKey` on that key ARN. Here's how an example policy looks like ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::your-bucket-name" }, { "Effect": "Allow", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::your-bucket-name/*" } // optional, only if the bucket uses **SSE-KMS** { "Effect": "Allow", "Action": "kms:GenerateDataKey", "Resource": "arn:aws:kms:{region}:{account_id}:key/{id}" } ] } ``` **Notes:** * Logs are gzip-compressed before upload * Batching: 10MB or 60 seconds (whichever comes first) ### Credential Management #### S3 Output - AWS Credentials Forwarder files are secured with `0640` permissions and `root:panther-log-forwarder` ownership - only root can write, only the agent service can read. S3 output defaults to the AWS credential chain (instance profile, environment variables) - no `auth` block needed. To supply explicit credentials, add an `auth` block: | Field | Required (if `auth` set) | Description | | ------------------- | ------------------------ | --------------------- | | `access_key_id` | Yes | AWS access key ID | | `secret_access_key` | Yes | AWS secret access key | Both fields accept either plaintext values or `${ENV_VAR}` references that are resolved when the agent starts. ```yaml output: type: s3 bucket: my-log-bucket region: us-east-1 auth: access_key_id: ${AWS_ACCESS_KEY_ID} secret_access_key: ${AWS_SECRET_ACCESS_KEY} ``` For env vars, provide the values via the systemd unit (`Environment=` inline or, `EnvironmentFile=` pointing at a `chmod 600` file) ### Disk Buffering Each forwarder has its own disk buffer (default: 3GB) to prevent data loss during network outages. Logs are stored on disk until they can be forwarded. When the buffer fills: * **File inputs** - the forwarder blocks and stops reading until buffer space frees up (no data loss, but log files may grow) * **Syslog inputs** - the forwarder drops newest incoming messages (prevents memory issues, but some logs may be lost) Configure buffer size globally in `panther.conf` or per-forwarder with `buffer_size` in the output section. The default 3GB is sufficient for most deployments. Consider increasing the buffer size for high-volume sources to ensure no data loss. ### Syslog Parsing By default, syslog messages are forwarded as raw text. Adding `parse: json` to the input parses syslog messages into structured JSON with fields like `timestamp`, `hostname`, `appname`, `severity`, and `message`. This works with both input types: * **Network syslog** (`type: syslog` + `parse: json`) — uses built-in syslog parsing * **File syslog** (`type: file` + `format: syslog` + `parse: json`) — parses each line as a syslog message #### Network syslog with parsing ```yaml forwarders: parsed_syslog: input: type: syslog mode: tcp address: 0.0.0.0:1514 parse: json output: type: s3 bucket: my-log-bucket region: us-east-1 ``` #### File syslog with parsing Use `format: syslog` and `parse: json` on the input to indicate the file contains syslog-formatted lines that should be parsed: ```yaml forwarders: os_syslog: input: type: file path: /var/log/syslog format: syslog parse: json output: type: s3 bucket: my-log-bucket region: us-east-1 ``` {% hint style="info" %} `parse: json` requires a syslog context, either `input.type: syslog` or `input.format: syslog`. It cannot be used with plain file inputs. {% endhint %} {% hint style="warning" %} When `parse: json` is enabled, parsing is best-effort, however it covers most common syslog formats including RFC 3164 / RFC 5424. Lines that fail to parse are dropped. {% endhint %} *** ## More Examples ### Syslog Server ```yaml # /etc/panther-log-forwarder/forwarders.d/syslog.yaml forwarders: my_syslog: input: type: syslog mode: tcp address: 0.0.0.0:1514 output: type: s3 bucket: my-log-bucket region: us-east-1 ``` ### Multiple Files with Exclusions ```yaml # /etc/panther-log-forwarder/forwarders.d/app.yaml forwarders: app_logs: input: type: file path: /var/log/myapp/*.log excludes: ["*.gz", "*.old"] output: type: s3 bucket: my-log-bucket region: us-east-1 ``` ### Multiple Forwarders in One File ```yaml # /etc/panther-log-forwarder/forwarders.d/all-logs.yaml forwarders: auth_logs: input: type: file path: /var/log/auth.log output: type: s3 bucket: my-log-bucket region: us-east-1 syslog_server: input: type: syslog mode: tcp address: 0.0.0.0:1514 output: type: s3 bucket: my-log-bucket region: us-east-1 ``` ### Syslog to S3 with JSON Parsing ```yaml # /etc/panther-log-forwarder/forwarders.d/syslog-s3.yaml forwarders: syslog_to_s3: input: type: syslog mode: udp address: 0.0.0.0:514 parse: json output: type: s3 bucket: my-log-bucket region: us-east-1 ``` ### S3 Output Forwarder running in AWS environment, or with env vars provided to the systemd unit (no auth block needed): ```yaml # /etc/panther-log-forwarder/forwarders.d/s3.yaml forwarders: s3_archive: input: type: file path: /var/log/app/*.log output: type: s3 bucket: my-log-bucket region: us-east-1 ``` ### S3 Output with Inline AWS Credentials Forwarder running outside AWS, with explicit credentials defined: ```yaml # /etc/panther-log-forwarder/forwarders.d/s3-inline.yaml forwarders: s3_archive: input: type: file path: /var/log/app/*.log output: type: s3 bucket: my-log-bucket region: us-east-1 auth: access_key_id: my_access_key_id secret_access_key: my_secret_access_key ``` *** ## Troubleshooting ### Checking Logs There are two log files to check when troubleshooting: * **`/var/log/panther-log-forwarder/log-forwarder.log`** - Check this for issues with the forwarder service itself, such as startup failures, configuration errors, and connectivity problems. * **`/var/log/panther-log-forwarder/vector.log`** - Check this for issues with the data pipeline, such as log ingestion problems, parsing errors, and delivery failures. ```bash # Forwarder service logs sudo tail -f /var/log/panther-log-forwarder/log-forwarder.log # Data pipeline logs sudo tail -f /var/log/panther-log-forwarder/vector.log ``` ### Log Forwarder Won't Start ```bash sudo panther-log-forwarder status ``` Common causes: * **Configuration errors** - Run `sudo panther-log-forwarder validate` * **Permission issues** - Verify `/etc/panther-log-forwarder/forwarders.d/` has `0750` permissions * **File access errors** - The log forwarder checks file paths on startup. Verify paths exist and are readable by the `panther-log-forwarder` user * **Port conflicts** - For syslog inputs, ensure the listening port is available ### No Logs Appearing in Panther ```bash sudo panther-log-forwarder check-connectivity sudo panther-log-forwarder status ``` Common causes: * **Invalid credentials** - Verify the AWS credentials in your configuration are valid. If you're using a Log Forwarder source, confirm the credentials match the generated ones from the Panther Console. * **Incorrect bucket or region** - Double-check the `bucket` and `region` values in your configuration. * **Network issues** - Test connectivity from the host to S3 * **File path issues** - Verify paths exist and are readable * **S3 permissions** - For S3 output, verify IAM permissions allow `s3:PutObject` and `s3:ListBucket` (used by HeadBucket). Add `kms:GenerateDataKey` if the bucket uses SSE-KMS with a customer-managed key. ### Configuration Errors ```bash sudo panther-log-forwarder validate ``` Common issues: * YAML syntax errors (indentation, quotes) * Missing required fields (`type` in input; `bucket` and `region`) * Invalid input type (must be `file` or `syslog`) * Invalid output type (must be `s3`) * Invalid forwarder name (must start with a letter, alphanumeric/underscore/hyphen only) *** ## Operations ### Adding New Forwarders 1. Create a new [Log Forwarder source](/data-onboarding/data-transports/log-forwarder.md) in the Panther Console (or configure an S3 output to your own bucket). 2. Add a new YAML file in `/etc/panther-log-forwarder/forwarders.d/`. 3. Validate: `sudo panther-log-forwarder validate` 4. Restart: `sudo systemctl restart panther-log-forwarder` ### Updating Configuration 1. Edit files in `/etc/panther-log-forwarder/`. 2. Validate: `sudo panther-log-forwarder validate` 3. Restart: `sudo systemctl restart panther-log-forwarder` ### Upgrading 1. Download the new release tarball and checksum for your platform (see [Download](#download)). 2. Verify the checksum, extract, and run the installer: ```bash sha256sum -c panther-log-forwarder_linux_amd64.tar.gz.sha256 tar -xzf panther-log-forwarder_linux_amd64.tar.gz cd panther-log-forwarder sudo ./install.sh ``` The installer automatically restarts the service if it was running. Verify the upgrade: ```bash sudo panther-log-forwarder --version sudo panther-log-forwarder status ``` Configuration files in `/etc/panther-log-forwarder/` are preserved during upgrades. ### Uninstalling ```bash sudo systemctl stop panther-log-forwarder cd panther-log-forwarder # unpacked installation directory sudo ./uninstall.sh ``` *** ## File Locations | Path | Purpose | | -------------------------------------------------- | ------------------ | | `/etc/panther-log-forwarder/panther.conf` | Global settings | | `/etc/panther-log-forwarder/forwarders.d/*.yaml` | Forwarder configs | | `/var/log/panther-log-forwarder/log-forwarder.log` | Log forwarder logs | | `/var/log/panther-log-forwarder/vector.log` | Vector logs | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.panther.com/data-onboarding/panther-log-forwarder.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.