# Panther Log Forwarder Onboarding Guide (Beta)

## Overview

{% hint style="info" %}
The Panther Log Forwarder is in open beta starting with Panther version 1.121, and is available to all customers. Please share any bug reports and feature requests with your Panther support team.
{% endhint %}

Panther Log Forwarder is a lightweight tool for forwarding logs from on-premises infrastructure to Panther. It collects logs from files or syslog and sends them to Panther through an HTTP source or S3 bucket.

## Download Links

Download the latest release for your platform:

| Platform        | Download Link                                                                                                                                                              | Checksum                                                                                                                                      |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| Linux (x86\_64) | [panther-log-forwarder\_linux\_amd64.tar.gz](https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder/latest/panther-log-forwarder_linux_amd64.tar.gz) | [SHA256](https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder/latest/panther-log-forwarder_linux_amd64.tar.gz.sha256) |
| Linux (ARM64)   | [panther-log-forwarder\_linux\_arm64.tar.gz](https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder/latest/panther-log-forwarder_linux_arm64.tar.gz) | [SHA256](https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder/latest/panther-log-forwarder_linux_arm64.tar.gz.sha256) |

## Get Started

Follow these steps to install the log forwarder and start sending logs in minutes. This is an example of sending logs from a file to an HTTP source in Panther with default configuration. For sending logs to an S3 bucket and more configuration options, see below.

### Prerequisites

* Linux with systemd (Ubuntu 20.04+, Debian 11+, RHEL 8+, Rocky Linux 8+)
* Sudo/root access
* An HTTP Source URL and Bearer Token from Panther Console (see [HTTP transport documentation](https://docs.panther.com/data-onboarding/data-transports/http))

### Step 1: Install

```bash
# Download (x86_64 — for ARM64, replace "amd64" with "arm64")
wget https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder/latest/panther-log-forwarder_linux_amd64.tar.gz

# Extract and install
tar -xzf panther-log-forwarder_linux_amd64.tar.gz
cd panther-log-forwarder
sudo ./install.sh
```

{% hint style="info" %}
**Optional: Verify checksum before installing**

```bash
wget https://panther-community-us-east-1.s3.us-east-1.amazonaws.com/log-forwarder/latest/panther-log-forwarder_linux_amd64.tar.gz.sha256
sha256sum -c panther-log-forwarder_linux_amd64.tar.gz.sha256
```

{% endhint %}

### Step 2: Configure a Forwarder

Create a YAML file in `/etc/panther-log-forwarder/forwarders.d/` and paste the following content. Replace the `url` and `token` values with the ones from your HTTP log source.

```yaml
forwarders:
  my_app_logs:
    input:
      type: file
      path: /var/log/testapp
    output:
      type: http
      url: https://logs.your-instance.runpanther.net/http/YOUR_SOURCE_ID
      auth:
        type: bearer
        token: YOUR_BEARER_TOKEN
```

### Step 3: Validate and Start

```bash
# Validate your configuration
sudo panther-log-forwarder validate

# Test connectivity to Panther
sudo panther-log-forwarder check-connectivity

# Start the service
sudo systemctl start panther-log-forwarder
sudo systemctl enable panther-log-forwarder

# Verify it's running
sudo panther-log-forwarder status
```

{% hint style="success" %}
Logs should appear in Panther's Data Explorer within a few minutes.
{% endhint %}

***

## Commands

| Command                                                     | Description                                   |
| ----------------------------------------------------------- | --------------------------------------------- |
| `panther-log-forwarder validate`                            | Validate configuration syntax                 |
| `panther-log-forwarder check-connectivity`                  | Test connectivity to all destinations         |
| `panther-log-forwarder check-connectivity --forwarder NAME` | Test a specific forwarder                     |
| `panther-log-forwarder start`                               | Run in foreground (for debugging)             |
| `panther-log-forwarder status`                              | Show health, metrics, and per-forwarder stats |
| `panther-log-forwarder --version`                           | Show version                                  |

For production, use systemd: `sudo systemctl start panther-log-forwarder`

***

## Configuration Reference

The log forwarder uses two types of configuration files:

1. **Global settings** in `/etc/panther-log-forwarder/panther.conf` — created automatically during installation with sensible defaults
2. **Forwarder definitions** in `/etc/panther-log-forwarder/forwarders.d/*.yaml` — you create these to define what logs to collect and where to send them

### Global Configuration

The installer creates `/etc/panther-log-forwarder/panther.conf` with default values. Edit it only if you need to change defaults:

```yaml
# /etc/panther-log-forwarder/panther.conf
default_source_buffer_size: 3GB   # Disk buffer per forwarder (default: 3GB)
log_level: info                   # debug, info, warn, error (default: info)
log_format: text                  # text or json (default: text)
```

### Forwarder Configuration

Each forwarder is added with a unique name in under `forwarders` and has an `input` (what to collect) and `output` (where to send) section.

Forwarder names must start with a letter, contain only alphanumeric characters, underscores, or hyphens, and be 1-64 characters long.

You can organize forwarders across multiple files or put them all in one — any `.yaml` file in `forwarders.d/` is loaded.

### Input Types

#### File Input

Tail log files using glob patterns:

```yaml
input:
  type: file
  path: /var/log/app/*.log       # Supports globs: *.log, **/*.log
  excludes: ["*.gz", "*.zip"]    # Optional: patterns to skip
  read_from: end                 # Optional: "end" (default) or "beginning"
```

| Field       | Required | Default | Description                                                                                                  |
| ----------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------ |
| `type`      | Yes      | —       | Must be `file`                                                                                               |
| `path`      | Yes      | —       | File path or glob pattern                                                                                    |
| `excludes`  | No       | —       | Array of glob patterns to exclude                                                                            |
| `read_from` | No       | `end`   | `end` = new lines only; `beginning` = read existing content                                                  |
| `format`    | No       | —       | Content format hint. Set to `syslog` when tailing syslog-formatted files (use with `parse: json`, see below) |
| `parse`     | No       | —       | Set to `json` to parse syslog lines into structured JSON (requires `format: syslog`)                         |

{% hint style="warning" %}
Using `read_from: beginning` will send all existing log lines in the file. For large files, this may take considerable time and disk buffer space.
{% endhint %}

**Notes:**

* Only newline-delimited logs are supported (one log entry per line)
* Multi-line logs are not currently supported
* On startup, the log forwarder validates that file paths are accessible and reports permission errors before starting

#### Syslog Input

Receive syslog messages over the network:

```yaml
input:
  type: syslog
  mode: tcp              # "tcp" or "udp"
  address: 0.0.0.0:1514  # Listen address
```

| Field     | Required | Description                                                 |
| --------- | -------- | ----------------------------------------------------------- |
| `type`    | Yes      | Must be `syslog`                                            |
| `mode`    | Yes      | `tcp` or `udp`                                              |
| `address` | Yes      | Host and port to listen on                                  |
| `parse`   | No       | Set to `json` to parse syslog messages into structured JSON |

**Notes:**

* Messages will be sent in raw format to Panther. If your messages adhere to RFC 3164 or RFC 5424 format, you can use the built in corresponding syslog schema in Panther ([`Syslog.RFC5424`](https://docs.panther.com/supported-logs/syslog#syslog.rfc5424)/[`Syslog.RFC3164`](https://docs.panther.com/supported-logs/syslog#syslog.rfc3164)) for proper parsing and classification.
* Use `parse: json` to send the message parsed as json to your Panther source. Parsing is best effort and covers most common syslog formats, additionally to RFC compliant ones.

### Output Types

#### HTTP Output

Send logs to a Panther HTTP source:

```yaml
output:
  type: http
  url: https://logs.your-instance.runpanther.net/http/SOURCE_ID
  auth:
    type: bearer
    token: ptl-your-bearer-token
  buffer_size: 10GB  # Optional: override global default
```

| Field         | Required | Default | Description                                                                 |
| ------------- | -------- | ------- | --------------------------------------------------------------------------- |
| `type`        | No       | `http`  | Output type (can be omitted for HTTP)                                       |
| `url`         | Yes      | —       | Panther HTTP source URL                                                     |
| `auth`        | Yes      | —       | Authentication config (see [Credential Management](#credential-management)) |
| `buffer_size` | No       | `3GB`   | Per-forwarder disk buffer size override                                     |

#### S3 Output

Send logs to an Amazon S3 bucket — no AWS keys needed in config.

```yaml
output:
  type: s3
  bucket: my-log-bucket
  region: us-east-1
  key_prefix: "custom/prefix/%Y/%m/%d/"  # Optional
```

| Field         | Required | Default                           | Description                                                 |
| ------------- | -------- | --------------------------------- | ----------------------------------------------------------- |
| `type`        | Yes      | —                                 | Must be `s3`                                                |
| `bucket`      | Yes      | —                                 | S3 bucket name                                              |
| `region`      | Yes      | —                                 | AWS region (e.g., `us-east-1`)                              |
| `key_prefix`  | No       | `logs/{forwarder_name}/%Y/%m/%d/` | S3 key prefix (supports strftime tokens, must end with `/`) |
| `buffer_size` | No       | `3GB`                             | Per-forwarder disk buffer size override                     |

**Notes:**

* Authentication: **IAM instance profiles** (recommended) or **environment variables** (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`). File-based credentials (`~/.aws/credentials`) are not supported because the service does not have access to home directories.
* The IAM identity used must have `s3:PutObject` and `s3:HeadBucket` permissions on the target bucket
* Logs are gzip-compressed before upload
* Batching: 10MB or 60 seconds (whichever comes first)

### Credential Management

HTTP output requires a bearer token. There are two ways to provide it:

#### Plaintext Token

```yaml
output:
  auth:
    type: bearer
    token: ptl-your-bearer-token
```

Forwarder files are secured with `0640` permissions and `root:panther-log-forwarder` ownership — only root can write, only the agent service can read.

#### Environment Variable

Use `${ENV_VAR}` syntax to reference an environment variable:

```yaml
output:
  auth:
    type: bearer
    token: "${PANTHER_TOKEN}"
```

Set the variable in systemd, Docker Compose, Kubernetes Secrets, etc.

### Disk Buffering

Each forwarder has its own disk buffer (default: 3GB) to prevent data loss during network outages. Logs are stored on disk until they can be forwarded.

When the buffer fills:

* **File inputs** — the forwarder blocks and stops reading until buffer space frees up (no data loss, but log files may grow)
* **Syslog inputs** — the forwarder drops newest incoming messages (prevents memory issues, but some logs may be lost)

Configure buffer size globally in `panther.conf` or per-forwarder with `buffer_size` in the output section.

The default 3GB is sufficient for most deployments. Consider increasing the buffer size for high-volume sources to ensure no data loss.

### Syslog Parsing

By default, syslog messages are forwarded as raw text. Adding `parse: json` to the input parses syslog messages into structured JSON with fields like `timestamp`, `hostname`, `appname`, `severity`, and `message`.

This works with both input types:

* **Network syslog** (`type: syslog` + `parse: json`) — uses built-in syslog parsing
* **File syslog** (`type: file` + `format: syslog` + `parse: json`) — parses each line as a syslog message

#### Network syslog with parsing

```yaml
forwarders:
  parsed_syslog:
    input:
      type: syslog
      mode: tcp
      address: 0.0.0.0:1514
      parse: json
    output:
      url: https://logs.your-instance.runpanther.net/http/SOURCE_ID
      auth:
        type: bearer
        token: ptl-your-token
```

#### File syslog with parsing

Use `format: syslog` and `parse: json` on the input to indicate the file contains syslog-formatted lines that should be parsed:

```yaml
forwarders:
  os_syslog:
    input:
      type: file
      path: /var/log/syslog
      format: syslog
      parse: json
    output:
      url: https://logs.your-instance.runpanther.net/http/SOURCE_ID
      auth:
        type: bearer
        token: ptl-your-token
```

{% hint style="info" %}
`parse: json` requires a syslog context — either `input.type: syslog` or `input.format: syslog`. It cannot be used with plain file inputs.
{% endhint %}

{% hint style="warning" %}
When `parse: json` is enabled, parsing is best-effort, however it covers most common syslog formats including RFC 3164 / RFC 5424. Lines that fail to parse are dropped.
{% endhint %}

***

## More Examples

### Syslog Server

```yaml
# /etc/panther-log-forwarder/forwarders.d/syslog.yaml
forwarders:
  my_syslog:
    input:
      type: syslog
      mode: tcp
      address: 0.0.0.0:1514
    output:
      type: http
      url: https://logs.your-instance.runpanther.net/http/SOURCE_ID
      auth:
        type: bearer
        token: ptl-your-token
```

### Multiple Files with Exclusions

```yaml
# /etc/panther-log-forwarder/forwarders.d/app.yaml
forwarders:
  app_logs:
    input:
      type: file
      path: /var/log/myapp/*.log
      excludes: ["*.gz", "*.old"]
    output:
      type: http
      url: https://logs.your-instance.runpanther.net/http/SOURCE_ID
      auth:
        type: bearer
        token: ptl-your-token
```

### Multiple Forwarders in One File

```yaml
# /etc/panther-log-forwarder/forwarders.d/all-logs.yaml
forwarders:
  auth_logs:
    input:
      type: file
      path: /var/log/auth.log
    output:
      type: http
      url: https://logs.your-instance.runpanther.net/http/source1
      auth:
        type: bearer
        token: ptl-token-auth

  syslog_server:
    input:
      type: syslog
      mode: tcp
      address: 0.0.0.0:1514
    output:
      type: http
      url: https://logs.your-instance.runpanther.net/http/source2
      auth:
        type: bearer
        token: ptl-token-syslog
```

### Syslog to S3 with JSON Parsing

```yaml
# /etc/panther-log-forwarder/forwarders.d/syslog-s3.yaml
forwarders:
  syslog_to_s3:
    input:
      type: syslog
      mode: udp
      address: 0.0.0.0:514
      parse: json
    output:
      type: s3
      bucket: my-log-bucket
      region: us-east-1
```

### S3 Output

```yaml
# /etc/panther-log-forwarder/forwarders.d/s3.yaml
forwarders:
  s3_archive:
    input:
      type: file
      path: /var/log/app/*.log
    output:
      type: s3
      bucket: my-log-bucket
      region: us-east-1
```

***

## Troubleshooting

### Checking Logs

There are two log files to check when troubleshooting:

* **`/var/log/panther-log-forwarder/log-forwarder.log`** — Check this for issues with the forwarder service itself, such as startup failures, configuration errors, and connectivity problems.
* **`/var/log/panther-log-forwarder/vector.log`** — Check this for issues with the data pipeline, such as log ingestion problems, parsing errors, and delivery failures.

```bash
# Forwarder service logs
sudo tail -f /var/log/panther-log-forwarder/log-forwarder.log

# Data pipeline logs
sudo tail -f /var/log/panther-log-forwarder/vector.log
```

### Log Forwarder Won't Start

```bash
sudo panther-log-forwarder status
```

Common causes:

* **Configuration errors** — Run `sudo panther-log-forwarder validate`
* **Permission issues** — Verify `/etc/panther-log-forwarder/forwarders.d/` has `0750` permissions
* **File access errors** — The log forwarder checks file paths on startup. Verify paths exist and are readable by the `panther-log-forwarder` user
* **Port conflicts** — For syslog inputs, ensure the listening port is available

### No Logs Appearing in Panther

```bash
sudo panther-log-forwarder check-connectivity
sudo panther-log-forwarder status
```

Common causes:

* **Invalid bearer token** — Verify token matches Panther Console
* **Network issues** — Test HTTPS connectivity to Panther endpoint
* **Incorrect URL** — Double-check URL from Panther Console
* **File path issues** — Verify paths exist and are readable
* **S3 permissions** — For S3 output, verify IAM permissions allow PutObject and HeadBucket

### Configuration Errors

```bash
sudo panther-log-forwarder validate
```

Common issues:

* YAML syntax errors (indentation, quotes)
* Missing required fields (`type` in input, `url`/`auth` for HTTP, `bucket`/`region` for S3)
* Invalid input type (must be `file` or `syslog`)
* Invalid output type (must be `http` or `s3`)
* Invalid forwarder name (must start with a letter, alphanumeric/underscore/hyphen only)

***

## Operations

### Adding New Forwarders

1. Create a new source in Panther Console (HTTP or S3).
2. Add a new YAML file in `/etc/panther-log-forwarder/forwarders.d/`.
3. Validate: `sudo panther-log-forwarder validate`
4. Restart: `sudo systemctl restart panther-log-forwarder`

### Updating Configuration

1. Edit files in `/etc/panther-log-forwarder/`.
2. Validate: `sudo panther-log-forwarder validate`
3. Restart: `sudo systemctl restart panther-log-forwarder`

### Credential Rotation

1. Generate a new bearer token in Panther Console.
2. Update the token in your forwarder configuration file.
3. Restart: `sudo systemctl restart panther-log-forwarder`

### Upgrading

1. Download the new release tarball and checksum for your platform (see [Download Links](#download-links)).
2. Verify the checksum, extract, and run the installer:

```bash
sha256sum -c panther-log-forwarder_linux_amd64.tar.gz.sha256
tar -xzf panther-log-forwarder_linux_amd64.tar.gz
cd panther-log-forwarder
sudo ./install.sh
```

The installer automatically restarts the service if it was running. Verify the upgrade:

```bash
sudo panther-log-forwarder --version
sudo panther-log-forwarder status
```

Configuration files in `/etc/panther-log-forwarder/` are preserved during upgrades.

### Uninstalling

```bash
sudo systemctl stop panther-log-forwarder
cd panther-log-forwarder   # unpacked installation directory
sudo ./uninstall.sh
```

***

## File Locations

| Path                                               | Purpose            |
| -------------------------------------------------- | ------------------ |
| `/etc/panther-log-forwarder/panther.conf`          | Global settings    |
| `/etc/panther-log-forwarder/forwarders.d/*.yaml`   | Forwarder configs  |
| `/var/log/panther-log-forwarder/log-forwarder.log` | Log forwarder logs |
| `/var/log/panther-log-forwarder/vector.log`        | Vector logs        |