Data Transports

Panther integrates with various common data transport log ingestion sources

Overview

A Data Transport is a type of log source that sends log types that are not natively supported by Panther—i.e., custom log types. Using Data Transports, you can process custom data types through Panther’s log processing pipeline, map existing detections to custom data types, and map data models to custom data types.

In addition to using a Data Transport to onboard your custom logs, you'll need to create a custom schema to normalize and classify the data.

Learn about sending compressed data to Panther, as well as latency expectations, below.

Panther-supported Data Transports

Panther currently supports the following Data Transport methods:

• HTTP (Webhook)

• AWS

  • S3

  • CloudWatch

  • SQS

    • SNS

  • Amazon EventBridge

• Google Cloud

  • Cloud Storage (GCS)

  • Pub/Sub

• Azure Blob Storage

Ingesting compressed data in Panther

The following Data Transport mechanisms support ingesting compressed data (according to one of the below algorithms):

• HTTP
• Azure Blob Storage
• AWS S3
• GCS

Panther will transparently decompress compressed data, meaning no extra headers are required. Panther's decompression works by first determining the compression algorithm each file was compressed with. This is not inferred from the file extension or metadata, but rather the content of the file itself.

The inner data of a compressed payload must match the stream type you've configured for your Data Transport source.

Supported compression algorithms

Panther supports the following compression formats:

• gzip
• zstd

  • Panther only supports zstd data that was compressed without the use of a dictionary.

Supported columnar formats

Parquet

Panther can transparently detect and ingest Parquet files with a maximum size of 100MB (compressed).

The inner data of Parquet files can be compressed with the following algorithms:

• gzip

• snappy

Avro

Avro file ingestion is in open beta starting with Panther version 1.109, and is available to all customers. Please share any bug reports and feature requests with your Panther support team.

Panther can transparently detect and ingest Apache Avro Object Container Files (OCF).

The inner data of the OCF files can be compressed with any of the following algorithms:

• gzip

• snappy

• zst

Data Transports latency

When forwarding data to Panther using a Data Transport source, you can expect data to be ingested within five minutes. After ingestion, it can then take up to two and a half minutes for an alert to be generated (when applicable).

These time frames are applicable to all Data Transport sources; there is not a particular source that has a shorter latency than the others.

Troubleshooting Data Transports

Visit the Panther Knowledge Base to view articles about data transports that answer frequently asked questions and help you resolve common errors and issues.