# Filtering (Beta)

## Overview

{% hint style="info" %}
Filtering is in open beta starting with Panther version 1.78, and is available to all customers. Please share any bug reports and feature requests with your Panther support team.
{% endhint %}

Raw event filters allow you to filter your log data ingested into Panther, using regex expressions or substrings patterns. Excluding certain events from ingestion can help improve the hygiene of your data lake, and lower costs.

### Types of raw event filters

There are currently two types of filters:

* Regex filters**:** Events that match the regex expression will be dropped.
  * Regex filters use [Google's RE2 engine](https://github.com/google/re2/wiki/Syntax).
* Substring filters**:** Events that include the pattern at least once will be dropped.

<figure><img src="https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FKkYyy8RbwVJXljWGc96O%2FScreenshot%202023-07-13%20at%203.27.26%20PM.png?alt=media&#x26;token=8649d7ab-d98a-43c7-ad10-fc27a3aed95f" alt="In a log event filter, an Exclusion Condition is shown. The filter reads, &#x22;Exclude if&#x22; and a select box is open, showing two options: &#x22;Matches Regex&#x22; and &#x22;Contains&#x22;"><figcaption></figcaption></figure>

## Creating a raw event filter

{% hint style="info" %}
Filters are applied on raw events—*not* normalized data visible in the data lake, which can differ. Ensure you are constructing filters based on raw data. Basing filters on normalized data could cause false positives and unintentionally dropped data.
{% endhint %}

To create a raw event filter:

1. In the left-hand navigation bar of your Panther Console, click **Configure** > **Log Sources**.
2. Click the name of the log source you'd like to add a filter to.
3. Click the **Filters** tab.
4. Click **New Filter**.\
   ![The "Filters" tab of an "Aurora logs" Log source is shown. There is an arrow drawn to a blue "New Filter" button.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2F2AK17Wcw57kzpy2KfDdF%2Ffilt.webp?alt=media\&token=ba20d4f5-9e4e-460f-8c7c-a962d8a8b0c4)
5. Provide a value in the **Add Filter name or short description** field, then click the checkmark to the right of the field to save it.
6. In the **Exclusion Condition** section, click the **+** to the right of **Exclude if**.\
   ![An "Exclusion Condition" is shown. There is an "Exclude if" statement, with a plus sign to its right. The plus sign is circled.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FPVjeqd5qEWRKiHqFP0pv%2FScreenshot%202023-07-13%20at%2012.42.52%20PM.png?alt=media\&token=009e84c7-5e73-445a-a249-bf1951a30ca2)
7. Click **Condition**, and select one of the options below. Learn more about the difference ways to construct exclusion statements in [Types of raw event filters](#types-of-raw-event-filters).
   * **Matches Regex**
   * **Contains**\
     ![In a log event filter, an Exclusion Condition is shown. The filter reads, "Exclude if" and a select box is open, showing two options: "Matches Regex" and "Contains"](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FlUN42yo5DutxJU2bv6wC%2FScreenshot%202023-07-13%20at%2012.48.03%20PM.png?alt=media\&token=4614ec9b-7396-4798-9cec-80fc96e231a0)
8. If you used the **Matches Regex** condition, enter a regular expression. If you used the **Contains** condition, enter a string value.
9. In the **Quick Test** section, enter a raw event to test against the filter you just created.&#x20;
   * You can click **View raw data** to see raw events received by the source. To the right of an event, click **Test event** to populate the **Raw Event** field in **Quick Test** with the event.\
     ![A list of raw events is shown. Each row has a "Test event" button on the right-hand side. The "Test event" button in the first row is circled.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FdiylJ94nYw5zdP03eaH5%2FScreenshot%202023-07-13%20at%2012.58.31%20PM.png?alt=media\&token=dfaebe3e-237f-464c-9ac6-546bef5e756b)
10. Click **Run Test**.
    * Notice whether the test event matches the exclusion pattern.
11. Click **Save**.

## Enabling or disabling a raw event filter

To enable or disable a filter, by clicking on the toggle in the righthand side corner.

1. In the left-hand navigation bar of your Panther Console, click **Configure** > **Log Sources**.
2. Click the name of the log source you'd like to add a filter to.
3. Click the **Filters** tab.
4. On the right-hand side of the tile of the filter you would like to enable or disable, click the toggle next to **Enabled**.\
   ![The "Filters" tab of a HTTP source is shown. To the right of the filter with the name "Filter non Json rows," the toggle next to "Enabled" is on.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FC08mDQlNSnF2S73HYq1v%2FScreenshot%202023-07-13%20at%201.15.59%20PM.png?alt=media\&token=c15d93f7-131b-4898-acc0-39448c2197e4)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/data-onboarding/filtering.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.