# Saved and Scheduled Searches ## Overview You can avoid repeatedly creating the same searches in Panther's [Data Explorer](https://docs.panther.com/search/data-explorer) and [Search](https://docs.panther.com/search/search-tool) by saving your searches. You can also schedule searches created in Data Explorer, which allows you to then run results against a rule. This workflow includes the following features: * [Create a Saved Search](#how-to-create-a-saved-search), a preserved search expression. * [Create a Scheduled Search](#how-to-create-a-scheduled-search), a Saved Search that you can schedule to run on a designated interval. * [Create a Scheduled Rule](https://docs.panther.com/detections/rules#how-to-write-scheduled-rules), a detection that's associated with a Scheduled Search. The data returned each time the search executes is run against the detection, alerting when matches are found. By default, each Panther account is limited to 10 active Scheduled Searches. This limit is only precautionary, and can be increased via a support request. There is no additional cost from Panther for raising this limit, however you may incur extra charges from the database backend, depending on the volume of data processed. {% hint style="info" %} In the [CLI workflow](https://github.com/panther-labs/panther-docs/blob/main/docs/gitbook/search/scheduled-searches/broken-reference/README.md), Saved and Scheduled Searches are often referred to as queries. {% endhint %} ## How to create a Saved Search A Saved Search is a preserved search expression. Saving the searches your team runs frequently can help reduce duplicated work. You can create Saved Searches in the Panther Console (in either Search or Data Explorer), using the CLI workflow (PAT), or with the Panther API. You can also add variables in your Saved Searches, creating Templated Queries. Learn more on [Templated Queries and Macros](https://docs.panther.com/search/scheduled-searches/templated-searches). {% tabs %} {% tab title="Console" %} **How to create a Saved Search in the Panther Console** You can save a search in Panther's Data Explorer or Search. Searches saved in both tools are considered Saved Searches. Follow [these instructions for how to save a search in Data Explorer](https://docs.panther.com/data-explorer#create-a-saved-or-scheduled-search-in-data-explorer), and [these instructions for how to save a search in Search](https://docs.panther.com/search-tool#creating-a-saved-search). {% endtab %} {% tab title="CLI (PAT)" %} **How to create a Saved Search in the CLI workflow** Writing a Saved Search locally means creating a file that defines a SQL query on your own machine, then uploading it to your Panther instance (typically via the [Panther Analysis Tool](https://docs.panther.com/resources/help/glossary#panther-analysis-tool-pat)). We recommend managing your local detection files in a version control system like GitHub or GitLab. {% hint style="info" %} It's best practice to create a fork of Panther's [open-source analysis repository](https://github.com/panther-labs/panther-analysis), but you can also create your own repo from scratch. {% endhint %} **File setup** Each Saved Search consists of: * A YAML file (`.yml` or `.json` extension) containing [metadata attributes of the Saved Search](#saved-search-specification-reference). **Folder setup** If you group your queries into folders, each folder name must contain `queries` in order for them to be found during upload (using either PAT or the bulk uploader in the Console). We recommend grouping searches into folders based on log/resource type. You can use the open source [Panther Analysis](https://github.com/panther-labs/panther-analysis) repo as a reference. **Write the Saved Search** In your Saved Search file (called, for example, `new-saved-search.yml`), write your Saved Search, following the template below. See the full list of available fields in the [Saved Search specification reference](#saved-search-specification-reference). ```yaml AnalysisType: saved_query QueryName: MySavedQuery Description: Example of a saved query for PAT Query: |- Your query goes here Tags: - Your tags ``` **Upload the content with PAT** * Use the PAT upload command:\ `panther_analysis_tool upload --path --api-token --api-host https://api..runpanther.net/public/graphql` * Replace the values: * `` : The [API key](https://docs.panther.com/panther-developer-workflows/api#step-1-creating-an-api-token) you generated. * `` : The fairytale name of your instance (e.g. **carrot-tuna**.runpanther.net). * `` : The path to your Saved Search on your own machine. When your Saved Search is uploaded, each of the fields you would normally populate in the Panther Console will be auto-filled. See [Saved Search Specification Reference](#saved-search-specification-reference) for a complete list of required and optional fields. {% endtab %} {% tab title="API" %} **How to create a Saved Search in the Panther API** * See the `POST` operation on [Queries](https://docs.panther.com/panther-developer-workflows/api/rest/queries). {% endtab %} {% endtabs %} ## How to create a Scheduled Search A Scheduled Search is a Saved Search that has been configured to run on a schedule. Using the Panther Console, currently only Saved Searches created in Data Explorer can be scheduled—Saved Searches created in Search (including those created in both SQL and [PantherFlow](https://docs.panther.com/pantherflow)) *cannot* be scheduled. You can alternatively create and upload Scheduled Searches using the CLI workflow or the Panther API. {% hint style="info" %} It's strongly recommended to use `p_occurs_since` in your Scheduled Search—[learn more below](#using-p_occurs_since-to-define-a-lookback-window). {% endhint %} Note that creating a Scheduled Search alone won't run the returned data against detections or generate [signals](https://docs.panther.com/detections/signals). To do this, you must also [create a Scheduled Rule](https://docs.panther.com/detections/rules#how-to-write-rules-and-scheduled-rules), and associate it with your Scheduled Search. {% hint style="warning" %} [Customer-configured Snowflake](https://docs.panther.com/system-configuration/panther-deployment-types/legacy-configurations/snowflake-setup#customer-configured-snowflake) accounts: Your company will incur costs on your database backend every time a Scheduled Search runs. Please make sure that your searches can complete inside the specified timeout period. This does not apply to accounts that use Panther-managed Snowflake. {% endhint %} {% tabs %} {% tab title="Data Explorer" %} **How to create a Scheduled Search in Data Explorer** To learn how to schedule your Saved Search created in Data Explorer, follow one of the below sets of instructions: * If you haven't yet created a Saved Search in Data Explorer, follow the [Create a Saved or Scheduled Search in Data Explorer](https://docs.panther.com/data-explorer#create-a-saved-or-scheduled-search-in-data-explorer) instructions, paying attention to **Create a scheduled search** in Step 4. * If you've already saved the search in Data Explorer, follow the [Update a Saved or Scheduled Search in Data Explorer](https://docs.panther.com/data-explorer#update-a-saved-or-scheduled-search-in-data-explorer) instructions, paying attention to Step 6. {% endtab %} {% tab title="CLI (PAT)" %} **How to create a Scheduled Search in the CLI workflow** Writing a Scheduled Search locally means creating a file that defines a SQL query on your own machine, then uploading it to your Panther instance (typically via the [Panther Analysis Tool](https://docs.panther.com/resources/help/glossary#panther-analysis-tool-pat)). We recommend managing your local detection files in a version control system like GitHub or GitLab. {% hint style="info" %} It's best practice to create a fork of Panther's [open-source analysis repository](https://github.com/panther-labs/panther-analysis), but you can also create your own repo from scratch. {% endhint %} **File setup** Each scheduled query consists of: * A YAML file (`.yml` or `.json` extension) containing [metadata attributes of the Scheduled Search](#scheduled-search-specification-reference). View an [example Scheduled Search YAML file here](https://github.com/panther-labs/panther-analysis/blob/master/templates/example_scheduled_query.yml). **Folder setup** If you group your searches into folders, each folder name must contain `queries` in order for them to be found during upload (using either PAT or the bulk uploader in the Console). We recommend grouping searches into folders based on log/resource type. You can use the open source [Panther Analysis](https://github.com/panther-labs/panther-analysis) repo as a reference. **Write the Scheduled Query** In your Scheduled Search file (called, for example, `new-scheduled-search.yml`), write your Scheduled Search, following the template below. See the full list of available fields in the [Scheduled Search specification reference](#scheduled-search-specification-reference). ```yaml AnalysisType: scheduled_query QueryName: ScheduledQuery_Example Description: Example of a scheduled query for PAT Enabled: true Query: |- Select 1 Tags: - Your tags Schedule: CronExpression: "0 0 29 2 *" RateMinutes: 0 TimeoutMinutes: 2 ``` **Upload the content with PAT** * Use the PAT upload command:\ `panther_analysis_tool upload --path --api-token --api-host https://api..runpanther.net/public/graphql` * Replace the values: * `` : The [API key](https://docs.panther.com/panther-developer-workflows/api#step-1-creating-an-api-token) you generated. * `` : The fairytale name of your instance (e.g. **carrot-tuna**.runpanther.net). * `` : The path to your Saved Query on your own machine. When your Scheduled Search is uploaded, each of the fields you would normally populate in the Panther Console will be auto-filled. See [Scheduled Search Specification Reference](#scheduled-search-specification-reference) for a complete list of required and optional fields. {% endtab %} {% tab title="API" %} **How to create a Scheduled Search in the Panther API** * See the `POST` operation on [Queries](https://docs.panther.com/panther-developer-workflows/api/rest/queries). {% endtab %} {% endtabs %} ### Using `p_occurs_since` to define a lookback window To define a lookback window in your Scheduled Search, it's strongly recommended to use the Panther SQL macro [`p_occurs_since`](https://docs.panther.com/data-explorer#time-offset-from-present-p_occurs_since). When a Scheduled Search runs, the "now" time in `p_occurs_since` is replaced with the scheduled time (rather than the actual current time). This makes the lookback window precise. With each run, the Search steps forward in time according to its schedule.\ \ Using `p_occurs_since` protects against disruption due to Snowflake or cloud provider outages. If the Scheduled Search cannot be run due to an outage, the scheduled time is not advanced. Once the outage is recovered and queries can again be run, the Scheduled Search steps forward in time until it is caught up to the current schedule and no longer behind. ### How to use the Scheduled Search crontab Panther's Scheduled Search crontab uses the standard crontab notation consisting of five fields: minutes, hours, day of month, month, day of week. Additionally, you will find a search timeout selector (with a maximum value currently set at 10 minutes). The expression will run on UTC. The interpreter uses a subset of the standard crontab notation: ``` ┌───────── minute (0 - 59) │ ┌──────── hour (0 - 23) │ │ ┌────── day of month (1 - 31) │ │ │ ┌──── month (1 - 12) │ │ │ │ ┌── day of week (0 - 6 => Sunday - Saturday) │ │ │ │ │ ↓ ↓ ↓ ↓ ↓ * * * * * ``` If you want to specify day by day, you can separate days with dashes (`1-5` is Monday through Friday) or commas, for example `0,1,4` in the `Day of Week` field will execute the command only on Sundays, Mondays and Thursdays. Currently, we do not support using named days of the week or month names. Using the crontab allows you to be more specific in your schedule than the Period frequency option:
The Cron expression screen displays options for selecting a time range for the scheduled query to run.
### How to configure a Scheduled Search to generate an email report You can configure a Scheduled Search to send an email report each time it runs. The email report contains a download link for the results (if there are more than zero), a link to the search in Data Explorer, and optionally a CSV attachment containing the search results. If the Scheduled Search is associated to any Scheduled Rules, those Scheduled Rules will process returned data as usual. {% hint style="info" %} Email reports are sent from `do-not-reply@panther-notifications.email`. It's recommended to add this address to your personal or organizational allowlist to avoid the emails being labeled as spam. {% endhint %}
Under the Panther logo, a title reads, "1 events found by your scheduled search." Below that is a "Download search results as CSV" button.
The following parameters for email reports apply: * Each email report can be sent to a maximum of 10 recipients. * The maximum size of the CSV attachment is 10 MB. If your query results are large enough that the CSV exceeds this size, the CSV will not be attached to the email. * You can send up to 20 email reports per day, across all Scheduled Searches in your Panther instance. * The number of email reports sent per day depends on how often your Scheduled Searches run. * For example, if you have one Scheduled Search that sends 15 email reports per day, and another that sends five email reports per day, you will be unable to configure an additional Scheduled Search to generate email reports unless you reduce the frequency of the existing configurations. Only Scheduled Searches can be configured for email reporting—because searches made in [Search](https://docs.panther.com/search/search-tool) with [PantherFlow](https://docs.panther.com/pantherflow) or [filter expressions](https://docs.panther.com/search-tool#creating-filter-expressions) can be [saved but not scheduled](https://docs.panther.com/search-tool#creating-a-saved-search), they cannot be configured to generate email reports. {% tabs %} {% tab title="Console" %} **How to configure a Scheduled Search to generate an email report in the Panther Console** 1. Follow the [Create a Saved or Scheduled Search in Data Explorer](https://docs.panther.com/data-explorer#create-a-saved-or-scheduled-search-in-data-explorer) instructions, ensuring the **Email scheduled search results** toggle is set to **ON**. * If you'd like to configure an existing Scheduled Search, access its **Update Search** modal by following the [Update a Saved Search's metadata](#update-a-saved-searchs-metadata) instructions, and set the **Email scheduled search results** toggle to **ON**.
In a "Save Search" modal, various fields, such as "Tags" and "Description" are visible. An "Email scheduled search results" toggle is circled.
2. Configure the email report settings: * **Recipients**: Select or enter up to 10 recipients. The values in the dropdown list are the users in your Panther Console, but you can also type in additional email addresses. * **Attach CSV data to the email**: Toggle **ON** if you'd like the email report to contain a CSV attachment containing the query results (up to 10MB). * **Send an email even if search returns 0 results**: Toggle **ON** if you'd like the email report to be sent even if the Scheduled Search generates zero results.
Three toggles are shown: Email scheduled search results, Attach CSV data to the email, and Send an email even if search returns 0 results.
{% endtab %} {% tab title="CLI (PAT)" %} **How to configure a Scheduled Search to generate an email report in the CLI workflow** You can configure a Scheduled Search for email reporting adding an `EmailConfig` object at the root level in the YAML file, along with the `Recipients`, `SendEmpty`, and `PreferAttachment` nested fields. Learn more about these fields below, in the [Scheduled Search specification reference](#scheduled-search-specification-reference). ```yaml AnalysisType: scheduled_query QueryName: ScheduledQuery_Example Description: Example of an email report for PAT Enabled: true Schedule: RateMinutes: 900 TimeoutMinutes: 2 EmailConfig: Recipients: - myemail@example.com - otheruser@otherexample.com SendEmpty: true PreferAttachment: true ``` {% endtab %} {% tab title="API" %} **How to configure a Scheduled Search to generate an email report in the Panther API** * See the `POST` operation on [Queries](https://docs.panther.com/panther-developer-workflows/api/rest/queries). {% endtab %} {% endtabs %} ## Using Saved and Scheduled Searches ### How to delete or download a Saved Search You can delete Saved Searches individually or in bulk. Note that if a Saved Search is scheduled (i.e., it's a Scheduled Search), it must be unlinked from any Scheduled Rules it's associated to in order to be deleted. 1. In the left-hand navigation bar of your Panther Console, click **Investigate** > **Saved Searches**. 2. In the list of Saved Searches, find the search or searches you'd like to download or delete. Check the box to the left of the name of each search. 3. At the top of the page, click either **Download** or **Delete**.\ ![The top of the Saved Queries page is shown, with two buttons: "Download" and "Delete"](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-d575f18561cfc2324d109d6e566113102a80c3d6%2FScreenshot%202023-03-27%20at%202.44.37%20PM.png?alt=media) * If you clicked **Download**, a `saved_queries.zip` file will be downloaded. * If you clicked **Delete**, an **Attention!** modal will pop up. Click **Confirm**.\ ![A modal titled "Attention!" is shown, with the text, "Are you sure you want to delete these (2) selected Saved Queries" and there are "Cancel" and "Confirm" buttons](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-4b46528c42e90ef03486dbf5730678500ca4d68a%2FScreenshot%202023-10-02%20at%2010.10.05%20AM.png?alt=media) ### How to deactivate a Scheduled Search 1. In the left-hand navigation bar of your Panther Console, click **Investigate** > **Saved Searches**. 2. Find the Scheduled Search you'd like to deactivate, and in the upper right corner of its tile, click the three dots icon.\ ![The image shows a query from the list of queries in the Panther Console. In the right side, there is a red arrow pointing to the 3 dots icon.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-65ce9f39260fd7bd2460c18f1a54f257a3852455%2Fquery-options.png?alt=media) 3. In the dropdown menu, click **Edit Search Metadata**. 4. In the **Update Search** form, toggle the setting **Is it active?** to **OFF** to disable the query.\ ![The "Update Search" form is displayed. It contains fields for Search Name, Tags, Description, and Default Database. The toggle next to "Is it active?" is set to "Off."](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-91f7ba5c3413baf5bd67bee782fe71f935573e5d%2FScreenshot%202023-10-02%20at%2010.12.13%20AM.png?alt=media) 5. Click **Update Query** to save your changes. ### Update a Saved Search's metadata To edit a Saved Search's name, tags, description, and default database (and, for Scheduled Searches, whether it's active, its frequency, and its [email report settings](#how-to-configure-a-scheduled-search-to-generate-an-email-report)): 1. In the left-hand navigation bar of your Panther Console, click **Investigate** > **Saved Searches**. 2. Locate the query you'd like to edit, and click the three dots icon in the upper right corner of its tile.\ ![The 3 dots icon is expanded to a dropdown menu. The option "Edit Search Metadata" is highlighted.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-217a7d93b9ddc70c74bf150a4063b62aab0eb309%2FScreenshot%202023-10-02%20at%2010.14.23%20AM.png?alt=media) 3. In the dropdown menu, click **Edit Search Metadata**. 4. Make changes in the **Update Search** form as needed. 5. Click **Update Search**. ### Search for Saved Searches On the Saved Searches page, you can search for queries using: * The search bar at the top of the queries list * The date range selector in the upper right corner * The **Filters** option in the upper right corner * Filter by whether the query is scheduled, whether its active, its type (**Native SQL**, **Search**, or **PantherFlow Search**), or by up to 100 tags.
in the Saved Searches list, use the date range or filters in the upper right corner to search for queries. In the image, the date range selector is circled and the Filters button is circled.
Click on the name of the Saved Search to be taken directly to Data Explorer (for **Native SQL** queries) or Search (for **Search** and **PantherFlow Search** searches) with the query populated. ### Use `LIMIT`s in Scheduled Searches In the Panther Data Lake settings page, you can optionally enable a setting that will check if a Scheduled Search has a `LIMIT` clause specified. Use this option if you're concerned about a Scheduled Search unintentionally returning thousands of results, potentially resulting in alert delays, Denial of Service (DoS) for downstream systems and general cleanup overhead from poorly tuned queries. {% hint style="info" %} Scheduled Searches that result in a timeout will generate a [`System Error`](https://docs.panther.com/system-configuration/notifications/system-errors) to identify that the Scheduled Search was unsuccessful. {% endhint %} 1. In the upper right corner of the Panther Console, click the gear icon. In the dropdown menu that appears, click **General**.\ ![The gear icon's dropdown menu is expanded, showing options for General, Users, User Roles, API Tokens, and API Playground.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-121bf556d2dff1ab321d37338c87c8b3ca75544c%2Fimage.png?alt=media) 2. Click the **Data Lake** tab. 3. Scroll down to the **Scheduled Queries** header. Below the header, you will see the LIMIT clause toggle setting:\ ![At the top, a tab labeled Data Lake is selected. Near the bottom of the screen, there is a header called "Scheduled Queries." The option "LIMIT Clause for Scheduled Queries" is set to "Off."](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-7154406b308fdb2c72bdaa1f4f64b362e5dba3b5%2Fimage.png?alt=media) 4. Toggle the **`LIMIT` Clause for Scheduled Queries** setting to **ON** to start enforcing LIMITs in Scheduled Queries.\ ![The toggle next to "LIMIT Clause for Scheduled Queries" is set to "On."](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-1b312a812cd23351da22239f65fe8b032fd0e368%2Fimage.png?alt=media) When this field is set to **ON**, any new Scheduled Searches marked as active cannot be saved unless a LIMIT clause is specified in the query definition. ![The image shows the query creation screen. There is a red banner at the top that says "Unable to create Saved query. This scheduled query does not contain a LIMIT clause in the SQL expression. Update the SQL expression to add a LIMIT clause to save this scheduled query."](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-72726a7347bb7e2fe80ed5278a0ffd8fc8a8864f%2Fimage.png?alt=media) Existing Scheduled Searches without a LIMIT clause will appear with a warning message in the list of Saved Searches, and edits cannot be saved unless a LIMIT clause is included.
A Scheduled Query without a LIMIT clause shows a warning banner that says "This Scheduled Query does not contain a LIMIT clause in the SQL expression."
The setting only checks for the existence of a LIMIT clause anywhere in the Saved Search. It does not check specifically for outer LIMIT clauses. ### Exporting Scheduled Searches from your Panther Console You can export a .zip file of all of the detections and Scheduled Searches in your Panther Console: 1. In the left-hand navigation bar of your Panther Console, click **Detections**. 2. In the upper-right corner, click **Upload**. 3. In the **Bulk Uploader** modal, click **Download all entities**. ## Saved Search specification reference Required fields are in **bold**. A complete list of Saved Search specification fields:
Field NameDescriptionExpected Value
AnalysisTypeIndicates whether this analysis is a Rule, Policy, Scheduled Search, Saved Search, or global.saved_query
QueryNameA friendly name to show in the UI.String
TagsTags used to categorize this rule.List of strings
DescriptionA brief description of the rule.String
QueryA data query.
Must be written in SQL (i.e., cannot be PantherFlow).		String
## Scheduled Search specification reference Required fields are in **bold**. A complete list of Scheduled Search specification fields:
Field NameDescriptionExpected Value
AnalysisTypeIndicates whether this analysis is a Rule, Policy, Scheduled Search, Saved Search, or global.scheduled_query
QueryNameA friendly name to show in the UI.String
EnabledWhether this rule is enabled.Boolean
TagsTags used to categorize this rule.List of strings
DescriptionA brief description of the rule.String
QueryA data query.String
Schedule

The schedule that this query should run. Expressed with a CronExpression or in Rate Minutes. TimeoutMinutes is required to release the query if it takes longer than expected. Note that cron and rate minutes are mutually exclusive.

CronExpression: "0 0 29 2 *"
  RateMinutes: 0
  TimeoutMinutes: 2
Map
EmailConfig

If provided, generating email reports for your Scheduled Search. Supports three nested fields:

  • (Required) Recipients: An array of strings, each containing an email address to send the email report to. There is a maximum of 10 recipients.
  • (Optional) PreferAttachment: A boolean field that defaults to false. If set to true, the email report will contain a CSV attachment of the Scheduled Search results (if the results fit within the 10 MB attachment size limit).
  • (Optional) SendEmpty: A boolean field that defaults to false. If set to true, the email report will be sent even if the Scheduled Search generates zero results.
EmailConfig:
  Recipients:
    - myemail1@example.com
    - myemail2@example.com
  PreferAttachment: true
  SendEmpty: false
Map