Data Replay (Beta)

Preview the outcome of a rule against real data before enabling it

Overview

Data Replay is currently in open beta, and is available to all customers. Please share any bug reports and feature requests with your account team.

With Panther's Data Replay, rules can be tested against historical log data to preview the outcome of a rule before enabling it.

When writing or updating a detection, use Data Replay to simulate what type of alerts you are likely to receive before deploying the detection. Alerts generated through a replay will not be delivered to the rule's destinations or displayed on the main Alerts & Errors dashboard.

Data Replay is available in the Panther Console. In the CLI workflow, you can use Panther Analysis Tool's benchmark command to evaluate rule performance.

Limitations

Before starting a Data Replay, note the following limitations:

• Time range is configurable but must be within the last 15 days

• Time range must be older than 24 hours

• Replay must complete in under an hour

• Access to the DynamoDB cache is blocked to prevent polluting production data

• Network calls from within a rule are blocked

Starting a replay

A rule's deduplication period and events threshold will be taken into account during Data Replay. If a rule has Rule Filters, they will be applied.

1. In the left-hand navigation bar of your Panther Console, click Build > Detections.

2. Click on the name of the detection you want to use in a Data Replay.

3. Scroll down to the Data Replay tile, within the Test section.
4. In the Log Types field, select the log types you wish to replay for the detection.

5. On the right side of the Data Replay section, select Time or Data(GB), depending on how you want to limit the set of data your replay will run over:

   • Time: Choose this option to run a replay on the data collected during a specified period of time.

     • Click the Date Start or Date End fields to open the date modal. You can select a preset day range or pick custom dates in the available range:

     
   • Data(GB): Choose this option to run a replay on a data set for a specified size (in gigabytes.)

     • Select one of the options in the Data dropdown.

     • Note that the data set is not guaranteed to be the most recent n number of gigabytes.

6. Click Create Replay.

   • The estimated run time and data size will be displayed.

   • Note: If you have made any changes to the rule, you must save your changes before starting a Replay.

As the Replay processes, the progress will be displayed live, including the estimated volume processed and matched events. You can exit this page at any time without cancelling the Replay.

If you have a high rate of matched events, we recommend stopping the replay to tune your detection.

Viewing replay results

After the Replay is complete, you will see an alert summary on the detection's page that includes total alerts, total rule errors, volume processed, and total matched events. All alerts will be shown as they would appear if the alert had been enabled during this range of time.

On this page, you can filter alerts and view the events associated with each alert.

Click on an alert for more information. On the alert's page, click the Details tab to see which destinations the alerts would have been sent to if the rule was live. This page also includes the start time, end time, and duration of the replay.

Note: No alerts are sent to their destinations while replaying data.