# Testing ## Overview Testing your detections ensures that once deployed, detections will behave as expected, generating alerts appropriately. Testing also promotes reliability and protects against regressions as code evolves over time. Testing works by defining a test log event for a certain detection, and indicating whether or not you'd expect an alert to be generated when the test event is processed by that detection. Panther doesn't enforce a required minimum or maximum number of tests for each detection, but it's recommended to configure at least two—one false positive and one true positive. You can create, edit, and delete tests for custom detections (i.e., those that aren't [Panther-managed](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/detections/panther-managed)). Tests for [Panther-managed](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/detections/panther-managed) detections are maintained by Panther and are read-only. Panther's [Data Replay](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/detections/testing/data-replay), which allows you to run historical log data through a rule to preview the outcome, is also useful while testing. ## Using tests ### How to create a test You can create tests for detections that are not [Panther-managed](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/detections/panther-managed) in the Panther Console, or programmatically with the [Panther Analysis Tool](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/panther-developer-workflows/ci-cd/deployment-workflows/pat). {% tabs %} {% tab title="Console: Detection page" %} {% hint style="info" %} This consolidated user interface for viewing and editing detections is in open beta starting with Panther version 1.74. Please share any bug reports and feature requests with your Panther support team. {% endhint %} 1. In the left-hand navigation bar of your Panther Console, click **Build > Detections**. 2. Click the name of an existing detection, or [create a new detection](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/detections/..#how-to-write-detections). 3. Scroll down to the **Test** section. 4. On the right-hand side of the **Unit Test** tile, click **Add New**.\ ![The Unit Test section of the detection page shows the names of one already-created test ("Failed login"). In the upper right corner there's a "Add New" button, which is circled.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FOQyne2h3r959acfXK3zy%2Fimage.png?alt=media\&token=a312da85-5cf2-4767-9ebd-f52b3df40e41) * If the detection is [Panther-managed](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/detections/panther-managed), **Add New** will be disabled, as you cannot create new tests. 5. The newly created test is given a placeholder name. To its right, click the three dots icon and select **Rename**. \ ![In the Unit Test section, a test with the placeholder name "Test-b789c4" is shown. The three dot icon to its right has been selected, which opened a menu with two options: Rename and Delete. ](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FUKkwbKg850TA3PqEquyO%2Fimage.png?alt=media\&token=43f60b3a-bb45-4508-8db3-dfacccf08512) 6. Provide a meaningful name for the test, and press enter, or return, to save it. 7. Set the **The detection should trigger based on the example event** toggle to`YES` or `NO`. 8. Compose a test JSON event in the text editor. 9. To see whether your test passes, click **Run Test**. 10. When you are finished, in the upper-right corner of the page, click **Update** or **Save**. {% endtab %} {% tab title="Console: Alert event" %} In the Panther Console, you can create a new unit test directly from an alert, using an actual event associated with the alert. Follow the instructions in [Add filters from an alert event](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/rule-filters#add-filters-from-an-alert-event), taking note of steps 9 and 10. Note that this functionality is only available for rules. {% endtab %} {% tab title="Developer Workflows" %} **Rules** In your spec file, add the `Tests` key with sample cases: ``` Tests: - Name: Name to describe our first test LogType: LogType.GoesHere ExpectedResult: true or false Log: { "hostName": "test-01.prod.acme.io", "user": "martin_smith", "eventTime": "June 22 5:50:52 PM" } ``` We recommend running as many test cases as possible, including both true and false positives. **Policies** In the spec file, add the following `Tests` key: ``` Tests: - Name: Name to describe our first test. ResourceType: AWS.S3.Bucket ExpectedResult: true Resource: { "PublicAccessBlockConfiguration": null, "Region": "us-east-1", "Policy": null, "AccountId": "123456789012", "LoggingPolicy": { "TargetBucket": "access-logs-us-east-1-100", "TargetGrants": null, "TargetPrefix": "acmecorp-fiancial-data/" }, "EncryptionRules": [ { "ApplyServerSideEncryptionByDefault": { "SSEAlgorithm": "AES256", "KMSMasterKeyID": null } } ], "Arn": "arn:aws:s3:::acmecorp-fiancial-data", "Name": "acmecorp-fiancial-data", "LifecycleRules": null, "ResourceType": "AWS.S3.Bucket", "Grants": [ { "Permission": "FULL_CONTROL", "Grantee": { "URI": null, "EmailAddress": null, "DisplayName": "admins", "Type": "CanonicalUser", "ID": "013ae1034i130431431" } } ], "Versioning": "Enabled", "ResourceId": "arn:aws:s3:::acmecorp-fiancial-data", "Tags": { "aws:cloudformation:logical-id": "FinancialDataBucket" }, "Owner": { "ID": "013ae1034i130431431", "DisplayName": "admins" }, "TimeCreated": "2020-06-13T17:16:36.000Z", "ObjectLockConfiguration": null, "MFADelete": null } ``` The value of Resource can be a JSON object copied directly from the Policies > Resources explorer. {% endtab %} {% endtabs %} ### How to rename or delete a test in the Panther Console You can rename or delete tests for detections that are not [Panther-managed](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/detections/panther-managed). {% hint style="info" %} This consolidated user interface for viewing and editing detections is in open beta starting with Panther version 1.74. Please share any bug reports and feature requests with your Panther support team. {% endhint %} 1. In the left-hand navigation bar of your Panther Console, click **Build > Detections**. 2. Click the name of a detection. 3. Scroll down to the **Test** section. 4. Within the **Unit Test** tile, locate the test you'd like to rename or delete. 5. To the right of the test's name, click the three dot icon.
In the Unit Test section, a test named "Reset by Company Admin" is shown. The three dot icon to its right has been selected, which has opened a menu with two options: Rename and Delete.
* If the detection is [Panther-managed](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/detections/panther-managed), its tests cannot be renamed or deleted. Instead of a three dot icon next to the name of each test, a Panther icon will appear.
The Unit Test section shows multiple test names, each with a Panther icon to its left.
6. Click **Rename** or **Delete**. * If renaming, enter the new name and press enter, or return, to save. * If deleting, a **Delete Test** confirmation modal will pop up. * Select **Confirm**. ### Test example * Click **Edit** in the upper right corner of the page. * Scroll down, below the **Rule Function** text editor, to the **Unit Test** text editor. Keeping with the previous example (in [Rules](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/detections/writing-and-editing-detections/rules)), let's write two tests for this detection: ```python def rule(event): return event.get('status') == 200 and 'admin-panel' in event.get('request') def title(event): return f"Successful admin panel login detected from {event.get('remoteAddr')}" def dedup(event): return event.get('remoteAddr') ``` `Name`: Successful admin-panel Logins `Test event should trigger an alert`: Yes ```javascript { "httpReferer": "https://domain1.com/?p=1", "httpUserAgent": "Chrome/80.0.3987.132 Safari/537.36", "remoteAddr": "180.76.15.143", "request": "GET /admin-panel/ HTTP/1.1", "status": 200, "time": "2019-02-06 00:00:38 +0000 UTC" } ``` `Name`: Errored requests to the access-policy page `Test event should trigger an alert`: No ```javascript { "httpReferer": "https://domain1.com/?p=1", "httpUserAgent": "Chrome/80.0.3987.132 Safari/537.36", "remoteAddr": "180.76.15.143", "request": "GET /access-policy/ HTTP/1.1", "status": 500, "time": "2019-02-06 00:00:38 +0000 UTC" } ``` {% hint style="info" %} Use as many combinations as you would like to ensure the highest reliability with your detections. {% endhint %} ## Mocks Panther's testing framework also allows for basic Python call mocking. Both policy and rule tests support unit test mocking. When writing a detection that requires an external API call, mocks can be utilized to mimic the server response in the unit tests without having to actually execute an API call. Mocks are defined by a given `Mock Name` and `Return Value` which respectively denote the name of the object to patch and the `string` to be returned when the patched object is invoked. Mocks are defined on the unit test level, allowing you to define different mocks for each unit test. ### How to use mocks {% tabs %} {% tab title="Panther Console" %} To add a mock to a unit test in the Console: 1. Within the **Unit Test** tile, locate the **Mock Testing** section, below the test event editor. 2. Add values for **Mock Name** and **Return Value**. 3. To test that your mock is behaving as expected, click **Run Test**.
In the Panther Console, the Unit Test section of a detection page is shown. Below the test event, there is a Mock Testing section. It contains fields for Mock Name and Return Value.
{% endtab %} {% tab title="Panther Developer Workflows" %} To configure this using a CI/CD workflow, add the `Mocks` key to your test case. The `Mocks` key is used to define a list of functions you want to mock, and the value that should be returned when that function is called. Multiple functions can be mocked in a single test. For example, if we have a rule test and want to mock the function `get_counter` to always return a `1` and the function `geoinfo_from_ip` to always return a specific set of geo IP info, we could write our unit test like this: ``` Tests: - Name: Test With Mock LogType: LogType.Custom ExpectedResult: true Mocks: - objectName: get_counter returnValue: 1 - objectName: geoinfo_from_ip returnValue: >- { "region": "UnitTestRegion", "city": "UnitTestCityNew", "country": "UnitTestCountry" } Log: { "hostName": "test-01.prod.acme.io", "user": "martin_smith", "eventTime": "June 22 5:50:52 PM" } ``` Mocking allows us to emulate network calls without requiring API keys or network access in our CI/CD pipeline, and without muddying the state of external tracking systems (such as the panther KV store). See the related documentation for more information on using [Panther Analysis Tool (PAT)](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/panther-developer-workflows/ci-cd/deployment-workflows/pat) and [CI/CD workflows](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/panther-developer-workflows/ci-cd/deployment-workflows/circle-ci). {% endtab %} {% endtabs %} Mocks are allowed on the **global** and **built-in** python namespaces, this includes: 1. Imported Modules and Functions 2. Global Variables 3. Built-in Python Functions 4. User-Defined Functions Python provides two distinct forms of importing, which can be mocked as such: * `import package` * Mock Name: `package` * `from package import module` * Mock Name: `module` ### Example mock This example is based on the [AWS Config Global Resources](https://github.com/panther-labs/panther-analysis/blob/cd220c87982011d4ad156c7daecd2857c358d154/policies/aws_config_policies/aws_config_global_resources.py) detection. The detection utilizes a global helper function `resource_lookup` from `panther_oss_helpers` which queries the `resources-api` and returns the resource attributes. However, the unit test should be able to be performed without any external API calls.
A Unit Test within a Detection's details page is shown. In the Mock Testing section, the fields for Mock Name and Return Value are blank.
This test fails as there is no corresponding resource mapping to the generic example data. #### Diving into the detection ```python # --- Snipped --- from panther_oss_helpers import resource_lookup # --- Snipped --- def policy(resource): # --- Snipped --- for recorder_name in resource.get("Recorders", []): recorder = resource_lookup(recorder_name) resource_records_global_resources = bool( deep_get(recorder, "RecordingGroup", "IncludeGlobalResourceTypes") and deep_get(recorder, "Status", "Recording") ) if resource_records_global_resources: return True return False # --- Snipped --- ``` The detection uses the `from panther_oss_helpers import resource_lookup` convention which means the mock should be defined for the `resource_lookup` function. Mocks provide a way to leverage real world data to test the detection logic. **The return value used**: ``` { "AccountId": "012345678910", "Name": "Default", "RecordingGroup": { "AllSupported": true, "IncludeGlobalResourceTypes": true, "ResourceTypes": null }, "Region": "us-east-1", "ResourceId": "012345678910:us-east-1:AWS.Config.Recorder", "ResourceType": "AWS.Config.Recorder", "RoleARN": "arn:aws:iam::012345678910:role/PantherAWSConfig", "Status": { "LastErrorCode": null, "LastErrorMessage": null, "LastStartTime": "2018-10-05T22:45:01.838Z", "LastStatus": "SUCCESS", "LastStatusChangeTime": "2021-05-28T17:45:14.916Z", "LastStopTime": null, "Name": "Default", "Recording": true }, "Tags": null, "TimeCreated": null } ```
A Unit Test within a Detection's details page is shown. The Mock Name field contains "resource_lookup" and the Return Value contains the value from the "Return Value Used" section in the documentation above this image.
While this resource should be compliant, the unit test fails. \ Detections that do not expect a `string` to be returned requires a small tweak for mocks. In order to get this unit test working as expected, the following modifications need to be made: ```python # --- Snipped --- import json # Another option is to use: from ast import literal_eval # --- Snipped --- def policy(resource): # --- Snipped --- recorder = resource_lookup(recorder_name) if isinstance(recorder, str): recorder = json.loads(recorder) # --- Snipped --- ``` Once this modification is added, you can now test the detection logic with real data!
A full Policy Function and Unit Test with Mock Testing is shown. The test is named "Global Recorders Present - None Recording Global Resources"
#### Mocks from the CLI Unit test mocking is also supported with CLI based workflows for writing detections. For details on adding unit test mocks to a CLI based detection, see the [unit test mocking](#how-to-use-mocks) section of the Panther Analysis Tool docs. ## Enrich test data If you have [Lookup Table(s)](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/enrichment/lookup-tables) configured and created a detection to leverage them, click **Enrich Test Data** in the upper right side of the JSON event editor to ensure that `p_enrichment` is populated correctly.
Within the Unit Test section, the test event editor is shown. In the top right corner of the editor is a circled button that reads Enrich Test Data.