Testing
Use unit tests to ensure your detections are working as expected
Testing your detections ensures that once deployed, detections will behave as expected, generating alerts appropriately. Testing also promotes reliability and protects against regressions as code evolves over time.
Testing works by defining a test log event for a certain detection, and indicating whether or not you'd expect an alert to be generated when the test event is processed by that detection. Panther doesn't enforce a required minimum or maximum number of tests for each detection, but it's recommended to configure at least two—one false positive and one true positive.
You can create, edit, and delete tests for custom detections (i.e., those that aren't Panther-managed). Tests for Panther-managed detections are maintained by Panther and are read-only.
Panther's Data Replay, which allows you to run historical log data through a rule to preview the outcome, is also useful while testing.
You can create tests for detections that are not Panther-managed in the Panther Console, or in the CLI workflow with the Panther Analysis Tool (PAT).
Console: Detection page
Console: Alert event
CLI
- 1.In the left-hand navigation bar of your Panther Console, click Build > Detections.
- 2.
- 3.Scroll down to the Test section.
- 4.On the right-hand side of the Unit Test tile, click Add New.​
- 5.The newly created test is given a placeholder name. To its right, click the three dots icon and select Rename.​
- 6.Provide a meaningful name for the test, and press enter, or return, to save it.
- 7.Set the The detection should trigger based on the example event toggle to
YES
orNO
. - 8.Compose a test JSON event in the text editor.
- 9.To see whether your test passes, click Run Test.
- 10.When you are finished, in the upper-right corner of the page, click Update or Save.
In the Panther Console, you can create a new unit test directly from an alert, using an actual event associated with the alert. Follow the instructions in Add filters from an alert event, taking note of steps 9 and 10.
Note that this functionality is only available for rules.
Rules and scheduled rules
In your YAML file (in both Python and YAML detections), add a
Tests
key with the following fields:Tests:
-
Name: Name to describe our first test
LogType: LogType.GoesHere
ExpectedResult: true or false
Log:
{
"hostName": "test-01.prod.acme.io",
"user": "martin_smith",
"eventTime": "June 22 5:50:52 PM"
}
Policies
In your YAML file, add a
Tests
key. The value of Resource
can be a JSON object copied directly from the Policies > Resources explorer.Tests:
-
Name: Name to describe our first test.
ResourceType: AWS.S3.Bucket
ExpectedResult: true
Resource:
{
"PublicAccessBlockConfiguration": null,
"Region": "us-east-1",
"Policy": null,
"AccountId": "123456789012",
"LoggingPolicy": {
"TargetBucket": "access-logs-us-east-1-100",
"TargetGrants": null,
"TargetPrefix": "acmecorp-fiancial-data/"
},
"EncryptionRules": [
{
"ApplyServerSideEncryptionByDefault": {
"SSEAlgorithm": "AES256",
"KMSMasterKeyID": null
}
}
],
"Arn": "arn:aws:s3:::acmecorp-fiancial-data",
"Name": "acmecorp-fiancial-data",
"LifecycleRules": null,
"ResourceType": "AWS.S3.Bucket",
"Grants": [
{
"Permission": "FULL_CONTROL",
"Grantee": {
"URI": null,
"EmailAddress": null,
"DisplayName": "admins",
"Type": "CanonicalUser",
"ID": "013ae1034i130431431"
}
}
],
"Versioning": "Enabled",
"ResourceId": "arn:aws:s3:::acmecorp-fiancial-data",
"Tags": {
"aws:cloudformation:logical-id": "FinancialDataBucket"
},
"Owner": {
"ID": "013ae1034i130431431",
"DisplayName": "admins"
},
"TimeCreated": "2020-06-13T17:16:36.000Z",
"ObjectLockConfiguration": null,
"MFADelete": null
}
- 1.In the left-hand navigation bar of your Panther Console, click Build > Detections.
- 2.Click the name of a detection.
- 3.Scroll down to the Test section.
- 4.Within the Unit Test tile, locate the test you'd like to rename or delete.
- 5.To the right of the test's name, click the three dot icon.
- If the detection is Panther-managed, its tests cannot be renamed or deleted. Instead of a three dot icon next to the name of each test, a Panther icon will appear.
- 6.Click Rename or Delete.
- If renaming, enter the new name and press enter, or return, to save.
- If deleting, a Delete Test confirmation modal will pop up.
- Select Confirm.
- Click Edit in the upper right corner of the page.
- Scroll down, below the Rule Function text editor, to the Unit Test text editor.
Python
YAML
def rule(event):
return event.get('status') == 200 and 'admin-panel' in event.get('request')
​
def title(event):
return f"Successful admin panel login detected from {event.get('remoteAddr')}"
​
​
def dedup(event):
return event.get('remoteAddr')
Detection:
- KeyPath: status
Condition: Equals
Value: 200
- KeyPath: request
Condition: Contains
Value: 'admin-panel'
​
AlertTitle: "Successful admin panel login detected from {remoteAddr}"
​
GroupBy:
- KeyPath: remoteAddr
Name
: Successful admin-panel LoginsTest event should trigger an alert
: Yes{
"httpReferer": "https://domain1.com/?p=1",
"httpUserAgent": "Chrome/80.0.3987.132 Safari/537.36",
"remoteAddr": "180.76.15.143",
"request": "GET /admin-panel/ HTTP/1.1",
"status": 200,
"time": "2019-02-06 00:00:38 +0000 UTC"
}
Name
: Errored requests to the access-policy pageTest event should trigger an alert
: No{
"httpReferer": "https://domain1.com/?p=1",
"httpUserAgent": "Chrome/80.0.3987.132 Safari/537.36",
"remoteAddr": "180.76.15.143",
"request": "GET /access-policy/ HTTP/1.1",
"status": 500,
"time": "2019-02-06 00:00:38 +0000 UTC"
}
Use as many combinations as you would like to ensure the highest reliability with your detections.
Panther's testing framework also allows for basic Python call mocking. Both policy and rule tests support unit test mocking.
When writing a detection that requires an external API call, mocks can be utilized to mimic the server response in the unit tests without having to actually execute an API call.
Mocks are defined with a Mock Name and Return Value (or
objectName
and returnValue
, in the CLI workflow) which respectively denote the name of the object to patch and the string
to be returned when the patched object is invoked.Mocks are defined on the unit test level, allowing you to define different mocks for each unit test.
Console
CLI
To add a mock to a unit test in the Console:
- 1.Within the Unit Test tile, locate the Mock Testing section, below the test event editor.
- 2.Add values for Mock Name and Return Value.
- 3.To test that your mock is behaving as expected, click Run Test.
.png?alt=media&token=0f83aa5f-f743-4806-b3a0-d76da88649d5)
To configure this using a CI/CD workflow, add the
Mocks
key to your test case. The Mocks
key is used to define a list of functions you want to mock, and the value that should be returned when that function is called. Multiple functions can be mocked in a single test. For example, if we have a rule test and want to mock the function
get_counter
to always return a 1
and the function geoinfo_from_ip
to always return a specific set of geo IP info, we could write our unit test like this:Tests:
-
Name: Test With Mock
LogType: LogType.Custom
ExpectedResult: true
Mocks:
- objectName: get_counter
returnValue: 1
- objectName: geoinfo_from_ip
returnValue: >-
{
"region": "UnitTestRegion",
"city": "UnitTestCityNew",
"country": "UnitTestCountry"
}
Log:
{
"hostName": "test-01.prod.acme.io",
"user": "martin_smith",
"eventTime": "June 22 5:50:52 PM"
}
Mocking allows us to emulate network calls without requiring API keys or network access in our CI/CD pipeline, and without muddying the state of external tracking systems (such as the panther KV store).
See the related documentation for more information on using Panther Analysis Tool (PAT) and CI/CD workflows.
Mocks are allowed on the global and built-in Python namespaces, this includes:
- 1.Imported Modules and Functions
- 2.Global Variables
- 3.Built-in Python Functions
- 4.User-Defined Functions
Python provides two distinct forms of importing, which can be mocked as such:
import package
- Mock Name:
package
from package import module
- Mock Name:
module
The detection utilizes a global helper function
resource_lookup
from panther_oss_helpers
which queries the resources-api
and returns the resource attributes. However, the unit test should be able to be performed without any external API calls..png?alt=media&token=492bc94b-5d05-4213-8a10-aeda8566fe3f)
This test fails as there is no corresponding resource mapping to the generic example data.
# --- Snipped ---
from panther_oss_helpers import resource_lookup
# --- Snipped ---
def policy(resource):
# --- Snipped ---
for recorder_name in resource.get("Recorders", []):
recorder = resource_lookup(recorder_name)
resource_records_global_resources = bool(
deep_get(recorder, "RecordingGroup", "IncludeGlobalResourceTypes")
and deep_get(recorder, "Status", "Recording")
)
if resource_records_global_resources:
return True
return False
# --- Snipped ---
The detection uses the
from panther_oss_helpers import resource_lookup
convention which means the mock should be defined for the resource_lookup
function.Mocks provide a way to leverage real world data to test the detection logic.
The return value used:
{ "AccountId": "012345678910", "Name": "Default", "RecordingGroup": { "AllSupported": true, "IncludeGlobalResourceTypes": true, "ResourceTypes": null }, "Region": "us-east-1", "ResourceId": "012345678910:us-east-1:AWS.Config.Recorder", "ResourceType": "AWS.Config.Recorder", "RoleARN": "arn:aws:iam::012345678910:role/PantherAWSConfig", "Status": { "LastErrorCode": null, "LastErrorMessage": null, "LastStartTime": "2018-10-05T22:45:01.838Z", "LastStatus": "SUCCESS", "LastStatusChangeTime": "2021-05-28T17:45:14.916Z", "LastStopTime": null, "Name": "Default", "Recording": true }, "Tags": null, "TimeCreated": null }
.png?alt=media&token=02341de8-92ba-4a02-98e8-53cd835a2be4)
While this resource should be compliant, the unit test fails.
Detections that do not expect a
string
to be returned requires a small tweak for mocks.In order to get this unit test working as expected, the following modifications need to be made:
# --- Snipped ---
import json
# Another option is to use: from ast import literal_eval
# --- Snipped ---
def policy(resource):
# --- Snipped ---
recorder = resource_lookup(recorder_name)
if isinstance(recorder, str):
recorder = json.loads(recorder)
# --- Snipped ---
Once this modification is added, you can now test the detection logic with real data!
.png?alt=media&token=e0424f4c-acac-4998-805e-b33a160b8521)
Unit test mocking is also supported with CLI based workflows for writing detections. For details on adding unit test mocks to a CLI based detection, see the unit test mocking section of the PAT documentation.
It's possible to enrich test events in both the Panther Console and CLI workflow:
Console
CLI
If you have Lookup Table(s) configured and created a detection to leverage them, click Enrich Test Data in the upper right side of the JSON event editor to ensure that
p_enrichment
is populated correctly..png?alt=media&token=dd40f4f7-d9ec-4d4a-8982-5d6bb956b356)
Last modified 1mo ago