Links

Testing

Overview

Detection Testing ensures that detections will behave as expected and will generate alerts once deployed and running. Testing also promotes reliability over time as code evolves and protects against regressions.
Testing works by defining a "test" input to be used as the event for the detection and expecting whether or not an alert would be generated. A Detection can have zero or many test cases and it's recommended to have at least two (one false positive and one true positive).

Using tests

Tests can be defined either in the Panther Console or programmatically with the Panther Analysis Tool.
Panther Console
Panther Developer Workflows
  1. 1.
    Log in to the Panther Console.
  2. 2.
    In the left sidebar, click Build > Detections.
  3. 3.
    Click into a Detection or create a new Detection.
  4. 4.
    Click the Functions & Tests tab.
  5. 5.
    Click Edit in the upper right corner of the page.
  6. 6.
    Scroll down below the Rule Function text editor to the testing text editor.
  7. 7.
    In the upper right corner of the testing text editor, click {} Create Test.
  8. 8.
    Enter a descriptive name for the test, and select whether the test event should trigger an alert.
  9. 9.
    Compose a sample case in the text editor. When you are finished, click Run Test.
Rules
In your spec file, add the Tests key with sample cases:
Tests:
-
Name: Name to describe our first test
LogType: LogType.GoesHere
ExpectedResult: true or false
Log:
{
"hostName": "test-01.prod.acme.io",
"user": "martin_smith",
"eventTime": "June 22 5:50:52 PM"
}
We recommend running as many test cases as possible, including both true and false positives.
Policies
In the spec file, add the following Tests key:
Tests:
-
Name: Name to describe our first test.
ResourceType: AWS.S3.Bucket
ExpectedResult: true
Resource:
{
"PublicAccessBlockConfiguration": null,
"Region": "us-east-1",
"Policy": null,
"AccountId": "123456789012",
"LoggingPolicy": {
"TargetBucket": "access-logs-us-east-1-100",
"TargetGrants": null,
"TargetPrefix": "acmecorp-fiancial-data/"
},
"EncryptionRules": [
{
"ApplyServerSideEncryptionByDefault": {
"SSEAlgorithm": "AES256",
"KMSMasterKeyID": null
}
}
],
"Arn": "arn:aws:s3:::acmecorp-fiancial-data",
"Name": "acmecorp-fiancial-data",
"LifecycleRules": null,
"ResourceType": "AWS.S3.Bucket",
"Grants": [
{
"Permission": "FULL_CONTROL",
"Grantee": {
"URI": null,
"EmailAddress": null,
"DisplayName": "admins",
"Type": "CanonicalUser",
"ID": "013ae1034i130431431"
}
}
],
"Versioning": "Enabled",
"ResourceId": "arn:aws:s3:::acmecorp-fiancial-data",
"Tags": {
"aws:cloudformation:logical-id": "FinancialDataBucket"
},
"Owner": {
"ID": "013ae1034i130431431",
"DisplayName": "admins"
},
"TimeCreated": "2020-06-13T17:16:36.000Z",
"ObjectLockConfiguration": null,
"MFADelete": null
}
The value of Resource can be a JSON object copied directly from the Policies > Resources explorer.

Test example

Keeping with the previous example (in Rules), let's write two tests for this detection:
def rule(event):
return event.get('status') == 200 and 'admin-panel' in event.get('request')
def title(event):
return f"Successful admin panel login detected from {event.get('remoteAddr')}"
def dedup(event):
return event.get('remoteAddr')
Name: Successful admin-panel Logins
Test event should trigger an alert: Yes
{
"httpReferer": "https://domain1.com/?p=1",
"httpUserAgent": "Chrome/80.0.3987.132 Safari/537.36",
"remoteAddr": "180.76.15.143",
"request": "GET /admin-panel/ HTTP/1.1",
"status": 200,
"time": "2019-02-06 00:00:38 +0000 UTC"
}
Name: Errored requests to the access-policy page
Test event should trigger an alert: No
{
"httpReferer": "https://domain1.com/?p=1",
"httpUserAgent": "Chrome/80.0.3987.132 Safari/537.36",
"remoteAddr": "180.76.15.143",
"request": "GET /access-policy/ HTTP/1.1",
"status": 500,
"time": "2019-02-06 00:00:38 +0000 UTC"
}
Use as many combinations as you would like to ensure the highest reliability with your detections.

Mocks

Panther's testing framework also allows for basic Python call mocking. Both policy and rule tests support unit test mocking.
When writing a detection that requires an external API call, mocks can be utilized to mimic the server response in the unit tests without having to actually execute an API call.
Mocks are defined by a given Mock Name and Return Value which respectively denote the name of the object to patch and the string to be returned when the patched object is invoked.

How to use Mocks

Panther Console
Panther Developer Workflows
Added mocks are defined on the unit test level allowing users to define different mocks for each unit test.
The section to add mocks to a unit test is located directly underneath the unit test resource definition:
To configure this using a CI/CD workflow, add the Mocks key to your test case. The Mocks key is used to define a list of functions you want to mock, and the value that should be returned when that function is called. Multiple functions can be mocked in a single test.
For example, if we have a rule test and want to mock the function get_counter to always return a 1 and the function geoinfo_from_ip to always return a specific set of geo IP info, we could write our unit test like this:
Tests:
-
Name: Test With Mock
LogType: LogType.Custom
ExpectedResult: true
Mocks:
- objectName: get_counter
returnValue: 1
- objectName: geoinfo_from_ip
returnValue: >-
{
"region": "UnitTestRegion",
"city": "UnitTestCityNew",
"country": "UnitTestCountry"
}
Log:
{
"hostName": "test-01.prod.acme.io",
"user": "martin_smith",
"eventTime": "June 22 5:50:52 PM"
}
Mocking allows us to emulate network calls without requiring API keys or network access in our CI/CD pipeline, and without muddying the state of external tracking systems (such as the panther KV store).
See the related documentation for more information on using Panther Analysis Tool (PAT) and CI/CD workflows.
Mocks are allowed on the global and built-in python namespaces, this includes:
  1. 1.
    Imported Modules and Functions
  2. 2.
    Global Variables
  3. 3.
    Built-in Python Functions
  4. 4.
    User-Defined Functions
Python provides two distinct forms of importing, which can be mocked as such:
  • import package
    • Mock Name: package
  • from package import module
    • Mock Name: module

Example Mock

This example is based on the AWS Config Global Resources detection.
The detection utilizes a global helper function resource_lookup from panther_oss_helpers which queries the resources-api and returns the resource attributes. However, the unit test should be able to be performed without any external API calls.
This test fails as there is no corresponding resource mapping to the generic example data.

Diving Into the Detection

# --- Snipped ---
from panther_oss_helpers import resource_lookup
# --- Snipped ---
def policy(resource):
# --- Snipped ---
for recorder_name in resource.get("Recorders", []):
recorder = resource_lookup(recorder_name)
resource_records_global_resources = bool(
deep_get(recorder, "RecordingGroup", "IncludeGlobalResourceTypes")
and deep_get(recorder, "Status", "Recording")
)
if resource_records_global_resources:
return True
return False
# --- Snipped ---
The detection uses the from panther_oss_helpers import resource_lookup convention which means the mock should be defined for the resource_lookup function.
Mocks provide a way to leverage real world data to test the detection logic.
The Return Value used:
{ "AccountId": "012345678910", "Name": "Default", "RecordingGroup": { "AllSupported": true, "IncludeGlobalResourceTypes": true, "ResourceTypes": null }, "Region": "us-east-1", "ResourceId": "012345678910:us-east-1:AWS.Config.Recorder", "ResourceType": "AWS.Config.Recorder", "RoleARN": "arn:aws:iam::012345678910:role/PantherAWSConfig", "Status": { "LastErrorCode": null, "LastErrorMessage": null, "LastStartTime": "2018-10-05T22:45:01.838Z", "LastStatus": "SUCCESS", "LastStatusChangeTime": "2021-05-28T17:45:14.916Z", "LastStopTime": null, "Name": "Default", "Recording": true }, "Tags": null, "TimeCreated": null }
While this resource should be compliant, the unit test fails. Detections that do not expect a string to be returned requires a small tweak for mocks.
In order to get this unit test working as expected, the following modifications need to be made:
# --- Snipped ---
import json
# Another option is to use: from ast import literal_eval
# --- Snipped ---
def policy(resource):
# --- Snipped ---
recorder = resource_lookup(recorder_name)
if isinstance(recorder, str):
recorder = json.loads(recorder)
# --- Snipped ---
Once this modification is added, you can now test the detection logic with real data!

Mocks from the CLI

Unit test mocking is also supported with CLI based workflows for writing detections. For details on adding unit test mocks to a CLI based detection, see the unit test mocking section of the Panther Analysis Tool docs.

Enrich Test Data

If you have Lookup Table(s) configured and created a detection to leverage them, click Enrich Test Data in the upper right side of the JSON event editor to ensure that p_enrichment is populated correctly.
This example is based on the Account Metadata Lookup Table scenario.