Custom Lookup Tables

Perhaps you'd like to store user data in a Lookup Table so that incoming log events associated with a certain user are enriched with additional personal information. You'd like to match on the user's email address, which means the email field will be the primary key in the Lookup Table and the selector in the log events.

You are deciding whether the primary key column in your Lookup Table should be of type string or string array. First, review the below two events you might expect to receive from your log source:

# Incoming log event one
{
    "actor_email": "[email protected]",
    "action": "LOGIN"
}

# Incoming log event two
{
    "actor_email": "[email protected]",
    "action": "EXPORT_FILE"
}

Note that the two email addresses ([email protected] and [email protected]) belong to the same user, Jane Doe.

When Panther receives these events, you would like to use a Lookup Table to enrich each of them with Jane's full name and role. After enrichment, these events would look like the following:

# Log event one after enrichment
{
    "actor_email": "[email protected]",
    "action": "LOGIN",
    "p_enrichment": {
        "<lookup_table_name>": {
            "actor_email": {
                "full_name": "Jane Doe",
                "p_match":  "[email protected]",
                "role": "ADMIN"
            }
        }
    }
}

# Log event two after enrichment
{
    "actor_email": "[email protected]",
    "action": "EXPORT_FILE",
    "p_enrichment": {
        "<lookup_table_name>": {
            "actor_email": {
                "full_name": "Jane Doe",
                "p_match": "[email protected]",
                "role": "ADMIN"
            }
        }
    }
}

You can accomplish this enrichment by defining a Lookup Table with either:

• (Recommended) A primary key column that is of type array of strings

• A primary key column that is of type string

Using a Lookup Table with a primary key column that is of type array of strings, you can include Jane's multiple email addresses in one primary key entry, associated to one row of data. This might look like the following:

Alternatively, you can define a Lookup Table with a primary key column that is of type string. However, because the match between the event and Lookup Table is made on the user's email address, and a user can have multiple email addresses (as is shown in Jane's case), you must duplicate the Lookup Table row for each email. This would look like the following:

While both options yield the same result (i.e., log events are enriched in the same way), defining a Lookup Table with an array of strings primary key is recommended for its convenience and reduced proneness to maintenance error.

1. In the left-hand navigation bar in your Panther Console, click Configure > Lookup Tables.

2. In the upper-right corner, click Create New.

3. On the Basic Information page:

   • Enter a descriptive Lookup Name.

   • Enter a Description (optional) and a Reference (optional). Description is meant for content about the table, while Reference can be used to hyperlink to an internal resource.

   • Next to Enabled? toggle the setting to Yes. Note: This is required to import your data later in this process.

4. Click Continue.

5. On the Associated Log Types page, optionally designate log types/Selectors:

   • Click Add Log Type.

   • Click the Log Type dropdown, then select a log type.

   • Choose one or more Selectors, the foreign key fields from the log type you want enriched with your Lookup Table.

     • You also can reference attributes in nested objects using JSON path syntax. For example, if you wanted to reference a field in a map, you could enter $.field.subfield.

   • Click Add Log Type to add another if needed. In the example screen shot above, we selected AWS.CloudTrail logs and typed in accountID and recipientAccountID to represent keys in the CloudTrail logs.

6. Click Continue.

7. Configure the Table Schema. Note: If you have not already created a new schema, please see our documentation on creating schemas. You can also use your Lookup Table data to infer a schema. Once you have created a schema, you will be able to choose it from the dropdown on the Table Schema page while configuring a Lookup Table. Note: CSV schemas require column headers to work with Lookup Tables.

   • Select a Schema Name from the dropdown.

   • Select a Primary Key Name from the dropdown. This should be a unique column on the table, such as accountID.

8. Click Continue.

9. Drag and drop a file or click Select File to choose the file of your Lookup Table data to import. The file must be in .csv or .json format.

10. Click Finish Setup. A source setup success page will populate.

11. Optionally, next to to Set an alarm in case this lookup table doesn't receive any data?, toggle the setting to YES to enable an alarm.

    • Fill in the Number and Period fields to indicate how often Panther should send you this notification.

    • The alert destinations for this alarm are displayed at the bottom of the page. To configure and customize where your notification is sent, see documentation on Panther Destinations.

    ​ ​

    Note: Notifications generated for a Lookup Table upload failing are accessible in the System Errors tab within the Alerts & Errors page in the Panther Console.

Once finished, you should be returned to the Lookup Table overview screen. Ensure that your new Lookup Table is listed.

File setup

A Lookup Table requires the following files:

• A YAML specification file containing the configuration for the table

• A YAML file defining the schema to use when loading data into the table

• A JSON or CSV file containing data to load into the table (optional, read further).

Folder setup

All files related to your Lookup Tables must be stored in a folder with a name containing lookup_tables. This could be a top-level lookup_tables directory, or sub-directories with names matching *lookup_tables*. You can use the Panther Analysis repo as a reference.

Writing the configuration files

It's usually prudent to begin writing the schema config first, because the table config will reference some of those values.

1. Create a YAML file for the schema, and save it with the rest of your custom schemas, outside the lookup_tables directory (for example, /schemas in the root of your panther analysis repo). This schema defines how to read the files you'll use to upload data to the table. If using a CSV file for data, then the schema should be able to parse CSV. The table schema is formatted the same as a log schema. For more information on writing schemas, read our documentation around managing Log Schemas.

2. Next, create a YAML file for the table configuration. For a Lookup Table with data stored in a local file, an example configuration would look like:

   Copy

   AnalysisType: lookup_table
   LookupName: my_lookup_table # A unique display name
   Schema: Custom.MyTableSchema # The schema defined in the previous step
   Filename: ./my_lookup_table_data.csv # Relative path to data
   Description: >
     A handy description of what information this table contains.
     For example, this table might convert IP addresses to hostnames
   Reference: >
     A URL to some additional documentation around this table
   Enabled: true # Set to false to stop using the table
   LogTypeMap:
     PrimaryKey: ip                # The primary key of the table
     AssociatedLogTypes:           # A list of log types to match this table to
       - LogType: AWS.CloudTrail
         Selectors:
           - "sourceIPAddress"     # A field in CloudTrail logs
           - "p_any_ip_addresses"  # A panther-generated field works too
       - LogType: Okta.SystemLog
         Selectors:
           - "$.client.ipAddress"  # Paths to JSON values are allowed

3. Upload the schema file by running the update schemas command from the repository root: panther_analysis_tool update-custom-schemas --path ./schemas

4. Finally, from the root of the repo, upload the lookup table: panther_analysis_tool upload

Update Lookup Tables via Panther Analysis Tool:

1. Locate the YAML configuration file for the Lookup Table in question.

2. Open the file, and look for the field Filename. You should see a file path which leads to the data file.

3. Update or replace the file indicated in Filename.

4. Push your changes to Panther with the following code:

   Copy

   panther_analysis_tool upload

   Optionally, you can specify only to upload the Lookup Table:

   Copy

   panther_analysis_tool upload --filter AnalysisType=lookup_table

1. In the left-hand navigation bar in your Panther Console, click Configure > Lookup Tables.

2. In the upper-right corner, click Create New.

3. On the Basic Information page:

   • Enter a descriptive Lookup Name.

   • Enter a Description (optional) and a Reference (optional).

     • Description is meant for content about the table, while Reference can be used to hyperlink to an internal resource.

   • Make sure the Enabled? toggle is set to Yes.

     • Note: This is required to import your data later in this process.

   

4. Click Continue.

5. On the Associated Log Types page, optionally designate log types/Selectors:

   • Click Add Log Type.

   • Click the Log Type dropdown, then select a log type.

   • Choose one or more Selectors, the foreign key fields form the log type you want enriched with your Lookup Table.

     • You also can reference attributes in nested objects using JSON path syntax. For example, if you wanted to reference a field in a map, you could enter $.field.subfield.

   • Click Add Log Type to add another if needed. In the example screen shot above, we selected AWS.VPCFlow logs and typed in account to represent keys in the VPC Flow logs.

6. Click Continue.

7. Configure the Table Schema. Note: If you have not already created a new schema, please see our documentation on creating schemas. Once you have created a schema, you will be able to select it from the dropdown on the Table Schema page while configuring a Lookup Table.

   1. Select a Schema Name from the dropdown.

   2. Select a Primary Key Name from the dropdown. This should be a unique column on the table, such as accountID.

8. Click Continue.

9. On the "Choose Import Method" page, click Set Up next to "Sync Data from an S3 Bucket."

10. Set up your S3 source. Note that your data must be in .csv or .json format.

    • Enter the Account ID, the 12-digit AWS Account ID where the S3 bucket is located.

    • Enter the S3 URI, the unique path that identifies the specific S3 bucket.

    • Optionally, enter the KMS Key if your data is encrypted using KMS-SSE.

    • Enter the Update Period, the cadence your S3 source gets updated (defaulted to 1 hour).

11. Click Continue.

12. Set up an IAM Role.

    • Please see the next section, Creating an IAM Role, for instructions on the three options available to do this.

13. Click Finish Setup. A source setup success page will populate.

14. Optionally, next to to Set an alarm in case this lookup table doesn't receive any data?, toggle the setting to YES to enable an alarm.

    • Fill in the Number and Period fields to indicate how often Panther should send you this notification.

    • The alert destinations for this alarm are displayed at the bottom of the page. To configure and customize where your notification is sent, see documentation on Panther Destinations.

​​

Note: Notifications generated for a Lookup Table upload failing are accessible in the System Errors tab within the Alerts & Errors page in the Panther Console.

Creating an IAM Role

There are three options for creating an IAM Role to use with your Panther Lookup Table using an S3 source:

• Create an IAM role using AWS Console UI.

• Create an IAM role using CloudFormation Template File.

• Create an IAM role manually.

Create an IAM role using AWS Console UI

1. On the "Set Up an IAM role" page, during the process of creating a Lookup Table with an S3 source, locate the tile labeled "Using the AWS Console UI". On the right side of the tile, click Select.

2. Click Launch Console UI.

   • You will be redirected to the AWS console in a new browser tab, with the template URL pre-filled.

   • The CloudFormation stack will create an AWS IAM role with the minimum required permissions to read objects from your S3 bucket.

   • Click the "Outputs" tab of the CloudFormation stack in AWS, and note the Role ARN.

3. Navigate back to your Panther account.

4. On the "Use AWS UI to set up your role" page, enter the Role ARN.

5. Click Finish Setup.

Create an IAM role using CloudFormation Template File

1. On the "Set Up an IAM role" page, during the process of creating a Lookup Table with an S3 source, locate the tile labeled "CloudFormation Template File". On the right side of the tile, click Select.

2. Click CloudFormation template, which downloads the template to apply it through your own pipeline.

3. Upload the template file in AWS:

   1. Open your AWS console and navigate to the CloudFormation product.

   2. Click Create stack.

   3. Click Upload a template file and select the CloudFormation template you downloaded.

4. On the "CloudFormation Template" page in Panther, enter the Role ARN.

5. Click Finish Setup.

Create an IAM role manually

1. On the "Set Up an IAM role" page, during the process of creating a Lookup Table with an S3 source, click the link that says I want to set everything up on my own.

2. Create the required IAM role. You may create the required IAM role manually or through your own automation. The role must be named using the format PantherLUTsRole-${Suffix}(e.g., PantherLUTsRole-MyLookupTable).

   • The IAM role policy must include the statements defined below:

     Copy

         "Version": "2012-10-17",
         "Statement": [
             {
                 "Action": "s3:GetBucketLocation",
                 "Resource": "arn:aws:s3:::<bucket-name>",
                 "Effect": "Allow"
             },
             {
                 "Action": "s3:GetObject",
                 "Resource": "arn:aws:s3:::<bucket-name>/<input-file-path>",
                 "Effect": "Allow"
             }
         ]
     }

   • If your S3 bucket is configured with server-side encryption using AWS KMS, you must include an additional statement granting the Panther API access to the corresponding KMS key. In this case, the policy will look something like this:

     Copy

         "Version": "2012-10-17",
         "Statement": [
             {
                 "Action": "s3:GetBucketLocation",
                 "Resource": "arn:aws:s3:::<bucket-name>",
                 "Effect": "Allow"
             },
             {
                 "Action": "s3:GetObject",
                 "Resource": "arn:aws:s3:::<bucket-name>/<input-file-path>",
                 "Effect": "Allow"
             },
             {
                 "Action": ["kms:Decrypt", "kms:DescribeKey"],
                 "Resource": "arn:aws:kms:<region>:<your-accound-id>:key/<kms-key-id>",
                 "Effect": "Allow"
             }
         ]
     }

3. On the "Setting up role manually" page in Panther, enter the Role ARN.

   • This can be found in the "Outputs" tab of the CloudFormation stack in your AWS account.

4. Click Finish Setup, and you will be redirected to the Lookup Tables list page with your new Employee Directory table listed.

File setup

A Lookup Table requires the following files:

• A YAML specification file containing the configuration for the table

• A YAML file defining the schema to use when loading data into the table

• A JSON or CSV file containing data to load into the table (optional, read further).

Folder setup

All files related to your Lookup Tables must be stored in a folder with a name containing lookup_tables. This could be a top-level lookup_tables directory, or sub-directories with names matching *lookup_tables*. You can use the Panther Analysis repo as a reference.

Writing the configuration files

It's usually prudent to begin writing the schema config first, because the table config will reference some of those values.

1. Create a YAML file for the schema, and save it in the lookup table directory (for example, lookup_tables/my_table/my_table_schema.yml). This schema defines how to read the files you'll use to upload data to the table. If using a CSV file for data, then the schema should be able to parse CSV. The table schema is formatted the same as a log schema. For more information on writing schemas, read our documentation around managing Log Schemas.

2. Next, create a YAML file for the table configuration. For a Lookup Table with data stored in a file in S3, an example configuration would look like this:

   Copy

   AnalysisType: lookup_table
   LookupName: my_lookup_table # A unique display name
   Schema: Custom.MyTableSchema # The schema defined in the previous step
   Refresh:
     RoleArn: arn:aws:iam::123456789012:role/PantherLUTsRole-my_lookup_table # A role in your organization's AWS account
     ObjectPath: s3://path/to/my_lookup_table_data.csv
     PeriodMinutes: 120 # Sync from S3 every 2 hours
   Description: >
     A handy description of what information this table contains.
     For example, this table might convert IP addresses to hostnames
   Reference: >
     A URL to some additional documentation around this table
   Enabled: true # Set to false to stop using the table
   LogTypeMap:
     PrimaryKey: ip                # The primary key of the table
     AssociatedLogTypes:           # A list of log types to match this table to
       - LogType: AWS.CloudTrail
         Selectors:
           - "sourceIPAddress"     # A field in CloudTrail logs
           - "p_any_ip_addresses"  # A panther-generated field works too
       - LogType: Okta.SystemLog
         Selectors:
           - "$.client.ipAddress"  # Paths to JSON values are allowed

1. Finally, from the root of the repo, upload the lookup table: panther_analysis_tool upload

2. Upload the schema file by running the update schemas command from the repository root: panther_analysis_tool update-custom-schemas --path ./schemas

Prerequisites

Before you can configure your Lookup Table to sync with S3, you'll need to have the following ready:

1. The ARN of an IAM role in AWS, which Panther can use to access the S3 bucket. For more information on setting up an IAM role for Panther, see the section on Creating an IAM Role.

2. The path to the file you intend to store data in. The path should be of the following format: s3://bucket-name/path_to_file/file.csv

Configuring Lookup Table for S3

1. Navigate to the YAML specification file for this Lookup Table.

2. In the file, locate (or add) the Refresh field.

3. Specify the RoleARN, ObjectPath, and PeriodMinutes fields. For specs on the allowed values, see our Lookup Table Config File Specification.

4. Save the config file, then upload your changes with panther_analysis_tool.​