Knowledge BaseCommunityRelease NotesTraining VideosRequest Demo
Search
⌃K
Links

Cribl Onboarding Guide

How to redact sensitive log data using AWS S3, Cribl, and Panther

Overview

The process outlined in this guide outlines how to redact sensitive log data using Cribl Cloud or Cribl Self-Hosted, working specifically within Amazon Web Services (AWS) S3 buckets and Panther. Broadly we will:
  • Authenticate an S3 Bucket in Cribl, with examples of the manual method and Event Breakers.
  • Configure an AWS S3 source and destination using a JSON data format.
  • Provide examples of JSON events, using pre and post-Cribl masks.

How to redact sensitive fields with Cribl

Step 1: Configuring the AWS S3 Source in Cribl

  1. 1.
    Log into Cribl. On the left-hand side, navigate to Groups. Then at the top of the page click the Routing drop down and select Quick Connect.
    In Cribl, the groups icon on the left is highlighted. The Routing tab's dropdown menu is displayed, showing the options "Data Routes" and "QuickConnect."
    ​
  2. 2.
    Click Add Source, then select Amazon S3 from the list of sources.
    The image shows a screen with the header "QuickConnect." The option "Unconnected" is selected. Beneath that, there is a button labeled "Add Source."
    ​
  3. 3.
    Under Configure > General Settings, enter a unique Input ID, the ARN of the SQS Queue, and optionally, additional filters.
    The image shows Cribl's Configure > General Settings page. It includes fields for Input ID, Queue, Filename Filter, Region, and Tags.
    ​
  4. 4.
    Click Authentication on the left. Click to choose one of the following authentication methods:
    • Manual - Requires Access and Secret Keys (You will need to generate the Access and Secret keys in AWS)
    • Secret - Secret key pair (You will need to generate this key pair in AWS)
    • Auto - Requires AWS Account ID and an ARN to be configured in the Assume Role section. This role should have access to S3 and/or SQS.
      The Configure section of Cribl is displayed. On the left sidebar, "Authentication" is highlighted. The Authentication Method is set to manual. There are fields for Access Key and Secret Key.
       In the example screen shot above, "Manual" is selected.
  5. 5.
    Click Event Breakers on the left. Choose the Cribl event breaker.
    • Note: This step allows Cribl to parse the incoming JSON data.

Step 2: Configuring the AWS S3 Destination in Cribl

  1. 1.
    Navigate back to the QuickConnect page. Click Add Destination and choose Amazon S3.
    In QuickConnect under "Sources", Amazon S3 is chosen.
    ​
  2. 2.
    In the General Settings page, enter the following:
    • S3 Bucket name. In AWS, these S3 buckets can be the same. Objects within the bucket can be differentiated by the prefix filter if needed.
    • Bucket region
    • Prefix for the outputted JSON files
    • File Name Prefix Expression
      The General Settings page for the S3 source is displayed. It contains fields for Output ID, S3 Bucket Name, Region, Staging Location, Key Prefix, Partitioning Expression, Data Format, File Name Prefix Expression, File Name Suffix Expression, Compress, Backpressure behavior, and Tags.
  3. 3.
    Click Authentication on the left. Click to choose one of the following authentication methods:
    • Manual - Requires Access and Secret Keys (You will need to generate the Access and Secret keys in AWS)
    • Secret - Secret key pair (You will need to generate this key pair in AWS)
    • Auto - Requires AWS Account ID and an ARN to be configured in the Assume Role section. This role should have access to S3 and/or SQS.
      The Configure section of Cribl is displayed. On the left sidebar, "Authentication" is highlighted. The Authentication Method is set to manual. There are fields for Access Key and Secret Key.
       In the example screen shot above, "Manual" is selected.

Step 3: Configuring the Pipeline in Cribl

  1. 1.
    Click the dotted line that connects the source and destination:
    The sources page in Cribl is displayed. At the bottom of the screen, there is a popup dialog labeled "Connection Configuration." "Pipeline" is selected.
    ​
    • You will be redirected to the Connection Configuration dashboard.
  2. 2.
    In the upper right side of the Connection Configuration dashboard, click + Pipeline. In the drop-down menu that appears, click Create Pipeline.
    In the Connection Configuration dashboard, in the upper right, the "Pipeline" dropdown menu is expanded.
    ​
  3. 3.
    In the "Create New Pipeline" popup dialog that appears, enter the name redacted into the ID field. Optionally enter an Async Function Timeout and a Description, then click Save.
    The "Create New Pipeline" form is open. There are fields are ID, Async Function Timeout (ms), and Description.
    ​
  4. 4.
    On the Processing / Pipelines page, click the blue gear icon.
    The "Processing / Pipelines" tab at the top is seleted. There is a red arrow pointing to a gear icon in the upper right.
    ​
  5. 5.
    In the upper right corner, click Edit as JSON.
    The "Processing / Pipelines" tab is selected at the top. In the upper right, there is an "Edit as JSON" link.
    ​
  6. 6.
    Paste in the block of JSON below. Note that the block of JSON below has two filters:
    • Mask: Replaces everything in the name field with REDACTED
    • Eval: Removes the fields _raw cribl_breaker crible_pipe _time
    {
    "id": "redaction",
    "conf": {
    "asyncFuncTimeout": 1000,
    "functions": [
    {
    "filter": "true",
    "conf": {
    "rules": [
    {
    "matchRegex": "/(.*)/i",
    "replaceExpr": "`REDACTED`"
    }
    ],
    "fields": [
    "name"
    ],
    "depth": 5,
    "flags": []
    },
    "id": "mask",
    "description": "Masking Filter",
    "final": false
    },
    {
    "filter": "true",
    "conf": {
    "remove": [
    "_raw",
    "cribl_breaker",
    "cribl_pipe",
    "_time"
    ]
    },
    "id": "eval",
    "final": true
    }
    ],
    "description": "redaction-pipeline",
    "groups": {}
    }
    }
  7. 7.
    Save the JSON.
Once the JSON has been saved, the UI will reflect your Masking Rules similar to the screenshot below:
The Processing / Pipelines page is open. Under "Masking Rules", the Match Regex value is /(.*) and the Replace Expression value is `REDACTED`.
The completed Data Route will look similar to the screen shot below:
The image shows the QuickConnect screen. On the left under Sources, "S3, cribl-source-s3" is listed. There is a dotted line connecting it to "S3 cribl-destination" under the Destinations header.

JSON Example Pre and Post Cribl

The following example shows a JSON event, using the mask filter mentioned in the previous steps of this guide. After using the mask filter, the name field Bella is replaced with REDACTED.
Pre-Cribl mask:
{ ... "name": "Bella", ...}
Post-Cribl mask:
{ ... "name":"REDACTED", ...}
We strongly advise that you verify the mask filter is working as expected after configuration. Once the mask filter is confirmed to be working correctly, ingest your masked logs into Panther using the S3 Data Transport.
Previous
Data Pipeline Tools
Next
Fluent Bit Onboarding Guide
Last modified 8mo ago