Knowledge BaseCommunityRelease NotesTraining VideosRequest Demo
Search
⌃K
Links

Azure Blob Storage Source

Onboarding Azure Blob Storage as a Data Transport log source in the Panther Console

Overview

With Azure Blob Storage as a log source, Panther can pull log data directly from your Azure container, allowing you to then write detections and perform investigations on this processed data.
Data can be sent compressed (or uncompressed). Learn more about compression specifications in Ingesting compressed data in Panther.

How to connect Azure Blob Storage as a log source in Panther

To ingest logs from Azure Blob Storage, you will first verify in Azure that certain resource providers are registered for your subscription. You'll begin setting up the source in Panther, then create necessary Azure infrastructure, either using a provided Terraform template or manually in the Azure Console.

Prerequisite

To create an Azure Blob Storage log source, you must first ensure that within your Azure subscription settings, Microsoft.EventGrid and Microsoft.Storage are registered resource providers. Verify this by following these steps:
  1. 1.
    In your Azure Console, navigate to Subscriptions.
  2. 2.
    Select the subscription you will be creating your Azure resources in.
  3. 3.
    Within the subscription settings, click Resource providers.
    In the Azure Console, the page of a Subscription called Azure subscription 1 is shown. There is a list of Resource providers, e.g., Microsoft.RecoveryServices and Microsoft.DBforMySQL. On the right is a column called Status.
    ​
  4. 4.
    In the Filter by name field, search for and locate Microsoft.EventGrid and Microsoft.Storage.
    • For each of these providers, ensure the Status column has a value of Registered.

Step 1: Begin source setup in Panther

  1. 1.
    In the left-hand navigation bar of your Panther Console, click Configure > Log Sources.
  2. 2.
    In the upper right corner, click Create New.
  3. 3.
    Click the Custom Log Formats tile.
  4. 4.
    In the Azure Blob Storage tile on the slide-out panel, click Start.
    In the Panther Console, the Custom Log Formats tile has been selected, and the slide-out panel shows a number of Data Transport options. An arrow is drawn to the Start button on the Azure Blob Storage option.
    ​
  5. 5.
    On the Basic Info page, fill in the following:
    • Name: Enter a descriptive name for your log source.
    • Logs Stream Type: Choose Lines, JSON, or JSON Array.
    • Log Types: Select one or more log types to associate with this log source.
      On the Basic Info page of the Create Azure Blob Storage source flow, there are three fields: Name, Logs Stream Type, and Log Types. At the bottom is a Setup button.
      ​
  6. 6.
    Click Setup.
    • Continue in Step 2: Create required Azure infrastructure.

Step 2: Create required Azure infrastructure

On the Infrastructure & Configuration page, you'll create required Azure infrastructure (either by using a Panther-provided Terraform template, or manually configuring resources in the Azure Console) and provide configuration values to Panther.
Panther-provided Terraform template
Manual setup in Azure Console

Using the Terraform template to create Azure infrastructure

After creating Azure resources using the Terraform template, Panther will ingest all logs written to any container in your created storage account. Ensure that the created Azure application has permission to read from each container.
  1. 1.
    Click Terraform Template to download the Terraform template.
    The Infrastructure & Configuration page shows 1. Create Required Infrastructure Component. There is a circled button labeled "Terraform Template"
    ​
  2. 2.
    If you do not already have the Azure CLI installed, install it by following Azure's How to install the Azure CLI documentation.
  3. 3.
    In a terminal, run az login.
  4. 4.
    Move the Terraform template to a new directory, and navigate to that directory.
  5. 5.
    Edit the panther.tfvars file to customize your deployment, e.g., by changing the region the infrastructure will be created in, and providing a custom storage account name.
  6. 6.
    Run the following Terraform commands to create the Azure resources:
    1. 1.
      terraform init
    2. 2.
      terraform apply -var-file="panther.tfvars"
  7. 7.
    After Terraform has finished creating the resources, copy the outputted values into the following fields in the Provide Azure configuration section of the Panther Console:
    • Tenant ID
    • Client ID
    • Storage Account Name
    • Storage Queue Name
    • Client Secret
      • The client secret value will be redacted in your terminal. To view it, run terraform output secret, and copy the value without quotation marks.
        • If you're running macOS, execute terraform output -raw secret | pbcopy to copy the value without printing it.
    ​
    This section of the source setup shows "2. Provide Azure configuration" and there are form fields for Tenant ID, Client ID, Storage Account Name, Storage Queue Name, and Client Secret.
    ​
  8. 8.
    Click Setup.
    • Continue in Step 3: Verify setup in Panther.

Manually creating infrastructure in the Azure Console

Step 1: Create resource group and storage account

  1. 1.
    In your Azure Console, navigate to Subscriptions.
  2. 2.
    Select the subscription you will be creating your Azure resources in.
  3. 3.
    Click Resource groups.
    1. 1.
      Click +Create.
      In the Azure console, the Azure subscription 1 Resource groups page is shown. A +Create button is circled.
      ​
    2. 2.
      Provide values for Name and Region.
      • Copy down or remember the value you provide for Name, as you'll need it later in this process.
    3. 3.
      Click Save.
  4. 4.
    Click the name of your newly created resource group.
  5. 5.
    In the search bar, enter "storage account" and within the Storage account tile that returns, click Create.
    In the Azure console, in the Marketplace, storage has been searched in a search bar. On a Storage account tile, the Create button is circled.
    ​
  6. 6.
    On the Create a storage account page, in the Instance details section, enter values for Storage account name and Region.
    1. 1.
      Click Review.
      On the Create a storage account page of the Azure console, there is a Project details section and a Instance details section. Within the latter, the following fields are circled: Storage account name and Region. A Review button at the bottom is circled.
      ​
    2. 2.
      Click Create.

Step 2: Add app registration and client secret

  1. 1.
    In the top search bar, search for "Active Directory" and click on Active Directory.
  2. 2.
    Click +Add, and in the dropdown menu that populates, App registration.
    In the Azure console, the Default Directory Overview page shows an +Add button. An arrow is drawn from +Add to an option in its dropdown, App registration.
    ​
    1. 1.
      Enter a Name.
    2. 2.
      Click Create.
    3. 3.
      Securely copy and store the Application (client) ID value, as you'll need it later in this process.
  3. 3.
    Click on your newly registered app.
  4. 4.
    On the right hand side, click Add a certificate or secret.
    In the Azure console, within an App called pantherapp3, an Add a certificate or secret button at the right side of the screen is circled.
    ​
  5. 5.
    Click +New Client Secret.
    1. 1.
      Provide a Description.
    2. 2.
      Click Add.
    3. 3.
      Securely copy and store the Client Secret value, as you'll need it later in this process.

Step 3: Create queue and add permission

  1. 1.
    Navigate to your newly created storage account.
  2. 2.
    In the left-hand navigation bar, select Queues.
    1. 1.
      Click +Queue to create a new queue.
      The panthertestacct3 Queues page shows an arrow drawn to the +Queue button. In the Queue name field is panthertestqueue3.
      ​
    2. 2.
      Enter a Name for the queue.
      • Copy down or remember the value you provide for Name, as you'll need it later in this process.
    3. 3.
      Click Ok.
  3. 3.
    Click on your newly created queue, then in the left-hand navigation bar, click Access Control (IAM).
    1. 1.
      Click +Add, then Add Role Assignment.
    2. 2.
      Search for "Storage Queue Data Message Processor" and select the matching role that populates.
    3. 3.
      Click on the Members tab.
    4. 4.
      Click +Select Members.
    5. 5.
      Search for the name of your registered app created in Step 2, and click Select.
    6. 6.
      Click Review+Assign.

Step 4: Create system topic and event subscription

  1. 1.
    In the top search bar, search for "Event Grid System Topic" and click on the matching page that populates.
    1. 1.
      Click +Create.
    2. 2.
      On the Create Event Grid System Topic page, fill in the following fields:
      • Topic Types: Select Storage Accounts (Blob & GPv2).
      • Subscription: Select the subscription you created your resource group in during Step 1.
      • Resource Group: Select the resource group you created in Step 1.
      • Resource: Select the storage account you created in Step 1.
      • Name: Enter a descriptive topic name.
        In the Azure Console's Create Event Grid System Topic page, there are various fields under a Topic Details header: Topic Types, Subscription, Resource Group, Resource, and under a System Topic Details header: Name and Location
        ​
    3. 3.
      Click Create.
  2. 2.
    Navigate back to your storage account.
  3. 3.
    In the left-hand navigation bar, click Events then +Event Subscription.
  4. 4.
    On the Create Event Subscription page, provide values for the following fields:
    1. 1.
      In the Event Subscription Details section, enter a Name.
    2. 2.
      In the Event Types section, for the Filter to Event Types field, leave the default options selected.
    3. 3.
      In the Endpoint Details section, make the following selections:
      • Endpoint Type: Select Storage Queues.
      • Endpoint: Select the queue you created in Step 3.
      ​
      On the Create Event Subscription page of the Azure console, various fields have been circled: Name, Filter to Event Types, Endpoint Type, and Endpoint.
      ​
    4. 4.
      Click Create.

Step 5: Create container and add permission

If you already have a container created, you only need to grant read permissions to the application you created in Step 2. In the instructions set below, start with Step 3.
Note that you will not need to provide information about this container to Panther, as all logs written to any container in your created storage account will be ingested.
  1. 1.
    Navigate to your newly created storage account.
  2. 2.
    In the left-hand navigation bar, select Containers.
    1. 1.
      Click +Container to create a new container.
      On the panthertestacct3 storage account's Containers page, an arrow is drawn to the +Container button.
      ​
    2. 2.
      Enter a Name for the container.
      • Copy down or remember the value you provide for Name, as you'll need it later in this process.
    3. 3.
      Click Create.
  3. 3.
    Click on your newly created container, then in the left-hand navigation bar, click Access Control (IAM).
    1. 1.
      Click +Add.
      In the panthertestcontainer3 Access Control (IAM) page, an arrow is drawn to the +Add button
      ​
    2. 2.
      Click Add Role Assignment.
    3. 3.
      Search for "Storage Blob Data Reader" and select the matching role that populates.
      In the Add role assignment page of the Azure console, "storage blob" has been searched for in the search box. One of the results, Storage Blob Data Reader, is circled.
      ​
    4. 4.
      Click on the Members tab.
    5. 5.
      Click +Select Members.
    6. 6.
      Search for the name of the registered app you created in Step 2, and click Select.
    7. 7.
      Click Review+Assign.

Step 6: Copy Azure configuration values back into the Panther Console

  1. 1.
    Return to the Infrastructure & Configuration page in your Panther Console.
  2. 2.
    In the Provide Azure configuration section, copy in values for the following fields:
    • Tenant ID: This value can be found on your Azure Console's Active Directory home page.
    • Client ID: The application (client) ID generated in Step 2.
    • Storage Account Name: The name you gave your storage account in Step 1.
    • Storage Queue Name: The name you gave your queue in Step 3.
    • Client Secret: The client secret value generated in Step 2.
      This section of the source setup shows "2. Provide Azure configuration" and there are form fields for Tenant ID, Client ID, Storage Account Name, Storage Queue Name, and Client Secret.
      ​
  3. 3.
    Click Setup.
    • Continue in Step 3: Verify setup in Panther.

Step 3: Verify setup in Panther

You will be directed to a success screen:
The success screen reads, "Everything looks good! Panther will now automatically pull & process logs from your account"
  • You can optionally enable one or more Detection Packs.
  • The Trigger an alert when no events are processed setting defaults to YES. We recommend leaving this enabled, as you will be alerted if data stops flowing from the log source after a certain period of time. The timeframe is configurable, with a default of 24 hours.
    The "Trigger an alert when no events are processed" toggle is set to YES. The "How long should Panther wait before it sends you an alert that no events have been processed" setting is set to 1 Day
Previous
Pub/Sub Source
Next
Monitoring Log Sources
Last modified 3d ago