# Pub/Sub Source ## Overview Panther supports configuring [Google Pub/Sub](https://cloud.google.com/pubsub?hl=en) as a Data Transport to pull log data directly from Pub/Sub topics. Panther can authenticate against your source using Google Cloud [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) or a [service account](https://cloud.google.com/iam/docs/service-account-overview). Panther supports using [regional endpoints](https://cloud.google.com/pubsub/docs/reference/service_apis_overview#list_regional_endpoints), which may be helpful if your organization has compliance requirements (e.g., your data must stay in a specific region in Google Cloud), and you need to activate the [`enforceInTransit` flag](https://docs.cloud.google.com/pubsub/docs/resource-location-restriction). ## How to set up a Cloud Pub/Sub log source in Panther ### Step 1: Configure the Pub/Sub source in Panther 1. In the left-hand navigation bar of your Panther Console, click **Configure** > **Log Sources**. 2. In the upper-right corner, click **Create New**. 3. Click the **Google Cloud Pub/Sub** tile. 4. On the **Basic Info** page, fill in the fields: * **Name**: Enter a descriptive name for the log source. * **Log Types**: Select the log types Panther should use to parse your logs. 5. Click **Setup.** 6. On the **Log Format** page, select the [stream type](https://docs.panther.com/custom-log-types/reference#stream-type) of the incoming logs: * **Auto** * **Lines** * **JSON** * **JSON Array** * **XML** 7. Click **Continue**. 8. On the **Configuration** page, follow the steps to create the required infrastructure components. * The instructions below are based on using a Terraform template. If you do not want to use Terraform, you can follow our alternative documentation to [complete the infrastructure components process manually](#manual-creation-in-gcp-console). ### Step 2: Create the required infrastructure in your GCP cloud Before creating GCP infrastructure, you'll need to decide: * The authentication method: You can use [Google Cloud Workload Identity Federation with AWS](https://cloud.google.com/iam/docs/workload-identity-federation) or a [service account](https://cloud.google.com/iam/docs/service-account-overview)—see the top-level tabs below. * The creation method: You can use Terraform to create the infrastructure, or create it manually in the GCP console—see the sub-tabs within each top-level tab below. {% hint style="warning" %} When creating the subscription, it's recommended to not enable `enable_exactly_once_delivery`, as this can lead to much lower throughputs and larger latencies. {% endhint %} {% tabs %} {% tab title="Service account authentication" %} If you'd like Panther to authenticate using a [Google Cloud service account](https://cloud.google.com/iam/docs/service-account-overview), follow the instructions in one of the tabs below. {% tabs %} {% tab title="Terraform" %} To create GCP infrastructure using Terraform (authenticating with a service account): 1. On the **Infrastructure & Credentials** page, click the **Service Account** tab.\ ![Under an "Infrastructure & Credentials" header, a tab labeled "Service Account" is circled.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-86cf52ca25fab148f209f35c7689bf9fce571efe%2FScreenshot%202025-02-24%20at%202.37.30%E2%80%AFPM.png?alt=media) 2. Click **Terraform Template** to download the Terraform template. * You can also find the Terraform template at [this GitHub link](https://github.com/panther-labs/panther-auxiliary/tree/main/terraform/panther_cloud_pubsub_transport_type_infra). 3. Fill out the fields in the `panther.tfvars` file with your configuration. * Set `authentication_method` to `"service_account"`. 4. Initialize a working directory containing Terraform configuration files and run `terraform init`. 5. Copy the corresponding **Terraform Command** provided and run it in your CLI. 6. Generate a JSON key file by copying the **gcloud Command** provided, replacing the value for your service account email address, and running it in your CLI. * You can find the service account email in the output of the **Terraform Command**. {% endtab %} {% tab title="Manual creation in GCP console" %} To create the GCP infrastructure components manually in the GCP console (authenticating with a service account): 1. [Create a topic](https://cloud.google.com/pubsub/docs/admin#creating_a_topic) for the data. * You can create a topic using the `gcloud` CLI tool with the following command format: `gcloud pubsub topics create $TOPIC_ID` 2. [Create a subscription](https://cloud.google.com/pubsub/docs/admin#pubsub_create_pull_subscription-gcloud) to be used with the topic you created. * You can create a subscription using the `gcloud` CLI tool with the following command format: `gcloud pubsub subscriptions create $SUBSCRIPTION_ID --topic $TOPIC_ID --topic-project $PROJECT_ID` {% hint style="warning" %} This subscription should not be used by any other service or source. {% endhint %} 3. [Enable the IAM API](https://console.cloud.google.com/apis/library/iam.googleapis.com). 4. [Create a new Google Cloud service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts). To create the account using the `gcloud` CLI tool, use the following command format: ``` gcloud iam service-accounts create SA-NAME \     --description="DESCRIPTION" \     --display-name="DISPLAY_NAME" ``` * Make sure to take note of the account email address, as Panther will use this to access the infrastructure created for this GCS integration. 5. Assign the required IAM roles to the account. * The following permissions are required for the project where the Pub/Sub subscription and topic lives:
Permission requiredRoleCondition
Permissions requiredRoleScope
pubsub.subscriptions.consumeroles/pubsub.subscribersubscription-name
pubsub.subscriptions.getroles/pubsub.viewersubscription-name
monitoring.timeSeries.listroles/monitoring.viewerproject
* The `monitoring.timeSeries.list` permission is optional, but is recommended for improved autoscaling. * **Note:** You can set conditions or IAM policies on permissions for specific resources. This can be done either in the IAM page of the service account (as seen in the example screenshot below) or in the specific resource's page.\ ![On the IAM page of the service account, under the header "Grant this service account access to project," there are fields for Rule and Condition.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-a76ce6b753aefd1a1a5b99869bb6773442a905cf%2Fgcp-grant-access.png?alt=media) * **Note:** You can create the permissions using the `gcloud` CLI tool: * `gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/pubsub.subscriber"` * `gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/pubsub.viewer"` * `gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/monitoring.viewer"` 6. [Generate a JSON key file](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) for the service account, which will be used in Panther to authenticate to the GCP infrastructure. * To create a JSON key file using the gcloud CLI tool, run the following command format:\ `gcloud iam service-accounts keys create $KEYFILE_PATH --iam-account=$SERVICE_ACCOUNT_EMAIL` * Alternatively, you can run the above command in GCP's terminal instead of locally. 1. Click the 3 dots icon menu in the top right, then click **Download**. * ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-6940cc677e233faedec16959c99926def5e052be%2Fimage.png?alt=media) 2. Click the folder icon for Browse. 3. Navigate to the key file and select it, then click **Download**. {% endtab %} {% endtabs %} {% endtab %} {% tab title="Workload Identity Federation authentication" %} If you'd like Panther to authenticate using [Google Cloud Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation), follow the instructions in one of the tabs below. {% tabs %} {% tab title="Terraform" %} To create the GCP infrastructure components using Terraform (authenticating with Workload Identity Federation): 1. On the **Infrastructure & Credentials page**, click the **Workload Identity Federation** tab.\ ![Under an "Infrastructure & Credentials" header, a tab labeled "Workload Identity Federation" is circled.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-7e7ca651c6e60e46eebec337cf723e11dc602ae7%2FScreenshot%202025-02-24%20at%202.39.43%E2%80%AFPM.png?alt=media) 2. Click **Terraform Template** to download the Terraform template. * You can also find the Terraform template at [this GitHub link](https://github.com/panther-labs/panther-auxiliary/tree/main/terraform/panther_cloud_pubsub_transport_type_infra). 3. Fill out the fields in the `panther.tfvars` file with your configuration. * Set `authentication_method` to `"workload_identity_federation"`. * Provide values for `panther_workload_identity_pool_id`, `panther_workload_identity_pool_provider_id`**,** and `panther_aws_account_id`. 4. Initialize a working directory containing Terraform configuration files and run `terraform init`. 5. Copy the corresponding **Terraform Command** provided and run it in your CLI. 6. Generate a credential configuration file for the pool by copying the **gcloud Command** provided, replacing the value for the project number, pool ID, and provider ID, and running it in your CLI. * You can find the project number, the pool ID and the provider ID in the output of the **Terraform Command**. {% endtab %} {% tab title="Manual creation in GCP console" %} To create the GCP infrastructure components manually in the GCP console (authenticating with Workload Identity Federation): 1. [Create a topic](https://cloud.google.com/pubsub/docs/admin#creating_a_topic) for the data. * You can create a topic using the `gcloud` CLI tool with the following command format: `gcloud pubsub topics create $TOPIC_ID` 2. [Create a subscription](https://cloud.google.com/pubsub/docs/admin#pubsub_create_pull_subscription-gcloud) to be used with the topic you created. * You can create a subscription using the `gcloud` CLI tool with the following command format: `gcloud pubsub subscriptions create $SUBSCRIPTION_ID --topic $TOPIC_ID --topic-project $PROJECT_ID` {% hint style="warning" %} This subscription should not be used by any other service or source. {% endhint %} 3. [Enable the IAM API](https://console.cloud.google.com/apis/library/iam.googleapis.com). 4. Configure Workload Identity Federation with AWS by following the [Configure Workload Identity Federation with AWS or Azure](https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds) documentation. 1. As you are [defining an attribute mapping(s) and condition](https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds#mappings-and-conditions), take note of the following examples: * Example [attribute mappings](https://cloud.google.com/iam/docs/workload-identity-federation#mapping):
GoogleAWS
google.subjectassertion.arn.extract('arn:aws:sts::{account_id}:')+":"+assertion.arn.extract('assumed-role/{role_and_session}').extract('/{session}')
attribute.accountassertion.account
* Example [attribute condition](https://cloud.google.com/iam/docs/workload-identity-federation#conditions):\ `attribute.account==""` {% hint style="warning" %} The value of the `google.subject` attribute [cannot exceed 127 characters](https://cloud.google.com/iam/docs/workload-identity-federation#mapping). You may use [Common Expression Language (CEL) expressions](https://cloud.google.com/iam/docs/workload-identity-federation#mapping) to transform or combine attributes from the token issued by AWS. The expression suggested in the table above takes this limit into account, and is an attempt at transforming the ARN into a value that uniquely identifies Panther entities. For more information on the AWS attributes, see "Example 2 - Called by user created with AssumeRole" on [this AWS documentation page](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html). {% endhint %} 5. When you are [adding a provider to your identity pool](https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds#aws), select **AWS**. 6. Assign the required IAM roles to the account. * The following permissions are required for the project where the Pub/Sub subscription and topic lives:
Permission requiredRoleCondition
Permissions requiredRoleScope
pubsub.subscriptions.consumeroles/pubsub.subscribersubscription-name
pubsub.subscriptions.getroles/pubsub.viewersubscription-name
monitoring.timeSeries.listroles/monitoring.viewerproject
* The `monitoring.timeSeries.list` permission is optional, but is recommended for improved autoscaling. * **Note:** You can set conditions or IAM policies on permissions for specific resources. This can be done either in the IAM section in GCP (as seen in the example screenshot below) or in the specific resource's page.\ ![In the Google Cloud console, the "IAM" navigation bar item is circled. In a slide-out panel, sections titled "Add principals" and "Assign roles" are circled.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-fde2bbdd808d22e67b3f2409a908d7a63596593b%2FScreenshot%202025-03-05%20at%2015.20.04.png?alt=media) * **Note:** You can create the permissions using the `gcloud` CLI tool, where the `$PRINCIPAL_ID` may be something like:\ `principalSet://iam.googleapis.com/projects//locations/global/workloadIdentityPools//attribute.account/` * `gcloud projects add-iam-policy-binding $PROJECT_ID --member="$PRINCIPAL_ID" --role="roles/pubsub.subscriber"` * `gcloud projects add-iam-policy-binding $PROJECT_ID --member="$PRINCIPAL_ID" --role="roles/pubsub.viewer"` * `gcloud projects add-iam-policy-binding $PROJECT_ID --member="$PRINCIPAL_ID" --role="roles/monitoring.viewer"` 7. [Download the credential configuration file](https://cloud.google.com/iam/docs/workload-download-cred-and-grant-access), which will be used in Panther to authenticate to the GCP infrastructure. * To generate a credential configuration file using the gcloud CLI tool, use the following command format:\ `gcloud iam workload-identity-pools create-cred-config projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$POOL_ID/providers/$PROVIDER_ID --aws --output-file=config.json` {% endtab %} {% endtabs %} {% endtab %} {% endtabs %} ### Step 3: Finish the source setup in Panther {% tabs %} {% tab title="Service account authentication" %} If you are using a [Google Cloud service account](https://cloud.google.com/iam/docs/service-account-overview) to authenticate: 1. Under **Provide pulling configuration & JSON Keyfile**, upload your JSON key file. 2. Enter your **Pub/Sub Subscription ID,** found in the **Subscriptions** section of your Google Cloud account.\ ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-ae31c88e5acfad0e4374bac8d748aa5c49526010%2FScreenshot%202025-03-12%20at%2012.10.49.png?alt=media) 3. Under **Regional Endpoint (Optional)**, optionally enter a [regional endpoint](https://cloud.google.com/pubsub/docs/reference/service_apis_overview#list_regional_endpoints). 4. Click **Setup**. You will be directed to a success screen:
The success screen reads, "Everything looks good! Panther will now automatically pull & process logs from your account"
* You can optionally enable one or more [Detection Packs](https://docs.panther.com/detections/panther-managed/packs). * If you have not done so already, click **Attach or Infer Schemas** to attach one or more schemas to the source. * The **Trigger an alert when no events are processed** setting defaults to **YES**. We recommend leaving this enabled, as you will be alerted if data stops flowing from the log source after a certain period of time. The timeframe is configurable, with a default of 24 hours.
The "Trigger an alert when no events are processed" toggle is set to YES. The "How long should Panther wait before it sends you an alert that no events have been processed" setting is set to 1 Day
{% endtab %} {% tab title="Workload Identity Federation authentication" %} If you are using [Google Cloud Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) to authenticate: 1. On the **Infrastructure & Credentials page**, if you have not already, click the **Workload Identity Federation** tab.\ ![Under an "Infrastructure & Credentials" header, a tab labeled "Workload Identity Federation" is circled.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-7e7ca651c6e60e46eebec337cf723e11dc602ae7%2FScreenshot%202025-02-24%20at%202.39.43%E2%80%AFPM.png?alt=media) 2. Under **Provide pulling configuration & credential configuration file**, upload your credential configuration file. 3. Enter your **Pub/Sub Subscription ID** and **Project ID**, found in the **Subscriptions** section of your Google Cloud account.\ ![In a tab titled "Workload Identity Federation," there are various buttons and form fields, including "GCS Bucket Name," "Pub/Sub Subscription ID," and "Project ID."](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-a308262722541528d234ad2e7bc70e45a7143742%2FScreenshot%202025-03-12%20at%2012.10.55.png?alt=media) 4. Under **Regional Endpoint (Optional)**, optionally enter a [regional endpoint](https://cloud.google.com/pubsub/docs/reference/service_apis_overview#list_regional_endpoints). 5. Click **Setup**. You will be directed to a success screen:
The success screen reads, "Everything looks good! Panther will now automatically pull & process logs from your account"
* You can optionally enable one or more [Detection Packs](https://docs.panther.com/detections/panther-managed/packs). * If you have not done so already, click **Attach or Infer Schemas** to attach one or more schemas to the source. * The **Trigger an alert when no events are processed** setting defaults to **YES**. We recommend leaving this enabled, as you will be alerted if data stops flowing from the log source after a certain period of time. The timeframe is configurable, with a default of 24 hours.
The "Trigger an alert when no events are processed" toggle is set to YES. The "How long should Panther wait before it sends you an alert that no events have been processed" setting is set to 1 Day
{% endtab %} {% endtabs %} ## Viewing ingested logs After your log source is configured, you can search ingested data using [Search](https://docs.panther.com/search/search-tool) or [Data Explorer](https://docs.panther.com/search/data-explorer). --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.panther.com/data-onboarding/data-transports/google/pubsub.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.