GCP Logs

Connecting GCP logs to your Panther Console

Overview
Panther supports ingesting Google Cloud Platform (GCP) logs via common Data Transport options: Amazon Web Services (AWS) S3, AWS SQS, and Google Cloud Storage (GCS).
GCP Logs Video Walkthrough
Walkthrough video showing how to onboard GCP logs to Panther
How to onboard GCP logs to Panther
To connect these logs into Panther:
1.Set up your Data Transport in the Panther Console.
Please follow Panther’s documentation for configuring the Data Transport option you will use:
​AWS S3 bucket​
​AWS SQS​
​GCS​
2.Configure GCP to push logs to the Data Transport source.
See GCP's documentation for instructions on pushing logs to your selected Data Transport source.
Panther-Built Detections
See Panther's built in rules for GCP in panther-analysis in Github.​
Supported log types
Required fields in the table are in bold.
GCP.AuditLog
Cloud Audit Logs maintains three audit logs for each Google Cloud project, folder, and organization: Admin Activity, Data Access, and System Event. Google Cloud services writes audit log entries to these logs to help answer the questions of "who did what, where, and when?" within your Google Cloud resources.
Reference: GCP Documentation on Cloud Audit Logs. ​
Column
Type
Description
logName
string
The resource name of the log to which this log entry belongs.
severity
string
The severity of the log entry. The default value is LogSeverity.DEFAULT.
insertId
string
A unique identifier for the log entry.
resource
{   "type":string,   "labels":{     string:string } }
The monitored resource that produced this log entry.
timestamp
timestamp
The time the event described by the log entry occurred.
receiveTimestamp
timestamp
The time the log entry was received by Logging.
labels
{   string:string }
A set of user-defined (key, value) data that provides additional information about the log entry.
operation
{   "id":string,   "producer":string,   "first":boolean,   "last":boolean }
Information about an operation associated with the log entry, if applicable.
trace
string
Resource name of the trace associated with the log entry, if any.
httpRequest
{   "requestMethod":string,   "requestURL":string,   "requestSize":bigint,   "status":smallint,   "responseSize":bigint,   "userAgent":string,   "remoteIP":string,   "serverIP":string,   "referer":string,   "latency":string,   "cacheLookup":boolean,   "cacheHit":boolean,   "cacheValidatedWithOriginServer":boolean,   "cacheFillBytes":bigint,   "protocol":string }
Information about the HTTP request associated with this log entry, if applicable.
spanId
string
The span ID within the trace associated with the log entry.
traceSampled
boolean
The sampling decision of the trace associated with the log entry.
sourceLocation
{   "file":string,   "line":bigint,   "function":string }
Source code location information associated with the log entry, if any.
protoPayload
{   "at_sign_type":string,   "serviceName":string,   "methodName":string,   "resourceName":string,   "numResponseItems":bigint,   "status":{     "code":int,     "message":string,     "details":string },   "authenticationInfo":{     "principalSubject":string,     "serviceAccountKeyName":string,     "principalEmail":string,     "authoritySelector":string,     "thirdPartyPrincipal":string,     "serviceAccountDelegationInfo":[{       "firstPartyPrincipal":{         "principalEmail":string,         "serviceMetadata":string },       "thirdPartyPrincipal":{         "thirdPartyClaims":string } }] },   "authorizationInfo":[{     "resource":string,     "permission":string,     "granted":boolean,     "resourceAttributes":{       "service":string,       "name":string,       "type":string,       "labels":string,       "uid":string } }],   "requestMetadata":{     "callerIP":string,     "callerSuppliedUserAgent":string,     "callerNetwork":string,     "requestAttributes":string,     "destinationAttributes":string },   "request":string,   "response":string,   "serviceData":json,   "metadata":string }
The AuditLog payload

Note: protoPayload.serviceData is type json because Google emits context-dependent logs. By having it as type:json, Panther can account for anything that can be present under this field.
p_log_type
string
Panther added field with type of log
p_row_id
string
Panther added field with unique id (within table)
p_event_time
timestamp
Panther added standardize event time (UTC)
p_parse_time
timestamp
Panther added standardize log parse time (UTC)
p_source_id
string
Panther added field with the source id
p_source_label
string
Panther added field with the source label
p_any_ip_addresses
[string]
Panther added field with collection of ip addresses associated with the row
p_any_domain_names
[string]
Panther added field with collection of domain names associated with the row
p_any_sha1_hashes
[string]
Panther added field with collection of SHA1 hashes associated with the row
p_any_md5_hashes
[string]
Panther added field with collection of MD5 hashes associated with the row
p_any_sha256_hashes
[string]
Panther added field with collection of SHA256 hashes of any algorithm associated with the row
GCP.HTTPLoadBalancer
External HTTP(S) Load Balancing distributes HTTP and HTTPS traffic to backends hosted on a variety of Google Cloud platforms (such as Compute Engine, Google Kubernetes Engine (GKE), Cloud Storage, and so on), as well as external backends connected over the internet or via hybrid connectivity. HTTP(S) load balancing logs provide information for monitoring and debugging web traffic.
Reference: HTTPLoadBalancer​
schema: GCP.HTTPLoadBalancer
parser:
  native:
    name: GCP.HTTPLoadBalancer
fields:
  - name: httpRequest
    required: true
    description: httpRequest
    type: object
    fields:
      - name: referer
        description: referer
        type: string
        indicators:
          - url
      - name: latency
        required: true
        description: latency
        type: string
      - name: remoteIp
        required: true
        description: remoteIp
        type: string
        indicators:
          - ip
      - name: requestMethod
        required: true
        description: requestMethod
        type: string
      - name: requestSize
        required: true
        description: requestSize
        type: bigint
      - name: requestUrl
        required: true
        description: requestUrl
        type: string
        indicators:
          - url
      - name: responseSize
        description: responseSize
        type: bigint
      - name: serverIp
        description: serverIp
        type: string
        indicators:
          - ip
      - name: status
        description: status
        type: bigint
      - name: userAgent
        description: userAgent
        type: string
  - name: insertId
    required: true
    description: insertId
    type: string
  - name: jsonPayload
    required: true
    description: jsonPayload
    type: object
    fields:
      - name: '@type'
        required: true
        description: '@type'
        type: string
      - name: remoteIp
        required: true
        description: remoteIp
        type: string
        indicators:
          - ip
      - name: statusDetails
        required: true
        description: statusDetails
        type: string
  - name: logName
    required: true
    description: logName
    type: string
  - name: receiveTimestamp
    required: true
    description: receiveTimestamp
    type: timestamp
    timeFormat: rfc3339
  - name: resource
    required: true
    description: resource
    type: object
    fields:
      - name: labels
        required: true
        description: labels
        type: object
        fields:
          - name: backend_service_name
            required: true
            description: backend_service_name
            type: string
          - name: forwarding_rule_name
            required: true
            description: forwarding_rule_name
            type: string
          - name: project_id
            required: true
            description: project_id
            type: string
          - name: target_proxy_name
            required: true
            description: target_proxy_name
            type: string
          - name: url_map_name
            required: true
            description: url_map_name
            type: string
          - name: zone
            required: true
            description: zone
            type: string
      - name: type
        required: true
        description: type
        type: string
  - name: severity
    required: true
    description: severity
    type: string
  - name: spanId
    required: true
    description: spanId
    type: string
  - name: timestamp
    required: true
    description: timestamp
    type: timestamp
    timeFormat: rfc3339
    isEventTime: true
  - name: trace
    required: true
    description: trace
    type: string
    indicators:
      - trace_id

Fluentd Logs

GitHub Logs

Last modified 10d ago

Column	Type	Description
`logName`	`string`	The resource name of the log to which this log entry belongs.
`severity`	`string`	The severity of the log entry. The default value is LogSeverity.DEFAULT.
`insertId`	`string`	A unique identifier for the log entry.
`resource`	`{ "type":string, "labels":{ string:string } }`	The monitored resource that produced this log entry.
`timestamp`	`timestamp`	The time the event described by the log entry occurred.
`receiveTimestamp`	`timestamp`	The time the log entry was received by Logging.
`labels`	`{ string:string }`	A set of user-defined (key, value) data that provides additional information about the log entry.
`operation`	`{ "id":string, "producer":string, "first":boolean, "last":boolean }`	Information about an operation associated with the log entry, if applicable.
`trace`	`string`	Resource name of the trace associated with the log entry, if any.
`httpRequest`	`{ "requestMethod":string, "requestURL":string, "requestSize":bigint, "status":smallint, "responseSize":bigint, "userAgent":string, "remoteIP":string, "serverIP":string, "referer":string, "latency":string, "cacheLookup":boolean, "cacheHit":boolean, "cacheValidatedWithOriginServer":boolean, "cacheFillBytes":bigint, "protocol":string }`	Information about the HTTP request associated with this log entry, if applicable.
`spanId`	`string`	The span ID within the trace associated with the log entry.
`traceSampled`	`boolean`	The sampling decision of the trace associated with the log entry.
`sourceLocation`	`{ "file":string, "line":bigint, "function":string }`	Source code location information associated with the log entry, if any.
`protoPayload`	{ "at_sign_type":string, "serviceName":string, "methodName":string, "resourceName":string, "numResponseItems":bigint, "status":{ "code":int, "message":string, "details":string }, "authenticationInfo":{ "principalSubject":string, "serviceAccountKeyName":string, "principalEmail":string, "authoritySelector":string, "thirdPartyPrincipal":string, "serviceAccountDelegationInfo":[{ "firstPartyPrincipal":{ "principalEmail":string, "serviceMetadata":string }, "thirdPartyPrincipal":{ "thirdPartyClaims":string } }] }, "authorizationInfo":[{ "resource":string, "permission":string, "granted":boolean, "resourceAttributes":{ "service":string, "name":string, "type":string, "labels":string, "uid":string } }], "requestMetadata":{ "callerIP":string, "callerSuppliedUserAgent":string, "callerNetwork":string, "requestAttributes":string, "destinationAttributes":string }, "request":string, "response":string, "serviceData":json, "metadata":string }	The AuditLog payload Note: `protoPayload.serviceData` is type `json` because Google emits context-dependent logs. By having it as `type:json`, Panther can account for anything that can be present under this field.
`p_log_type`	`string`	Panther added field with type of log
`p_row_id`	`string`	Panther added field with unique id (within table)
`p_event_time`	`timestamp`	Panther added standardize event time (UTC)
`p_parse_time`	`timestamp`	Panther added standardize log parse time (UTC)
`p_source_id`	`string`	Panther added field with the source id
`p_source_label`	`string`	Panther added field with the source label
`p_any_ip_addresses`	`[string]`	Panther added field with collection of ip addresses associated with the row
`p_any_domain_names`	`[string]`	Panther added field with collection of domain names associated with the row
`p_any_sha1_hashes`	`[string]`	Panther added field with collection of SHA1 hashes associated with the row
`p_any_md5_hashes`	`[string]`	Panther added field with collection of MD5 hashes associated with the row
`p_any_sha256_hashes`	`[string]`	Panther added field with collection of SHA256 hashes of any algorithm associated with the row