Data Explorer SQL Search Examples

Overview

These examples demonstrate how to use Data Explorer in the Panther Console to investigate and answer questions using the data within your Security Data Lake.

For examples on syntax and correct use of statements for Snowflake environments, please see the Snowflake documentation: SQL Command Reference and Querying semi-structured data.

The examples on this page apply to all log types, while the sub-pages contain examples that apply to specific log types:

• VPC Flow logs

• CloudTrail logs

• GuardDuty logs

• S3 Access logs

• NGINX and ALB Access logs

• Github Audit logs

Considerations

All queries should control the result size. This can be done with a LIMIT or GROUP BY clause.

Please note that all Athena queries should be qualified with partition columns (year, month, day, hour) for performance reasons.

Did this IP address have any activity in my network (and in what logs)?

This is often one of the first questions asked in an investigation. Given there is some known bad indicator such as an IP address, then if there is related activity in your network/systems, a detailed investigation will be needed.

You can quickly search your data in Panther using the all_logs Athena view, which will search all data for the indicator of interest.

Athena's contains method requires an exact match for data you're searching. Snowflake's contains method can be a partial match for the data, and Snowflake's array_contains method requires an exact match for the data in question.

In the example below, the Panther field p_any_ip_addresses is used. Panther extracts a number of common indicator fields over all data sources into standard fields (see Panther Fields).

Snowflake

SELECT
 p_log_type, count(1) AS row_count
FROM panther_views.public.all_logs
WHERE p_occurs_between('2021-01-01', '2021-01-02')
     AND array_contains('95.123.145.92'::variant, p_any_ip_addresses)
GROUP BY p_log_type
LIMIT 100

Athena

SELECT
 p_log_type, count(1) AS row_count
FROM panther_views.all_logs
WHERE p_occurs_between('2021-01-01', '2021-01-02')
     AND contains(p_any_ip_addresses, '1.2.3.4')
GROUP BY p_log_type
LIMIT 100

What are the top 10 IPs by row count over all logs?

Ranking activity (top or bottom) is a useful technique to gain visibility into a network. High ranking activity might help locate IP addresses involved in a DDOS attack while low ranking (change ORDER BY to ASC) might highlight sneaky activity.

Snowflake

SELECT
  ip.value as ip,
  count(1) as total_rows
FROM panther_views.public.all_logs, LATERAL FLATTEN(input => p_any_ip_addresses) ip
WHERE p_occurs_between('2021-01-01', '2021-01-02')
GROUP BY ip.value
ORDER BY  total_rows DESC
LIMIT 100

Athena

SELECT
  ip,
  count(1) as total_rows
FROM panther_views.all_logs
CROSS JOIN UNNEST(p_any_ip_addresses) AS t(ip)
WHERE p_occurs_between('2021-01-01', '2021-01-02')
GROUP BY ip
ORDER BY  total_rows DESC
LIMIT 100

What are the top 10 IPs by log type over all logs?

This is a variant of the above query where we are ranking the IPs by how many data sources they show activity. This shows the degree of "reach" the IP address has over all your systems.

Snowflake

SELECT
  ip,
  count(distinct p_log_type) as datasets
FROM
(
SELECT
  p_log_type,
  ip. value as ip
FROM panther_views.public.all_logs, LATERAL FLATTEN(input => p_any_ip_addresses) ip
WHERE p_occurs_between('2021-01-01', '2021-01-02')
GROUP BY ip.value, p_log_type
)
GROUP BY ip
ORDER BY  datasets DESC
LIMIT 100

Athena

SELECT
  ip,
  count(distinct p_log_type) as datasets
FROM
(
SELECT
  p_log_type,
  ip
FROM panther_views.all_logs
CROSS JOIN UNNEST(p_any_ip_addresses) AS t(ip)
WHERE p_occurs_between('2021-01-01', '2021-01-02')
GROUP BY ip, p_log_type
)
GROUP BY ip
ORDER BY  datasets DESC
LIMIT 100