Exporting AWS Security Hub Findings using Python script

Overview

In today’s digital landscape, security is paramount. With the increasing sophistication of cyber threats, organizations must leverage enhanced security solutions to safeguard their assets. Amazon Web Services (AWS) offers AWS Security Hub, a comprehensive security service that provides a centralized view of your security posture across your AWS accounts. This blog post will guide you through exporting AWS Security Hub findings using a Python script, empowering you to enhance your security operations.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Introduction

AWS Security Hub helps you examine your security patterns and pinpoint the most important security risks by gathering security data from your AWS accounts, services, and approved third-party products.

As organizations embrace cloud services, effective security measures become crucial. AWS Security Hub acts as a nerve center, aggregating and prioritizing security findings from various AWS services. While AWS Security Hub provides a unified view, exporting findings for further analysis or integration with other tools can be essential. This guide will walk you through the steps to export AWS Security Hub findings using a Python script, providing greater flexibility and control.

Purpose of AWS Security Hub

Unified Security Dashboard: Provides a consolidated view of security and compliance findings from various sources for rapid assessment and response.
Automated Threat Detection: Employs automated mechanisms to continuously analyze AWS resources, proactively identifying and prioritizing potential security threats.
Prioritization of Findings: Categorizes and prioritizes security issues, allowing security teams to focus on critical issues for swift mitigation.
Integration with AWS Services: Seamlessly integrates with AWS security services to enhance security monitoring and analysis depth and breadth.

Prerequisites

Before diving into the hands-on steps, ensure you have the following prerequisites in place:

AWS Account: Access to an AWS account with sufficient permissions to work with the AWS Security Hub.
AWS CLI: Installing and configuring the AWS Command Line Interface (CLI) is necessary to communicate with AWS services.

Architecture Diagram

In this architecture, we use an AWS Lambda function with appropriate AWS IAM roles to fetch AWS Security Hub findings and store them in an Amazon S3 bucket, ensuring secure and automated data management.

Step-by-Step Guide

Step 1: Create an Amazon S3 bucket to store the log, which we will get from the AWS Lambda function.

step1

Step 2: Created a role for the AWS Lambda function with the required permission of the AWS Security Hub and Amazon S3

This role contains:

AWSLambdaBasicExecutionRole
AmazonS3FullAccess
AWSSecurityHubFullAccess

step2

Step 3: Create an AWS Lambda function for the script with the required configurations.

step3

Using the AWS Lambda function, we will extract these findings.

step3b

AWS Lambda Function Code

Below is an AWS Lambda function code in Python that exports AWS Security Hub findings to an Amazon S3 bucket:

import json
import boto3
import botocore
import csv

securityhub = boto3.client('securityhub')
s3 = boto3.resource('s3')

_filter = Filters={
    'WorkflowStatus': [
        {
            'Value': 'NEW',
            'Comparison': 'EQUALS'
        },
        {
            'Value': 'NOTIFIED',
            'Comparison': 'EQUALS'
        }
    ],
    'RecordState': [
        {
            'Value': 'ACTIVE',
            'Comparison': 'EQUALS'
        }
    ],
}

_sort = SortCriteria=[
    {
        'Field': 'ComplianceStatus',
        'SortOrder': 'desc'
    },
    {
        'Field': 'SeverityNormalized',
        'SortOrder': 'desc'
    }
]

MAX_ITEMS = 99
BUCKET_NAME = 'ssc123'
KEY = 'schubfindings.csv'

def lambda_handler(event, context):
    result = securityhub.get_findings(Filters=_filter, 
                                SortCriteria=_sort, 
                                MaxResults=MAX_ITEMS)
    print(result)
    with open("data.csv", "w") as file:
        csv_file = csv.writer(file)

        count = 0

        while(result != None): 
            
            items = []
            findings = result['Findings']
            
            for finding in findings: 
                count += 1
                item = {}
                item['Severity']=finding["Severity"]["Label"]
                item['Workflow']=finding["Workflow"]['Status']
                item['RecordState']=finding["RecordState"]
                item['Region']=finding["Region"]
                item['AccountId']=finding["AwsAccountId"]
                item['CompanyName']=finding["CompanyName"]
                item['Product']=finding["ProductName"]
                item['Title']=finding["Title"]
                item['Resource']=finding["Resources"][0]['Id'].split(":")[-4]+" "+finding["Resources"][0]['Id'].split(":")[-1]
                if "Compliance" in finding:
                    item['Compliance']=finding["Compliance"]["Status"]
                
                print("--------")
                print(count)
                # item['json'] = finding
                items.append(item)
                if (len(keys) == 0):
                    keys = list(item.keys())
                    csv_file.writerow(keys)
            
            for d in items:
                csv_file.writerow(list(d.values()))

            if 'NextToken' in result:
                result = securityhub.get_findings(Filters=_filter, SortCriteria=_sort, MaxResults=MAX_ITEMS, NextToken=result['NextToken'])

    csv_binary = open('data.csv', 'rb').read()

    try:
        obj = s3.Object(BUCKET_NAME, KEY)
        obj.put(Body=csv_binary)
    except botocore.exceptions.ClientError as e:
        if e.response['Error']['Code'] == "404":
            print("The object does not exist.")
        else:
            raise

    s3client = boto3.client('s3')

    try:
        download_url = s3client.generate_presigned_url(
            'get_object',
            Params={
                'Bucket': BUCKET_NAME,
                'Key': KEY
            },
            ExpiresIn=3600
        )
        return {
            "csv_link": download_url,
            "total": count
        }
    except Exception as e:
        raise utils_exception.ErrorResponse(400, e, Log)

    return {
        'message': 'Error found, please check your logs',
        'total': 0
    }

lambda_handler("ss", "context")

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

import json

import boto3

import botocore

import csv

securityhub = boto3.client('securityhub')

s3 = boto3.resource('s3')

_filter = Filters={

'WorkflowStatus': [

{

'Value': 'NEW',

'Comparison': 'EQUALS'

{

'Value': 'NOTIFIED',

'Comparison': 'EQUALS'

}

'RecordState': [

{

'Value': 'ACTIVE',

'Comparison': 'EQUALS'

}

_sort = SortCriteria=[

{

'Field': 'ComplianceStatus',

'SortOrder': 'desc'

{

'Field': 'SeverityNormalized',

'SortOrder': 'desc'

}

]

MAX_ITEMS = 99

BUCKET_NAME = 'ssc123'

KEY = 'schubfindings.csv'

def lambda_handler(event, context):

result = securityhub.get_findings(Filters=_filter,

SortCriteria=_sort,

MaxResults=MAX_ITEMS)

print(result)

with open("data.csv", "w") as file:

csv_file = csv.writer(file)

count = 0

while(result != None):

items = []

findings = result['Findings']

for finding in findings:

count += 1

item = {}

item['Severity']=finding["Severity"]["Label"]

item['Workflow']=finding["Workflow"]['Status']

item['RecordState']=finding["RecordState"]

item['Region']=finding["Region"]

item['AccountId']=finding["AwsAccountId"]

item['CompanyName']=finding["CompanyName"]

item['Product']=finding["ProductName"]

item['Title']=finding["Title"]

item['Resource']=finding["Resources"][0]['Id'].split(":")[-4]+" "+finding["Resources"][0]['Id'].split(":")[-1]

if "Compliance" in finding:

item['Compliance']=finding["Compliance"]["Status"]

print("--------")

print(count)

# item['json'] = finding

items.append(item)

if (len(keys) == 0):

keys = list(item.keys())

csv_file.writerow(keys)

for d in items:

csv_file.writerow(list(d.values()))

if 'NextToken' in result:

result = securityhub.get_findings(Filters=_filter, SortCriteria=_sort, MaxResults=MAX_ITEMS, NextToken=result['NextToken'])

csv_binary = open('data.csv', 'rb').read()

try:

obj = s3.Object(BUCKET_NAME, KEY)

obj.put(Body=csv_binary)

except botocore.exceptions.ClientError as e:

if e.response['Error']['Code'] == "404":

print("The object does not exist.")

else:

raise

s3client = boto3.client('s3')

try:

download_url = s3client.generate_presigned_url(

'get_object',

Params={

'Bucket': BUCKET_NAME,

'Key': KEY

ExpiresIn=3600

)

return {

"csv_link": download_url,

"total": count

}

except Exception as e:

raise utils_exception.ErrorResponse(400, e, Log)

return {

'message': 'Error found, please check your logs',

'total': 0

}

lambda_handler("ss", "context")

Advantages

Unified Visibility: AWS Security Hub offers a centralized dashboard for a consolidated view of security findings from diverse AWS services and third-party tools.
Automated Threat Detection: AWS Security Hub employs automated mechanisms to continuously analyze AWS resource configurations, proactively identifying and prioritizing potential security threats.
Prioritization and Insights: AWS Security Hub categorizes and prioritizes security findings, providing actionable insights into potential impacts on the overall security posture.
Integration Capabilities: AWS Security Hub seamlessly integrates with various AWS security services like Amazon GuardDuty, AWS Inspector, and AWS Config, enhancing the depth and breadth of security monitoring.
Customization and Automation: AWS Security Hub allows users to create custom insights and automate responses based on predefined rules and playbooks, enabling organizations to tailor security measures efficiently.
Compliance Monitoring: AWS Security Hub includes pre-built compliance standards and checks, aiding organizations in assessing adherence to industry-specific regulations and best practices.

Conclusion

AWS Security Hub emerges as a vital ally in safeguarding assets against sophisticated cyber threats in the ever-evolving digital landscape.

This blog has navigated readers through the significance of AWS Security Hub, outlining its diverse functionalities and advantages. The upcoming section will explore a hands-on lab, guiding users step-by-step by exporting AWS Security Hub findings using Python.

Drop a query if you have any questions regarding AWS Security Hub and we will get back to you quickly.

Making IT Networks Enterprise-ready – Cloud Management Services

Accelerated cloud migration
End-to-end view of the cloud environment

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, Microsoft Gold Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.

FAQs

1. Why should I export AWS Security Hub findings?

ANS: – Exporting findings allows for deeper analysis, custom reporting, and integration with other security tools, enhancing your overall security strategy.

2. Can I schedule the Python script for automated exports?

ANS: – Yes, you can leverage AWS Lambda or other scheduling mechanisms to automate the script execution, enabling regular exports of AWS Security Hub findings.

WRITTEN BY Rohit Kumar

Rohit Kumar works as a Research Associate (Infra, Migration, and Security Team) at CloudThat. He is focused on gaining knowledge of the Cloud environment. He has a keen interest in learning and researching emerging technologies.