At some point during your AWS cloud journey, you will likely find yourself repeatedly resolving predictable incidents in your cloud environment. You may find, for example, that contrary to your company’s standards, Amazon EC2 instances are launched without encrypted EBS Volumes, security groups have ingress rules that allow SSH traffic from 0.0.0.0/0., and Amazon S3 buckets are frequently created without encryption. To release engineers from having to manually remediate repetitive events and instead direct their energies to innovate and use their minds creatively, you can build auto remediation in your AWS environment.
Automatic Remediation
Automatic remediation, also known as security response automation, or just auto remediation, is a way to describe the “planned and programmed action taken to achieve a desired state for an application or resource based on a condition or event,” (more on this is available in this AWS security blog).
The long-term benefits of setting up programmatic actions in response to events within your AWS environment are quickly appreciated. Some of these include:
- Enforcement of security requirements your company has established within your AWS environment
- Reduction of monotonous tasks
- Time given back to the engineers to tackle the types of problem solving best suited for the human brain (rather than repetitive tasks easily accomplished by a computer)
- Behavioral training of teams from alerts triggered by auto remediation events
- Automatic triaging of specific patterns or situations
- Minimization of time between event, detection, and follow-up response (particularly important for security-related events)
AWS Services for Auto Remediation
A variety of AWS services can be used for auto remediation in your environment, ranging from using individual services for creating custom architectures to using AWS-built playbooks. Here is a quick list of services to underline the idea that, depending on your needs and AWS environment setup, your architecture will vary. Ultimately, your use case should be the deciding factor for what you deploy as one or more long-term solutions.
- AWS Lambda
- Amazon SQS
- Amazon SNS
- Amazon EventBridge
- AWS Step Functions
- AWS Security Hub rules and custom actions and remediation playbook addons
- AWS Config rules
In this blog post, I’m going to focus on a common pattern that uses a combination of Amazon EventBridge and AWS Lambda. This pairing quickly adapts to a wide range of situations and supports adding functionality with iteration using other AWS services such as Amazon SNS. I’m going to walk you through it using AWS CloudFormation, so that you can try deploying it into your AWS account. Alternatively, you can deploy it via a CloudFormation stack set into one or more organizational units (OUs) in your AWS organization. This allows you to deploy the resources from the CloudFormation template into all the accounts associated with the target OUs.
Let’s Build Auto Remediation for Amazon S3 Bucket Encryption
Pretend for a moment that to fulfill one of your company’s security requirements, you need to make sure that any Amazon S3 bucket created within your AWS account is encrypted. You are finding that here and there, unencrypted buckets are being created by engineers that now need to be remediated. You want to automatically encrypt a newly created S3 bucket so you can prevent the list of unencrypted buckets from getting longer. Your preference is to start with auto remediation that uses a simple, replicable template that you can iterate on, that gives you the option to deploy it into specific accounts in your AWS organization.
This could look like the following:
- A user creates an unencrypted S3 bucket.
- AWS CloudTrail logs the API call of “CreateBucket.”
- EventBridge receives the “CreateBucket” event, recognizes that the event matches an event pattern, and triggers a Lambda function.
- A Lambda function receives the event data (which includes data about the newly created S3 bucket) and checks the encryption status of the bucket. If the bucket is not encrypted, it immediately encrypts it with default server-side encryption.
- The user can now check the encryption status of the bucket in the console or via the AWS CLI to see that the bucket has been automatically encrypted.
We only need to build four AWS resources to create a minimum viable product (MVP):
| AWS Resource | CloudFormation Resource Type | Demo CloudFormation Template’s Logical Id for Resource |
| EventBridge Rule | AWS::Events::Rule | CreateBucketEventRule |
| Lambda Function | AWS::Lambda::Function | EncryptBucketFunction |
| IAM Role | AWS::IAM::Role | EncryptBucketFunctionRole |
| Permission resource | AWS::Lambda::Permission | PermissionForEventsToInvokeLambda |
The EventBridge Rule
Using Amazon CloudFormation, we can create a rule that looks for a very specific pattern of data that comes through from CloudTrail. This pattern has an event source of S3, with an API call of “CreateBucket.” We additionally want the rule to invoke a resource when the rule gets triggered; in this case, we want it to invoke our Lambda function. Note that for this invocation to successfully occur, we will need to build an additional resource that gives permission for events to invoke the function. We’ll do this in a moment.
CreateBucketEventRule:
Type: AWS::Events::Rule
Properties:
Description: Identifies S3 bucket creation events
State: ENABLED # by default
Targets:
- Arn: !GetAtt EncryptBucketFunction.Arn
Id: TargetEncryptBucketFunction
EventPattern:
source:
- aws.s3
detail-type:
- AWS API Call via CloudTrail
detail:
eventSource:
- s3.amazonaws.com
eventName:
- CreateBucket
The use of “AWS API Call via CloudTrail” in the above example provides a wide variety of events that can be used for automation because it gives us a way to harness AWS API calls as a trigger. I’d recommend taking a moment to navigate to the AWS CloudTrail console and examine the Event history. As an example, change the pulldown menu on the left from Read only to Resource type and then use the pulldown menu to the right to choose AWS::EC2::VPC. Note all the events, or API calls, that have been logged in your account.
It’s worth noting that many services emit events, which means you can instead set up a rule to use that emitted event versus an API call made by that service. More information about events is available in the EventBridge docs.
The Lambda Function
When the EventBridge rule finds a match to the pattern it is looking for, it triggers the Lambda function to check the newly created S3 bucket for encryption. If the bucket is not encrypted, the function adds default encryption with an Amazon S3-managed key.
Notice that logging is used to send logs to Amazon CloudWatch, where logs and outputs are provided within a log group every time the function is successfully invoked. These logs can be useful for debugging and troubleshooting issues with the function.
EncryptBucketFunction:
Type: AWS::Lambda::Function
Properties:
Handler: index.lambda_handler
Runtime: python3.8
Role: !GetAtt EncryptBucketFunctionRole.Arn
Timeout: 300
Code:
ZipFile: |
import boto3
from botocore.exceptions import ClientError
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
"""Lambda function triggered by EventBridge rule that detects the
creation of an S3 bucket.
"""
try:
# gather data from event
curr_region = event['region']
s3 = boto3.client('s3', region_name=curr_region)
detail = event['detail']
bucket_name = detail['requestParameters']['bucketName']
account_id = detail['userIdentity']['accountId']
check_encryption_status(bucket_name, s3, account_id, logger)
except ClientError as e:
logger.info(
f'{e}. Enabling encryption on Bucket {bucket_name}...'
)
set_up_encryption(s3, bucket_name, account_id, logger)
check_encryption_status(bucket_name, s3, account_id, logger)
except Exception as e:
raise e
def check_encryption_status(bucket_name, s3, account_id, logger):
response = s3.get_bucket_encryption(
Bucket=bucket_name,
ExpectedBucketOwner=account_id
)
logger.info(f'Current encryption status: {response}')
return response
def set_up_encryption(s3, bucket_name, account_id, logger):
response = s3.put_bucket_encryption(
Bucket=bucket_name,
ServerSideEncryptionConfiguration={
'Rules': [
{
'ApplyServerSideEncryptionByDefault': {
'SSEAlgorithm': 'AES256'
}
},
]
},
ExpectedBucketOwner=account_id
)
return response
In this CloudFormation example, Python is written inline to the Lambda resource. There are pros and cons to doing it this way. For this use case, we want to, with minimal effort, be able to deploy the CloudFormation template as a stack or a stack set. Setting the code inline to the AWS::Lambda::Function lets us simply upload the single file to AWS, whether it’s to a stack or stack set. This lets us avoid the slightly increased complexity of using multiple files to write our infrastructure as code—a YAML file and a Python file for example—which would require us to take the steps to store the function to an S3 bucket (you can read about macros, transforms, and stack sets HERE, how to create a ZIP file and upload to an S3 bucket in the CloudFormation User Guide HERE, and about the very useful but more elaborate AWS SAM HERE).
The biggest downsides to embedding the code within the CloudFormation is that you cannot use external libraries, your IDE will not catch errors, and you are limited to a max of 4,096 characters for your CloudFormation template. However, this is a short and simple template, so here the benefit outweighs the costs.
IAM Role
Our Lambda function will need an IAM role to allow it to check the S3 bucket for encryption status and encrypt that bucket as needed. We also want the function to be able to log to CloudWatch, which the AWS managed policy called AWSBasicLambdaExecutionRole is designed for.
EncryptBucketFunctionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
Policies:
- PolicyName: CheckEncryptionStatusAndAddBucketEncryption
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- s3:GetEncryptionConfiguration
- s3:PutEncryptionConfiguration
Resource: "*"
Lambda Permission
Lastly, we need a resource called a Lambda permission. This is a very specific agreement for a service (or account) to use a Lambda. In our case, we need to provide permission for EventBridge to trigger the Lambda function. If you deploy the template with the Lambda permission commented out, and then navigate to the AWS Lambda console and click on the “configuration” tab, you will notice that there is no trigger visible for the function. The permission resource will change that; once it is deployed, the function will show that it can now be triggered by the “CreateBucketEventRule.”
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref EncryptBucketFunction
Action: "lambda:InvokeFunction"
Principal: "events.amazonaws.com"
SourceArn: !GetAtt CreateBucketEventRule.Arn
Try Out the Code Yourself
All the code used in this blog post is available to you in the 1Strategy GitHub S3 encryption auto remediation repository.
You can either deploy the CloudFormation template via your AWS CLI or from the AWS CloudFormation console. Instructions to deploy using your AWS CLI are in the GitHub repo documentation. To deploy via the CloudFormation console, click on the button in the upper right corner that says, “Create stack.” Follow the console’s instructions to upload and deploy the YAML template.
Conclusion
In this blog post we have discussed what auto remediation is and its benefits, identified AWS services that are often used to architect auto remediation solutions, explored a frequently used pattern using EventBridge and Lambda to automatically respond to events in an AWS cloud environment, and used that pattern to automatically encrypt any newly created S3 bucket in an AWS account.
If you would like hands-on help setting up auto remediation or strengthening your security stature in your AWS cloud environment, please let us know. Reach out to info@1strategy.com to start a conversation with us.
