Problem
Despite Amazon S3’s being only an object store storage solution, this service can be leveraged to support some pretty complex architectural designs and business requirements. I was recently presented with an interesting problem to solve for one of our clients. For simplicity, I would refer to the first AWS account as Alice’s account and the second AWS account as Bob’s account. The request was to copy a file from Alice’s S3 bucket located in her AWS account into an S3 bucket located in a Bob’s AWS account. In addition to the buckets’ being located in different AWS accounts, the contents of both buckets had to be encrypted. Alice’s S3 bucket was encrypted with her KMS key while Bob’s S3 bucket was encrypted with his own KMS key. Alice was writing data into her bucket and Bob wanted to copy this encrypted data into his own S3 bucket and encrypt it with his own key in order to do further analysis on the data.
Solution
After reading through the documentation, I came up with a solution. The first thing that had to be considered was that enabling proper cross-account S3 access would not be sufficient. Bob not only needed permission to the S3 bucket but also needed permission to use Alice’s KMS key to decrypt the data. Since this solution had many components, I have provided an architectural diagram and also IAM, Bucket, and KMS Key policy samples to help you with the implementation of this solution.
Below are the steps I took to enable copying the encrypted S3 objects across AWS accounts:
- Bob’s Account: Create an IAM Role that will be used to access Alice’s data.
- Alice’s Account: Update Alice’s bucket policy to allow Bob’s IAM role to read data from the bucket (Step 1 in the diagram).
- Alice’s Account: Grant permission to Bob’s AWS Account to utilize Alice’s KMS Key (Step 2 in the diagram).
- Bob’s Account: Grant permission in Bob’s account to allow his IAM role to utilize his KMS Key (Step 3 in the diagram).
- Bob’s Account: Add IAM policy to allow Bob’s IAM role to read and write to his staging bucket (Step 4 in the diagram).
- Bob’s Account: Add IAM policy to allow Bob’s IAM role to read from Alice’s bucket (Step 4 in the diagram).
- Bob’s Account: Add IAM policy to allow Bob’s IAM role to utilize Alice’s KMS key (Step 4 in the diagram).
After these steps are complete, I can run a copy command while the decryption and re-encryption occur seamlessly in the background.
aws s3 cp s3://alice-data/ s3://bob-staging-bucket –recursive
Architectural Diagram:
IAM Policy Samples:
Alice’ Bucket Policy
This bucket policy should be applied to Alice’s S3 bucket from where data is going to be read.
{
"Version": "2012-10-17",
"Id": "Policy1536196380857",
"Statement": [
{
"Sid": "Stmt1536196369980",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam:::role/Bob’s_Data_Access_Role"
},
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::alice-data",
"arn:aws:s3:::alice-data/*"
]
}
]
}
Alice’s KMS Key Inline Policy
This KMS key policy should be applied to the KMS key that is used to encrypt the S3 objects located in the Alice’s S3 Bucket. This would allow Bob’s account, to not only get S3 object, but also decrypt them.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowUseOfTheKey",
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:us-west-2:<Alice’s-AWSAccountNumber>:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
]
}
]
}
Bob’s Staging S3 Bucket Access Policy
This IAM policy should be attached to the IAM role that will need permission to get data from Alice’s data bucket. This policy however, grants permission to Bob’s bucket so the role has permission to put objects that it receives from Alice’s data bucket. The next policy below will grant permissions to Alice’s data bucket.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:AbortMultipartUpload",
"s3:ListBucket",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::bob-staging-bucket",
"arn:aws:s3:::bob-staging-bucket/*"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"s3:HeadBucket"
],
"Resource": "*"
}
]
}
Bob’s Access to Alice’s S3 Bucket Policy
This policy allows the IAM role to talk to Alice’s data bucket that is located in her AWS account. This policy should be paired with the previous one so the principal assuming this role has a place to store the data once it receives it from Alice’s data bucket.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::alice-data/*",
"arn:aws:s3:::alice-data"
]
}
]
}
Summary
As mentioned earlier, S3 is an excellent object storage solution in the cloud for a variety of use cases. From simple file storage, to complex multi-account encrypted data pipelines, S3 is able to provide value. In this solution I showed you how to copy encrypted data from an S3 bucket in one AWS account into an S3 bucket in a separate AWS account. You can use this solution whenever you need to copy S3 data from a customer’s or partner’s AWS account while keeping the data secure. If you have further questions regarding S3 or how it can be leveraged in your infrastructure, feel free to schedule a consultation with us at info@1Strategy.com or email me directly at pavel@new.1strategy.com.
