EC2, Encrypted EBS, and IAM Users

Background

Recently while working with one of our clients, we ran into an issue where an IAM user (we’ll call him John) with full EC2 permissions could not start an EC2 instance after it was stopped for a maintenance task. When starting the instance, the instance state would change to “Pending,” but after a few seconds it would switch back to “Stopped.”
Upon further inspection, we discovered that the instance had attached EBS volumes that were encrypted using a custom Customer Managed Key (CMK). When these encrypted volumes were detached, John could start the EC2 instances without any issues. This wasn’t what we wanted to happen because we needed these volumes for our instance.

Problem

To find the root of the problem. Let’s take a look at how EBS and EC2 use encryption. From the AWS documentation:

1. When you create an encrypted EBS volume, Amazon EBS sends a GenerateDataKeyWithoutPlaintext request to AWS KMS, specifying the CMK that you chose for EBS volume encryption.
2. AWS KMS generates a new data key, encrypts it under the specified CMK, and then sends the encrypted data key to Amazon EBS to store with the volume metadata.
3. EC2 sends the encrypted data key to AWS KMS with a Decrypt request.
4. AWS KMS decrypts the encrypted data key and then sends the decrypted (plaintext) data key to Amazon EC2.
5. Amazon EC2 uses the plaintext data key in hypervisor memory to encrypt disk I/O to the EBS volume. The data key persists in memory as long as the EBS volume is attached to the EC2 instance.

The problem we’re facing deals with steps 3 through 5. Although John has full access to EC2 resources, he does not have permissions to allow EC2 access the CMK in KMS. This prevents EC2 from retrieving a decryption key for the EBS volume(s).

Solution

A quick fix would be to give John full kms:* permissions, but this is a bad idea. We want to follow best practices, right? We should follow the principle of least privilege, and only give John the permissions necessary to allow him to start EC2 instances with CMK encrypted EBS volumes. How do we this?
Let’s talk about kms:CreateGrant. This permission allows grants to be added to CMKs. Grants are an alternative way to specify permissions rather than key policies. A grant specifies who can use the CMK and under what conditions. For example, allow IAM user John to encrypt and decrypt with the key, or allow the EC2 service to decrypt with the specified key. Having permissions to create KMS grants is essentially like having the power to give anyone kms:* privileges for a key.
At this point you’re probably saying, “Wait! Hold up a minute. You mean to tell me that the kms:CreateGrant permission allows someone to give out any permissions they want to my key?” In short, yes. However, let’s continue with the principle of least privilege.
We must ensure that grants are only given to AWS services that require KMS permissions. This is done with the kms:GrantIsForAWSResource condition in your policy. When the kms:GrantIsForAWSResource condition is set to true, the IAM user or resource with the kms:CreateGrant permission can only create grants to AWS services integrated with AWS KMS, not to IAM users.
If you’re setting a policy on your IAM user, you could use a policy similar to the following (using your CMK’s ARN in the Resources section):

{
     "Version": "2012-10-17",
     "Statement": [
          {
               "Effect": "Allow",
               "Action": [
                    "kms:CreateGrant"
               ],
               "Resource": [
                    "arn:aws:kms:<region>:<account #>:key/<key id>"
               ],
               "Condition": {
                    "Bool": {
                         "kms:GrantIsForAWSResource": true
                    }
               }
          }
     ]
}

If you need a user to have access to multiple keys, you’d add each key’s ARN in the Resources section.

Conclusion

By using the kms:CreateGrant permission, in combination with the kms:GrantIsForAWSResource policy condition, we can allow an IAM user to start EC2 instances with encrypted EBS volumes without exposing the key to them. This causes less headache on the IAM user’s end and still allows you to keep your keys secure.