September 19th, 2017
IAM Permissions Can Be Tricky
By Justin Iravani

A significant amount of the time I spend with customers is fine tuning and troubleshooting IAM permissions. Permissions are easy if your policy has lots of asterisks in it, however once you start minimizing the scope of your policy, things get more complex.

On the surface, AWS permissioning is straightforward. Let’s say I want to grant someone permission to delete an object in my S3 bucket. This should work right?

Not so fast.

The s3:DeleteObject only applies to objects within the bucket, not the S3 bucket itself. So the correct Resource would be all of the objects contained within MyBucket. i.e.

The above is an example of different actions having Resource-level permissions. In addition to Resource-level, there is also Action-level, Resource-based, and Tag-Based permissioning schemes. Each has its own limitations and quirks. You can find out more HERE.

As you can see, there is quite a bit to know in order to build proper IAM Policies. These types of misunderstandings can be very frustrating to get right. Luckily, AWS is looking out for us.

Being customer obsessed as always, last week, AWS updated the IAM console to help troubleshoot some of these non-obvious IAM Policy issues.

Now in the IAM Console you can see if you’ve applied permissions on the proper level:

There are all sorts of nuances when creating proper IAM policies, especially when you’re creating a secure environment; this feature makes it much easier to build properly formulated secure IAM Policies.

Principle of Least Privilege is critical in any environment, but AWS even more so. It’s very difficult for someone to accidentally delete physical servers. In a virtual environment, it can be trivial for someone to accidentally terminate production servers because they were given admin access to “get something working” and then those permissions were never reined in.

On the flipside, it can be easy to overshoot and remove permissions that a system or individual actually does need to perform their job. In such cases, it can be very frustrating to encounter and remediate every single permission, especially if you have to hunt through logs to figure out what went wrong or if you’re not being made aware of the access denials in real-time.

To cite an example, part of what we do at 1Strategy is build custom hands-on trainings. Therein students are given user accounts in our training AWS account. If a student were to decide to go exploring and buy a $5000 CloudHSM module, that’s going be an awkward phone call to have with my boss. As such, we have to make sure that the students have ONLY the permissions they need to complete the training.

A big part of the process is running through the lab with the student credentials and making sure that the permissions are just right—not too permissive and not too restrictive. To assist the team in better fine tuning the student IAM policies, I created a Lambda function that watches the CloudTrail S3 bucket, extracts the access denied AWS API calls from CloudTrail Logs, and sends an SNS notification to inform the team that a permission is missing.

It’s a nice way to get quasi-real-time notification if a user or system has a missing permission and it can easily be added to the correct policy. You can see a sample of the Lambda function HERE.

You could arguably use CloudWatch Events to do something similar, however CloudWatch Events has a limitation in that rules only trigger off of WRITE API actions (e.g. Create*, Delete*, Modify) and don’t trigger off of READ API actions (e.g. List, Describe*). While there may not be sufficient value in automated actions triggered by READ API actions, it still isn’t a great experience if a student can’t see something they should be able to see, or a task fails to execute because it is missing a read permission.

There are also some edge cases where denials are obfuscated from the user. For example, a user may not see an access denied message, instead the action they are trying to perform just fails (Looking at you, Athena) and the CloudWatch Event would not trigger.

Now that I’ve laid out some of the challenges, here are some tools to help you get IAM permissions right:

  1. The AWS Policy Simulator is built into AWS to help you test the permissions in real-time to determine if the correct permissions are in place.

  2. The AWS Policy Generator is a great place to start creating IAM policies assuming you have some idea of what your application/user should be doing.

    There are thousands of API Actions on AWS and trying to remember them all is ludicrous, the Policy Generator has a (mostly) up-to-date list of the actions supported for IAM policies as well as resource-based permissions (S3/SQS/SNS etc.).

  3. The IAM Console has been recently updated several times by AWS to help users make more sense of the policies that are already in place and can be a very useful resource.


  4. Thanks to the folks over at Cloudonaut for creating an awesome IAM reference. It lists virtually all of the AWS API calls, the resource level they apply to and which conditions are supported. This is an immensely helpful resource, make liberal use of it.


  5. Should all else fail, you can always go directly to the AWS documentation HERE.

Please reach out to me at justin.iravani@new.1strategy.com if you have other IAM resources that you find particularly useful or your organization could use some help crafting IAM permissions.