If you are using Amazon Web Services in any capacity, you inevitably are using Amazon Simple Storage Service (S3). Amazon S3 is a scalable, reliable, and fast storage infrastructure built for storage of objects or BLOBs. Some common use cases for using S3 are to store and distribute static web content and media, store data for analytics, and for backup and data archival.

S3 offers the following storage classes to fit various needs and use cases:

• Standard (S3-Standard), for general purpose storage of data that is frequently accessed
• Standard-Infrequent Access (S3-IA), for data that is accessed less frequently
• One-Zone IA, for non-critical data that is infrequently accessed
• Glacier, for archival data that does not need to be accessed immediately
• Glacier Deep Archive, for archival data that is accessed rarely if ever
• (New!) Intelligent Tiering, for data with varying or unknown access needs

Perhaps your application has some objects that are accessed many times a day, and others that can be stored for months without needing to be retrieved. Maybe you are currently using Lifecycle Configuration Management and are wondering if a switch to this new Intelligent Tiering class is right for your objects. In this blog post, we will compare these two options for cost-effective and performance-optimized S3 storage: Intelligent Tiering and Lifecycle Configuration.

What is Intelligent Tiering?

Amazon S3 Intelligent Tiering is the newest storage class in S3. Objects in the Intelligent Tiering storage class are monitored for access patterns and placed automatically in the most cost-effective and performant storage class based on those patterns. One thing to note, however, is that Intelligent Tiering will NOT move objects to One-Zone IA or Glacier, as it only supports S3 Standard and S3 Standard-IA at this time.

If a given object in S3 Intelligent Tiering has not been accessed for 30 consecutive days, it is automatically moved to Infrequent Access storage. Then, if an object in IA is accessed, it will be returned to frequent access. This way you can automatically realize cost savings from moving objects to Standard-Infrequent Access in a way that fits into your application’s variable timeline, rather than blindly moving objects based on arbitrary periods of time. This makes Intelligent Tiering a great option for storage of objects with unpredictable access needs.

Intelligent Tiering has no additional charges for moving objects between the frequent and infrequent access tiers, as well as no retrieval costs for accessing objects in S3-IA. However, there is a cost for monitoring the objects.

What is Lifecycle Configuration?

Object Lifecycle Management is supported by all S3 storage classes. It is a set of rules or policies defined by you to move and/or delete projects according to your desired timeline. Some common use cases for employing lifecycle policies are for objects which need to be accessed multiple times for a few weeks, and then are never accessed again, such as logs. Perhaps compliance needs dictate that all logs are retained, but keeping them in S3 Standard is not at all cost-effective for objects that are infrequently or never accessed.

A lifecycle policy can be applied to an entire bucket, a group of objects (filtered by prefix or tag), or to a single object.

Comparison

This is why you’re here! You want to know which of these options makes the most sense for your use case. Unfortunately, “it depends” will almost always be the answer, but hopefully this comparison will give you the tools to make an informed decision.

Let’s say I have 10,000 objects in a bucket in us-west-2 (Oregon). Each object is 1GB in size. I’m not sure what the access patterns of these objects will be yet. All pricing data was taken from AWS S3 Pricing, and is subject to change.

Scenario 1: Intelligent Tiering

The table below summarizes a breakdown of the monthly costs for this scenario. Following the table is a detailed explanation of the costs.

Month 1

Because I’m not sure which of my 10,000 objects need to be accessed frequently and which can move to an infrequent or archival storage tier, I upload all 10,000 objects into the Intelligent Tiering class. For the first month, I pay the frequent access tier price as well as the monitoring price:

Storage: 10,000 GB = 10TB, priced at $0.023 per GB = $230.00
Monitoring: 10,000 objects, priced at $0.0025 per 1,000 objects = $0.025

Month 2

Let’s say it turns out that 5,000 of my objects need frequent (daily) access, and the other 5,000 are accessed infrequently enough to be moved to S3-IA. After 30 consecutive days of not being accessed, 5,000 of my objects are automatically moved to S3-Infrequently Accessed with no action required on my part. Now I pay the lower storage cost for half of my objects, while avoiding the hassle of moving objects manually, or monitoring access manually. I also forego paying retrieval costs for objects that are accessed from IA.

Storage:
5,000 GB priced at $0.023 per GB = $115.00
5,000 GB priced at $0.0125 per GB = $62.50
Monitoring: 10,000 objects, priced at $0.0025 per 1,000 objects = $0.025

Total cost for two months: $407.55

Month 3

In month 3 of this scenario, I unexpectedly need to access 2,500 of my objects that were moved to S3-IA automatically. Because they are in Intelligent Tiering, I pay no retrieval cost, and these objects are moved back to S3-standard. For this month I pay for 7,500 GB at the standard price, 2,500 at the IA price, and the monitoring price for all objects.

Storage:
7,500 GB priced at $0.023 per GB = $172.50
2,500 GB priced at $0.0125 per GB = $31.25
Monitoring: 10,000 objects, priced at $0.0025 per 1,000 objects = $0.025

Total cost for Scenario 1: $611.30

Scenario 2: Lifecycle Configuration

The table below summarizes a breakdown of the monthly costs for this scenario. Following the table is a detailed explanation of the costs.

Month 1

I upload all 10,000 objects to a bucket in S3 Standard. For the first month, I pay the S3 Standard price for all 10 TB. I choose to monitor the data access patterns using S3 Storage Class Analysis, which costs $0.10 per million objects monitored per month. So, for the first month I pay the S3-Standard price for all objects and the Storage Class Analysis price:

Storage: 10,000 GB = 10TB, priced at $0.023 per GB = $230.00
Monitoring: 10,000 objects, priced at $0.10 per million objects = $0.10

Month 2

After 30 days, I check my Storage Class Analysis dashboard, and find all 10,000 objects in the 30-44 days old category, as expected. Half of them show being frequently accessed over the last 30 days, while the other half were infrequently accessed.

I can now identify the 5,000 infrequently accessed objects to be moved to S3-IA, and that would require some manual tagging and/or sorting of objects. After tagging the objects I want to apply the lifecycle policy to, I create a lifecycle policy for my S3 bucket, specifying the scope to affect the infrequently accessed objects. Because objects must be stored in S3-Standard for a minimum of 30 days before transitioning to S3-IA, I would pay the same price for month two as month one:

Storage: 10,000 GB = 10TB, priced at $0.023 per GB = $230.00
Monitoring: 10,000 objects, priced at $0.10 per million objects = $0.10

Total cost for two months: $460.20

Month 3

For month 3 of this scenario, I would pay for 5,000 objects at the standard price for the first 50TB, and for 5,000 objects at the S3-IA price. I choose to turn off Storage Class Analysis at this time because I have already identified my access patterns.

Storage:
5,000 GB priced at $0.023 per GB = $115.00
5,000 GB priced at $0.0125 per GB = $62.50

Should I need to retrieve 2,500 objects from S3-IA as I did in scenario 1, I would incur the following retrieval fees:

S3-IA Retrieval: 2,500 GB priced at $0.01 per GB = $25.00

Total cost for scenario 2: $662.70

Conclusion

Clearly, both scenarios mentioned above are oversimplifications of real-world examples. At this small of a scale, both options were comparable in cost with a difference of only $51.40 over the course of three months. It is likely to take more time to analyze what your object access patterns look like. Your objects are also not likely to be 1GB each in size. Perhaps your situation calls for a combination of the two scenarios, using Intelligent Tiering with Lifecycle Policies to move data to more cost-effective storage classes. It is up to you to determine which storage class management makes the most sense for your data.

Some general guidelines when making decisions about how to cost-effectively store your data are to ask these questions:

1. What do the access patterns look like for objects in this bucket? Are they known?
2. Is access to objects predictable or unpredictable?
3. Would I prefer to monitor the objects myself, or let S3 handle monitoring and moving objects between classes for a fee?

Keep in mind that whenever your objects have a defined lifecycle, such as needing frequent access for a month and then never needing to be accessed again, a lifecycle policy is the most efficient and cost-effective solution for you. If your access patterns are unpredictable or difficult to identify, Intelligent Tiering is an excellent solution for hands-off storage access monitoring while still taking advantage of cost savings from S3-IA. Also note that S3-Intelligent Tiering supports lifecycle policies, so you could even configure rules to keep objects in Intelligent Tiering for a few months, and then move them to S3-Glacier.

If you want assistance analyzing your access patterns and customizing your storage solutions on AWS, or on any other AWS-related tasks, reach out to the 1Strategy team at info@1strategy.com and we will be happy to help!