Unexplained S3 ListBuckets cost in cost explorer

Question

We are trying to optimise AWS S3. The cost explorer shows very high cost of ListBuckets calls.

To get the source of these calls, we enabled the CloudTrails with configuration to log all the management and data eventtype logs for S3. In the cloudtrail logs, there is not even 0.0001% of the calls which are reflected in the cost explorer. What can be the possible reasons for this?

Is the ListBuckets in cost explorer related to some other API in CloudTrail? Or are the ListBuckets events not getting logged in CloudTrail becuase am I doing something wrong while configuring the CloudTrails?

• jellycsc: Are you talking about the s3:ListBucket action?
• John Rotenstein: The API call is ListBuckets - Amazon Simple Storage Service, which needs the s3:ListBucket permission. According to Amazon S3 CloudTrail events - Amazon Simple Storage Service, this bucket-level action should be included in CloudTrail. Is the Trail configured to record actions in all regions?
• Aditya Agarwal: @JohnRotenstein! Yes, the CloudTrail was configured to record actions in all regions. A few ListBuckets events also showed up in the CloudTrails logs, but the number is nowhere close to the one in the cost explorer.
• Aditya Agarwal: @jellycsc! In the cost explorer, there is a filter called 'API operation'. In that filter, the operation name is 'ListBucket', which is responsible for high costs.

Answer 1

According to Actions, resources, and condition keys for Amazon S3 - Service Authorization Reference:

ListBucket action
Grants permission to list some or all of the objects in an Amazon S3 bucket (up to 1000)

I always find this naming confusing. The ListBucket operation lists the contents of buckets, not to be confused with ListBuckets permission that is associated with the ListAllMyBuckets action. Confusing indeed!

So, I think it is actually charging you for listing the contents of buckets. In boto3 this is called list_objects(). So, look for anything in Cost Explorer that might talk about listing objects within a bucket and it will probably show more data.

Answer 2

I assume you mean ListBucket, and I had a similar problem. You need to enable S3 server access logging to see the requester IAM role. I tried Cloudtrail S3 data events and found they don't cover ListBucket calls. This is done after you figure which S3 bucket the calls are made to with Cost Explorer.

Answer 3

I observed the same issue in AWS S3: high amount of ReadAccountPublicAccessBlock and ListAllMyBuckets in Cost Explorer, but almost none in CloudTrail. After consulting with AWS Support here are my findings:

• CUR ReadAccountPublicAccessBlock is GetAccountPublicAccessBlock in CloudTrail
• CUR ListAllMyBuckets is ListBuckets in CloudTrail

The confusing part is that GetAccountPublicAccessBlock, and ListBuckets are service/account level requests. This means that the GetAccountPublicAccessBlock, and ListBuckets requests made to your home region, e.g. eu-west-1 will be reflected under the us-east-1 region in Cost Explorer. But CloudTrail will still reflect the correct region the request was actually made to.

So in our case, the CloudTrail logs were there, just in other region, not us-east-1.

If you are doing some experiments, watch out, there is a similar api call for getting bucket level public access block, like aws s3api get-public-access-block --bucket bucket-name --region us-east-1. If your bucket is not in us-east-1 region, aws cli will do a hidden redirect, and get results from a proper region, so your call would be seen in Cost Explorer results for proper region.