S3 Object Querying with JMESPath

A quick post with some useful querying patterns when using JMESPath queries to find keys in a target S3 bucket.

Finding and filtering with JMESPath expressions

Find keys ending or starting with a certain value, and sort by Size

Here is a JMESPath query using s3api to find and sort keys based on the ending with a certain value, with the sort then being applied based on the resulting key sizes.

aws s3api list-objects-v2 --bucket example-bucket --query "Contents[?ends_with(Key, 'example')] | sort_by(@, &Size)"

To do the same as above, but for keys starting with a specific value, change the ends_with boolean expression to starts_with.

List all objects in the bucket, selecting only specific target keys, you can use a command like:

aws s3api list-objects-v2 --bucket example-bucket --query "Contents[*].[Key,Size]"

To refine that down to the first 3 x items only, add [-3:] to the end. For example:

aws s3api list-objects-v2 --bucket example-bucket --query "Contents[*].[Key,Size][-3:]"

Pipe operator

The pipe operator is used to stop projections in the query, or group expressions together.

Here is an example of filtering objects in a bucket down, followed by another expression to find only those with a key containing the value example_string:

aws s3api list-objects-v2 --bucket example-bucket --query "Contents[*] | [? contains(Key, 'example_string')]"

Another example, filtering down to include only objects on the STANDARD StorageClass, and then only those starting with a specific value:

aws s3api list-objects-v2 --bucket example-bucket --query "Contents[?StorageClass == 'STANDARD'] | [? starts_with(Key, 'ffc302a')]"

Transforming property names

Transforming keys / properties can be done using curly braces. For example, Key can be changed to become lowercase key:

aws s3api list-objects-v2 --bucket example-bucket --query "Contents[*].{key:Key}[-1:]"

This can be useful if you have a large, nested object structure and wish to create a short property in the pipeline for use in expressions further down the line. This wouldn’t be the case in the S3 object structure we’re primarily working with here, but a query example would be:

"InitialResults[*].{shortkey:Some.Nested.Object.Key} | [? starts_with(shortkey, 'example')]"

Leave a Comment