A quick post with some useful querying patterns when using JMESPath queries to find keys in a target S3 bucket.
Finding and filtering with JMESPath expressions
Find keys ending or starting with a certain value, and sort by Size
Here is a JMESPath query using s3api to find and sort keys based on the ending with a certain value, with the sort then being applied based on the resulting key sizes.
aws s3api list-objects-v2 --bucket example-bucket --query "Contents[?ends_with(Key, 'example')] | sort_by(@, &Size)"
To do the same as above, but for keys starting with a specific value, change the ends_with boolean expression to starts_with.
List all objects in the bucket, selecting only specific target keys, you can use a command like:
aws s3api list-objects-v2 --bucket example-bucket --query "Contents[*].[Key,Size]"
To refine that down to the first 3 x items only, add [-3:] to the end. For example:
aws s3api list-objects-v2 --bucket example-bucket --query "Contents[*].[Key,Size][-3:]"
Pipe operator
The pipe operator is used to stop projections in the query, or group expressions together.
Here is an example of filtering objects in a bucket down, followed by another expression to find only those with a key containing the value example_string:
aws s3api list-objects-v2 --bucket example-bucket --query "Contents[*] | [? contains(Key, 'example_string')]"
Another example, filtering down to include only objects on the STANDARD StorageClass, and then only those starting with a specific value:
aws s3api list-objects-v2 --bucket example-bucket --query "Contents[?StorageClass == 'STANDARD'] | [? starts_with(Key, 'ffc302a')]"
Transforming property names
Transforming keys / properties can be done using curly braces. For example, Key can be changed to become lowercase key:
aws s3api list-objects-v2 --bucket example-bucket --query "Contents[*].{key:Key}[-1:]"
This can be useful if you have a large, nested object structure and wish to create a short property in the pipeline for use in expressions further down the line. This wouldn’t be the case in the S3 object structure we’re primarily working with here, but a query example would be:
"InitialResults[*].{shortkey:Some.Nested.Object.Key} | [? starts_with(shortkey, 'example')]"