AWS S3: listing directories with tailored AWS S3 API
We started to generate partitioned datasets to S3, so they are now spread in a great number of folders. And usually have 1 file in them.
And now I want to download/delete these folders and "listing" stage takes too much time going through folders 1-by-1.
AWS has an API
That should speed things up a lot. And I think it is also suitable/tailored to get all the keys from "sub-folders" quite easily.
Thanks for the great tool, btw.
And now I want to download/delete these folders and "listing" stage takes too much time going through folders 1-by-1.
AWS has an API
ListObjectsV2
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html
That should speed things up a lot. And I think it is also suitable/tailored to get all the keys from "sub-folders" quite easily.
Thanks for the great tool, btw.