Skip to content

Instantly share code, notes, and snippets.

@colegatron
Created January 4, 2021 14:42
Show Gist options
  • Save colegatron/00c0822c7d6e5eafdb871d6b14af6ee7 to your computer and use it in GitHub Desktop.
Save colegatron/00c0822c7d6e5eafdb871d6b14af6ee7 to your computer and use it in GitHub Desktop.
How to delete quickly millions of files on AWS S3 (fastest, quickest way)
#!/bin/bash
BUCKET="mybucket"
PREFIX="elasticsearch-backup/all-indexes"
[ "$(which pv)" != "" ] && echo "Required: 'apt install -y pv'" && exit 1
aws s3api list-objects --output text --bucket $BUCKET --prefix $PREFIX --query 'Contents[].[Key]' | pv -l > /tmp/${BUCKET}.keys
split -d -l 1000 /tmp/${BUCKET}.keys ${BUCKET}-
# If for any reason the process stops (you loose network connection, for instance), you can continue the job from here
for i in $( ls ${BUCKET}-* )
do
echo "Deleting $i"
cat $i | pv -l | grep -v -e "'" | tr '\n' '\0' | xargs -0 -P1 -n1000 bash -c "aws s3api delete-objects --bucket $BUCKET --delete \"Objects=[$(printf \"{Key=%q},\" \"$@\")],Quiet=true\"" _
rm $i
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment