Skip to content

Instantly share code, notes, and snippets.

@colegatron
colegatron / delete-millions-s3-files.sh
Created January 4, 2021 14:42
How to delete quickly millions of files on AWS S3 (fastest, quickest way)
#!/bin/bash
BUCKET="mybucket"
PREFIX="elasticsearch-backup/all-indexes"
[ "$(which pv)" != "" ] && echo "Required: 'apt install -y pv'" && exit 1
aws s3api list-objects --output text --bucket $BUCKET --prefix $PREFIX --query 'Contents[].[Key]' | pv -l > /tmp/${BUCKET}.keys
split -d -l 1000 /tmp/${BUCKET}.keys ${BUCKET}-
# If for any reason the process stops (you loose network connection, for instance), you can continue the job from here
for i in $( ls ${BUCKET}-* )
@colegatron
colegatron / .bashrc
Last active March 15, 2022 13:48
Shell prompt with support for Kubernetes, Docker and Git
# Add to your ~/.bashrc:
source /home/user/bin/myprompt.sh
PROMPT_COMMAND=myprompt
#Enjoy. Any improvement is welcome.
@colegatron
colegatron / gist:7aa726be4417d01636e288418bb1f500
Created January 31, 2019 11:08 — forked from tjcorr/gist:3baf86051471062b2fb7
CloudFormation to demo etcd-aws-cluster
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "An etcd cluster based off an auto scaling group",
"Mappings" : {
"RegionMap" : {
"eu-central-1" : {
"AMI" : "ami-840a0899"
},
"ap-northeast-1" : {
"AMI" : "ami-6c5ac56c"
@colegatron
colegatron / c.py
Created August 21, 2018 10:58
Convert configuration YAML file into properties file or shell environment variables
#!/usr/bin/python3
import os
import re
import yaml
import sys
if not os.getenv("CONFIG_FILE"):
print("Usage:")
print("$ CONFIG_FILE=the/config.yaml " + sys.argv[0] + " < exports to yaml by default")
print("$ CONFIG_FILE=the/config.yaml " + sys.argv[0] + " yaml")
usix15y22zbix30y14zbix29y24zbix29y19zbix29y21zbix30y16zbix30y12zbix30y7zbix30y9zbmix14y20r20zpix10y23zdtix45y19zbix43y19zdtix43y17zlsix68y23zlsix67y22zlsix68y19zeix57y23zeix52y24zeix48y23zeix8y8zeix10y14znetix43y11znwtix52y10zutix53y8zpix28y15zbmix25y17r7zbmix24y12r6zpix28y11zbmix55y6r8zbmix23y7r13zpix44y-4zlsvx62y24zbmvx71y4r12zbmvx65y20r0zpvx41y17zpix39y18zpix37y20zpix36y20zusix33y18zlsix39y18zlsix39y18zevx33y22zevx54y12zdtvx67y-4zutvx72y23zpvx74y23zutvx76y22z
@colegatron
colegatron / _etc_init_web-a5.conf
Last active May 31, 2018 15:21
Run node app as a service forwarding logs to local syslog. Can also be remote syslog using logger -n
# This is one of those little treasures hidden in the filesystem
# you always forget how you did it and which server you have it running on
# so, here it is
#
description "node web-a5-app"
start on (net-device-up and local-filesystems and runlevel [2345])
stop on runlevel [!2345]
respawn
script
[ -p /var/run/web-a5-out ] && rm /var/run/web-a5-out
@colegatron
colegatron / crontab every 5 min
Created September 30, 2016 07:31
Java too much open files; restart and notify
*/5 * * * * [ $( lsof -p $(pgrep java) | wc -l ) -gt 60000 ] && service tomcat7 restart && notify-adm "${HOSTNAME}: [ERROR] JVM open files limit reached, restarting."
@colegatron
colegatron / bash
Created August 25, 2016 17:57
delete all old kernels
sudo apt-get purge $(\
for tag in "linux-image" "linux-headers"; \
do dpkg-query -W -f'${Package}\n' "$tag-[0-9]*.[0-9]*.[0-9]*" | \
sort -V | \
awk 'index($0,c){exit} //' c=$(uname -r | cut -d- -f1,2); \
done)
@colegatron
colegatron / Move Ubuntu to the new machine
Created July 21, 2016 13:57
Move Ubuntu to the new machine
Whenever is possible, try to do not install anything on your desktop.
Use virtualbox.
Better, use Docker.
Don't feel bad. I have lots of shit installed in my desktop too.
------
On source machine:
@colegatron
colegatron / README-saltstack-postfix-forwarder
Last active May 12, 2016 09:48
saltstack postfix formula as forwarder (for root emails)
This is an update of the Russel's state to use it basically as mail forwarder in auto scaled environments.
The tipycal problem is that if you get notification emails from "[email protected]" in an environment where you really have 5 or 6 webservers for the same application, it is difficult to track back the issue to the right server.
I have solved it renaming the server and the headers:
All the emails sent from the server for <myhostname>.domain.com will renamed to <senderuser>@<myhostname-my-ipv4>.domain.com.
Also adds a header replacement to avoid problems with Office365 mail servers, which rejects to send emails with different names on the hostname and the email headers
Original job: http://russell.ballestrini.net/postfix-salt-state-formula/