Mike Sparr mikesparr
mikesparr
/ gke-clickhouse-operator-parquet.sh
Created
October 21, 2024 23:59
Example deploying Clickhouse database operator on Google Kubernetes Engine (GKE) Autopilot cluster and importing parquet data from storage bucket using HMAC key
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| ##################################################################### | |
| # REFERENCES | |
| # - https://cloud.google.com/kubernetes-engine/docs/how-to/creating-an-autopilot-cluster | |
| # - https://cloud.google.com/storage/docs/creating-buckets | |
| # - https://cloud.google.com/storage/docs/access-control/using-iam-permissions | |
| # - https://cloud.google.com/storage/docs/authentication/managing-hmackeys#command-line | |
| # - https://github.com/Altinity/clickhouse-operator/blob/master/docs/quick_start.md | |
| # - https://clickhouse.com/docs/en/getting-started/quick-start |
mikesparr
/ gcp-asset-inv-bq-feed.sh
Last active
July 17, 2024 21:24
Experiment with GCP Asset Inventory feeds publishing to Pub/Sub and importing into BigQuery using subscriptions and Avro schema
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| ##################################################################### | |
| # REFERENCES | |
| # - https://cloud.google.com/asset-inventory/docs/monitoring-asset-changes#organizations | |
| # - https://cloud.google.com/asset-inventory/docs/reference/rest/v1/TopLevel/batchGetAssetsHistory#temporalasset | |
| # - https://cloud.google.com/iam/docs/creating-custom-roles#creating | |
| # - https://cloud.google.com/sdk/gcloud/reference/organizations/add-iam-policy-binding | |
| # - https://gist.github.com/mikesparr/38260f9d457fc2f9452f1910213b4a71 | |
| # - https://aravind-ramaiahk.medium.com/streamlining-cloud-asset-management-automating-the-export-of-google-cloud-organization-asset-46cc8b428038 |
mikesparr
/ cloud-run-nat-serial.sh
Last active
August 29, 2024 15:07
Experiment on Google Cloud with Cloud Run, Cloud NAT, Private Google Access, and Secure Web Proxy with NAT only for external requests
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| ##################################################################### | |
| # REFERENCES | |
| # - https://cloud.google.com/sdk/gcloud/reference/compute/networks/create | |
| # - https://cloud.google.com/sdk/gcloud/reference/compute/networks/subnets/create | |
| # - https://cloud.google.com/vpc/docs/configure-private-google-access | |
| # - https://cloud.google.com/network-connectivity/docs/router/how-to/create-router-vpc-network#gcloud | |
| # - https://cloud.google.com/nat/docs/set-up-manage-network-address-translation | |
| # - https://cloud.google.com/nat/docs/using-nat-rules |
mikesparr
/ cloud-next-24-demo.sh
Last active
April 8, 2024 03:57
Snippets from my Cloud Next '24 demo talk "Hello Gateway, Goodbye Ingress" in Las Vegas April 9, 2024
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| ##################################################################### | |
| # REFERENCES | |
| # - https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-api | |
| # - https://cloud.google.com/kubernetes-engine/docs/how-to/container-native-load-balancing | |
| # - https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/ | |
| # - https://cloud.google.com/gemini/docs/quickstart | |
| # - https://cloud.google.com/kubernetes-engine/docs/best-practices/rbac | |
| # - https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster#group_authentication |
mikesparr
/ gcp-secure-web-proxy-1.sh
Created
March 29, 2024 18:35
Experiment using Google Cloud Secure Web Proxy and Cloud NAT
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| ##################################################################### | |
| # REFERENCES | |
| # - https://cloud.google.com/secure-web-proxy/docs/initial-setup-steps | |
| # - https://cloud.google.com/certificate-manager/docs/deploy-google-managed-regional | |
| # - https://cloud.google.com/secure-web-proxy/docs/quickstart | |
| # - https://cloud.google.com/secure-web-proxy/docs/enable-tls-inspection (OPTIONAL) | |
| ##################################################################### |
mikesparr
/ gcp-gae-cr-private.sh
Last active
October 15, 2024 22:46
Example Google Cloud Platform (GCP) serverless apps communicating via private network
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| ##################################################################### | |
| # REFERENCES | |
| # - https://cloud.google.com/run/docs/securing/private-networking#from-other-services | |
| # - https://cloud.google.com/run/docs/securing/private-networking#from-vpc | |
| # - https://cloud.google.com/appengine/docs/flexible/disable-external-ip | |
| # - https://cloud.google.com/dns/docs/records#adding_or_removing_a_record | |
| # - https://cloud.google.com/vpc/docs/configure-private-google-access | |
| # - https://cloud.google.com/run/docs/quickstarts/build-and-deploy/deploy-go-service |
mikesparr
/ advanced-ddos-gcp.sh
Last active
March 6, 2024 22:24
Cheat sheet for enabling advanced DDoS protection on Google Cloud
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| ##################################################################### | |
| # REFERENCES | |
| # - https://cloud.google.com/armor/docs/managed-protection-using#gcloud | |
| # - https://cloud.google.com/armor/docs/advanced-network-ddos | |
| # - https://cloud.google.com/armor/docs/configure-adaptive-protection | |
| # - https://cloud.google.com/armor/docs/adaptive-protection-auto-deploy | |
| ##################################################################### |
mikesparr
/ 01_setup.sh
Created
March 6, 2024 20:25
Experiment using Langchain, OpenAI and Streamlit, along with FAISS for CPU vector store, that analyzes YouTube transcripts and answers questions
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| # reference: https://www.youtube.com/watch?v=lG7Uxts9SXs (some code was deprecated so fix along way) | |
| # create folder | |
| mkdir youtube-assistant | |
| cd youtube-assistant | |
| # set up virtual env | |
| python3 -m venv .venv |
mikesparr
/ word2vec_import.py
Created
March 5, 2024 00:40
Experiment with Word2Vec embedding of words in Python for study of GenAI and NLP solutions
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import pandas as pd | |
| import nltk | |
| import string | |
| import matplotlib.pyplot as plt | |
| from nltk.corpus import stopwords | |
| from nltk import word_tokenize | |
| from gensim.models import Word2Vec as w2v | |
| from sklearn.decomposition import PCA |
mikesparr
/ mini-qa.py
Last active
August 24, 2024 18:27
Experiment with Langchain, OpenAI, and Datastax AstraDB (vector database) for custom LLM Q&A bot in Python
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env python3 | |
| # Reference: https://www.youtube.com/watch?v=yfHHvmaMkcA (Tutorial - some dated libs fixed in here) | |
| # set constants | |
| ASTRA_DB_APPLICATION_TOKEN="<< YOUR APP TOKEN (starts with AstraCS:) >>" | |
| ASTRA_DB_SECURE_BUNDLE_PATH="<< FULL PATH TO YOUR ASTRA DB BUNDLE >>" | |
| ASTRA_DB_CLIENT_ID="<< YOUR ASTRA DB CLIENT ID >>" | |
| ASTRA_DB_CLIENT_SECRET="<< YOUR ASTRA DB CLIENT SECRET >>" | |
| ASTRA_DB_KEYSPACE="search" |
NewerOlder