Skip to content

Instantly share code, notes, and snippets.

@justinTM
justinTM / write_previous_deployment_yml.py
Created April 6, 2022 20:29
Downloads a GitLab pipeline job artifact based on branch name, status, and job name criteria.
import gitlab
import os
import subprocess
from gitlab.v4.objects.jobs import ProjectJob
from gitlab.v4.objects.pipelines import ProjectPipeline
def pprint_job(job: ProjectJob):
return f"ProjectJob<name={job.attributes.get('name', None)}, finished_at={job.attributes.get('finished_at')[0:16]}>"
@justinTM
justinTM / example.py
Created March 2, 2022 03:57
Apache Spark RDD parallelize pipe JSON file through jq multi-core
import os
from pyspark.sql import SparkSession
# create the spark session on a cluster of multiple cores
DIR_JSONS = '/tmp/in/jsons'
SPARK = SparkSession.builder.appName('APP_NAME').getOrCreate()
sc = SPARK.sparkContext
# execute a shell command using python
def os_shell_jq(filepath):
@justinTM
justinTM / 2021-12-28-tf-trace-grafana-plugin-crash.txt
Created December 28, 2021 18:33
Terraform TRACE level debug output from Grafana Provider Synthetic Monitoring plugin crash
grafana_synthetic_monitoring_check.all: Creating...
2021-12-28T18:29:30.583Z [WARN] Provider "registry.terraform.io/grafana/grafana" produced an invalid plan for grafana_synthetic_monitoring_check.all, but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .basic_metrics_only: planned value cty.True for a non-computed attribute
- .frequency: planned value cty.NumberIntVal(60000) for a non-computed attribute
- .probes: planned value cty.NullVal(cty.Set(cty.Number)) does not match config value cty.SetValEmpty(cty.Number)
- .alert_sensitivity: planned value cty.StringVal("none") for a non-computed attribute
- .timeout: planned value cty.NumberIntVal(3000) for a non-computed attribute
2021-12-28T18:29:30.583Z [TRACE] checkPlannedChange: Verifying that actual change (action Create) matches planned change (action Create)
2021-12-28T18:29:30.583Z [TRACE] readResourceInstanceSt
@justinTM
justinTM / 2021-12-17-error-console.txt
Created December 28, 2021 16:59
grafana_synthetic_monitoring_check.all: Creating... 84╷ 85│ Error: Plugin did not respond
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# grafana_synthetic_monitoring_check.all will be created
+ resource "grafana_synthetic_monitoring_check" "all" {
+ alert_sensitivity = "none"
+ basic_metrics_only = true
+ enabled = true
+ frequency = 60000
@justinTM
justinTM / error.txt
Created December 15, 2021 01:08
Console output Error: grafana_synthetic_monitoring_probes
Error: authorization token required
59│
60│ with data.grafana_synthetic_monitoring_probes.main,
61│ on probes.tf line 2, in data "grafana_synthetic_monitoring_probes" "main":
62│ 2: data "grafana_synthetic_monitoring_probes" "main" {}
63│
64╵
@justinTM
justinTM / get_gitlab_project_id_from_project_path.sh
Last active November 1, 2021 03:42
gitlab get project id from path with namespace from open merge requests. jq iterate string in array, jq select get matching item in list element in array, jq string split
#!/bin/sh
OPEN_MERGE_REQUESTS="[\"my/project!14\", \"my/other/project!2\"]"
# parse input args and environment variables
if [ -z "$OPEN_MERGE_REQUESTS" ]; then echo "$0: need OPEN_MERGE_REQUESTS"; exit 1; fi
if [ -z "$GITLAB_TOKEN" ]; then echo "$0: need GITLAB_TOKEN"; exit 1; fi
curl_token="PRIVATE-TOKEN: ${GITLAB_TOKEN}"
@justinTM
justinTM / confluence_python_get_page_source.py
Created September 29, 2021 01:00
Confluence will not show the true underlying string contents as Confluence Storage Format. This helps.
import os
from atlassian import Confluence
CONFLUENCE = Confluence(
url=f"https://{os.environ['CONFLUENCE_SITE']}.atlassian.net",
username=os.environ['CONFLUENCE_USER'],
password=os.environ['CONFLUENCE_TOKEN'],
cloud=True)
@justinTM
justinTM / make-grafana-cloud-alerting-yaml-rules-from-dashboard-alerts-by-dashboard-slug.sh
Created July 16, 2021 07:12
given an environment variable for the grafana api token and any number of dashboard slug arguments, generate prometheus yaml rules config files from legacy grafana dashboard alerts. makes a lot of assumptions like there only being one condition in each alert (one query reference). requires hand-editing of results rule ".expr" attributes to pass …
if [ -z "${GRAFANA_API_TOKEN}" ]; then
echo Error: missing environment variable: GRAFANA_API_TOKEN
exit 1
fi
DASHBOARD_SLUGS="${*}"
if [ -z "${DASHBOARD_SLUGS}" ]; then
echo Error: missing argument1...n: DASHBOARD_SLUGS
exit 1
@justinTM
justinTM / format-diff-file-as-markdown.sh
Last active June 2, 2022 20:58
Format cortextool diff output as Markdown diff code section
#!/bin/bash
# this script will:
# 1. read in file argument 1, $DIFF_FILE_IN
# 2. format text to Markdown diff code section
# 3. write out file argument 2, $DIFF_FILE_OUT
# https://gitlab.com/gitlab-org/gitlab/-/issues/15582
set -e
@justinTM
justinTM / GetSharedDriveDownloadLinksFromFolderId.gs
Last active May 4, 2020 18:07
Write a CSV to Google Drive folder (by id) for filenames and download URLs of all files within a Shared Drive folder (by Drive id and folder id)
/**
* Writes a CSV file to a Drive folder (by id) containing filenames and Google Drive download urls (File.webContentLink)
* for all files in a specified folder (by id) of a Shared Drive (by id).
*/
function ExportFilesFromSharedDriveToCSV() {
// id of Shared Drive (go to Shared Drive, copy-paste id from browser URL)
var sharedDriveId = "0AFR1j1Oo3fnlUk9PVA";
// id of folder in Shared Drive to search (go to folder, copy-paste if from browser URL)
var folderIdToSearch = "1Ua6mIE8kL4Q1r-LEEVmAlo-oyrsY2gg9";