Skip to content

Instantly share code, notes, and snippets.

View GregHilston's full-sized avatar

Greg Hilston GregHilston

View GitHub Profile
@GregHilston
GregHilston / metaflow_conda_environment_error_local_execution_s3_datastore.txt
Created December 11, 2020 15:12
The following is my Metaflow conda environment error when running locally, on OSX, using a S3 datastore. I do not get any environment issues when running remotely on AWS Batch and get a different set of Conda issues when running locally with a local datastore.
CONDA_CHANNELS=anaconda,conda-forge,maciejkula python3 flows/my_flow.py --environment=conda --datastore=s3 --no-pylint run 130 ?
Metaflow 2.2.5 executing MyFlow for user:ghilston
Validating your flow...
The graph looks good!
cli.py about to create a MetaflowPackage
MetaflowPackage about to init_environment for flow {flow} in environment {environment}
2020-12-11 09:47:33.212 Bootstrapping conda environment...(this could take a few minutes)!
step <bound method MyFlow.start of <__main__.MyFlow object at 0x7fb8c833ff98>> of type <class 'method'>
deco conda<decorated libraries={} python=None disabled=True> of type <class 'metaflow.plugins.conda.conda_step_decorator.CondaStepDecorator'>
step <bound method MyFlow.get_data of <__main__.MyFlow object at 0x
@GregHilston
GregHilston / metaflow_conda_environment_error_local_execution_local_datastore.txt
Last active December 11, 2020 15:12
The following is my Metaflow conda environment error when running locally, on OSX, using a local datastore. I do not get any environment issues when running remotely on AWS Batch and get a different set of Conda issues when running locally with a S3 datastore.
CONDA_CHANNELS=anaconda,conda-forge,maciejkula python3 flows/my_flow.py --environment=conda --no-pylint run 130 ?
Metaflow 2.2.5 executing MyFlow for user:ghilston
Validating your flow...
The graph looks good!
cli.py about to create a MetaflowPackage
MetaflowPackage about to init_environment for flow {flow} in environment {environment}
2020-12-11 09:47:30.165 Bootstrapping conda environment...(this could take a few minutes)!
step <bound method MyFlow.start of <__main__.MyFlow object at 0x7fbfb89cff98>> of type <class 'method'>
deco conda<decorated libraries={} python=None disabled=True> of type <class 'metaflow.plugins.conda.conda_step_decorator.CondaStepDecorator'>
step <bound method MyFlow.get_data of <__main__.MyFlow object at 0x
@GregHilston
GregHilston / wpa_supplicant.conf
Last active April 21, 2019 05:24
Add an empty file called "ssh" and this file to the root of your Raspberry Pi SD card to automate the connection to wifi
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=US
network={
ssid=""
psk=""
key_mgmt=WPA2-PSK
}
@GregHilston
GregHilston / file_cleaner.py
Last active August 24, 2024 10:03
Deletes all files that are missing a given string in their name
# Hey everyone, I was having a tough time cleaning up my ROMs, as there would be multiple versions of the same game, like:
# Super-Smash(Japan).rom
# Super-Smash(Europe).rom
# Super-Smash(USA).rom
# I ended up writing this small Python (3.6) script that you can use to cleanup duplicate files.
# Execution looks like this, if you want to keep only USA files:
@GregHilston
GregHilston / test.ipynb
Created May 2, 2018 22:20
Testing hour gists handle ipynb files
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@GregHilston
GregHilston / visualizing-geospatial-data-read.py
Created April 25, 2018 18:15
First we'll create a `Pandas.DataFrame` out of a `json` file hosted by NASA.
# Data from NASA on meteorite landings
df = pd.read_json(data_directory + "y77d-th95.json")
@GregHilston
GregHilston / visualizing-geospatial-data-setup.py
Created April 25, 2018 18:14
Prepares our notebook, defining our imports and global variables
# ensure our graphs are displayed inline
%matplotlib inline
import os
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sb
import numpy as np
import folium
from folium import plugins
@GregHilston
GregHilston / gist-to-png.js
Last active April 18, 2018 18:25 — forked from fgeorges/blog--gist-to-png-01.js
This Gist assists you in creating a screen shot of a Gist. I use it to post screen shots on LinkedIn.
#!/usr/local/bin/phantomjs
var page = require('webpage').create();
var system = require('system');
// ***** Argument handling
function isInt(value) {
return !isNaN(value) &&
parseInt(Number(value)) == value &&
# contains helpful premade actions we'll be using later.
# Anything that starts with os. came from this
import os
# gets a list of all the items in the current directory
file_names = os.listdir(".")
# we'll loop over every file in the current directory
for file_name in file_names:
# define the prefix we're looking to remove
@GregHilston
GregHilston / eprint.py
Created December 18, 2017 19:31
Prints a message to stderr
import sys
def eprint(*args, **kwargs):
"""Prints a message to stderr
:param args: variable amount of arg Strings to print to stderr
:param kwargs: variable amount of keyword arg Strings to print to stderr
"""