Aslan

Setting up ssh public key authentication on macOS using a YubiKey 4

I largely followed Florin's blog post, but have a few notes to add regarding issues I encountered:

Basic setup notes

1. I used a YubiKey 4, while the blog describes using a YubiKey NEO. I'm sure a YubiKey 5 would also work. I'm also running macOS 10.13.6.
2. I installed GPGTools as recommended. However, as I'll note later, it seems that gpg-agent only automatically starts when gpg is used; for ssh, you'll need to ensure it's running.
3. Before generating your keys, decide what key size you want to use. If you run the list command inside gpg --edit-card, look for the Key attributes line to see what is currently selected. On my YubiKey 4, it defaulted to 2048 bits for all keys:

Key attributes ...: rsa2048 rsa2048 rsa2048

Aslan

Each of these commands will run an ad hoc http static server in your current (or specified) directory, available at http://localhost:8000. Use this power wisely.

Discussion on reddit.

Python 2.x

$ python -m SimpleHTTPServer 8000

Aslan

Initialize

gitflow	git
git flow init	git init
	git commit --allow-empty -m "Initial commit"
	git checkout -b develop master

Connect to the remote repository

Aslan

  * service[opscode-erchef] action nothing (skipped due to action :nothing)
Recipe: private-chef::opscode-erchef
  * runit_service[opscode-erchef] action enable
  Recipe: <Dynamically Defined Resource>
    * directory[/opt/opscode/sv/opscode-erchef] action create
      - create new directory /opt/opscode/sv/opscode-erchef
      - change mode from '' to '0755'
    * template[/opt/opscode/sv/opscode-erchef/run] action create
      - create new file /opt/opscode/sv/opscode-erchef/run
      - update content in file /opt/opscode/sv/opscode-erchef/run from none to 8ef29e

Aslan

❯❯❯ aws s3 ls s3://xxxxxxxxxxxxxx/production/MPX/2016-04-04/                                                                                                                             
2016-04-05 01:20:07          0 _SUCCESS
2016-04-05 01:16:41      93211 part-r-00000-68103d9b-8201-49c5-91f8-f832337648db.gz.parquet
2016-04-05 01:16:42      97383 part-r-00001-68103d9b-8201-49c5-91f8-f832337648db.gz.parquet
2016-04-05 01:16:43      92641 part-r-00002-68103d9b-8201-49c5-91f8-f832337648db.gz.parquet
2016-04-05 01:16:44      97808 part-r-00003-68103d9b-8201-49c5-91f8-f832337648db.gz.parquet
2016-04-05 01:16:45      96843 part-r-00004-68103d9b-8201-49c5-91f8-f832337648db.gz.parquet
❯❯❯ aws s3 ls s3://xxxxxxxxxxxxxxxx/production/MPX/2016-04-05/                                                                                                     
2016-04-06 01:21:08          0 _SUCCESS
2016-04-06 01:17:45      93367 part-r-00000-ba75a2d7-6e02-45f6-a1fe-dca02e3f2620.gz.parquet

Aslan

org.apache.spark.SparkException: Job aborted.
	at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:156)
	at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
	at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
	at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:108)
	at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57)
	at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57)
	at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:69)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:140)

Aslan

# upgrade from java 1.6 to 1.7
apt-get install openjdk-7-jre
update-alternatives --config java # chose java 1.7

# install scala 2.10.4 (same version as AWS EMR)
apt-get remove scala-library scala
wget http://www.scala-lang.org/files/archive/scala-2.10.4.deb
apt-get update
apt-get install scala

Aslan

[hadoop@ip-10-65-200-150 ~]$ /usr/bin/spark-submit --master yarn-cluster --class com.truex.prometheus.CLIJob /home/hadoop/Prometheus-assembly-0.0.1.jar  -e 'select x.id, x.title, x.description, x.mediaavailableDate as available_date, x.mediaexpirationDate as expiration_date, mediacategories.medianame as media_name, x.mediakeywords as keywords, mediaratings.scheme as rating_scheme, mediaratings.rating, cast(mediaratings.subRatings as String) as sub_ratings, content.plfileduration as duration, x.plmediaprovider as provider, x.ngccontentAdType as ad_type, x.ngcepisodeNumber as episode, ngcnetwork as network, x.ngcseasonNumber as season_number, x.ngcuID as ngc_uid, x.ngcvideoType as video_type from etl lateral view explode(entries) entries as x lateral view explode(x.mediacategories) cat as mediacategories lateral view explode(x.mediaratings) r as mediaratings lateral view explode(x.mediacontent) mediacontent as content lateral view outer explode(x.ngcnetwork) net as ngcnetwork' -j http://feed.theplatform.com/f/n

Aslan

15/10/26 19:19:38 INFO client.RMProxy: Connecting to ResourceManager at ip-10-65-200-150.ec2.internal/10.65.200.150:8032


Container: container_1444274555723_0062_02_000003 on ip-10-169-170-124.ec2.internal_8041
==========================================================================================
LogType:stderr
Log Upload Time:26-Oct-2015 19:18:20
LogLength:9332
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.

Aslan

15/10/24 01:00:34 INFO client.RMProxy: Connecting to ResourceManager at ip-10-65-200-150.ec2.internal/10.65.200.150:8032


Container: container_1444274555723_0060_01_000003 on ip-10-169-170-124.ec2.internal_8041
==========================================================================================
LogType:stderr
Log Upload Time:24-Oct-2015 01:00:10
LogLength:4299
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.