giefferre

# read a part of the whole datalake just to extract the schema
part = spark.read.json("s3a://path/to/json/part")

# create a temporary rdd in order to store the schema as binary file
temp_rdd = sc.parallelize(part.schema)
temp_rdd.coalesce(1).saveAsPickleFile("s3a://path/to/destination_schema.pickle")

# from now on, the schema will be saved.
# it could be used to improve the speed of reading json files.

giefferre

Keybase proof

I hereby claim:

• I am giefferre on github.
• I am giefferre (https://keybase.io/giefferre) on keybase.
• I have a public key ASC5MZ4LhY4fbsPqLY6kt-PA8ifPTGW5zEAt5P-jkDmLCwo

To claim this, I am signing this object:

giefferre

package main

import (
	"log"
	"log/syslog"
)

func main() {
	logwriter, e := syslog.Dial("tcp","123.123.123.123:12345", syslog.LOG_DEBUG, "your.software.identifier")
	if e != nil {