Skip to content

Instantly share code, notes, and snippets.

@afloesch
Last active August 9, 2019 22:40
Show Gist options
  • Save afloesch/ebbb911cbda3d2d4f7e30a62ffa14509 to your computer and use it in GitHub Desktop.
Save afloesch/ebbb911cbda3d2d4f7e30a62ffa14509 to your computer and use it in GitHub Desktop.
Python signed requests to AWS Elasticsearch

Python Signed Requests to AWS Elasticsearch

AWS provides a basic example for making signed requests to managed Elasticsearch clusters, but the example doesn't cover how to deal with EC2 instance profiles, which will require passing the session token.

Here's a simple example, which also handles authenticating when using an instance profile instead of a generated user level access key and secret, leverages the elastic bulk upload API to add multiple records at the same time, and threads the request so it can be made in a non-blocking fashion.

from elasticsearch import Elasticsearch, RequestsHttpConnection, helpers
from requests_aws4auth import AWS4Auth
import boto3
import json
import uuid
import threading
import datetime
def upload(records):
credentials = boto3.Session().get_credentials()
if credentials.token != None:
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, 'us-east-1', 'es', session_token=credentials.token)
else:
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, 'us-east-1', 'es')
es = Elasticsearch(
hosts = [{'host': 'es-asdf1234.us-east-1.es.amazonaws.com', 'port': 443}],
http_auth = awsauth,
use_ssl = True,
verify_certs = True,
connection_class = RequestsHttpConnection
)
body = []
for r in records:
i = {}
i["_id"] = str(uuid.uuid4())
i["_index"] = 'Elastic-Index-' + str(datetime.datetime.now().date())
i["_type"] = 'Elastic-Type'
i["_source"] = json.dumps(r)
body.append(i)
t = threading.Thread(target=helpers.bulk, args=(es, body,))
t.start()
t.join()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment