Skip to content

Instantly share code, notes, and snippets.

View barakseri1's full-sized avatar

Barak barakseri1

View GitHub Profile
@barakseri1
barakseri1 / data_node
Created August 17, 2020 08:21
data node cant re-join the cluster
[2020-08-17T08:08:14,479][INFO ][o.e.e.NodeEnvironment ] [ip-172-30-1-61.ec2.internal] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/mapper/vg_elastic-lv_elastic)]], net usable_space [886.4gb], net total_space [984.1gb], types [ext4]
[2020-08-17T08:08:14,482][INFO ][o.e.e.NodeEnvironment ] [ip-172-30-1-61.ec2.internal] heap size [31gb], compressed ordinary object pointers [true]
[2020-08-17T08:08:14,622][INFO ][o.e.n.Node ] [ip-172-30-1-61.ec2.internal] node name [ip-172-30-1-61.ec2.internal], node ID [heumpdUlSRezT69xwljSgg], cluster name [elasticsearch-prod]
[2020-08-17T08:08:14,624][INFO ][o.e.n.Node ] [ip-172-30-1-61.ec2.internal] version[7.7.1], pid[24785], build[default/rpm/ad56dce891c901a492bb1ee393f12dfff473a423/2020-05-28T16:30:01.040088Z], OS[Linux/4.14.97-90.72.amzn2.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.1/14.0.1+7]
[2020-08-17T08:08:14,625][INFO ][o.e.n.Node ] [ip-172-30-1-61.ec2.internal] JVM home [/usr/share/el
@barakseri1
barakseri1 / gist:4e2ccffa185c918d5fe9f07974063e2a
Created August 17, 2020 08:19
data_node
[2020-08-17T08:08:14,479][INFO ][o.e.e.NodeEnvironment ] [ip-172-30-1-61.ec2.internal] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/mapper/vg_elastic-lv_elastic)]], net usable_space [886.4gb], net total_space [984.1gb], types [ext4]
[2020-08-17T08:08:14,482][INFO ][o.e.e.NodeEnvironment ] [ip-172-30-1-61.ec2.internal] heap size [31gb], compressed ordinary object pointers [true]
[2020-08-17T08:08:14,622][INFO ][o.e.n.Node ] [ip-172-30-1-61.ec2.internal] node name [ip-172-30-1-61.ec2.internal], node ID [heumpdUlSRezT69xwljSgg], cluster name [elasticsearch-prod]
[2020-08-17T08:08:14,624][INFO ][o.e.n.Node ] [ip-172-30-1-61.ec2.internal] version[7.7.1], pid[24785], build[default/rpm/ad56dce891c901a492bb1ee393f12dfff473a423/2020-05-28T16:30:01.040088Z], OS[Linux/4.14.97-90.72.amzn2.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.1/14.0.1+7]
[2020-08-17T08:08:14,625][INFO ][o.e.n.Node ] [ip-172-30-1-61.ec2.internal] JVM home [/usr/share/el
@barakseri1
barakseri1 / gist:30ee99c6d135c74f0115e23babe1219f
Created August 16, 2020 14:35
elasticsearch log
[2020-08-16T14:24:45,784][WARN ][o.e.d.HandshakingTransportAddressConnector] [ip-172-30-0-168.ec2.internal] handshake failed for [connectToRemoteMasterNode[172.30.0.123:9300]]
org.elasticsearch.transport.SendRequestTransportException: [][172.30.0.123:9300][internal:transport/handshake]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:719) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:162) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$300(SecurityServerTransportInterceptor.java:53) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.lambda$sendRequest$0(SecurityServerTransportInterceptor.java:114) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsUser(SecurityContext.java:127) ~[?:?]
at org.elasticsearch.xpack.security.transport.Security
@barakseri1
barakseri1 / G1GC.log
Created July 16, 2020 10:39
[2020-07-16T03:45:20.269+0000][4227][gc,start ] GC(3335) Pause Young (Concurrent Start) (G1 Humongous Allocation)
[2020-07-16T03:45:20.269+0000][4227][gc,task ] GC(3335) Using 13 workers of 13 for evacuation
[2020-07-16T03:45:20.269+0000][4227][gc,age ] GC(3335) Desired survivor size 356515840 bytes, new threshold 15 (max threshold 15)
[2020-07-16T03:45:20.279+0000][4227][gc,age ] GC(3335) Age table with threshold 15 (max threshold 15)
[2020-07-16T03:45:20.279+0000][4227][gc,age ] GC(3335) - age 1: 6609936 bytes, 6609936 total
[2020-07-16T03:45:20.279+0000][4227][gc,age ] GC(3335) - age 2: 9849736 bytes, 16459672 total
[2020-07-16T03:45:20.279+0000][4227][gc,age ] GC(3335) - age 3: 204408 bytes, 16664080 total
[2020-07-16T03:45:20.279+0000][4227][gc,age ] GC(3335) - age 4: 172984 bytes, 16837064 total
[2020-07-16T03:45:20.279+0000][4227][gc,age ] GC(3335) - age 5: 268000 bytes, 17105064 total
[2020-07-16T03:45:20.279+000
@barakseri1
barakseri1 / gist:36370cb2757a02b7b2ec498e41defebc
Created January 2, 2020 14:50
[2020-01-02T10:42:55,200][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [ip-172-30-0-37.ec2.internal] failed to execute on node [zJiqfiIZQIGtELtjY7jnxg]
org.elasticsearch.transport.RemoteTransportException: [ip-172-30-0-148.ec2.internal][172.30.0.148:9300][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<transport_request>] would be [4145616528/3.8gb], which is larger than the limit of [4063657984/3.7gb], rea
l usage: [4145608968/3.8gb], new bytes reserved: [7560/7.3kb]
at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:343) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:173) [elasticsearch-7.2.0.jar:7.2.
@barakseri1
barakseri1 / Manual request and repsonse
Last active January 1, 2020 15:26
//Request
POST /.monitoring-es-6-*%2C.monitoring-es-7-*/_search?size=10000&ignore_unavailable=true&filter_path=hits.hits._index%2Chits.hits._source.cluster_uuid%2Chits.hits._source.cluster_name%2Chits.hits._source.version%2Chits.hits._source.license.status%2Chits.hits._source.license.type%2Chits.hits._source.license.issue_date%2Chits.hits._source.license.expiry_date%2Chits.hits._source.license.expiry_date_in_millis%2Chits.hits._source.cluster_stats%2Chits.hits._source.cluster_state%2Chits.hits._source.cluster_settings.cluster.metadata.display_name\n{\"query\":{\"bool\":{\"filter\":[{\"term\":{\"type\":\"cluster_stats\"}},{\"term\":{\"cluster_uuid\":\"vd63AAWCTc6rZ1RiIRSK4g\"}},{\"range\":{\"timestamp\":{\"format\":\"epoch_millis\",\"gte\":1577829973994,\"lte\":1577830091905}}}]}},\"collapse\":{\"field\":\"cluster_uuid\"},\"sort\":{\"timestamp\":{\"order\":\"desc\"}}}"
//Response
{
"hits" : {
"hits" : [
{
"_index" : ".monitoring-es-7-2019.12.26",
@barakseri1
barakseri1 / gist:9282cef2afa82c8e481eb60056c6ce92
Created January 1, 2020 15:22
//Monitoring request for "gap" period
Jan 1 15:14:21 ip-172-30-0-37 kibana: {"type":"log","@timestamp":"2020-01-01T15:14:21Z","tags":["debug","elasticsearch","monitoring","query"],"pid":11073,"message":"200\nPOST /.monitoring-es-6-*%2C.monitoring-es-7-*/_search?size=10000&ignore_unavailable=true&filter_path=hits.hits._index%2Chits.hits._source.cluster_uuid%2Chits.hits._source.cluster_name%2Chits.hits._source.version%2Chits.hits._source.license.status%2Chits.hits._source.license.type%2Chits.hits._source.license.issue_date%2Chits.hits._source.license.expiry_date%2Chits.hits._source.license.expiry_date_in_millis%2Chits.hits._source.cluster_stats%2Chits.hits._source.cluster_state%2Chits.hits._source.cluster_settings.cluster.metadata.display_name\n{\"query\":{\"bool\":{\"filter\":[{\"term\":{\"type\":\"cluster_stats\"}},{\"term\":{\"cluster_uuid\":\"vd63AAWCTc6rZ1RiIRSK4g\"}},{\"range\":{\"timestamp\":{\"format\":\"epoch_millis\",\"gte\":1577829973994,\"lte\":1577830091905}}}]}},\"collapse\":{\"field\":\"cluster_
@barakseri1
barakseri1 / gaps-period
Created December 23, 2019 15:01
{
"clusterStatus":{
"status":"green",
"indicesCount":18,
"documentCount":92971348,
"dataSize":529705350030,
"nodesCount":16,
"upTime":9567913160,
"version":[
"7.2.0"
@barakseri1
barakseri1 / cluster_stats
Created December 23, 2019 08:03
{
"took": 665,
"timed_out": false,
"_shards": {
"total": 7,
"successful": 7,
"skipped": 0,
"failed": 0
},
"hits": {