Kafka.md

Kafka

http://kafka.apache.org

https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations

https://cwiki.apache.org/confluence/display/KAFKA/Index

https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics

http://blog.cloudera.com/blog/2014/09/apache-kafka-for-beginners/

Producer

https://kafka.apache.org/0101/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html

http://kafka.apache.org/documentation/#producerconfigs

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_secure-kafka-ambari/content/ch_secure-kafka-produce-events.html

https://github.com/gwenshap/kafka-examples/blob/master/SimpleCounter/src/main/java/com/shapira/examples/producer/simplecounter/DemoProducerNewJava.java

https://github.com/cakesolutions/scala-kafka-client/wiki/Message-Exchange-Patterns

http://ingest.tips/2015/07/19/tips-for-improving-performance-of-kafka-producer/

Consumer

https://kafka.apache.org/0101/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html

http://kafka.apache.org/documentation/#consumerconfigs

http://doc.akka.io/docs/akka-stream-kafka/current/consumer.html

https://github.com/akka/reactive-kafka/blob/master/core/src/test/scala/akka/kafka/internal/ConsumerTest.scala

Interceptors

https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/ProducerInterceptor.html

https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors

https://github.com/appsignal/appsignal-kafka-statsd-interceptor/blob/master/src/main/kotlin/com/appsignal/kafka/statsd_consumer_interceptor.kt

Partition

http://www.javaworld.com/article/3066873/big-data/big-data-messaging-with-kafka-part-2.html

https://github.com/jkorab/ameliant-tools/blob/master/kafka/kafka-perf-tool/src/main/java/com/ameliant/tools/kafka/perftool/drivers/partitioning/KeyAllocationStrategy.java

http://stackoverflow.com/questions/39574328/kafka-multiple-partition-ordering

http://www.jakubkorab.net/2015/12/message-distribution-and-topic-partitioning-in-kafka.html

Performance

http://linuxthrill.blogspot.co.uk/2016/04/kafka-performance-tuning.html

https://www.cloudera.com/documentation/kafka/1-4-x/topics/kafka_performance.html

https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

Topics

http://stackoverflow.com/questions/32761598/is-it-possible-to-create-a-kafka-topic-with-dynamic-partition-count

http://stackoverflow.com/questions/38024514/understanding-kafka-topics-and-partitions?rq=1

https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/

Serializer

https://github.com/nielsutrecht/kafka-serializer-example/blob/master/src/main/java/com/nibado/example/kafka/Example.java

https://kafka.apache.org/0100/javadoc/org/apache/kafka/common/serialization/Serializer.html

http://stackoverflow.com/questions/23755976/kafka-writing-custom-serializer

Monitoring:

https://github.com/linkedin/Burrow/wiki/What-is-Burrow

https://github.com/tulios/burrow-stats

https://github.com/splee/burrower

https://github.com/yahoo/kafka-manager

yahoo/CMAK#187

https://signalfx.com/blog/how-we-monitor-and-run-kafka-at-scale/

https://signalfx.com/kafka-monitoring/

Deployment

http://docs.confluent.io/3.0.1/kafka/deployment.html

https://www.confluent.io/product/control-center/

Operations

https://www.cloudera.com/documentation/kafka/latest/topics/kafka_command_line.html

netstat -nlp | grep 2181 netstat -nlp | grep 9092

topic partition = directory . Consistent Hash required to achieve paralellism

http://docs.confluent.io/3.1.2/kafka/deployment.html?highlight=production

https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas

https://issues.apache.org/jira/browse/KAFKA-1489

https://cwiki.apache.org/confluence/display/KAFKA/KIP-61%3A+Add+a+log+retention+parameter+for+maximum+disk+space+usage+percentage

https://github.com/apache/kafka/pull/1348/files

https://github.com/linkedin/Burrow/wiki/Consumer-Lag-Evaluation-Rules

https://github.com/softwaremill/kmq/blob/master/core/src/main/scala/com.softwaremill.kmq/redelivery/CommitMarkerOffsetsActor.scala

http://stackoverflow.com/questions/29647656/effective-strategy-to-avoid-duplicate-messages-in-apache-kafka-consumer/29656353

http://docs.confluent.io/3.1.2/kafka/monitoring.html

https://www.cloudera.com/documentation/kafka/latest/topics/kafka_performance.html

http://stackoverflow.com/questions/21020347/kafka-sending-a-15mb-message