|Most Shared

Spark + Kafka tech coding spark kafka avro
29 Jan 2016at Alpharetta
Consolidated all the Spark+Kafka+Scala working samples here

Sample-1

https://github.com/vijayaraj210/scribbles/blob/master/spark/spark-kafka/src/main/scala/com/vijayrc/spark/KafkaReadJob.scala

Covers the basics of reading from kafka topic

  • with fixed offset range into a spark RDD
  • with reading into stream RDDs

Sample-2

https://github.com/vijayaraj210/scribbles/blob/master/spark/spark-kafka/src/main/scala/com/vijayrc/spark/KafkaWordCount.scala

Covers the wordcount example using both old and new(direct stream) spark-kafka apis

Sample-3

https://github.com/vijayaraj210/scribbles/blob/master/spark/spark-kafka/src/main/scala/com/vijayrc/spark/KafkaWriteJob.scala

Uses Cloudera Spark Kafka Writer to write string data into Kafka topics

Sample-4

https://github.com/vijayaraj210/scribbles/blob/master/spark/spark-kafka/src/main/scala/com/vijayrc/spark/KafkaAvroReadJob.scala

Reads Avro objects from Kafka

Sample-5

https://github.com/vijayaraj210/scribbles/blob/master/spark/spark-base/src/main/scala/com/vijayrc/spark/AvroJob.scala

Uses Databricks spark-avro to read/write avro from spark dataframes

Reference

  • http://spark.apache.org/docs/latest/streaming-programming-guide.html
  • https://databricks.com/blog/2015/03/30/improvements-to-kafka-integration-of-spark-streaming.html
  • http://blog.cloudera.com/blog/2015/03/exactly-once-spark-streaming-from-apache-kafka/
  • http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-example-tutorial/
  • http://capykoa.com/articles/14
  • https://github.com/databricks/spark-avro
  • https://github.com/cloudera/spark-kafka-writer
comments powered by Disqus

All content except noted photos and videos copyright © Vijayaraj Chakravarthy. All rights reserved. *Any images or videos not listed as mine are copyright to their respective owners and were used under creative common license or fair use standards. If a photo or video is your material and you do not wish it to be on the site, please email me vijayrc@outlook.com and I will remove it immediately.