1. Spark

Introduced  by  Apache,  it  is  a  powerful  big  data  processing engine which can stream huge volume of data and ingestion of data. The best thing about spark is its ability to process the billions  of  records  of  data  and  using  powerful  in-memory technique  to achieve  it.  Spark  comes in different  languages like phython, scala, java and more. Spark comes with whole of different components like streaming, sql ingestion, graph, machine learning libraries  and more.

Spark is well know for powerful computing of big data over distributed  clusters.  Spark  gives  flexibiity  for  the  user  to override default spark properties and gives full control to the developers  to  fine  tune  the  application  according  to  their needs.

2. Spark Steaming

Spark Streaming basically streams incoming data and ingest the data into any sql engines or file systems or publish the data. The spark streaming flow allows the application to consume the  incoming  data,  the  data  can  be  coming  through  any producers   who   is   publishing   data   using   any   distributed messaging frameworks like kafka. The spark streaming can be done either by using direct streaming, or structured streaming and  write  the  data  to  wherver  it  is  intended  for. The  initial version of spark 1.x  was mainly supporting direct streaming and later Apache introduced spark structured streaming which reduced lot of bioler plate code.

In most case, spark streaming is used along with Kafka, and it is well integrated with any No SQl databases.

2.1  Spark Streaming Flow

To set up a streaming job,

 We require configuration details like for instance in case of Kafka we need what are kafka brokers, kafka  topic  name,  and  Kafka  Consumer,  Kafka Java   Keystore   location   and   password.   These configuration  can  be  passed  as  parameters  for either  structured  streaming  or  direct  streaming api's.

 Once  the  streaming  code  is  implemented,  next task  is  to  implement  data  ingestion  flow,  like where we want to write this data. Spark goes well along with hive, hbase, cassandra etc, so based on requirement spark streaming can convert its RDD to dataset and further dataset can write to tables or file system.

 To kickstart spark streaming flow, we need spark submit  command  along  with  spark  properties, which is a triggerring point for spark job.

Once we are ready with all above steps, spark streaming job is good to start. Spark also comes with UI, which can be used for monitoring the job statistics and metrics.

Spark submit is simple command which can be configurable

in  a  shell  script  to  run.  The  command  has  path  for  spark- submit along with spark properties and actual class to invoke and  the  jar  location.  Spark  job  can  be  submitted  in  either cluster mode or locally.

Spark Configuration plays a key role in overall performance

of job and settings. In the configuration you can specify driver memory, number of executor cores, and number of executor instances  and  executor  memory.  This  configuration  purely depends on volume of data your job is going to process and tune it accordingly. There are tons of different configuration settings that spark provides which is documented in official site. Some additional settings like hearbeat interval, cleaning intervals , backpressure all these also plays vital role.

2.2 Spark Streaming Uses

Spark  streaming  job  serves  different  uses  cases.  Unlike traditional maprecude hadoop jobs which depends on disk for read  and  write,  spark  can  do  it  in  memory  and  also  it  can customize to do both in memory and disk by simple settings. This makes Spark a great tool to process and transform huge volume data.

Spark is well integrable with any stacks like kafka, cassandra, hive, hdfs, MQ, flumes, Kinesis and more. This makes great choice whenever we want to ingest into any of the data stores. Development wise it less coding and more time is spent on tunning the job to meet the performance goal.

Spark   help   processing   and   computing   big   data   over distributed  cluster.  This  data  can  be  used  for  analytics, machine  learning   and   also   building   artifical  intelligence insights.

Spark streaming ultimately serves the purpose of building live dashboards,  reporting,  smart  data  analytics.  The  uses  are unlimited, due to its unique processing power it is a number one choice when huge volume of data needed to be processed.

Acknowledgements

My profound experiences I have in infirmation technology field,  give  me  a  vision  to  contribute  to  build  these  kind  of tools. I am very thankful to my past and present company to give me an opportunity to explore a way to solve a industry problems.

Sudhish  Koloth  is  a  Lead  developer  working  in  Banking  and Financial  company.  He  has  13+  years  of  working  experience  in information   technology.   He   worked   in   various   technologies including Full stack, big data, automation and android development.