Building (Kafka and Spark Structured) Streaming Pipelines using Databricks

4 года назад

14,003 Просмотров

As part of this topic we will see how to ingest data in real time using Kafka eco system and process using Spark Structured Streaming on top of Databricks.

Here is the link for complete material - http://discuss.itversity.com/t/integration-of-kafka-and-spark-structured-streaming-using-databricks/20087

## Agenda
Here is the agenda for this module or section. We will see end to end streaming analytics pipeline using Kafka and Spark with Scala as Programming Language on top of Databricks Platform.

* Define Problem Statement
* Setup EC2 Instance
* Simulate Web Server Logs
* Setup and Start Kafka Broker
* Create Kafka Topic and Validate
* Ingest Web Server Logs into Kafka Topic
* Setup Databricks Cluster
* Integrate and Validate with Spark Structured Streaming
* Streaming Data Processing using Data Frame APIs
* Perform Data Analysis using Spark SQL

Тэги:

#databricks #kafka #spark #scala

Скачать видео

Комментарии:

Takshit Mathur - 05.09.2023 01:48

This has been so helpful, Thank you so much!

Ответить

fahad siddiqui - 15.03.2023 13:54

Hi i just want to do shell in putty or cmd in local

Ответить

Steve Goldstein - 05.11.2022 17:16

My concerns about using jdbc as a producer lets say with Kafka is scalability in terms of the amount of connections that can be established to the db

Ответить

ashok gupta - 01.01.2022 21:42

It's really nice video. Could you please throw some light on check point files. How can we recover data in case we want to get prior data based on some time window. In case some data is corrupted.

Ответить