An End to End Azure Data Engineering Real Time Project Demo | Get Hired as an Azure Data Engineer

I hope you find this video useful. If you have learned something or found it helpful, please consider giving it a like.

What topic should I cover in the next end-to-end project video? Please let me know your thoughts in the comments. Thank you!😊😊😊

Thank you for video MR K. I just wonder what made you to use ADF pipeline rather than DLT one?

Mr. K, you are the gem man. Your way of explaining the things superb. Thanks a lot for this free content.

For the final section of data loading in Synapse, doesn't that need to be added to the pipeline run in Data Factory?

good

It is a great effort that you have made in supporting those who cannot afford to spend money on courses. This channel may become successful in the coming days. The content you provide here is not taught in courses themselves. Please be motivated to continue teaching in this way. We are here to support anyway whatever possibility we have in our hands.

Very very helpful, appreciate your efforts, thank you very much

Where is tony blaiaaaa

One more end to end project, with different scenarios would be great

Hello Mr K,
Thank you so much for the detailed explanation , I am new to cloud systems and this is really helpful for someone like me trying to understand how Azure resources can be put together to build a meaningful Use case. Quick question: Do we really need Azure Synapse Analytics for ? can't we use Azure SQL DB instead? and may be Power BI talks to SQL DB to fetch and build reports/ dashboard visualization? If possible, could you please share pros/cons with this design?

Thanks in advance and appreciate your content sharing,
Eswar

Excellent video.. all topics are explained in detail and covered all topics.Thanks for the video.

Thanks!

very useful content, thanks for sharing!

Thank you Mr. K for sharing this project. Please share the database that needs to be ingested from on-premises to cloud?

Excellent video with great explanations - thanks so much!

can I do all this with azure free trail?

Are you providing any training on azure data engineering..?

Sir, what a in-dept explanation end to end project. I am working as Power Bi developer just begin to learn Azure Data engineering. You explained very well I got confidence that I can do it. Thanks a lot once again👍

Best content with excellent clarity.

Awesome content very clear explanation sir. You are a great teacher! Lucky to have found this channel.

hello are tools free to use ? I'm beginner and not sure if I should start with this and thank you

I wouldn't be surprised if this channel hits a million subscribers soon !!! Excellent explanations and extra ordinary presentations 👏👏. WIll continue to watch every videos of this channel...love the structured way of going with the data related topics 👌

I am a newbie. I am trying to implement this project but I just cannot establish the connection between ADF and SSMS/SQL Server. I have followed the same steps and made sure if the username and passwords are all correct.

The error that I receive is: Cannot connect to SQL Database. Please contact SQL server team for further support. Server: 'localhost', Database: 'AdventureWorksLT2017', User: 'molu'. Check the linked service configuration is correct, and make sure the SQL Database firewall allows the integration runtime to access.
Login failed for user 'molu'., SqlErrorNumber=18456,Class=14,State=1,

I tried to find out how to make SQL Database firewall to allow the runtime. Found some solutions and I implemented them but it did not help.

Any help would be appreciated.

This video was really helpful thanks for bringing this up 😊
Hope to see more end to end projects

hello! excellent tutorial! It helps me a lot in my data engineering jobs! One question, once the lake house is created, can power bi connect directly to databricks and consume the data from there? Is it necessary to have a cluster on for this? thank you!

I'm getting an error while testing sql connection
Cannot connect to SQL Database. Please contact SQL server team for further support. Server: 'localhost', Database: 'AdventureWorksLT2019', User: 'mrk'. Check the linked service configuration is correct, and make sure the SQL Database firewall allows the integration runtime to access.
Login failed for user 'mrk'., SqlErrorNumber=18456,Class=14,State=1,

This is amazing, end to end, thorough and clear concept and example detail, love it

Can you share the dataset link or script ?

It's my first video from your channel and I loved it. I liked the clear and concise explanation, along with utilizing various services in creating a truly end to end Data Engineering Project. I was able to successfully implement the pipelines after learning from your video. Thank you!

Gold

You sir, have earned my respect. Excellent communication style, clear, and concise. Very well done. 👍🏼

Thanks!

Amazing work. Appreciate the help. just one question , Can we use azure data brick instead Synapse to avoid vendor lock in ?

very useful video thank you so much brother.

Reached here today accidentally and subscribed immediately.

Could you please tell that why you have used ADF as separate resource for data ingestion ?

Why not make use of the ADF instance which is provided under Azure Synapse ?

Mr. k Sirji I went through the first part of your video - Environment setup. You mentioned that after going through end to end one can have the architecture in one's resume.

Surprised that basics like Linked Service, Datasets were just created on the fly. These are extremely important components of ADF and no concept was provided. I am lucky that I have previously gone through certain ADF data movement tutorials

All essential components were already created and creation is services forms a basis of any tutorial.

You create a SHR IR service for one premise SQL Server but the most important thing is the configuration of Security Lists / Groups (Virtual Cloud Nw Firewall).

Anyway the presentation is good , voice volume is optimum and speed is also okay.


Namaste 🙏

Thaks, you're amazing !

In order for us to follow along - how do we setup the resource group and also what do i need to do in order to setup the sql database?

This is a hidden Gem for sure. Will make sure to go through all your videos.

It is very helpful, Thank you very much :)

You explained bronze silver gold but your session is diamond .

Do you conduct live sessions

Hello sir
Its humble request to you
Pls tell me
Roles n responsibilities of this project to include on resume

Highest effort you have given. Kindly share the all code you have used in the video will be helpful. Thank you

make another project and add CDC,SURROGATE KEY and SCD2 IN IT

Cool !

I am new to Data stream. Why we need adf ? we cant copy the database tables from on prem to azure data lake directly?

"Great video! Your explanation of End-to-end Azure Data Engineering was clear and easy to follow. I appreciate your expertise in breaking it down for us.

hi, great tutorial and indeed good learning for starters as me. Can you also please make end to end azure data engineering real time project with continuous data stream & readily available big data (so that we can readily download from your link). It would be of great help for us.