Azure Data Factory Mapping Data Flows Tutorial | Build ETL visual way!

Azure Data Factory Mapping Data Flows Tutorial | Build ETL visual way!

Adam Marczak - Azure for Everyone

4 года назад

218,244 Просмотров

With Azure Data Factory Mapping Data Flow, you can create fast and scalable on-demand transformations by using visual user interface. In just minutes you can leverage power of Spark with not a single line of code written.

In this episode I give you introduction to what Mapping Data Flow for Data Factory is and how can it solve your day to day ETL challenges. In a short demo I will consume data from blob storage, transform movie data, aggregate it and save multiple outputs back to blob storage.

Sample code and data: https://github.com/MarczakIO/azure4everyone-samples/tree/master/azure-data-factory-mapping-data-flows

Next steps for you after watching the video
1. Azure Data Factory introduction video
- https://youtu.be/EpDkxTHAhOs
2. Check mapping data flow documentation
- https://docs.microsoft.com/en-us/azure/data-factory/concepts-data-flow-overview?WT.mc_id=AZ-MVP-5003556
3. Helpful tips and samples
- https://github.com/kromerm/adfdataflowdocs/blob/master/data-flow-expression-samples.md

### Want to connect?
- Blog https://marczak.io/
- Twitter https://twitter.com/MarczakIO
- Facebook https://www.facebook.com/MarczakIO
- LinkedIn https://www.linkedin.com/in/adam-marczak/
- Site https://azure4everyone.com

Тэги:

#Azure #Data_Factory #Data_Flow #Azure_4_Everyone #Adam_Marczak #Mapping_Data_Flow #Spark #ADF #big_data #SSIS
Ссылки и html тэги не поддерживаются


Комментарии:

Kiran Kumarreddykkr
Kiran Kumarreddykkr - 10.11.2023 19:24

can you pyspark or sql in Expression functions ?
are only scale

Ответить
Joshua Odeyemi
Joshua Odeyemi - 23.05.2023 19:44

I love you, Adam!

I have been struggling with using expression builder in Data Flow. I can't seem to figure out how to write the code. This video just made it look less complex. I'll be devoting more time to it.

Ответить
Sape Cyrille
Sape Cyrille - 13.04.2023 16:07

Great! You are the best Adam.

Ответить
Willy Donkeng
Willy Donkeng - 10.02.2023 12:49

Perfect

Ответить
Sanjay Sinha
Sanjay Sinha - 25.01.2023 03:55

Thanks!

Ответить
Hafid Azer
Hafid Azer - 17.01.2023 01:42

I owe you my paycheck tbh 😅🤣

Ответить
Mohmmed shahrukh
Mohmmed shahrukh - 30.12.2022 14:45

best video on azure I have ever seen❤❤

Ответить
Abhimanyu Pandey
Abhimanyu Pandey - 15.12.2022 23:48

Will it work with pipe (“|”) separated value file instead of csv?

Ответить
Ahmed MJ
Ahmed MJ - 08.12.2022 14:49

Hello Adam , i follow these steps but i have a problem : i didn't find the source columns when i go to derived column component to write expression based on existing column. in your video , total columns in source component show = 3 , for me =0 ? i changed the source from csv to sql table and i didn't found the solution.

Ответить
Dimitar Krastev
Dimitar Krastev - 17.11.2022 15:01

Great video! Most videos seem to focus mostly on the evertisement material straight from Azure. At best they show you the very dumb step of copying data from a file to DB.
This is the first video I saw where you actually show how you can do something useful with the data and close to real life scenario.

Thank you.

Ответить
Affan Ahmed
Affan Ahmed - 30.10.2022 04:31

Good explanation there.

Ответить
POWER BI S.
POWER BI S. - 19.10.2022 21:26

HELLO I"M FROM RRRRRRUSSIA

Ответить
Jay Ong
Jay Ong - 10.10.2022 09:16

Thank you Adam.

Ответить
Khurram Shehzad
Khurram Shehzad - 22.08.2022 14:15

very nice

Ответить
Cuong Lam
Cuong Lam - 13.07.2022 08:49

Very good explaining the Data Flow. Thanks Mr.Adam.

Ответить
Hills Trail
Hills Trail - 05.07.2022 13:33

👍 Its amazing , Practical implementation of Data Flow.

Ответить
Sudarshan Bhattacharjee
Sudarshan Bhattacharjee - 29.06.2022 15:30

Thanks for such good video

Ответить
Bijou Bakson
Bijou Bakson - 24.05.2022 10:54

It must be very challenging to do all this thing in English for you I imagine, Adam! Congratulations for pushing through despite the difficulty. 🙂

Ответить
Achraf Erraji
Achraf Erraji - 16.05.2022 13:06

Amazing Video, we want other parts !

Ответить
Yash Negi
Yash Negi - 11.05.2022 14:18

Video is excellent. I want to know the problem statement which Data flow is solving?

Ответить
Kevin Abraham
Kevin Abraham - 20.04.2022 23:06

Nice video.
Just curious. Can you explain toInteger(trim(right(title,6),'()')) in detail please. Like how this command executes?

Ответить
subhraz
subhraz - 13.04.2022 17:11

Very well explained and demonstrated. Really helpful to get started with Data flows.

Ответить
Thanh Ngo
Thanh Ngo - 11.04.2022 22:32

Thank you, Adam. As always, you rock.

Ответить
Zibeh Zakka
Zibeh Zakka - 17.03.2022 19:57

Great tutorial

Ответить
Low Roar
Low Roar - 17.03.2022 16:25

So helpful! Thank you very much Adam!

Ответить
Jav M
Jav M - 04.03.2022 01:43

I really like your tutorials. I have been looking for a "table partition switching" tutorial but haven't found any good ones. May be you could do one for us? I am sure it'll be very popular as there aren't any good ones out there and it is an important topic in certifications :-)

Ответить
Sam B
Sam B - 01.03.2022 03:35

Great video.
Question: Under "New Datasets", is there a capability to drop data into Snowflake? I see S3, Redshift, etc.
I appreciate the video and feedback!

Ответить
Joy Young
Joy Young - 06.02.2022 15:48

an error message e.g. handshake_failure when the data flow source retrieve data from API, can anyone help? thanks.

Ответить
NiCk6
NiCk6 - 30.12.2021 07:20

Excellent tutorials

Ответить
Hari babu
Hari babu - 15.12.2021 08:18

So nice of your talent explaining the data flow in simple way. Thank you so much Mr.Adam.

Ответить
nidhi sharma
nidhi sharma - 03.12.2021 20:12

Adam, great video.I m new to Data Flow and I have one doubt, I want to implement File level checks in Data Flow but not able to do it. All tasks are performing data level checks like exist or conditional split. Is it possible to implement File level check like whether file exist or not in storage account?

Ответить
Alfreds Futterkiste
Alfreds Futterkiste - 28.10.2021 11:28

Great

Ответить
rajan arora
rajan arora - 23.09.2021 22:21

Your videos are really great and helped me understand lot of concepts of Azure. Can you please make one using SSIS package and show how to use that within Azure Data Factory

Ответить
Bhavani Metla
Bhavani Metla - 13.09.2021 19:44

Hello Adam, thanks a bunch for this excellent video. The tutorial was very thorough and anyone new can easily follow. I do have a question though. I am trying to replicate an SQL query into the Data Flow, however, I have had no luck so far.
The query is as follows:

Select ZipCode, State
From table
Where State in ('AZ', 'AL', 'AK', 'AR', 'CO', 'CA', 'CT'...... LIST OF 50 STATES);

I tried using Filter, Conditional Split and Exists transforms, but could not achieve the desired result. Being new to the Cloud Platform, I am having a bit of trouble.

Might I request you please cover topics like Data Subsetting/Filtering (WHERE and IN Clauses etc.) in your tutorials.

Appreciate your time and help in putting together these practical implementations.

Ответить
fadi abusafat
fadi abusafat - 12.09.2021 18:41

Nice one Adam. Cool one. Keep doing fabulous videos always fella.

Many THanks.

Ответить
Aaron Tian
Aaron Tian - 03.09.2021 01:28

Your channel is totally underrated, man

Ответить
Akiraikonics !!
Akiraikonics !! - 27.08.2021 16:53

How do you delete from target based on data from the Source? I'm really struggling to understand if i have a column with a value that I want to delete in the target table. Everything seems to be geared up to altering source data coming in

Ответить
Isuru Eranga
Isuru Eranga - 23.08.2021 21:19

best tutorial ever... 💪🏻💪🏻💪🏻

Ответить
JingTeng Liu
JingTeng Liu - 10.08.2021 07:20

Would you plan to make video for introduction of each transforamtion components? Thanks

Ответить
Mohit Joshi
Mohit Joshi - 25.07.2021 15:23

Does any of these option changed now? Because I am not able to see any data debug option to be enabled, and directly preview data in dataset itself.

Ответить
mp
mp - 21.07.2021 02:33

-1979 and ,12
This is why complex logic is needed. Nice tutorial :)

Ответить
Mustafa Kamal
Mustafa Kamal - 07.07.2021 19:33

Hi Adam, Thanks for making this videos, very clear and concise. I have a question (sorry not related to this video) regarding Conditional split - Can the output stream activities, run in parallel ?

Ответить
Giovanni Orlando
Giovanni Orlando - 29.06.2021 21:33

Great video! Thanks Adam!

Ответить
Indhu Mathi
Indhu Mathi - 28.06.2021 18:12

I need to join header with data. header is dynamic. how can i retain the order of merge ?

Ответить
Hamid Mushtaq
Hamid Mushtaq - 18.06.2021 18:40

Wouldn't it be simpler to do all of this using code.

Ответить
Vivek Chaudhary
Vivek Chaudhary - 15.06.2021 18:29

very good explanation Adam. keep it up.

Ответить
Ringo vski
Ringo vski - 14.06.2021 12:23

Can you how to add the aggregation column to the same output?

Ответить
Mariya Susnerwala
Mariya Susnerwala - 23.04.2021 02:53

Adam, excellent presentation of ADF concept. I find all your videos really helpful in understanding the ADF concept. One question in regards to the sink dataset in dataflow, how can I create dynamic folder in my blob storage based on the year, month and day when this dataflow was triggered?

Ответить
Chandrasekhar Nallam
Chandrasekhar Nallam - 15.04.2021 21:43

excellent explanation with simple scenario. Thank you.

Ответить