How to build and automate a python ETL pipeline with airflow on AWS EC2 | Data Engineering Project

I'm so much blessed to be learning from a fellow Nigerian who is not interested in unnecessary promotions and advertorials. Just great lessons and for free. Thanks Dr Yemi.

Legend

Awesome Hands on Tutorial!! Thank you🎉

Perfect one video to get started with AIRFLOW. Thank you for sharing. your valuable knowledge
your teaching skills are highly impressive. Subscribed :)

Great video, super easy to understand and well explained!

I really liked the way you teach. Even a layman will understand everything.. Thank you Sir

@tuplespectra can we use shell scripting for orchestrating this instead of Apache Airflow..?

are all what you used included in the free tier of aws services or not, please?

Wow. I love how you explain.

So I'm doing a similar project but using DBT for transformation. Is it possible to put DBT into same DAG? Is that how people do it as a good practice?

Thank's from Angola

I have used airflow as a user but never spun up an instance on my own. This is easily the best video that exists today for AWS + Airflow, this has taken so much guess work out of the process. Thanks so much!

Thank you for this awesome project, it took me 2weeks to complete. Had to fix lots of errors but I was determined to get past it all. I honestly learnt a whole lot from all the errors really. Thank you once again🙏

Penato is way simpler, have nice gui, performs fantastic. What is the sense to write code for each data transformation in 2024...

Na Nigerian guy be this

Very clear explanation

I am able to install airflow on free-tier EC2 instance but not able to connect to : Public IPv4 DNS . can anyone suggest on this ?

What if I use CSV file instead of API so how can I load the file into airflow file

Excellent job

Hello thank you for the great video. I am running into an issue where the EC2 instance keeps freezing and disconnecting every time I manually trigger a dag. I believe it's because the t2.small instance does not have enough memory but I see this is not an issue for you or others. Just wondering if anyone else is going through the same thing or whether it is something I did wrong? Any help is appreciated, thank you.

wonderful video

So far to me, Airflow is cron with a half baked UI and functionality, for starters, you can't create a DAG using the UI

Thank you, for this! Love the enthusiasm, exuberance and energy brother. Usually, most tutorials are dull, monotonous and riddled with confusing, and abstract terms and acronyms. You really take the time to distill things in plain speak!

Just a remark: you forgot to tell that here you are using the default vpc. I don't think it is a best practise. I understand the video would be longer but it would worth mentioning it.
As well depending on how you created your VPC, AWS now can create for you automatically a connection between for ec2 instance and S3.

when I ssh using vs code it keeps disconnecting. anyone found a solution?

Since you are using session token, isnt it a manual effort and hence DAG will not be automated?

Where does the airflow folder in vscode came from?

Great work ! Thank for this project. I have already completed and i learn a lot.

Thank you so much for this tutorial, was very helpful for an interview !!!

Thank you very very very Much !

Command not working sudo pip install pandas in airflow _venv , what to do?

This is a masterpiece. Thank you again and again !!!

It's like T'Challa is teaching AirFlow! 🤣

Tutorial is really great. However, I am having some issues with loading the airflow UI. When I use ''airflow standalone'', and go to the 8080 port on the url from s3 instance it takes a long time to load the UI. Most times it doesn't load at all. I can't seem to understand what the reason is. Could you please help me?

Hi bro. Congrats for the video!!
I have experiencing a major issue, the airflow service and the SSH connection collapses after a few minutes of the instance initiation... What can I do?

The scheduler does not appear to be running. Last heartbeat was received 32 minutes ago.

The DAGs list may not update, and new tasks will not be scheduled.I have followed all commands you mentioned but i was getting like this in my airflow user interface and the command prompt which i connected to ec2 was running airflow standalone where to run new commands so that i can run airflow scheduler ? pls reply

amazing bro thanhs

Thank you very much for making the concepts so easy to understand👌

I am getting following error while creating a virtual environment.
E: Unable to locate package python3.11.9-venv
E: Couldn't find any package by glob 'python3.11.9-venv'

If anyone can help then it would be great!!

i get my load_data task failed i configured everything right but still get failed for the last task i couldnt figure it out anyone with the same scenario got any soln?

Excellent information, thank you so much for posting this video here

If you have problems with installing dependencies it is because instead of sudo apt install3.10-venv, replate it to sudo apt install3-venv to get the latest version. Currently, it's at 3.12

yay, i managed to finish it! and i have the csv file in s3. thx, u deserve the like lol.

dont ask for likes, i will only like if i can finish and add this to my resume

Amazing video❤

my ec2 instance is not loading using a t2 micro instance. could that be the only reason?

Good beginners video. I am personally not a fan of running your processing code and orchestration on same instance. How do you ensure package dependency,manage virtual environments,resource allocation for different workloads.

Thanks for the tuto, just as remarque no need for hard coding credentials, since you gave iam role full access and assumed by the EC2. you can write directly in S3 from EC2