Airflow XCom for Beginners - All you have to know in 10 mins

Best xcom video I watched

great tutorial and explanantion! thank you!

Please don't read the underscore!!!! Just read it a "choose model" and not "choose underscore model"

thank you!
as it gets very often, some detail is missed by me in order to use new functionality and that detail was ti

thanks for explaining!

Nice video but is there a way to do it between 2 DockerOperator? And if not with the DockerOperator creating some kind of CustomDockerOperator to make it able to do it?

Hi all! Please suggest how to implement the following scenario:

Have a dag with sshoperator that executes shell script. Shell script returns different return codes. How to analyze which return code is received??

Thank u for your sharing, keep it up because u doing that right clearly and easy to understand. Love from Viet Nam

Whats a clearly explanation! Thankyou so much!
Love from Indonesia!❤‍🔥

Thank you so much. It was really quick and clear.

useful!

Very well explained...! Very awesome...! I am just enjoying the series

Thank you so much! By following your videos closely I not only avoided bugs but also learned in detail how to set up an airflow pipeline!

How to access xcom in subdag from parent dag?

Amazing tutorial as always. Love yout channel.

can we pass xcom as params in redshiftsqloperator?

Excellent thank you

I have the same xcom key from same task and it does not get replace or overwrite, instead it creates other xcom record with same key and task id

Hi Marc, how to cross communication data (data sharing) between master_dag and child_dag(this is triggered by master dag)?

Does someone have the same issue like me? I got 3 “None” -> [None, None, None]

Thanks for all your videos, Marc. Brilliant content, very straightfoward!

Sir, I tried to do the needful to do good and nice Airflow in the XCom, but my hit chance was still saying 99% chance of to becoming the hit sir, but it still missed most of the time. Sir I do believe you should do the needful by improving the Airflow by using the Minecraft Mushroom Soup Windtunnel.

Thank you! I spent a few hours trying to figure out how it works, but your tutorial did it in 10 minutes

Very useful video! after reading airflow documentation and example for me this XCOM was still not clear :( Now I m ok! Thanks a lot!!!

hi, thank for the video
could you answer the question?
what happens with xcoms after pulling it from metastore? whether they stay there or delete? and if they stay there after pulling then for how long?

Commander, the aliens continue to make progress on the Avatar Project. If we're going to slow them, we need to move fast.

Pretty enlightening!
Thank you very much!

Thank you marc, I did the course in udemy and it's very recommended

The three people who disliked this video are nifi fans

thank you for clear explanation

Thanks, Marc. Great info
Can You Please Confirm This
I have Two Tasks in DAG1
Task1 - Normal Function Which does some process
Task2 - TriggerDagRunOperator (Trigger DAG2)
Can I Use the return value/XCom of task1 in Task2 and use this value in DAG2?
Thanks

add provide_context = True in task which uses ti in older version of airflow

Hi Marc,

I have a question with regards of using xcom in subdags. How would you approach this? I tried e.g. adding it as a parameter in my subdag using Jinja template (see below), but it causes the following error (ValueError: unknown url type: "'").

The goal is to extract the xcom created by extract_all_bag_links() (stored in key='links'), in my created subdag function process_all_municipalities_subdag().

Just to give some more context, we are still currently using airflow 10.8, we are going to change in a few months (I saw the vid about the tasks instead of the subdags haha!)

#################### CODE ####################

# Extracts all the links to each municipality bag data
extract_bag_links = PythonOperator(
task_id='extract_bag_links',
python_callable=extract_all_bag_links,
provide_context=True,
dag=dag,
)

process_all_municipalities = SubDagOperator(
task_id='process_all_municipalities',
subdag=process_all_municipalities_subdag(parent_dag_name=DAG_NAME,
child_dag_name='process_all_municipalities',
args=default_args,
links="'{{ ti.xcom_pull(task_ids='extract_bag_links', key='links') }}'"),
dag=dag,
)

extract_bag_links >> process_all_municipalities

Thanks Marc. Great info for a beginner like me :)

Thanks Marc as always