# Pushing data manually def push_custom_data(**kwargs): ti = kwargs['ti'] # Get the Task Instance ti.xcom_push(key='model_accuracy', value=0.94) # Pulling data manually def pull_custom_data(**kwargs): ti = kwargs['ti'] accuracy = ti.xcom_pull(key='model_accuracy', task_ids='push_custom_data_task') print(f"The model accuracy is accuracy") Use code with caution. The Hidden Trap: Why Big Data Breaks XCom
In Apache Airflow, tasks are isolated by design to ensure reliability across distributed workers. However, real-world workflows often require sharing state—like a dynamically generated filename, a processing timestamp, or a specific API token. (short for Cross-Communication) is the native mechanism that makes this possible. What is Airflow XCom?
If your tasks pass sensitive connection tokens, temporary credentials, or PII via XCom, you must protect that data from appearing in plain text within the Airflow Web UI. Airflow automatically masks keys containing terms like secret , password , auth , or token . airflow xcom exclusive
If you are interested, I can provide a more detailed comparison of XComs vs. Airflow Variables vs. Airflow Pools. XComs — Airflow 3.2.0 Documentation
Example:
@task def use_conf(**context): value = context['dag_run'].conf['xcom_value']
This guide dives into the "exclusive" nature of XComs: how they work, why they are limited, how to use them efficiently, and how the newer has changed the game. 1. What Exactly is an Airflow XCom? # Pushing data manually def push_custom_data(**kwargs): ti =
In Apache Airflow, tasks are isolated by design. This isolation is great for reliability, but it creates a challenge when one task needs to share information—like a filename, a record count, or a status flag—with a downstream task. (short for "cross-communication") is the built-in mechanism that solves this problem. What is XCom?
Simple design:
By default, when a task returns a value, Airflow automatically serializes it and stores it in the metadata database ( airflow.db ).