Exclusive | Airflow Xcom

XComs are small pieces of metadata, such as file paths, API response tokens, or status flags, that tasks push to the Airflow metadata database. By default, tasks in Airflow are fully isolated and may run on completely different workers. XCom bridges this gap. An XCom is defined by a few key attributes: : The name of the data (default is return_value ). Task ID : The task that pushed the data. DAG ID : The DAG to which the task belongs. Defining "Airflow XCom Exclusive"

: Explicitly pushes a key-value pair to the database.

: By setting multiple_outputs=True , a task can return a dictionary that Airflow automatically unrolls into separate XCom entries for each key, allowing downstream tasks to pull only what they need.

Apache Airflow XComs should be reserved exclusively for small metadata pointers, such as S3 keys or row IDs, to prevent metadata database bottlenecks. For large data transfers, utilizing custom XCom backends for object storage like S3 or GCS is recommended to optimize DAG performance. Read more on best practices at Astronomer Documentation Apache Airflow XComs — Airflow 3.2.0 Documentation airflow xcom exclusive

Last updated: 2025 Applies to Apache Airflow 2.0+

def push_explicit(**context): context['ti'].xcom_push(key='my_key', value='my_value')

The evolution of Airflow has dramatically simplified how developers interact with XComs. Understanding the distinction between the legacy syntax and the modern TaskFlow API is essential for writing clean code. The Legacy Approach XComs are small pieces of metadata, such as

# Pushing XCom (implicitly via return) def push_task(**context): return "some_value"

. This allows you to store the actual data "exclusively" in external object storage while only keeping a reference in the Airflow DB. Apache Airflow Object Storage Backend : You can configure Airflow to use Google Cloud Storage Azure Blob Storage Implementation : To build a custom one, you must subclass and override the serialize_value deserialize_value Thresholding : You can set a size threshold (e.g., xcom_objectstorage_threshold

When using other classic operators (like the BashOperator ), setting do_xcom_push=True forces the operator to push the last line of standard output ( stdout ) to the metadata database. Manual Pushing and Pulling An XCom is defined by a few key

Relational database columns have strict size limits (e.g., BLOB or TEXT limits). Attempting to push an asset larger than the database column capacity will hard-fail the task with serialization or database write errors. 3. The TaskFlow API: Modernizing XComs

[core] xcom_backend = my_project.xcom_backend.ExclusiveRedisXCom