XCom Backend
The custom XCom backend adds special handling to Astro’s custom constructs (see Concepts) so they can
be used without enabling XCom picking (the xcom_pickling
configuration). When the custom constructs are
not accessed, this is simply a wrapper around Airflow’s default XCom backend, so a migration from the default
backend is seamless and fully compatible.
Configure the backend
To use Astro’s custom XCom backend, set the [core] xcom_backend
configuration like this:
[core]
xcom_backend = astro.custom_backend.astro_custom_backend.AstroCustomXcomBackend
The same may also be achieved by setting the environment variable AIRFLOW__CORE__XCOM_BACKEND
.
You should also tell the backend where to store data, by providing a storage URL and connection ID:
[astro_sdk]
xcom_storage_url = <url here>
xcom_storage_conn_id = <conn id here>
The same may also be achieved by setting the environment variables AIRFLOW__ASTRO_SDK__XCOM_STORAGE_URL
and AIRFLOW__ASTRO_SDK__XCOM_STORAGE_CONN_ID
.
If you don’t have a storage (in local development scenarios, for example), the data can be stored in Airflow’s metadatabase instead, by setting
[astro_sdk]
store_data_local_dev = true
or the environment variable AIRFLOW__ASTRO_SDK__STORE_DATA_LOCAL_DEV
instead. Note that this is considered
suboptimal, and should not be used in a production environment.