Description
Google Cloud system tests (BigQuery, and likely others) fail due to a DetachedInstanceError thrown by the OpenLineage listener in on_dag_run_failed / on_dag_run_success. All operator tasks complete successfully, but the watcher task marks the DAG run as failed because the OpenLineage listener errors out.
Reproduction
Run any Google Cloud BigQuery system test:
export SYSTEM_TESTS_GCP_PROJECT=<your-project>
export SYSTEM_TESTS_ENV_ID=test01
export PYTHONPATH=providers/google/tests:providers/openlineage/tests:$PYTHONPATH
uv run --project providers/google pytest --system \
providers/google/tests/system/google/cloud/bigquery/example_bigquery_queries.py -xvs --with-db-init
Error
WARNING airflow.providers.openlineage.plugins.listener:listener.py:971 OpenLineage received exception in method on_dag_run_failed
...
File ".../openlineage/utils/utils.py", line 756, in <lambda>
"note": lambda dagrun: getattr(dagrun, "note", None) if AIRFLOW_V_3_2_PLUS else None,
...
sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <DagRun at 0x...> is not bound to a Session;
lazy load operation of attribute 'dag_run_note' cannot proceed
Root Cause
The DagRunInfo._cast_fields() method in providers/openlineage/src/airflow/providers/openlineage/utils/utils.py accesses dagrun.note, which triggers a lazy load of the dag_run_note association proxy on a DagRun instance that is no longer bound to a SQLAlchemy session.
This was likely introduced by #62347 ("feat: Add DagRun note to OL events").
Impact
All system tests with a watcher task fail, even though all actual operator tasks complete successfully. This blocks contributors from getting clean system test results.
Environment
- Airflow main branch (latest)
- Running locally via
uv run (not breeze)
- SQLite backend
Description
Google Cloud system tests (BigQuery, and likely others) fail due to a
DetachedInstanceErrorthrown by the OpenLineage listener inon_dag_run_failed/on_dag_run_success. All operator tasks complete successfully, but thewatchertask marks the DAG run as failed because the OpenLineage listener errors out.Reproduction
Run any Google Cloud BigQuery system test:
Error
Root Cause
The
DagRunInfo._cast_fields()method inproviders/openlineage/src/airflow/providers/openlineage/utils/utils.pyaccessesdagrun.note, which triggers a lazy load of thedag_run_noteassociation proxy on aDagRuninstance that is no longer bound to a SQLAlchemy session.This was likely introduced by #62347 ("feat: Add DagRun note to OL events").
Impact
All system tests with a
watchertask fail, even though all actual operator tasks complete successfully. This blocks contributors from getting clean system test results.Environment
uv run(not breeze)