astro.lineage.extractor
Module Contents
Classes
OpenLineageFacets are pieces of metadata that can be attached to the core entities: Run, |
|
This extractor provides visibility on what different python-sdk operator does by |
- class astro.lineage.extractor.OpenLineageFacets
OpenLineageFacets are pieces of metadata that can be attached to the core entities: Run, Job and Dataset as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#facets
- inputs :list[openlineage.client.run.Dataset]
- outputs :list[openlineage.client.run.Dataset]
- run_facets :dict[str, openlineage.client.facet.BaseFacet]
- job_facets :dict[str, openlineage.client.facet.BaseFacet]
- class astro.lineage.extractor.PythonSDKExtractor
Bases:
openlineage.airflow.extractors.base.BaseExtractor
This extractor provides visibility on what different python-sdk operator does by extracting operator specific facets by calling get_openlineage_facets on each operator
- classmethod get_operator_classnames()
- Return type
list[str]
- extract()
Empty extract implementation for the abstractmethod of the
BaseExtractor
class.- Return type
openlineage.airflow.extractors.TaskMetadata
- extract_on_complete(task_instance)
Callback on
get_openlineage_facets(ti)
task completion to fetch metadata extraction details that are to be pushed to the Lineage server.- Parameters
task_instance (airflow.models.taskinstance.TaskInstance) –
- Return type
TaskMetadata | None