astro.lineage.facets
Module Contents
Classes
Facet that represents input file for load file |
|
Facet that represents input file dataset Facets for load file |
|
Facet that represents output database dataset Facets for load file |
|
Facets that represent may be available for Table object |
|
Facets that represent may be available for Source Table object in MergeOperator |
|
Facets that represent may be available for Target Table object in Merge Operator |
- class astro.lineage.facets.InputFileFacet
Bases:
openlineage.client.facet.BaseFacet
Facet that represents input file for load file
- Parameters
filepath – path of each file.
file_size – size of the file.
file_type – type of the file.
- filepath :str
- file_size :int | None
- file_type :astro.constants.FileType
- description :str | None
- class astro.lineage.facets.InputFileDatasetFacet
Bases:
openlineage.client.facet.BaseFacet
Facet that represents input file dataset Facets for load file
- Parameters
file_size – size of the file.
number_of_files – number of files to be loaded.
file_type – type of the file.
description – description of the dataset.
is_pattern – True when file path is a pattern(eg.
s3://bucket/folder
or/folder/sample_*
etc).files – list of filepaths to be loaded from dataset.
- number_of_files :int | None
- description :str | None
- is_pattern :bool = False
- files :list[InputFileFacet] = []
- class astro.lineage.facets.OutputDatabaseDatasetFacet
Bases:
openlineage.client.facet.BaseFacet
Facet that represents output database dataset Facets for load file
- Parameters
metadata – metadata of the table.
columns – columns defined in table
schema – schema used.
used_native_path – use native support for data transfer if available on the destination.
enabled_native_fallback – use enable_native_fallback=True to fall back to default transfer.
native_support_arguments – args to be used by method involved in native support flow
description – description of the dataset.
- metadata :astro.table.Metadata
- columns :list[astro.table.Column]
- schema :str | None
- used_native_path :bool
- enabled_native_fallback :bool | None = False
- native_support_arguments :dict | None
- description :str | None
- class astro.lineage.facets.TableDatasetFacet
Bases:
openlineage.client.facet.BaseFacet
Facets that represent may be available for Table object
- Parameters
table_name – Name of the table
columns – columns defined in table
source_table_rows – Total rows in source table
metadata – metadata of the table.
- table_name :str
- columns :list[str] | tuple[str] | dict[str, str] | None
- source_table_rows :int = 0
- metadata :Metadata | None
- class astro.lineage.facets.SourceTableMergeDatasetFacet
Bases:
openlineage.client.facet.BaseFacet
Facets that represent may be available for Source Table object in MergeOperator
- Parameters
table_name – Name of the table
if_conflicts – The strategy to be applied if there are conflicts.
columns – columns defined in table
source_table_rows – Total rows in source table
metadata – metadata of the table.
- table_name :str
- if_conflicts :astro.constants.MergeConflictStrategy
- columns :list[str] | tuple[str] | dict[str, str] | None
- source_table_rows :int = 0
- metadata :Metadata | None
- class astro.lineage.facets.TargetTableMergeDatasetFacet
Bases:
openlineage.client.facet.BaseFacet
Facets that represent may be available for Target Table object in Merge Operator
- Parameters
table_name – Name of the table
target_conflict_columns – List of cols where we expect to have a conflict while combining
columns – columns defined in table
metadata – metadata of the table.
- table_name :str
- target_conflict_columns :list[str]
- columns :list[str] | tuple[str] | dict[str, str] | None
- metadata :Metadata | None