astro.lineage.facets

Module Contents

Classes

InputFileFacet

Facet that represents input file for load file

ExportFileFacet

Facet that represents output file for export file

InputFileDatasetFacet

Facet that represents input file dataset Facets for load file

OutputDatabaseDatasetFacet

Facet that represents output database dataset Facets for load file

TableDatasetFacet

Facets that represent may be available for Table object

SourceTableMergeDatasetFacet

Facets that represent may be available for Source Table object in MergeOperator

TargetTableMergeDatasetFacet

Facets that represent may be available for Target Table object in Merge Operator
class astro.lineage.facets.InputFileFacet

Bases: astro.lineage.BaseFacet

Facet that represents input file for load file

Parameters:

  • filepath – path of each file.

  • file_size – size of the file.

  • file_type – type of the file.

filepath: str
file_size: int | None
file_type: astro.constants.FileType
description: str | None
class astro.lineage.facets.ExportFileFacet

Bases: astro.lineage.BaseFacet

Facet that represents output file for export file

Parameters:

  • filepath – path of each file.

  • file_size – size of the file.

  • file_type – type of the file.

  • if_exists – Overwrite file if exists.

filepath: str
file_size: int | None
file_type: astro.constants.FileType
if_exists: astro.constants.ExportExistsStrategy
class astro.lineage.facets.InputFileDatasetFacet

Bases: astro.lineage.BaseFacet

Facet that represents input file dataset Facets for load file

Parameters:

  • file_size – size of the file.

  • number_of_files – number of files to be loaded.

  • file_type – type of the file.

  • description – description of the dataset.

  • is_pattern – True when file path is a pattern(eg. s3://bucket/folder or /folder/sample_* etc).

  • files – list of filepaths to be loaded from dataset.

number_of_files: int | None
description: str | None
is_pattern: bool = False
files: list[InputFileFacet] = []
class astro.lineage.facets.OutputDatabaseDatasetFacet

Bases: astro.lineage.BaseFacet

Facet that represents output database dataset Facets for load file

Parameters:

  • metadata – metadata of the table.

  • columns – columns defined in table

  • schema – schema used.

  • used_native_path – use native support for data transfer if available on the destination.

  • enabled_native_fallback – use enable_native_fallback=True to fall back to default transfer.

  • native_support_arguments – args to be used by method involved in native support flow

  • description – description of the dataset.

metadata: astro.table.Metadata
columns: list[astro.table.Column]
schema: str | None
used_native_path: bool
enabled_native_fallback: bool | None = False
native_support_arguments: dict | None
description: str | None
class astro.lineage.facets.TableDatasetFacet

Bases: astro.lineage.BaseFacet

Facets that represent may be available for Table object

Parameters:

  • table_name – Name of the table

  • columns – columns defined in table

  • source_table_rows – Total rows in source table

  • metadata – metadata of the table.

table_name: str
columns: list[str] | tuple[str] | dict[str, str] | None
source_table_rows: int = 0
metadata: astro.table.Metadata | None
class astro.lineage.facets.SourceTableMergeDatasetFacet

Bases: astro.lineage.BaseFacet

Facets that represent may be available for Source Table object in MergeOperator

Parameters:

  • table_name – Name of the table

  • if_conflicts – The strategy to be applied if there are conflicts.

  • columns – columns defined in table

  • source_table_rows – Total rows in source table

  • metadata – metadata of the table.

table_name: str
if_conflicts: astro.constants.MergeConflictStrategy
columns: list[str] | tuple[str] | dict[str, str] | None
source_table_rows: int = 0
metadata: astro.table.Metadata | None
class astro.lineage.facets.TargetTableMergeDatasetFacet

Bases: astro.lineage.BaseFacet

Facets that represent may be available for Target Table object in Merge Operator

Parameters:

  • table_name – Name of the table

  • target_conflict_columns – List of cols where we expect to have a conflict while combining

  • columns – columns defined in table

  • metadata – metadata of the table.

table_name: str
target_conflict_columns: list[str]
columns: list[str] | tuple[str] | dict[str, str] | None
metadata: astro.table.Metadata | None