astro.files.locations.google.gcs

Module Contents

Classes

GCSLocation

Handler GS object store operations
class astro.files.locations.google.gcs.GCSLocation(path, conn_id=None, load_options=None)

Bases: astro.files.locations.base.BaseFileLocation

Handler GS object store operations

Parameters:
property hook: airflow.providers.google.cloud.hooks.gcs.GCSHook
Return type:

airflow.providers.google.cloud.hooks.gcs.GCSHook

property transport_params: dict

get GCS credentials for storage

Return type:

dict

property paths: list[str]

Resolve GS file paths with prefix

Return type:

list[str]

property size: int

Return file size for GCS location

Return type:

int

property openlineage_dataset_namespace: str

Returns the open lineage dataset namespace as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

Return type:

str

property openlineage_dataset_name: str

Returns the open lineage dataset name as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

Return type:

str

location_type
supported_conn_type
databricks_auth_settings()

Required settings to upload this file into databricks. Only needed for cloud storage systems like S3 :return: A dictionary of settings keys to settings values

Return type:

dict

exists()

Check if the file exists or not

Return type:

bool