astro.files.locations.base

Module Contents

Classes

BaseFileLocation

Base Location abstract class
class astro.files.locations.base.BaseFileLocation(path, conn_id=None, load_options=None)

Bases: abc.ABC

Base Location abstract class

Parameters:
property smartopen_uri: str

Changes the object URI (self.path) to a SmartOpen supported URI if necessary. By default, does not change the self.path.

Returns:

URI compatible with SmartOpen for desired location.

Return type:

str

abstract property hook
abstract property location_type: astro.constants.FileLocation

Property to identify location type

Return type:

astro.constants.FileLocation

abstract property paths: list[str]

Resolve patterns in path

Return type:

list[str]

property transport_params: dict | None

Get credentials required by smart open to access files

Return type:

dict | None

abstract property size: int

Return the size in bytes of the given file

Return type:

int

abstract property openlineage_dataset_namespace: str

Returns the open lineage dataset namespace as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

Return type:

str

abstract property openlineage_dataset_name: str

Returns the open lineage dataset name as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

Return type:

str

property databricks_uri: str

Return a Databricks compatible URI. In most scenarios, it will be self.path. An exception is Microsoft WASB.

Returns:

self.path

Return type:

str

property snowflake_stage_path: str

Get the altered path if needed for stage creation in snowflake stage creation

Return type:

str

template_fields = ('path', 'conn_id')
supported_conn_type: set[str]
validate_conn()

Check if the conn_id matches with provided path.

static is_valid_path(path)

Check if the given path is either a valid URI or a local file

Parameters:

path (str) – Either local filesystem path or remote URI

Return type:

bool

static check_non_existing_local_file_path(path)

Check if the path is valid by creating and temp file and then deleting it. Assumes the file don’t exist

Parameters:

path (str) –

Return type:

bool

static get_location_type(path)

Identify where a file is located

Parameters:

path (str) – Path to a file in the filesystem/Object stores

Return type:

astro.constants.FileLocation

exists()

Check if the file exists or not

Return type:

bool

databricks_auth_settings()

Required settings to upload this file into databricks. Only needed for cloud storage systems like S3 :return: A dictionary of settings keys to settings values

Return type:

dict

get_stream()

Create a file in the desired location using the smart_open.

Parameters:

df – pandas dataframe

get_snowflake_stage_auth_sub_statement()
Return type:

str