astro.files.locations.base
Module Contents
Classes
Base Location abstract class |
- class astro.files.locations.base.BaseFileLocation(path, conn_id=None, load_options=None)
Bases:
abc.ABCBase Location abstract class
- Parameters
path (str) –
conn_id (str | None) –
load_options (LoadOptions | None) –
- property smartopen_uri: str
Changes the object URI (self.path) to a SmartOpen supported URI if necessary. By default, does not change the self.path.
- Returns
URI compatible with SmartOpen for desired location.
- Return type
str
- abstract property hook
- abstract property location_type: astro.constants.FileLocation
Property to identify location type
- Return type
- abstract property paths: list[str]
Resolve patterns in path
- Return type
list[str]
- property transport_params: dict | None
Get credentials required by smart open to access files
- Return type
dict | None
- abstract property size: int
Return the size in bytes of the given file
- Return type
int
- abstract property openlineage_dataset_namespace: str
Returns the open lineage dataset namespace as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md
- Return type
str
- abstract property openlineage_dataset_name: str
Returns the open lineage dataset name as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md
- Return type
str
- template_fields = ('path', 'conn_id')
- supported_conn_type: set[str]
- validate_conn()
Check if the conn_id matches with provided path.
- static is_valid_path(path)
Check if the given path is either a valid URI or a local file
- Parameters
path (str) – Either local filesystem path or remote URI
- Return type
bool
- static check_non_existing_local_file_path(path)
Check if the path is valid by creating and temp file and then deleting it. Assumes the file don’t exist
- Parameters
path (str) –
- Return type
bool
- static get_location_type(path)
Identify where a file is located
- Parameters
path (str) – Path to a file in the filesystem/Object stores
- Return type
- exists()
Check if the file exists or not
- Return type
bool
- databricks_auth_settings()
Required settings to upload this file into databricks. Only needed for cloud storage systems like S3 :return: A dictionary of settings keys to settings values
- Return type
dict
- get_stream()
Create a file in the desired location using the smart_open.
- Parameters
df – pandas dataframe