astro.files.locations.base
Module Contents
Classes
Base Location abstract class |
- class astro.files.locations.base.BaseFileLocation(path, conn_id=None)
Bases:
abc.ABCBase Location abstract class
- Parameters
path (str) –
conn_id (str | None) –
- abstract property hook
- abstract property location_type
Property to identify location type
- abstract property paths: list[str]
Resolve patterns in path
- Return type
list[str]
- property transport_params: dict | None
Get credentials required by smart open to access files
- Return type
dict | None
- abstract property size: int
Return the size in bytes of the given file
- Return type
int
- abstract property openlineage_dataset_namespace: str
Returns the open lineage dataset namespace as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md
- Return type
str
- abstract property openlineage_dataset_name: str
Returns the open lineage dataset name as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md
- Return type
str
- template_fields = ['path', 'conn_id']
- static is_valid_path(path)
Check if the given path is either a valid URI or a local file
- Parameters
path (str) – Either local filesystem path or remote URI
- Return type
bool
- static check_non_existing_local_file_path(path)
Check if the path is valid by creating and temp file and then deleting it. Assumes the file don’t exist
- Parameters
path (str) –
- Return type
bool
- static get_location_type(path)
Identify where a file is located
- Parameters
path (str) – Path to a file in the filesystem/Object stores
- Return type
- exists()
Check if the file exists or not
- Return type
bool