astro.files.locations.azure.wasb
Module Contents
Classes
Handler WASB object store operations |
- class astro.files.locations.azure.wasb.WASBLocation(path, conn_id=None, load_options=None)
Bases:
astro.files.locations.base.BaseFileLocationHandler WASB object store operations
- Parameters:
path (str) –
conn_id (str | None) –
load_options (LoadOptions | None) –
- property hook: airflow.providers.microsoft.azure.hooks.wasb.WasbHook
- Return type:
airflow.providers.microsoft.azure.hooks.wasb.WasbHook
- property transport_params: dict
get WASB credentials for storage
- Return type:
dict
- property paths: list[str]
Resolve WASB file paths with prefix
- Return type:
list[str]
- property smartopen_uri: str
SmartOpen does not support URIs prefixed with wasb, so we need to change them to azure.
- Returns:
URI compatible with SmartOpen for Azure BlobStorage.
- Return type:
str
- property size: int
Return file size for WASB location
- Return type:
int
- property openlineage_dataset_namespace: str
Returns the open lineage dataset namespace as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md
- Return type:
str
- property openlineage_dataset_name: str
Returns the open lineage dataset name as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md
- Return type:
str
- property snowflake_stage_path: str
- Get the altered path if needed for stage creation in snowflake stage creation. We need to modify the path since
Snowflake only accepts paths of format for stage creation: “azure://<storage_account>.blob.core.windows.net/<container_name>/load/files/” But SDK accepts paths “wasb://<container_name>/<filename>” or “wasbs://<container_name>/<filename>” To bridge the gap we use this method
- Return type:
str
- location_type
- supported_conn_type
- LOAD_OPTIONS_CLASS_NAME = WASBLocationLoadOptions
- AZURE_HOST = blob.core.windows.net
- exists()
Check if the file exists or not
- Return type:
bool