astro.files.locations

Subpackages

• astro.files.locations.amazon
  • astro.files.locations.amazon.s3
• astro.files.locations.google
  • astro.files.locations.google.gcs

Submodules

• astro.files.locations.base
• astro.files.locations.http
• astro.files.locations.local

Package Contents

Functions

get_class_name(module_ref[, suffix])	Get class name to be dynamically imported. Class name are expected to be in following formats
get_dict_with_module_names_to_dot_notations(base_path)	Given a directory, recursively identify which modules exist within it
create_file_location(path[, conn_id])	Location factory method to generate location class

Classes

FileLocation	Generic enumeration.
BaseFileLocation	Base Location abstract class

Attributes

DEFAULT_CONN_TYPE_TO_MODULE_PATH

astro.files.locations.get_class_name(module_ref, suffix='Location')

Get class name to be dynamically imported. Class name are expected to be in following formats example - module name: test suffix: Abc

expected class names -

1. TESTAbc

2. TestAbc

Parameters

• module_ref (Any) – Module from which to get class location type implementation

• suffix (str) – suffix for class name

Return type

str

astro.files.locations.get_dict_with_module_names_to_dot_notations(base_path)

Given a directory, recursively identify which modules exist within it (ignoring __init__.py & base.py) and create a dictionary which has module names as keys and the values are the dot notation import paths.

An example:

├── package

├── module.py ├── subpackage

├── __init__.py └── subpackage_module.py

Running:

from pathlib import Path from astro.utils.path import get_dict_with_module_names_to_dot_notations values = get_dict_with_module_names_to_dot_notations(Path(“package”)) print(values)

Prints:

{

“module”: “package.module”, “subpackage_module”: “package.subpackage.subpackage_module”

}

Parameters

base_path (pathlib.Path) –

Return type

Dict[str, str]

astro.files.locations.create_file_location(path, conn_id=None)

Location factory method to generate location class :param path: Path to a file in the filesystem/Object stores :param conn_id: Airflow connection ID

Parameters

• path (str) –

• conn_id (Optional[str]) –

Return type

base.BaseFileLocation

astro.files.locations.DEFAULT_CONN_TYPE_TO_MODULE_PATH

class astro.files.locations.FileLocation

Bases: enum.Enum

Generic enumeration.

Derive from this class to define new enumerations.

LOCAL = local

HTTP = http

HTTPS = https

GS = gs

S3 = s3

class astro.files.locations.BaseFileLocation(path, conn_id=None)

Bases: abc.ABC

Base Location abstract class

Parameters

• path (str) –

• conn_id (Optional[str]) –

template_fields = ['path', 'conn_id']

property location_type

Property to identify location type

property paths

Resolve patterns in path

Return type

List[str]

property transport_params

Get credentials required by smart open to access files

Return type

Union[Dict, None]

property size

Return the size in bytes of the given file

static is_valid_path(path)

Check if the given path is either a valid URI or a local file

Parameters

path (str) – Either local filesystem path or remote URI

Return type

bool

static check_non_existing_local_file_path(path)

Check if the path is valid by creating and temp file and then deleting it. Assumes the file don’t exist

Parameters

path (str) –

Return type

bool

static get_location_type(path)

Identify where a file is located

Parameters

path (str) – Path to a file in the filesystem/Object stores

Return type

astro.constants.FileLocation

exists()

Check if the file exists or not

Return type

bool

__repr__()

Return repr(self).

__str__()

String representation of location