astro.files

Subpackages

• astro.files.locations
  • astro.files.locations.amazon
    • astro.files.locations.amazon.s3
  • astro.files.locations.google
    • astro.files.locations.google.gcs
  • astro.files.locations.base
  • astro.files.locations.http
  • astro.files.locations.local
• astro.files.operators
  • astro.files.operators.files
• astro.files.types
  • astro.files.types.base
  • astro.files.types.csv
  • astro.files.types.json
  • astro.files.types.ndjson
  • astro.files.types.parquet

Submodules

• astro.files.base

Package Contents

Functions

check_if_connection_exists(conn_id)	Given an Airflow connection ID, identify if it exists.
get_file_list(path, conn_id, **kwargs)	List file path from a remote object store or the local filesystem based on the given path pattern.

astro.files.check_if_connection_exists(conn_id)

Given an Airflow connection ID, identify if it exists. Return True if it does or raise an AirflowNotFoundException exception if it does not.

Parameters

conn_id (str) – Airflow connection ID

Return bool

If the connection exists, return True

Return type

bool

astro.files.get_file_list(path, conn_id, **kwargs)

List file path from a remote object store or the local filesystem based on the given path pattern. It is not advisable to fetch huge number of files since it would overload the XCOM and also, if you are using response of this method in dynamic task map expand method it can create huge number of parallel task

Supported filesystem: Supported File Location

Supported file pattern: Patterns in File path

Parameters

• conn_id (str) – Airflow connection id. This will be used to identify the right Airflow hook at runtime to connect with storage services

• path (str) – Path pattern to the file in the filesystem/Object stores

Return type

airflow.models.xcom_arg.XComArg