astro.files

Subpackages

• astro.files.locations
  • astro.files.locations.amazon
    • astro.files.locations.amazon.s3
  • astro.files.locations.azure
    • astro.files.locations.azure.wasb
  • astro.files.locations.google
    • astro.files.locations.google.gcs
    • astro.files.locations.google.gdrive
  • astro.files.locations.base
  • astro.files.locations.ftp
  • astro.files.locations.http
  • astro.files.locations.local
  • astro.files.locations.sftp
• astro.files.operators
  • astro.files.operators.files
• astro.files.types
  • astro.files.types.base
  • astro.files.types.csv
  • astro.files.types.excel
  • astro.files.types.json
  • astro.files.types.ndjson
  • astro.files.types.parquet
  • astro.files.types.xls
  • astro.files.types.xlsx

Submodules

• astro.files.base

Package Contents

Functions

get_file_list(path, conn_id, **kwargs)

List file path from a remote object store or the local filesystem based on the given path pattern.

astro.files.get_file_list(path, conn_id, **kwargs)

List file path from a remote object store or the local filesystem based on the given path pattern. It is not advisable to fetch huge number of files since it would overload the XCOM and also, if you are using response of this method in dynamic task map expand method it can create huge number of parallel task

Supported filesystem: Supported File Location

Supported file pattern: Patterns in File path

Parameters:

• conn_id (str) – Airflow connection id. This will be used to identify the right Airflow hook at runtime to connect with storage services

• path (str) – Path pattern to the file in the filesystem/Object stores

Return type:

airflow.models.xcom_arg.XComArg