astro.files.operators.files

Module Contents

Classes

ListFileOperator

List the file available at path and storage

Functions

get_file_list_func(path, conn_id)

Function to get list of files from a bucket
class astro.files.operators.files.ListFileOperator(path, conn_id, task_id='', **kwargs)

Bases: airflow.models.BaseOperator

List the file available at path and storage

Parameters:

  • task_id (str) – The task id for uniquely identify a task in a DAG

  • path (str) – A path pattern for which you want to get a list of file

  • conn_id (str) – Airflow connection id. This will be used to identify the right Airflow hook at runtime to connect with storage services

template_fields = ('path', 'conn_id')
execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Parameters:

context (astro.utils.compat.typing.Context) –

Return type:

Any

astro.files.operators.files.get_file_list_func(path, conn_id)

Function to get list of files from a bucket

Parameters:

  • path (str) –

  • conn_id (str) –

Return type:

list[str]