astro.sql

Subpackages

• astro.sql.operators
  • astro.sql.operators.append
  • astro.sql.operators.base_decorator
  • astro.sql.operators.base_operator
  • astro.sql.operators.cleanup
  • astro.sql.operators.dataframe
  • astro.sql.operators.drop
  • astro.sql.operators.export_file
  • astro.sql.operators.export_table_to_file
  • astro.sql.operators.export_to_file
  • astro.sql.operators.load_file
  • astro.sql.operators.merge
  • astro.sql.operators.raw_sql
  • astro.sql.operators.transform
  • astro.sql.operators.upstream_task_mixin

Submodules

• astro.sql.table

Package Contents

Classes

AppendOperator	Append the source table rows into a destination table.
CleanupOperator	Clean up temporary tables at the end of a DAG run.
DataframeOperator	Converts a SQL table into a dataframe. Users can then give a python function that takes a dataframe as
DropTableOperator	Airflow Operator for dropping SQL tables.
ExportFileOperator	Write SQL table to csv/parquet on local/S3/GCS.
ExportTableToFileOperator	Write SQL table to csv/parquet on local/S3/GCS.
ExportToFileOperator	Write SQL table to csv/parquet on local/S3/GCS.
LoadFileOperator	Load S3/local file into either a database or a pandas dataframe
MergeOperator	Merge the source table rows into a destination table.
TransformOperator	Given a SQL statement and (optional) tables, execute the SQL statement and output
Metadata	Contains additional information to access a SQL Table, which is very likely optional and, in some cases, may
Table	User-facing class that has information necessary to access a SQL Table. It is agnostic to the database type.

Functions

append(*, source_table, target_table[, columns])	Append the source table rows into a destination table.
cleanup([tables_to_cleanup])	Clean up temporary tables once either the DAG or upstream tasks are done
dataframe([python_callable, multiple_outputs, ...])	This decorator will allow users to write python functions while treating SQL tables as dataframes
drop_table(table, **kwargs)	Drops a table.
export_file(input_data, output_file[, if_exists, task_id])	Convert ExportFileOperator into a function. Returns XComArg.
export_table_to_file(input_data, output_file[, ...])	Convert ExportTableToFileOperator into a function. Returns XComArg.
export_to_file(input_data, output_file[, if_exists, ...])	Convert ExportToFileOperator into a function. Returns XComArg.
load_file(input_file[, output_table, task_id, ...])	Load a file or bucket into either a SQL table or a pandas dataframe.
merge(*, target_table, source_table, columns, ...)	Merge the source table rows into a destination table.
run_raw_sql([python_callable, conn_id, parameters, ...])	Given a python function that returns a SQL statement and (optional) tables, execute the SQL statement and output
transform([python_callable, conn_id, parameters, ...])	Given a python function that returns a SQL statement and (optional) tables, execute the SQL statement and output
transform_file(file_path[, conn_id, parameters, ...])	This function returns a Table object that can be passed to future tasks from specified SQL file.
get_value_list(sql, conn_id, **kwargs)	Execute a sql statement and return the result.

astro.sql.get_value_list(sql, conn_id, **kwargs)

Execute a sql statement and return the result. By default, the response size is less than equal to value of max_map_length conf. You can call a callable handler to alter the response by default it call fetchall on database result set.

Parameters:

• sql (str) – sql query to execute. If the sql query will return huge number of row then it can overload the XCOM. also, If you are using output of this method to expand a task using dynamic task map then it can create lots of parallel task. So it is advisable to limit your sql query statement.

• conn_id (str) – Airflow connection id. This connection id will be used to identify the database client and connect with it at runtime

Return type:

airflow.models.xcom_arg.XComArg