astro.sql

Subpackages

• astro.sql.operators
  • astro.sql.operators.append
  • astro.sql.operators.base_decorator
  • astro.sql.operators.base_operator
  • astro.sql.operators.cleanup
  • astro.sql.operators.dataframe
  • astro.sql.operators.drop
  • astro.sql.operators.export_file
  • astro.sql.operators.load_file
  • astro.sql.operators.merge
  • astro.sql.operators.raw_sql
  • astro.sql.operators.transform
  • astro.sql.operators.upstream_task_mixin

Submodules

• astro.sql.table

Package Contents

Classes

AppendOperator	Append the source table rows into a destination table.
CleanupOperator	Clean up temporary tables at the end of a DAG run. Temporary tables are the ones that are
DataframeOperator	Converts a SQL table into a dataframe. Users can then give a python function that takes a dataframe as
DropTableOperator	Airflow Operator for dropping SQL tables.
ExportFileOperator	Write SQL table to csv/parquet on local/S3/GCS.
LoadFileOperator	Load S3/local file into either a database or a pandas dataframe
MergeOperator	Merge the source table rows into a destination table.
TransformOperator	Given a SQL statement and (optional) tables, execute the SQL statement and output
Metadata	Contains additional information to access a SQL Table, which is very likely optional and, in some cases, may
Table	User-facing class that has information necessary to access a SQL Table. It is agnostic to the database type.

Functions

append(*, source_table, target_table[, columns])	Append the source table rows into a destination table.
cleanup([tables_to_cleanup])	Clean up temporary tables once either the DAG or upstream tasks are done
dataframe([python_callable, multiple_outputs, ...])	This decorator will allow users to write python functions while treating SQL tables as dataframes
drop_table(table, **kwargs)	Drops a table.
export_file(input_data, output_file[, if_exists, task_id])	Convert ExportFileOperator into a function. Returns XComArg.
load_file(input_file[, output_table, task_id, ...])	Load a file or bucket into either a SQL table or a pandas dataframe.
merge(*, target_table, source_table, columns, ...)	Merge the source table rows into a destination table.
run_raw_sql([python_callable, conn_id, parameters, ...])	Given a python function that returns a SQL statement and (optional) tables, execute the SQL statement and output
transform([python_callable, conn_id, parameters, ...])	Given a python function that returns a SQL statement and (optional) tables, execute the SQL statement and output
transform_file(file_path[, conn_id, parameters, ...])	This function returns a Table object that can be passed to future tasks from specified SQL file.