astro.databases.aws.redshift

AWS Redshift table implementation.

Module Contents

Classes

RedshiftDatabase

Handle interactions with Redshift databases.

Attributes

DEFAULT_CONN_ID

NATIVE_PATHS_SUPPORTED_FILE_TYPES

astro.databases.aws.redshift.DEFAULT_CONN_ID
astro.databases.aws.redshift.NATIVE_PATHS_SUPPORTED_FILE_TYPES
class astro.databases.aws.redshift.RedshiftDatabase(conn_id=DEFAULT_CONN_ID)

Bases: astro.databases.base.BaseDatabase

Handle interactions with Redshift databases.

Parameters

conn_id (str) –

property sql_type
property hook: airflow.providers.amazon.aws.hooks.redshift_sql.RedshiftSQLHook

Retrieve Airflow hook to interface with the Redshift database.

Return type

airflow.providers.amazon.aws.hooks.redshift_sql.RedshiftSQLHook

property sqlalchemy_engine: sqlalchemy.engine.base.Engine

Return SQAlchemy engine.

Return type

sqlalchemy.engine.base.Engine

property default_metadata: astro.sql.table.Metadata

Fill in default metadata values for table objects addressing redshift databases

Return type

astro.sql.table.Metadata

NATIVE_LOAD_EXCEPTIONS :Any
DEFAULT_SCHEMA
NATIVE_PATHS
illegal_column_name_chars :List[str] = ['.']
illegal_column_name_chars_replacement :List[str] = ['_']
schema_exists(schema)

Checks if a dataset exists in the Redshift

Parameters

schema (str) – Redshift namespace

Return type

bool

table_exists(table)

Check if a table exists in the database.

Parameters

table (astro.sql.table.BaseTable) – Details of the table we want to check that exists

Return type

bool

load_pandas_dataframe_to_table(source_dataframe, target_table, if_exists='replace', chunk_size=DEFAULT_CHUNK_SIZE)

Create a table with the dataframe’s contents. If the table already exists, append or replace the content, depending on the value of if_exists.

Parameters

  • source_dataframe (pandas.DataFrame) – Local or remote filepath

  • target_table (astro.sql.table.BaseTable) – Table in which the file will be loaded

  • if_exists (astro.constants.LoadExistStrategy) – Strategy to be used in case the target table already exists.

  • chunk_size (int) – Specify the number of rows in each batch to be written at a time.

Return type

None

merge_table(source_table, target_table, source_to_target_columns_map, target_conflict_columns, if_conflicts='exception')

Merge the source table rows into a destination table. The argument if_conflicts allows the user to define how to handle conflicts.

Parameters

  • source_table (astro.sql.table.BaseTable) – Contains the rows to be merged to the target_table

  • target_table (astro.sql.table.BaseTable) – Contains the destination table in which the rows will be merged

  • source_to_target_columns_map (Dict[str, str]) – Dict of target_table columns names to source_table columns names

  • target_conflict_columns (List[str]) – List of cols where we expect to have a conflict while combining

  • if_conflicts (astro.constants.MergeConflictStrategy) – The strategy to be applied if there are conflicts.

Return type

None

is_native_load_file_available(source_file, target_table)

Check if there is an optimised path for source to destination.

Parameters

  • source_file (astro.files.File) – File from which we need to transfer data

  • target_table (astro.sql.table.BaseTable) – Table that needs to be populated with file data

Return type

bool

load_file_to_table_natively(source_file, target_table, if_exists='replace', native_support_kwargs=None, **kwargs)

Checks if optimised path for transfer between File location to database exists and if it does, it transfers it and returns true else false.

Parameters

  • source_file (astro.files.File) – File from which we need to transfer data

  • target_table (astro.sql.table.BaseTable) – Table that needs to be populated with file data

  • if_exists (astro.constants.LoadExistStrategy) – Overwrite file if exists. Default False

  • native_support_kwargs (Optional[Dict]) – kwargs to be used by method involved in native support flow

load_s3_file_to_table(source_file, target_table, native_support_kwargs=None, **kwargs)

Load content of multiple files in S3 to output_table in Redshift by: - Creating a table - Using the COPY command

Parameters

  • source_file (astro.files.File) – Source file that is used as source of data

  • target_table (astro.sql.table.BaseTable) – Table that will be created on the redshift

  • if_exists – Overwrite table if exists. Default ‘replace’

  • native_support_kwargs (Optional[Dict]) – kwargs to be used by method involved in native support flow

See also

Redshift official documentation on CREATE TABLE Redshift official documentation on COPY

static get_merge_initialization_query(parameters)

Handles database-specific logic to handle constraints for Redshift.

Parameters

parameters (tuple) –

Return type

str