astro.sql.operators.merge

Module Contents

Classes

MergeOperator

Merge the source table rows into a destination table.

Functions

merge(*, target_table, source_table, columns, ...)

Merge the source table rows into a destination table.
class astro.sql.operators.merge.MergeOperator(*, target_table, source_table, columns, if_conflicts, target_conflict_columns, task_id='', **kwargs)

Bases: airflow.models.baseoperator.BaseOperator

Merge the source table rows into a destination table.

Parameters

  • source_table (astro.sql.table.Table) – Contains the rows to be merged to the target_table (templated)

  • target_table (astro.sql.table.Table) – Contains the destination table in which the rows will be merged (templated)

  • columns (list[str] | tuple[str] | dict[str, str]) – List/Tuple of columns if name of source and target tables are same. If the column names in source and target tables are different pass a dictionary of source_table columns names to target_table columns names. Examples: ["sell", "list"] or {"s_sell": "t_sell", "s_list": "t_list"}

  • target_conflict_columns (list[str]) – List of cols where we expect to have a conflict while combining

  • if_conflicts (astro.constants.MergeConflictStrategy) – The strategy to be applied if there are conflicts.

  • task_id (str) –

  • kwargs (Any) –

template_fields = ['target_table', 'source_table']
execute(context)

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Parameters

context (dict) –

Return type

astro.sql.table.Table

astro.sql.operators.merge.merge(*, target_table, source_table, columns, target_conflict_columns, if_conflicts, **kwargs)

Merge the source table rows into a destination table.

Parameters

  • source_table (astro.sql.table.Table) – Contains the rows to be merged to the target_table (templated)

  • target_table (astro.sql.table.Table) – Contains the destination table in which the rows will be merged (templated)

  • columns (list[str] | tuple[str] | dict[str, str]) – List/Tuple of columns if name of source and target tables are same. If the column names in source and target tables are different pass a dictionary of source_table columns names to target_table columns names. Examples: ["sell", "list"] or {"s_sell": "t_sell", "s_list": "t_list"}

  • target_conflict_columns (list[str]) – List of cols where we expect to have a conflict while combining

  • if_conflicts (astro.constants.MergeConflictStrategy) – The strategy to be applied if there are conflicts.

  • kwargs (Any) – Any keyword arguments supported by the BaseOperator is supported (e.g queue, owner)

Return type

airflow.models.xcom_arg.XComArg