astro.sql.operators.merge

Module Contents

Classes

MergeOperator

Merge the source table rows into a destination table.

Functions

merge(*, target_table, source_table, columns, ...)

Merge the source table rows into a destination table.
class astro.sql.operators.merge.MergeOperator(*, target_table, source_table, columns, if_conflicts, target_conflict_columns, task_id='', **kwargs)

Bases: astro.sql.operators.base_operator.AstroSQLBaseOperator

Merge the source table rows into a destination table.

Parameters:

  • source_table (astro.table.BaseTable) – Contains the rows to be merged to the target_table (templated)

  • target_table (astro.table.BaseTable) – Contains the destination table in which the rows will be merged (templated)

  • columns (list[str] | tuple[str] | dict[str, str]) – List/Tuple of columns if name of source and target tables are same. If the column names in source and target tables are different pass a dictionary of source_table columns names to target_table columns names. Examples: ["sell", "list"] or {"s_sell": "t_sell", "s_list": "t_list"}

  • target_conflict_columns (list[str]) – List of cols where we expect to have a conflict while combining

  • if_conflicts (astro.constants.MergeConflictStrategy) – The strategy to be applied if there are conflicts.

  • task_id (str) –

  • kwargs (Any) –

template_fields = ('target_table', 'source_table')
execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Parameters:

context (astro.utils.compat.typing.Context) –

Return type:

astro.table.BaseTable

get_openlineage_facets_on_complete(task_instance)

Collect the input, output, job and run facets for merge operator

astro.sql.operators.merge.merge(*, target_table, source_table, columns, target_conflict_columns, if_conflicts, **kwargs)

Merge the source table rows into a destination table.

Parameters:

  • source_table (astro.table.BaseTable) – Contains the rows to be merged to the target_table (templated)

  • target_table (astro.table.BaseTable) – Contains the destination table in which the rows will be merged (templated)

  • columns (list[str] | tuple[str] | dict[str, str]) – List/Tuple of columns if name of source and target tables are same. If the column names in source and target tables are different pass a dictionary of source_table columns names to target_table columns names. Examples: ["sell", "list"] or {"s_sell": "t_sell", "s_list": "t_list"}

  • target_conflict_columns (list[str]) – List of cols where we expect to have a conflict while combining

  • if_conflicts (astro.constants.MergeConflictStrategy) – The strategy to be applied if there are conflicts.

  • kwargs (Any) – Any keyword arguments supported by the BaseOperator is supported (e.g queue, owner)

Return type:

airflow.models.xcom_arg.XComArg