`astro.databases.google.bigquery`

Google BigQuery table implementation.

Module Contents

Classes

BigqueryDatabase

Handle interactions with Bigquery databases. If this class is successful, we should not have any Bigquery-specific

Attributes

DEFAULT_CONN_ID

astro.databases.google.bigquery.DEFAULT_CONN_ID

class astro.databases.google.bigquery.BigqueryDatabase(conn_id=DEFAULT_CONN_ID)

Bases: astro.databases.base.BaseDatabase

Handle interactions with Bigquery databases. If this class is successful, we should not have any Bigquery-specific logic in other parts of our code-base.

Parameters: conn_id (str) –

illegal_column_name_chars :List[str] = ['.']

illegal_column_name_chars_replacement :List[str] = ['_']

property sql_type

property hook

Retrieve Airflow hook to interface with the BigQuery database.

Return type: airflow.providers.google.cloud.hooks.bigquery.BigQueryHook

property sqlalchemy_engine

Return SQAlchemy engine.

Return type: sqlalchemy.engine.base.Engine

property default_metadata

Fill in default metadata values for table objects addressing bigquery databases

Returns
Return type: astro.sql.table.Metadata

schema_exists(schema)

Checks if a dataset exists in the BigQuery

Parameters: schema (str) – Bigquery namespace
Return type: bool

static get_merge_initialization_query(parameters)

Handles database-specific logic to handle constraints for BigQuery. The only constraint that BigQuery supports is NOT NULL.

Parameters: parameters (Tuple) –
Return type: str

load_pandas_dataframe_to_table(source_dataframe, target_table, if_exists='replace', chunk_size=DEFAULT_CHUNK_SIZE)

Create a table with the dataframe’s contents. If the table already exists, append or replace the content, depending on the value of if_exists.

Parameters

source_dataframe (pandas.DataFrame) – Local or remote filepath
target_table (astro.sql.table.Table) – Table in which the file will be loaded
if_exists (astro.constants.LoadExistStrategy) – Strategy to be used in case the target table already exists.
chunk_size (int) – Specify the number of rows in each batch to be written at a time.

Return type

None

merge_table(source_table, target_table, source_to_target_columns_map, target_conflict_columns, if_conflicts='exception')

Merge the source table rows into a destination table. The argument if_conflicts allows the user to define how to handle conflicts.