astro.options

Module Contents

Classes

LoadOptions
LoadOptionsList
SnowflakeLoadOptions	Load options to load file to snowflake using native approach.
WASBLocationLoadOptions

Functions

contains_required_option(load_options, option_name)	Check required options in load_option class
list_to_dict(value)	Convert list object to dict

class astro.options.LoadOptions

empty()

to_dict()

Convert options class to dict

Return type:

dict

astro.options.contains_required_option(load_options, option_name)

Check required options in load_option class

Parameters:

• load_options (LoadOptions | None) –

• option_name (str) –

Return type:

bool

astro.options.list_to_dict(value)

Convert list object to dict

Parameters:

value (list[LoadOptions] | None) –

Return type:

dict[str, LoadOptions] | None

class astro.options.LoadOptionsList

get(option_class)

Check LOAD_OPTIONS_CLASS_NAME attribute and select the correct load_options :param option_class: FileType | FileLocation | BaseDatabase

Return type:

LoadOptions | None

get_by_class_name(option_class_name)

Get load_option by class name :return:

Return type:

LoadOptions | None

class astro.options.SnowflakeLoadOptions

Bases: LoadOptions

Load options to load file to snowflake using native approach.

Parameters:

• copy_options – Specify one or more of the copy option as key-value pair. Read more at: https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#copy-options-copyoptions

• file_options – Depending on the file format type specified, use one or more of the format-specific options as key-value pair. Read more at: https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#format-type-options-formattypeoptions

• metadata_columns – Specify one or more metadata columns to be included from Snowflake stage into the target table. Read more at: https://docs.snowflake.com/en/user-guide/querying-metadata#example-3-loading-metadata-columns-into-a-table

• storage_integration – Specify the previously created Snowflake storage integration

• validation_mode – Defaults to validation_mode=None. This instructs the COPY command to validate the data files instead of loading them into the specified table; i.e. the COPY command tests the files for errors but does not load them. Supported validation mode; RETURN_n_ROWS | RETURN_ERRORS | RETURN_ALL_ERRORS

Note

Specify the supported validation mode;

• RETURN_n_ROWS: validates the specified n rows if no errors are encountered; otherwise, fails at the first error encountered in the rows.

• RETURN_ERRORS: returns all errors (parsing, conversion, etc.) across all files specified in the COPY statement.

• RETURN_ALL_ERRORS: returns all errors across all files specified in the COPY statement, including files with errors that were partially loaded during an earlier load because the ON_ERROR copy option was set to CONTINUE during the load.

Read more at: https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#optional-parameters

Note

If you specify validation_mode and metadata_columns together, metadata_columns will not be loaded as the transformed COPY INTO command fails with SQL compilation error, and the load file operation will fall back to the non-native approach using pandas.

copy_options: dict

file_options: dict

metadata_columns: list[str]

storage_integration: str

validation_mode: str

empty()

class astro.options.WASBLocationLoadOptions

Bases: LoadOptions

storage_account: str