astro.options
Module Contents
Classes
Load options to load file to snowflake using native approach. |
|
Functions
|
Check required options in load_option class |
|
Convert list object to dict |
- class astro.options.LoadOptions
- empty()
- to_dict()
Convert options class to dict
- Return type:
dict
- astro.options.contains_required_option(load_options, option_name)
Check required options in load_option class
- Parameters:
load_options (LoadOptions | None) –
option_name (str) –
- Return type:
bool
- astro.options.list_to_dict(value)
Convert list object to dict
- Parameters:
value (list[LoadOptions] | None) –
- Return type:
dict[str, LoadOptions] | None
- class astro.options.LoadOptionsList
- get(option_class)
Check LOAD_OPTIONS_CLASS_NAME attribute and select the correct load_options :param option_class: FileType | FileLocation | BaseDatabase
- Return type:
LoadOptions | None
- get_by_class_name(option_class_name)
Get load_option by class name :return:
- Return type:
LoadOptions | None
- class astro.options.SnowflakeLoadOptions
Bases:
LoadOptions
Load options to load file to snowflake using native approach.
- Parameters:
copy_options – Specify one or more of the copy option as key-value pair. Read more at: https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#copy-options-copyoptions
file_options – Depending on the file format type specified, use one or more of the format-specific options as key-value pair. Read more at: https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#format-type-options-formattypeoptions
metadata_columns – Specify one or more metadata columns to be included from Snowflake stage into the target table. Read more at: https://docs.snowflake.com/en/user-guide/querying-metadata#example-3-loading-metadata-columns-into-a-table
storage_integration – Specify the previously created Snowflake storage integration
validation_mode – Defaults to validation_mode=None. This instructs the COPY command to validate the data files instead of loading them into the specified table; i.e. the COPY command tests the files for errors but does not load them. Supported validation mode; RETURN_n_ROWS | RETURN_ERRORS | RETURN_ALL_ERRORS
Note
Specify the supported validation mode;
RETURN_n_ROWS: validates the specified n rows if no errors are encountered; otherwise, fails at the first error encountered in the rows.
RETURN_ERRORS: returns all errors (parsing, conversion, etc.) across all files specified in the COPY statement.
RETURN_ALL_ERRORS: returns all errors across all files specified in the COPY statement, including files with errors that were partially loaded during an earlier load because the ON_ERROR copy option was set to CONTINUE during the load.
Read more at: https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#optional-parameters
Note
If you specify validation_mode and metadata_columns together, metadata_columns will not be loaded as the transformed COPY INTO command fails with SQL compilation error, and the load file operation will fall back to the non-native approach using pandas.
- copy_options: dict
- file_options: dict
- metadata_columns: list[str]
- storage_integration: str
- validation_mode: str
- empty()
- class astro.options.WASBLocationLoadOptions
Bases:
LoadOptions
- storage_account: str