astro.utils.dataframe

Module Contents

Functions

convert_columns_names_capitalization(df, ...)

Convert cols of a dataframe to required case. Options - lower/Upper

convert_dataframe_to_file(df)

Passes a dataframe into a File using parquet as an efficient storage format. This allows us to use
astro.utils.dataframe.convert_columns_names_capitalization(df, columns_names_capitalization)

Convert cols of a dataframe to required case. Options - lower/Upper

Parameters:

  • df (pandas.DataFrame) – dataframe whose cols will be altered

  • columns_names_capitalization (astro.constants.ColumnCapitalization) – String Literal with possible values - lower/Upper

astro.utils.dataframe.convert_dataframe_to_file(df)

Passes a dataframe into a File using parquet as an efficient storage format. This allows us to use Json as a storage method without filling the metadata database. the values for conn_id and bucket path can be found in the airflow.cfg as follows:

[astro] xcom_storage_conn_id=… xcom_storage_url=/// :param df: Dataframe to convert to file :return: File object with reference to stored dataframe file

Parameters:

df (pandas.DataFrame) –

Return type:

astro.files.File