astro.files.types.ndjson

Module Contents

Classes

NDJSONFileType

Concrete implementation to handle NDJSON file type

class astro.files.types.ndjson.NDJSONFileType(path, normalize_config=None, load_options=None)

Bases: astro.files.types.base.FileType

Concrete implementation to handle NDJSON file type

Parameters:

• path (str) –

• normalize_config (dict | None) –

• load_options (LoadOptions | None) –

property name

get file type

LOAD_OPTIONS_CLASS_NAME = PandasLoadOptions

export_to_dataframe(stream, columns_names_capitalization='original', **kwargs)

read ndjson file from one of the supported locations and return dataframe

Parameters:

• stream – file stream object

• load_options – Pandas option to pass to the Pandas lib while reading ndjson

• columns_names_capitalization – determines whether to convert all columns to lowercase/uppercase in the resulting dataframe

create_from_dataframe(df, stream)

Write ndjson file to one of the supported locations

Parameters:

• df (pandas.DataFrame) – pandas dataframe

• stream (io.TextIOWrapper) – file stream object

Return type:

None

static flatten(normalize_config, stream, **kwargs)

Flatten the nested ndjson/json.

Parameters:

• normalize_config (dict) – parameters in dict format of pandas json_normalize() function. https://pandas.pydata.org/docs/reference/api/pandas.json_normalize.html

• stream (io.TextIOWrapper) – io.TextIOWrapper object for the file

Returns:

return dataframe containing the loaded data

Return type:

pandas.DataFrame