astro.databases.databricks.load_file.load_file_job

Module Contents

Functions

load_file_to_delta(input_file, delta_table, ...[, ...])

Load a file object into a databricks delta table

Attributes

cwd

supported_file_locations

log

astro.databases.databricks.load_file.load_file_job.cwd
astro.databases.databricks.load_file.load_file_job.supported_file_locations
astro.databases.databricks.load_file.load_file_job.log
astro.databases.databricks.load_file.load_file_job.load_file_to_delta(input_file, delta_table, databricks_job_name, delta_load_options, if_exists='replace')

Load a file object into a databricks delta table

Parameters

  • if_exists (astro.constants.LoadExistStrategy) – How to handle the data if the table exists. Defaults to replace (but can be set to “append”)

  • delta_load_options (astro.databases.databricks.load_options.DeltaLoadOptions) – Loading options for passing data into delta. Things like COPY_OPTIONS, cluster_id, etc.

  • databricks_job_name (str) – The name of the job as it will show up in databricks.

  • input_file (astro.files.File) – File to load into delta

  • delta_table (astro.table.BaseTable) – a Table object with necessary metadata for accessing the cluster.