export_to_file operator
When to use the export_to_file
operator
The export_to_file
operator allows you to write SQL tables to CSV or parquet files and store them locally, on S3, or on GCS. The export_to_file
function can export data from Supported Databases or a Pandas dataframe.
There are two main uses for the export_to_file
operator.
Case 1: Export data from a table.
gcs_bucket = os.getenv("GCS_BUCKET", "gs://dag-authoring") aql.export_to_file( task_id="save_file_to_gcs", input_data=t1, output_file=File( path=f"{gcs_bucket}/{{{{ task_instance_key_str }}}}/all_movies.csv", conn_id=ASTRO_GCP_CONN_ID, ), if_exists="replace", )
Case 2: Export data from a Pandas dataframe.
aql.export_file( task_id="save_dataframe_to_gcs", input_data=t2, output_file=File( path=f"{gcs_bucket}/{{{{ task_instance_key_str }}}}/top_5_movies.csv", conn_id=ASTRO_GCP_CONN_ID, ), if_exists="replace", )
Default Datasets
Input dataset - Source table for the operator.
Output dataset - Target file of the operator.