export_to_file operator

When to use the export_to_file operator

The export_to_file operator allows you to write SQL tables to CSV or parquet files and store them locally, on S3, or on GCS. The export_to_file function can export data from Supported Databases or a Pandas dataframe.

There are two main uses for the export_to_file operator.

Case 1: Export data from a table.

    gcs_bucket = os.getenv("GCS_BUCKET", "gs://dag-authoring")

    aql.export_to_file(
        task_id="save_file_to_gcs",
        input_data=t1,
        output_file=File(
            path=f"{gcs_bucket}/{{{{ task_instance_key_str }}}}/all_movies.csv",
            conn_id="gcp_conn",
        ),
        if_exists="replace",
    )

Case 2: Export data from a Pandas dataframe.

    aql.export_file(
        task_id="save_dataframe_to_gcs",
        input_data=t2,
        output_file=File(
            path=f"{gcs_bucket}/{{{{ task_instance_key_str }}}}/top_5_movies.csv",
            conn_id="gcp_conn",
        ),
        if_exists="replace",
    )

Default Datasets

  • Input dataset - Source table for the operator.

  • Output dataset - Target file of the operator.