Skip to content
This repository was archived by the owner on May 22, 2025. It is now read-only.
This repository was archived by the owner on May 22, 2025. It is now read-only.

output_table control in "aql.transform" #2082

@Andrew-Wichmann

Description

@Andrew-Wichmann

Please describe the feature you'd like to see
I would really like to be able to configure the output_table for the TransformOperator when using the @transform decorator.

Describe the solution you'd like

PRODUCTION_DATABASE = Table(
    name="IMAGE",
    conn_id=SNOWFLAKE_CONN_ID,
)

@dag(
    schedule="@daily",
    start_date=datetime(2023, 12, 12),
)
def fetch_missing_files():
    @aql.transform(output_table=Table(conn_id="some_other_db"))
    def fetch_image(prod_database):
        return """
        SELECT image
        FROM {{prod_database}}
        WHERE <condition>
        """
    fetch_image(PRODUCTION_DATABASE)

Are there any alternatives to this feature?
🤔 I'm not sure. I mean, I could just not use the decorator and initialize the Operator myself, but that's not as fun.

Additional context
Nope. I love the product though. I'm really excited to start using more Airflow =)

Acceptance Criteria

  • All checks and tests in the CI should pass
  • Unit tests (90% code coverage or more, once available)
  • Integration tests (if the feature relates to a new database or external service)
  • Example DAG
  • Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • Exception handling in case of errors
  • Logging (are we exposing useful information to the user? e.g. source and destination)
  • Improve the documentation (README, Sphinx, and any other relevant)
  • How to use Guide for the feature (example)

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions