This site uses different types of cookies, including analytics and functional cookies (its own and from other sites). To change your cookie settings or find out more, click here. If you continue browsing our website, you accept these cookies.
At our company we have large amounts of data (2bn records per month) partitioned and stored at Redshift as monthly tables. This setup complicates our queries, since 1) the table name changes (increments) over time, and 2) we have to do joins to reduce the amount of data and this limits the use of views.
Because of the amount of data we have to operate in In-DB workflows. I am trying to build a dynamic query in a standard workflow that could replace a placeholder query within an In-DB workflow. The query builder works; Both checkbox lists populates from external source (sql query). The comma separated list of tables is split into rows, and the join criteria (from "Select classified type(s)") are appended to each table row. Lastly, it's all summed up by concatenating all rows into a single field:
My question(s) is: Can I somehow use the resulting query string in a new workflow where I replace a placeholder string with the query string?
Or even better: Can I achieve the same results in a pure In-DB setup, using the output from one app in the next app?
I am not sure how I can use this with an in-db stream? I can't find a way to stream the generated query into an in db flow and then replace the query string. A standard workflow seems pretty straight forward, but I need to avoid streaming data locally.