Azure Data Lake Output with Dynamic Name

[Salvador_Laborde]

Hello everyone,

I am currently doing an extract from an SQL database, this database is really big so the process takes quite some time to complete, this data is then saved as a CSV in a BLOB storage for later use.

What I thought would work out to reduce the time of processing is splitting the dataset into 3 parts

1) Historical data (from beginning to say OCT 2024) - only runs once, does not affect the question

2) Last month data

3) This month data

The idea is to be able to save the files with a variable name and upload this to BLOB, that way I will be uploading

202411.csv + 202412.csv during Dec 2024

202412.csv + 202501.csv next month

and so on. with the idea of keeping the historical data but making sure any change in the last month is passed onto the historical files.

With other output tools there is the simple option to use a column for the name, however for the ADL Output tool I've been unable to find a way to do this variable name.

Has anyone encountered this issue and a way to solve it?

[apathetichell · Accepted solution]

Throw everything in a batch macro -> update the ADL Output tool in the batch macro via a control parameter -> action tool. update the query via control parameter -> action tool. your datastream shoudl be two columns (query/outputfile) ---> and should feed in to the macro.

having said that I am very proud of knowing zero about Azure --- but if this was AWS or GCP and I wanted to take a DB or query from BQ/Postgres/Red**bleep**/etc --> and shoot it to GCS/S3 ---> I would do that via Command line because it's like I dunnoo 1000X faster?

[Salvador_Laborde]

As a new user I had a tough time finding out how to do that, but with a single batch and an input I managed to make it all work, however the idea guided me pretty well, thanks!