Is that possible to use a var into my recipe step so every time that recipe runs, it will generate the profile into my storage with the dataset name. Example attached does not matter which file is being wrangled, but the profile will always be generated as Standard Clean, but I wish to have each time it run, using the dataset name as var. Is it possible?
Solved! Go to Solution.
2 distinctions i would like to highlight here.
Sorry not sure I understood step 1 you mentioned. Let me try to explain further, for the example where i call there Transient file its parameterized to take any CSV file from a specific cloud storage bucket as image below, once any CSV is detected it will trigger the recipe (I created a cloud function to trigger the recipe thru API) so the what I call Standard Clean is some standard treatment that applies to any csv file. But I want instead Standard Clean it change every run to generate the profile using FileName, not Standard Clean.
I think i am a bit confused. Here's what i understood.
So FileName is a raw dataset that are parameterised as input dataset(Transient File) to Flow DDC Standard Transient to Staging.
Now if you run an output on recipe Standard Clean, it will present a profile based on transformation steps applied on Transient File.
But instead, you want a profile of parameterised raw dataset FileName(i.e Transient File) only. If that's the case, then you can add a new recipe to the dataset Transient File and add no steps, perhaps call it Profile Recipe. Next create an output on Profile Recipe and Run job(make sure profile results are checked). This should theoretically not perform any transformation as there are no recipe steps.
Once the job run completes successfully you can retrieve the profile as JSON and further unnest it in dataprep.
On Step 1, if you write $filename under a transform like New Formula, it gives you the full path of the dataset(in this case parameterised files) as a new column. Then one can think of extracting only the name of the file. I thought that was one of the asks.
No problem buddy but I will try to get back and explain entire flow.
Better now?
Aah it seems like you would like to change the recipe name from Flow view itself. Unfortunately, I don't think that's possible.