Is that possible to user FileName var into recipe?

Is that possible to use a var into my recipe step so every time that recipe runs, it will generate the profile into my storage with the dataset name. Example attached does not matter which file is being wrangled, but the profile will always be generated as Standard Clean, but I wish to have each time it run, using the dataset name as var. Is it possible?

Trifacta Classic

Accepted answers

Trifacta_Alumni

Aah it seems like you would like to change the recipe name from Flow view itself. Unfortunately, I don't think that's possible.

All comments

Trifacta_Alumni

2 distinctions i would like to highlight here.

Dataset name in normal sense as per your screenshot is Transient File and NOT Standard Clean. To extract file name as a part of recipe, metadata transform $filepath can be used and original dataset name can be extacted.
Second distinction, to pass a variable value as a part of profile, it needs to be passed via recipe. From flow view one can create a flow parameter and reuse it along the current and downstream recipes. This is easily achievable. In recipe editor, New Formula one can reference flow parameters using a $ symbol and call the parameter. This documentation will be helpful https://cloud.google.com/dataprep/docs/html/Create-Flow-Parameter_156863207

unknown

Sorry not sure I understood step 1 you mentioned. Let me try to explain further, for the example where i call there Transient file its parameterized to take any CSV file from a specific cloud storage bucket as image below, once any CSV is detected it will trigger the recipe (I created a cloud function to trigger the recipe thru API) so the what I call Standard Clean is some standard treatment that applies to any csv file. But I want instead Standard Clean it change every run to generate the profile using FileName, not Standard Clean.

Trifacta_Alumni

I think i am a bit confused. Here's what i understood.

So FileName is a raw dataset that are parameterised as input dataset(Transient File) to Flow DDC Standard Transient to Staging.

Now if you run an output on recipe Standard Clean, it will present a profile based on transformation steps applied on Transient File.

But instead, you want a profile of parameterised raw dataset FileName(i.e Transient File) only. If that's the case, then you can add a new recipe to the dataset Transient File and add no steps, perhaps call it Profile Recipe. Next create an output on Profile Recipe and Run job(make sure profile results are checked). This should theoretically not perform any transformation as there are no recipe steps.

Once the job run completes successfully you can retrieve the profile as JSON and further unnest it in dataprep.

On Step 1, if you write $filename under a transform like New Formula, it gives you the full path of the dataset(in this case parameterised files) as a new column. Then one can think of extracting only the name of the file. I thought that was one of the asks.

unknown

No problem buddy but I will try to get back and explain entire flow.

The flow will detect any CSV file in a bucket
A Cloud function will detect this csv file (any name) and trigger the recipe with some standard treatments
The job recipe will run performing the treatments and generating the profile
Where I mentioned in the picture Standard Clean is the recipe NAME, and I would like this name to be changed dynamically every run, so instead name is Standard Clean, it changes to filename

Better now?

Trifacta_Alumni

Aah it seems like you would like to change the recipe name from Flow view itself. Unfortunately, I don't think that's possible.

Quick Links

This months top contributors

atcodedog05 19598

Qiu 15879

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703