topic Do Parameterized Datasets automatically Union the inputs where the parameters match multiple files? in Alteryx Designer Cloud Discussions

Do Parameterized Datasets automatically Union the inputs where the parameters match multiple files?

4a0098b7f9b243add27a — Thu, 15 Nov 2018 09:00:29 GMT

It is not clear in the GCDp documentation how multiple matched files are handled other than to say they should be of the same type.

Is this a Union operation?

Re: Do Parameterized Datasets automatically Union the inputs where the parameters match multiple files?

Trifacta_Alumni — Thu, 15 Nov 2018 16:18:56 GMT

Hey Jonathon,

Yes, in essence this is a Union operation (though happens outside of the Wrangle language so differs in practice from the Union transform). Adding an entire directory will union all files within into a single dataset, and using parameters is a way of specifying a subset of a directory to union into a single dataset.

We will look into modifications to the Trifacta docs to add some clarity on this.

thank you!

David

Re: Do Parameterized Datasets automatically Union the inputs where the parameters match multiple files?

4a0098b7f9b243add27a — Fri, 16 Nov 2018 10:11:44 GMT

Thanks for the clarification.

Having experimented a little it is clear that it is a simple file concatenation rather than a data union in the spirit of the Union tool within the wrangler. To whit, the header rows of the constituent files are included as data rows.

I shall explore what process I can use to 'normalise' the columns between csv's i was trying to gather in this way. Working within a flow with defined inputs is no problem and Dataprep unions these files no problem. But in a scenario where the number of inputs is variable, this is more complicated.

Re: Do Parameterized Datasets automatically Union the inputs where the parameters match multiple files?

4a089ce890eccf4a5c26 — Tue, 04 Dec 2018 17:06:41 GMT

My solution to this so far is to bring the dataset in unstructured and then to filter rows that match my header.

Honestly, I have found that just explicitly setting the structure in unbounded data sets works far better than having Dataprep infer the structure.