Is that possible to interact with Dataprep through API or Python so I can execute transformations through my programs, instead need to login to web portal and do all the wrangling in web interface ?
Solved! Go to Solution.
Hi Gustavo,
Dataprep have REST API you can use to start Dataprep jobs from an external script or tool.
API doc https://cloud.google.com/dataprep/docs/html/API-Reference_145281441
Examples with Cloud Functions https://github.com/victorcouste/google-cloudfunctions-dataprep
Example of data pipeline using Cloud Function and Dataprep API https://medium.com/google-cloud/how-to-automate-a-cloud-dataprep-pipeline-when-a-file-arrives-9b85f2...
Victor
thanks a lot @Victor Coustenoble? really appreciated
Hello @Victor Coustenoble? your tutorial at medium about a pipeline is great, im trying to create my pipeline and when im trying to parameterize my FileName variable to take all *.csv files in a storage it is not allowing me to update the parameter.
See the picture. Am I doing something wrong?
@Victor Coustenoble? another issue is regarding the code, i did the necessary adjustments (token, landingzone, etc) but is getting this error:
"Result : {'code': 16, 'message': 'JWT validation failed: Bad JWT format: should have 2 dots', 'details': [{'@type': 'type.googleapis.com/google.rpc.DebugInfo', 'stackEntries': [], 'detail': 'auth'}]}"
Hi Gustavo,
For your first question, you need to use Pattern parameter (with wildcard for example for your need). Note that you can mix date, pattern or variable parameter type :
For your second question, difficult to say, there must be an error in your source code.
Happy to have a look if you send it to me at vcoustenoble@trifacta.com
Victor
Ok I got, but if I will use wildcard is it going to impact my code since it will not have FileName variable anymore, right?
Hum, yes in this case you don't need anymore a variable, and I think your code won't work as you will try to overwrite a variable that won't exist anymore.
Don't know how you could easily switch between a Pattern parameter and a Variable parameter, as it's not manage in the same way.