Don't forget to register for our panel discussion with Dean Stoecker, Amy Holland, and Mark Frisch occurring next Wednesday, June 1!

2022-05-26 Updates: Email: If you're not seeing emails be delivered from the Community, please check your spam and mark the Community emails as not junk. Thank you for your patience.

Alteryx Connect Discussions

Find answers, ask questions, and share expertise about Alteryx Connect.

Alteryx loader - Relationship to created file

HarryM123
6 - Meteoroid

Hi there,

 

I am using the alteryx loader as a base for a custom loader.

 

I am trying to make a relationship to a custom file i created on connect but i am not able to achieve this, see process below:

 

1. Created file named 'Kafka_Topics', connect folder path is Data Sources/files/kafka/ , formatted as a file as is parent folder.

 

2. Using the AYX_WF_FS staging table in my workflow i am inputting the below:

 

USAGE_TYPESYSTEM_NAMESYSTEM_ENVIRONMENTNAMEPATHPARENTPATH_HASHTABLE_NAMETABLE_PATH_HASHTYPEWORKFLOW_IDLOAD_CODE
2FilesKafkaKafka_Topicsfiles/kafka/kafka_topicskafka   FDATABRICKS_2598393148941662DATABRICKS CUSTOM LOADER

 

i have also tried a few different permutations but not really sure as to which should be working.

 

3. I check the workflow that should be outputting to this file, and whilst the details in the table above are shown there is no relationship made.

 

My queries are:

 

1. Is this approach feasible ? or is there a better way?

 

2. If feasible what variables should i be inputting into the AYX_WF_FS staging table to get the connection made?

 

Any ideas or suggestions would be appreciated,

 

Cheers,

 

Harry

4 REPLIES 4
VojtechT
Alteryx
Alteryx

Hi @HarryM123 , out of curiosity - is it possible that you are trying to load metadata from Databricks? If not, could you share what data source are you building the custom loader for? Maybe we have a better option to start with than ayx loader. 

Thx

Vojta T., PM for Data Connectors & Connect
Try our Beta data connectors at https://bit.ly/3Ae8HgY
HarryM123
6 - Meteoroid

Hey @VojtechT , i am using the databricks rest API as an input. All notebooks are being downloaded and parsed to then isolate the read/write commands. I then join these commands back to other notebooks to find the paths and names of the objects the notebooks reads/writes to.

 

The rest API is first queried for directory paths using an iterative macro, these directories are then fed into another API call which downloads the notebooks text is a subsequent iterative macro. The result is the complete download of all notebooks in the specified directory. 

 

This seems to work well for the time being, I have successfully created relationships from these to other data sources in the database/database servers locations its just when trying to create relationships to objects stored in the files location i am running into the above issue.

 

I thought that the Alteryx loader would provide the best functionality for this use case as i am able to create different subsections for each category of the notebooks. If there is a more suitable loader/method i would be keen to hear your thoughts on it !

 

Cheers,

Harry

VojtechT
Alteryx
Alteryx

@HarryM123 , we actually have a Beta version of Databricks metadata loader already, just looking for someone with a real instance who could verify it works also in "real environment", not just our artificial one. And also to confirm all the main use cases are covered. However, the loader requires new Connect as well. Would you be willing to give it a shot and install a Beta Connect and run the Beta Databricks loader?

Vojta T., PM for Data Connectors & Connect
Try our Beta data connectors at https://bit.ly/3Ae8HgY
HarryM123
6 - Meteoroid

@VojtechT i may be able to help with this, as my current databricks instance is on a clients real environment i would not be able to do this myself. I can send to another colleague who could test on their server instance. It would be interesting to see the approach taken and how it pulls the data through to connect.

 

As i would not be able to deploy it on my clients work, do you think the above approach i have taken would give me the desired results (linking of my databricks workflow to a file source) ?

 

Is there something i am doing wrong in the configuration that is not making the relationship? Do you know what i should be inputting into the AYX_WF_FS staging table to get the connection made?