Community Spring Cleaning week is here! Join your fellow Maveryx in digging through your old posts and marking comments on them as solved. Learn more here!

Engine Works

Under the hood of Alteryx: tips, tricks and how-tos.
DavidM
Alteryx
Alteryx

We are all very excited about Alteryx Connect, our Social Data Cataloging and Data Discovery piece of the Alteryx Analytics platform.

 

Last week we released Alteryx Connect v2.0 and one of the great new features as part of this release is the Loader SDK (software development kit) which will allow everyone, our partners and customers alike, to extend the existing functionality of the product.

 

This blog post introduces the SDK functionality to provide everyone with clear, big-picture understanding of how the Connect loaders, SDK and extensibility works.

 

Why is this such a big game changer? With Alteryx Connect, we are agnostic to the type of metadata we are collecting and with the SDK, we now enable you to easily push the metadata from any third-party software of your choice to Connect and therefore extend the Connect functionality.

 

The best part about this - you can also use the Alteryx Designer, another key piece of the Alteryx Analytics platform, to develop your own loaders for Connect, nice and simple.

 

How Alteryx Connect works recap & How we use loaders to populate metadata?

 Picture1.png

Alteryx Connect Refresher

To get started with Alteryx Connect, the first thing you want to do with it is to populate it with metadata from the source systems you use in your data landscape. For instance - database platforms, file systems, data lakes, applications, reporting platforms, and more. You also want to be able to do this automatically, by running what we call Metadata Loaders.

 

How do the Metadata Loaders work?

The Metadata Loaders can connect to your Source Systems; and extract, load, and index the metadata from these systems in Alteryx Connect. Once this takes place, you are on the way to capturing the technical metadata, which is a crucial starting point for creating a catalog. This will allow you to understand what you have inside of all of the in-house systems and be able to access this information easily in a centralized fashion.

 

All the out-of-the-box loaders we ship with the product were developed using the standard toolset available in Alteryx Designer, with the addition of two new plugins (I will talk about these later).

 

We use the Alteryx Designer to get to the metadata fast and easy, transforming it into the dataset we need and publishing it to Connect. Again, nice and simple.

Let me note that we populate Connect with more information than just what you use or access from the Alteryx portfolio. We let you paint the full picture of your company's data landscape - from the entire data warehouse, all reports from visualization and reporting platforms, and everything you store in your data lake and more. 

 

Adding annotations & business detail

So, once the metadata is loaded in Connect, we represent them in pre­configured, out-of-the-box, asset types on the frontend. An asset type is a type of object we store in Connect (analytic apps, reports, databases, files, applications, glossary, people - are all examples of default asset types in Connect).

 

Typically the loaded metadata is raw and missing that much- needed business detail. So, with Connect, we let you annotate it, add business details and descriptions, as well as tag, certify, rate your data and assign owners.  This way, you capture all the tribal knowledge that resides in “people's brains" or is scattered over emails, tickets, wikis, blogs, and all other sorts of communication channels.

 

Ultimately, we help you create a centralized catalog, agnostic of the source technology, enabling search, collaboration, and cooperation of your teams. This allows users to easily find the data assets when they need them, using extremely simple, yet powerful UI.

 

Creating the perfect catalog

To sum up - we want Alteryx Connect to be the one central place, where you can find all your information assets - whether it is reports, data sources, workflows, business concepts, people, you name it. And to speed up this process,  we let you collect all the metadata from your existing systems and push it to Connect automatically. There is no need for long implementation projects because with Connect you can show viable results to anyone you need, in just a few days.

 

How we let anyone extend Connect functionality - “Bring your own metadata”


Picture2.png

Intro to Connect SDK

Now, that we’ve done a very fast recap of how Alteryx Connect works, let me talk about the actual SDK functionality and what it allows you to do.

As much as we would like to have an out-of-the-box loader ready for every possible source system that exists, there are simply too many data source options available these days, and different organizations have different data needsFor this reason, we have decided to give Connect a proper boost and let anyone create their own loaders and extend the product with the SDK. 

 

What can you actually do with the SDK?

The SDK helps organizations take advantage of existing systems, sources and infrastructures that Alteryx Connect does not support right out of the box.  For instance, if Connect does not fully support an RDBMS, reporting platform, data lake, or analytics tool of your choice, you can easily develop and create the extensibility yourself. This way, you can publish any type of metadata information from your source system to Connect and start capturing that much-needed business-oriented detail and tribal knowledge we talked about earlier.

 

Just to give you a real-life example - I have been beta testing the SDK with a customer of Connect and they wanted to capture information from SAP Hana (a source we are currently missing in Connect). With a 1-hour onboarding session to the SDK, the customer was able to write their own proof of concept (PoC) of an SAP Hana DB loader in less than 1 man-day, which is an incredible time to value. 

 

SDK Components 

First and foremost, we have developed a new set of REST API endpoints that allow you to easily push and synchronize the metadata from your source systems to Alteryx Connect. We leave it fully up to you, whether you want to design your custom loaders using Alteryx Designer - just like we do for all out-of-the-box loaders; or if you want to use a programming language of your choice (Java, Python, R, C#, ...).  We offer both a code-friendly and code-free approach.

 

Secondly, we have developed a pair of new plugin tools for Alteryx Designer to support the ability to push metadata to Alteryx Connect. So, if you decide to design the loaders in Alteryx Designer, you just connect to the source technology; transform the metadata the way you need it to be, and then using our plugins simply publish the metadata to Connect.  What is great about the new Connect plugin tools, is the fact that we take care of refreshing the data in Connect, create any new metadata tables for you dynamically and run the sync jobs in Connect for you. In short, everything is taken care of for you.

 

Lastly, once the metadata is in Connect, you can specify how the front-end representation of the metadata will look. For this, you can either reuse the existing asset types (analytic apps, reports, databases, files, applications, glossary, people, ...) or create your own asset types. All of this with a simple configuration of the product and without the need for custom development.

 

SDK Technical documentation and questions

 For more details on the Alteryx Connect SDK, you can read the full technical documentationYou can also download the existing loaders to learn how the Alteryx team develops the out-of-the-box loaders and uses the new plugin tools.

 

If you have questions, post them on our new Developer Community and use the "Connect SDK" tag so others can find the questions as well!

 

Short Disclaimer: Upgrades to future major versions and releases of Alteryx Connect may require you to do some rework on the loaders and customizations you developed.

While Alteryx makes every effort to ensure version compatibility, new objects created with the SDK may not migrate as-is to future major releases of Alteryx Connect.

 

 

 

David Matyas
Sales Engineer

David is a Sales Engineer in Alteryx, based in a beautiful regional office in the city of Prague, Czech Republic. He is a highly universal analytics pro, having 10 years worth of experience with Consulting, Sales Engineering, Development, and Product Management. David is now leading technical success across strategic accounts in Northern Europe, helping customers to get the most out of Alteryx Analytics Platform && beyond.

David is a Sales Engineer in Alteryx, based in a beautiful regional office in the city of Prague, Czech Republic. He is a highly universal analytics pro, having 10 years worth of experience with Consulting, Sales Engineering, Development, and Product Management. David is now leading technical success across strategic accounts in Northern Europe, helping customers to get the most out of Alteryx Analytics Platform && beyond.

Comments
npandya
7 - Meteor

David,

 

Great article. I am pretty green to connect and I'm want to build a custom loader that can load the metadata from a client site to connect. I have a workflow which accesses the site data in JSON format using api url, key and download tool. I am trying to output that in connect using connect JSON builder and connect output, but its not pushing data in connect. There should be steps in between, but I'm not able to figure about which ones from the existing loaders.

DavidM
Alteryx
Alteryx

Hi @npandya,

 

what loader are you trying to build? are you utilizing some of the existing tables to push the metadata to with the Puiblish to Connect tools?

 

it may be worth to check out the links of the article above, especially

https://help.alteryx.com/developer/current/Connect/Overview.htm

 

cheers,

d

npandya
7 - Meteor

@DavidM 

 

Yes, I am trying to use the existing tables resides on a web application to push meta data with publish to connect tools. Now I'm no sure which connect job I need to execute and what should be the load code.

 

Below is the format of JSON I'm getting using API where medical is the schema name, treatments is the object or table name and attribute is column in the table treatment. Now, I am not sure how to load in on connect!

BigID data.png   

 

Looking forward to get some insights.

 

Thank you,

Nisarg

DavidM
Alteryx
Alteryx

@npandya yeah, the load jobs are missing more robust documentation. my approach would be to check out existing Connect loaders (you can install them from licenses.alteryx.com) into your Designer. And check out, for instance, a SQL server or FILES loader to utilize the metadata tables they use. Also, these loaders will have the LOAD JOBS already present. 

The LOAD JOBS are actually defined in Alteryx Connect and it would probably be extremely hard to write your own custom one at this point without detailed knowledge of the Connect's backend.

So bottom line - i would try to check an existing loader and reuse the TABLES and LOAD JOBS from those.

npandya
7 - Meteor

@DavidM 

 

Thank you! I am almost there. I do have one question. Can I create a new job in alteryx connect or I can only use the existing one? When I'm trying to enter a different job name that does not exist, my workflow gives me an error. 

DavidM
Alteryx
Alteryx

@npandya i would strongly advice to utilize the load job of existing loader. I.e. - check out the existing files/ db loaders of Connect and in the tool all the way on the right in those loaders, you have the PublishToConnect tool which will tell you a load job name.

Writing a custom load job would require scripting in velocity, javascript, html, sql. It's probably not the easies thing to start with, especially without robust SDK documentation. This is hopefully coming but not in place as of yet.

npandya
7 - Meteor

@DavidM 

 

Thanks David. I was able to publish it using load-alteryx-db.

 

I have one more question.

 

I have tags associated with columns in my source data. I see a tag field in the connect as well, but I know I can't attach a column in DB_TABLES/DB_COLUMNS for tags and upload that.

 

How can I upload the tag information on connect so that it will show up in the assets?

 

-Nisarg

DavidM
Alteryx
Alteryx

Hi @npandya

 

I would suggest you either open up the Tableau or Alteryx Workflows loader as both utilize loading TAGs, and just reuse that logic.

 

David

N_FirstMileAnalytics
6 - Meteoroid

@DavidM ,

 

How to ingest reports metadata into Alteryx connect's report section rather than Data sources section?

DavidM
Alteryx
Alteryx

Hi @N_FirstMileAnalytics ,

 

It all depends into which metadata tables the Connect tools make the push via the Alteryx worklfow.

 

I would probably suggest investigating existing loaders from Alteryx - Tableau, PowerBI for instance.


And utilizing the metadata writes from there, rather than from DB loaders.

 

david