Alteryx Connect BYOM – “Bring your own metadata” with an SDK

[DavidM]

We are all very excited about Alteryx Connect, our Social Data Cataloging and Data Discovery piece of the Alteryx Analytics platform.

Last week we released Alteryx Connect v2.0 and one of the great new features as part of this release is the Loader SDK (software development kit) which will allow everyone, our partners and customers alike, to extend the existing functionality of the product.

This blog post introduces the SDK functionality to provide everyone with clear, big-picture understanding of how the Connect loaders, SDK and extensibility works.

Why is this such a big game changer? With Alteryx Connect, we are agnostic to the type of metadata we are collecting and with the SDK, we now enable you to easily push the metadata from any third-party software of your choice to Connect and therefore extend the Connect functionality.

The best part about this - you can also use the Alteryx Designer, another key piece of the Alteryx Analytics platform, to develop your own loaders for Connect, nice and simple.

How Alteryx Connect works recap & How we use loaders to populate metadata?

 

Alteryx Connect Refresher

To get started with Alteryx Connect, the first thing you want to do with it is to populate it with metadata from the source systems you use in your data landscape. For instance - database platforms, file systems, data lakes, applications, reporting platforms, and more. You also want to be able to do this automatically, by running what we call Metadata Loaders.

How do the Metadata Loaders work?

The Metadata Loaders can connect to your Source Systems; and extract, load, and index the metadata from these systems in Alteryx Connect. Once this takes place, you are on the way to capturing the technical metadata, which is a crucial starting point for creating a catalog. This will allow you to understand what you have inside of all of the in-house systems and be able to access this information easily in a centralized fashion.

All the out-of-the-box loaders we ship with the product were developed using the standard toolset available in Alteryx Designer, with the addition of two new plugins (I will talk about these later).

We use the Alteryx Designer to get to the metadata fast and easy, transforming it into the dataset we need and publishing it to Connect. Again, nice and simple.

Let me note that we populate Connect with more information than just what you use or access from the Alteryx portfolio. We let you paint the full picture of your company's data landscape - from the entire data warehouse, all reports from visualization and reporting platforms, and everything you store in your data lake and more. 

Adding annotations & business detail

So, once the metadata is loaded in Connect, we represent them in pre­configured, out-of-the-box, asset types on the frontend. An asset type is a type of object we store in Connect (analytic apps, reports, databases, files, applications, glossary, people - are all examples of default asset types in Connect).

Typically the loaded metadata is raw and missing that much- needed business detail. So, with Connect, we let you annotate it, add business details and descriptions, as well as tag, certify, rate your data and assign owners.  This way, you capture all the tribal knowledge that resides in “people's brains" or is scattered over emails, tickets, wikis, blogs, and all other sorts of communication channels.

Ultimately, we help you create a centralized catalog, agnostic of the source technology, enabling search, collaboration, and cooperation of your teams. This allows users to easily find the data assets when they need them, using extremely simple, yet powerful UI.

Creating the perfect catalog

To sum up - we want Alteryx Connect to be the one central place, where you can find all your information assets - whether it is reports, data sources, workflows, business concepts, people, you name it. And to speed up this process,  we let you collect all the metadata from your existing systems and push it to Connect automatically. There is no need for long implementation projects because with Connect you can show viable results to anyone you need, in just a few days.

How we let anyone extend Connect functionality - “Bring your own metadata”

Intro to Connect SDK

Now, that we’ve done a very fast recap of how Alteryx Connect works, let me talk about the actual SDK functionality and what it allows you to do.

As much as we would like to have an out-of-the-box loader ready for every possible source system that exists, there are simply too many data source options available these days, and different organizations have different data needs. For this reason, we have decided to give Connect a proper boost and let anyone create their own loaders and extend the product with the SDK. 

What can you actually do with the SDK?

The SDK helps organizations take advantage of existing systems, sources and infrastructures that Alteryx Connect does not support right out of the box.  For instance, if Connect does not fully support an RDBMS, reporting platform, data lake, or analytics tool of your choice, you can easily develop and create the extensibility yourself. This way, you can publish any type of metadata information from your source system to Connect and start capturing that much-needed business-oriented detail and tribal knowledge we talked about earlier.

Just to give you a real-life example - I have been beta testing the SDK with a customer of Connect and they wanted to capture information from SAP Hana (a source we are currently missing in Connect). With a 1-hour onboarding session to the SDK, the customer was able to write their own proof of concept (PoC) of an SAP Hana DB loader in less than 1 man-day, which is an incredible time to value. 

SDK Components 

First and foremost, we have developed a new set of REST API endpoints that allow you to easily push and synchronize the metadata from your source systems to Alteryx Connect. We leave it fully up to you, whether you want to design your custom loaders using Alteryx Designer - just like we do for all out-of-the-box loaders; or if you want to use a programming language of your choice (Java, Python, R, C#, ...).  We offer both a code-friendly and code-free approach.

Secondly, we have developed a pair of new plugin tools for Alteryx Designer to support the ability to push metadata to Alteryx Connect. So, if you decide to design the loaders in Alteryx Designer, you just connect to the source technology; transform the metadata the way you need it to be, and then using our plugins simply publish the metadata to Connect.  What is great about the new Connect plugin tools, is the fact that we take care of refreshing the data in Connect, create any new metadata tables for you dynamically and run the sync jobs in Connect for you. In short, everything is taken care of for you.

Lastly, once the metadata is in Connect, you can specify how the front-end representation of the metadata will look. For this, you can either reuse the existing asset types (analytic apps, reports, databases, files, applications, glossary, people, ...) or create your own asset types. All of this with a simple configuration of the product and without the need for custom development.

SDK Technical documentation and questions

 For more details on the Alteryx Connect SDK, you can read the full technical documentation. You can also download the existing loaders to learn how the Alteryx team develops the out-of-the-box loaders and uses the new plugin tools.

If you have questions, post them on our new Developer Community and use the "Connect SDK" tag so others can find the questions as well!

Short Disclaimer: Upgrades to future major versions and releases of Alteryx Connect may require you to do some rework on the loaders and customizations you developed.

While Alteryx makes every effort to ensure version compatibility, new objects created with the SDK may not migrate as-is to future major releases of Alteryx Connect.