Alteryx Connect Ideas

Hello Alteryx Team,

Would you consider creating a metadata loader for SharePoint? A lot of companies that we are working with have a lot of data stored in SharePoint and they would like to catalog their metadata.

I am aware that it is possible to use the Files Loader when the SharePoint drive is mapped as a mapped drive in Windows Explorer, but the mapping and subsequent scheduling is very problematic, also because of security policies in large enterprises.

Having a SharePoint dedicated loader would be a great benefit.

Thank you very much for considering this idea.

Regards,

Jan Laznicka

Hello Alteryx Team,

Would you consider creating a metadata loader for Databricks? I can imagine this being a part of the Analytical workflows section and be catalogued in a similar way like Alteryx workflows. On a number of clients, we are seeing Databricks jobs being used as a complement to Alteryx workflows and a possibility to have them both catalogued in Alteryx Connect would be of great benefit.

Thank you very much for considering this idea.

Regards,

Jan Laznicka

Hello Alteryx,

The Alteryx loader does an excellent job capturing the Data Connection information from workflows and visualizing the data in Connect. It would be an additional benefit if the Data Connection information could get more granular and include the columns utilized from each table in the workflow.

An example where this will provide valuable benefits:

If you have a table with 100 columns and it's utilized extensively in more than 50 workflows, how do you analyze the impact when you're notified that column ABCXYZ is going to be altered? Connect can tell me which workflows are using the table, but it can't tell me which workflow is specifically using column ABCXYZ.

This would greatly aid users who may be searching workflows for specific column usage.

Thank you,

Jason

Hello Alteryx Team,

We would like to be scheduling the loaders from the administration console, Connections tab. However, when scheduling from there, the loaders are being sent to the Alteryx Gallery under low priority and therefore can easily end up in a queue. As we are scheduling a number of loaders that run for a significant amount of time, it is essential to have the schedules set on precise times and have a certainty that they will not end up in a queue. However, the low priority in the Gallery prevents us from doing that.

It would be great if there was a priority switch directly in the Connect's Connections section, where it would be possible to select how prioritized should the loaders be in the Alteryx Gallery. Also a possibility to select a specific worker would be greatly appreciated.

Thank you very much for considering this idea.

Regards,

Jan Laznicka

Hi Alteryx team,

would it be possible to include a project filter in the Tableau loader? Currently, the loader brings in all the projects from the selected sites. However, we have come across a situation when each department in a company has a couple of projects that they do not want to load.

The idea is that it would be possible to list the names of the projects that won't be loaded. It could be done in a way that all the projects containing the listed words won't be loaded. This would give us the possibility to, for example, not load the projects containing 'Ad hoc' in their names.

We are aware that we might keep these projects 'invisible' using permissions, however, for operational purposes we believe that the project filtering capability would be much better.

Thank you very much.

Regards,

Jan Laznicka

Hi Alteryx team,

It would be great if it was possible to load SAP HANA systems using ODBC DSN instead of the normal host name.

From our experience, there is often more than 1 node present for SAP HANA systems and when entering the master node into the loader interface, we can't easily switch to a failover node when an unexpected event happens. Sometimes, it is not only the matter of nodes but for example a migration of an instance to cloud might result in a different host name. As a consequence, the loadcode is different and it is not possible to migrate the previous enrichment to the asset pages created under the new loadcode.

A solution for this would be the possibility of loading SAP HANA systems using the ODBC DSNs (this is already possible for example for Hive and it would be great if it was available for all applicable systems).

Thank you very much for considering this idea.

Regards,

Jan Laznicka

Hi Alteryx team,

Would it be possible to implement custom URL in the HDFS loader when using Knox Gateway? This could be implemented in the same way as in the normal Input Data tool.

When connecting to HDFS using the Input Data tool, the URL is automatically generated, however, it is possible to change it. Sometimes this is necessary as the 'sandbox' part of the URL is for some companies not applicable, they have for example their company names instead of it:

However, in the Alteryx Connect HDFS loader, this is not possible as there is no such field in the interface:

The 'sandbox' part of the URL is then automatically brought through 2 formula tools:

This makes it impossible to load the HDFS metadata into Connect using the out-of-the-box Alteryx loader. Implementing a field into the interface like in the standard input data tool would solve this problem.

Thank you very much for considering this idea.

Regards,

Jan Laznicka

Hi Alteryx team,

Would it be possible to implement an overview table with a complete history of metadata loads? I can imagine there could be a table in the administration console (in Jobs) where you could see the history and number of runs for each source system so that it can be referred to at any time when you need to know the frequency and past runs?

Thank you

Michal

Establish lineage/relationship between SAP HANA and Tableau (reports)

Issue

By default in Connect, we are unable to see data lineage between SAP HANA and Tableau. The standard Alteryx loaders (i.e. Tableau & SAP HANA) in version 2019.2 are unable to detect the HANA object link due to a misaligned naming convention (“::” vs “/”). This specifically occurs when Tableau dashboards are pointed to HANA synonyms (“::”) as opposed to the views (“/”).  

Background

In SAP Hana, there are two types of objects – views (which are present in relevant schemas, e.g. “_SYS_BIC”) and synonyms. The synonyms are created for each view by default and always appear to be in the schema “PUBLIC”, however, this is not actually an explicit schema. According to SAP, public synonyms are the official access interface for the system views.

When loading the objects from SAP HANA, the object names contain the package name as well as the object name (view etc.). These two are differentiated with a “/” when loaded from the “_SYS_BIC” schema (e.g. “GSTP_VALIDATION/CA_01_C_PDA_GRA_001”), however synonyms from the “PUBLIC” schema are loaded with a differentiator of “::” (e.g. “GSTP_VALIDATION::CA_01_C_PDA_GRA_001”). An example below, where first row is the view and the second row provides the related synonym:

Thus, the lineage is not established when the SAP HANA objects from schemas are loaded (containing the “/”, and at the same time, the Tableau dashboards linked to the synonyms (“::”). This is because the object names are quite clearly different, and the loader doesn’t currently resolve the synonym to view translation (as per standard SAP HANA SQL parsing behaviour).

Loading problem analysis (using standard Alteryx loaders)

On SAP HANA side, when running the loader, user can (optionally) define schema name(s) they wish to load, however:

1. When schema is set as “_SYS_BIC”, the loader only loads the views from this particular schema, with no synonyms.
2. When schema is left as blank (i.e. no value entered), the loader loads everything, however in the “_SYS_BIC” schema, only synonyms are loaded and not the views.
3. When schema is set as “PUBLIC”, the loader does not return anything - fails as there is no corresponding schema in SAP HANA (it is only virtual space, not an explicit schema)

The SAP HANA instance in question is very, very large and due to volume limitations, as well as the potential risk of performance degradation, it is not feasible to load everything.

When we’ve examined the H2 staging tables, we find that Connect has loaded (therefore is technically aware of) the synonyms from the “PUBLIC” schema but they are not ever loaded beyond the staging tables. This is obviously required if we wish to represent these relationships within Connect.

Idea/Fix:

• SAP HANA loader – it seems there is a problem of loading synonyms from the “PUBLIC” schema and the loader should be adjusted in order to allow this.
• Tableau loader – needs to be able to refer to synonyms defined in a SAP HANA data source, meaning whenever there is a “::” in object name and the schema name is blank (i.e. empty string representing the “PUBLIC” schema), then it should be defaulted to “PUBLIC”. This logic would ensure that a schema name is always present, even in cases where the schema name is retrieved as blank (which then would become “PUBLIC”).
• In case the schema name is left blank for any reason, it should be populated as “PUBLIC”.
• @DanH suggested to customize the Tableau loader as a temporary fix to replace the “::” with “/”, which is certainly possible but comes with a risk of incompatibility with future releases. Also the customers strong preference is to deliver a solution which only utilises standard loaders.

We're working through an implementation of Connect - and it appears that every time Connect scans the Alteryx or Tableau environment, it does a full reload of all canvasses / workbooks.

However - we have several thousand tableau dashboards & alteryx canvasses - so this creates a significant delay on running the loaders, and we'd want to run these loaders every few hours so that Connect has up-to-date information (or at most 24 hours out of date).    Running a full export and scan is causing this load to take a very long time to run.

Can we change the default behaviour for all the loaders to use a delta-load rather than doing a full scan - i.e. only pull out assets that have changed since the last load?

 cc: @nganesha @Kosi

Our Data Catalogue in Connect has about 2 millions items (tables, views, columns).

I see next issues:

1. We collect metadata from about 10+ DBMS. So after each Metadata loader run, Alteryx Connect will start load_alteryx_db script and process whole staging area (DB_*) tables, not only current extracted metadata set from single DBMS. It will lead huge redundancy.
2. Follows from first issue: One-by-one comparison of loaded metadata will take a lot of time in real environment with 1-2 millions items (ordinary situation in large Bank). And this comparison will be executed several times. It will increase the redundancy in the number of DBMS servers.

3. All queries in this script containing column or table name as a parameter (e.g. src.TABLE_NAME='${query_table_name}' AND                                 src.COLUMN_NAME='${query_column_name}') will be executed as many times as number of columns in Data Catalogue (millions times). It will work very slow because it executes a lot of queries.

Can you optimize somehow this process?

Version upgrade of metaloaders gets difficult when customers change or add tools to already complex metaloader workflows.

Please better support customer customization by placing macros in the metaloaders at critical points

• just after data input
• just before data output

These macros would do nothing except pass data in and out

with the intent that customers build these out as they need to improve the resulting data catalog.

Most customer modifications could fit within limits of what these macros could do.

It would be easier to upgrade version when most often it only required replacing entire macros that had been modified.

The improvement would be worth the nominal impact on performance.

Where performance is a problem, un-customized macros could be removed.

.

Hi,

In standard Alteryx Gallery loader (to Connect), would it be possible to implement an option of selecting the public workflows only and ignoring the ones stored in users' private studios?

Thanks

Michal

Hi,

Are you considering building a standard SAP BW loader for Alteryx Connect. I am concious there are some custom solutions available on the market (like DVW loader), but that is not standard and does not allow data lineage with other systems?

Thank you

Michal

Hi Alteryx team,

It would be great to have a loader for Azure HDInsight. I am aware that there is a loader available for HDFS but a lot of companies are migrating to HDInsight.

Thank you very much for considering this idea.

Regards,

Jan Laznicka

Currently, the Files Loader in Connect loads all files in a directory regardless of whether they are used in an asset (e.g Alteryx workflow, Tableau dashboard, etc). This becomes a problem when you have folders containing hundreds or even thousands of files that may not actually be linked in anyway to Alteryx or Tableau as those are all populated into Connect.

It would be beneficial if the Files loader was more selective about what files to load into the Connect UI by picking up file relationships discovered using the Alteryx and Tableau loaders for example.

PowerBI Report Server is now offered as an on-prem solution. It would be great if there was a loader available to capture PowerBI reports not in Azure. If this could also capture SSRS reports which also reside on the PowerBI Report Server this would also be very beneficial.

Hi Alteryx team,

I understand that tags applied in the Alteryx Gallery are a completely different tags from the tags within Connect.

Would it be possible to sync these two and when loading Gallery metadata into Connect (workflows), load and apply also the tags?

Thank you

Michal

I'm using HP Vertica as a data source of Tableau.

At this time I'm considering to install metadata management tool in my company.

But there is no name of HP Vertica on available list of Alteryx Connect.

I would be grateful if Alteryx could release a metadata loader of HP Vertica.

The share of Vertica increase gradually in Japan.