Alteryx Connect Ideas

Our Data Catalogue in Connect has about 2 millions items (tables, views, columns).

I see next issues:

1. We collect metadata from about 10+ DBMS. So after each Metadata loader run, Alteryx Connect will start load_alteryx_db script and process whole staging area (DB_*) tables, not only current extracted metadata set from single DBMS. It will lead huge redundancy.
2. Follows from first issue: One-by-one comparison of loaded metadata will take a lot of time in real environment with 1-2 millions items (ordinary situation in large Bank). And this comparison will be executed several times. It will increase the redundancy in the number of DBMS servers.

3. All queries in this script containing column or table name as a parameter (e.g. src.TABLE_NAME='${query_table_name}' AND                                 src.COLUMN_NAME='${query_column_name}') will be executed as many times as number of columns in Data Catalogue (millions times). It will work very slow because it executes a lot of queries.

Can you optimize somehow this process?

Version upgrade of metaloaders gets difficult when customers change or add tools to already complex metaloader workflows.

Please better support customer customization by placing macros in the metaloaders at critical points

• just after data input
• just before data output

These macros would do nothing except pass data in and out

with the intent that customers build these out as they need to improve the resulting data catalog.

Most customer modifications could fit within limits of what these macros could do.

It would be easier to upgrade version when most often it only required replacing entire macros that had been modified.

The improvement would be worth the nominal impact on performance.

Where performance is a problem, un-customized macros could be removed.

.

A lot of information is not captured when colleagues run numerous SQL codes on server. Oracle, SQL Server, Azure and others...

Would there be a clever way of capturing and archiving all this queries run?

It may be wise to collect these for several reasons;

1. SQL code profiling is an important matter especially for DB admins. You can see the most required tables and fields etc.
2. Also have a grasp on most frequent and time consuming joins to enhance DB performance
3. Figure the queries that can be replicated in Alteryx and deployed to a server so that no business user needs to run SQL code instead  they will be reverted to the gallery.

So found out a similar feature is now available in SQL Server 2016 -->  https://docs.microsoft.com/en-us/sql/relational-databases/performance/monitoring-performance-by-usin...