Shared Macros - ability use a central shared macro (not package INTO the file)

[SeanAdams]

Situation:

As your analytics work grows - you find yourself using the power of Alteryx to create shared macros.    These act as an accelerator for a team because one team member can us a reusable solution created by another team member.   For example - many teams need to get data out of JIRA (or some other system) so you create a connector that everyone can use.

That's going well - and now you have 20 teams all publishing canvasses to your server (possibly 100s of canvasses running in production) which make use of your JIRA connector - all good so-far!

Problem:

BUT THEN - you discover an issue with the JIRA connector and you need to fix it and publish a new version!

It's at this point that you realise that the canvasses on your server which use your JIRA connector are NOT pointing to it, but they have made a copy and included this inside their canvas.   So when you fix the problem with the JIRA connector - no-one gets the fix!

This is because every application uploaded to the server is a yxzp file, which zips up a COPY of all the shared macros and uses this in an isolated way.

So - in order to get the new JIRA connector (with the defect repaired) used instead of the old one you now need to:

- Download EVERY canvas on your server

- Unpack them all to expose the sub-macros being used

- Inspect them to see if they are actually an instance of the JIRA Macro

- Make a list of the owner and application IDs

- reach out by e-mail or phone to every one of these folk to ask them to republish their Alteryx workflow with your new version of the JIRA connector.

Proposal:

Please can we revisit this - we really do need the power of shared macros - and we also need the ability to fix and manage these like a product over time.   This will have an impact on the engine (hence copying @AdamR_AYX ) 

Desired end-state:

- When you build a canvas using a shared macro - it doesn't store the macro itself, but rather a reference to the version on the server - unless you explicitly decide to break the connection and take a copy.

- When you check this canvas into the server - your application / yxzp does NOT include a copy of the shared macro - instead it has a reference link

- this means that Alteryx Server can now track which canvasses use this shared macro very simply

- When I fix this shared macro - I can then do an in-place update; or if the interface is not the same (i.e. different inputs or outputs) then this has to be a new version and the users will stay pointing to version 1.

This is how shared assets are managed in a micro-service world, which is the way that all of our architecture is going - and it seems important that we build this thinking into the Alteryx infrastructure too.

@AdamR_AYX ; @Treyson ; @SteveA @DerekK ; @BlytheE 

[DerekK]

This is definitely something we are looking at and will address for all the reasons you state.  There are many technical considerations around this that we will have to evaluate (e.g. resolution in the engine) but again the value is there.

[cplewis90]

I wish I could upvote this more than just by 1. It is tough as developers to build something and keep it updated for all people who use it. 

[Korri]

I completely agree with the thoughts above!  This type of feature is necessary for building the type of critical mass that is needed and it reduces the "Overhead" cost of leveraging macros and similar within Alteryx, thus changing the dialogue with IT and the greater business.  Frankly, I have built macros at my company, and have had to reach out to people individually to have them update their workflows in order to get the most up to date version, which creates some level of drag.

This is definitely one that I would upvote a million times.  #JourneyToScale.  And thanks for chiming in here @DerekK !  We appreciated our chat with you last year and look forward to seeing what is in store!  

[KylieF]

Hello!

After reviewing this idea thoroughly with our Product team, we’ve set the status for this idea to Revisit.

This feature is being heavily considered for our next generation Server product that was demonstrated at Inspire 2019. The team felt that our new Server product would be the appropriate place to release such a feature.

To properly implement this feature into our current Server product would be a large undertaking requiring a fair amount of time and careful planning. We would prefer to invest that effort into building it on the new platform at this time. 

[brianscott]

This is a serious problem that is going to grow quickly as adoption increases and people try to create more re-usable components.  Upvote given. 

[STP]

One thing that we have chosen to do for custom macros is to save the macros to a shared network location that the server also has access to and reference them all that way both in our instance of designer and the server's instance of designer. This has seemed to minimize issues with changes to the underlying macros affecting published work, but maybe I'm missing something. Regardless, differences in macro versions and where they are stored have caused issues and headaches so to see this improved would be great.

[David-Carnes]

If the SARS-CoV-2 pandemic had not struck, I would have been presenting a session called "Macro Madness - One City's Attempt to Govern the Macro-verse" in which I address this exact issue.

The solution lies in the use of .yxi files to distribute macros. When you run Designer as Admin and open a YXI file, you get the option to install for all users.  When chosen, the macro gets installed into C:\ProgramData\Alteryx\Tools\[your macro].

When the macro is installed on the server (as Admin of course), then anyone who has installed the macro in their environment will be able to publish a workflow without the macro getting a checkbox in the Manage Assets list.

Now here's where it gets magically and scary.  You can change the macro, create a new YXI file, then install it on server.  This will overwrite the existing macro.  All workflows will use the that macro at runtime.  That's the magical part.  The scary part is that all the workflows will use the new version of the macro.  So tread with care.  There's also the problem of ensuring all users keep their macros up with the latest version.  More scariness.

If you need to have concurrent versions then you need to have the new version have the same name as the existing one but in a different folder.  E.g.    C:\ProgramData\Alteryx\Tools\[your_macro_name]\macro.yxmc and C:\ProgramData\Alteryx\Tools\[your_macro_name_V2]\macro.yxmc

When two or more versions of the same named palette\macro are installed, the version can be selected by a right-click on the macro then Choose Tool Version from the context menu.  This is how some of the Predictive tools work.

If you have questions, I guess post them below.  Or message me.  Been thinking lately that I need to get some of the procedures I'm putting in place, written down for others to use and comment on.  I had hoped to get a conversation going in my presentation in New Orleans.  

Best of luck with your version of macro madness.

Ciao,

David

[David-Carnes]

@STP  We have done that but there are a couple of downsides.  If you are cut off from the network share then (a) you no longer have access and (b) Designer will freeze for a chuck of time every few minutes whilst it attempts to make the connection.  If you are in the same building as the server then you probably won't have an issue.  But working from home, in a forested city, in inclememt weather, well, the odds are not great.

If you choose to stay this route, I'd advise you to limit who can write to that directory and have some procedures in place regarding macro versioning and palette naming.  Things can get ugly quickly.

Ciao,

David

[karenmtheis]

We absolutely need centralized macro capability.  We are storing macros on a file server, and deselect the asset when publishing.  But we lose a lot of functionality like visibility of the macros, version control etc.  The available solutions are not intuitive.  Love the product, but this part is disappointing..

[JohnPelletier]

Hey everyone thanks for the feedback. We are aware of the need for this feature, and we continue to evaluate where it fits in the product lines and timelines. We'll let you know when we have more info. Thanks for your patience.