Python packages / dependencies / venv's

Question

Dear all,

At the moment I am trying to make more use of python scripting inside Alteryx, but control of different python environments seems possibly tricky.

At the moment we have a structure where we point to the specific venv inside a specific script, but it feels possibly suboptimal:

import os
sys.path.insert(1, "path to venv lib\site-packages folder")

Is there a recommended way to ensure proper dependency management, or might it even be better to not use complex python scripting but to place everything inside an Alteryx tool using the Python SDK?

PS: we have used the following guide to implement version control:

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Python-Code-Tool-Script-Runner-Macro-Code-Injection/m-p/419146#M77446

PPS: image in attachment show roughly how it would work

Alteryx_python_dependency.drawio.png

roelof · Answer

For those interested in this topic:
atm we do not see the benefit of the jupyter flow tool, it only creates additional venv's and needs a rebuild every time there is a change in the environment, and it does not have any build in checks for compatibility between external packages and internal python kernel of Alteryx.

Also briefly spoke to Alteryx themselves, python code injection with proper version control in gitlab/github/bitbucket does not seem to be a common use-case, and Alteryx has no recommendations.

The plan we have now consists of:
1. on scheduler: daily automatic pull from gitlab if the python scripts have passed the checks from gitlab CI-CD (inside containerized python env)
2. on scheduler: after the pull trigger alteryx test flows in order to check integration of scripts with flows (through powershell atm, but ideally done through alteryx API)

checks from CI CD tooling will contain standard unit testing, etc, and also consist of checking that each script points to the correct venv in order to integrate with Alteryx. Checks on the server will consist of
-checking venv exist
-package requirements are met
-that python versions match between venv and alteryx
-> additionally integration tests with the actual workflows

In parallel to this solution we also note that even though Python code injection is possible in Alteryx it has no semblance of a stable solution, we are going to explore other workflow programs, like for example google DataFlow, google functions, and possibly other ETL tools that have out of the box integrations with git-systems

Cheers,
Roelof

roelof · Answer

Hi apathetichell,

No, I had not looked at the jupyter-flow tool, it looks to provide similar functionality indeed as the solution I am working with atm. What would be the advantage of creating a separate environment everytime instead of pointing to the Lib\site-packages folder using sys.path.insert('path')?

is there also a way to easily check which python version Alteryx is relying on in order to build in a check to make sure the venv has the same underlying python version?

Cheers,

Roelof

apathetichell · Answer

Have you looked into the Jupyter Flow tool?