This site uses different types of cookies, including analytics and functional cookies (its own and from other sites). To change your cookie settings or find out more, click here. If you continue browsing our website, you accept these cookies.
When using multiple sdk tools in the same workflow the modules imports packages from the other modules. Thus, if the sdk tools do not use compatible versions of imported tools, the sdk tools fail.
Reason for this: When starting a tool the path to the python-environment is added to the global PATH. Thereby every tool imports the package version of the python-sdk tool that is inserted in the workflow FIRST and not the package version which is specified in the environment.txt of the corresponding sdk.
I have added a small example. It has 2 almost equal tools. The only difference is that the first tool uses tqdm version 4.60.0 and the second tool uses tqdm version 4.39.0. In the first workflow, the first tool is inserted first (has a lower id) resulting in both tools importing tqdm version 4.60.0 (it is printed in the messages). In the second workflow, the second tool is inserted first leading to both tools importing tqdm version 4.39.0.
This is a small example that should show the problem. In our production work, this leads to major issues since we have to use each sdk tool in a separate workflow, save intermediate data and schedule workflows accordingly, which is an organizational mess and is very frustrating.
Would be very helpful if you remove this bug and keep sdk environments separated as it is intended.
Hello @MichaelE . I'm not speaking officially here, but I believe this is a known issue with the original Python SDK. Have you tried using the new platform SDK? It leverages the Python library "shiv" which bootstraps packages on top of each tool's environment.
Another option, if your needs aren't too complicated, would be to use the new Jupyter Flow tool. If you place your Python code in a Jupyter notebook, add some IO tags, and save it, you can reference it along with the site-packages you'd like bootstrapped along with it. Every Jupyter Flow tool will run a notebook in an isolated environment determined by the site-packages provided.
Let me know if you have any questions or would like a more thorough answer!
Jeff Arnold Sr. Full Stack Software Engineer | Alteryx