Python tool *.yxi offline installer - by pass network block to PyPI packages

[haphan_tran]

Hi data people,

I had this question a while back. In our company, users will not have access to pypi to download Python packages. All the *.yxi tools cannot be installed for our users due to this reason.

In this post, I will use Azure Data Lake tools as an example: Azure Data Lake Tools - Alteryx Community

*.yxi tool can be changed to *.zip and be unzip, then you can explore the structure of the tool.

Normally it will come with HTML file for the GUI, XML for information about the tool and icon.png file for, you know, icon of the tool. Along with that, there is a requirements.txt file. And most importantly, at least one *.py file to host the code of the tool (in this example, this contain a ton of *.py in subfolders as well).

Any time you install a *.yxi tool, Alteryx will use its internal python to run download and install those packages inside requirements.txt. On your personal machine, that's not a problem. On the corporate devices, this tt is a huge challenge. Even when I requests access to PyPI, it is terribly slow to install any tool (~30 mins).

I checked several docs and articles and talked to support team and figured out a way to pre-download packages into *.yxi file.

First step is download those packges in requirements.txt into a wheels folder (the name is not important, you can do whl or whatever). You need to use Alteryx's python to download, not your PC's python. Normally Alteryx's Python is not in Path, so you need the full path to call it.

Create new folder Wheels, go inside and Shift right click to open PowerShell

Run this command to check that you get the Alteryx's Python (Alteryx 2020 comes with Python 3.6, 2021 comes with 3.8.5) 

C:\"Program Files"\Alteryx\bin\Miniconda3\python --version
Python 3.8.5

Next, run this command to download all the Python Packages in requirements.txt

 C:\"Program Files"\Alteryx\bin\Miniconda3\python -m pip download -r ..\requirements.txt
.
.
Successfully downloaded azure-common azure-datalake-store azure-mgmt-datalake-nspkg...

Here is the original requirements.txt

azure-common==1.1.14
azure-datalake-store==0.0.27
azure-mgmt-datalake-nspkg==2.0.0
azure-mgmt-datalake-store==0.5.0
azure-mgmt-nspkg==2.0.0
azure-mgmt-resource==2.0.0
azure-nspkg==2.0.0
cryptography==2.9.2
protobuf==3.5.1
google-cloud-bigquery==1.1.0
google-cloud-core==0.28.1
fastavro==1.0.0
oauthlib==2.0.6
onedrivesdk==1.1.8
openpyxl==3.0.5
defusedxml==0.7.1
pypac==0.13.0

We need to change this one to trick Alteryx to install from offline packages, not download it on the fly. We need the list of wheels file we downloaded. I use python os.listdir() to get all the files, then go into excel to append "./wheels/" to file name. I changed the requirements.txt to requirements_og.txt for future reference and create a new requirements.txt. Pay attention to the first 2 lines to trick ALteryx to install from local folder wheels

--find-links wheels
--no-index
./wheels/adal-1.2.7-py2.py3-none-any.whl
./wheels/azure_common-1.1.14-py2.py3-none-any.whl
./wheels/azure_datalake_store-0.0.27-py2.py3-none-any.whl
./wheels/azure_mgmt_datalake_nspkg-2.0.0-py2.py3-none-any.whl
./wheels/azure_mgmt_datalake_store-0.5.0-py2.py3-none-any.whl
./wheels/azure_mgmt_nspkg-2.0.0-py2.py3-none-any.whl
./wheels/azure_mgmt_resource-2.0.0-py2.py3-none-any.whl
./wheels/azure_nspkg-2.0.0-py2.py3-none-any.whl
./wheels/cachetools-4.2.4-py3-none-any.whl
./wheels/certifi-2021.10.8-py2.py3-none-any.whl
./wheels/cffi-1.15.0-cp38-cp38-win_amd64.whl
./wheels/charset_normalizer-2.0.12-py3-none-any.whl
./wheels/cryptography-2.9.2-cp38-cp38-win_amd64.whl
... there are more, but you got the idea.

After that, you only need to go back one level and select the tool folder + config.xml -> zip the file.

Then change the file from *.zip to *.yxi. You got a new tool that can be install offline very fast (for me, I took 3 mins vs. 30 mins originally).

Note: If the tool has some old packages that doesn't have a wheel, you may need to include Cython to compile the packages.

[BrandonB]

Thanks for sharing! Very useful for organizations requiring offline installation capabilities. 

[haphan_tran]

I wanted to edit the original post but don't know how to, so I will post a few more information here to help this post reach out to more people with the error.

Situation: you want to install a Python based tool for Alteryx (Sharepoint Input files, Azure Data Lake, Dynamics CRM, Dataverse, ...). You can install it just find on your personal device, but with your corporate laptop, you get the error: No Module named 'pip'

Root cause: When you install the new tool, Alteryx will go to internet to download required packages. Most organization will block that connection.

Solution: In above article, I show you how you can pre-download all the required package and add it into the tool installer itself. That way, you can install it offline on your corporate laptop.