Community Spring Cleaning week is here! Join your fellow Maveryx in digging through your old posts and marking comments on them as solved. Learn more here!

Alteryx IO Discussions

Customize and extend the power of Alteryx with SDKs, APIs, custom tools, and more.

Unable to use Tensorflow with AYX Python SDK v2

sf_sky_scientist
5 - Atom

How is it possible to use TensorFlow with the new python SDK v2? 

 

First of all, the TensorFlow package requirements are not compatible with "ayx-python-sdk." After lots of trial and error, I used the following version requirements list to solve the issues:

 

 

 

 

ayx-python-sdk==2.1.0
protobuf==3.20.1
tensorflow==2.9
MarkupSafe==2.0.1

 

 

 

 

Using this requirements list, ayx_plugin_cli generated the YXI file, installed it on my machine, and I was able to run a workflow with that plugin in it. However, when I try using this plugin on a different machine, it gives the error shown below:

 

 

 

Error: myplugin(1): Internal error: Failed to read port assignment
Error: myplugin(1): All incoming anchors closed, but plugin has errors.

 

 

 

This error indicates that Alteryx could not load the required packages before initiating the plugin's main class. Is there any solution to this problem?

 

Also, I have noticed that using "shiv" is probably not the best solution for oversized packages like TensorFlow. It makes  Alteryx very slow every time that tries to load packages, and as you can see, it might end up with platform compatibility issues.

 

 

4 REPLIES 4
shawnto
Alteryx
Alteryx

Hello!

 

I can start by sharing some debugging advice. If you don't find the cause of your error with that, please share your yxi here. Or, at minimum, your tools' code so we can do some more investigation.


@sf_sky_scientist wrote:

How is it possible to use TensorFlow with the new python SDK v2? 

 

First of all, the TensorFlow package requirements are not compatible with "ayx-python-sdk." After lots of trial and error, I used the following version requirements list to solve the issues:

 

 

 

 

 

 

ayx-python-sdk==2.1.0
protobuf==3.20.1
tensorflow==2.9
MarkupSafe==2.0.1

 

 

 

 

 

 

Using this requirements list, ayx_plugin_cli generated the YXI file, installed it on my machine, and I was able to run a workflow with that plugin in it. However, when I try using this plugin on a different machine, it gives the error shown below:

 

 

 

 

 

Error: myplugin(1): Internal error: Failed to read port assignment
Error: myplugin(1): All incoming anchors closed, but plugin has errors.

 

 

 

 

 

This error indicates that Alteryx could not load the required packages before initiating the plugin's main class. Is there any solution to this problem?

 

Also, I have noticed that using "shiv" is probably not the best solution for oversized packages like TensorFlow. It makes  Alteryx very slow every time that tries to load packages, and as you can see, it might end up with platform compatibility issues.

 


To this point, unless you've already done some investigation not shown here to specifically verify, this is not what this error always means. Something we've learned from posts and feedback like this, is that a critical detail is missing haha. This is a general exception for python SDK tools that could not start their process for any unknown (by designer as a process owner) errors. To get a better picture of what the cause is, you can check the PythonSDK.log here: 

SDK Log location for general errors:
C:\Users\sf_sky_scientist\AppData\Local\Alteryx\Log

Here, you can see anything that the SDK process throws itself, or anything you throw with `self.provider.io.*` where * is `.info | .warn | .error`. Additionally, you may run your tool(s) as a standalone process (albeit one that can't talk to anything since designer needs to launch it to actually chat with data). You can do this with:

Running an SDK Tool stand-alone

`C:\Users\Your.Name\AppData\Local\Alteryx\bin\Python\python-3.8.5-embed-amd64\python.exe main.pyz start-sdk-tool-service ayx_plugins YouToolClassName`

This will let you directly see the stack trace without needing to run a workflow in designer for quick iteration on debugging like this. You can also edit your "installed" code by accessing and editing the cached shiv version at ~/.shiv. I understand this is a bit tedious, that's cause its not technically a supported feature but a common workaround when devs want to rapidly iterate on code. (Which we do recognize and are working to make a much more intuitive experience btw).

Of the above, I recommend trying first to run your tool standalone. TBH, we even see this (your main issue) the most with minor coding errors, since the generic message sounds so much worse than its scope of possibilities lol.

I'm currently not aware of anything that should prevent Tensorflow's use, as someone who's also investigated pytorch's use with SDK tools. If this IS the case though, it's certainly something I'd like to fix and enable. So, if you get progress with the above but still need some tips, please don't hesitate to respond here. Please provide any of the following that you can:
* log files relevant to the error and for said tool
* code or a link to a full yxi
* screenshots of errors or the workflow if there are additional tools in the workflow

* Your designer version 

 

Thanks, 

SHawn

dcates
6 - Meteoroid

A little help, please. Tensorflow won't install with 2023.1. Thanks!

 

 
 
 
Collecting tensorflow
  Using cached tensorflow-2.12.0-cp38-cp38-win_amd64.whl (1.9 kB)
Collecting tensorflow-intel==2.12.0; platform_system == "Windows"
  Using cached tensorflow_intel-2.12.0-cp38-cp38-win_amd64.whl (272.8 MB)
Collecting tensorflow-estimator<2.13,>=2.12.0
  Using cached tensorflow_estimator-2.12.0-py2.py3-none-any.whl (440 kB)
Collecting typing-extensions>=3.6.6
  Using cached typing_extensions-4.7.1-py3-none-any.whl (33 kB)
Collecting gast<=0.4.0,>=0.2.1
  Using cached gast-0.4.0-py3-none-any.whl (9.8 kB)
Collecting numpy<1.24,>=1.22
  Using cached numpy-1.23.5-cp38-cp38-win_amd64.whl (14.7 MB)
Collecting keras<2.13,>=2.12.0
  Using cached keras-2.12.0-py2.py3-none-any.whl (1.7 MB)
Requirement already satisfied: six>=1.12.0 in c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\site-packages (from tensorflow-intel==2.12.0; platform_system == "Windows"->tensorflow) (1.12.0)
Collecting tensorboard<2.13,>=2.12
  Using cached tensorboard-2.12.3-py3-none-any.whl (5.6 MB)
Collecting astunparse>=1.6.0
  Using cached astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting google-pasta>=0.1.1
  Using cached google_pasta-0.2.0-py3-none-any.whl (57 kB)
Collecting opt-einsum>=2.3.2
  Using cached opt_einsum-3.3.0-py3-none-any.whl (65 kB)
Collecting libclang>=13.0.0
  Using cached libclang-16.0.0-py2.py3-none-win_amd64.whl (24.4 MB)
Collecting wrapt<1.15,>=1.11.0
  Using cached wrapt-1.14.1-cp38-cp38-win_amd64.whl (35 kB)
Requirement already satisfied: packaging in c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\site-packages (from tensorflow-intel==2.12.0; platform_system == "Windows"->tensorflow) (21.3)
Collecting flatbuffers>=2.0
  Using cached flatbuffers-23.5.26-py2.py3-none-any.whl (26 kB)
Requirement already satisfied: setuptools in c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\site-packages (from tensorflow-intel==2.12.0; platform_system == "Windows"->tensorflow) (59.4.0)
Collecting absl-py>=1.0.0
  Using cached absl_py-1.4.0-py3-none-any.whl (126 kB)
Collecting protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3
  Using cached protobuf-4.23.3-cp38-cp38-win_amd64.whl (422 kB)
Collecting termcolor>=1.1.0
  Using cached termcolor-2.3.0-py3-none-any.whl (6.9 kB)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\site-packages (from tensorflow-intel==2.12.0; platform_system == "Windows"->tensorflow) (1.28.1)
Collecting h5py>=2.9.0
  Using cached h5py-3.9.0-cp38-cp38-win_amd64.whl (2.7 MB)
Processing c:\users\tripo\appdata\local\pip\cache\wheels\46\d9\15\d2800d4089dc4c77299ac7513c6aa1036f5491edbd2bf6ba16\jax-0.4.13-py3-none-any.whl
Collecting tensorflow-io-gcs-filesystem>=0.23.1; platform_machine != "arm64" or platform_system != "Darwin"
  Using cached tensorflow_io_gcs_filesystem-0.31.0-cp38-cp38-win_amd64.whl (1.5 MB)
Collecting werkzeug>=1.0.1
  Using cached Werkzeug-2.3.6-py3-none-any.whl (242 kB)
Collecting markdown>=2.6.8
  Using cached Markdown-3.4.3-py3-none-any.whl (93 kB)
Collecting tensorboard-data-server<0.8.0,>=0.7.0
  Using cached tensorboard_data_server-0.7.1-py3-none-any.whl (2.4 kB)
Requirement already satisfied: requests<3,>=2.21.0 in c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\site-packages (from tensorboard<2.13,>=2.12->tensorflow-intel==2.12.0; platform_system == "Windows"->tensorflow) (2.22.0)
Collecting google-auth-oauthlib<1.1,>=0.5
  Using cached google_auth_oauthlib-1.0.0-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: wheel>=0.26 in c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\site-packages (from tensorboard<2.13,>=2.12->tensorflow-intel==2.12.0; platform_system == "Windows"->tensorflow) (0.37.0)
Collecting google-auth<3,>=1.6.3
  Using cached google_auth-2.21.0-py2.py3-none-any.whl (182 kB)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\site-packages (from packaging->tensorflow-intel==2.12.0; platform_system == "Windows"->tensorflow) (2.4.0)
Collecting importlib-metadata>=4.6; python_version < "3.10"
  Using cached importlib_metadata-6.7.0-py3-none-any.whl (22 kB)
Collecting scipy>=1.7
  Using cached scipy-1.10.1-cp38-cp38-win_amd64.whl (42.2 MB)
Collecting ml-dtypes>=0.1.0
  Using cached ml_dtypes-0.2.0-cp38-cp38-win_amd64.whl (938 kB)
Collecting MarkupSafe>=2.1.1
  Using cached MarkupSafe-2.1.3-cp38-cp38-win_amd64.whl (17 kB)
Requirement already satisfied: idna<2.9,>=2.5 in c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\site-packages (from requests<3,>=2.21.0->tensorboard<2.13,>=2.12->tensorflow-intel==2.12.0; platform_system == "Windows"->tensorflow) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\site-packages (from requests<3,>=2.21.0->tensorboard<2.13,>=2.12->tensorflow-intel==2.12.0; platform_system == "Windows"->tensorflow) (1.25.10)
Requirement already satisfied: certifi>=2017.4.17 in c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\site-packages (from requests<3,>=2.21.0->tensorboard<2.13,>=2.12->tensorflow-intel==2.12.0; platform_system == "Windows"->tensorflow) (2021.10.8)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\site-packages (from requests<3,>=2.21.0->tensorboard<2.13,>=2.12->tensorflow-intel==2.12.0; platform_system == "Windows"->tensorflow) (3.0.4)
Collecting requests-oauthlib>=0.7.0
  Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting rsa<5,>=3.1.4
  Using cached rsa-4.9-py3-none-any.whl (34 kB)
Collecting cachetools<6.0,>=2.0.0
  Using cached cachetools-5.3.1-py3-none-any.whl (9.3 kB)
Collecting pyasn1-modules>=0.2.1
  Using cached pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB)
Collecting zipp>=0.5
  Using cached zipp-3.15.0-py3-none-any.whl (6.8 kB)
Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.2.2-py3-none-any.whl (151 kB)
Collecting pyasn1>=0.1.3
  Using cached pyasn1-0.5.0-py2.py3-none-any.whl (83 kB)
Installing collected packages: tensorflow-estimator, typing-extensions, gast, numpy, keras, protobuf, MarkupSafe, werkzeug, zipp, importlib-metadata, markdown, tensorboard-data-server, absl-py, pyasn1, rsa, cachetools, pyasn1-modules, google-auth, oauthlib, requests-oauthlib, google-auth-oauthlib, tensorboard, astunparse, google-pasta, opt-einsum, libclang, wrapt, flatbuffers, termcolor, h5py, scipy, ml-dtypes, jax, tensorflow-io-gcs-filesystem, tensorflow-intel, tensorflow
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'c:\\program files\\alteryx\\bin\\miniconda3\\envs\\designerbasetools_venv\\Lib\\site-packages\\tensorflow_estimator'
Consider using the `--user` option or check the permissions.
 
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
<ipython-input-4-b963a67deeaa> in <module>
      2 # script here (only missing packages will be installed)
      3 from ayx import Package
----> 4 Package.installPackages('tensorflow')

c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\site-packages\ayx\Package.py in installPackages(package, install_type, debug)
    200     print(pip_install_result["msg"])
    201     if not pip_install_result["success"]:
--> 202         raise pip_install_result["err"]

c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\site-packages\ayx\Utils.py in runSubprocess(args_list, debug)
    118 
    119     try:
--> 120         result = subprocess.check_output(args_list, stderr=subprocess.STDOUT)
    121         if debug:
    122             print("[Subprocess success!]")

c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\subprocess.py in check_output(timeout, *popenargs, **kwargs)
    409         kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b''
    410 
--> 411     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,    412                **kwargs).stdout
    413 

c:\program files\alteryx\bin\miniconda3\envs\designerbasetools_venv\lib\subprocess.py in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    510         retcode = process.poll()
    511         if check and retcode:
--> 512             raise CalledProcessError(retcode, process.args,    513                                      output=stdout, stderr=stderr)
    514     return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['c:\\program files\\alteryx\\bin\\miniconda3\\envs\\designerbasetools_venv\\python.exe', '-I', '-m', 'pip', 'install', 'tensorflow']' returned non-zero exit status 1.
Chunxin
5 - Atom

I have the exact same problem here! Hope to get answers soon.

dcates
6 - Meteoroid

Try:

 

Package.installPackages(['tensorflow'], install_type="install --user")

 

I got it from another post, Solved: Update the Python Packages like pandas - Alteryx Community at the bottom of the page. I was able to install Pulp, which was also giving me problems. Tensorflow got further but the install paths are bad, I think. Another problem to work through, at least!!