Jupyter Flow tool NoSuchKernel error

[JeffA]

Originally posted by @jrgo here :

Trying to learn this tool, but get stuck on step V9 (first run) finishing with an error directing me review the log file (below).

C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\papermill\iorw.py:50: FutureWarning: pyarrow.HadoopFileSystem is deprecated as of 2.0.0, please use pyarrow.fs.HadoopFileSystem instead.
  from pyarrow import HadoopFileSystem
Input Notebook:  C:\Users\<redact>\my_env\first_notebook_post_processed.ipynb
Output Notebook: C:\Users\<redact>\my_env\first_notebook_post_processed.ipynb
Executing:   0%|          | 0/3 [00:00<?, ?cell/s]Kernel 'Python 3 (ipykernel)' is referencing a kernel provisioner ('local-provisioner') that is not available.  Ensure the appropriate package has been installed and retry.
Executing:   0%|          | 0/3 [00:00<?, ?cell/s]
Traceback (most recent call last):
  File "C:\ProgramData\Alteryx\Tools\JupyterFlow_venv\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\ProgramData\Alteryx\Tools\JupyterFlow_venv\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\<redact>\.jupyter_flow\b01b3cec3977e9e-cv-0-1.pyz\__main__.py", line 3, in <module>
  File "C:\Users\<redact>\.jupyter_flow\b01b3cec3977e9e-cv-0-1.pyz\_bootstrap\__init__.py", line 233, in bootstrap
  File "C:\Users\<redact>\.jupyter_flow\b01b3cec3977e9e-cv-0-1.pyz\_bootstrap\__init__.py", line 36, in run
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\click\core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\click\core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\click\core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\papermill\cli.py", line 250, in papermill
    execute_notebook(
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\papermill\execute.py", line 107, in execute_notebook
    nb = papermill_engines.execute_notebook_with_engine(
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\papermill\engines.py", line 49, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\papermill\engines.py", line 343, in execute_notebook
    cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\papermill\engines.py", line 402, in execute_managed_notebook
    return PapermillNotebookClient(nb_man, **final_kwargs).execute()
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\papermill\clientwrap.py", line 43, in execute
    with self.setup_kernel(**kwargs):
  File "C:\ProgramData\Alteryx\Tools\JupyterFlow_venv\lib\contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\nbclient\client.py", line 456, in setup_kernel
    self.start_new_kernel(**kwargs)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\nbclient\util.py", line 78, in wrapped
    return just_run(coro(*args, **kwargs))
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\nbclient\util.py", line 57, in just_run
    return loop.run_until_complete(coro)
  File "C:\ProgramData\Alteryx\Tools\JupyterFlow_venv\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\nbclient\client.py", line 412, in async_start_new_kernel
    await ensure_async(self.km.start_kernel(extra_arguments=self.extra_arguments, **kwargs))
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\nbclient\util.py", line 89, in ensure_async
    result = await obj
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\jupyter_client\manager.py", line 331, in _async_start_kernel
    kernel_cmd, kw = await ensure_async(self.pre_start_kernel(**kw))
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\jupyter_client\utils.py", line 33, in ensure_async
    return await obj
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\jupyter_client\manager.py", line 295, in _async_pre_start_kernel
    self.kernel_spec,
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\jupyter_client\manager.py", line 131, in kernel_spec
    self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\jupyter_client\kernelspec.py", line 294, in get_kernel_spec
    return self._get_kernel_spec_by_name(kernel_name, resource_dir)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\jupyter_client\kernelspec.py", line 258, in _get_kernel_spec_by_name
    raise NoSuchKernel(kernel_name)
jupyter_client.kernelspec.NoSuchKernel: No such kernel named python3

Re-posting here with solution to ensure others can find this workaround while I look into a permanent fix.

[jrgo]

@JeffA 

The log does seem to have different issues depending on whether a Run As account is configured or not. Here's the log for the run WITHOUT the Run As. 

[JeffA]

Having a lot of trouble reproducing this issue. But I have some information that might help you.

The jupyter_flow_a6c1e8... log you attached indicates that you indeed need a service runner to run your workflow, as it seems JupyterFlow was trying to access a file which the runner of the workflow (and hence the runner of JupyterFlow) was not allowed access to. That issue is probably masking the real issue, which is that first log you attached (jupyter_flow_ed663c51...). The only time I've seen that issue occur in JupyterFlow is when I built my environment with a python version other than 3.8.5.

If you indeed built your environment with python 3.8.5, then it's possible that there is some python install on your machine that is mucking up the module discovery. The PATH might be getting DLL references in a bad order (this was an issue with pywin32 - and apparently now with cython). I'll see if I can make this issue occur by adding new python installs to the machine. If we can narrow it down to this as the issue, I should be able to patch the tool to prevent that from happening again.

If you could show me what your User (the service user) and System PATH variables look like, that might provide a clue. It would also help to know what your Python ecosystem is like on that machine; how many python installs it has, what versions they are, etc...

I'd also be happy to get on a call with you so we can try to get this sorted faster. Shoot me a message on Community and maybe we can connect.

[MatthewP]

@jrgo 

Would you be able to email clientsvcs@alteryx.com so that we in the support team can try to help further? If you could just include your current progress when you email us, so we can have one location for details. 

[jrgo]

@JeffA 

Apologies for my delay, got caught up with other priorities. Here's the Env Variables for the SA account I created. In case it matters, it's a local, not a domain account. My sandbox is virtual running on Parallels and not connected to any AD domain.

And per @MatthewP request, I'll send an email if we wish to continue troubleshooting outside of Community.

Thank you,

Jimmy