Jupyter Flow tool NoSuchKernel error

[JeffA]

Originally posted by @jrgo here :

Trying to learn this tool, but get stuck on step V9 (first run) finishing with an error directing me review the log file (below).

C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\papermill\iorw.py:50: FutureWarning: pyarrow.HadoopFileSystem is deprecated as of 2.0.0, please use pyarrow.fs.HadoopFileSystem instead.
  from pyarrow import HadoopFileSystem
Input Notebook:  C:\Users\<redact>\my_env\first_notebook_post_processed.ipynb
Output Notebook: C:\Users\<redact>\my_env\first_notebook_post_processed.ipynb
Executing:   0%|          | 0/3 [00:00<?, ?cell/s]Kernel 'Python 3 (ipykernel)' is referencing a kernel provisioner ('local-provisioner') that is not available.  Ensure the appropriate package has been installed and retry.
Executing:   0%|          | 0/3 [00:00<?, ?cell/s]
Traceback (most recent call last):
  File "C:\ProgramData\Alteryx\Tools\JupyterFlow_venv\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\ProgramData\Alteryx\Tools\JupyterFlow_venv\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\<redact>\.jupyter_flow\b01b3cec3977e9e-cv-0-1.pyz\__main__.py", line 3, in <module>
  File "C:\Users\<redact>\.jupyter_flow\b01b3cec3977e9e-cv-0-1.pyz\_bootstrap\__init__.py", line 233, in bootstrap
  File "C:\Users\<redact>\.jupyter_flow\b01b3cec3977e9e-cv-0-1.pyz\_bootstrap\__init__.py", line 36, in run
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\click\core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\click\core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\click\core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\papermill\cli.py", line 250, in papermill
    execute_notebook(
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\papermill\execute.py", line 107, in execute_notebook
    nb = papermill_engines.execute_notebook_with_engine(
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\papermill\engines.py", line 49, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\papermill\engines.py", line 343, in execute_notebook
    cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\papermill\engines.py", line 402, in execute_managed_notebook
    return PapermillNotebookClient(nb_man, **final_kwargs).execute()
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\papermill\clientwrap.py", line 43, in execute
    with self.setup_kernel(**kwargs):
  File "C:\ProgramData\Alteryx\Tools\JupyterFlow_venv\lib\contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\nbclient\client.py", line 456, in setup_kernel
    self.start_new_kernel(**kwargs)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\nbclient\util.py", line 78, in wrapped
    return just_run(coro(*args, **kwargs))
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\nbclient\util.py", line 57, in just_run
    return loop.run_until_complete(coro)
  File "C:\ProgramData\Alteryx\Tools\JupyterFlow_venv\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\nbclient\client.py", line 412, in async_start_new_kernel
    await ensure_async(self.km.start_kernel(extra_arguments=self.extra_arguments, **kwargs))
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\nbclient\util.py", line 89, in ensure_async
    result = await obj
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\jupyter_client\manager.py", line 331, in _async_start_kernel
    kernel_cmd, kw = await ensure_async(self.pre_start_kernel(**kw))
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\jupyter_client\utils.py", line 33, in ensure_async
    return await obj
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\jupyter_client\manager.py", line 295, in _async_pre_start_kernel
    self.kernel_spec,
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\jupyter_client\manager.py", line 131, in kernel_spec
    self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\jupyter_client\kernelspec.py", line 294, in get_kernel_spec
    return self._get_kernel_spec_by_name(kernel_name, resource_dir)
  File "C:\Users\<redact>\.shiv\b01b3cec3977e9e-cv-0-1_830b0357a27e50e91c83b7a77e0750042db9a80cd6fde18310e1a11b29ef90c1\site-packages\jupyter_client\kernelspec.py", line 258, in _get_kernel_spec_by_name
    raise NoSuchKernel(kernel_name)
jupyter_client.kernelspec.NoSuchKernel: No such kernel named python3

Re-posting here with solution to ensure others can find this workaround while I look into a permanent fix.

[JeffA · Accepted solution]

Solution:

@jrgo try doing this in your environment and re-running the tool:

C:\path\to\your\environment\root\folder\Scripts\pip.exe install jupyter-client==6.1.12

[jrgo]

@JeffA 

Thanks again for the assist. I am running into another issue that I wanted to run by you and get your thoughts. I followed the steps to prepare for packaging/publishing and when I run in Gallery, it returns an error. 

I published it to 2 different environments after installing the Jupyter Flow tool (as admin and completed without error), one running 21.2 with a service account set as the Run As and the other on the latest, v21.3, that has no Run as set up. I'm mentioning the Run as because that looks to have an impact to the error return as well as the errors showing in the log file the mentioned in the result page.

Attached is the packaged workflow and I included a copy of the log from each server within the yxzp.

Also want to note that I opened the workflow from the server (RDP) from their respective Gallery and ran them in Designer and it finishes without error.

AMP turned on or off doesn't seem to have any impact.

Any ideas here?

Thank you,

Jimmy

[JeffA]

Hey again @jrgo . Sorry you're having this issue. But hopefully we can get it squared away and make sure this tool is solid for everyone.

I don't see anything weird in your notebook, and I'm currently working on reproducing the issue.

One quick checklist item is that JupyterFlow has to be installed as Admin if you didn't do it that way. To do this, open Designer as administrator, and do not drag the yxi or double click the yxi. Instead, go to open a workflow and open that yxi instead. This will ensure that the tool is installed as admin.

If that doesn't fix it, could you upload the log files if possible? Even just a screenshot of their contents would be helpful.

[jrgo]

Hi @JeffA 

Yes, I made sure that the tool was installed from an elevated instance of Designer and was presented and selected the option the install for all users.

[JeffA]

@jrgo Ok so I reproduced an error throwing on Server 2021.3 with your workflow. It did not fail in Designer but it did fail on Server.

In my case, this was a long paths not enabled issue. The reason it passed in Designer was because the user context was different, so the path in which shiv (one of the libs we use for Jupyter Flow) was saving its files was short. When running from server the path was a lot longer, and so failure ensued. After enabling long paths on that machine, it worked. I enabled long paths using the command shown in this guide: https://www.msftnext.com/how-to-enable-ntfs-long-paths-in-windows-10/

I'm personally looking forward to the day when Windows stops defaulting long paths to disabled 😕

Let me know if that was not it! If that was the problem, do you think it would be helpful to include that step in my guide?

[jrgo]

@JeffA that fix didn't seem to work for me. I only attempted this on my local sandbox environment. I've attached the Jupyter log for the last attempt.

[JeffA]

@jrgo 

A few questions to help me reproduce this faster:

1. What is the server version that this error log you linked was generated from?

2. Was this the version with no "Run As" set up, or the version with "Run As" set up?

[jrgo]

@JeffA 

Was on 2021.3. It was ran WITH a service account, but that was on the same server that I had previously tested without one, in case that matters. I did also test it before the SA was added (after fix you suggested was applied).

[JeffA]

@jrgo 

Has the logfile been showing basically the same issue each time? Or is it significantly different based on whether or not a service account was used?