feat: Connect to synchronized kernel #77

fecet · 2023-04-13T06:43:29Z

I managed to implement an MVP for my feature request on #76. The implementation follows your suggested approach:

start sync
Request the connection info by executing javascript Jupyter.notebook.kernel.id
Open a terminal and connect to it by running jupyter console --existing.

This implementation supports connecting to a remote kernel by explicitly passing a hostname parameter and a naive approach(ssh host -t), and this require https://github.com/akinsho/toggleterm.nvim as a dependency

You can view the workflow here:

output.mp4

In my opinion, to improve this functionality, we may need to:

Better function name
Native terminal
Expose user configuration interfaces to modify the running command and terminal style.
Document
Check windows support

Long-term plan

Improve the process of connecting to a remote kernel to make it more intelligent.
autocompletion Tab completion inside Jupyter QtConsole ipdb shell jupyter/qtconsole#99 Question: Completion in nvim's terminal window hrsh7th/nvim-cmp#1219
Notebook 7 support Jupyter Notebook 7 compatibility (discussion) #74

As a beginner in JavaScript and Neovim plugin development, I would be grateful if you could share any additional ideas you may have to further improve this functionality. Additionally, I would appreciate your guidance on what else I need to do to merge this pull request.

kiyoon · 2023-04-13T08:10:24Z

Hi, thanks a lot for the PR! This looks awesome.

Is there any reason you used toggleterm? I haven't used the plugin but it's better not to have too many dependencies. Is it possible to just open a terminal natively and connect to the kernel? You can follow the implementation of :JupyniumStartAndAttachToServerInTerminal
To keep the naming consistent, we can change the name to :JupyniumKernelConnectInTerminal. Or, :JupyniumKernelOpenInTerminal may be a more user-friendly name?
Can you add this command in README?
I know this can be annoying, but Jupynium also supports Windows. Make sure the command runs on different types of shells.

src/jupynium/events_control.py

src/jupynium/lua/helpers.lua

fecet · 2023-04-13T17:00:52Z

Thank you for your suggestion. I will start making the changes immediately. The following points may require further input:

Toggleterm exposes an interface(https://github.com/akinsho/toggleterm.nvim#custom-terminals) for creating terminals, which makes it easier to configure and use. To balance this point and keep the dependencies minimal, it may be possible to fallback to Neovim's native terminal (just like Notify) when the user is detected not to have installed it.
Personally, I don't use Windows, so I would like to know which functions may not be compatible with Windows? As far as I can see, an incompatibility occurs with SSH because many people on Windows do not configure SSH directly but use some SSH management tools. However, I cannot think of a simple way to address this.

kiyoon · 2023-04-13T17:43:38Z

If the configuration is not absolutely game-changing, I think the native terminal still can do most. We already have quite many (optional) dependencies and things like this can be hard to maintain and test. I also know many people like to use floaterm, so toggleterm is not necessarily the one and only choice when it comes to terminal management?
Maybe it will work directly on PowerShell / CMD if your command is simple. If you have strings with quotes etc. you have to escape the commands in a slightly different way. You can assume that the command ssh is installed already, as I think it is installed by default on PowerShell? I also don't develop on Windows but I realised many people develop on Windows just because it's a company requirement related to software security etc. etc.

If you don't have Windows at all, I'll do some simple tests once all implementation is done. But do let me know what kind of example (complicated) shell commands should be executed, so I can see if they all work well.

fecet · 2023-04-14T06:59:57Z

OK, I will use the native terminal as default and expose the kernel id, so the user can define their own terminal interface if they want. As for Windows, the commands I am currently using are relatively simple, so they should be able to run without issue. However, I will keep in mind that I may need to use more complex commands in the future.

kiyoon · 2023-04-14T10:05:46Z

Let's first see how many people uses this feature. I think it's a niche functionality only a few people may use (but still very useful, thanks!), so if they ever wanted to customise terminal interface we can later add some more options without a dependency. First of all, is there any requirement for you or is horizontal split enough most of the time?

And the kernel ID is able to be achieved with the Lua API with one line of javascript. We don't need to really expose it I think.

fecet · 2023-04-14T16:09:24Z

And the kernel ID is able to be achieved with the Lua API with one line of javascript. We don't need to really expose it I think.

What I meant was to add a function similar to :lua Jupynium_connect_info(bufnr), which would only return the kernel ID. There is no need to create a Vim command for it, but it could be mentioned in the README file along with an example of how to require it to create a terminal. This way, users can configure it in their init file.

Or do we already have something like :lua Jupynium_execute_js(code)? Then that's definitely unnecessary.

kiyoon · 2023-04-14T17:29:50Z

:lua Jupynium_connect_info(bufnr)

This can be done with Jupynium_execute_javascript(bufnr, "Jupyter.notebook.kernel.id"). It's in the Lua API section in the README. So we can provide your functionality without needing an extra dependency, but for people who really want to customise it can still achieve that easily.

kiyoon · 2023-04-17T09:54:44Z

In the README, I think instead of long description you can just add in the Vim Commands section, add one line of description would be enough? I appreciate it though!

fecet · 2023-04-17T09:58:34Z

In the README, I think instead of long description you can just add in the Vim Commands section, add one line of description would be enough? I appreciate it though!

Good for me, I also feel that the long description now looks a bit inconsistent.

src/jupynium/lua/helpers.lua

kiyoon · 2023-04-17T15:15:23Z

src/jupynium/events_control.py

@@ -522,6 +522,12 @@ def process_request_event(nvim_info: NvimInfo, driver, event):
        event[3].send(ret_obj)
        return True, None

+    elif event[1] == "kernel_connect_info":
+        driver.switch_to.window(nvim_info.window_handles[bufnr])
+        kernel_id = driver.execute_script("Jupyter.notebook.kernel.id")


Do you also get this error?

Error executing Lua callback: [string "<nvim>"]:718: attempt to concatenate local 'kernel_id' (a userdata value) stack traceback: [string "<nvim>"]:718: in function 'Jupynium_get_kernel_connect_shcmd' [string "<nvim>"]:726: in function <[string "<nvim>"]:723>

I think the kernel_id value is wrong. You should add return.

kernel_id = driver.execute_script("return Jupyter.notebook.kernel.id")

Sorry I only test no-sync cases (connect to latest kernel) today since it's just implemented.

That's okay. I think it works alright now. Thanks a lot for your work. The only things hasn't been done are

Test on Windows

Mention pip install jupyter-console dependency in README

# jupyter-console is optional and used for `:JupyniumKernelOpenInTerminal` pip install notebook jupyter-console

I think you can add that command in the README Jupyter Notebook version dependency area. Then we're good to go.

I see that you have requested someone else to do the testing work for Windows. If they lack the time, since this plugin does not require any additional installation, I should be able to find an opportunity to complete the testing on Windows.

And as for the document, it's suitable to mention an installation command in the requirements area? But I also don't find other where to place that.

If you can do that, that'd be helpful. He's actually using it on Windows so I thought it would be good to have it from an actual user. Once we make sure it runs on Windows it's good to go.

kiyoon · 2023-04-17T15:55:07Z

@sho-87 Hi, it would be really appreciated if you can test this feature on Windows. It runs a shell command so I think potentially it will be buggy on Windows shell.

Only if you care about this feature though!

You can just test :JupyniumKernelOpenInTerminal and see if you can interact with the Jupyter kernel of the opened notebook.

sho-87 · 2023-04-18T03:52:41Z

A couple of things:

JupyniumStartAndAttachToServerInTerminal is not available to me without first starting/attaching to the server. I presume I should be running this command instead of the traditional startattach?
The command JupyniumKernelOpenInTerminal itself seems to work fine for me. I created a cell and executed the code, then running JupyniumKernelOpenInTerminal opens up an nvim terminal running ipython, and I have access through that to the variables I created. neat stuff

Isn't this basically a REPL window that is connected to the notebook kernel? Calling it a REPL might make it more obvious what this feature does

fecet · 2023-04-18T04:17:36Z

Basically, yes. But it would be a bit weird to use it as REPL since we don't know which kernel to connect without sync with jupynium. And jupynium already work as a better REPL.

Or we can let JupyniumStartAndAttachToServerInTerminal available without syncing/attaching (then it only connects to recent created kernel), and we can use it also a REPL if we cannot access GUI at some moment. But this seems inconsistent with current jupynium workflow

kiyoon · 2023-04-18T07:35:08Z

@sho-87

I think it's because you lazy load on command. However, you don't need to run that to be using this feature. :JupyniumStartAndAttachToServerInTerminal just shows the output logs of Jupynium, and I don't see a good way to combine this two features
I'm open for suggestions. I used the name kernel for consistency, but REPL might be a better name. If we do change the nameit would be JupyniumOpenREPLInTerminal

sho-87 · 2023-04-18T08:14:01Z

I think it's because you lazy load on command. However, you don't need to run that to be using this feature. :JupyniumStartAndAttachToServerInTerminal just shows the output logs of Jupynium, and I don't see a good way to combine this two features

yup, I was just confused about the correct way to test this, but the JupyniumKernelOpenInTerminal by itself was working fine

I'm open for suggestions. I used the name kernel for consistency, but REPL might be a better name. If we do change the nameit would be JupyniumOpenREPLInTerminal

tbh I'd be ok with it either way. I was just trying to create a mental model for myself regarding what the feature would be for and REPL was the first thing that came to mind

fecet · 2023-04-18T11:49:03Z

To make it a real REPL we should also implement a method to send code cell to the terminal, currently it's just a plain terminal

kiyoon · 2023-04-18T12:48:46Z

Okay, I think it's ready and the current name sounds good enough. Thank you @fecet and @sho-87 for this cool feature and testing it!

fecet added 2 commits April 13, 2023 14:18

first draft

60e725b

style

adae64d

kiyoon reviewed Apr 13, 2023

View reviewed changes

src/jupynium/events_control.py Outdated Show resolved Hide resolved

kiyoon reviewed Apr 13, 2023

View reviewed changes

src/jupynium/lua/helpers.lua Outdated Show resolved Hide resolved

rename and use native terminal

7eb3463

fecet force-pushed the feat/connect-kernel branch from f1597a5 to cc86ac8 Compare April 17, 2023 10:10

fecet marked this pull request as ready for review April 17, 2023 10:11

kiyoon reviewed Apr 17, 2023

View reviewed changes

src/jupynium/lua/helpers.lua Show resolved Hide resolved

kiyoon reviewed Apr 17, 2023

View reviewed changes

fecet added 3 commits April 18, 2023 01:47

docs: open kernel in terminal

f176279

renmae to shcmd

de63782

fix js code

29efdd2

fecet force-pushed the feat/connect-kernel branch from b08a36b to 29efdd2 Compare April 17, 2023 17:47

kiyoon merged commit 4f2c1ea into kiyoon:master Apr 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Connect to synchronized kernel #77

feat: Connect to synchronized kernel #77

fecet commented Apr 13, 2023 •

edited

Loading

kiyoon commented Apr 13, 2023 •

edited

Loading

fecet commented Apr 13, 2023 •

edited

Loading

kiyoon commented Apr 13, 2023 •

edited

Loading

fecet commented Apr 14, 2023

kiyoon commented Apr 14, 2023 •

edited

Loading

fecet commented Apr 14, 2023 •

edited

Loading

kiyoon commented Apr 14, 2023

kiyoon commented Apr 17, 2023

fecet commented Apr 17, 2023

kiyoon Apr 17, 2023

fecet Apr 17, 2023

kiyoon Apr 17, 2023 •

edited

Loading

fecet Apr 17, 2023 •

edited

Loading

kiyoon Apr 17, 2023

kiyoon commented Apr 17, 2023

sho-87 commented Apr 18, 2023

fecet commented Apr 18, 2023 •

edited

Loading

kiyoon commented Apr 18, 2023

sho-87 commented Apr 18, 2023

fecet commented Apr 18, 2023

kiyoon commented Apr 18, 2023

feat: Connect to synchronized kernel #77

feat: Connect to synchronized kernel #77

Conversation

fecet commented Apr 13, 2023 • edited Loading

Long-term plan

kiyoon commented Apr 13, 2023 • edited Loading

fecet commented Apr 13, 2023 • edited Loading

kiyoon commented Apr 13, 2023 • edited Loading

fecet commented Apr 14, 2023

kiyoon commented Apr 14, 2023 • edited Loading

fecet commented Apr 14, 2023 • edited Loading

kiyoon commented Apr 14, 2023

kiyoon commented Apr 17, 2023

fecet commented Apr 17, 2023

kiyoon Apr 17, 2023

Choose a reason for hiding this comment

fecet Apr 17, 2023

Choose a reason for hiding this comment

kiyoon Apr 17, 2023 • edited Loading

Choose a reason for hiding this comment

fecet Apr 17, 2023 • edited Loading

Choose a reason for hiding this comment

kiyoon Apr 17, 2023

Choose a reason for hiding this comment

kiyoon commented Apr 17, 2023

sho-87 commented Apr 18, 2023

fecet commented Apr 18, 2023 • edited Loading

kiyoon commented Apr 18, 2023

sho-87 commented Apr 18, 2023

fecet commented Apr 18, 2023

kiyoon commented Apr 18, 2023

fecet commented Apr 13, 2023 •

edited

Loading

kiyoon commented Apr 13, 2023 •

edited

Loading

fecet commented Apr 13, 2023 •

edited

Loading

kiyoon commented Apr 13, 2023 •

edited

Loading

kiyoon commented Apr 14, 2023 •

edited

Loading

fecet commented Apr 14, 2023 •

edited

Loading

kiyoon Apr 17, 2023 •

edited

Loading

fecet Apr 17, 2023 •

edited

Loading

fecet commented Apr 18, 2023 •

edited

Loading