Custom Api Url?

[Andoramb]

Hi!

Would it be possible to provide custom API url when setting up the node?
This could potentially play along when running local llm models, such as text-generation-webui-docker or LM Studio.

Then tokens don't really matter.

Thanks :)

[allanbunch]

Hey there! Thanks for the suggestion.

I'll look into this. My initial thought is that this would take the project in a new direction--which I am very much open to as it evolves. As things currently exist, the node is quite specific to OpenAI's REST API endpoints, request paths, and payload, so we'd be looking at more than simply providing an option for a custom API URL in the node options.

Be sure to update to the latest version of the node if you haven't done so in the past day. I fixed several initial bugs with the OpenAI Assistants requests.

Cheers!

[Andoramb]

Sure, great initiative!👍
I am experimenting with running LLM locally with text-generation-webui.
And for what I've seen, the API implementation is taken from OpenAI API, a few examples here
Payload, requests look all similar 😃

So at a first try, I just wanted to put these 2 together, by changing the base url.

I probably could do the changes locally for me, but wanted to know your opinion and do a collab 😄

And as Oobabooga evolves, it might get additional features as well.

[allanbunch]

Got it! Thanks for the details. I ran a few tests with the node's domain pointing to a local Text Generation Web UI (TGWU 😀) instance and things are looking good! I'm able to chat with the zephyr-7B-beta-GGUF model in a flow by injecting an OpenAI API messages payload.

I'll send a note your way after I add the option to to the node and publish a preview branch.

[allanbunch]

@Andoramb

Branch is up! You can install it and give it a go using:

npm install github:allanbunch/node-red-openai-api#feature/custom-service-host

You'll find that the node now has a configurable "Service Host". Let me know how things go for you. Thanks!

[Andoramb]

Awesome! 🎉
I'll check it as soon as I get the chance 🔥

[allanbunch]

Excellent! 🚀
I'll mention that you suggested this feature in my next NPM release if you like. Just let me know!

[Andoramb]

So let's see here:

1. I did not manage to connect it to text-generation-webui, but it's probably something with my setup:
   "POST /v1/chat/completions HTTP/1.1" 404 -
   (I tried with the example for also from chat completions, tried it with postman but still not found)

2. With LM studio it went better:
   
   Chat completion worked as expected, but probably other functions are not implemented yet.
   But it works! 😄 🎉

You don't really have to mention me, you are developing it, I just barely sit in the passenger seat 😄but thanks for asking!

[Andoramb]

I'll try to debug TGW in the meantime

[Andoramb]

*Edit:
Fixed TGW in docker now 👍 (I had to completely recreate the container)

• Chat completions should work (assuming it gets the right payload and configurations)
• Listing the running models also returns the found models

I might have some ideas for the future, but let me know how it works out for you 😀

[allanbunch]

Awesome! Thanks for the update. 😀 I'll check LM Studio too!

Future ideas would be fantastic. Thanks for this one. Cheers!