Add function calling ability to openai extension

[yhyu13]

Summery

This is a experimental implemention of function calling ability for the openai extension.

Essentially, the function calling is simply user adding functions with json format into url request body, and server process these functions into part of system prompt. Either with 1 shot prompting or some sft function calling ability from the base llm, a llm model is able spit out function api name and arguments in json format. Finally, the server post process the llm ouput into url reponse body.

Openai has provide solid reference for api spec, https://platform.openai.com/docs/api-reference/chat/create and cookbook exmaple that I used for testing https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb

I fine tuned a phi2 model https://huggingface.co/Yhyu13/dolphin-2_6-phi-2-sft-glaive-function-calling-v2-ep1 using a curated dataset https://huggingface.co/datasets/Yhyu13/glaive-function-calling-v2-llama-factory-convert?row=2 for this implementation, though I believe sft model is not a pre-prequsite because llm can be 1 or N-shot prompted.

Progress so far

• Support both function calling(deprecated, but some project like memgpt still use it!) and tool calling api for openai chat completion api (shold not support stream though, but I am not sure)
• Implement function calling context that parse user functions and function calls into sysmte prompt,
• Handle function role as part of historical user input
• Post process function calling finish reason
• Very basic run in the above openai cook book example, and it worked (though the response is not as accurate as chatgpt3.5 or above)!

Caveats:

• Only support generating single function calling in a single response so far. Parsing multi function call json str is simply not implemented yet. That is because I have not seen any example in the cookbook where multi function calling in a single response, though the openai response format implies there could be multi function calling (as it uses a list to store every function calling response)
• I personally use a xml wrapper for <functioncall> and for function role, I used <functionresponse> as indicators. Those indicators I also used in training sft model for function calling. THIS IS NOT UNIVERSAL, and base models that have not exposed with glaive-function-calling-v2 might not follow it even with 1/N shot prompting
• Stream should be set to False in the function call request

TODO:

• Test 1 shot prompt for models w/o function calling sft, I will do SOLAR-instruct and Mistral
• Identify edges cases from calling openai api (Multi function calling?, Escape crtl char in json str)
• Parallel function calling (i.e. calling multiple function in single reply)
• Do more openai cook book function call example and idenitfy bugs

Checklist:

• I have read the Contributing guidelines.

[yhyu13]

As an example, here is openai extension debug output from console

<|im_start|>system
Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.You are given access to the following functions, use them if required -
{'name': 'get_current_weather', 'description': 'Get the current weather', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA'}, 'format': {'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The temperature unit to use. Infer this from the users location.'}}, 'required': ['location', 'format']}}

{'name': 'get_n_day_weather_forecast', 'description': 'Get an N-day weather forecast', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA'}, 'format': {'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The temperature unit to use. Infer this from the users location.'}, 'num_days': {'type': 'integer', 'description': 'The number of days to forecast'}}, 'required': ['location', 'format', 'num_days']}}
If you find it necessary to call function, you must reply in the format only when necessary: <functioncall> json_str </functioncall>, e.g <functioncall> {"name": "calculate_loan_payment", "arguments": '{"principal": 50000, "interest_rate": 5, "loan_term": 10}'} </functioncall>.<|im_end|>
<|im_start|>user
What's the weather like today<|im_end|>
<|im_start|>assistant
I'm sorry, but I can't provide the current weather. The function `get_current_weather` requires me to know the location and current format of the temperature which is not current in your function call. Please provide more information if possible.<|im_end|>
<|im_start|>user
I'm in Glasgow, Scotland.<|im_end|>
<|im_start|>assistant

--------------------

Output generated in 2.01 seconds (17.91 tokens/s, 36 tokens, context 485, seed 1282986997)
process_finish_msg Try match '<functioncall>(.*?)</functioncall>' from llm reply '<functioncall> {"name": "get_current_weather", "arguments": '{"location": "Glasgow, Scotland"}'} </functioncall>'
process_finish_msg response:
{
    "id": "call_aeq9go1d65oi92tmaqe2777h",
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "arguments": "{\"location\": \"Glasgow, Scotland\"}"
    }
}

For the cookbook blocks in the above link

As you can see, the sft phi-2 missed required argument format in the replying function call. But the function calling ability is there.

chatgpt3.5 cookbook block for reference

[oobabooga]

Could you

1. Move the new code to a new file (maybe extensions/openai/function_calling.py)
2. Create a very simple example using a curl command for me to put in the documentation

After these the PR can be merged when you say it's ready.

[yhyu13]

@oobabooga

2, For the curl cmd, I am following this stackoverflow answer for windows https://stackoverflow.com/a/7173621

a, You would need to create a body.json in the cwd

{"model": "gpt-3.5-turbo-0613", "messages": [{"role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."}, {"role": "user", "content": "What's the weather like today"}], "tools": [{"type": "function", "function": {"name": "get_current_weather", "description": "Get the current weather", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "format": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The temperature unit to use. Infer this from the users location."}}, "required": ["location", "format"]}}}, {"type": "function", "function": {"name": "get_n_day_weather_forecast", "description": "Get an N-day weather forecast", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "format": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The temperature unit to use. Infer this from the users location."}, "num_days": {"type": "integer", "description": "The number of days to forecast"}}, "required": ["location", "format", "num_days"]}}}]}

b, curl cmd would be

curl --request POST --url http://127.0.0.1:5000/v1/chat/completions --header "Content-Type: application/json" --data "@body.json"

c, and the functioncall response

{"id":"chatcmpl-1704901878340349696","object":"chat.completions","created":1704901878,"model":"dolphin-2_6-phi-2-sft-glaive-function-calling-v2-ep1","choices":[{"index":0,"finish_reason":"tool_calls","message":{"role":"assistant","content":"","tool_calls":["{\n    \"id\": \"call_iand1aym1txbyjtxsqv4k4z6\",\n    \"type\": \"function\",\n    \"function\": {\n        \"name\": \"get_current_weather\",\n        \"arguments\": \"{\\\"location\\\": \\\"San Francisco, CA\\\"}\"\n    }\n}"]}}],"usage":{"prompt_tokens":417,"completion_tokens":34,"total_tokens":451}}

Edit:
Example added to openai doc md in 5e98683

[yhyu13]

Update:

I've added few shot example in the commit 5e98683 for llm that are not sft on function calling, and turns out it does well for one of my favorite model https://huggingface.co/Yhyu13/LMCocktail-10.7B-v1

Here is the output from the openai cook book, where LMCocktail-10.7B-v1 behave the same as chatgpt3.5, by asking x number of days for whether report before performing function calling for the uer

Actually here is the full response that I copied from webui terminal log. It's quite intersting that LMCocktail-10.7B actually spit out contextual infor described in the few shot example

To get a 5-day weather forecast for Glasgow, Scotland, I will now call the "get_n_day_weather_forecast" function with the required parameters:

<functioncall>
{
  "name": "get_n_day_weather_forecast",
  "arguments": {
   "location": "Glasgow, Scotland",
   "format": "celsius",
   "num_days": 5
  }
}
</functioncall>

Please wait while I retrieve the weather data. As an AI, I can't give you the forecast immediately, but once you see a <functionresponse> tag in the response, it will contain the requested weather information.'

Since it contains <functioncall> wrapper, I'm able to interpret the llm output as a functino calling.

[yhyu13]

@oobabooga All tasks should be done now, let me know if you have more questions before mr

Edit:
Still working on a some edge cases on json escaping control characters, hang onFixed 4bcbcd9
Edit2:
Found another bug that assitant fuction call history message is not wrapper by <functioncall> Fixed 4f27ca4

[yhyu13]

@oobabooga
(Maybe final) Update on summrize caveats: also mentioned in docs/12 - OpenAI API.md the function call section

• Function calling success rate is not 100%, it's totally up to the llm model to decide calling a function or not. Models fine tuned on function calling would expect to performane better.

• Parallel function calling is supported, we can parse multiple function call calls in a single llm reply, but again, it's up to the llm model to decide how many function to call.

• Instead of compliance with openai function call specification where the response content is None, we deliberately fill content with function called wrapped in a internal syntax. Unlike what demonstrated in the openai cookbook, users are not expected to change content in order to continue the conversation. So no longer assistant_message['content'] = str(assistant_message["tool_calls"][0]["function"]) on user side, just leave the heavy work to webui.
  Here is an example for this on modifying the openai cook funtion call
  Before
  
  After, you should leave the content of webui response alone, the webui has handled it already, whereas official openai response set it to None
  

• There still exists edge cases where the llm model output function call json message that contain control character (e.g. \n \t) that fails the json.loads() method. If that happen, user should try json.loads(..., strict=False) to bypass. The openai cookbook simply assume all function call json returned are valid, we do not gurantee that since it is up to the llm model to spit out the function call message.

[teddybear082]

This will be amazing, looking forward to trying it! Open source community needs function calling llms!

[AiratGaliev]

Hello, community!
There is a project in which there is a full-fledged alternative to function calling from openai https://github.com/MeetKai/functionary?tab=readme-ov-file#the-differences-between-related-projects
And here https://llama-cpp-python.readthedocs.io/en/latest/server/ is an example of running an older version of this model functionary-7b-v1
It would be great to update this old implementation

[Katehuuh]

@yhyu13 In documentation, please use a single curl line instead of vim to edit vim body.json. Also can you show module openai (openai cook funtion call does not work for me) with for

base example i used phi-2 ...with work 1/10

#> Get me news from china
# Function: get_news_headlines
# Arguments: {'country': 'China'}
#> <functionresponse>News from China think function calling is cool!</functionresponse>

import requests
import re
import ast

def get_news_headlines(country):
    json_str = "News from " + country + " think function calling is cool!"
    return json_str

def function_handler(assistant_message):
    pattern = r'<functioncall>(.*?)</functioncall>'
    match = re.search(pattern, assistant_message, re.DOTALL)

    if match:
        json_str = match.group(1)
        json_str = json_str.strip()
        json_dict = ast.literal_eval(json_str)
        print("Function:", json_dict['name'])
        arguments = ast.literal_eval(json_dict['arguments'])
        print("Arguments:", arguments)

        # Call the placeholder function if the function name matches
        if json_dict['name'] == 'get_news_headlines':
            country = arguments['country']
            result = get_news_headlines(country)
            return f'<functionresponse>{result}</functionresponse>', True
    else:
        return assistant_message, False

url = "http://127.0.0.1:5000/v1/chat/completions"

headers = {
    "Content-Type": "application/json"
}

history = [
    {"role": "user", "content": "You are a helpful assistant with access to the following functions. Use them if required -\n{\n \"name\": \"get_news_headlines\",\n \"description\": \"Get the latest news headlines\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"country\": {\n \"type\": \"string\",\n \"description\": \"The country for which to fetch news\"\n }\n },\n \"required\": [\n \"country\"\n ]\n }\n}"}
]

return_function = False

while True:
    if not return_function:
        user_message = input("> ")
    else:
        user_message = handled_message + " Please report this to the user."
        return_function = False
    history.append({"role": "user", "content": user_message})
    data = {
        "mode": "chat",
        "character": "Example",
        "messages": history
    }

    response = requests.post(url, headers=headers, json=data, verify=False)
    assistant_message = response.json()['choices'][0]['message']['content']
    history.append({"role": "assistant", "content": assistant_message})

    # Handle the assistant's message with the function handler
    handled_message, return_function = function_handler(assistant_message)
    print(handled_message)

[yhyu13]

@yhyu13 In documentation, please use a single curl line instead of vim to edit vim body.json. Also can you show module openai (openai cook funtion call does not work for me) with for

base example i used phi-2 ...with work 1/10

@Katehuuh
1, A single curl call would not work unfortunatelly, as the quotes escaping worksly really wiredly.
2, I am not sure if you have provided 'tools' or 'fucntions' (deprecated, but also supported by this pr) in the posted request? IMO, phi-2 might not work ideally w/o sft on function-calling, that's why I prepared this variant of phi-2 https://huggingface.co/Yhyu13/dolphin-2_6-phi-2-sft-glaive-function-calling-v2-ep1. The most ideal model that I trained so far is https://huggingface.co/Yhyu13/dolphin-2.6-mistral-7b-dpo-laser-function-calling.
3, For the open ai cookbook to work, one extra thing is to set openai.base_url = http://127.0.0.1:5000/v1. Do you need any other assiatant on running the openai cookbook?

[Katehuuh]

variant of phi-2 https://huggingface.co/Yhyu13/dolphin-2_6-phi-2-sft-glaive-function-calling-v2-ep1.

Yes i used the same finetuned variant dolphin-2_6-phi-2.

Can you provide a simple python code such as show in documentation?

[yhyu13]

variant of phi-2 https://huggingface.co/Yhyu13/dolphin-2_6-phi-2-sft-glaive-function-calling-v2-ep1.

Yes i used the same finetuned variant dolphin-2_6-phi-2.

Can you provide a simple python code such as show in documentation?

@Katehuuh
Ok, I forgot how phi2 sucks at function calling, it does fail miserbally from my latest local run on the openai cook book. Another model that I trained Yhyu13/dolphin-2.6-mistral-7b-dpo-laser-function-calling would do a better job. The original version of olphin-2.6-mistral-7b-dpo is good enough, I couldn't tell much difference.

Below is the code scrapped from openai cook book

import json
import openai
import requests
from tenacity import retry, wait_random_exponential, stop_after_attempt
from termcolor import colored

GPT_MODEL = "gpt-3.5-turbo-0613"
openai.api_key = "sk-1111"

@retry(wait=wait_random_exponential(multiplier=1, max=40), stop=stop_after_attempt(3))
def chat_completion_request(messages, tools=None, tool_choice=None, model=GPT_MODEL):
    headers = {
        "Content-Type": "application/json",
        "Authorization": "Bearer " + openai.api_key,
    }
    json_data = {"model": model, "messages": messages}
    if tools is not None:
        json_data.update({"tools": tools})
    if tool_choice is not None:
        json_data.update({"tool_choice": tool_choice})
    try:
        response = requests.post(
            "http://127.0.0.1:5051/v1/chat/completions",
            headers=headers,
            json=json_data,
        )
        return response
    except Exception as e:
        print("Unable to generate ChatCompletion response")
        print(f"Exception: {e}")
        return e


tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    },
                    "format": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "The temperature unit to use. Infer this from the users location.",
                    },
                },
                "required": ["location", "format"],
            },
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_n_day_weather_forecast",
            "description": "Get an N-day weather forecast",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    },
                    "format": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "The temperature unit to use. Infer this from the users location.",
                    },
                    "num_days": {
                        "type": "integer",
                        "description": "The number of days to forecast",
                    }
                },
                "required": ["location", "format", "num_days"]
            },
        }
    },
]

messages = []
messages.append({"role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."})
messages.append({"role": "user", "content": "What's the weather like today"})
chat_response = chat_completion_request(
    messages, tools=tools
)
assistant_message = chat_response.json()["choices"][0]["message"]
messages.append(assistant_message)
print(assistant_message)

[Katehuuh]

Below is the code scrapped from openai cook book...

In, I replace /127.0.0.1:5051/ by /127.0.0.1:5000/ instead of

For the open ai cookbook to work, one extra thing is to set openai.base_url = "http://127.0.0.1:5000/v1".

This return {'role': 'assistant', 'content': "I'm sorry, but I don't have access to the current weather."} for phi2 sft.

curl in single line, base of last doc.

curl --request POST --url http://127.0.0.1:5000/v1/chat/completions --header "Content-Type: application/json" --data "{\"model\": \"gpt-3.5-turbo-0613\", \"messages\": [{\"role\": \"system\", \"content\": \"Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\"}, {\"role\": \"user\", \"content\": \"What's the weather like today for San Francisco\"}], \"tools\": [{\"type\": \"function\", \"function\": {\"name\": \"get_current_weather\", \"description\": \"Get the current weather\", \"parameters\": {\"type\": \"object\", \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The city and state, e.g. San Francisco, CA\"}, \"format\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"], \"description\": \"The temperature unit to use. Infer this from the users location.\"}}, \"required\": [\"location\", \"format\"]}}}, {\"type\": \"function\", \"function\": {\"name\": \"get_n_day_weather_forecast\", \"description\": \"Get an N-day weather forecast\", \"parameters\": {\"type\": \"object\", \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The city and state, e.g. San Francisco, CA\"}, \"format\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"], \"description\": \"The temperature unit to use. Infer this from the users location.\"}, \"num_days\": {\"type\": \"integer\", \"description\": \"The number of days to forecast\"}}, \"required\": [\"location\", \"format\", \"num_days\"]}}}]}"

Return:

{"id":"chatcmpl-1709437719252631040","object":"chat.completions","created":1709437719,"model":"dolphin-2_6-phi-2-sft-glaive-function-calling-v2-ep1","choices":[{"index":0,"finish_reason":"tool_calls","message":{"role":"assistant","content":"<functioncall> {'name': 'get_current_weather', 'arguments': '{\"location\": \"San Francisco, CA\"}'} </functioncall>","tool_calls":[{"id":"call_tgs4qp6lyxru3azb2ntmdpge","type":"function","function":{"name":"get_current_weather","arguments":"{\"location\": \"San Francisco, CA\"}"}}]}}],"usage":{"prompt_tokens":1474,"completion_tokens":34,"total_tokens":1508}}

same phi "model" no sft:

{"id":"chatcmpl-1709438463586460416","object":"chat.completions","created":1709438463,"model":"dolphin-2_6-phi-2.Q4_K_M.gguf","choices":[{"index":0,"finish_reason":"tool_calls","message":{"role":"assistant","content":"<functioncall> {'name': 'get_current_weather', 'arguments': '{\"location\": \"San Francisco, CA\"}'} </functioncall>","tool_calls":[{"id":"call_34tf3i6n084pfr9klozkpq3f","type":"function","function":{"name":"get_current_weather","arguments":"{\"location\": \"San Francisco, CA\"}"}}]}}],"usage":{"prompt_tokens":1536,"completion_tokens":75,"total_tokens":1611}}

Some of the simple api return JSONDecodeError

text-generation-webui/docs/12 - OpenAI API.md

Lines 32 to 44 in ba85271

	#### Completions
	```shell
	curl http://127.0.0.1:5000/v1/completions \
	-H "Content-Type: application/json" \
	-d '{
	"prompt": "This is a cake recipe:\n\n1.",
	"max_tokens": 200,
	"temperature": 1,
	"top_p": 0.9,
	"seed": 10
	}'
	```

Or

import requests
import json
url = "http://127.0.0.1:5000/v1/completions"
headers = {
    "Content-Type": "application/json"
}
data = {
    "prompt": "This is a cake recipe:\n\n1.",
    "max_tokens": 200,
    "temperature": 1,
    "top_p": 0.9,
    "seed": 10
}
response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())

[yhyu13]

@Katehuuh

1, For you first case where phi-2 failed, larger models CodeBooga-34B would succeed (even w/o fine tuning)

{"id":"chatcmpl-1709622264268705792","object":"chat.completions","created":1709622264,"model":"CodeBooga-34B-v0.1-4.0bpw-h6-exl2","choices":[{"index":0,"finish_reason":"tool_calls","message":{"role":"assistant","content":"<functioncall> {'name': 'get_current_weather', 'arguments': '{\"location\": \"San Francisco, CA\", \"format\": \"fahrenheit\"}'} </functioncall>","tool_calls":[{"id":"call_77spvhlc1kxo7o0oe98ns9fm","type":"function","function":{"name":"get_current_weather","arguments":"{\"location\": \"San Francisco, CA\", \"format\": \"fahrenheit\"}"}}]}}],"usage":{"prompt_tokens":1537,"completion_tokens":43,"total_tokens":1580}}

I am not sure at moment how to improve phi2, I will try other models with 2B size

2, Thans for exposing the bug on using completion api, I fix it in 19bc209

[Katehuuh]

I am not sure at moment how to improve phi2, I will try other models with 2B size

Comparing to “old dolphin-2_6-phi-2”, new phi sft model show promising score on YALL Leaderboard like phi-2-orange-v2.

[Wladastic]

I have to push this up a bit with another comment.
Llama 3 8b variants with high context work very well as well as codeqwen 7b and should be considered.
phi 3 is not usable for json at all still.

[zhenweiding]

https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3#function-calling

[teddybear082]

I found mistral 7b instruct v0.3 to be very good too. I wonder if this can ever be committed?

[yhyu13]

If some one still looking for a out of the box function calling soloution locally, here it is : https://github.com/MeetKai/functionary with campion model https://huggingface.co/meetkai/functionary-small-v2.5 which works like a charm!

With the following setup,

python3 server_vllm.py --model "meetkai/functionary-small-v2.5" --host 127.0.0.1 --port 5000 --max-model-len 8192 --served-model-name "chat-gpt-1106"

it manage to cut throw the openai function calling cookbook like buffer (100% success rate and lightning fast)