[Bug]: KnowledgeGraphIndex.from_documents ignoring set llm

[mackenziedott]

Bug Description

While attempting to load documents into a KnowledgeGraphIndex, i keep timing out on _openAI completion call, Checking my path to the OpenAILike and running a different method, it seems to be executing just fine otherwise. I've set my OpenAIKey to 'fake' as I don't want any communication to OpenAI. I'm unsure what could be causing this issue as my normal RAG pipeline using the same model is working fine.

Version

0.10.50

Steps to Reproduce

from llama_index.llms.openai_like import OpenAILike
llmProp = configs['llm_properties']
  
print(configs['llm'])
llm = OpenAILike(
              
                model=llmProp['model'],
                api_base = configs['llm'].replace('/completions', ''),
                api_key="fake",
                max_tokens=llmProp['max_tokens'],
                logprobs=llmProp['logprobs'],
                temperature=llmProp['temperature'],
                top_p=llmProp['top_p'],
                top_k=llmProp['top_k'],
                additional_kwargs = {'echo':llmProp['echo']},
                )
Settings.llm = llm
graph_store = Neo4jGraphStore(
    username=username,
    password=password,
    url=url,
    database=database,
)
storage_context = StorageContext.from_defaults(graph_store=graph_store)

reader = SimpleDirectoryReader(
   input_files=pdfs
)
Settings.chunk_overlap=50
Settings.chunk_size=512
docs = reader.load_data(show_progress=True, )


index = KnowledgeGraphIndex.from_documents(
    docs,
    storage_context=storage_context,
    llm=llm,
    max_triplets_per_chunk=10
)

Relevant Logs/Tracbacks

INFO:openai._base_client:Retrying request to /completions in 0.802545 seconds
Retrying request to /completions in 0.802545 seconds
INFO:openai._base_client:Retrying request to /completions in 1.782454 seconds
Retrying request to /completions in 1.782454 seconds
INFO:openai._base_client:Retrying request to /completions in 3.479021 seconds
Retrying request to /completions in 3.479021 seconds
WARNING:llama_index.llms.openai.utils:Retrying llama_index.llms.openai.base.OpenAI._complete in 0.25277889074673177 seconds as it raised APIConnectionError: Connection error..
Retrying llama_index.llms.openai.base.OpenAI._complete in 0.25277889074673177 seconds as it raised APIConnectionError: Connection error..
INFO:openai._base_client:Retrying request to /completions in 0.799029 seconds
Retrying request to /completions in 0.799029 seconds
INFO:openai._base_client:Retrying request to /completions in 1.908156 seconds
Retrying request to /completions in 1.908156 seconds
INFO:openai._base_client:Retrying request to /completions in 3.990974 seconds
Retrying request to /completions in 3.990974 seconds
WARNING:llama_index.llms.openai.utils:Retrying llama_index.llms.openai.base.OpenAI._complete in 1.2667151135404542 seconds as it raised APIConnectionError: Connection error..
Retrying llama_index.llms.openai.base.OpenAI._complete in 1.2667151135404542 seconds as it raised APIConnectionError: Connection error..
INFO:openai._base_client:Retrying request to /completions in 0.881817 seconds
Retrying request to /completions in 0.881817 seconds
INFO:openai._base_client:Retrying request to /completions in 1.745545 seconds
Retrying request to /completions in 1.745545 seconds
INFO:openai._base_client:Retrying request to /completions in 3.904172 seconds
Retrying request to /completions in 3.904172 seconds

[dosubot]

Hey @mackenziedott, I'm here to help you with any issues you're facing. I can assist with bug resolutions, answer your questions, and guide you on becoming a contributor. Let's tackle this together!

You are experiencing timeouts on the _openAI completion call because the fake API key you are using is not valid. The OpenAI API cannot authenticate the request with a fake key, resulting in the API not responding and leading to a timeout.

The CachedOpenAIApiKeys class sets a fake key ("sk-" + "a" * 48) when set_fake_key is True, which is not a valid API key. Any attempt to make a completion call with this fake key will fail, causing the retries and eventual timeouts you are seeing.

Here is a relevant code snippet that demonstrates this behavior:

class CachedOpenAIApiKeys:
    ...
    def __enter__(self) -> None:
        ...
        if self.set_fake_key:
            os.environ["OPENAI_API_KEY"] = "sk-" + "a" * 48

Additionally, the test_completion_model_with_retry function shows that the OpenAI class will retry the completion call up to the specified max_retries if it encounters an openai.APITimeoutError:

@patch("llama_index.llms.openai.base.SyncOpenAI")
def test_completion_model_with_retry(MockSyncOpenAI: MagicMock) -> None:
    mock_instance = MockSyncOpenAI.return_value
    mock_instance.completions.create.side_effect = openai.APITimeoutError(None)

    llm = OpenAI(model="text-davinci-003", max_retries=3)
    prompt = "test prompt"
    with pytest.raises(openai.APITimeoutError) as exc:
        llm.complete(prompt)

    assert exc.value.message == "Request timed out."
    assert mock_instance.completions.create.call_count == 3

To resolve this issue, you need to use a valid OpenAI API key. The fake key will not work for making actual API calls and will always result in timeouts and retries ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[mackenziedott]

@dosu

The issue is that I've set my LLM and Embedmodel to models that don't require an OpenAI key

[dosubot]

To resolve the timeouts and retries occurring during the _openAI completion call when loading documents into a KnowledgeGraphIndex using models that don't require an OpenAI key, you can use alternative models such as Konko, which do not rely on OpenAI's API. Here is an example of how you can use the Konko API to avoid using the OpenAI API:

import logging
from unittest.mock import patch
from llama_index.llms.konko import Konko
from llama_index.core.base.llms.types import ChatMessage
from llama_index.legacy.llms.konko_utils import completion_with_retry, acompletion_with_retry

# Configure logging
logger = logging.getLogger(__name__)

# Mock the completion call to avoid using OpenAI's API
@patch("llama_index.llms.konko.base.SyncKonko")
def test_completion_model_basic(MockSyncKonko):
    mock_instance = MockSyncKonko.return_value
    mock_instance.completions.create.return_value = {
        "id": "cmpl-uqkvlQyYK7bGYrRHQ0eXlWi7",
        "object": "text_completion",
        "created": 1589478378,
        "model": "text-davinci-003",
        "choices": [
            {
                "text": "\n\nThis is indeed a test",
                "index": 0,
                "logprobs": None,
                "finish_reason": "length",
            }
        ],
        "usage": {"prompt_tokens": 5, "completion_tokens": 7, "total_tokens": 12},
    }

    llm = Konko(model="text-davinci-003")
    prompt = "test prompt"
    message = ChatMessage(role="user", content="test message")

    response = completion_with_retry(is_chat_model=False, max_retries=3, prompt=prompt)
    assert response["choices"][0]["text"] == "\n\nThis is indeed a test"

    chat_response = completion_with_retry(is_chat_model=True, max_retries=3, messages=[message])
    assert chat_response["choices"][0]["message"]["content"] == "\n\nThis is indeed a test"

# Run the test
test_completion_model_basic()

This code uses the unittest.mock.patch decorator to replace the actual Konko completion call with a mock function that returns a predefined response. Additionally, it uses the completion_with_retry function to handle retries, avoiding timeouts and retries associated with the OpenAI API ^[1][2].

To continue talking to Dosu, mention @dosu.

[mackenziedott]

@dosu
Other functions (such as query_engine) that are set-up similarly do not suffer this issue

[dosubot]

To resolve the timeouts on the _openAI completion call when loading documents into a KnowledgeGraphIndex using models that don't require an OpenAI key, you can use the retry mechanism provided by the tenacity library. This approach will help handle various API errors and retry the request automatically.

Here's how you can implement the retry mechanism:

from llama_index.legacy.llms.konko_utils import completion_with_retry

# Define the parameters for the completion call
is_chat_model = False  # Set to True if using a chat model
max_retries = 5  # Number of retries

# Define the kwargs for the completion call
kwargs = {
    # Add your specific parameters here
}

# Use the completion_with_retry function to handle retries
result = completion_with_retry(is_chat_model, max_retries, **kwargs)

# Use the result to load documents into the KnowledgeGraphIndex
# Example:
# knowledge_graph_index.load_documents(result, documents)

By using the completion_with_retry function, you can handle timeouts and other API errors more gracefully, reducing the likelihood of timeouts occurring during the document loading process ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[logan-markewich]

@mackenziedott you are using openai-like, which uses the openai client 🤔 so the logs about using openai are kind of expected?

I'm not sure what the issue is here, but you can check the llm attached to the index with print(index._llm). This is the llm used for extracting triplets

[mackenziedott]

@dosu
I want to try something. How do i turn off async document loading?

[mackenziedott]

@mackenziedott you are using openai-like, which uses the openai client 🤔 so the logs about using openai are kind of expected?

I'm not sure what the issue is here, but you can check the llm attached to the index with print(index._llm). This is the llm used for extracting triplets

Thanks! I think I might know the issue, I wonder if i'm overloading my API with aysnc calls. Its weird, because i can call the endpoints with other functions (standard RAG completion) just fine. I experimented with the PropertyGraphIndex, which i run into the same issue after it parses the nodes and starts attempting to extract from text, it can do up to 6 before running into the same issue.