PostgreSQL Error while using GPT-4 Turbo endpoint on Azure

[plopezamaya]

On version v0.3.75

When using gpt4-turbo on azure. The partial stream takes several minutes (azure endpoint issue).

Because of this 👆 it seems that the connexion to postgresql is closed when trying to insert the answer :

05/30/2024 09:08:24 AM   process_message.py 511 : (psycopg2.OperationalError) SSL connection has been closed unexpectedly
[SQL: INSERT INTO chat_message (chat_session_id, parent_message, latest_child_message, message, rephrased_query, prompt_id, token_count, message_type, citations, files, error) VALUES (%(chat_session_id)s, %(parent_message)s, %(latest_child_message)s, %(message)s, %(rephrased_query)s, %(prompt_id)s, %(token_count)s, %(message_type)s, %(citations)s, %(files)s, %(error)s) RETURNING chat_message.id, chat_message.time_sent]
[parameters: {'chat_session_id': 439, 'parent_message': 753, 'latest_child_message': None, 'message': "It appears you've gathered detailed feedback from a survey assessing satisfaction rates on various search terms related to online shopping for home a ... (14 characters truncated) ... lies, tools, and more. The rates range from 1 (lowest satisfaction) to 5 (highest satisfaction). \n\nHere's a summary of key observations:\n1. **High", 'rephrased_query': None, 'prompt_id': 167, 'token_count': 62, 'message_type': 'ASSISTANT', 'citations': 'null', 'files': 'null', 'error': None}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1968, in _exec_single_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 920, in do_execute
    cursor.execute(statement, parameters)
psycopg2.OperationalError: SSL connection has been closed unexpectedly
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/app/danswer/chat/process_message.py", line 493, in stream_chat_message_objects
    gen_ai_response_message = partial_response(
                              ^^^^^^^^^^^^^^^^^
  File "/app/danswer/db/chat.py", line 292, in create_new_chat_message
    db_session.flush()
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4154, in flush
    self._flush(objects)
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4290, in _flush
    with util.safe_reraise():
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4251, in _flush
    flush_context.execute()
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 467, in execute
    rec.execute(self)
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 644, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
    _emit_insert_statements(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 1223, in _emit_insert_statements
    result = connection.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1413, in execute
    return meth(
           ^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1637, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1987, in _exec_single_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2344, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1968, in _exec_single_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 920, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL connection has been closed unexpectedly
[SQL: INSERT INTO chat_message (chat_session_id, parent_message, latest_child_message, message, rephrased_query, prompt_id, token_count, message_type, citations, files, error) VALUES (%(chat_session_id)s, %(parent_message)s, %(latest_child_message)s, %(message)s, %(rephrased_query)s, %(prompt_id)s, %(token_count)s, %(message_type)s, %(citations)s, %(files)s, %(error)s) RETURNING chat_message.id, chat_message.time_sent]
[parameters: {'chat_session_id': 439, 'parent_message': 753, 'latest_child_message': None, 'message': "It appears you've gathered detailed feedback from a survey assessing satisfaction rates on various search terms related to online shopping for home a ... (14 characters truncated) ... lies, tools, and more. The rates range from 1 (lowest satisfaction) to 5 (highest satisfaction). \n\nHere's a summary of key observations:\n1. **High", 'rephrased_query': None, 'prompt_id': 167, 'token_count': 62, 'message_type': 'ASSISTANT', 'citations': 'null', 'files': 'null', 'error': None}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
05/30/2024 09:08:24 AM            timing.py  74 : stream_chat_message took 104.46927809715271 seconds
05/30/2024 09:08:24 AM      chat_backend.py 174 : Received rename request for chat session: 439
05/30/2024 09:08:25 AM chat_session_naming.py  47 : New Session Name: Search Relevancy Survey Results
INFO:     127.0.0.6:45691 - "PUT /chat/rename-chat-session HTTP/1.1" 200 OK
INFO:     127.0.0.1:46814 - "GET /auth/type HTTP/1.1" 200 OK
INFO:     127.0.0.1:46822 - "GET /manage/me HTTP/1.1" 200 OK
INFO:     127.0.0.1:46868 - "GET /chat/get-user-chat-sessions HTTP/1.1" 200 OK

The UI shows the following error Failed to parse LLM outputand deletes the LLM response.

For the moment one quick fix could be done on the backend/danswer/chat/process_message.py file that only queries the database with a build in function that you implemented :

for packet in answer.processed_streamed_output:
            # Keeping alive session for long running LLM calls
            _ = get_db_current_time(db_session)

If that suits you well we can create a PR.