New rag agent

[thinkall]

Why are these changes needed?

Refactor RAG agents. In this PR, only core functionalities are added. After we agree on the new architecture, enhancements could be added in other PRs.

Example Code:

import autogen
from autogen.agentchat.contrib.rag import RagAgent, logger
import logging

logger.setLevel(logging.DEBUG)

config_list = autogen.config_list_from_json(
    "OAI_CONFIG_LIST",
    file_location=".",
    filter_dict={
        "model": ["gpt-3.5-turbo", "gpt-35-turbo", "gpt-35-turbo-0613", "gpt-4", "gpt4", "gpt-4-32k"],
    },
)

print("LLM models: ", [config_list[i]["model"] for i in range(len(config_list))])

llm_config = {
    "timeout": 60,
    "config_list": config_list,
}


def termination_msg(x):
    return isinstance(x, dict) and "TERMINATE" == str(x.get("content", ""))[-9:].upper()


userproxy = autogen.UserProxyAgent(
    name="userproxy",
    is_termination_msg=termination_msg,
    human_input_mode="ALWAYS",
    code_execution_config={"use_docker": False, "work_dir": ".tmp"},
    default_auto_reply="Reply `TERMINATE` if the task is done.",
    description="The boss who ask questions and give tasks.",
)


rag_config = {
    "docs_path": "./website/docs",
}

rag = RagAgent(
    name="rag",
    is_termination_msg=termination_msg,
    human_input_mode="NEVER",
    max_consecutive_auto_reply=5,
    llm_config=llm_config,
    rag_config=rag_config,
    code_execution_config=False,
    description="Assistant who has extra content retrieval power for solving difficult problems.",
)

userproxy.initiate_chat(recipient=rag, message="What is AutoGen?")

Related issue number

Closes #1726

Tasks

• Core functionalities
• Notebook example
• RAG capability
• unstructured

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[jackgerrits]

The front_matter needs to be moved to the notebook metadata instead of comment in the notebook. See here: https://github.com/microsoft/autogen/blob/main/notebook/contributing.md