[RAGFlow] RAG Evaluation feature request

[SUNbrightness]

Now the RAG development every phase need to manually test each QA, assess the RAG result, which cost too much. All of these are hard to quantify.

I would like to have a "test evaluation function". Manually maintain multiple test sets containing questions, answers, index_ids, and use NLP or a LLM to do the evaluation, record the document hit rate and the answer correct rate. Having such a metric is very beneficial for RAG tuning and project delivery.
Now Dify, FastGPT, etc. are missing such a feature!

[JinHai-CN]

We intend to create an international community, so we encourage using English for questions and answers to help others with similar queries. 😊 So, I have translated the original text into English, so if anything is wrong or missing, please correct the description.

[JinHai-CN]

Thank you for your advice.

Here are some details that I would like to discuss with you.

1. Which data sets do you suggest? Or what data sets do you typically use to evaluate RAG system?
2. How to assess the quality of data chunking?

[SUNbrightness]

thank you for the response.

1. Our Rag Project require our customer offer a excel which include four colums
   question, answer, ref file, documents fragment
   Now, of course, we do some of the evaluation manually using these QA
2. I'm sure you have more experience with the methods of evaluation, because there are a lot of theoretical methods.
   I have some simple ideas.
   Using the above evaluation data, the index hit rate and answer accuracy are evaluated. This can be done using some of the methods of Nlp assessment or using gpt.

The index hit ratio and answer accuracy are the two metrics that I find most meaningful。