RAG-Instruct-Benchmark-Tester

This is an updated benchmarking test dataset for "retrieval augmented generation" (RAG) use cases in the enterprise, especially for financial services, and legal. This test dataset includes 200 questions with context passages pulled from common 'retrieval scenarios', e.g., financial news, earnings releases, contracts, invoices, technical articles, general news and short texts.
The questions are segmented into several categories for benchmarking evaluation:
Core Q&A Evaluation (Samples 0-99) - 100 samples - fact-based 'core' questions- used to assign a score between 0-100 based on correct responses.
Not Found Classification (Samples 100-119) - 20 samples - in each sample, the context passage does not contain a direct answer to the question, and the objective is to evaluate whether the model correctly identifies as "Not Found" or attempts to answer using information in the context.
Boolean - Yes/No (Samples 120-139) - 20 samples - each sample is a Yes/No question.
Basic Math (Samples 140-159) - 20 samples - these are "every day" math questions - basic increments, decrements, percentages, multiplications, sorting, and ranking with amounts and times.
Complex Q&A (Samples 160-179) - 20 samples - tests several distinct 'complex q&a' skills - multiple-choice, financial table reading, multi-part extractions, causal, and logical selections.
Summary (Sample 180-199) - 20 samples - tests long-form and short-form summarization.