Orchestrating Multi-Agent Systems for Multi-Source Information Retrieval and Question Answering with Large Language Models

Orchestrating Multi-Agent Systems forMulti-Source Information Retrieval and QuestionAnswering with Large Language ModelsAntony Seabra1,2, Claudio Cavalcante1,2, Joao Nepomuceno1, Lucas Lago1, NicolaasRuberg1, and Sergio Lifschitz21 BNDES - ´Area de Tecnologia da Informa¸c˜ao, Rio de Janeiro, Brazil2 PUC-Rio - Departamento de Inform´atica, Rio de Janeiro, BrazilAbstract.

 We present a novel framework for developing robust multi-source questionanswersystems by dynamically integrating Large Language Models with diverse data sources.This framework leverages a multi-agent architecture to coordinate the retrieval and synthesisof information from unstructured documents, like PDFs, and structured databases. Specializedagents, including SQL agents, Retrieval-Augmented Generation agents, and routeragents, dynamically select and execute the most suitable retrieval strategies for each query.To enhance contextual relevance and accuracy, the framework employs adaptive prompt engineering,fine-tuned to the specific requirements of each interaction. We demonstrate theeffectiveness of this approach in the domain of Contract Management, where answering complexqueries often demands seamless collaboration between structured and unstructured data.The results highlight the framework’s capability to deliver precise, context-aware responses,establishing a scalable solution for multi-domain question-answer applications.

Keywords: Information Retrieval, Question Answer, Large Language Models, Documents,Databases, Prompt Engineering, Retrieval Augmented Generation, Text-to-SQL.