Benchmark Dataset for DevNous: An LLM-Based Multi-Agent System for Grounding IT Project Management in Unstructured Conversation

Overview This dataset contains team conversation data with ground truth annotations for actionable intent tuples. The data were synthetically generated and annotated via human annotators. Dataset Structure The dataset contains 8 individual conversation files, each with 20 messages (160 total messages). Each conversation file represents a separate team conversation session. Additionally the original initial state of tasks and team configuration that was used to generate the dataset is attached. Data Fields Each conversation file contains: conversation_id: Unique identifier for the conversation total_messages: Number of messages in this conversation messages: Array of message entries, each containing: conversation_id: Conversation identifier message_index: Sequential index of the message within the conversation (0-19) input_message: user: Username of the person who sent the message message: The actual text content of the message time: Timestamp when the message was sent ground_truth: Annotated classification in the format [category, action] Classification Categories The dataset uses the following classification scheme: Message Categories: NEW_TASK: Messages that introduce a new task or requirement EXISTING_TASK: Messages related to ongoing tasks or work WORKFLOW_RESPONSE: Messages that respond to or continue workflow processes REGULAR_CONVERSATION: General team communication and social messages SUMMARY_TRIGGER: Messages that request summaries or status updates OTHER: Miscellaneous or unclear message types Actions: create_task: Creation of a new task or work item update_context: Updates the context or information about existing work continue_workflow: Continues or responds to ongoing workflow generate_summary: Triggers summary generation no_action: No specific action required Citation If you use this dataset in your research, please cite the original DevNous paper. This dataset is provided for research and educational purposes.