C-PASS: An Organization-centric Framework for Compliance and Penalty Assessment Using Large Language Models

Supplementary Materials – Annotated Corpora and Outputs This package contains the annotated corpora and intermediate outputs associated with the paper submission "C-PASS: An Organization-centric Framework for Compliance and Penalty Assessment Using Large Language Models". Contents Results from each stage of the processing pipeline and the baseline approaches are provided in separate folders: deontic_modality_classification (Stage 1.1)/ Deontic_modality_ground_truth_CDCA: Ground truth deontic modality classification (Obligation, Permission, Prohibition Other) for CDCA. Deontic_modality_ground_truth_EPMA: Ground truth deontic modality classification (Obligation, Permission, Prohibition, Other) for EPMA. Setfit_predictions_CDCA_and_EPMA: Deontic modality classification outputs using Setfit for CDCA and EPMA. definition_retrieval (Stage 1.2)/ definitions_CDCA: Definitions retrieved from 'Other' clauses in CDCA. definitions_EPMA: Definitions retrieved from 'Other' clauses in EPMA. binding_clause_identification (Stage 1.3)/ CDCA_profile_1_output_with_ground_truth: Binding clause identification output and ground truth for CDCA profile 1. CDCA_profile_2_output_with_ground_truth: Binding clause identification output and ground truth for CDCA profile 2. EPMA_profile_1_output_with_ground_truth: Binding clause identification output and ground truth for EPMA profile 1. EPMA_profile_2_output_with_ground_truth: Binding clause identification output and ground truth for EPMA profile 2. compliance_task_generation (Stage 2)/ CDCA_compliance_tasks: Compliance tasks and associated elements from CDCA clauses. CDCA_compliance_tasks_evaluation: "LLM-as-a-Judge" evaluation scores and justifications for CDCA compliance tasks for GPT-4.1. EPMA_compliance_tasks: Compliance tasks and associated elements from EPMA clauses. EPMA_compliance_tasks_evaluation: "LLM-as-a-Judge" evaluation scores and justifications for EPMA compliance tasks for GPT-4.1. per_LLM_evaluation_output_profile_1/: "LLM-as-a-Judge" evaluation scores for profile 1 on CDCA and EPMA for all LLMs (GPT-4.1, GPT-4o, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct). per_LLM_evaluation_output_profile_2/: "LLM-as-a-Judge" evaluation scores for profile 2 on CDCA and EPMA for all LLMs (GPT-4.1, GPT-4o, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct). baseline_approaches/ baseline_deontic_modality_classification (Stage 1.1)/ deontic_modality_classification_baseline: Zero-shot GPT-4.1 results for deontic modality classification for CDCA and EPMA. baseline_binding_clause_identification (Stage 1.3)/ binding_clause_identification_baseline_CDCA: Baseline results for CDCA profile 1 and profile 2. binding_clause_identification_baseline_EPMA: Baseline results for EPMA profile 1 and profile 2. baseline_deontic_modality_classification (Stage 2)/ Profile 1/: Baseline results and "LLM-as-a-Judge" evaluation scores for profile 1 on CDCA and EPMA for all LLMs (GPT-4.1, GPT-4o, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct). Profile 2/: Baseline results and "LLM-as-a-Judge" evaluation scores for profile 2 on CDCA and EPMA for all LLMs (GPT-4.1, GPT-4o, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct). Each folder contains the corresponding outputs and/or ground truth for that stage.