TrustDABench

# TrustDABench Data This ModelScope repository hosts the full TrustDABench data release as a multi-part ZIP archive. ## Download Download all archive parts: ```text TrustDABench-Data-Full.zip.part001 TrustDABench-Data-Full.zip.part002 TrustDABench-Data-Full.zip.part003 TrustDABench-Data-Full.zip.part004 TrustDABench-Data-Full.zip.part005 TrustDABench-Data-Full.zip.part006 TrustDABench-Data-Full.zip.part007 TrustDABench-Data-Full.zip.part008 TrustDABench-Data-Full.zip.part009 ``` Then reconstruct the archive. Linux/macOS: ```bash cat TrustDABench-Data-Full.zip.part* > TrustDABench-Data-Full.zip sha256sum -c SHA256SUMS.txt unzip TrustDABench-Data-Full.zip ``` Windows PowerShell: ```powershell Get-Content -Encoding Byte -Path .\TrustDABench-Data-Full.zip.part* | Set-Content -Encoding Byte .\TrustDABench-Data-Full.zip Get-FileHash .\TrustDABench-Data-Full.zip -Algorithm SHA256 Expand-Archive .\TrustDABench-Data-Full.zip ``` Expected SHA256 for the reconstructed archive: ```text 6ce82e4fa168012349333a989a9fe5eb1929913a3caddb0c7a9c9247822652d6 ``` `SHA256SUMS.txt` also contains checksums for each individual part. ## Package Structure After extraction, the data directory has this structure: ```text TrustDABench-Data-Full/ reliability/ aida_qa/ attack_dataset.jsonl files/<sample_id>/ dabench/ attack_dataset.jsonl files/<sample_id>/ robustness/ aida_qa/ attack_dataset.jsonl files/<sample_id>/ dabench/ attack_dataset.jsonl files/<sample_id>/ ``` Each `attack_dataset.jsonl` is a JSON Lines file. Each line is one benchmark sample. The field `file_root` points to the relative directory containing the attacked input files for that sample. If `input_file` contains multiple file names separated by newlines, all listed files are located under the same `file_root` directory. ## Subsets | Path | Task Source | Dimension | Samples | | --- | --- | --- | ---: | | `reliability/aida_qa` | AIDA-QA | reliability | 562 | | `reliability/dabench` | DABENCH | reliability | 643 | | `robustness/aida_qa` | AIDA-QA | robustness | 672 | | `robustness/dabench` | DABENCH | robustness | 463 | ## Record Fields Common fields: | Field | Meaning | | --- | --- | | `id` | Unique attacked sample ID. | | `source_id` | ID of the original clean task. | | `scenario` | Benchmark scenario. Currently `table_reason`. | | `dimension` | Reliability sub-dimension or `robustness`. | | `attack_type` | Perturbation operator. | | `question` | Question to answer against the attacked files. | | `answer` | Gold answer or refusal target. | | `input_file` | File name or newline-separated file names provided to the model. | | `file_root` | Relative directory containing the attacked files. | | `source_question` | Original clean question. | | `source_reference` | Original clean reference answer. | | `construction` | Public validation summary; only `validation_status` is retained. | Robustness records may also include: | Field | Meaning | | --- | --- | | `answer_relation` | Relation between clean and attacked answers. Currently `invariant`. | | `source_input_file` | Original clean input file name(s). | | `source_metadata` | Optional source-dataset metadata. | | `transformation_record` | Structured description of the answer-preserving transformation. | ## Reliability Operators ```text field_missing data_missing evidence_conflict file_missing header_conflict deep_analysis_missing structural_context_missing ``` Reliability samples test whether a model refuses to answer when the attacked table package no longer supports a unique answer. ## Robustness Operators ```text row_order_shuffle column_order_shuffle header_synonym_substitution semantic_distractor_column equivalent_value_reencoding unit_scale_conversion csv_wide_long_reshape csv_relational_decomposition excel_hierarchical_header_relayout excel_cross_sheet_relayout decoy_feature_pack_injection non_observation_row_injection ``` Robustness samples test whether a model can still produce the correct answer when the table representation is changed or redundant information is injected.