ASE26_AVR_study

# ASE26_DRSCV ## Downstream Repair of Supply Chain Vulnerabilities (DR-SCV): Dataset and Evaluation Artifacts This repository provides the dataset and experimental artifacts for our ASE 2026 paper on downstream repair of supply chain vulnerabilities. --- ## 1. Introduction Third-party library vulnerabilities are often difficult to address through upstream fixes due to version constraints and the high maintenance cost of patch adoption. Consequently, repairing vulnerabilities directly in downstream projects has become an increasingly important strategy. To support research in this setting, we construct a dataset for **Downstream Repair of Supply Chain Vulnerabilities (DR-SCV)** and provide evaluation artifacts for Automated Vulnerability Repair (AVR) tools. A key challenge in DR-SCV is that the vulnerability root cause often resides in third-party libraries and is therefore not accessible or localizable within downstream project code. This repository is designed to facilitate evaluation under this realistic constraint. --- ## 2. Dataset Overview The dataset consists of downstream proof-of-concept (PoC) cases derived from real-world projects. - **24 upstream projects**- **63 downstream projects**- Covers diverse supply chain vulnerability scenarios- Includes downstream contexts where root cause code may be unavailable Each dataset instance represents a DR-SCV case, including:- Vulnerability information- Downstream project context- Proof-of-concept (PoC) triggering the vulnerability --- ## 3. Repository Structure ```textASE26_DRSCV/├── dataset/ # Downstream repositories with PoCs, upstream repositories, and upstream PoCs├── prompt/ # Prompt files used in different evaluation settings│ ├── prompt1/ # Prompts for RQ1 and RQ2│ └── prompt2/ # Prompts with injected root-cause functions(RQ3)├── evaluated_tools_cli_scripe/ # CLI scripts for the evaluated repair tools(The tool versions used in our experiments should be downloaded)│ ├── Agentless_JAVA/ # Our adapted Java-compatible version of Agentless│ ├── OpenHands/ # CLI scripts for OpenHands│ ├── ReinFix/ # CLI scripts for ReinFix│ └── SWE-agent/ # CLI scripts for SWE-agent├── code/ # Scripts for running experiments and evaluations(RQ1-4)├── result/ # Generated patches(RQ1-3) and evaluation outputs examples(RQ4)├── LICENSE # MIT License└── README.md # Project documentation``` ## 4. Evaluation Framework We propose a multidimensional evaluation framework tailored to DR-SCV, which jointly considers: - **Patch Quality** - Correctness and validity of generated patches - **Preservation Integrity** - Detection of misleading behaviors (e.g., modification of test cases) This framework enables systematic evaluation of AVR tools in downstream repair scenarios. --- ## 5. Quick Start ## 5.1 Requirements Before starting, ensure your development environment meets the following requirements: ### Prerequisites - **Java**: Version 1.8 - **Python**: Version 3.12 ### 5.2 Downstream Project POC Usage ### Example (run a specific test class) ```bashmvn test -Dtest=XStreamEventSerializer_ESTest``` The PoCs of our dataset: | targetID | pocClassName(Dtest arg) ||----------|--------------|| CVE-2017-7957_cqrs-lottery-master/cqrs-example | XStreamEventSerializer_ESTest || CVE-2017-7957_source | XMLConfig_ESTest.class || CVE-2018-1324_common-util | ZipUtils_ESTest || CVE-2018-1002202_brigen-base | ZipDelegaterZip4j_ESTest || CVE-2019-12415_poi-examples | CustomXMLMapping_Test || CVE-2019-12415_PoiSamples | CVE_2019_12415_Test || CVE-2020-13956_crawler-jsoup-maven | CSDNLoginApater_ESTest || CVE-2020-13956_wechat-ssm | HttpUtils_ESTest || CVE-2020-26217_cqrs-lottery-master/cqrs-example | XStreamEventSerializer_ESTest || CVE-2020-26217_source | XMLConfig_ESTest.class || CVE-2020-26258_cqrs-lottery-master/cqrs-example | CVE_2020_26258_ComparisonTest.class || CVE-2021-21341_cqrs-lottery-master/cqrs-example | CVE_2021_21341_Test.class || CVE-2021-21341_source | CVE_2021_21341_PoC_Test.class || CVE-2021-23899_json-sanitizer | JsonSanitizer_ESTest || CVE-2021-29425_FastJoin | FileReader_ESTest.class || CVE-2021-31812_Framework-Mobile | Pdf_ESTest.class || CVE-2021-31812_PDFConvert14 | CVE_2021_31812_ESTest || CVE-2021-35516_JavaUtils-master | ZipPwdUtil_ESTest || CVE-2021-37714_phoenix_interface | InterfaceAPI_ESTest || CVE-2021-37714_pickaxe | Scraper_ESTest || CVE-2021-39144_cqrs-lottery-master/cqrs-example/ | CVE_2021_39144_Test.class || CVE-2021-39144_source | XMLConfig_ESTest.class || CVE-2022-41966_cqrs-lottery-master/cqrs-example | XStreamEventSerializer_ESTest || CVE-2022-41966_source | XMLConfig_ESTest || CVE-2022-42889_java | SearchController_ESTest || CVE-2022-45688_base-starter | XMLUtil_ESTest || CVE-2022-45688_virtress | Converter_ESTest.class || CVE-2023-1370_microservice-with-jwt-and-microprofile | TokenUtil_ESTest.class || CODEC-263_DBlog-master/blog-core | PasswordUtil_ESTest || CODEC-270_BurpCrypto-master | AesUtil_ESTest || CODEC-270_DBlog-master/blog-core | PasswordUtil_ESTest || IO-611_FastJoin | FileReader_ESTest.class || IO-611_velocity-engine/velocity-engine-core | FileResourceLoader_ESTest || LANG-1385_ewallet/wallet-base | NumberUtil_ESTest.class || LANG-1385_wechat-ssm | MyNumberUtils_ESTest.class || LANG-1484_jjdz7-drony-refactor/Web | Validator_ESTest.class || LANG-1645_ewallet/wallet-base | NumberUtil_ESTest.class || LANG-1645_wechat-ssm | MyNumberUtils_ESTest || TEXT-215_geoportal-esri/geoportal | Val_TEXT215_Test || TEXT-215_geoportal-server/geoportal | Val_TEXT215_Test || Zip-263_CarStoreApi/account/account-web | ZipUtil_ESTest || Zip-263_ZingClient | ZFile_ESTest || CVE-2023-43642_flow | CVE_2023_43642_Testcase2Test || CVE-2023-1370_axon-server-se/axonserver | CVE_2023_1370_Testcase3 || CVE-2022-42004_serritor | CVE_2022_42004_DownstreamTest || CVE-2022-42004_vsqdeveloper-serritor | CVE_2022_42004_DownstreamTest || CVE-2020-13973_son-editor/son-validate-web | CVE_2020_13973_Test || CVE-2020-13956_apache/httpcomponents-client | HttpUtils_CVE_2020_13956_Test || CVE-2019-10086_bean-query | CVE_2019_10086_Testcase2 || CVE-2021-27568_json-configuration | JsonConfigurationTest || CVE-2022-25845_geek_framework | DownstreamFastjsonCVE25845Test || CVE-2022-25845_base-starter | ReadDataPoCTest || CVE-2023-34453_UltraPlaytime | CVE202334453PoCTest || CVE-2022-29631_ucloud-java-sdk | CrlfInjectionPocTest || CVE-2015-2156_webbit | NettyCookieDecodePrintTest / NettyMalformedCookieDecoderTest / HttpRequestWrapperCookieVulTest || CVE-2022-22976_gerenciador-viagens | CVE_2022_22976_Test || CVE-2017-7957_rpki-commons | CVE_2017_7957_Test || CVE-2021-43859_rpki-commons | CVE_2021_43859_Test || CVE-2021-23900_OmegaTester | CVE_2021_23900_Test || CVE-2018-1002201_neo | CVE_2018_1002201_Test || CVE-2018-1000632_tcpser4j | CVE_2018_1000632_Test || CVE-2018-1002201_elasticsearch-maven-plugin | CVE_2018_1002201_Test || CVE-2018-15756_mirage/mirage-core | CVE_2018_15756_Test || CVE-2022-22976_RuoYi-Vue-Multi-Tenant/multi-tenant-server | CVE_2022_22976_Test || CVE-2021-23899_OmegaTester | CVE_2021_23899_Test | ### 5.3 Upstream Project POC Usage Refer to the approach used in https://github.com/WordDealer/ASE2024-Magneto ## 6. how to use our Evaluation Framework #### 6.1 Execution on the Dataset Run the in-house tool on the full set of 63 datasets. #### 6.2 Evaluation Framework Usage Use the evaluation framework, where the output is required to be a single standardized patch file with a `.diff` or `.patch` suffix. ### 6.3 Verify Whether the Test Patches Can Be Successfully Compiled Using **SWE-agent** as an example: ```bash./run_mvn_install_all.sh \ --root /VESTA/resource \ --patch-dir /VESTA/resource/patch_swe-agent/patch_swe-agent/swe-agent \ --checker ./mvn_install_one_agentless.py \ --result ./result.txt \ --pass-dir ./swe-agent_installed_patch \ --label swe-agent-2-track``` ### 6.4 Verify Whether the Test Patches Can Pass All Original Test Suites Using **SWE-agent** as an example: ```bashpython run_mvn_test_all.py \ --patchs=/VESTA/resource/patch_scripes/swe-agent_installed_patch/ \ --repo=/VESTA/resource/ \ --output=./swe-agent_patch_test_success_2_track``` ### 6.5 Verify Whether the Test Patches Can Pass the PoC Tests Using **SWE-agent** as an example: ```bashpython patch_Dtest_all.py \ --yaml swe-agent-gpt-4o-tasks.yaml \ --validator ./patch_validator.py \ --repo_root /VESTA/resource \ --patch_root /VESTA/resource/patch_scripes/swe-agent_installed_patch \ --out_dir ~/swe_success_patch``` ### 6.6 Verify the Integrity of the Test Patches Using **SWE-agent** as an example: Count how many patches modify test files:```bashpython count_test_in_patch.py swe-agent/``` Count how many patches modify PoC files:```bashpython count_Dtest_in_patch.py --dir swe-agent/ --rules Dtest_class.yaml``` --- ## 7. Disclaimer This dataset is intended for research purposes only. The included vulnerability information and proof-of-concept (PoC) cases are provided solely to support evaluation research and should be handled responsibly.