A dataset to assess Microsoft Copilot Answers \\ in the Context of Swiss, Bavarian and Hesse Elections.

This readme file was generated on 2024-01-15 by Salvatore Romano

GENERAL INFORMATION

Title of Dataset: A dataset to assess Microsoft Copilot Answers in the Context of Swiss, Bavarian and Hesse Elections. 

Author/Principal Investigator InformationName: Salvatore RomanoORCID: 0000-0003-0856-4989Institution: Universitat Oberta de Catalunya, AID4So.Address: Rambla del Poblenou, 154. 08018 Barcelona.Email: [e-mail]

Author/Associate or Co-investigator InformationName: Riccardo AngiusORCID: 0000-0003-0291-3332Institution: Ai ForensicsAddress: Paris, France.Email: [e-mail]

Date of data collection: from 2023-09-21 to 2023-10-02.

Geographic location of data collection: Switzerland and Germany.

Information about funding sources that supported the collection of the data: The data collection and analysis was supported by AlgorithmWatch's DataSkop project, funded by Germany’s Federal Ministry of Education and Research (BMBF) as part of the program “Mensch-Technik-Interaktion” (human-technology interaction). dataskop.netIn Switzerland, the investigation was realized with the support of Stiftung Mercator Schweiz.AI Forensics contribution was supported in part by the Open Society Foundations.AI Forensics data collection infrastructure is supported by the Bright Initiative.

SHARING/ACCESS INFORMATION

Licenses/restrictions placed on the data: This publication is licensed under a Creative Commons Attribution 4.0 International License.https://creativecommons.org/licenses/by/4.0/deed.en

Links to publications that cite or use the data: https://aiforensics.org//uploads/AIF_AW_Bing_Chat_Elections_Report_ca7200fe8d.pdf

Links to other publicly accessible locations of the data: NA

Links/relationships to ancillary data sets: NA

Was data derived from another source? NAIf yes, list source(s): 

Recommended citation for this dataset: S. Romano, R. Angius, N. Kerby, P. Bouchaud, J. Amidei, A. Kaltenbrunner. 2024. A dataset to assess Microsoft Copilot Answers in the Context of Swiss, Bavarian and Hesse Elections. https://aiforensics.org//uploads/AIF_AW_Bing_Chat_Elections_Report_ca7200fe8d.pdf

DATA & FILE OVERVIEW

File List: Microsof-Copilot-Answers_in-Swiss-Bavarian-Hess-Elections.csvThe only dataset for this research. It includes rows with prompts and responses from Microsoft Copilot, along with associated metadata for each entry.

Relationship between files, if important: NA

Additional related data collected that was not included in the current data package: NA 

Are there multiple versions of the dataset? NAIf yes, name of file(s) that was updated: Why was the file updated? When was the file updated? 

METHODOLOGICAL INFORMATION

Description of methods used for collection/generation of data:In our algorithmic auditing research, we adopted for a sock-puppet audit methodology (Sandvig at Al., 2014). This method aligns with the growing interdisciplinary focus on algorithm audits, which prioritize fairness, accountability, and transparency to uncover biases in algorithmic systems (Bandy, 2021). Sock-puppet auditing offers a fully controlled environment to understand the behavior of the system.

Every sample was collected by running a new browser instance connected to the internet via a network of VPNs and residential IPs based in Switzerland and Germany, then accessing Microsoft Copilot through its official URL. Every time, the settings for Language and Country/Region were set to match those of potential voters from the respective regions (English, German, French, or Italian, and Switzerland or Germany). We did not simulate any form of user history or additional personalization. Importantly, Microsoft Copilot's default settings remained unchanged, ensuring that all interactions occurred in the ``Conversation Style" set as ``Balanced". 

Sandvig, C.; Hamilton, K.; Karahalios, K.; and Langbort, C. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry,22(2014): 4349–4357.

Bandy, J. 2021. Problematic machine behavior: A systematic literature review of algorithm audits. Proceedings of the acm on human-computer interaction, 5(CSCW1): 1–34

Methods for processing the data: The process involved analyzing the HTML code of the web pages that were accessed. During this examination, key metadata were identified and extracted from the HTML structure. Once this information was successfully extracted, the rest of the HTML page, which primarily consisted of code and elements not pertinent to the needed information, was discarded. This approach ensured that only the most relevant and useful data was retained, while all unnecessary and extraneous HTML components were efficiently removed, streamlining the data collection and analysis process.

Instrument- or software-specific information needed to interpret the data: NA

Standards and calibration information, if appropriate: NA

Environmental/experimental conditions: NA

Describe any quality-assurance procedures performed on the data: NA

People involved with sample collection, processing, analysis and/or submission: Salvatore Romano, Riccardo Angius, Natalie Kerby, Paul Bouchaud, Jacopo Amidei, Andreas Kaltenbrunner.

DATA-SPECIFIC INFORMATION FOR:Microsof-Copilot-Answers_in-Swiss-Bavarian-Hess-Elections.csv

Number of variables: Number of Variables: 33

Number of cases/rows: 5562

Variable List:prompt - (object) Text of the prompt.answer - (object) Text of the answer.country - (object) Country information.language - (object) Language of the text.input_conversation_id - (object) Identifier for the conversation.conversation_group_ids - (object) Group IDs for the conversation.conversation_group_names - (object) Group names for the conversation.experiment_id - (object) Identifier for the experiment group.experiment_name - (object) Name of the experiment group.begin - (object) Start time.end - (object) End time.datetime - (int64) Datetime stamp.week - (int64) Week number.attributions - (object) Link quoted in the text.attribution_links - (object) Links for attributions.search_query - (object) Search query used by the chatbot.unlabelled - (int64) Unlabelled flag.exploratory_sample - (int64) Exploratory sample flag.very_relevant - (int64) Very relevant flag.needs_review - (int64) Needs review flag.misleading_factual_error - (int64) Misleading factual error flag.nonsense_factual_error - (int64) Nonsense factual error flag.rejects_question_framing - (int64) Rejects question framing flag.deflection - (int64) Deflection flag.shield - (int64) Shield flag.wrong_answer_language - (int64) Wrong answer language flag.political_imbalance - (int64) Political imbalance flag.refusal - (int64) Refusal flag.factual_error - (int64) Factual error flag.evasion - (int64) Evasion flag.absolutely_accurate - (int64) Absolutely accurate flag.macrocategory - (object) Macro-category of the content.

Missing data codes:NA

Specialized formats or other abbreviations used: NA