rpaweb

# Dataset Card for rpaweb <!-- Provide a quick summary of the dataset. --> The **rpaweb** dataset contains multilingual annotations of **Robotic Process Automation (RPA)** actions in a web environment in English, Spanish and Catalan. It has been designed to train multimodal models capable of autonomously performing browser-based tasks based on natural language user requests. The corpus documents each user request along with the full sequence of web automation actions needed to fulfil it, including structured JSON code, natural language descriptions, web page screenshots, and HTML dumps. The dataset covers five main usage scenarios, each divided into 15 intents in total: 1. **[Agència Tributària de la Generalitat de Catalunya](https://atc.gencat.cat/ca/tributs/impost-emissions-vehicles/)**: Vehicle CO₂ tax procedures. Accessing, consulting, modifying vehicle records, reporting status changes, etc. 2. **[Agencia Tributaria del Gobierno de España](https://sede.agenciatributaria.gob.es/)**: Personal finance procedures. Booking appointments, checking tax status. 3. **[Aladí - Xarxa de Biblioteques Municipals de la Diputació de Barcelona](https://aladi.diba.cat/)**: Book queries. 4. **[TimeOut](https://www.timeout.cat/barcelona/ca/search#/?viewstate=list)**: Search and browse city activities and events. 5. **[Trello](https://trello.com/)**: Project management actions with boards, lists, cards. ## Dataset Details <!-- Provide a longer summary of what this dataset is. --> - **Curated by:** Barcelona Supercomputing Center (BSC) - **Funded by:** [AINA](https://projecteaina.cat/) - **Languages:** Catalan (`ca`), English (`en`), Spanish (`es`) - **License:** CC BY 4.0 ## Uses <!-- Address questions around how the dataset is intended to be used. --> ### Direct Use <!-- This section describes suitable use cases for the dataset. --> - Training and evaluating multimodal RPA agents. - Research on grounding natural language instructions into executable browser actions. - Studying multilingual RPA workflows. ### Out-of-Scope Use <!-- This section addresses misuse, malicious use, and uses that the dataset will not work well for. --> - Not intended for processing or generating real personal data. - Not designed for high-stakes automation without human oversight. ## Dataset Structure <!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. --> The dataset has a single split (`train`) for each of the 3 language subsets (`en`, `es`, `ca`) and includes the following fields per action: - `id`: Unique action identifier which consists of {intent_id}.{lang}.{action_id} - `language`: Language of the action. - `intent`: Natural language description of the action. - `step`: Step of the action input. - `input`: User request text in natural language. - `requirements`: Set of preconditions or input parameters needed before executing the RPA workflow for a given user request. - `outputs`: The expected result(s) or goal state after executing the RPA workflow. - `anonymized_data`: A mapping of personal data elements detected during annotation to synthetic placeholder values. - `png_file`: PNG screenshot of the web page before execution. - `json_data`: JSON code of the RPA step. - `html_data`: HTML dump of the web page before execution. The default format of the dataset is a JSONL file, where each line has the following structure: ```json {"id": "i11.en.0015", "language": "en", "intent": "saving/emailing a bibliographic record", "step": 7, "input": "Save the books’ information", "requirements": "{\"*Com guardar registres: enviar com a fitxer o a correu?\": \"FITXER\", \"*S'especifica com guarder els registres?\": \"NO\", \"*S'especifica l'adreça d'email?\": \"no rellevant\", \"*S'especifica el títol de l'email?\": \"no rellevant\", \"*format del registre:- Visualització completa (TOT)- Visualització breu (BREU)- MARC\": \"MARC\", \"*S’especifica el format del registre?\": \"NO\"}", "outputs": "{\"classe\": \"GuardarEnviarRegistreBibliografic\", \"email\": \"no rellevant\", \"assumpte\": \"no rellevant\"}", "anonymized_data": "{}", "png_file": "data/en/i11.en.0015.7.png", "json_data": "{\"browserCode\": \"{\\\"name\\\":\\\"click\\\",\\\"selector\\\":\\\"internal:role=button[name=\\\\\\\"Submit\\\\\\\"i]\\\",\\\"signals\\\":[{\\\"name\\\":\\\"download\\\",\\\"downloadAlias\\\":\\\"2\\\"}],\\\"button\\\":\\\"left\\\",\\\"modifiers\\\":0,\\\"clickCount\\\":1,\\\"pageAlias\\\":\\\"page\\\",\\\"locator\\\":{\\\"kind\\\":\\\"role\\\",\\\"body\\\":\\\"button\\\",\\\"options\\\":{\\\"attrs\\\":[],\\\"exact\\\":false,\\\"name\\\":\\\"Submit\\\"}}}\\n\", \"nlpDescription\": \"submit\", \"subintent\": \"browser_action\"}", "html_data": "<!DOCTYPE html><html lang=\"en\"><head><meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\">\n\n<title>Diputació de Barcelona [...]"" } ``` The source files consist of: - **`peticions_usuari/`**: CSV files with user requests, parameters, and anonymized personal data placeholders. - **`accions_rpa/`**: directories per request containing JSON, PNG, HTML, and anonymization mapping. ## Dataset Creation ### Curation Rationale <!-- Motivation for the creation of this dataset. --> The **rpaweb** dataset was developed to support research and development of multilingual, multimodal systems capable of performing **Robotic Process Automation (RPA)** actions in web environments. Its main objective is to connect natural language understanding with structured browser automation, enabling models to convert user requests into reproducible, executable workflows. The dataset includes a broad range of domains—government services, library catalog searches, event listings, and project management platforms—to ensure coverage of diverse interaction patterns and action types. ### Source Data <!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). --> #### Data Collection and Processing <!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. --> The dataset content was produced through a controlled, tool-assisted annotation workflow: 1. **User request creation** - Requests were manually authored for each intent in **Catalan (ca)**, **English (en)**, and **Spanish (es)**. - Each language’s dataset was generated independently (not translations) to maximize linguistic and phrasing diversity. - Relevant parameters (e.g., year, license plate, search terms) were explicitly annotated for each request. 2. **Execution environment** - Annotators used a dedicated RPA annotation tool integrated with the [Playwright](https://playwright.dev/) automation framework. - All actions were executed on live websites or simulated environments corresponding to the target intent. 3. **Annotation outputs** - Each user request was paired with a sequence of action quadruples: 1. JSON code representing the RPA command. 2. Natural language description of the action. 3. PNG screenshot of the webpage before action execution. 4. HTML dump of the webpage before action execution. 4. **Validation** - Automated checks ensured JSON validity, action type correctness, and anonymization completeness. - Manual inspection of sampled data validated task flow consistency. #### Who are the source data producers? <!-- This section describes the people or systems who originally created the data. It should also include self-reported demographic or identity information for the source data creators if this information is available. --> The dataset was created by annotators who are trained in RPA tool usage, multilingual content creation, and data anonymization. ### Annotations <!-- If the dataset contains annotations which are not part of the initial data collection, use this section to describe them. --> #### Annotation process <!-- This section describes the annotation process such as annotation tools used in the process, the amount of data annotated, annotation guidelines provided to the annotators, interannotator statistics, annotation validation, etc. --> - **Tooling**: Custom-built annotation platform with Playwright integration. - **Granularity**: Each interaction, browser command, and user feedback request was recorded as an individual annotation step. - **Metadata**: Each action is linked to the intent, language, and required parameters. - **Itineraries**: Canonical action sequences defined per intent to ensure coverage and consistency. - **Review**: All annotations were anonymized automatically; a sample was manually checked for compliance. #### Personal and Sensitive Information <!-- State whether the dataset contains data that might be considered personal, sensitive, or private (e.g., data that reveals addresses, uniquely identifiable names or aliases, racial or ethnic origins, sexual orientations, religious beliefs, political opinions, financial or health data, etc.). If efforts were made to anonymize the data, describe the anonymization process. --> The dataset contains **no real personal data**. During annotation, realistic values (e.g., NIF, IBAN, license plates) were required to progress in real systems, but: - Real values remained **local** to the annotator’s machine. - An anonymization configuration file (`config_anonymizer.csv`) mapped real values to **synthetic placeholders**. - The public dataset only includes anonymized values stored in `anonymized_data.json`. ### Recommendations <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. --> - Before deploying models trained on this dataset in production, validate them in controlled environments. - Although the dataset is domain-accurate; for production, complement with real in-domain RPA logs. - Especially in high-stakes contexts (e.g., government filings, financial operations), always keep a human in the loop. - Each language variant is unique, so models should be tested per language to avoid performance gaps. ## Glossary [optional] <!-- If relevant, include terms and calculations in this section that can help readers understand the dataset or dataset card. --> - **Intent**: High-level description of the task (e.g., “consult vehicle tax data”). - **Itinerary**: Ordered sequence of RPA steps to complete a specific request. - **Browser action**: Automation step executed directly without user input (e.g., click, navigate). - **User feedback**: Step requiring user input before continuing (e.g., entering SMS code). - **Quadruple**: Four-part action annotation (JSON code, description, screenshot, HTML). - **Anonymized data**: Synthetic replacements for sensitive personal information. ## More Information This work/research has been promoted and financed by the Government of Catalonia through the [Aina project](https://projecteaina.cat/). ## Dataset Card Contact Language Technologies Laboratory ([e-mail]) at the Barcelona Supercomputing Center (BSC).