joelniklaus/eurlex_resources

---
annotations_creators:
- other
language_creators:
- found
language:
- bg
- cs
- da
- de
- el
- en
- es
- et
- fi
- fr
- ga
- hr
- hu
- it
- lt
- lv
- mt
- nl
- pl
- pt
- ro
- sk
- sl
- sv
license:
- cc-by-4.0
multilinguality:
- multilingual
paperswithcode_id: null
pretty_name: "EurlexResources: A Corpus Covering the Largest EURLEX Resources"
size_categories:
- 1M<n<10M
source_datasets:
- original
task_categories:
- fill-mask

---

# Dataset Card for EurlexResources: A Corpus Covering the Largest EURLEX Resources

## Table of Contents

- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:**
- **Repository:** [GitHub](https://github.com/JoelNiklaus/LegalDatasets/tree/main/pretrain/eurlex)
- **Paper:**
- **Leaderboard:**
- **Point of Contact:** [Joel Niklaus](mailto:[e-mail])

### Dataset Summary

This dataset contains large text resources (~179GB in total) from EURLEX that can be used for pretraining language models.

Use the dataset like this:
```python
from datasets import load_dataset
config = "de_caselaw" # {lang}_{resource}
dataset = load_dataset("joelito/eurlex_resources", config, split='train', streaming=True)
```

### Supported Tasks and Leaderboards

The dataset supports the task of masked language modeling.

### Languages

The following languages are supported: bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv

## Dataset Structure

### Data Instances

The file format is jsonl.xz and there is one split available ("train").
The following resource types are supported: caselaw, decision, directive, intagr, proposal, recommendation, regulation

More information about the resource types can be found here:
- Caselaw: [EU](https://eur-lex.europa.eu/collection/eu-law/eu-case-law.html)
- Decision: [EU](https://eur-lex.europa.eu/EN/legal-content/summary/european-union-decisions.html), [Wikipedia](https://en.wikipedia.org/wiki/Decision_(European_Union))
- Directive: [EU](https://european-union.europa.eu/institutions-law-budget/law/types-legislation_en), [Wikipedia](https://en.wikipedia.org/wiki/Directive_(European_Union))
- Recommendation: [EU](https://eur-lex.europa.eu/EN/legal-content/glossary/recommendation.html), [Wikipedia](https://en.wikipedia.org/wiki/Recommendation_(European_Union))
- Regulation: [EU](https://european-union.europa.eu/institutions-law-budget/law/types-legislation_en), [Wikipedia](https://en.wikipedia.org/wiki/Regulation_(European_Union))
- Intagr: [EU](https://eur-lex.europa.eu/collection/eu-law/inter-agree.html), [Wikipedia](https://en.wikipedia.org/wiki/Treaties_of_the_European_Union)
- Proposal: No resource found

| Source | Size (MB) | Words | Documents | Words/Document |
|:-------------------|------------:|------------:|------------:|-----------------:|
| all_all | 180668 | 12106556233 | 8306749 | 1457 |
| all_caselaw | 34939 | 3413551598 | 2487794 | 1372 |
| all_decision | 28519 | 1698585620 | 1267402 | 1340 |
| all_directive | 4786 | 368577940 | 104187 | 3537 |
| all_intagr | 11421 | 743271516 | 274485 | 2707 |
| all_proposal | 26526 | 2087989530 | 702392 | 2972 |
| all_recommendation | 1886 | 164979037 | 80277 | 2055 |
| all_regulation | 72590 | 3629600992 | 3390212 | 1070 |
| bg_all | 7819 | 398067053 | 348691 | 1141 |
| bg_caselaw | 1588 | 109749174 | 104434 | 1050 |
| bg_decision | 1248 | 58817972 | 54075 | 1087 |
| bg_directive | 263 | 15731608 | 4388 | 3585 |
| bg_intagr | 603 | 31292848 | 11581 | 2702 |
| bg_proposal | 1083 | 60674956 | 29251 | 2074 |
| bg_recommendation | 89 | 5588991 | 3321 | 1682 |
| bg_regulation | 2943 | 116211504 | 141641 | 820 |
| cs_all | 8360 | 471961631 | 449793 | 1049 |
| cs_caselaw | 1163 | 110005022 | 104519 | 1052 |
| cs_decision | 1102 | 58921128 | 54075 | 1089 |
| cs_directive | 186 | 13951134 | 4388 | 3179 |
| cs_intagr | 449 | 28106332 | 11581 | 2426 |
| cs_proposal | 840 | 61838692 | 29252 | 2113 |
| cs_recommendation | 64 | 5416549 | 3323 | 1630 |
| cs_regulation | 4557 | 193722774 | 242655 | 798 |
| da_all | 8932 | 671484862 | 332500 | 2019 |
| da_caselaw | 1746 | 185589641 | 88234 | 2103 |
| da_decision | 1356 | 89498535 | 54085 | 1654 |
| da_directive | 207 | 17525792 | 4388 | 3994 |
| da_intagr | 506 | 35596169 | 11582 | 3073 |
| da_proposal | 1399 | 119759476 | 29257 | 4093 |
| da_recommendation | 100 | 9463897 | 3352 | 2823 |
| da_regulation | 3618 | 214051352 | 141602 | 1511 |
| de_all | 9607 | 695512401 | 348290 | 1996 |
| de_caselaw | 1930 | 193232441 | 104228 | 1853 |
| de_decision | 1449 | 93688222 | 53980 | 1735 |
| de_directive | 218 | 17337760 | 4385 | 3953 |
| de_intagr | 531 | 36791153 | 11580 | 3177 |
| de_proposal | 1556 | 126987454 | 29219 | 4346 |
| de_recommendation | 109 | 9608034 | 3318 | 2895 |
| de_regulation | 3813 | 217867337 | 141580 | 1538 |
| el_all | 12469 | 696216541 | 349667 | 1991 |
| el_caselaw | 2951 | 202027703 | 105138 | 1921 |
| el_decision | 1823 | 94919886 | 54150 | 1752 |
| el_directive | 321 | 19411959 | 4390 | 4421 |
| el_intagr | 701 | 38965777 | 11584 | 3363 |
| el_proposal | 2085 | 128005737 | 29290 | 4370 |
| el_recommendation | 145 | 9344866 | 3357 | 2783 |
| el_regulation | 4443 | 203540613 | 141758 | 1435 |
| en_all | 9217 | 769465561 | 348641 | 2207 |
| en_caselaw | 1846 | 222891827 | 104422 | 2134 |
| en_decision | 1504 | 114626013 | 54054 | 2120 |
| en_directive | 204 | 18860876 | 4388 | 4298 |
| en_intagr | 499 | 39029843 | 11581 | 3370 |
| en_proposal | 1538 | 140781768 | 29242 | 4814 |
| en_recommendation | 97 | 10091809 | 3320 | 3039 |
| en_regulation | 3530 | 223183425 | 141634 | 1575 |
| es_all | 8588 | 725125274 | 348443 | 2081 |
| es_caselaw | 1870 | 220621730 | 104312 | 2115 |
| es_decision | 1334 | 98163499 | 54001 | 1817 |
| es_directive | 221 | 21484479 | 4385 | 4899 |
| es_intagr | 516 | 41841805 | 11581 | 3612 |
| es_proposal | 1366 | 133674486 | 29224 | 4574 |
| es_recommendation | 82 | 8864018 | 3319 | 2670 |
| es_regulation | 3199 | 200475257 | 141621 | 1415 |
| et_all | 6090 | 328068754 | 349615 | 938 |
| et_caselaw | 1074 | 93096396 | 105111 | 885 |
| et_decision | 1069 | 50752324 | 54159 | 937 |
| et_directive | 177 | 11555930 | 4390 | 2632 |
| et_intagr | 436 | 24018147 | 11584 | 2073 |
| et_proposal | 810 | 51600852 | 29283 | 1762 |
| et_recommendation | 61 | 4451369 | 3355 | 1326 |
| et_regulation | 2464 | 92593736 | 141733 | 653 |
| fi_all | 7346 | 404265224 | 349633 | 1156 |
| fi_caselaw | 1596 | 126525296 | 105119 | 1203 |
| fi_decision | 1227 | 59659475 | 54163 | 1101 |
| fi_directive | 204 | 12766491 | 4389 | 2908 |
| fi_intagr | 463 | 25392311 | 11584 | 2192 |
| fi_proposal | 1075 | 69198401 | 29288 | 2362 |
| fi_recommendation | 73 | 5070392 | 3356 | 1510 |
| fi_regulation | 2707 | 105652858 | 141734 | 745 |
| fr_all | 9937 | 828959218 | 348295 | 2380 |
| fr_caselaw | 2158 | 246262666 | 104228 | 2362 |
| fr_decision | 1473 | 108648744 | 53981 | 2012 |
| fr_directive | 222 | 20308801 | 4385 | 4631 |
| fr_intagr | 536 | 41986012 | 11580 | 3625 |
| fr_proposal | 1592 | 149134298 | 29218 | 5104 |
| fr_recommendation | 112 | 11510415 | 3318 | 3469 |
| fr_regulation | 3845 | 251108282 | 141585 | 1773 |
| ga_all | 1028 | 65030095 | 349778 | 185 |
| ga_caselaw | 11 | 696305 | 105205 | 6 |
| ga_decision | 87 | 4415457 | 54189 | 81 |
| ga_directive | 18 | 1512027 | 4390 | 344 |
| ga_intagr | 19 | 1820723 | 11586 | 157 |
| ga_proposal | 289 | 26106889 | 29298 | 891 |
| ga_recommendation | 10 | 902390 | 3361 | 268 |
| ga_regulation | 594 | 29576304 | 141749 | 208 |
| hr_all | 4594 | 258816068 | 348691 | 742 |
| hr_caselaw | 617 | 62432734 | 104434 | 597 |
| hr_decision | 596 | 31911903 | 54075 | 590 |
| hr_directive | 156 | 10855913 | 4388 | 2474 |
| hr_intagr | 450 | 24962086 | 11581 | 2155 |
| hr_proposal | 552 | 33437815 | 29251 | 1143 |
| hr_recommendation | 40 | 3612247 | 3321 | 1087 |
| hr_regulation | 2183 | 91603370 | 141641 | 646 |
| hu_all | 6653 | 375253894 | 349605 | 1073 |
| hu_caselaw | 1278 | 110179375 | 105144 | 1047 |
| hu_decision | 1147 | 57108172 | 54156 | 1054 |
| hu_directive | 200 | 13568304 | 4389 | 3091 |
| hu_intagr | 470 | 27258501 | 11586 | 2352 |
| hu_proposal | 912 | 60882750 | 29291 | 2078 |
| hu_recommendation | 70 | 5312868 | 3357 | 1582 |
| hu_regulation | 2576 | 100943924 | 141682 | 712 |
| it_all | 9586 | 768605772 | 333631 | 2303 |
| it_caselaw | 1889 | 206117726 | 89560 | 2301 |
| it_decision | 1445 | 102848859 | 53983 | 1905 |
| it_directive | 217 | 19687773 | 4385 | 4489 |
| it_intagr | 528 | 40134330 | 11580 | 3465 |
| it_proposal | 1533 | 140713925 | 29218 | 4816 |
| it_recommendation | 109 | 10923431 | 3318 | 3292 |
| it_regulation | 3865 | 248179728 | 141587 | 1752 |
| lt_all | 6400 | 364361783 | 200565 | 1816 |
| lt_caselaw | 1137 | 101808706 | 105477 | 965 |
| lt_decision | 1096 | 55850308 | 21990 | 2539 |
| lt_directive | 185 | 13078983 | 3239 | 4037 |
| lt_intagr | 452 | 27009631 | 7481 | 3610 |
| lt_proposal | 850 | 58553579 | 29272 | 2000 |
| lt_recommendation | 64 | 5121089 | 3363 | 1522 |
| lt_regulation | 2617 | 102939487 | 29743 | 3460 |
| lv_all | 6349 | 363239195 | 349919 | 1038 |
| lv_caselaw | 1153 | 103456811 | 105242 | 983 |
| lv_decision | 1103 | 55512944 | 54224 | 1023 |
| lv_directive | 186 | 13023024 | 4392 | 2965 |
| lv_intagr | 452 | 26693107 | 11630 | 2295 |
| lv_proposal | 96 | 58176216 | 29298 | 1985 |
| lv_recommendation | 64 | 5074494 | 3361 | 1509 |
| lv_regulation | 2545 | 101302599 | 141772 | 714 |
| mt_all | 6540 | 367834815 | 350292 | 1050 |
| mt_caselaw | 1164 | 100423543 | 105479 | 952 |
| mt_decision | 1109 | 55239141 | 54280 | 1017 |
| mt_directive | 203 | 14355266 | 4392 | 3268 |
| mt_intagr | 470 | 27701991 | 11675 | 2372 |
| mt_proposal | 878 | 59749277 | 29274 | 2041 |
| mt_recommendation | 65 | 5039600 | 3363 | 1498 |
| mt_regulation | 2650 | 105325997 | 141829 | 742 |
| nl_all | 9586 | 770312808 | 349407 | 2204 |
| nl_caselaw | 1847 | 206271837 | 105005 | 1964 |
| nl_decision | 1456 | 104060901 | 54152 | 1921 |
| nl_directive | 217 | 19529361 | 4388 | 4450 |
| nl_intagr | 529 | 40247634 | 11584 | 3474 |
| nl_proposal | 1540 | 141258274 | 29279 | 4824 |
| nl_recommendation | 111 | 11002405 | 3355 | 3279 |
| nl_regulation | 3886 | 247942396 | 141644 | 1750 |
| pl_all | 6677 | 406648795 | 350349 | 1160 |
| pl_caselaw | 1231 | 115824759 | 105479 | 1098 |
| pl_decision | 1125 | 60407576 | 54287 | 1112 |
| pl_directive | 197 | 14672157 | 4392 | 3340 |
| pl_intagr | 466 | 28543668 | 11680 | 2443 |
| pl_proposal | 886 | 64728230 | 29317 | 2207 |
| pl_recommendation | 68 | 5769893 | 3363 | 1715 |
| pl_regulation | 2703 | 116702512 | 141831 | 822 |
| pt_all | 8450 | 675152149 | 348449 | 1937 |
| pt_caselaw | 1763 | 198084937 | 104312 | 1898 |
| pt_decision | 1327 | 93278293 | 54007 | 1727 |
| pt_directive | 217 | 19831549 | 4385 | 4522 |
| pt_intagr | 504 | 37999753 | 11581 | 3281 |
| pt_proposal | 1361 | 127461782 | 29224 | 4361 |
| pt_recommendation | 81 | 8396661 | 3319 | 2529 |
| pt_regulation | 3197 | 190099174 | 141621 | 1342 |
| ro_all | 6315 | 415038571 | 350300 | 1184 |
| ro_caselaw | 1110 | 114780999 | 105516 | 1087 |
| ro_decision | 1047 | 59479553 | 54281 | 1095 |
| ro_directive | 206 | 16101628 | 4392 | 3666 |
| ro_intagr | 481 | 31497000 | 11675 | 2697 |
| ro_proposal | 805 | 62130419 | 29274 | 2122 |
| ro_recommendation | 63 | 5977913 | 3363 | 1777 |
| ro_regulation | 2603 | 125071059 | 141799 | 882 |
| sk_all | 6484 | 392235510 | 350570 | 1118 |
| sk_caselaw | 1160 | 110125141 | 105608 | 1042 |
| sk_decision | 1111 | 59576875 | 54349 | 1096 |
| sk_directive | 188 | 14132755 | 4393 | 3217 |
| sk_intagr | 458 | 28298155 | 11676 | 2423 |
| sk_proposal | 859 | 63726047 | 29290 | 2175 |
| sk_recommendation | 66 | 5654790 | 3364 | 1680 |
| sk_regulation | 2642 | 110721747 | 141890 | 780 |
| sl_all | 6222 | 394814289 | 350574 | 1126 |
| sl_caselaw | 1071 | 111238184 | 105608 | 1053 |
| sl_decision | 1075 | 59454906 | 54349 | 1093 |
| sl_directive | 176 | 13908097 | 4393 | 3165 |
| sl_intagr | 441 | 28239078 | 11676 | 2418 |
| sl_proposal | 812 | 63391970 | 29290 | 2164 |
| sl_recommendation | 62 | 5628775 | 3364 | 1673 |
| sl_regulation | 2585 | 112953279 | 141894 | 796 |
| sv_all | 7419 | 500085970 | 351051 | 1424 |
| sv_caselaw | 1585 | 162108645 | 105980 | 1529 |
| sv_decision | 1213 | 71744934 | 54357 | 1319 |
| sv_directive | 195 | 15386273 | 4393 | 3502 |
| sv_intagr | 463 | 29845462 | 11676 | 2556 |
| sv_proposal | 1059 | 86016237 | 29292 | 2936 |
| sv_recommendation | 79 | 7152141 | 3366 | 2124 |
| sv_regulation | 2825 | 127832278 | 141987 | 900 |

### Data Fields

[More Information Needed]

### Data Splits

[More Information Needed]

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

The data has been downloaded using the R package [eurlex](https://cran.r-project.org/web/packages/eurlex/vignettes/eurlexpkg.html) between June and August 2022.

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)
[see also the legal notice](https://eur-lex.europa.eu/content/legal-notice/legal-notice.html)

### Citation Information

[More Information Needed]

### Contributions

Thanks to [@JoelNiklaus](https://github.com/joelniklaus) for adding this dataset.

原始信息汇总

数据集概述

数据集名称

• 名称: EurlexResources: A Corpus Covering the Largest EURLEX Resources
• 别名: EurlexResources

数据集基本信息

• 语言: 支持多种语言，包括bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv
• 许可证: cc-by-4.0
• 多语言性: 多语言
• 大小: 1M<n<10M
• 源数据集: 原始数据
• 任务类别: fill-mask

数据集详细描述

• 摘要: 该数据集包含来自EURLEX的大量文本资源（总计约179GB），适用于预训练语言模型。
• 支持任务: 支持掩码语言建模任务。
• 数据结构: 数据文件格式为jsonl.xz，包含一个分割（"train"），支持的资源类型包括caselaw, decision, directive, intagr, proposal, recommendation, regulation。

数据集创建

• 来源数据: 数据集为原始数据，未经过第三方处理。
• 注释: 注释创建者为"other"，语言创建者为"found"。

使用注意事项

• 社会影响: 未提供具体信息。
• 偏见讨论: 未提供具体信息。
• 其他已知限制: 未提供具体信息。

附加信息

• 数据集管理者: 未提供具体信息。
• 许可证信息: 遵循cc-by-4.0许可证。
• 引用信息: 未提供具体信息。
• 贡献: 未提供具体信息。