Cleaned LargeRDFBench dumps

Dumps for each of the LargeRDFBench datasets in two formats:

A single N-Triples (no prefixes, no unquoted numbers/booleans) file compressed with zstd

A HDT file with sidecar index file (.hdt.index.v1-1) for faster querying.

.mark files, which are JSON files storing the SHA-256 hashes of the above files and the input dump file from the original LargeRDFBench.

In addition to individual datasets, there is LargeRDFBench-all.hdt, containing the union of all triples in all datasets.

The files in this dataset were generated using the "fix" subcommand of the freqel-driver command-line utility. The files in this Zenodo dataset were generated from commit 47cea26 of said repository. Nearly all of the cleanup code, however, is from rdfit version 1.0.6, which is available from maven central.

There are four reasons to use this dataset as a substitute for the original:

Flatter file structure: there is a single file per dataset

All data is in N-Triples (no RDF/XML or Turtle syntax in .nt named files)

Valid IRIs and valid N-Triples syntax (no parsers errors, at most warnings)

Provided .hdt files are directly queryable

Since the original dumps have syntax errors and invalid IRIs, there are multiple ways to handle such issues and this dataset is one set of choices for handling them. For example, the Virtuoso endpoints of the original (as of commit 49d1401) ingest and expose invalid IRIs and langtags without complaining. Thus, there are SPARQL queries for which the results obtained using this cleaned version and the original Virtuoso endpoint bundles will differ. As far as we know, such possibility does not apply to the LargeRDFBench SPARQL queries. No triples were discarded in the cleaning process, rather triples with invalid IRIs (as per RFC 3987) and invalid language tags are mapped to valid counterparts. Literals are mostly unaffected, except for one particular syntax violation in the Affymetrix dataset: non-escaped null characters (U+0000) in lexical forms were replaced with spaces (U+0020) to make HDT files possible. The syntax fixes were made using RIt.tolerant() functionality of the rdfit library, version 1.0.6. The list of transformations (beyond flattening the file structure and storing as N-Ttiples and HDT) was:

Percent-encode characters not allowed at their current position in the IRI by RFC 3987.

If percent-encoding is not allowed at that position by RFC 3987 (e.g., port rule), the character will be erased

Erase invalid character encodings (when the binary representation is so messed up it does not appear as the wrong character but is straight up invalid UTF-8)

Replace '_' in language tags with '-' (e.g., en_US becomes en-US)

For NT/Turtle, \-escape occurrences of \r (0x0D) and \n (0x0A) inside single-quoted lexical forms.

For NT/Turtle, replace \ with \\ in any \x-escape where x is not in tbnrf"' (see ECHAR).

For NT/Turtle, identify UCHAR) escape sequences that represent an UTF-8 encoding instead of an unicode code point. Such sequences are composed of only byte-sized code points, which value sequence correspond to a valid UTF-8 sequence and where at least one such byte has a value that is the code point of a control character. Given such conditions, the sequence of UCHARs is replaced by a single UCHAR for the character encoded in UTF-8. Example: \x00C3\x0085, which corresponds to Å in UTF-8 becomes \u00C5 since U+0085 is a control character.

For NT/Turtle, @PREFIX and @BASE are rewritten to @prefix and @base

For NT/Turtle, literals true and false with any variation in case (e.g., True) are replaced

with the standard true and false.

For NT/Turtle, a lexical form followed by an <IRI> without space or with a number of ^

characters different from 2 is replaced with ^^<IRI>

For NT/Turtle, replace invalid unquoted plain literals with plain string literals. For this, the code assumes the invalid unquoted literal has no spaces (i.e., whitespace is a separator and never part of the invalid literal). Examples of this fix in action:

<an-iri> t:chromosome X becomes <an-iri> t:chromosome "X", <s> <p> 2e-3.4 becomes <s> <p> "2e-3.4" (exponent must be an integer) and <s> <p> falseful becomes <s> <p> "falseful"

Strip leading whitespace, %20, %09, %0A %0D and strip underlines at any position from IRI schemes. Affymetrix and Jamendo are affected

For Turtle/NT/TriG, replace NULL characters (U+0000) in string literals with (U+0020). Use case: only Affymetrix

Changelog

1.0.1: Re-generated LMDB.index.v1-1 to fix wrong results on queries with unbound subject, owl:sameAs predicate and bound object.

1.0.2: Added LargeRDFBench-all.hdt and sidecar index file

本数据集包含所有LargeRDFBench（LargeRDFBench）数据集的两种格式转储文件：
1. 单个N-Triples（N-Triples，无前缀、无未引用的数值/布尔值）文件，采用zstd（zstd）压缩
2. 附带辅助索引文件（.hdt.index.v1-1）的HDT（HDT）文件，用于加速查询
此外还包含.mark文件，该类文件为JSON格式，用于存储上述转储文件以及原始LargeRDFBench输入转储文件的SHA-256（SHA-256）哈希值。

除单个数据集文件外，本数据集还提供LargeRDFBench-all.hdt，整合了所有数据集中的全部RDF三元组。

本数据集的文件通过freqel-driver命令行工具的"fix"子命令生成，本Zenodo（Zenodo）数据集基于该仓库的47cea26提交版本构建。绝大多数清理代码来自rdfit 1.0.6版本，该版本可从Maven Central（Maven Central）获取。

选用本数据集作为原始数据集替代方案的四大核心理由如下：
1. 扁平化文件结构：每个数据集仅对应单个文件
2. 全量数据采用N-Triples格式（.nt命名文件中无RDF/XML或Turtle语法）
3. 符合规范的IRI与合法的N-Triples语法（无解析错误，仅可能存在警告信息）
4. 提供的HDT文件可直接用于SPARQL（SPARQL）查询

由于原始转储文件存在语法错误与非法IRI，针对此类问题存在多种处理方案，本数据集即为其中一套标准化处理选择。例如，原始数据集的Virtuoso端点（截至49d1401提交版本）会接收并暴露非法IRI与语言标签而不触发报错。因此，部分SPARQL查询在本清洗后数据集与原始Virtuoso端点数据集上的查询结果可能存在差异。据我们所知，此类差异并不适用于LargeRDFBench官方的SPARQL查询集。

本清洗过程未丢弃任何三元组，仅将符合RFC 3987（RFC 3987）规范的非法IRI与非法语言标签映射为合法形式。字面量（Literal）基本不受影响，仅Affymetrix数据集存在一处特殊语法问题：词法形式中的未转义空字符（U+0000）被替换为空格（U+0020），以确保HDT文件可正常生成。语法修复工作通过rdfit库1.0.6版本的RIt.tolerant()功能完成。除扁平化文件结构并转换为N-Triples与HDT格式外，具体转换操作列表如下：
1. 依据RFC 3987，对IRI中当前位置不允许出现的字符进行百分号编码
2. 若RFC 3987不允许对某位置进行百分号编码（例如端口规则场景），则直接删除该字符
3. 删除无效字符编码（当二进制表示损坏到无法显示为错误字符，而是直接为非法UTF-8时）
4. 将语言标签中的下划线"_"替换为连字符"-"（例如en_US转换为en-US）
5. 针对NT/Turtle格式，对单引号词法形式内的
（0x0D）与
（0x0A）进行反斜杠转义
6. 针对NT/Turtle格式，将所有不在tbnrf'"范围内的x转义中的反斜杠替换为双反斜杠（参考ECHAR规范）
7. 针对NT/Turtle格式，识别表示UTF-8编码而非Unicode码点的UCHAR转义序列。此类序列需满足：仅由字节大小的码点组成，其值序列对应合法UTF-8序列，且至少有一个字节的值为控制字符的码点。满足上述条件时，将该UCHAR序列替换为代表该UTF-8编码字符的单个UCHAR。示例：x00C3x0085（对应UTF-8编码的Å）将被替换为u00C5，因U+0085为控制字符
8. 针对NT/Turtle格式，将@PREFIX与@BASE重写为@prefix与@base
9. 针对NT/Turtle格式，将任意大小写变体的true与false字面量（例如True）替换为标准形式的true与false
10. 针对NT/Turtle格式，将未留空格或^字符数量不为2的紧跟在词法形式后的<IRI>修正为^^<IRI>
11. 针对NT/Turtle格式，将非法的未引用纯字面量替换为普通字符串字面量。此处代码假设非法未引用字面量不含空格（即空格为分隔符，不会成为非法字面量的一部分）。修复示例如下：
- <an-iri> t:chromosome X 修正为 <an-iri> t:chromosome "X"
- <s> <p> 2e-3.4 修正为 <s> <p> "2e-3.4"（指数必须为整数）
- <s> <p> falseful 修正为 <s> <p> "falseful"
12. 移除IRI方案中的前导空白字符、%20、%09、%0A、%0D以及任意位置的下划线，此修复影响Affymetrix与Jamendo数据集
13. 针对Turtle/NT/TriG格式，将字符串字面量中的空字符（U+0000）替换为空格（U+0020），仅影响Affymetrix数据集

### 更新日志
1.0.1：重新生成LMDB.index.v1-1，修复了存在未绑定主题、owl:sameAs谓词与绑定对象的查询结果错误问题
1.0.2：新增LargeRDFBench-all.hdt及其辅助索引文件