Web Archive of Independent News Sites on Turkish Affairs derivatives

Derivatives of the Web Archive of Independent News Sites on Turkish Affairs collection from the Ivy Plus Libraries Confederation. The derivatives were created with the Archives Unleashed Toolkit and Archives Unleashed Cloud. The <strong>ivy-12911-parquet.tar.gz</strong> derivatives are in the Apache Parquet format, which is a columnar storage format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See this notebook for examples. <strong>Domains</strong> <pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre> Produces a DataFrame with the following columns: domain count <strong>Web Pages</strong> <pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre> Produces a DataFrame with the following columns: crawl_date url mime_type_web_server mime_type_tika content <strong>Web Graph</strong> <pre><code class="language-java">.webgraph()</code></pre> Produces a DataFrame with the following columns: crawl_date src dest anchor <strong>Image Links</strong> <pre><code class="language-java">.imageLinks()</code></pre> Produces a DataFrame with the following columns: src image_url <strong>Binary Analysis</strong> Audio Images PDFs Presentation program files Spreadsheets Text files Word processor files<br> The <strong>ivy-12911-auk.tar.gz </strong>derivatives<strong> </strong>are the standard set of web archive derivatives produced by the Archives Unleashed Cloud. <strong>Gephi </strong>file, which can be loaded into Gephi. It will have basic characteristics already computed and a basic layout. <strong>Raw Network</strong> file, which can also be loaded into Gephi. You will have to use that network program to lay it out yourself. <strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content. <strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.本数据集为常春藤盟校图书馆联盟（Ivy Plus Libraries Confederation）收录的土耳其事务独立新闻网站网络档案衍生数据集。所有衍生文件均由Archives Unleashed Toolkit与Archives Unleashed Cloud生成。 名为**ivy-12911-parquet.tar.gz**的衍生文件采用Apache Parquet列式存储格式。此类衍生文件普遍体积小巧，可在本地设备上直接处理，亦可轻松转换为Pandas数据框（Pandas DataFrames），相关使用示例可参考配套笔记文档。 **域名统计** 通过以下Java代码可生成对应数据框： java .webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc) 该数据框包含以下字段：`domain`（域名）、`count`（计数） **网页内容** 通过以下Java代码可生成对应数据框： java .webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content")) 该数据框包含以下字段：`crawl_date`（爬取日期）、`url`（统一资源定位符）、`mime_type_web_server`（服务器MIME类型）、`mime_type_tika`（Tika识别的MIME类型）、`content`（正文内容） **网页图谱** 通过以下Java代码可生成对应数据框： java .webgraph() 该数据框包含以下字段：`crawl_date`（爬取日期）、`src`（源节点）、`dest`（目标节点）、`anchor`（锚文本） **图片链接** 通过以下Java代码可生成对应数据框： java .imageLinks() 该数据框包含以下字段：`src`（源节点）、`image_url`（图片URL） **二进制文件分析**类别涵盖以下文件类型：音频文件、图片文件、PDF文件、演示文稿文件、程序文件、电子表格文件、文本文件、文字处理文件。 名为**ivy-12911-auk.tar.gz**的衍生文件为Archives Unleashed Cloud生成的标准网络档案衍生套装，包含以下内容： 1. **Gephi文件**：可直接导入Gephi可视化工具，已预先计算图谱基础特征并完成基础布局。 2. **原始网络文件**：同样支持导入Gephi，但需手动使用该工具完成图谱布局。 3. **全文本文件**：该文件中，网络档案集合内的每个网站的全文本将单独占一行，同时附带爬取时间、所属域名及内容完整URL等元数据信息。 4. **域名计数文件**：纯文本文件，记录本次网络档案采集到的所有域名的出现频次。