madmoonslice/German-Instruct-Icarus

---
license: apache-2.0
language:
- de
- aa
task_categories:
- text-generation
size_categories:
- 1B<n<10B
tags:
- chemistry
- finance
- biology
- legal
- music
pretty_name: Icarus Dataset
---

# German-Instruct-Icarus Dataset

## Overview
The **German-Instruct-Icarus** dataset is a high-quality, fully copyright-free collection of 10 million German-language examples, carefully curated for instruction tuning, fine-tuning, and pre-training of large language models (LLMs). It emphasizes natural dialogue, conversational patterns, and diverse textual interactions. All data is sourced exclusively from public domain and openly licensed materials, including transcribed dialogues from public domain films, excerpts from copyright-free books, and anonymized emails/chats from open repositories. The dataset has been rigorously cleaned to remove any pornographic, illegal, or copyrighted content, making it safe and ethically sound for broad AI research and development.

## Data Sources
- **Public Domain Books**: Selected passages rich in narrative and dialogue from copyright-free German literature.
- **Film Dialogues**: Accurate transcriptions of conversations from public domain films.
- **Emails and Chats**: Anonymized real-world exchanges sourced from open-source archives and public datasets, reflecting authentic everyday German communication.

All materials were primarily extracted from Common Crawl and other verified public repositories, with strict verification that every item is copyright-free.

## Dataset Statistics
- **Language**: German (Deutsch)
- **Size**: 10 million examples
- **Format**: Instruction-following pairs and conversational sequences optimized for supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and general language modeling.
- **Cleaning**: Comprehensive filtering to exclude sensitive, illegal, or inappropriate content.

## Usage
Ideal for:
- Instruction tuning of German LLMs
- Building conversational AI and chatbots in German
- Research in dialogue systems, sentiment analysis, and machine translation

Load the dataset on Hugging Face:
```python
from datasets import load_dataset
dataset = load_dataset("madmoonslice/German-Instruct-Icarus")
```

## Credits
- **Nikias Ochs**, CEO of Aquatix AI Labs: Led the sourcing of copyright-free materials from Common Crawl and oversaw the extensive cleaning process to remove pornographic and illegal content.
- **Martin Manzek**: Handled transcription of film dialogues and audio-to-text conversion from copyright-free sources.
- **Stefan Weißner**: Curated copyright-free books and sourced emails/chats from open, public repositories.
- Special thanks to the many public domain archives and copyright-free websites that provided the raw materials.

## Future Plans
An English version is in preparation and will be released soon, followed by Russian and French editions to support multilingual model development.

The dataset is released under a permissive open license (e.g., CC0 or MIT) to promote innovation in the open-source AI community. For questions or contributions, please contact Aquatix AI Labs or open an issue on the repository.