Topic model of English-language fiction, 1880-1999, with 200 topics.

A topic model of 29,341 volumes of fiction, written in English and published between 1880 and 1999. The underlying corpus was organized by Ted Underwood for an experiment on period and cohort effects in cultural change. Metadata is in finalcorpus.tsv (which also has rows for 10 volumes not actually included in the model). 

To identify volumes as fiction, we relied on the NovelTM Dataset of English-Language Fiction (https://culturalanalytics.org/article/13147-noveltm-datasets-for-english-language-fiction-1700-2009). To confirm birth years of authors and publication dates of books, we compared NovelTM metadata both to the Chicago Novel Corpus and to a copy of the US Copyright Registry, digitized by the New York Public Library (https://github.com/NYPL/catalog_of_copyright_entries_project).

The corpus itself is in cohort4.txt.gz; each line represents a roughly 10,000-word "chunk" of a document. The first 15% and last 5% of pages in each volume were discarded; the remaining pages were divided into chunks of roughly equal size. Chunk id is the first token on each line; it is formed by taking a HathiTrust volume id and adding an underscore + sequential integer (chunk number). Removing the underscore and integer produces a "document id" that can be paired to the metadata. The words in the line are not presented in original order; they are taken from HathiTrust Extracted Features, which records only page-level word counts.

The topic model was produced using MALLET (http://mallet.cs.umass.edu/index.php), and has 200 topics.

The top words in each topic are listed in the "keys" file; document-topic proportions are listed in "doctopics."

For more information on the construction of the corpus and the experiment it is designed to support, see https://github.com/tedunderwood/period-cohort and/or a permanent Zenodo object created from that repository.