Dataset of sentence pairs from renowned authors in North American literature

This dataset contains pairs of sentences taken from 35 literary works by three renowned authors of North American literature, namely: William Cuthbert Faulkner, Ernest Miller Hemingway and Philip Milton Roth.

There are three versions of the dataset:

1 - Pairs of pre-processed sentences with removal of punctuation, removal of alphanumeric characters and normalization of words in lowercase;
2 - Pairs of pre-processed sentences including the removal of stopwords;
3 - Pairs of pre-processed sentences including the removal of stopwords and the lemmatization of words.

The datasets were created inspired by a Kaggle challenge of identifying duplicate sentence pairs. Each dataset contains 72600 sentences by each author, 72000 being reserved for training and validation of the LSTM Siamese neural network used and 600 for testing/prediction.