Data set to train a natural language classifier able to differentiate between 15 topics relevant to biodiversity informatics

<strong>Scope and size</strong><br> This data set is used to train a natural language processing classifier. The classifier shall be able to differentiate between 15 topics relevant to biodiversity informatics. The list of relevant topics was adapted from Searls (2012). The data set was split into training data, testing data (for tweaking and unit-testing the classifier) and validation data. Each data set is stored as PDF files in a separate directory. Training data (5494 pages) Test data (977 pages) Validation data (215 pages) <strong>Data sources and licenses</strong><br> Details about the licenses for each data set can be found in the corresponding directories. Training data was compiled from MIT OpenCourseWare resources provided by MIT under a Creative Commons BY-NC-SA License. Testing data was compiled from MIT OpenCourseWare exams, provided by MIT under a Creative Commons License BY-NC-SA. Validation data was compiled from Wikipedia, provided under a Creative Commons License by Wikipedia editors and contributors. <strong>Topic references</strong><br> Each topic references one or more MIT OpenCourseWare courses: Algorithms (Demaine, and Devadas, 2011) Artificial Intelligence (Winston, 2010) Building Dynamic Websites (Abelson, and Greenspun, 2003) Computational Biology (Kellis, 2015) Computer Graphics (Matusik, and Durand, 2012) Computer Science and Programming (Bell, Grimson, and Guttag, 2016) Databases (Madden, Morris, Stonebraker, and Curino, 2010) Data Structures (Demaine, 2012) Digital Image Processing (Clifford, Fisher, Greenberg, and Wells, 2007; Golland, 2005) Machine Learning (Singh, Jaakkola, and Mohammad, 2006) Machine Structures (Morris, and Madden, 2009) Natural Language Processing (Berwick, 2003; Collins, and Barzilay, 2005) Parallel Computing (Edelman, 2011) Software Engineering (Jackson, and Devadas, 2005) Structure and Interpretation of Computer Programs (Miller, and Goldman, 2016). <strong>References</strong> Harold Abelson, and Philip Greenspun. 6.171 Software Engineering for Web Applications. Fall 2003. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Ana Bell, Eric Grimson, and John Guttag. 6.0001 Introduction to Computer Science and Programming in Python. Fall 2016. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Robert Berwick. 6.863J Natural Language and the Computer Representation of Knowledge. Spring 2003. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Gari Clifford, John Fisher, Julie Greenberg, and William Wells. HST.582J Biomedical Signal and Image Processing. Spring 2007. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Michael Collins, and Regina Barzilay. 6.864 Advanced Natural Language Processing. Fall 2005. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Erik Demaine. 6.851 Advanced Data Structures. Spring 2012. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Erik Demaine, and Srini Devadas. 6.006 Introduction to Algorithms. Fall 2011. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Alan Edelman. 18.337J Parallel Computing. Fall 2011. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Polina Golland. 6.881 Representation and Modeling for Image Analysis. Spring 2005. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Daniel Jackson, and Srini Devadas. 6.170 Laboratory in Software Engineering. Fall 2005. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Manolis Kellis. 6.047 Computational Biology. Fall 2015. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Robert Miller, and Max Goldman. 6.005 Software Construction. Spring 2016. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Samuel Madden, Robert Morris, Michael Stonebraker, and Carlo Curino. 6.830 Database Systems. Fall 2010. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Wojciech Matusik, and Frédo Durand. 6.837 Computer Graphics. Fall 2012. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Morris, Robert, and Madden, Samuel. 6.033 Computer System Engineering. Spring 2009. Massachusetts Institute of Technology: MIT OpenCourseWare, http://hdl.handle.net/1721.1/118791. License: Creative Commons BY-NC-SA. David B. Searls. An online bioinformatics curriculum. 2012. PLoS computational biology, 8(9), p.e1002632. Rohit Singh, Tommi Jaakkola, and Ali Mohammad. 6.867 Machine Learning. Fall 2006. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. Patrick Winston. 6.034 Artificial Intelligence. Fall 2010. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA.