LORELEI Somali Representative Language Pack - Monolingual and Parallel Text

<h3>Introduction</h3><br>
<p>LORELEI Somali Representative Language Pack - Monolingual and Parallel Text was developed by the Linguistic Data Consortium (LDC) and is comprised of approximately 13 million words of monolingual Somali text, approximately 800,000 of which are translated into English. Another 100,000 words are also translated from English into Somali.</p><br>
<p>The LORELEI (Low Resource Languages for Emergent Incidents) Program was concerned with building human language technology for low resource languages in the context of emergent situations like natural disasters or disease outbreaks. Linguistic resources for LORELEI include Representative Language Packs and Incident Language Packs for over two dozen low resource languages, comprising data, annotations, basic natural language processing tools, lexicons and grammatical resources. Representative languages were selected to provide broad typological coverage, while incident languages were selected to evaluate system performance on a language whose identity was disclosed at the start of the evaluation.</p><br>
<h3>Data</h3><br>
<p>Data was collected in the following genres: discussion forums, news, reference, social network and weblog. Both monolingual text collection and parallel text creation involved a combination of manual and automatic methods, which are detailed in the included documentation. All harvested content was initially converted from its original HTML form into a relatively uniform XML format.</p><br>
<p>XML data is presented in two formats: a "homogenized" XML format that preserves the minimum set of tags needed to represent the structure of the relevant text as seen by the human web-page reader and a fully segmented and tokenized version of the text. All text data is encoded as UTF-8.</p><br>
<p>Also included in this release are two tools: one to recreate original source data from the processed XML material and the other to condition text data users download from Twitter.</p><br>
<h3>Acknowledgement</h3><br>
<p>This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011-15-C-0123. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of DARPA.</p><br>
<h3>Samples</h3><br>
<p>Please view the following samples:</p><br>
<ul><br>
<li><a href="desc/addenda/LDC2018T11.ltf.xml">LTF Somali Source</a></li><br>
<li><a href="desc/addenda/LDC2018T11.eng.ltf.xml">LTF English Translation</a></li><br>
<li><a href="desc/addenda/LDC2018T11.psm.xml">PSM Somali Source</a></li><br>
<li><a href="desc/addenda/LDC2018T11.eng.psm.xml">PSM English Translation</a></li><br>
</ul><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2010-2016 afweyne.com, © 2002-2007, 2009-2010 Agence France Presse, © 2000 American Broadcasting Company, © 2012-2016 aqbaar.com, © 2014-2015 BaligubadleMedia.com, © 2016 BBC, © 2015-2016 BooramaOnline.Com, © 2000 Cable News Network, LP, LLLP, © 2015 Cadalool, © 2015-2016 Calayaale, © 2008 Central News Agency (Taiwan), © 2015-2016 Dayaxside, © 2006 Dow Jones & Company, Inc., © 2015-2016 Gorgor Online, © 2014-2016 HAATUF.NET, © 2012-2016 Harowo.com, © 2011-2016 Ilays, © 2014-2016 Jowhar.com, www.jowharnews.so, © 2015-2016 Kismaayo, © 2015 Kismaayonews.com, © 2015-2016 Kulmiye News Network, © 2005-2006 Los Angeles Times - Washington Post News Service, Inc., © 2015-2016 Maankaab, © 2016 Mareeg.com, © 2013-2016 Markacadeey, © 2000 National Broadcasting Company, Inc., © 1999, 2005-2006, 2008 New York Times, © 2015 Ogadenworld, © 2000 Public Radio International, © 2011-2016 Radio Ergo, © 2014-2016 Radio Muqdisho, © 2008-2016 SBC, © 2015-2016 Simbanews, © 2003, 2015 Singapore Press Holdings Ltd. Co. Regn., © 2008-2016 SomaliTalk.com, © 2015-2016 starfmke.com, © 2003, 2005-2010 The Associated Press, © 2014-2016 Waaheen Media Group, © 2015 Warbaahinta Warar.net, © 2016 Warkii.com, © 2010, 2012-2016 Warfaafiye, © 2011-2013, 2016 www.hiiraan.com, © 2014-2016 Yoobsannews.com, © 2003, 2005-2008 Xinhua News Agency, © 2016, 2018 Trustees of the University of Pennsylvania