National Library as Corpus: Introducing DeLiKo@DNB – a Large Synchronous German Fiction Corpus

Introduction

Fiction books are weakly represented in the German Reference Corpus DeReKo (Kupietz et al. 2010, 2018). This is primarily due to the tremendously higher costs associated with licensing and converting raw fiction data into TEI-encoded XML format, compared to newspaper articles (Kupietz et al. 2014, p. 2). In the following, we discuss how we addressed these challenges and successfully created a large extensible corpus of recent fiction books.

The Legal Challenge

Linguistics and literary studies both face the challenge that their research data is affected by third-party rights. Obtaining transferable, uniform licenses for German fiction books is particularly costly as no licensing models for non-expressive use (previously also called “non-consumptive use”, see Kamocki 2018) of entire texts as primary research data are generally established. Moreover, individual author permissions are often required, since the use of texts as research data is not covered by standard licensing agreements between authors and publishers.

Our solution to this challenge consists of two main pillars: The first leverages § 14 of the German National Library Act (DNBG) requiring submission of all digitally published media works to the DNB. The second pillar is part of the general strategy adopted for DeReKo to address legal issues through infrastructural means, following Jim Gray's (2003) famous principle: If the data cannot move, put the computation near the data (Kupietz, Diewald, and Margaretha 2022, 163ff) . In this application case, our approach is to leave the full texts in the DNB and make them accessible via an instance of the corpus analysis platform KorAP (Bański et al. 2012) running within DNB. This ensures that the data can be made available more openly than under the text and data mining exception, ¹  and be used as effectively as possible for linguistic and related research purposes, but within the limits set by copyright law. The main restriction is that the context size of each search result is limited to a maximum of 50 words.

Data and Methods

To address the second challenge, and to make conversion into high-quality TEI-XML encoded corpora feasible, we ignored PDF ebooks and limited our focus to the 273,976 books available in the XML-based EPUB format and drew a 10% random sample from these as a first step, stratified by year of publication, resulting in a sample of 26,091 ebooks. As a second step, on the occasion of the 20th anniversary of the German Book Prize in October 2024, we added all 362 digitally available longlisted titles from the past two decades to the corpus, in cooperation with the German Publishers and Booksellers Association. To convert the data to the TEI I5 format (Lüngen and Sperberg-McQueen 2012) used by DeReKo, we applied XSLT 3.0 stylesheets in three passes, via the Saxon XSLT processor and GNU Make, using the DNB SRU API to retrieve consistent metadata, a heuristic genre classifier based on this, and a MALLET (McCallum 2002) based implementation of the standard DeReKo topic domain classifier (Weiß 2005; Klosa, Lüngen, and Kupietz 2012). In subsequent steps, the TEI-XML data was converted to KorAP-XML format and annotated for POS and lemma using the TreeTagger (Schmid 1994), for POS and morphosyntactic properties using MarMoT (Mueller et al. 2013), and dependencies using MaltParser (Nivre et al. 2007). These tools were selected due to their good balance between accuracy and performance. ²  If required for specific applications, additional annotation layers can be added later, since KorAP supports any number of them. ³ 

The entire conversion and annotation process was completed in 48 hours on a Linux server with 96 cores and 1.5 TB of RAM. The composition of the resulting corpus, categorized by genres and publication years, is presented in Figure 1. Genre classifications were derived from the DNB metadata using string matching heuristics. ⁴  It is important to note that the relative proportions and the ‘representativeness’ or ‘balance’ of strata are not relevant in the case of DeLiKo@DNB. Instead, only the minimum absolute sizes of the strata are important, as users are invited to define their own virtual subcorpora. This allows them to create stratified, task-specific subsamples based on metadata constraints (see Kupietz et al. 2010; Kupietz 2016 for a detailed account of this primordial sample approach).

Using DeLiKo@DNB

DeLiKo@DNB, currently comprising 2.02 billion words, is freely accessible through the website https://korap.dnb.de (see Figure 1), leveraging the full range of KorAP features, such as flexible metadata-based definition of virtual subcorpora, multiple annotation layers, and complex annotation search in six supported query languages, simplified by a query assistant and query-by-match functionalities. The corpus is also accessible via the KorAP API, supported by libraries for R ⁵  and Python ⁶  (Kupietz, Diewald, and Margaretha 2022).

Outlook

We aim to regularly expand the corpus by incorporating newly published books. Additionally, we plan to enhance the search and analysis capabilities by integrating advancements from the long-term KorAP project, including updates to the user interface and client libraries. Further additions, improvements, and extensions, concerning e.g. the addition of books, text classifications, or annotation layers, will be driven by the demands of the user communities engaging with DeLiKo@DNB.