Operationalizing operationalizing

Introduction

In this paper we suggest a comprehensive account of operationalization. The goal is to define a workflow which specifies the normative components that ensure that the results match the research question. The focus here lies on epistemic norms – as opposed to social norms – which can be understood as rules that govern processes, thoughts, and actions that ought to be followed in order to produce the intended result (cf. Wedgwood 2018). This workflow is based on a measurement workflow, which is based on input-process-output components (cf. Mari, Wilson, and Maul 2023). In order to make the workflow implementable in Digital Humanities research, we need to identify the components of a generalized method for getting from a research question to its answer. While some of the components have already been discussed in Digital Humanities’ research on operationalization, we propose to put more focus on explication, measurement, and validation.

In our view, measurement and validation are underdeveloped in the Digital Humanities. This becomes apparent when looking at discussions on operationalization. Before doing that we briefly line out our understanding of central terms in the discussion

A remark on theory

In order to develop our account, we will make the following assumptions:

The term theory is both much needed and much contested, not only in the Digital Humanities, but also in the philosophy of science, which often leads to confusion, since natural sciences (especially physics) consider very general statements and laws, which rarely, if at all, play a role in the humanities. For this reason, we will adopt a minimal reading of theory that simply says, that a theory is a collection of statements that forms the background knowledge, which is needed to do science. This may include empirical assumption, factual knowledge, formalisms, inference schemes, etc.

A mini-theory is a small set of concepts and statement formulating the relationship between them, e.g. the necessary and sufficient conditions for the application of said concept.¹  A mini-theory could be understood as an explicit explanation of a concept and how it should be used. In contrast to theory – especially in the humanities – we consider a mini-theory to be a precise formulation of the means to identify, classify and understand a phenomenon in a given context. Example: The notions of narratological concepts as detailed in the Living Handbook of Narratology (Hühn et al. 2021).

Operationalization

The leading question for scientific workflows involving the analysis of data is this:

(Q) How do we get from theory to empirical data?

In the case of text based digital humanities, we can specify the scope of theory and data further. Theory and text depend on the disciplinary background of the research question. For example, in Computational Literary Studies approaches, theory means literary theory and empirical data respectively text.

(Q') How do we get from literary theory to text?

A first approach to address this question attempted by Franco Moretti, who introduced operationalization to the discourse of the digital humanities with his seminal paper “Operationalizing or, the Function of Measurement in Modern Literary Theory” (Moretti 2013). In this paper, Moretti goes back to Bridgman’s ([1927] 1958) original conception of operationalization and applies it to literary concepts. His answer to question Q’ famously is:

“Operationalizing means building a bridge from concepts to measurement, and then to the world. In our case: from the concepts of literary theory, through some form of quantification, to literary texts.” (Moretti 2013, 1)

We must imagine a bridge that is built by operations, which take us from the (qualitative) concepts of literary theory to (quantitative) measurement results. This implies existence of a simple and easy way to connect a concept C to an indicator I, resulting in a straightforward measure, which allows inferences, like the more I, the more C. Finally numbers are assigned to produce a measurement metric, be it by counting words or by using more elaborate techniques, like network analysis. In his paper, Moretti gives examples to show how this can be done, but he does not provide a detailed and generalized method that can be followed in different contexts. We therefore specify our question even further:

(Q’’) How does a generalized method look like that gets us from literary theory to text?

In order to tackle this question, we have to address at least three ambivalent aspects of operations that are at least implicitly underlying understandings meaning of operationalization:

Operations as activities. Operations are things that are done, usually involving regulated sets of steps that specify a process, thus leading to pipelines and workflows.

Operations as definitions to work with. Operations are the result of a thought process that fix the empirical meaning of a concept, thus leading to the definition of a concept.

Operations as processes. Operations are parts of the specification of measurement processes that guarantee epistemic warrant, i.e., what is measured conforms with what is intended to be measured, thus leading to the validity of a measurement process.

In digital humanities research operationalization often only means (a), i.e. that in the research design some kind of workflow or pipeline is developed, that is used to automatically extract some features from the empirical material (text). Often literary concepts are adopted uncritically, when, in fact, critical reflection on the usage of the concepts in the sense of (b) is needed to guarantee that the workflow returns results that (c) are actually valid. The last point holds even then, when statistical validity metrics are incorporated in the workflow, since those tests can only answer how good the machine learning model or the algorithm worked, but not, if the quantified results translate back to literary concepts. This objection follows the main line of argumentation of Pichler and Reiter (2022), who argue for the importance of validity while maintaining its untenability for many real-world scenarios in the digital humanities.

Overall, we understand operationalization as a workflow or process in the sense of (c), specifically in terms of the scientific input-process-output workflow (Mari, Wilson, and Maul 2023, see Fig. 1). In the following, we will introduce explication, measurement, and validity as components that make this workflow consistent and appropriate for the respective scientific context.

Explication

The starting point of operationalization in our account are concepts which are transformed to empirical mini-theories that fit the research task at hand.

To reach this goal, we resort to Carnapian explication (cf. Carnap 1950; Brun 2016) which allows us to accommodate various epistemic goals and permits different concept variations for different purposes.

Explications involve two areas, the one of the explicandum and the one of the explicatum. These domains are sometimes viewed as pre-scientific and scientific uses of the term, but we can assume that explications can occur in any area. This construes a more exact explication in a more restricted or specialized area. For instance, moving from chemistry to molecular quantum chemistry or literary theory to computational literary theory (CLS).

Measurement

Measurement can be thought of as a process that takes some inputs and returns some outputs, where the input is some kind of empirical data and the output a value, see Fig 1a. But not any arbitrary process is suitable for measurement, it is necessary to specify the properties of the process, which will result in a procedure which is designed for that specific task (Mari, Wilson, and Maul 2023, 27), see Fig. 1b.

Validation

A common conception of validation is the following:

(M) measuring what is intended to be measured

A e.g. thermometer is intended to measure the actual temperature of an object, where temperature is a property of said object. This notion seems to be sufficiently clear, but on closer inspection it turns out that the terms in (M) are underspecified. How do we determine ‘measurement’, ‘measure’, and ‘intended’? What exactly are we referring to, when we use those terms?

‘Measurement’ is ambiguous at least respective its target (Mari, Wilson, and Maul 2023; Maul, Mari, and Wilson 2019). Measurement may either denote a measuring process or the result of such process. To avoid misunderstandings, we will adopt a strict terminology which discriminates measurement processes and measurement results. Where process means a set of operations which are implemented as an instrument, that can range from physical artefacts, like thermometers, to computational devices, like machine learning algorithms. The result is an assigned value on an appropriate scale.

Calibration

Calibration denotes the inferential activity of assigning values to the measurement instrument indications and thus producing the measurement results, where:

“A measurement outcome (or ‘measurement result’) is a knowledge claim attributing one or more parameter values to the object or event being measured, a claim that is inferred from one or more instrument indications along with relevant background knowledge.” (Tal 2017, 2)

Calibration links the outcomes of the measurement process to the epistemic claims about the object that is measured, this step is necessary since the indications of the measurement instrument are not of the same kind as the properties of the object, see Fig. 1c.

The scale of a thermometer e.g. presents its indications visually a further step is needed to ensure that those values can be used to represent temperature. The same applies to literary concepts, like, in Moretti’s case character space. Network analysis or word counts provide indicators that need to be mapped to the possible values of the intended property being measured.

Operationalizing operationalized

We can now revise Moretti’s picture of operationalization as a simple measurement process, that connects theory to world directly (see Fig. 2a) and replace it with the refined scheme developed above (Fig. 2b).

For our account of operationalization we consider operations as activities that are performed on some type of input, which return some type of outputs. The kind of activity performed in operations can vary wildly, ranging from simple reasoning about concepts over doing physical activity to actual computation. In constructing an operationalization workflow we denote operations as connected boxes:

We are finally able to put all the pieces together and construct a workflow that starts with concepts and ends with measurement results.

The operationalization workflow has two sources: (i) The concept(s) originating in the source domain (humanities, literary theory, CLS) and (ii) the empirical data / text. Starting with the concepts, the workflow follows these steps:

The concept is explicated, resulting a mini-theory, consisting in a list of conditions and positive/negative examples.

The conditions are adjusted to construct a catalogue of criteria which are a) used to formulate a specification for the measurement procedure and b) help to construct the scale used later for the calibration function.

Data is processed in the measurement procedure resulting in indicators

The indicators are calibrated resulting in values

The values are assigned and result in candidates for measurement results

The results are assessed and, if necessary, returned to the explication step, iterating and adjusting the workflow until the evaluations are passed

¹ Mini-theories could be considered as models, but since the term model is also used in other contexts, we prefer this term to mark a distinction between the various usages.