FakeNarratives – First Forays in Understanding Narratives of Disinformation in Public and Alternative News Videos

Introduction: Narratives of Disinformation

Audiovisual media, such as film, TV, webseries, YouTube-videos and so on, have long become one, if not the, dominant cultural communication form of our times (Mirzoeff, 1999; Mitchell, 2001). Their role in shaping sociocultural configurations of all kinds appears incontestable. However, description and analysis of such media presents immense challenges which have so far resisted scalable solutions. Although it is becoming increasingly possible to conduct larger-scale stylistic and formal feature analysis (shot lengths, brightness, color profiles, dynamicity, etc.: cf., e.g., Heftberger, 2018), productive engagement with such media as powerful sociocultural artifacts demands in addition hermeneutic and functional interpretations as pursued in many branches of the humanities. Analyses of these kinds have not so far been possible within the digital humanities.

In an ongoing BMBF project on FakeNarratives¹ , we combine expertise, experience and tools crucial for making general advances towards these goals, practically directed at a soundly delimited and, at the same time, socially significant class of audiovisual artifacts: TV news. We address the research hypothesis that news of this kind increasingly employs strategies of audiovisual narrative that may undercut, undermine or construct ideological positions and evaluations of news content independently of what may be simply stated, for example, in accompanying spoken text. The increasing sophistication and technical possibilities available to news channels makes it possible to draw on techniques for storytelling well established in film, but this can by no means always be guaranteed to work as intended. Indeed, filmic storytelling techniques may also serve to more effectively dis-inform.

Although consideration of news reporting as narrative already has a considerably history and is nowadays hardly controversial (Sperry, 1981; Bell, 1999; Liebes, 1994; Langer, 1998; Hickethier, 2000; Dunn, 2005; Machill et al., 2006), methods for engaging specifically with their audiovisual non-verbal properties remain limited. Analyses addressing the audiovisual are overwhelmingly based on small datasets and, as is usual within the humanities, are performed interpretatively. Consequently, just how widespread ‘narrativization’ strategies are, and to what effect and aims they are being applied, urgently demands in depth and larger-scale research from within the digital humanities.

Case study: Tagesschau vs. Bild TV

The current paper presents some early results of the FakeNarratives project illustrating how narrative audiovisual strategies might begin to be addressed at variable scales. Several distinct kinds of narrative strategies have been discussed in the journalism and newsreporting literature, but for the purposes of our first case study we focus on the production of ‘individual-centered narrative’ by means of editing devices such as showing layperson’s talk, individualized events and actions, close- ups of individual faces, and emotion shown either visually or exhibited in accompanying language. These features appear to be widely used across alternative and public news outlets (Vettehen et al., 2008). Our research hypothesis is that both kinds of news outlets use narrative strategies to some degree, but that there may still be significant differences in why and how often they are actually used. We consider it highly implausible that the use of narrative strategies can be reduced to any binary “good vs. bad” or “informing vs. dis-informing” characterisation and so more finely discriminating accounts are in any case necessary. The basic challenge we address here is then to set out how hypotheses concerning narrative construction may be made amenable to larger-scale data analysis.

Here we select some of the most commonly discussed narrative features in news that contribute to highlighting individual stories, a technique that can effectively increase the viewers' memories of news contents and their perception of news severity (Aust & Zillmann, 1996; Zillmann & Brosius, 2012) . We examine whether and how these features are presented in both public and online alternative news channels. In particular, we consider how these features are employed to individualise, personalise or emotionalise people shown in news videos. In other words, we seek to operationalise just what constitutes the narrative strategy of individualisation so that we can explore how the strategy is used across our selected news channels.

What’s ahead: Automatic annotation and exploratory visualization tool

The results of the previous section are encouraging concerning the use of more abstract audiovisual analyses as a means for revealing differences and similarities between audiovisual materials in ways that naturally relate to issues of narrativization. The characterizations of our selected news media clearly indicate where differences in broader strategies are occurring. It is also nevertheless clear that for more reliable and extensive results, expanding both the kinds of audiovisual materials analyses and the kinds of strategies at issue, it will be essential to move beyond time-intensive manual annotation. Augmenting this work further by the development and application of automatic and semi-automatic annotation is consequently a further central goal of the FakeNarratives project, for which we adopt a two-pronged approach combining foci on visualization and automated analysis.

We now combine a range of computational techniques for which there are already good solutions available, such as shot detection, face detection, etc., into a flexible tool named Zoetrope, supporting the progressive annotation of larger data sets by means of a more interactive interface (see Figure 3). Zoetrope enhances the existing landscape of tools for audiovisual annotation and analysis (for an overview see Pustu-Iren et al., 2020), as it is not only capable of visualizing automatic annotations, but also provides some basic functionalities for querying the video, for instance for specific key words or named entities. By these means, annotation becomes an activity that can leverage both automatic results of audiovisual processing and more abstract characterizations of data in terms such as cohesive structures as used above.

The current Zoetrope 1.0 prototype already allows researchers to search for keywords ( segment 1). Any keyword can be used as a query, which will then be searched in the spoken (Mozilla DeepSpeech²  with a German model by Agarwal and Zesch³ ) and written (scene detection framework easyOCR⁴ ) language of the video. We also embed the query using word embeddings⁵  so that we can find words semantically related to the query as well. Via a slider, the similarity threshold can be set looser or more exact (=100%). The results of such a query are visualized in segment 2, which basically uses a timeline metaphor. Query results found in spoken language are rendered in green, results in written language are rendered in red. Note that for the query “Krieg” we also find semantically related concepts such as “Truppen”, “Invasion” or “Militär”, due to the semantic similarity threshold, which was set here rather low, at 50%. In segment 3, we see some exemplary automatically determined features, including face detection and audio analysis by means of a spectrogram. Finally, segment 4 shows an interactive video player, which can be navigated by means of the timeline or by clicking on specific results, such as a keyword or a face. Visible features, such as written keywords or faces, can also be rendered with their bounding box withing the video.

The tools developed within the project will continue to pursue advanced visualization and more extensive automatic analysis. The latter goal is supported by our project partners in Hanover and their continuing development of the web-based audiovisual analytics platform TIBAVA (TIB AV- Analytics).

Conclusions and Outlook

We have shown in this paper some of the benefits that may accrue when the analysis of audiovisual media can build on more abstract discourse patterns carried out at scale. We have argued that to take this further, however, a progressive increase in computational support is necessary. To this end, we have set out some first steps taken towards bridging some of the gaps between current technical feature processing possibilities and discourse-related characterizations of data. This is being enabled first and foremost by an interactive tool which already seamlessly incorporates a variety of state of the art processing techniques relevant for more abstract annotation practices. We believe such a tool will greatly speed up the manual annotation that our current corpus analysis relies on and will allow us to discover and annotate many more narrative strategies in the future. Once the tool is beyond its prototype stage, we also plan to release it for the DH community, which might use it to search other video material than news videos, for instance YouTube Let’s Play videos for game studies, or any movie or TV series for film studies scenarios. A live demo of the prototype will be included in the presentation at DHd 2023.