Multimodal History – Convergence of Text, Image, Diagram and Virtual Space

What is the role of images for historians – not just as visual primary sources, but also in terms of the workflow of knowledge creation and scholarly communication? How do images compare to texts – our traditional and primary modality in historical research and historiography? Must we assume an unbridgeable difference between the two modalities, or can we utilize forms of interplay between them in multimodal formats? What do other media formats and environments, such as virtual reality, contribute to the ensemble of modalities? These questions were the focus of a workshop at the Center for Theories in Historical Research, on May 9, 2025. The event was inspired by an exchange that began in 2024 at the University of Graz, initiated by Philipp Hofeneder and later continued at the DIPF – Leibniz Institute for Research and Information in Education in Berlin. The workshop held in Bielefeld was a continuation and systemization of that ongoing conversation. In this blog post, I will refer briefly to the key arguments of the presenters and give some follow-up thoughts on the potential of multimodality for epistemic and communicative practices in our discipline.
1 A hegemony of text?
In the traditional self-understanding of history as an academic discipline, the written and printed word has long occupied a privileged position. With the notable exception of art history, historical sources were largely conceived as textual artefacts well into the latter half of the twentieth century. Since then, however, images, audio and video recordings, physical objects, and 3D reconstructions have progressively found their way into the broader historian toolkit. Their non-textual qualities have gradually been acknowledged, not just in explicit art history or visual history – with interdisciplinary theory providing a robust apparatus for grasping the social and political significance of such artefacts. When the purely pictorial properties of images are at the forefront of their analysis, historians often tend to draw on classical art theory, for instance, from Aby Warburg, Erwin Panofsky, Max Imdahl, Edward Tufte, and others. At the same time, the growing proliferation of (digital) images and interactive visual formats calls for renewed methodological attention – an imperative long emphasized by advocates of the Iconic and Pictorial Turn.
Still rare, however, is a systematic reflection on the semiotic and aesthetic foundations that shape the historian’s engagement with visual media. What, after all, distinguishes a text from an image? How does visual representation shape meaning and communication? Is there truly an unbridgeable gap between the figurative order of the image and the discursive order of language, as French philosopher Sarah Kofman once suggested?1 Or how, precisely, do the two modalities overlap or interact, for example, when a caption frames an image? Furthermore, individual images and texts often only reveal their full meaning when seen in trans- and intermedial environments. Gaining a deeper insight into the epistemic frameworks and interpretive layers that matter for historical inquiry in such cases calls for consideration of an entire (historical) media ecology.
At the same time, non-linear approaches to history – present in forms such as histoire croisée, historiography attentive to the sediments of time, or poststructuralist perspectives – coexisted for decades with the predominance of linear, text-based publication formats. More recently, however, how we communicate historical knowledge has become increasingly multimodal, interactive and non-linear, amplified by the possibilities digital media offer for these forms of expression. Arguably, this communicative evolution resonates belatedly with forms of historical argumentation and narration that move beyond strictly linear patterns. As Stefan Haas has aptly characterized images: “Due to their synchronicity, they make it possible to express complex patterns of argumentation and constellations of conditions more adequately than in the linear functioning of language.”2 A similar line of reasoning is pursued by David J. Staley, who elaborates on visualization not merely in the sense of data graphics, but as a broader visual format category for scholarly inquiry, such as concept maps. “Far from serving in a subordinate role to writing,” he portrays visualizations as “meaningful conceptual tools of inquiry and insight in their own right.”3 Staley attributes this epistemic potential to the “two- or three-dimensional syntax” of visual media, which he contrasts with the “one-dimensional syntax” of prose.
Despite such analytical distinctions between text and image, their respective strengths can be strategically combined, especially in digital formats. A compelling example of such interplay is Shahzad Bashir’s innovative online book A New Vision for Islamic Pasts and Futures, which I discussed earlier on this blog.4 Bashir conceptualizes Islam as an evolving tapestry of voices that relate the faith and its histories to multiple temporalities, thereby resisting reduction to a single explanatory plane. To reflect this multiplicity, Bashir chose a hypertextual and multimodal format as a meaningful translation of his core argument. Hypertext stands for the non-linear connection of information, forming a network or branching structure of information chunks and links. Bashir’s digital book design is far more than a modern or didactic flourish. It enables readers to experience his vision of Islam as a discursive web via their movements while navigating its content. In this way, his argument unfolds both in the substance of his writing and the manner of the reader’s engagement. The process of interactive navigation itself becomes a trope, revealing Bashir’s understanding of how Islam and its histories are to be conceived. In other words, the interactive design showcases Islam as a multivocal web. In essence, this is an epistemic feature – one that Bashir could scarcely have realized without his hypertextual, multimodal and interactive book design, unless perhaps with endless descriptions and repeated contextualizing. In which case, however, readers might well disengage.
Fig. 2: The main menu displays the section web. Hovering over an entry reveals the topic of the node and highlights its connection to other parts of the book. Bashir, A New Vision, URL: https://islamic-pasts-futures.org/#allChapters (21.03.2024).
Speaking of visualization and mapping – multimodal communication plays a key role in digital historical research as well, not least through the use of data visualization. Its growing prominence in historical scholarship shapes both how historians analyse material and how they communicate their findings. Here, multimodal interplay is particularly noticeable when visualizations are used to represent quantitative textual data or when these are embedded in complex text-image constellations. This is more and more the case in digital publishing, with emerging formats such as data stories or scholarly articles that include executable code alongside interactive visualizations, opening the door to polyphonic narratives and new modes of engaging with data-driven research. This is because such formats allow readers to explore, with full transparency, the programming code and underlying data of the presented research. Filtering and selection mechanisms allow them to conduct their own analyses, occasionally using settings and tools other than those chosen by the respective authors. In other words, they may even arrive at findings unanticipated in the original work. It is this combination of high-level transparency and interactivity that turns these digital publication formats into epistemic instruments unmatched by traditional publications or static data tables.
2 Theoretical ground work as enabler of applied multimodality
Despite the diversity of this panoply of domains and forms of multimodal history, it converges around a shared insight: bringing text and image together has the potential to significantly shape how historians perceive and express their work. Yet this interplay is rarely examined through a systematic, comparative lens that considers the semiotic, epistemic and practical functions of the respective modalities.
The workshop came up with several compelling responses in this regard. In the realm of virtual reality (VR), Silke Schwandt (Bielefeld University) presented the eTaRDiS (“Exploration of Temporal and Spatial Data in Immersive Scenarios”) project.5 Unlike VR applications that aim to reconstruct the past through immersive renderings by recreating historical sites, eTaRDiS adopts a more reflexive approach. Drawing inspiration from the visualization model of Der digitale Peters,6 with its map-like representation of history, the eTaRDiS project positions historical entities (individuals, institutions, events, etc.) as abstract spheres in a virtual space. Users can interact with these spheres to access further (primarily textual) information. As Schwandt demonstrated, this setup allows for visualization and exploration of historical data and its internal relationships within a navigable virtual environment. The spatial arrangement is dynamic: entities can be filtered, rearranged and recombined. This interactivity gives free rein to the central epistemic aim of eTaRDiS: to foster critical reflection on the perspectival nature of historical knowledge and the user’s own positionality in the act of selection and interpretation. Much like Bashir’s online book, this subject-centered – I would definitely call it constructivist – approach obliges us to critically reflect on our own ways of knowledge formation. Each interactive act of choosing and filtering information compels the user to consider the relevance of specific selections and constellations. In turn, each decision leads to further possibilities of navigating the virtual knowledge space in meaningful ways. Equally significant is the experience of one’s own situatedness within the network of information units arranged in virtual space – a powerful eTaRDiS trope that actively prompts self-reflection. In my view, this resonates strongly with the notion of second-order observation as formulated by cybernetic theorists such as Heinz von Foerster: observing ourselves as we observe (history) reveals how we construct our understanding (of the past).7 In this sense, eTaRDiS foregrounds a theoretically informed, reflexive layer of historical representation. David J. Staley would call this a “meta-narrative”8 that emerges precisely through this interplay of interactive spatial visualization and textual information. The epistemic impact of eTaRDiS is inseparable from its multimodal design.
In my own talk, I explored a similar notion of historical sense-making, focusing on digital publication formats as scholarly output. Projects such as A New Vision for Islamic Pasts and Futures, the early digital landmark The Differences Slavery Made,9 or Bodies and Structures 2.010 illustrate how the multimodal form follows the function of non-linear historiography. These works emphasize multivocality, contemporality and other dimensions of historical complexity. In this context, the concept of hypertext returns to the foreground, as already discussed with Bashir’s online book. Hypertext theory has long upheld this non-linear medium as better equipped to convey complexity than the constraints of printed text. Theodore ‘Ted’ Nelson, who coined the term “hypertext,” puts it succinctly: “The problem with creating sequential documents is this: any sequence cuts connections, just as any grouping omits items. The cutting of connections is the loss of information. The problem is worsened with publication. […] Before hypertext, these problems were intrinsic to writing and publication.”11 Several historians have echoed this call to explore the epistemic potential of hypertext. Jakob Krameritsch, for instance, sees in hypertext the ideal medium for what he terms “situative scholarly narratives.”12 Building on Jörn Rüsen’s typology of historical narrative structures, Krameritsch envisions a form of storytelling that avoids master narratives and embraces provisional truths in order to reflect the fluid identities and temporalities of postmodern societies. I myself have argued that multimodal hypertext mirrors the intricate architecture of historical argumentation and historiographical narratives, particularly when multimodality is employed to render visible the constructed nature of historical knowledge and the very ‘architecture’ of historical narratives and arguments.13 This is not merely a stylistic preference but an epistemic stance.
Such theoretical interventions, however, can only unfold their full analytical force if they clarify the underlying aesthetic and semiotic principles at play. This allows us to reason more precisely about how different modalities can be meaningfully combined, how their respective affordances can be leveraged and their limitations offset by the strengths of the other. There are numerous ways to approach this question, some of which surfaced during the workshop. Gunther Kress and Theo van Leeuwen laid the groundwork for interdisciplinary multimodality studies with their seminal book Reading Images.14 Here, they explore the “grammar” of images with rules and structures that govern how visual elements relate to one another and to the accompanying text. To me, one of the most promising developments following this foundational work is the field of Bildlinguistik (‘image linguistics’), which has gained traction notably in German-speaking scholarship. Hartmut Stöckl, for example, offers a nuanced analytical framework to distinguish the semiotic, perceptual, semantic, and pragmatic effects of language (spoken and written) and images.15 According to Stöckl, images offer holistic perception: they present spatial relationships at a glance and in concrete, immediate form. In their gestalt appearance, images can seem explicit, providing a simultaneous overview of all visual elements. Semantically, however, they tend to be vague and underdetermined – we often need textual explanation to make pictorial meaning legible. Language, by contrast, unfolds a successive perception and provides greater semantic precision. Understanding these differences and how they can be brought into productive interplay is crucial to developing genuinely multimodal forms of historical scholarship.
In her presentation, Linda Freyberg (DIPF – Leibniz Institute for Research and Information in Education, Berlin) took this line of thought a step further by discussing data visualizations as tools of reasoning and meaning-making. Drawing on Charles Sanders Peirce’s semiotic theory and philosophy, she emphasized that all thinking is conducted through signs. This cognitive dimension was linked to the ‘icon,’ one of Peirce’s fundamental sign types: an icon signifies by resemblance or similarity.16 In this sense, a diagram structure, for example, mirrors the logical, causal or other relation between its data points. This aligns with what Peirce calls “diagrammatic reasoning”: the process of reasoning by means of visual representation.17 When concepts and ideas cohere in a particular way, and this coherence is made visible in an isomorphic diagram, the visualization itself becomes a form of iconic signification. By drawing on Peirce, Freyberg added a crucial anthropological dimension to the discussion. She suggested that Digital History could embrace this perspective more fully by treating visualizations as epistemic objects – or tools that not only present information but stimulate reasoning, foster communication, amplify cognition, and help to untangle complexity. In my view, Freyberg’s approach makes a powerful case for discarding the persistent myth that visualizations and other images are mere illustrations. On the contrary: used as analytical and communicative tools, diagrams open up epistemic insights that neither text nor spoken language can achieve on their own.
Florian Windhager (University for Continuing Education Krems) pursued a similar line of thought in his talk on the role of texts, images and diagrams in broader debates in the digital and non-digital humanities. He argued that visualization offers scholars the potential to engage in critical, polyvocal “viscourses.” The term ‘viscourse’ was originally coined by Karin Knorr-Cetina, based on her praxeological observation of physicists at CERN.18 There, scientists used visualizations of their experiments and measurements not merely to illustrate results, but as key devices for in-depth project discussions. They served as instruments for refining methodology and planning subsequent research steps. These visual communication practices gave rise to distinct discursive forms – hence the neologism ‘viscourse.’ According to Florian, scholars in the humanities, particularly those engaging with data-driven research, could benefit from this model as well. Yet, he observed that disciplinary tensions often stand in the way. He identified a clash of values: on the one hand, the humanities are marked by a long-standing iconoclasm with a tendency to treat visual media as subordinate and limit their role to illustration. On the other hand, he diagnosed logoclastic tendencies, especially in computational research, to frequently dismiss texts as verbose, slow or overly speculative. These divergent traditions meet in interdisciplinary spaces such as the digital humanities, where they can cause a perceived yet unnecessary antagonism.
I find this diagnosis particularly thought-provoking, as it invites reflection on our own scholarly self-conceptions, academic socialization and the epistemic cultures of our disciplines. Are we limited by inherited preferences that derive from tradition rather than necessity? Is the tension between text and image perhaps a pseudo-conflict? Florian argued precisely this. Rather than viewing such friction as unfortunate noise, he suggested reading it as a sign of unbalanced preferences and uneven multimodal literacies. This would enable us to become more aware of monomodal strengths and limitations, and gain a deeper understanding of multimodal standards and solutions. Visualizations, in this view, do not displace texts but rather expand our critical toolbox, notably when it comes to (1) analysing subject matter, (2) evaluating the epistemic potential of that engagement, and (3) acting on/reacting to the results in order to improve research. Windhager sees this as the true ‘viscursive’ potential of visualizations: their capacity to become instruments of critique. He also added an anthropological dimension to his argument, suggesting that the human mind operates as a genuinely multimodal, hypertextual-visual content meshwork. Our complex thoughts are not structured along a single linear thread, but take the form of multidimensional, multimodal cognitive models. To represent them adequately, we also require multimodal media ensembles. Clearly, individuals differ in their preferences and even in their cognitive predispositions towards (more) textual or (more) visual formats. There is no one-size-fits-all solution. That is not what Windhager claims. Instead, he invites us to generally embrace multimodality as a default mode of cognition, one that in individual cases may lean towards one end of the spectrum or the other.
3 Conclusion: There is no dichotomy and no need to choose sides – just potential to be explored
The various insights offered by the workshop presentations demonstrated that modality boundaries are highly permeable. Text, image, diagram, and virtual reality can be combined into new multimodal forms, each with its own distinctive (semiotic, pragmatic, etc.) effect. This resonates with a key assumption in multimodality research: hybrid formats are the norm rather than the exception. Even this blog post blends text and image simply by means of visual cues, including headings that differ from the body text in size, weight and position.
Multimodal approaches such as Bildlinguistik, Peircean semiotics and cognitive studies of information processing constitute valuable tools when it comes to engage with historical sources. Indeed, many other approaches could be added to the mix; those highlighted in the workshop should be understood as an invitation to deepen the dialogue around multimodality rather than as an exhaustive framework. At the same time, they offer practical models for exploring historical information and reflecting on our practices. In addition, multimodal formats provide new ways of expressing and communicating non-linear histories, allowing for what Staley has called epistemic “meta-narratives”.
That said, however, this potential can only be exploited by deliberately cultivating multimodal literacy. This includes not only the necessary theoretical knowledge to understand multimodal ensembles but also the practical skills to produce hybrid media formats. It is a prerequisite for engaging with multimodal possibilities in a reflective, informed way. At the same time, media-rich publications are not ‘better’ by default simply because they are more visual or digital. As Bashir’s example shows, multimodal design must be driven by conceptual and argumentative decisions. Multimodal practices must be purposeful, they are not self-evident. They succeed only when supported by a broader openness and a greater degree of literacy within the historical discipline.
Christian Wachter is a research associate in the Digital History Working Group and a member of the Center for Uncertainty Studies (CeUS) at Bielefeld University. He is also a member of the Center for Theories in Historical Research. His academic interests include hypertext as a digital publication format in the humanities, theories and methods of the digital humanities, theory and methods of history, and the European cultural history of politics (early 20th century). His book Geschichte digital schreiben was published in 2021 by Transcript. He is currently investigating political discourse in the Weimar Republic, using digital techniques. ORCID: https://orcid.org/0000-0003-2937-0868


This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
- Sarah Kofman, Mélancolie de l’art. Paris: Éd. Galilée, 1985. [↩]
- My translation from the German original: “Sie geben die Möglichkeit, aufgrund ihrer Synchronität komplexe Argumentationsmuster und Bedingungskonstellationen adäquater denn in der linearen Funktionsweise von Sprache auszusagen.” Stefan Haas, “Vom Schreiben in Bildern. Visualität, Narrativität und digitale Medien in den historischen Wissenschaften”, zeitenblicke 5, no. 3, (2006), paragraph 23, URL: https://dipp.archiv.hbz-nrw.de/journals/https://www.zeitenblicke.de/2006/3/Haas/ (30.06.2025). [↩]
- David J. Staley, Computers, Visualization, and History. How New Technology Will Transform Our Understanding of the Past. 2nd edition. Armonk: Sharpe, 2014, 39-40. [↩]
- Christian Wachter, “Redefining Boundaries by Multivocality: The Synergy of Content and Form in Shahzad Bashir’s Online Book A New Vision for Islamic Pasts and Futures.” Geschichtstheorie am Werk (26.03.2024), URL: https://doi.org/10.58079/w3xc (31.07.2025). [↩]
- https://www.uni-bielefeld.de/fakultaeten/geschichtswissenschaft/abteilung/arbeitsbereiche/digital-history/projects/etardis/ (31.07.2025). [↩]
- Arno Peters et al., Der Digitale Peters. Arno Peters Synchronoptische Weltgeschichte. DVD-ROM, program version DDP-2010-03. Wiener Neudorf, Wiesbaden: ÖBZ und Büro W, 2010. [↩]
- von Foerster, Heinz: Cybernetics of Cybernetics. “The Control of Control and the Communication of Communication”, Urbana, IL: Univ. of Illinois. 1974. [↩]
- David J. Staley, Computers, Visualization, and History. How New Technology Will Transform Our Understanding of the Past. 2nd edition. Armonk: Sharpe, 2014, 156. [↩]
- William G. Thomas III and Edward L. Ayers, “The Differences Slavery Made. A Close Analysis of Two American Communities”, URL: http://www2.vcdh.virginia.edu/AHR/ (30.07.2025). [↩]
- David R. Ambaras and Kate McDonald (eds), “Bodies and Structures 2.0. Deep-Mapping Modern East Asian History”, URL: https://bodiesandstructures.org/bodies-and-structures-2 (30.06.2025). [↩]
- Theodore Nelson: Philosophy of Hypertext, Tokyo 2002, 24. [↩]
- Jakob Krameritsch, “Die fünf Typen des historischen Erzählens – im Zeitalter digitaler Medien”, Zeithistorische Forschungen / Studies in Contemporary History 6, no. 3 (2009), 413-32. [↩]
- Christian Wachter, Geschichte digital schreiben. Hypertext als non-lineare Wissensrepräsentation in der Digital History. Bielefeld: transcript, 2021, DOI: https://doi.org/10.14361/9783839458013. [↩]
- Gunther R. Kress and Theo van Leeuwen, Reading Images. The Grammar of Visual Design. London: Routledge, 1996. [↩]
- Hartmut Stöckl, “Beyond Depicting. Language-Image-Links in the Service of Advertising”, AAA – Arbeiten aus Anglistik und Amerikanistik 34, no. 1 (2009), 3-28. [↩]
- Cf. Charles Hartshorne and Paul Weiss (eds), Collected Papers of Charles Sanders Peirce. Volume I. Principles of Philosophy. Cambridge (MA): Harvard University Press, 1960, 195; Charles Hartshorne and Paul Weiss (eds), Collected Papers of Charles Sanders Peirce. Volume V. Pragmatism and Pragmaticism. Cambridge (MA): Harvard University Press, 1960, 50. [↩]
- This idea recurs across multiple sections of Peirce’s Collected Papers. [↩]
- Karin Knorr Cetina, “‘Viskurse’ in der Physik. Konsensbildung und visuelle Darstellung,” in Mit dem Auge denken, ed. Bettina Heintz and Jörg Huber. Zürich/Wien/New York: Springer, 2001, 305-320. [↩]
OpenEdition suggests that you cite this post as follows:
Christian Wachter (September 9, 2025). Multimodal History – Convergence of Text, Image, Diagram and Virtual Space. Geschichtstheorie am Werk. Retrieved November 16, 2025 from https://doi.org/10.58079/14m4o

